My Load Test

Performance testers must switch between different technology platforms almost every time they join a new project. Here are a few things you should know so you don’t look like a complete n00b on your first SAP project.

Transactions and Sessions

Unless a restriction has been put in place, users may create multiple sessions from a single workstation. Using the menu, this can be done through System > Create Session or by left-clicking on the top right of the SAP client window and selecting the Create Session option.

This menu is also the only place a transaction may be stopped if a dialog step is taking a long time to complete. Ending a transaction that has stopped responding by using Windows Task Manager to kill your SAP client is a bad solution, and I have seen it leave zombie processes running in the system.

Most of your navigation probably won’t be done using the menu tree, but by entering a transaction code in the command field.

In the above screenshot I am jumping directly to the Data Browser, which has a transaction code of SE16. There are several other things we can do in the command field:

• /x is a shortcut for exiting the system
• A new session can be opened with /o[Transaction Code]. E.g. “/oSE16″ would open another session and start the Data Browser
• If you are already in a transaction and want to jump to another, you would use /n[TransactionCode]

Be aware of where you invoke a transaction in your LoadRunner script. If you are restarting a transaction every time you iterate through the Action section of your script, you may be creating an unrealistic usage pattern (unless your users really will restart the transaction every time they use it).

Extracting Data

If you need to automate anything within SAP, like small-scale data loads or data extractions through the GUI, it is easier to use QuickTest Pro (if you are licensed) rather than CATT/eCATT scripts, which are a big pain in the butt.

Regular cut and paste does not work for all of the SAPGUI widgets. Where this is the case, you will have to use Ctl-y to select text before you can copy the text to the clipboard.

If a data grid does not have an export button to allow you to export data to a local file in Excel format, you have another option. Choose System > List > Save > Local File, and choose your preferred file format.

If you have access to the Data Browser (SE16), you can extract data from the database tables directly. Because SAP want their software to be independent of the underlying database, they discourage the use of SQL, so you must query the table using their forms.

Extracting data from tables rapidly becomes painful if your data spans multiple tables, because you can’t do a join. The (hacky) solution to this is to either extract both tables to Excel and join there (small data sets only), or right click in the field in the second table that is the key and choose multiple selection from the context menu. This will allow you to run a query on the table to match all the records that have your key value.

Finding the correct database table can be tricky as most tables and fields have cryptic names (very short German abbreviations). The field and table behind an edit field in your application can be found by placing the cursor in the edit field and pressing F1 (help).

Click on the icon with Technical Information icon (with the hammer and spanner), and the technical details behind the field will be displayed.

If you are lucky, you will be given a table and field name, otherwise you will be given a Struct. This is a data structure in the ABAP code that may have pulled data from multiple tables (or wherever). You will also be able to see the field in the struct and the underlying field (data element) in the database. Double click on the data element and you will be taken to a data dictionary view. Press the Where-Used List button to see a list of all the tables that have this field. Double click on the field name under the correct table and you will be taken to the data dictionary view of the table. Confirm you have the correct table and then jump back to SE16 to construct your query.

End-user Performance

I’ve talked about this before, but I think that the transaction timer in the SAP client is quite a neat feature.

It measures how long each dialog step takes, so if you press a button and you have wait for a response from the server, the timer will run until the response is received. Note that if multiple dialog steps are executed when you press the button, the timer will measure each individually (causing the timer value to update with a new value for each dialog step), and you will finish with only the last value.

Note from the screenshot, you can also see the transaction code for the current transaction in this menu.

LoadRunner Scripting

SAPGUI is one of the easiest vuser types you will ever use. Read all the LoadRunner documentation and you shouldn’t have many problems. Be aware that you can only run a limited number of virtual users on each generator due to GDI resource limitations. The work-around for this is to open up multiple Terminal Services sessions, each of which has its own allocation of GDI objects. You may need to run fewer virtual users in each TSC session if your application is graphically rich.

Further Reading

• Make sure to read enough that you will understand the SAP architecture for your project, also any tuning guides are useful
• Read the relevant sections in the LoadRunner manual
• Search the Mercury Support Knowledge Base for SAP-related tips and tricks. Read Problem ID: 11907 – How-to and troubleshooting guide for SAP Vuser
• Read the slides from my Mercury World Australia presentation on Mercury Diagnostics for SAP
• Finally, I plan to make a list of SAP transaction codes that are useful to performance testers, so check back soon

If the application that you are performance testing will be operating over a WAN, it is really important to test whether the reduced bandwidth and higher latency of the network link will increase transaction response times to an unacceptable level.

If you are being thorough, you should also run some tests to determine whether your response times will be impacted by other network traffic on the WAN (and vice versa).

Let the following anecdotes about testing two different applications serve as examples:

The first application was to be used by staff who would be interacting directly with customers. There would be up to 10 concurrent users per site, each spending the majority of their time using this application. In addition to the network traffic from the application, there would be back-office traffic like web browsing, email, and software packages being deployed to workstations. The Project had some concerns about the impact of this traffic (particularly package deployment) on end-user response times for this business-critical application.

A leased line with the same specifications as the (future) Production WAN links was provisioned for the test environment. A LoadRunner agent was set up at the end of the WAN link to generate traffic equivalent to 10 concurrent users. The background network traffic was generated with NetIQ’s Chariot tool.

Chariot sends packets (with a typical packet size for each type of traffic to be emulated) at specified volumes between user-defined ports on each end-point, which sit at either end of the WAN link.

The network team spent several days determining the profile of the background traffic and creating a number of traffic scenarios in Chariot. The different traffic scenarios were run at the same time as a worst-case profile of heavy application usage. The response times from these scenarios were compared with end-user performance from a scenario with no background traffic (and, obviously, against the performance requirements).

The second application was to by used by staff working in a warehouse. There would only be 1 or 2 concurrent users per site, and the traffic would be low volume – mainly related to printing picking slips and labels. This was tested using one of the warehouse WAN links; using real Production infrastructure meant that the tests had to be run ridiculously early in the morning to ensure that they would not interfere with normal Business activities.

The testing was nowhere near as sophisticated as the first example. A user was located at the warehouse with a stopwatch to measure response times. It was not necessary to use LoadRunner as there would only be a small number of users at the site, and LoadRunner would not be able to measure the time it took for output to appear at the printers.

A baseline with no additional network traffic was run. Multiple samples were taken for each business process to provide a meaningful average response time that could be reported on.

To emulate network activity, I ran a batch file that repeatedly copied a zipped file across the WAN (the file was zipped in case any network components were trying to compress traffic). The network team, who were monitoring the link, reported that the link was at 40% utilisation. As file transfers had a lower QoS priority than application or printer traffic, no significant change in response time was expected.

The business processes were re-run, and results were recorded. As expected, there was no statistically significant difference in response times.

Running a second instance of my batch file increased the network link utilisation to 90%. This time response times were higher, but still within requirements.

Just for the sake of completeness, here is the batch file:

:: Copy file.zip from shared drive
:: to the local D:\TEMP directory.
:: Loop forever (or until Ctl-C).
:: Rename the copied file as file<iteration>.zip.
@echo off
SET /A i=1
:copy
echo copying file%i%...
copy /Y \\dws0368\Temp\file.zip D:\TEMP\file%i%.zip
SET /A i=%i+1
goto copy

As with any sociability test, you can put in a level of effort that is likely to reduce the risk of unexpected behaviour in Production to an appropriate level. You could spend a lot of time making your usage profile “perfect” and testing every scenario, but it saves valuable time to concentrate on a probable-worst-case and simplify you usage profile as much as is practical.

Of the two examples, one took a couple of hours to prepare and execute, the other took a few days and involved significant work from other groups. While I would prefer to do this testing the first way if possible, the quick-and-dirty second way is an example of “good enough” testing (and, with historical graphs of network usage, is almost as defensible as the first example).

Performance testers sometimes get hung up on tool-based testing. Admittedly, most of the really interesting problems appear when there are more than a few concurrent users on a multi-user system, but it is important to verify that an application performs acceptably when it is not under load too.

Even if response times seem fine during functional testing, the functional test team may not have thought to look for the effect of large datasets on response times. It’s not just a case of “do we need to put an index on a table somewhere”, but sometimes response times increase exponentially instead of linearly as the dataset increases.

This is especially noticeable with poorly written batch jobs. The job might take 1 minute to process 1,000 records, but it does not necessarily follow that processing 100,000 records will take 100 minutes.

For an example of an application that seems to have implemented Shlemiel the painter’s algorithm, take the Replace function in Notepad. Performing a find-and-replace on a small file is very fast, but performance quickly degrades as the file gets larger.

As you can see from the graph, the increase in response times is definitely not linear as the file size increases.

Using an even larger file containing 10,000 lines, the operation completes in 28 minutes. Compare this to the response times for another text editor such as TextPad – all files up to 10,000 lines were processed in less than 1 second.

If you would like to experiment with this feature, text files of different sizes are available here (155kB zipped).

No one is ever going to argue that Mercury don’t know how to throw a party. Mercury World Australia was a great opportunity to hear experts talk about the direction our industry is headed and ways that business can increase the value they get from technology but, more importantly, it was also an opportunity to network (or schmooze if you prefer) and enjoy their excellent catering.

While I definitely wasn’t going to miss the Gala dinner on Thursday, due to client commitments I only ended up seeing one presenter – Paul McLean from RPM Solutions and his talk on “Analysing System Behaviour During a Load Test”.

His main point was that we, as Performance Testers, should use the scientific method as a framework to help diagnose the root cause of a performance problem.

The scientific method is designed so that the results of a test are not biased by the results you are expecting, or the results that you want to see. Physicist and academic Richard Feynman summarised the scientific method as 1. Make a guess; and 2. See if you’re wrong, but it is more traditionally defined as an iterative process comprised of the following four steps…

1. Observation – accurately characterising the behaviours that were witnessed, making sure to differentiate carefully between cause and effect (correlation does not equal causation).
2. Hypothesis – when you formulate a hypothesis, you determine a possible reason for the behaviour your are witnessing.
3. Predict Behaviour – use the hypothesis you formulated in the previous step to predict the behaviour of the system. Paul used the example of a system problem that he believed was related to user concurrency. To determine if this was likely, he repeated the test with the same number of users, but half the transaction rate. If the number of concurrent users was the key factor, he would expect the problem to still manifest.
4. Test Prediction – devising an experiment to determine if your predictions are correct, and therefore if your underlying hypothesis is valid is demonstrated in the previous example. The results of Paul’s test would tell him whether his predictions were correct or not, and whether he should formulate an alternative hypothesis.

It’s always nice to hear a presentation where, even if you do not necessarily learn anything new, you agree with everything the presenter is saying. Paul even used a project we were both involved in as an illustrative example of applying the scientific method to isolate the root cause of a real-world problem.

Even if all the talks aren’t this good, I will be making every effort to attend the next Mercury World in Australia.

I just received an evaluation copy of the new version of Rational Performance Tester. IBM Rational have re-written the tool from scratch and based it on the open-source Eclipse framework. The tool has only just come out of beta testing and currently supports web protocols only. Everyone in the industry seems to be keen to know how it stacks up against the competition.

First off, my impressions of the old version…

Coming from a LoadRunner background, my opinions on the tool were a little mixed. Performance Tester was easy to learn and easy to use but it is definitely missing some features when compared to LoadRunner and, possibly because of my background, the way the tool did some things just felt wrong. It must also be noted that I have only used the web protocol. Other protocols may behave differently.

In my opinion, the downsides were

• It uses its own custom scripting language. Once you know how to program in one language, it’s easy enough to pick up another (as long as it doesn’t use a totally different paradigm, like functional programming), but it all feels a little unnecessary.
• Manual correlation is done using regular expressions. So sites like the Regular Expression Library are useful.
• The tool automatically correlates everything. This makes the scripts ugly. This is great if your script happens to work for all the data you are going to use with it. It is painful to debug if it doesn’t work sometimes. This is why a Rational tool expert will tell you it is easier to re-record a script than edit it. This is also why Rational users don’t generally put text checks in their pages.
• When you are running a test, you have nowhere near the visibility into what is going on that LoadRunner gives you.
• I found the way it handled chunked encoding really weird.
• Connection handling is not transparent to the user.

But on the plus-side…

• The automatic correlation works most of the time. This can be a big time saver.
• Their support is excellent.
• It is significantly cheaper than LoadRunner.
• The tool gives you the option of using a persistent cursor for your data tables. So if you had a test case that, for instance, deleted a user from a database, you would not have to restore the database or re-add users between test runs, you could just continue from the same point in your list of users. This is a feature that LoadRunner is missing.

I will post a review of the new version of IBM Rational Performance Tester once I have had a chance to play with it for a while.

I recently worked on a J2EE project where pauses due to garbage collection seemed to be causing some problems with response times.

As the project was using the 1.4.2 JRE, they had four different garbage collectors to choose from. A short series of performance tests was conducted to examine their behaviour.

Using the -verbose:gc JVM flag gave visibility into the behaviour of the garbage collector as it logged important information each time the garbage collector ran.

[GC 325407K->83000K(776768K), 0.2300771 secs]
[GC 325816K->83372K(776768K), 0.2454258 secs]
[Full GC 267628K->83769K(776768K), 1.8479984 secs]

Excel was used to create a graph of (roughly) heap used before, heap used after, total heap size, time taken, and whether the gc was a minor collection or a major one.

To save myself time, I wrote a short awk script to filter the logfile so that its contents could be graphed nicely in Excel.

I found the following article especially interesting when learning about the garbage collection options that were available for the version of Java that the project was using…

Tuning Garbage Collection with the 1.4.2 Java Virtual Machine

I am yet to work on a project where environmental issues haven’t impacted my schedule to some degree. But it seems that I only start to document the outages once they start to seriously impact my schedule.

This stops now.

From now on I will document every outage (no matter how trivial) before they start to have any impact. Once the cumulative downtime reaches more than a few hours, I will include a link to the outage document as part of my weekly (or daily) status report.

Here is a template for my Environment Availability Spreadsheet.

I have decided to release an early draft of this document so that others may provide feedback. Please let me know what you think.

Writing test cases for performance testing requires a different mindset to writing functional test cases. Fortunately it is not a difficult mental leap. This article should give you enough information to get you up and running.

First, lets set out some background and define some terms that are used in performance testing.

• Test case – a test case is the same as a use case or business process. Just as with a functional test case, it outlines test steps that should be performed, and the expected result for each step.
• Test script – a test script is a program created by a Performance tester that will perform all the steps in the test case.
• Virtual user – a virtual user generally runs a single test script. Virtual users do not run test scripts using the Graphical User Interface (like a functional test case that has been automated with tools like WinRunner, QuickTest, QARun or Rational Robot); they simulate a real user by sending the same network traffic as a real user would. A single workstation can run multiple virtual users. [Would a diagram be useful here?]
• Scenario – a performance test scenario is a description of how a set of test scripts will be run. It outlines how many times an hour they will be run, how many users will run each test script, and when each test script will be run. The aim of a scenario is to simulate real world usage of a system.

Writing a test case for performance testing is basically writing a simple Requirements Specification for a piece of software (the test script). Just as with any specification, it should be unambiguous and as complete as possible.

Every test case will contain the steps to be performed with the application and the expected result for each step. As a performance tester will generally not know the business processes that they will be automating, a test case should provide more detail than may be included in a functional test case intended for a tester familiar with the application.

It is important that the test case describes a single path through the application. Adding conditional branches to handle varying application responses, such as error messages, will greatly increase script development time and the time taken to verify that the test script functions as expected. If a test script encounters an error that it does not expect, it will usually just stop. If the Project Manager decides that test scripts should handle errors the same way a real user would, then information should be included on how to reproduce each error condition, and additional scripting time should be included in the project plan.

The main reason a user may be presented with a different flow through the application is the input data that is used. Each test case will be executed with a large amount of input data. Defining data requirements is a critical part of planning for a performance test, and is the most common area to get wrong on a first attempt. It is very easy to forget that certain inputs will present the user with different options.

The other important data issues to identify are any data dependencies and any potential problems with concurrency. Is it important that data is used in some business functions before they are used in others? And, will data modified by virtual users cause other virtual users to fail when they try to use the same data? The test tool can partition the data used by each virtual user if these requirements can be identified. It can be difficult for a performance tester to debug test script failures with little knowledge of the application, especially if the failures only occur when multiple virtual users are running at once.

One of the most important pieces of information a performance test is designed to discover is the response time of the system under test – both at the overall business function level and at the low level of individual steps in the test case, such as the time it takes for a search to return a result set. Any test cases provided to a performance tester should clearly define the start and end points for any transaction timings that should be included in the test results.

It is important to remember that the test script is only creating the network traffic that would normally be generated by the application under test. This means that any operations that happen only on the client do not get simulated and therefore do not get included in any transaction timing points. A good example would be a client application that runs on a user’s PC, and communicates with a server. Starting the client application takes 10 seconds and logging in takes 5 seconds but, since only the login is sending network traffic to the server, the transaction timing point will only measure 5 seconds.

Operations that only happen on the client, including the time users take to enter data or spend looking at the screen is simulated with user think time – an intentional delay that is inserted into the test script. If no think time is included, virtual users will execute the steps of the test case as fast as they can, resulting in greater (and unrealistic) load on the system under test. Depending on the sophistication of the performance test tool, the user think time may be automatically excluded from the transaction timing points. Think times are generally inserted outside of any transaction timing points anyhow.

While a functional test case will be run once from start to finish, a performance test case will be run many times (iterated) by the same virtual user in a single scenario. Information on how the test steps will be iterated should be included in the test case. For example, if a test case involves a user logging in and performing a search, and the entire test case is iterated by the virtual user; then a test scenario may be generating too many logins if the real users generally stay logged into the application. [Is there a way to break up this sentence?] A more realistic test case may have the virtual user log in once and then keep doing the same action for as long as the virtual user is run.

When a script is iterated, consideration should be given to the non-obvious details of how it is iterated. A good example would be a test script simulating users using an Internet search engine. When the test script is iterated, simulating a new search operation, should the virtual user establish a new network connection and empty their cache or should every iteration simulate the same user conducting another search?

As all performance test tools have different default behaviours, a good performance tester should clarify this type of detail with business and technical experts. Some performance test tools make these details easier to change than others. If it is not practical to emulate all the attributes of the expected system traffic with a particular tool, then it should be noted as a limitation of the performance test effort, and a technical expert should assess the likely impact.

Hopefully this article has provided some insight into the extra considerations that must be given when writing a performance test case, rather than a functional test case. As with any software specification, a performance test case may need to be refined as questions are raised by the performance tester.

An example test case:

Here is a list of things I would like to write a short article on. They are either topics I am interested in or something I would like to be able to show to a client as background information that may save me from having to explain concepts related to performance testing.

• Estimating scripting time for performance testing
• How to write a Detailed Test Plan for Performance Testing
• How to write a Test Summary Report
• Maths for Performance testers (statistical significance, sample sizes, significant figures, queue theory)
• Preparing for a Performance Test – what the Business can do before the consultant arrives to save some time.
• A quick guide to Windows performance monitoring with Perfmon
• Staff utilisation rates. Different ways to calculate them, and common utilisation targets for different companies.
• How to write a Performance Test Case [done!]
• LoadRunner HTML mode vs. URL mode. Understand the differences or pay the price.
• Glossary of terms for Performance Testing (designed to be referenced in Performance Test documentation
• Introduction to LoadRunner scripting with CORBA (my current project)

If an idea grabs your imagination feel free to run with it – you’ll save me some time, and I will be happy to link to it.