My Load Test

I have made a list of the top ten things load testers frequently fail to do that make me feel like smiting them.

1. Thou shalt know how thy test tool works.
   The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic. Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. Thou shalt gather realistic usage data.
   Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. Thou shalt have testable requirements.
   Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects. This shouldn’t stop you from trying to gather the requirements you need for your tests. The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough. Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. Thou shalt write a test plan.
   Even if you already know what you’re going to be doing, other people would probably like to know too – they might even be able to help; besides, a signed-off test plan has saved many a tester from the wrath of project management.
5. Thou shalt test for the worst case.
   Don’t test with transactions from an average day, test for the busiest day your business has ever had. Add a margin for growth. Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. Thou shalt monitor your test environment infrastructure.
   I feel that I have to spell it out, because I still see people who don’t do this. Monitoring your servers allows you to more easily figure out where the problem is. You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%” When I say “monitor your servers”, this includes your load generators.
7. Thou shalt enforce change control on your environment.
   The final thing you tested should be what is deployed into Production – same application version, same system configuration. It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing. Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. Thou shalt use a defect tracking tool.
   An untracked defect is a little like a tree that fall in the forest when no-one is around – no-one cares. Raising defects lets everyone know there is a problem (not just the people who should be working to fix it). It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. Thou shalt rule out thy own errors before raising a defect.
   “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects. If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts. Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system). It always helps if you have followed commandment #1 Thou shalt know how thy test tool works.
10. Thou shalt pass on your knowledge.
    Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting. Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC. Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them. Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.

This is a quick walkthrough of a LoadRunner script I created to load test a “web” service that communicates with SOAP messages sent over JMS.

Note that this script supports the queuing/point-to-point model of JMS (rather than the publish and subscribe model), works with JMS version 1.1 rather than 1.02b (as it uses the BytesMessage.getBodyLength() method among others), and has been tested with Java 1.5 using the Tibco implementation of JMS.

The LoadRunner script can be downloaded here, the source code is available here, and a mirror of the JMS 1.1 specification can be downloaded here. People using Tibco’s JMS implementation can download their Javadocs here (10MB). Feedback is appreciated.

From looking at the script, you may wonder why I chose to write my own script using the Java vuser type instead of using the Web Services vuser (version 8.1, Feature Pack 4 supports web services using JMS as well as HTTP), but that is a story for some other time.

Anyway…for the tiny number of people that I haven’t lost already, here’s how it all works…

At a high level, we are trying to put a SOAP message on a queue (MyLoadTest.Warehouse.Request), receive the corresponding response from a different queue (MyLoadTest.Warehouse.Response), and check that the response message matches our expected message. Download my source code and follow along.

On with the detail…

The javax.jms.* classes provide interfaces only, other vendors have provided implementations of these interfaces (I am using Tibco’s implementation – found in com.tibco.tibjms.*). This should mean that any JMS code you write will be portable between implementations; whether this is true or not is something I haven’t tested.

Initially we perform a JNDI lookup to find a ConnectionFactory and get the details for the queues we will be using.

We then create a Connection and a Session, and then create the MessageProducer (using the request queue details) that we will be using to send our message to the request queue.

We create either BytesMessage or a TextMessage (MapMessage, ObjectMessage and StreamMessage have not been implemented in the script), and set the JMS header properties. The important JMS headers are the JMSDestination (the request queue), and the JMSReplyTo (the response queue). We can also set Message properties, which are simple name/value pairs. We then set the Message body, using a string constant containing the SOAP XML message.

We only want to measure system response times, so we only put our LoadRunner transaction timing points around the code that sends and receives the message.

Sending the message is as simple as calling myMessageProducer.send(myMessage).

In the following extract from the Tibco EMS logs, you can see the JMS headers, the Message properties and the Message body (shown here with a size only). The JMS headers show the request and response queues along with a unique msgID.

2007-02-26 13:20:07 [MSG:73422]
received from user='anonymous':
connID=1669
prodID=16906
msgID='ID:SVP-EMS-SERVER.391E45DA4ADA6C8:3083'
Time=1172456256000
mode=PERSISTENT
queue='MyLoadTest.Warehouse.Request'
msg=BytesMessage=
{
  Header=
  {
    JMSDestination={QUEUE:'MyLoadTest.Warehouse.Request'}
    JMSReplyTo={QUEUE:'MyLoadTest.Warehouse.Response'}
    JMSDeliveryMode={PERSISTENT}
    JMSPriority={4}
    JMSMessageID={ID:SVP-EMS-SERVER.391E45DA4ADA6C8:3083}
    JMSType={BytesMessage} JMSTimestamp={1172456256000}
  }
  Properties=
  {
    "SOAPJMS_requestIRI"={string:''}
    "SOAPJMS_targetService"={string:''}
    "SOAPJMS_soapAction"={string:''}
    "SOAPJMS_isFault"={string:'false'}
    "SOAPJMS_contentType"={string:''}
    "SOAPJMS_soapMEP"={string:'http://www.w3.org/2003/05/soap/mep/request-response/'}
    "SOAPJMS_bindingVersion"={string:''}
  }
  Body=
  {
    byte[]:1126 bytes
  }
}

To receive a Message, we must create a MessageConsumer (using the response queue). Rather than simply receiving the messages from the queue in the order they arrived (FIFO – which would mean that we could be receiving the wrong response message when we are running load against the system), we can use a message selector to make sure that we receive the message that we want from the queue.

In the log above, we saw that each JMS message has a unique JMSMessageID. When a response is sent by the server, the response message has a JMSCorrelationID that matches the JMSMessageID of the message it is replying to. A message selector is defined in the same way as the WHERE clause of an SQL statement, so our selector would be
"JMSCorrelationID = '" + myBytesMessage.getMessageID() + "'".
Note that the unique message ID is only set when the message is sent so, if you are using a selector, you must create your MessageConsumer *after* you have called myMessageProducer.send(myMessage). A selector can use any of the JMS headers and Message properties, but cannot use the Message body.

Using the MessageConsumer, receiving the correct message from the queue is done by simply calling myMessageConsumer.receive(timeout). The script will wait to receive the message until it reaches the timeout value. If no message is received before the timeout, the response message will be null. Remember to close the MessageConsumer, or you will leak and end your test with thousands of open receivers.

Here is the Tibco EMS log entry for the response message. You can see that the message has a different JMSMessageID, and has a JMSCorrelationID with the message ID of the request it is replying to.

2007-02-26 13:20:09 [MSG:73425]
received from user='anonymous':
connID=1714
prodID=19956
msgID='ID:SVP-EMS-SERVER.391E45DA4ADA6F5:7835'
Time=1172456409014
mode=PERSISTENT
queue='MyLoadTest.Warehouse.Response'
msg=BytesMessage=
{
  Header=
  {
    JMSDestination={QUEUE:'MyLoadTest.Warehouse.Response'}
    JMSDeliveryMode={PERSISTENT}
    JMSPriority={4}
    JMSMessageID={ID:SVP-EMS-SERVER.391E45DA4ADA6F5:7835}
    JMSCorrelationID={ID:SVP-EMS-SERVER.391E45DA4ADA6C8:3083}
    JMSTimestamp={1172456409014}
  }
  Properties=
  {
    "SOAPJMS_requestIRI"={string:''}
    "SOAPJMS_contentType"={string:'text/xml'}
    "SOAPJMS_soapMEP"={string:'http://www.w3.org/2003/05/soap/mep/request-response/'}
    "SOAPJMS_bindingVersion"={string:''}
  }
  Body=
  {
    byte[]:681 bytes
  }
}

Extracting contents of the Message body from the JMS Message is done by calling myMessage.getText() if it is a TextMessage (if it is a BytesMessage, it is a bit trickier – see the attached script).

And finally comparing the received SOAP message to the expected message is just done using String.equals().

Remember to close the Session and Connection at the end of the script.

The exercise of modifying the script to send many different messages to different queues (which would mean you only have to maintain one script instead of one script per SOAP operation) is left as an exercise for the reader…

A Java thread dump is a way of finding out what every thread in the JVM is doing at a particular point in time. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck.

You can generate a thread dump under Unix/Linux by running kill -QUIT <pid>, and under Windows by hitting Ctl + Break.

A great example of where this would be useful is the well-known Dining Philosophers deadlocking problem. Taking example code from Concurrency: State Models & Java Programs, we can cause a deadlock situation and then create a thread dump.

In the example below (shown using tda), we can see that the 5 Philosopher threads each have a lock on a Fork object and are each waiting to obtain a lock on a second Fork object before they can eat. Unfortunately this never happens and all the philosophers starve.

Download the thread dump from here (7 KB).

Note that not all hangs are going to be due to deadlocks, and there are many tools (including Eclipse) that will help you analyse thread dumps.

Yet another application to add to your performance testing toolkit is Wireshark (previously called Ethereal). A packet sniffer like Wireshark will let you see your network traffic at all the different protocol layers.

A naturally curious person might use it to investigate why the lights on their router started flashing all the time just after they installed that really useful piece of software that let them change their mouse cursor.

As a performance tester, I use it when I’m trying to figure out what the application I am trying to record with LoadRunner is sending and receiving, and the VuGen log doesn’t contain enough information (and I can’t get it by recording in Winsock mode). 99% of the time this isn’t necessary but occasionally, when the application uses some sort of weird (or custom) protocol, it helps to show what is going on.

Recently I was running a LoadRunner training session and was asked to take a look at the application the company was developing. It was a web-based application that had an embedded ActiveX object that also communicated with the server. Recording just HTTP did not record any traffic to or from the object. Recording with the Web/Winsocket Dual Protocol vuser type recorded a message from the object to the server (as well as all the HTTP traffic from the web page), but no communication from the server to the object; nothing appeared in the recording log either.

To double check that the object really was getting information from the server, rather than values being passed from the HTML, a network trace was run. This showed that there really was a message being sent to the ActiveX object, and that for some reason VuGen was not recording it correctly. In this situation, it is easy to give Mercury Support your script and your tracefile and they will pass it on to the R&D team who will usually give you a work-around or write you a patch in a couple of days.

So, anyway, sometimes you will come across an application that talks to the server in some kind of unconventional way; and sometimes a packet sniffer helps you figure out what is going on. Wireshark is your best option because it is free and it has a comparable feature-level with expensive proprietary tools like Sniffer Pro.

Download it from the Wireshark website.

Some other tools that I have talked about previously:

• Free WAN emulator (Dummynet on FreeBSD)
• Unix utilities on Windows
• WebScarab (a security test tool for web applications)
• FreeQueryBuilder (a better SQL tool; now called SQLeonado)
• LogParser (a website log analyser)

A LoadRunner feature that has made my life a lot easier has been ContentCheck rules, which are available in the script runtime settings. If you are using a web-based vuser type, you can configure your LoadRunner script to search through all returned pages for strings that match known error messages.

Using web_reg_find functions is fine, but when you get an error LoadRunner reports it as “failed to find text” instead of something more descriptive.

I will always create rules for any error messages I find during scripting and, if I receive an error while running a scenario, I will add the error message from the snapshot in the scenario results directory (the snapshot on error feature is very useful).

All this is pretty obvious if you have taken the time to explore LoadRunner’s features or you have attended a Mercury training session, but I recommend taking things a step further.

• Ask your developers for a list of all the error messages that the application can throw. This should be easy for them to provide if the application is well designed and stores all the message in some kind of message repository instead of sprinkling them throughout the source code.
• Include error message for functional errors that you are likely to encounter. Creating a rule for “incorrect username or password” may save someone 20 minutes of investigation when they first run the script after the database has been refreshed.

If you prefer to have error message you are checking for in the script (where you can add comments to them) instead of the runtime settings, you can use the web_global_verification function instead. The only difference between the two is the error message that LoadRunner will include in its log:

Action.c(737): Error -26368: “Text=A runtime error occurred” found for web_global_verification (“ARuntimeErrorOccurred”) (count=1), Snapshot Info [MSH 0 21]

…compared to:

Action.c(737): Error -26372: ContentCheck Rule “ARuntimeErrorOccurred” in Application “Webshop” triggered. Text “A runtime error occurred” matched (count=1), Snapshot Info [MSH 0 21]

And finally, ContentCheck rules can be easily exported and shared between scripts, which can be a nice time-saver.

Mercury has just announced that they will no longer be re-selling the Shunra WAN Emulator. This means that I can let you in on a little secret – you can get most of the WAN emulation features for free by using a simple open-source program.

As performance testers, we know that an application operating over the network will perform poorly if there is not enough bandwidth available. We also know that response times for some applications are more affected by latency than others no matter how much bandwidth you have (eg/ an interactive multiplayer game like Quake is playable over a 56k modem, but completely useless over a satellite link that has 10 times the bandwidth). Sidenote: a good job interview question for a performance tester might be to explain the difference between latency and bandwidth, and its impact on application performance.

Load testing with bandwidth limitations is easy; LoadRunner gives you this feature for free. Latency is harder as it requires either a real WAN link, or something to introduce an artificial delay. An artificial delay can be introduced by a black box that you plug into your network (like those offered by Anue, East Coast DataCom, and Apposite Technologies) or by a piece of software like the Shunra WAN Emulator.

The free alternative to Shunra’s software is Dummynet, which was created by an Italian academic researcher.

Unfortunately Dummynet only runs under FreeBSD, but a tiny version of FreeBSD with Dummynet that fits on a bootable floppy disk is available for download. Personally, I haven’t seen a floppy disk for years and I don’t quite trust that FreeBSD (let alone a tiny version of FreeBSD) will support the variety of hardware it will encounter.

My preferred solution is to install FreeBSD as a guest operating system inside VMware. The hardware in the virtual machine is virtualised, so you don’t have to worry about driver support, and it is easy to distribute a VMware image between computers. The only other thing you will need to do is to set up a second network card in your computer and add it to the virtual machine.

The good thing about this solution is that it makes it easy to demonstrate latency and bandwidth-related performance problems manually, rather than expecting people to just accept your tool’s measurements. The only tricky part may be getting permission to plug your laptop into the network (or install the software) at a client site.

I just found something interesting on Google Video; it is a 1 hour Google TechTalk presentation by Goranka Bjedov about Using Open Source Tools for Performance Testing.

Goranka has some interesting things to say. She makes the point that there is really no standard terminology in performance testing circles, and goes on to prove this by giving her own definitions of performance, stress, load, scalability, and reliability testing. As an example of reliability testing she notes that “typically, when I was at AT&T, we would run for about a month at a time after everything was done just to find out that the system can actually stand up and can work fine with the load for a prolonged period of time.” In my testing circles, we would call that a soak test, but I would have been interested to hear more about the types of systems she was testing at AT&T.

The main body of her talk is about the different tools available for load and performance testing. These can be broken down into in-house, vendor and open-source tools.

Google has 55 in-house load and performance testing tools that have been developed by different groups for testing different Google products. These are very expensive to maintain and may only be used in-house, which makes any benchmarks impossible to verify by a third-party. Goranka says “Before you decide to develop your own, please take a look at what is out there…”

Goranka slams vendor tools (like LoadRunner, SilkPerformer and WebLOAD) for being overly expensive and using proprietry scripting languages. Personally, I have always thought that it was pointless having to learn another language just to use a load testing tool. Unfortunately she uses LoadRunner’s scripting language as an example “it’s C, minus the pointers”, and is incorrect – unlike many other tools, Mercury uses standard C (and Java and VBScript).

Her recommended solution is the open source tools – “five years ago they just weren’t there, but today they are.” Her personal preference is JMeter, but also recommends OpenSTA and The Grinder. Open-source tools have the advantage of being a good price, and having source code available; she also makes the point that they use standard programming languages for scripting (although this is incorrect when talking about OpenSTA).

The disadvantages of open-source tools are that they have a steep learning curve and do not support many protocols. “The vendor tools support far more protocols than the open-source tools, but as long as you are staying in the web space, and your looking at HTTP/S, IMAP and POP3, the open-source tools are pretty good”.

Goranka does not say that the open-source tools are free because it is occasionally necessary to write code to extend their features. “Free software is free in the sense that a puppy is free.” Features that Google engineers have written for JMeter have been added back to the main code tree by the people maintaining the project, meaning that Google is at least saved the cost of maintaining their forked code.

She uses JMeter for testing web-based applications through the GUI, uses The Grinder for API-based testing, and does not use OpenSTA because it only works on Windows.

Other points during the presentation:

• You should use the same monitoring for Load testing that you use for Production monitoring (so you don’t have to account for the differences in load that a different monitoring system will put on the system).
• If you are running Unix-based systems, don’t sustain CPU above 80%.
• Google tracks a summary of every performance test in a central database. The database also contains information on every piece of software that is installed on the machines in the test environment.
• If I am unfamiliar with the system, I don’t trust it. One of the things that I have realised is that
  A) the system will fail in the place where they tell me that nothing could go wrong.
  B) developers are totally delusional about their own software, and frequently they will just forget about things that they’ve done two weeks ago.
• I run every test 5 times. I want to see that I have some sort of statistical consistency.
• Performance testing should not be used as a tool to find memory leaks; but it can.
• Performance testing without monitoring? Don’t bother. Why waste your time?
• If you are going to do any performance testing, make sure that database sizes are somewhat realistic. They don’t have to be exactly the same, but they have to be the same order of magnitude otherwise the results you are getting are completely off.
• Execute a stress test. Find how your system is failing. Find where it is failing. Do find out how the system handles overload. There are no good defence mechanisms against people out there, and you can’t predict sudden popularity (eg/ Google Earth).
• Start a test after a decent warm-up period. Don’t start 100000 users all at once.
• Quite often people don’t know about everything that is running on a complex system. Maybe there is a low priority process that is running with high priority. This can usually be fixed by niceing the process down. Quite often there are debug things that are still running also.
• Monitor the machines that are collecting the monitoring data and the load generators (not just the system under test).
• Performance Testing and QA is about risk analysis. If I believe it is high risk, I want to take a look at it.
• When I am doing performance testing, the first thing I try to do is eliminate the network. I want to simplify my problem. I am interested in the machines, and my hope is that the network provided will handle everything I need. Once everything is profiled and understood, we will do some tests that include the network. If you can, put everything on the same subnet and same switch. It will make you a much happier performance tester in the first pass. Debugging networking problems is (not) fun.
• (When talking about testing on smaller sized systems than Production). You can’t test on a 386. Extrapolation will kill you. You will run out of some resource that you never expected, and you can’t predict this ahead of time. For final validation, you really want to get some time on the Production hardware before it goes live. If the system is not being used for Production, it should not be that hard to get hold of it for a week or a weekend.
• Find more open-source performance tools at opensourcetesting.org

There is another summary of her talk available on Robert Baillie’s blog.

You might also want to have a look at Becoming a Software Testing Expert; a 1 hour presentation delivered by software testing expert, and author of Lessons Learned in Software Testing, James Bach on June 13, 2006. His presentation is available for download from his website.

As part of the LoadRunner 8.1 Hands-On Lab training, there is an exercise to enhance a VuGen script that books a flight using the Mercury Tours application.

Company “A” wants the script to generate 10 digit credit card numbers instead of supplying them from an external source on every iteration and for an unlimited number of iterations.

How do you make the script generate credit card numbers for every iteration?

As the Mercury Tours application does not check if the credit card numbers conform to a valid format, the answer to this question is just to replace the file-based credit card parameter with a random parameter using a number format of %010lu.

The purpose of Mercury’s training is to cover as much of the basic concepts as possible in a short space of time, so they can’t make the exercises too hard; but I don’t like to leave people with the incorrect idea that just because they’ve finished some training they are totally prepared for the level of difficulty they will encounter in the real world.

I have created an exercise where you will need to submit different credit card numbers, but I have made it more difficult as they must be valid credit card numbers.

Start the Credit Card Validation Exercise here.

Some quick notes:

• The exercise is quite simple. If you work intelligently (and do some research), it should not take long.
• Make sure that you really do generate the cards at runtime. Finding 60 different card numbers on the web, trying card numbers at random until you find ones that work, and generating card numbers outside of LoadRunner all miss the point of the scripting exercise.
• Do not run load against my server. Don’t run more than 1 virtual user at a time and be kind with your thinktime and pacing settings.
• Please report functional defects to me via email: stuart at myloadtest dot com
• Once some people have emailed me their solution scripts, I will post some comments about my preferred solution.

Here are some credit card numbers to get you started:

• 30100000000002 (Diners)
• 30100000000010 (Diners)
• 4000000000000002 (VISA)
• 4000000000000010 (VISA)
• 5400000000000005 (MasterCard)
• 5400000000000013 (MasterCard)
• 340000000000009 (American Express)
• 340000000000017 (American Express)

Good luck!

Last week I posted a question asking how I could recover or change the password for a VMware guest operating system (Windows 2000) that I had forgotten the password for. After receiving no useful suggestions, this week I allocated some time to solving the problem.

Windows password recovery tools usually consist of a bootable CD image containing a version of Linux that will overwrite the NT password with a known value or will extract the hashed password from the filesystem.

To boot your virtual machine from a CD, you must change the boot order in the virtual machine’s BIOS. Press F2 while the VM is starting up to access the BIOS.

I used the free software available from Windows XP Login Recovery. This is good because it does not try to write to your file system, it just retrieves the hashed password value.

Once you have the password hash, you enter it in a form on their website and they look up the hash value in their database and give you a password that matches the hash. Note that even though they ask for your email address the password is displayed on a web page rather than being sent to your inbox.

After all that effort, I discovered that my password was blank.

The Mercury Certified Instructor qualification is not about having mastery over the Mercury toolset (although if you are sitting for the CI exam, you probably have this anyway); it is about ensuring that you have the ability to effectively deliver training material to a classroom of students.

Officially, Mercury training must be delivered by someone who has a CI qualification and the appropriate CPC qualification for the tool that they are delivering the training for. In practice, there aren’t enough Certified Instructors yet, so this is not enforced.

After wanting to do it for a while, I completed the Mercury Certified Instructor training and passed the exam in December 2005 (making me the only non-Mercury employee technically allowed to deliver the LoadRunner training in Australia).

The course is run by the instructor according to the same teaching theories that are taught in the course (which I am rather embarrassed to admit I didn’t fully realise until the afternoon of the first day). The first two days run through all the theory, and the third day is dedicated to the final exam – presenting a module of Mercury training in a way that will satisfy all the grading criteria.

Even though I had presented the training module before, presenting it while remembering to “create a non-threatening environment”, “address multiple learning modes”, and “exhibit energy and enthusiasm” was like juggling and spinning plates at the same time.

The final exam was one of the hardest I have ever done, despite my previous teaching experience delivering Mercury training and as a class tutor for 1st year university C++ and Digital Design subjects.

The only people in the class who managed to make it seem effortless were a former teacher and a Mercury employee who has run hundreds of Mercury training courses.

I highly recommend this course to anyone who is interested in teaching others how to use the Mercury toolset, although the concepts can easily be applied to any training situation.