My Load Test

Yet another application to add to your performance testing toolkit is Wireshark (previously called Ethereal). A packet sniffer like Wireshark will let you see your network traffic at all the different protocol layers.

A naturally curious person might use it to investigate why the lights on their router started flashing all the time just after they installed that really useful piece of software that let them change their mouse cursor.

As a performance tester, I use it when I’m trying to figure out what the application I am trying to record with LoadRunner is sending and receiving, and the VuGen log doesn’t contain enough information (and I can’t get it by recording in Winsock mode). 99% of the time this isn’t necessary but occasionally, when the application uses some sort of weird (or custom) protocol, it helps to show what is going on.

Recently I was running a LoadRunner training session and was asked to take a look at the application the company was developing. It was a web-based application that had an embedded ActiveX object that also communicated with the server. Recording just HTTP did not record any traffic to or from the object. Recording with the Web/Winsocket Dual Protocol vuser type recorded a message from the object to the server (as well as all the HTTP traffic from the web page), but no communication from the server to the object; nothing appeared in the recording log either.

To double check that the object really was getting information from the server, rather than values being passed from the HTML, a network trace was run. This showed that there really was a message being sent to the ActiveX object, and that for some reason VuGen was not recording it correctly. In this situation, it is easy to give Mercury Support your script and your tracefile and they will pass it on to the R&D team who will usually give you a work-around or write you a patch in a couple of days.

So, anyway, sometimes you will come across an application that talks to the server in some kind of unconventional way; and sometimes a packet sniffer helps you figure out what is going on. Wireshark is your best option because it is free and it has a comparable feature-level with expensive proprietary tools like Sniffer Pro.

Download it from the Wireshark website.

Some other tools that I have talked about previously:

• Free WAN emulator (Dummynet on FreeBSD)
• Unix utilities on Windows
• WebScarab (a security test tool for web applications)
• FreeQueryBuilder (a better SQL tool; now called SQLeonado)
• LogParser (a website log analyser)

Mercury has just announced that they will no longer be re-selling the Shunra WAN Emulator. This means that I can let you in on a little secret – you can get most of the WAN emulation features for free by using a simple open-source program.

As performance testers, we know that an application operating over the network will perform poorly if there is not enough bandwidth available. We also know that response times for some applications are more affected by latency than others no matter how much bandwidth you have (eg/ an interactive multiplayer game like Quake is playable over a 56k modem, but completely useless over a satellite link that has 10 times the bandwidth). Sidenote: a good job interview question for a performance tester might be to explain the difference between latency and bandwidth, and its impact on application performance.

Load testing with bandwidth limitations is easy; LoadRunner gives you this feature for free. Latency is harder as it requires either a real WAN link, or something to introduce an artificial delay. An artificial delay can be introduced by a black box that you plug into your network (like those offered by Anue, East Coast DataCom, and Apposite Technologies) or by a piece of software like the Shunra WAN Emulator.

The free alternative to Shunra’s software is Dummynet, which was created by an Italian academic researcher.

Unfortunately Dummynet only runs under FreeBSD, but a tiny version of FreeBSD with Dummynet that fits on a bootable floppy disk is available for download. Personally, I haven’t seen a floppy disk for years and I don’t quite trust that FreeBSD (let alone a tiny version of FreeBSD) will support the variety of hardware it will encounter.

My preferred solution is to install FreeBSD as a guest operating system inside VMware. The hardware in the virtual machine is virtualised, so you don’t have to worry about driver support, and it is easy to distribute a VMware image between computers. The only other thing you will need to do is to set up a second network card in your computer and add it to the virtual machine.

The good thing about this solution is that it makes it easy to demonstrate latency and bandwidth-related performance problems manually, rather than expecting people to just accept your tool’s measurements. The only tricky part may be getting permission to plug your laptop into the network (or install the software) at a client site.

It’s always nice to base your load test on real-world usage patterns rather than someone’s “best guess”. If the system is used internally, Business Analysts are usually quite good at providing this information, but if the system is used by external users (like most high-traffic web applications), then they can typically only tell you “big picture” information; e.g. they know how many orders are created, but not how many searches are made per hour.

Basic web log analysis tools provide low-level technical detail that isn’t very useful – hits per hour. To be useful for load testing, we need to translate the data in the logs into end-user business processes and their respective volumes during a peak hour.

Typically this analysis effort is worthwhile when:

• You are testing a web application that is a replacement for an existing web application, and can therefore be assumed to have a similar usage profile.
• Your web application has gone live, but you want to validate the “best guess” usage profile that was used for load testing against actual usage patterns. If you tested with less load or used a significantly different usage profile, you might want to run some more tests to make sure that you won’t see problems in the future. You will also be able to use this new profile for future regression tests.

An e-commerce website may have a number of common functions – register, login, update details, search for product, browse catalogue, add to basket, checkout, logout, etc. Using an analysis tool such as LogParser (which I have discussed before), we can map the page requests in the log file onto these functions.

If you are in a hurry to get started, the steps are as follows:

1. Get a copy of the log files for a peak day.
2. Extract the top page requests from the file.
3. Determine which page requests map to which business processes.
4. Query the log file for the relevant page request that identifies each business process. Pick the busiest hour.

Everyone else, read on…

Hopefully your system administrators are keeping the web server logs for at least a week. Log files will probably be split into a separate file for each day. Ask for the largest log file. Make sure you get the log files for the same day from each web server.

Run a LogParser query to extract all the URLs from the log and sort them by popularity. This gives a good overview of system usage. We can safely exclude any static content (images, stylesheets, JavaScript files) and any errors. The size of the output file can be substantially reduced by excluding pages with a small number of hits (we don’t care about any pages that are low-volume anyway).

SELECT
   COUNT(cs-uri-stem) as Hits,
   cs-uri-stem AS URL
FROM
   C:\TEMP\logs\ex060910_*.log
WHERE
   sc-status <> 404 AND
   EXTRACT_EXTENSION(cs-uri-stem) NOT LIKE 'jpg' AND
   EXTRACT_EXTENSION(cs-uri-stem) NOT LIKE 'gif' AND
   EXTRACT_EXTENSION(cs-uri-stem) NOT LIKE 'css' AND
   EXTRACT_EXTENSION(cs-uri-stem) NOT LIKE 'ico' AND
   EXTRACT_EXTENSION(cs-uri-stem) NOT LIKE 'js'
GROUP BY
   URL
HAVING
   Hits >=5
ORDER BY
   Hits DESC

The overview will generally show that the majority of the requests to the web application are hitting a small number of pages (80/20 rule).

Next, record the key business processes (that you know of) with VuGen using URL mode. This is a simple way of creating a list of all the URLs that each business process covers. Try to pick a URL that uniquely identifies each business process. Take note of the following attributes that are included in the web server log files:

• cs-uri-stem – usually the filename on the webserver, like “/search.php”
• cs-uri-query – if the URL has any arguments that are passed in after the filename, these are included here; like “?query=foo&lang=en”
• cs(Referer) – the complete URL of the previous page
• cs-method – the HTTP method of the request, usually GET or POST.

Check that there aren’t any popular requests in your overview list that aren’t covered by the business processes that you have recorded.

Finally, write a query using the above information to count the number of times a given business process was run during your peak hour. E.g.

SELECT
  TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date, time), 3600)) AS TimeStamp,
   COUNT(cs-uri-stem) as Hits
FROM
   C:\TEMP\logs\ex060910_*.log
WHERE
  cs-uri-stem = '/addToBasket.php' AND
  cs-uri-query LIKE 'item=%' AND
  cs-method = 'GET' AND
  cs(Referer) = 'https://www.store.com/display.php?item=%'
GROUP BY
   TimeStamp
ORDER BY
   TimeStamp

…which gives the following output (in CSV format). Note that the original timestamps were in UTC+0, which was converted to Australian Eastern Standard Time (UTC+10).

TimeStamp,Hits
2006-09-10 10:00:00,933
2006-09-10 11:00:00,935
2006-09-10 12:00:00,761
2006-09-10 13:00:00,655
2006-09-10 14:00:00,705
2006-09-10 15:00:00,680
2006-09-10 16:00:00,565
2006-09-10 17:00:00,231
2006-09-10 18:00:00,105
2006-09-10 19:00:00,66
2006-09-10 20:00:00,60
2006-09-10 21:00:00,65
2006-09-10 22:00:00,98
2006-09-10 23:00:00,36
2006-09-11 00:00:00,26
2006-09-11 01:00:00,15
2006-09-11 02:00:00,16
2006-09-11 03:00:00,17
2006-09-11 04:00:00,23
2006-09-11 05:00:00,24
2006-09-11 06:00:00,20
2006-09-11 07:00:00,43
2006-09-11 08:00:00,182
2006-09-11 09:00:00,398

From this, we can easily see that the peak hour for this business process is between 11 and 12, and we can feed the transaction volumes into the usage profile that will be used for load testing. Rinse and repeat for each business process.

Frequently moving between companies means that I sometimes find myself jumping from using Toad one week to Microsoft Query (yuck!) the next.

Now that I have had the chance to use it on a couple of different projects, I think that FreeQueryBuilder is a pretty good replacement for most of the SQL tools that I have had to use; and it certainly saves the effort of learning a new interface every time I move projects.

The best part of it is that it is shipped as a jar file, so you don’t have to install anything to use it – allowing you to bypass the week-long delay you are likely to experience at large companies waiting for the desktop support team to install software you need to do your job with.

Off the top of my head, the other things that make it a neat tool are:

• It works with all the popular databases
• Shows relationships between tables
• Makes it easy to search through the data dictionary for table or field names
• Allows you to write queries by clicking buttons instead of typing, but gives you a text editor option too

Give it a go!

Update: The FreeQueryBuilder project has been discontinued. The software is now known as SQLeonado, and has some additional features.

Sometimes when creating performance testing scripts, you will stumble upon vulnerabilities in the application under test.

Maybe you will miss a value you should have correlated before sending to the server, and when you go back and check your script you will find that (hypothetically) the web store is allowing you to purchase every item in their inventory for $9.95.

This is the sort of problem that will never show up under functional testing, as the client application should always sent good (validated on the client side) data to the server. Once you expose your application to the world by putting it on the Internet, you can no longer expect to receive only sanitised inputs.

A recent project I was involved with brought in a security test team from PwC to audit the application. It was mind-blowing how many small security problems a relatively stable piece of software can have.

The test team were using Open Web Application Security Project’s WebScarab tool. At a basic level, this tool is just a proxy that will allow you to modify the HTTP requests that are sent to the application.

This tool is a lot of fun to play with (for web geeks like me, anyway); just don’t get carried away and start trying out SQL injection attacks you find on BugTraq on other people’s websites.

Once you know how to use some of the Unix command line utilities, it is hard to go back to doing things the Microsoft way – where data manipulation seems to always be an awkward cut and paste/find and replace/filter with Excel.

I have found that Native Win32 ports of some GNU utilities fills the gap nicely. It is a collection of some of the more useful utilities generally found on Unix, and the 3 MB zip file fits nicely on a USB key. Ideal for the times when you just want to grep through a file and don’t want to (or are not permitted to) install a more complex Unix-like environment such as Cygwin.

Some of the utilities I have found useful are…

comm
If I had two files that contained a list of usernames and I wanted to remove any duplicates, I could use comm to create a list of usernames only in file1, usernames only in file2, and usernames in both files. comm file1.txt file2.txt

cut
If I had a csv file containing columns for username, first name, surname, and password, I could easily create a new file with columns for just username and password with cut -d, -f1,4 userfile.txt.

diff
diff shows the difference between two files. diff3 shows the difference between three files. sdiff shows the differences side by side (kind of like windiff, but text-based).

gawk
GNU Awk is a very powerful text manipulation tool.

grep
Searching through a file with a regular expression is much more powerful than the Windows search facility.

join
If I had two csv files, one with columns username, and password, and the other with columns username, first name, and surname, I could use join to achieve the same result as a join in SQL – adding first name and surname to the same line as password, wherever the usernames matched. Join can also be used to join columns where there is no match; it could be considered the opposite operation to cut.

md5sum
Generates an md5 hash of the input file. This is useful for checking the integrity of some of the software you download. Once I had to create a large number of user accounts before running a load test. Rather than creating them through the application front-end by hand or with a LoadRunner script, I dumped them directly into the database. The password field was an md5 hash of the user’s password.

sed
The stream editor is generally used for simple one-line find and replace operations. I have never used it as Awk seems to do much the same task. Note that LoadRunner uses sed to convert C-based web scripts to Java (look for the file web_to_java.sed in your LoadRunner installation directory).

sort
Sorts a file. Sort can accept multiple files as input, so it is also useful if you want to join web server logs from multiple servers and sort them by time before analysing them.

split
Splits a single file into multiple files. This can be based on the number of lines or the number of bytes. If you’ve ever needed to quickly divide a file of usernames between automated test scripts, this is much less painful than the old Excel cut-and-paste.

uniq
Output unique lines in the input file. This utility only compares adjacent lines, so it is usually necessary to sort the list first.

wc
Counts the number of lines, words and characters in the input file.

wget
If you’ve ever wanted to download web content without having to sit in front of your computer while you do it, wget can be a very powerful tool. Making a copy of an entire website is as simple as wget −−mirror −−domains=www.myloadtest.com −−wait==5 www.myloadtest.com

Following a load test, I often need to perform some additional analysis on the web server logs. It is not practical to use any commercial tools, and the free tools are all aimed at people who want graphs and statistics for entire weeks or months, rather than a few hours of high load. So far, I have usually been forced into a roll-your-own solution.

While it is easy to create graphs with Microsoft Excel, it’s 65536 row limit makes analysing any sort of non-trivial load futile unless the log files have been filtered before importing them. Even with filtering, it is hard to do anything useful with such a small number of records.

Microsoft’s Log Parser tool allows you to perform SQL queries on web log files. If you know exactly what you want, this is an extremely powerful tool. Having a large amount of memory will dramatically improve query times for large files, as will truncating the log files to just the period you are interested in. I found that debugging my queries could be a little painful.

The industrial strength solution is always to just dump the log files into a database and analyse them there.