My Load Test

All performance testing tools that I know of are bad at collecting system metrics from Unix-like system (AIX, Solaris, Linux, BSD etc). If you are lucky you might (might!) be able to collect CPU utilisation and maybe one or two memory utilisation counters.

Generally you have to create a shell script to run perfstat (or similar) then, when your load test is over, you process the output file and import it into your analysis tool.

Yesterday I learnt that you can write whatever data you want to an SNMP MIB so, if your performance test tool is able to gather metrics from SNMP, it follows that you can write your performance stats to the SNMP MIB and cut out the painful process of processing and importing for each test.

A little extra research shows that some operating system versions already have MIB objects with performance counters for metrics like CPU and memory use, although collection of these may have to be enabled. The OpenSTA performance testing tool website even has an article on “Monitoring CPU usage with OpenSTA via SNMP“.

This is a useful technique to add to any performance tester’s repertoire.

Some changes were made to a web application to reduce the amount of bandwidth used. To determine the impact of the changes, a previous benchmark test was compared with a new test. Both tests measured Bytes Received/sec on IIS, Bytes Sent/sec on IIS and Throughput (in Bytes/sec) from the perspective of the load testing tool.

Comparing the tests showed that Bytes Received/sec had been reduced by 1/3 but the Total Throughput had been reduced by 2/3 (Bytes Received/sec was a trivially small value). Where were the missing bytes?

Throughput: Displays the amount of throughput (in bytes) on the Web server during the load test. Throughput represents the amount of data that the Vusers received from the server at any given second. This graph helps you to evaluate the amount of load Vusers generate, in terms of server throughput. (taken from LoadRunner Analysis graph description)
Sent Bytes/sec: Bytes Sent/sec is the rate that data bytes are sent by the Web service. (taken from Perfmon explaination text)

In theory, the values from these two counters should match. But there is one very big gotcha. The Web Service counters that Perfmon uses (which are the same as the Web Server Resources > MS IIS counters in LoadRunner) measure values before IIS hands off its data to any filters.

The web server uses a GZip compression filter. One counter was measuring the data when it was compressed. One was not.

Next time I have to do this, it might be a good idea to also measure the throughput at the NIC level, even though it will include bits from the ethernet packets as well as the higher level encapsulating protocols such as TCP and IP. In Perfmon, this is Network Interface > Bytes Sent/sec. Note that throughput is harder to measure at the TCP or IP level as Perfmon does not give statistics in Bytes, only segments and datagrams respectively.