My Load Test

I like posting LoadRunner code snippets on My Load Test and, judging by the emails and comments that I get, a lot of people find them really useful. To make my code more readable, I have added syntax highlighting that uses the same colours you see in VuGen. If you would like to download a copy of the syntax highlighting code to use on your own blog or forum, or you would like to learn more about how I did it, read on…
(more…)

I just got back from the US after presenting at HP Software Universe 2010. I will save the content of my presentation for another time, so this is a quick overview of what I saw at the Conference and some of the things I learned from the people who attended.

(more…)

On Tuesday March 2nd, I will be presenting on “The Non-Functional Requirements Every Project Forgets” at the ANZTB 2010 testing conference in Melbourne, Australia. Please come and say “hi”.

(more…)

I’ve spent the last year or so writing notes and fleshing out chapters for a book called Performance Testing Web Applications, so imagine my very slight feeling of annoyance when I did a google search for my book title and found that someone had released a very similar book 3 months earlier…

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea have collaborated on a book called Performance Testing Guidance for Web Applications, which is available either as a free download, or in dead-tree format through Amazon.

The book is really good, so I highly recommend that you grab a copy and have a read.

James Bach visited Melbourne this week to teach his Rapid Software Testing course. While he was here, he took the time to talk to about 30 developers and testers from the Melbourne XP Enthusiasts Group (MXPEG) and the Melbourne Association of Software Testers (MAST).

James is an acomplished speaker and is passionate about testing. He spent the first few minutes railing against the traditional testing world that is “obsessed with techniquism and artifactism” – blindly following a testing process without thinking (which I guess I talked about a couple of weeks ago in my post on Cargo Cult Testing); and observed that thinking was a skill that could be taught and practiced.

As an example of a training exercise that could develop exploratory testing skills (and therefore thinking skills), James introduced the Art Show game. This game is similar to Mastermind, except played with cards and much more informal. It boils down to teams determining the the algorithm inside a black box (an art critic’s preferences) through iterative exploratory tests (art showings).

I won’t spoil the game for you by giving too many hints but, rather than simply getting the correct answer, the exploratory testing process is more important. It gives valuable practice in forming a hypothesis and validating or refuting the hypothesis though testing while optimising for speed, coverage and cost (because you may lose cards each time you run a test).

The highlight of the night was probably seeing James in vigorous debate with someone over ISEB’s definition of exploratory testing as “an ad-hoc technique”.

I am sure that I will not be the only attendee who will be dusting off my art critic’s beret to run this as a training exercise with my team sometime soon.

Richard Feynman popularised the idea of the cargo cult in his essay on cargo cult science, which was “science” that followed all the forms of scientific investigation, but lacked real critical scientific thought. The idea was transferred to the world of software courtesy of the Jargon File with its entry on cargo cult programming.

The Jargon File defines cargo cult programming as:

A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose. A cargo cult programmer will usually explain the extra code as a way of working around some bug encountered in the past, but usually neither the bug nor the reason the code apparently avoided the bug was ever fully understood.

As someone who works in one of the more technical areas of testing, I see the same thing in the testing world too.

Functional testing is pretty straight-forward; ask a test manager why they are testing, they will probably say something like “we are seeing if the business functions work” (really smart test managers will add that they are seeing that the actions that are meant to fail do actually fail too).

But once you add a layer of technology, it becomes a bit harder to grasp. If your functional test regression suite is automated, then the quality of the testing is less easy for most people to see. Were the test cases that were automated good test cases to begin with? After a few months or a year, regression testing becomes a little bit like a ritual; you wave the magical testing tool over a new version of the application and declare it “tested”. Whether it is tested well is an entirely different question…and one which is unlikely to occur to those deeply involved in the ritualistic behaviour.

In the stress/volume/performance/load/reliability testing world things are even worse. Occasionally I visit companies that I have consulted at in the past to see how they are going with their testing. They generally know to run a peak load test for new builds (and platform changes) before applying the change to Prod, but if you ask a question like “have the transaction volumes changed in Production since I wrote the original Detailed Test Plan, and have the LoadRunner scenarios been updated to reflect this?”, you will draw a blank.

Performance testing during the maintenance phase of software frequently seems to degrade to a situation where the testers are more interested in whether the test cases and LoadRunner scripts still run successfully, rather than whether the test cases reflect reality or even cover the point of change that is driving the current round of testing. And a lack of understanding of what each test case is designed to test leads to wasteful re-running of test cases that are not impacted by a change.

Unfortunately I don’t really have any solutions for this besides companies paying me to occasionally come in and review their performance testing activities (which kind of smacks of self-interest).

I have made a list of the top ten things load testers frequently fail to do that make me feel like smiting them.

1. Thou shalt know how thy test tool works.
   The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic. Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. Thou shalt gather realistic usage data.
   Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. Thou shalt have testable requirements.
   Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects. This shouldn’t stop you from trying to gather the requirements you need for your tests. The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough. Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. Thou shalt write a test plan.
   Even if you already know what you’re going to be doing, other people would probably like to know too – they might even be able to help; besides, a signed-off test plan has saved many a tester from the wrath of project management.
5. Thou shalt test for the worst case.
   Don’t test with transactions from an average day, test for the busiest day your business has ever had. Add a margin for growth. Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. Thou shalt monitor your test environment infrastructure.
   I feel that I have to spell it out, because I still see people who don’t do this. Monitoring your servers allows you to more easily figure out where the problem is. You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%” When I say “monitor your servers”, this includes your load generators.
7. Thou shalt enforce change control on your environment.
   The final thing you tested should be what is deployed into Production – same application version, same system configuration. It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing. Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. Thou shalt use a defect tracking tool.
   An untracked defect is a little like a tree that fall in the forest when no-one is around – no-one cares. Raising defects lets everyone know there is a problem (not just the people who should be working to fix it). It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. Thou shalt rule out thy own errors before raising a defect.
   “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects. If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts. Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system). It always helps if you have followed commandment #1 Thou shalt know how thy test tool works.
10. Thou shalt pass on your knowledge.
    Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting. Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC. Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them. Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.

A Java thread dump is a way of finding out what every thread in the JVM is doing at a particular point in time. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck.

You can generate a thread dump under Unix/Linux by running kill -QUIT <pid>, and under Windows by hitting Ctl + Break.

A great example of where this would be useful is the well-known Dining Philosophers deadlocking problem. Taking example code from Concurrency: State Models & Java Programs, we can cause a deadlock situation and then create a thread dump.

In the example below (shown using tda), we can see that the 5 Philosopher threads each have a lock on a Fork object and are each waiting to obtain a lock on a second Fork object before they can eat. Unfortunately this never happens and all the philosophers starve.

Download the thread dump from here (7 KB).

Note that not all hangs are going to be due to deadlocks, and there are many tools (including Eclipse) that will help you analyse thread dumps.

I just found something interesting on Google Video; it is a 1 hour Google TechTalk presentation by Goranka Bjedov about Using Open Source Tools for Performance Testing.

Goranka has some interesting things to say. She makes the point that there is really no standard terminology in performance testing circles, and goes on to prove this by giving her own definitions of performance, stress, load, scalability, and reliability testing. As an example of reliability testing she notes that “typically, when I was at AT&T, we would run for about a month at a time after everything was done just to find out that the system can actually stand up and can work fine with the load for a prolonged period of time.” In my testing circles, we would call that a soak test, but I would have been interested to hear more about the types of systems she was testing at AT&T.

The main body of her talk is about the different tools available for load and performance testing. These can be broken down into in-house, vendor and open-source tools.

Google has 55 in-house load and performance testing tools that have been developed by different groups for testing different Google products. These are very expensive to maintain and may only be used in-house, which makes any benchmarks impossible to verify by a third-party. Goranka says “Before you decide to develop your own, please take a look at what is out there…”

Goranka slams vendor tools (like LoadRunner, SilkPerformer and WebLOAD) for being overly expensive and using proprietry scripting languages. Personally, I have always thought that it was pointless having to learn another language just to use a load testing tool. Unfortunately she uses LoadRunner’s scripting language as an example “it’s C, minus the pointers”, and is incorrect – unlike many other tools, Mercury uses standard C (and Java and VBScript).

Her recommended solution is the open source tools – “five years ago they just weren’t there, but today they are.” Her personal preference is JMeter, but also recommends OpenSTA and The Grinder. Open-source tools have the advantage of being a good price, and having source code available; she also makes the point that they use standard programming languages for scripting (although this is incorrect when talking about OpenSTA).

The disadvantages of open-source tools are that they have a steep learning curve and do not support many protocols. “The vendor tools support far more protocols than the open-source tools, but as long as you are staying in the web space, and your looking at HTTP/S, IMAP and POP3, the open-source tools are pretty good”.

Goranka does not say that the open-source tools are free because it is occasionally necessary to write code to extend their features. “Free software is free in the sense that a puppy is free.” Features that Google engineers have written for JMeter have been added back to the main code tree by the people maintaining the project, meaning that Google is at least saved the cost of maintaining their forked code.

She uses JMeter for testing web-based applications through the GUI, uses The Grinder for API-based testing, and does not use OpenSTA because it only works on Windows.

Other points during the presentation:

• You should use the same monitoring for Load testing that you use for Production monitoring (so you don’t have to account for the differences in load that a different monitoring system will put on the system).
• If you are running Unix-based systems, don’t sustain CPU above 80%.
• Google tracks a summary of every performance test in a central database. The database also contains information on every piece of software that is installed on the machines in the test environment.
• If I am unfamiliar with the system, I don’t trust it. One of the things that I have realised is that
  A) the system will fail in the place where they tell me that nothing could go wrong.
  B) developers are totally delusional about their own software, and frequently they will just forget about things that they’ve done two weeks ago.
• I run every test 5 times. I want to see that I have some sort of statistical consistency.
• Performance testing should not be used as a tool to find memory leaks; but it can.
• Performance testing without monitoring? Don’t bother. Why waste your time?
• If you are going to do any performance testing, make sure that database sizes are somewhat realistic. They don’t have to be exactly the same, but they have to be the same order of magnitude otherwise the results you are getting are completely off.
• Execute a stress test. Find how your system is failing. Find where it is failing. Do find out how the system handles overload. There are no good defence mechanisms against people out there, and you can’t predict sudden popularity (eg/ Google Earth).
• Start a test after a decent warm-up period. Don’t start 100000 users all at once.
• Quite often people don’t know about everything that is running on a complex system. Maybe there is a low priority process that is running with high priority. This can usually be fixed by niceing the process down. Quite often there are debug things that are still running also.
• Monitor the machines that are collecting the monitoring data and the load generators (not just the system under test).
• Performance Testing and QA is about risk analysis. If I believe it is high risk, I want to take a look at it.
• When I am doing performance testing, the first thing I try to do is eliminate the network. I want to simplify my problem. I am interested in the machines, and my hope is that the network provided will handle everything I need. Once everything is profiled and understood, we will do some tests that include the network. If you can, put everything on the same subnet and same switch. It will make you a much happier performance tester in the first pass. Debugging networking problems is (not) fun.
• (When talking about testing on smaller sized systems than Production). You can’t test on a 386. Extrapolation will kill you. You will run out of some resource that you never expected, and you can’t predict this ahead of time. For final validation, you really want to get some time on the Production hardware before it goes live. If the system is not being used for Production, it should not be that hard to get hold of it for a week or a weekend.
• Find more open-source performance tools at opensourcetesting.org

There is another summary of her talk available on Robert Baillie’s blog.

You might also want to have a look at Becoming a Software Testing Expert; a 1 hour presentation delivered by software testing expert, and author of Lessons Learned in Software Testing, James Bach on June 13, 2006. His presentation is available for download from his website.

Last week I posted a question asking how I could recover or change the password for a VMware guest operating system (Windows 2000) that I had forgotten the password for. After receiving no useful suggestions, this week I allocated some time to solving the problem.

Windows password recovery tools usually consist of a bootable CD image containing a version of Linux that will overwrite the NT password with a known value or will extract the hashed password from the filesystem.

To boot your virtual machine from a CD, you must change the boot order in the virtual machine’s BIOS. Press F2 while the VM is starting up to access the BIOS.

I used the free software available from Windows XP Login Recovery. This is good because it does not try to write to your file system, it just retrieves the hashed password value.

Once you have the password hash, you enter it in a form on their website and they look up the hash value in their database and give you a password that matches the hash. Note that even though they ask for your email address the password is displayed on a web page rather than being sent to your inbox.

After all that effort, I discovered that my password was blank.