My Load Test

A common question on the LoadRunner support forums is “I’ve got an application that uses this new thing called AJAX. Will LoadRunner work with my application?” …and the answer is “yes…but…”.

The “but” is a big one . An “AJAX” application can mean anything from a simple webpage that updates some fields with information from the server (without reloading the whole page), to a horribly complicated rich GUI interface created with JavaScript.

Of course LoadRunner is going to be able to handle anything that is sent over HTTP, but scripting might not be the usual simple Record-Correlate-Playback, and the chances of the new Click and Script vuser type working are much lower than with a standard web application.

As a quick introduction to AJAX I have prepared an exercise using the simplest AJAX application that I could find.

Google Suggest is just like Google’s regular search interface, except that every time you type a letter in the search field, a request is made to the server that returns search suggestions.

The exercises are as follows:

1. Become familiar with the application – both at an end-user level, and at the HTTP level. Record the script in HTML and URL mode, then try the Click and Script vuser type (at the time of writing, this vuser type will not correctly record the application).
2. Create a script for this application. It should accept a parameter file that contains a valid search term on each line. Create the following transactions load_search_page, enter_character (called every time a character is entered in the search field), and final_search (where the Google Search button is pressed).
3. Imagine that you’ve noticed an odd step-like pattern in the Percentile graph in LoadRunner analysis. You hypothesise that response times differ depending on how many characters have been entered in the search field. Modify the enter_character transaction name to include the length of the search term every time it is called.

Treat this with the same level of care that you would give a script that you were creating for a paying client.

I have added some notes and a solution script to the comments section of this post. If you are going to attempt this exercise, please do so before reading the comments.

Useless trivia time…

Back in 1983, Broderbund released a simple (and hugely popular) platform game called Lode Runner.

Many years later, Mercury Interactive released a hugely popular load testing tool called LoadRunner.

Notice that the animated figures look very similar. Do you think that this is subtle homage, or just a case of all stick figures looking alike?

These are the long-awaited slides from my Mercury World Australia presentation. If you are currently performance testing an SAP implementation with LoadRunner or Performance Center, it is worthwhile evaluating Mercury’s diagnostic tools.

In terms of revenue, SAP is the fourth largest software company (behind Microsoft, IBM, and Oracle).

If you have any questions or feedback, I am happy to reply via email.

I will be presenting at Mercury World Australia this year. If you are interested in hearing about my experiences using Mercury’s diagnostic tools for SAP (both R/3 and NetWeaver), please come and see my presentation at 9am on the 12th of May.

I will probably post the PowerPoint slides some time after the presentation.

Update: slides are here.

There’s a saying in software circles that “writing the first 90 percent of a computer program takes 90 percent of the time… The remaining ten percent also takes 90 percent of the time.” A year ago, Mercury started telling people about their new GUI level virtual users for the web. Their promise was an impressive one – no more correlation. Anyone who has scripted for the web knows that correlating a web script can be either incredibly easy or painfully difficult if, for example, the web application constructs HTTP POSTs with an unholy mess of JavaScript. Mercury was promising to make this unnecessary by creating a playback engine that understood what the web page was doing just as well as your web brower, allowing you to create scripts that were all about clicking on buttons and links in a browser rather than sending HTTP requests to the web server.

Presumably, getting the remaining ten percent of this functionality working perfectly before the release of LoadRunner 8.1 was too difficult, so the software was shipped with this Vuser type disabled. The good news is that if you have a copy of LoadRunner (or Performance Center) 8.1, you can enable this vuser type yourself and have a play before it is officially released.

To enable this functionality do the following…

1. Open the \dat\protocols\webjs.lrp file under your LoadRunner or Vugen installation directory.
2. It is in standard Windows ini format. Find the [Protocol] section and change the line that says “Hidden=1″ to “Hidden=0″ .
3. Now open Vugen and create a new script. You will see that there is a new option to create a Web (GUI level) Virtual User

Rumour has it that the GUI level web vusers will be officially released by Mercury in 4-6 weeks with the release of service pack 2 for LoadRunner and Performance Center. This will mean that it will be officially supported by Mercury.

I do not normally recommend that users jump into brand new technology, but this vuser type offers such a huge advantage over regular HTML and URL-mode virtual users (not to mention every other web load testing tool from all other companies), that it is worth spending an afternoon seeing if it works with your application. If it doesn’t, you’ve lost an afternoon; if it does, you’ve saved weeks of your time.

A question that regularly comes up in the Mercury Support forum is how to get a LoadRunner script to handle websites that require you to type in the characters from a picture of a funny-looking piece of text before you can perform an action such as creating a new account. “Surely someone has come across this before, and has figured out how to correlate these values?”

Well…uh…no they haven’t. These are Captchas – they are used to prevent bots from doing nasty things like signing up for thousands of Hotmail accounts or leaving comment spam. If it was possible to write a LoadRunner script that could handle these automatically, then they wouldn’t be very effective.

Automated solutions used by spammers revolve around text recognition software and are inneffective. A spammer might be happy with a 0.01% success rate when signing up for webmail accounts, but it is hardly useful to a load tester who wants a 100% success rate. Boing Boing’s solution is also not very useful for load testing.

If you need to load test a website that uses captchas, then the only practical solution is for the developers to include the captcha value somewhere in the web page that is presented to the user. You can then correlate the value the same way you would normally. Just make sure that the code that does this is removed before the website goes live.

I don’t ever like to feel that I am wasting a client’s time by doing unnecessary load tests, but a long time ago I had an experience that altered my definition of what a necessary test is.

I had come to this project after the scope had been decided and the Detailed Test Plan had been written. The overall system was made up of four applications; privately I queried why we were bothering to test ApplicationX, which was an exceedingly low volume application.

ApplicationX was not business-critical or customer-facing. It was performing acceptably with a production-sized dataset and a small number of test users. Its functionality was unlikely to put significant load on the system, and there would be a maximum of 6 concurrent users in Production. Not load testing ApplicationX seemed like a perfect way to save some time.

Fortunately the Test Manager wanted to stick with the original plan (rather than having to get sign-off on an amended plan), because I was very very very wrong.

Here’s a graph of what happened as I slowly ramped up my virtual users…

I had never seen a web application fall over under a load of only two hits per second before. I didn’t think it was possible. My grandmother could handle two hits per second creating HTTP headers on a typewriter and sending the packets via carrier pigeon.

The worst part of this was that ApplicationX was running in the same JVM as an application that was business critical, so when it went down, it brought the other system down with it. A severity-3 defect suddenly became a severity-1 “oh my god, we can’t do business” defect.

So, the lesson I took away from that experience was that even if an application is not necessarily a candidate for load testing, but it shares infrastructure components with an application that is business-critical, then it is a good idea to consider at least some rudimentary sociability testing under load to shake out any really nasty problems like this one.

Okay, this is a bit of a tease because I haven’t actually written this function, but it would be nice if someone out there would (maybe it will be me when I get some more free time).

web_reg_save_param() is a useful little function for easily extracting text from the last HTTP response received by the virtual user. You specify left and right boundaries, the occurrence of the text you want to find and a large number of other options which, admittedly, don’t give you the power of regular expressions but are a lot easier to use.

The only problem is that once you’ve got a piece of text from the web page (or some other source), you can’t use this function to help with further manipulations of the text. The closest equivalent function is lr_save_searched_string(), which is useful, but only allows you to specify a left boundary and the number of characters to save. To get the equivalent of web_reg_save_param() requires writing string manipulation code using the standard C libraries which, we all know, some people find a little difficult.

It would be nice to have a function that was the equivalent of web_reg_save_param(), but that you could pass an arbitrary string to. Various people have written functions that implement a small sub-set of the required functionality (like returning text between boundaries), but I have never seen anything which is even close being feature-complete; besides, code in Mercury’s User Contributed Knowledge Base is notoriously buggy.

A good implementation of this function – let’s call it lr_reg_save_param() – might have the following attributes…

• Would accept the same arguments as web_reg_save_param(), except with an additional argument for the input string.
  int lr_reg_save_param (char* InputString, const char *ParamName, <list of Attributes>, LAST);
• Would work in exactly the same way as web_reg_save_param() – I don’t want to have to learn how to use another function.
• Would be threadsafe – users shouldn’t have to chase down this type of bug.
• Would not create any LoadRunner parameters except for the parameter name specified – this prevents any problems when users inadvertently have a parameter name that is the same as one that is used internally by the function.
• Would be well-tested and bug-free.

People don’t generally read click-through licenses on the software they install. The only license I’ve read with a lawyer-like (or tester-like) attention to detail has been the GPL. The rest have just had a cursory check for any unduly onerous conditions…like the right to spy on my computer or the soul of my first-born child or something equally sneaky.

The LoadRunner license has a couple of surprises in it. I’m not sure how enforceable the license would be, but it’s probably in the best interests of anyone who uses Mercury tools that they are aware of the terms and do their best to comply.

Here are the sections I found interesting…

You may not:

• use the Licensed Program to provide services to any third party, including affiliates or subsidiaries of Licensee, including but not limited to installation, integration, support and testing or monitoring of hardware, software, web sites and/or networking solutions;

Did you ever wonder why Mercury is the only company that will include the use of their products in the price for short-term work? If it wasn’t prohibited by the license, a consultancy company could do a lot of business if they bought a LoadRunner license with a large number of virtual users and took it from company to company. I expect that Rational and Compuware have similar clauses in their licensing agreements, or else you would see consultancy companies doing it with their products instead. I guess that the open source tools are not quite mature enough to do this with yet.

• use, evaluate or view the Licensed Program or Documentation for the purpose of designing, modifying, or otherwise creating any software program, or any portion thereof, which performs functions similar to the functions performed by the Licensed Program;

I heard a story from one of the other load testing tool vendors; they asked for feedback from their users about what features they would like added to the next version of the product. When they combined all the feature requests, they basically had a list of most of the features in LoadRunner.

The license agreement also has a section about reverse engineeing. Mercury, understandably, don’t want anyone making a LoadRunner clone, whether they do it by decompiling the code or just by recreating the features from the documentation or GUI.

• provide externally or to third parties any oral or written communication describing or summarizing the features, functions or performance characteristics of a Licensed Program or Documentation or that compares a Licensed Program to any similar product of Yours or any third party;

This is a bit of a difficult one for anyone who, say, runs a website on performance testing. It means that you can’t really do a comparison of LoadRunner with other tools, even if it is favourable. Fortunately I’m not planning to write a LoadRunner feature list anytime soon.

Definitions, Product Permissions and Restrictions

• A Concurrent License allows for use of a Licensed Program by a specified number of users ordered and paid for by You, over a LAN at the Site, or via browser when the Licensed Program is browser based. A Concurrent License accessed and used by browser may be accessed and used only from the applicable territory or region as ordered and paid for by You, as reflected on MIC’s quote or then-current local published price list, and Your order document ordering a regional or global license as quoted or listed thereon.

I had no idea that Mercury had a regional pricing model, but I guess that makes sense. This just means that companies can’t do what I do on my holidays – bring home cheap products from third-world countries.

• The “LoadRunner” Licensed Program may be used on the Designated Computer to test a website, software program or networking solution located at the Site (“Application under Test” or “AUT”), and execution of the test on the AUT must occur at the Designated Computer. Load generation with Virtual Users (“VUs”) or Virtual User Days (“VUDs”) may take place locally or remotely. LoadRunner may not be accessed or used over a LAN or WAN or via remote access technology, including but not limited to the pcAnywhere product or any products marketed by Citrix Systems, Inc. VUs, VUDs and monitors are licensed to a single Controller and may not be transferred or relocated to a different Controller.
• The “Topaz” Licensed Program may be accessed over a LAN or WAN owned or leased by You or via remote access technology such as pcAnywhere or Citrix products provided the usage ordered and paid for by You is not thereby exceeded.

In other words, don’t use the Terminal Services client (or similar products) with LoadRunner unless you pay for Topaz. I think a lot of people do this anyway, preferring to have their controller box in the server room or data-centre rather than at their desk, and not wanting to sit in a cold noisy room while they do their load test.

I think that the following question from the Mercury Support forum neatly illustrates one of the major problems with LoadRunner.

Hi all,

I have a financial application that submits the data in $ amounts. When I looked in the extended log the value is in number format like (43251) but during submission it is converted into dollar format $43,251.00.

How to convert the string into $ format? Is there any function?

Thanks in advance,

The majority of people using LoadRunner aren’t great programmers, and the ones who are aren’t necessarily great programmers in C. Fortunately, the level of skill required for most problems is not very high, and everyone muddles along as best they can.

Not surprisingly, the small proportion of problems that require more than basic knowledge of C are done poorly (or by someone else).

The most common problem that requires additional programming is basic string manipulation. Most LoadRunner scripts are written in C, and manipulating strings with C is like pulling teeth.

I would argue that Mercury’s choice to use C as LoadRunner’s main scripting language, while having the advantages of it being a solid standard, having a low performance overhead, and there being a large pool of people at least partially familiar with the language, has the unfortunate trade-off of it being…well…kind of hard for most people.

Here’s my solution to the above problem.

// This function converts a numeric string to a dollar value.
// eg. 123456 becomes $123,456.00
LPCSTR ConvertStringToDollars(char input[20]) {
    int i;
    // To account for the extra characters that will be added, the size of output
    // should be 4 + strlen(input) + (strlen(input)/3)
    char output[30];

    strcpy(output,"$");

    // insert a comma every third character, but not at the start.
    for (i=0; i<strlen (input); i++) {
        if ( ((strlen(input)-i)%3==0) && (i!=0) ) {
            output[strlen(output)] = ′,′;
        }
        output[strlen(output)] = input[i];
    }

    strcat(output, ".00");

    return output;
}

Action()
{
    lr_output_message(ConvertStringToDollars("123456")); // Prints "$123,456.00"
    return 0;
}

This illustrates my point perfectly. Even though the code performs a simple operation, it’s complicated, and requires knowledge of how C stores data in memory in order to implement. Possibly more importantly, implementing the solution in C takes a lot longer than it might in another language. If there are any mistakes in my code, it just proves my point even more.

As computers increase in power and the performance advantage of writing performance test scripts in C shrinks even further compared to other languages, the relative disadvantages of C will grow in the minds of testers.