My Load Test

On Tuesday March 2nd, I will be presenting on “The Non-Functional Requirements Every Project Forgets” at the ANZTB 2010 testing conference in Melbourne, Australia. Please come and say “hi”.

Read the rest of this entry »

Subject: HTML return value

Hi Stu, you probably know the answer,

what I need to know if there is an option in Vugen to run a url and to save the url content to a string, something like XML apis in Vugen.
Later I can parse the string to see if I got the right value.
There is surely an add-on to web_url to save the return value to a string.

and I need to save it as a variable in Vugen and not writing it to a file.

Tell me if you know if it achievable.

Thanks
Txxxx

Here is some example code…

// save headers to a parameter
web_save_header(REQUEST, “ParamRequestHeader”);
web_save_header(RESPONSE, “ParamResponseHeader”);

// save entire body to a parameter
web_set_max_html_param_len(“262144″); // default max length is 256 characters
web_reg_save_param(“ParamResponseBody”, “LB=”, “RB=”, “Search=Body”, LAST);

// check that page contains expected value
web_reg_find(“Text=Google”, LAST);

web_url(“Google”,
“URL=http://www.google.com.au/”,
“TargetFrame=Main”,
“Resource=0″,
“RecContentType=text/html”,
“Snapshot=t3.inf”,
“Mode=HTTP”, // If this is HTML, then ParamResponseBody will contain the most recently downloaded item on the page (eg/ a GIF).
LAST); // but note that this means that the resources referenced on the page will not be downloaded.
// An easier way to do this is to define a left and right boundary around the item that you want to check.

lr_output_message(“# Request Header:\n %s”, lr_eval_string(“{ParamRequestHeader}”));
lr_output_message(“# Response Header:\n %s”, lr_eval_string(“{ParamResponseHeader}”));
lr_output_message(“# Response Body:\n %s”, lr_eval_string(“{ParamResponseBody}”));

thanks stu, look promising, my HTML return code going to be in XML format so I will probably going to use XML api to find a specific values.

Thanks
Txxxx

If you are using this with web services, you just need to use…

// save entire body to a parameter
web_set_max_html_param_len(“262144″); // default max length is 256 characters
web_reg_save_param(“ParamResponseBody”, “LB=”, “RB=”, “Search=Body”, LAST);

…then you can use the lr_xml functions to extract values from the parameter containing the SOAP body.

Subject: Help wanted in SAP with LR..

Hey Stuart,

I Guess u have got good experience in LR with SAP…
I got into a problem in my work place.. hope u can figure it out.. pelase let me know if u have a solution for his..

I created a Script for R/3, ( Order to Cash ) which will have 10 materials in a single transaction..
When i run the script with 1 Iteration, it is successfull.

But when i run it with 2 Iterations,, its bouncing from the program..
This happens even in Controller…

So please let me know if there is a solution for this..

Vxxxxx.

You need to make sure that the Action part of your script starts and ends in the same place (and with the same state) in your application.

If the screen is different on the second iteration, your script will probably fail.

Note that sometimes the screen looks the same, but object references are different. You can check this by recording two iterations of your business process in a single script and comparing iteration 1 with iteration 2 (including the object IDs).

Hey Stuart,

What you said was absolutely right..

We resolved the issue by adding a set focus function in the loop..so that on each and every iteration, it comes back to the focus..

Thanks a lot!

I frequently receive emails with LoadRunner-related questions. I try to reply to questions when I have time. Maybe posting my answers to MyLoadTest will be an incentive for me to answer more questions, and maybe it will save other people some time.

Subject: Sending email from loadrunner script

Hi Stuart,

we would like to run loadrunner script (HTTP WEB Protocol) which will send an email to user Whenever transaction response time for web page is more than specified time or application is down for some reason.Basically we would like to monitor the application continuously. How can we achieve this using loadrunner? or suggest us best solution for monitoring application which would send email notification to monitoring team.

Thanks,
Rxxx

The best solution for monitoring a web application’s availability and performance is going to be HP Business Availability Center (BAC). BAC has a component called Business Process Monitor (BPM). BPMs are VuGen scripts that create synthetic transactions against your application. BAC handles the alerting, collection of historical data, and dashboard functionality.

The advantage of using BPMs as well as traditional infrastructure-based monitoring is that a BPM will tell you if your web application is really available because it is performing a real business process. Pinging a server every 5 minutes doesn’t tell you that the database service has hung. Requesting just the front page of the website every 5 minutes doesn’t tell you that all of your online store functions are working except the “checkout” function.

As well as alerting on response times which are outside of SLAs, capturing response time data shows historical performance trends – maybe you will see a jump in response times for searches after all those extra products were added to the catalog? Or maybe a slow degradation in response times as more and more old orders sit in the database?

But let’s assume that you are unable to convince your management that they should do things the correct way, and they decide to go with the unreliable but cheap solution of monitoring an application with a copy of VuGen which is running constantly on a workstation.

Problem #1 – stability of the monitor over time

Using VuGen by itself as a monitor is a solution you would have to babysit. You would probably want to restart it every day as logs will accumulate and eventually start slowing everything down.

Problem #2 – accuracy of response time measurements

As well as the possibility of very large logs distorting your response time measurements by slowing down VuGen, running a script in VuGen is far slower than running the same script through the LoadRunner agent (mmdrv.exe which is used by LoadRunner Controller and by BAC). The overhead can be reduced by disabling “Animated Run” and “Output Window”, but there should still be a significant difference.

Problem #3 – historical data

I don’t generally like collecting data if I am not going to keep a copy of it somewhere. I like to see trends over time. Extracting data out of the logs would probably be painful so, if I were doing this, I would probably write some extra code to write to a CSV file so that I could graph the data in Excel.

Problem #4 – email alerting on slow response times

If you have licensed the SMTP vuser type, then you can create a Multiple Protocol (Web + SMTP) script so that you can send emails with your script. Alternatively you might be able to find a command line tool that can send email (like Blat), and call it with the system() function; or you could send an email using a web-based interface – maybe set up a private web service on a completely separate server.

Actually detecting that the response time for a transaction does not meet your SLA would require you to set a timer at the start of each transaction, and write some conditional logic at the end of each transaction e.g. if ((endTime – startTime )> SLA) { raiseAlert(); }. Using BAC is obviously a much more robust way of doing this, and eliminates all of the extra code you would have to write.

Problem #5 – alerting on errors

Alerting on slow response times is reletively simple, but alerting on errors becomes much more complicated with my hacky VuGen solution. The problem is that control is passed from your code back to VuGen and its Run Logic as soon as you fail a Content Check Rule or a web_reg_find. The only way to get around this is to enable the “Continue on error” runtime setting and do all of your error handling manually. This will mean writing big ugly chunks of code like this for every request that you want to check…

lr_continue_on_error(LR_ON_ERROR_CONTINUE);
// Check for known error messages
web_reg_find(“Text=Your session has expired”, “SaveCount=errorSessionExpired”, LAST);
web_reg_find(“Text=A runtime error occurred”, “SaveCount=errorRuntimeError”, LAST);
// Check for message to indicate success (positive text check)
web_reg_find(“Text=Your order has been submitted successfully”, “SaveCount=checkOrderSubmitted”, LAST);

web_submit_data(…); // snip!

if (atoi(lr_eval_string(“{errorSessionExpired}”)) > 0) {
sendEmailAlert(“Error: Session expired for user %s”, lr_eval_string(“{UserName}”));
} else if (atoi(lr_eval_string(“{errorRuntimeError}”)) > 0) {
sendEmailAlert(“Error: Runtime error for user %s”, lr_eval_string(“{UserName}”));
} else if (atoi(lr_eval_string(“{errorSessionExpired}”)) < 1) {
sendEmailAlert(“Error: Text Check failed for \”Your order has been submitted successfully\”.);
}

Subject: LR JDBC Protocol

Hi Stuart,

I have a Java Swing-based client that communicates directly with a database server using JDBC. What are my options for recording this application with LoadRunner

Thanks,
Jxxxx Cxxxx.

As you probably noticed, there is no native vuser type which will record your application successfully. Fortunately, you still have several options for recording your application.

Option #1 – Winsock (a bad option)

If you record your application with a Windows Sockets vuser, you will see uninteligible data being sent and received.

Just because you can record something with a Winsock vuser doesn’t mean it is a good idea to try to create a script with this vuser type. The rule-of-thumb is don’t try it unless you can understand what you have recorded at a glance. This means avoid any incomprehensible binary formats and only use Windows Sockets recording for the simplest text-based protocols.

Option #2 – Citrix or Terminal Services

Citrix vusers (and, with the release of LoadRunner 9.0, Terminal Services vusers) are kind of like a get out of jail free card for applications that are difficult to script. Rather than recording the network traffic between your client and the server, these record the traffic between your Citrix client and the server that is running your application client, giving you scripts that describe what is happening to the GUI (e.g. ctrx_mouse_double_click(), ctrx_sync_on_window() etc.)

The downside to this approach is that you will…

• need to be licensed for LoadRunner Citrix (or Terminal Services) vusers.
• need to have access to a server to run many Citrix sessions on. As you are adding another layer to your test, you must ensure that the Citrix server is not the bottleneck when running your load test.
• need to have enough Citrix licenses to run your required number of virtual users.

Option #3 – QuickTest Pro vusers

LoadRunner has an option to run QuickTest Pro scripts instead of scripts created with VuGen. As with the Citrix vusers, your scripts are interacting with the GUI of the application you are creating the script for, so you will never have to worry about any difficult correlation situations.

The downsides are…

• You need a special LoadRunner license.
• You will need a separate QTP license for each vuser you intend to run; e.g. if you want to run 100 virtual users, you will need 100 QTP licenses. This rapidly becomes expensive.
• You will only be able to run 1 vuser per load generator, so to run 100 virtual users, you may need to find 100 workstations and install QTP on them. You can increase this slightly by fiddling with VMWare or Terminal Services, but you will still need lots of generators.
• As it operates through the GUI, it requires extra time to identify objects. Using WinRunner vusers in the past, I would get radically different response measurements between my WinRunner vusers and my web vusers.

Option #5 – Buy from a third-party

On June 13 2007, Clay Roach from J2EE 911 announced that they would be releasing a LoadRunner JDBC protocol through the HP partnering program.

J2EE 911 announces HP LoadRunner and HP Performance Center protocol extensions for the Java/J2EE platform – including the first pure JDBC Protocol which can be used in conjunction with existing HP LoadRunner scripts to test and validate database data on any database that supports JDBC. The JDBC protocol will allow customers to harness their existing investments in HP Quality Management and enable quality assurance (QA) and development teams to easily collaborate and ensure scalability of their Java/J2EE applications.

The new protocol hasn’t been officially released, but Clay will be happy to discuss licensing options for this product, and may make an early release available to you if you ask nicely.

Option #4 – Java recording with Custom Hooks

The Rmi-Java and Corba-Java vuser types (which I have written about previously) behave differently to other vuser types. They hook into the JVM and record the method calls and arguments to the methods, and generate a Java-based script with the same API calls that your application makes.

The RMI and CORBA methods that LoadRunner hooks to record these scripts can be found in \LoadRunner\dat\rmi.hooks and \LoadRunner\dat\corba.hooks.

An example hook might look like this…

[CORBA_2_3-ORB-init1]
;---------- ORB.init ----------
class      = com.sun.corba.se.internal.corba.ORB
method     = init
signature  = ()Lorg/omg/CORBA/ORB;
cb_class   = mercury.corba.CORBASupport

This hook instructs LoadRunner to hook (add extra code to the class as it is loaded to record method calls and object values) the init method in the ORB class in the com.sun.corba.se.internal.corba package.

It is possible to define your own hooks in \LoadRunner\dat\user.hooks

; This is a sample hook -

[Hook-Name]
;---------- ClassName-MethodName --------
class      = MyPackage.MyClass
method     = MyMethod
signature  = ()V
cb_class   = mercury.ProtocolSupport
general_cb = true

When choosing which classes and methods to hook, you will generally pick any which send network traffic, so for an application which communicates using JDBC, you would choose all of the JDBC methods (which can easily be found in the java.sql and javax.sql Javadocs).

There is further information in the Hooking “General” Java API calls with LoadRunner document, which was in the Mercury Support knowledgebase until HP took it down.

I would not advise that you try this approach unless you are competent at programming with Java.

I’ve spent the last year or so writing notes and fleshing out chapters for a book called Performance Testing Web Applications, so imagine my very slight feeling of annoyance when I did a google search for my book title and found that someone had released a very similar book 3 months earlier…

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea have collaborated on a book called Performance Testing Guidance for Web Applications, which is available either as a free download, or in dead-tree format through Amazon.

The book is really good, so I highly recommend that you grab a copy and have a read.

James Bach visited Melbourne this week to teach his Rapid Software Testing course. While he was here, he took the time to talk to about 30 developers and testers from the Melbourne XP Enthusiasts Group (MXPEG) and the Melbourne Association of Software Testers (MAST).

James is an acomplished speaker and is passionate about testing. He spent the first few minutes railing against the traditional testing world that is “obsessed with techniquism and artifactism” – blindly following a testing process without thinking (which I guess I talked about a couple of weeks ago in my post on Cargo Cult Testing); and observed that thinking was a skill that could be taught and practiced.

As an example of a training exercise that could develop exploratory testing skills (and therefore thinking skills), James introduced the Art Show game. This game is similar to Mastermind, except played with cards and much more informal. It boils down to teams determining the the algorithm inside a black box (an art critic’s preferences) through iterative exploratory tests (art showings).

I won’t spoil the game for you by giving too many hints but, rather than simply getting the correct answer, the exploratory testing process is more important. It gives valuable practice in forming a hypothesis and validating or refuting the hypothesis though testing while optimising for speed, coverage and cost (because you may lose cards each time you run a test).

The highlight of the night was probably seeing James in vigorous debate with someone over ISEB’s definition of exploratory testing as “an ad-hoc technique”.

I am sure that I will not be the only attendee who will be dusting off my art critic’s beret to run this as a training exercise with my team sometime soon.

Richard Feynman popularised the idea of the cargo cult in his essay on cargo cult science, which was “science” that followed all the forms of scientific investigation, but lacked real critical scientific thought. The idea was transferred to the world of software courtesy of the Jargon File with its entry on cargo cult programming.

The Jargon File defines cargo cult programming as:

A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose. A cargo cult programmer will usually explain the extra code as a way of working around some bug encountered in the past, but usually neither the bug nor the reason the code apparently avoided the bug was ever fully understood.

As someone who works in one of the more technical areas of testing, I see the same thing in the testing world too.

Functional testing is pretty straight-forward; ask a test manager why they are testing, they will probably say something like “we are seeing if the business functions work” (really smart test managers will add that they are seeing that the actions that are meant to fail do actually fail too).

But once you add a layer of technology, it becomes a bit harder to grasp. If your functional test regression suite is automated, then the quality of the testing is less easy for most people to see. Were the test cases that were automated good test cases to begin with? After a few months or a year, regression testing becomes a little bit like a ritual; you wave the magical testing tool over a new version of the application and declare it “tested”. Whether it is tested well is an entirely different question…and one which is unlikely to occur to those deeply involved in the ritualistic behaviour.

In the stress/volume/performance/load/reliability testing world things are even worse. Occasionally I visit companies that I have consulted at in the past to see how they are going with their testing. They generally know to run a peak load test for new builds (and platform changes) before applying the change to Prod, but if you ask a question like “have the transaction volumes changed in Production since I wrote the original Detailed Test Plan, and have the LoadRunner scenarios been updated to reflect this?”, you will draw a blank.

Performance testing during the maintenance phase of software frequently seems to degrade to a situation where the testers are more interested in whether the test cases and LoadRunner scripts still run successfully, rather than whether the test cases reflect reality or even cover the point of change that is driving the current round of testing. And a lack of understanding of what each test case is designed to test leads to wasteful re-running of test cases that are not impacted by a change.

Unfortunately I don’t really have any solutions for this besides companies paying me to occasionally come in and review their performance testing activities (which kind of smacks of self-interest).

The Virtual Table Server is great for most situations where your virtual users need a common data pool, but the limitations of the API mean that it is a bad fit in some cases.

Say you need to find a row in the data table with a specific value. On a real database you could just do a simple SELECT statement. With VTS, you would have to write code to iterate through every row in the table and check the column value each time.

So sometimes you need to use a real database. Interfacing with a real database is not much harder than interfacing with VTS (as the attached script demonstrates). You will spend more time setting up the database than writing code.

Interfacing with a real database will also allow you talk to the database your application uses (only do read-only operations, and be careful of any additional overhead you might introduce during your load test).

This example users the Java vuser type, and interfaces with a MySQL database. The beauty of JDBC is that to port this code to another database just requires changing the line of code that specifies the database driver (assuming that you have not used database-specific SQL code).

On with the example…

Setting up the database

1. Download and install MySQL.
2. Create a database
   mysql> CREATE DATABASE loadtest;
mysql> SHOW DATABASES;
mysql> USE loadtest;
3. Create a table
   mysql> CREATE TABLE message (time_sent BIGINT UNSIGNED, identifier VARCHAR(127));
mysql> SHOW TABLES;
mysql> DESCRIBE message;

4. Load data into the table
   mysql> INSERT INTO message VALUES ('1178858071111','asdf1234');
mysql> INSERT INTO message VALUES ('1178858071112','asdf1235');
mysql> INSERT INTO message VALUES ('1178858071113','asdf1236');
mysql> INSERT INTO message VALUES ('1178858071114','asdf1237');
mysql> INSERT INTO message VALUES ('1178858071115','asdf1238');
mysql> SELECT * FROM message;
5. Create a user account with access to the database.
mysql> GRANT ALL PRIVILEGES ON loadtest.* TO 'loadrunner'@'%' IDENTIFIED BY 'loadrunner' WITH GRANT OPTION;
Note username/password is loadrunner/loadrunner.
mysql> SELECT * FROM mysql.user where User = 'loadrunner';

Verifying external connectivity to the database

1. Download an SQL query tool to verify that you can successfully connect to the database.
   I recommend that you check out SQLeonardo.
2. Download the JDBC driver for MySQL.
   MySQL now produce their own JDBC driver (Connector/J)
3. Configure the SQL query tool.
   nickyb (the creator of SQLeonardo says)…

   Suppose you have downloaded and unzipped sqleonardo into c:\sqleonardo and the mysql jdbc driver into c:\sqleonardo\mysql

   Run SQLeonardo and into the “metadata explorer” do:
   - choose the menu “actions>new driver…”
   - check “add library (browse filesystem)” and click “next >”
   - select the jar file into c:\sqleonardo\mysql and click “next >”
   - type into the textfield named “name:” => MySQL
   - select into the combobox named “driver:” => com.mysql.jdbc.Driver
   - into “example:” => jdbc:mysql://[host][:port]/[database]
   - click ok.
   now you have registered the driver!

   Select the item “MySQL” appeared into the tree and:
   - choose the menu “actions>new datasource…”
   - replace jdbc:mysql://[host][:port]/[database] => jdbc:mysql://localhost:3306/
   - put username and password and click ok.
   now you have added your database profile.

   Select the item under “MySQL”…you need now to test the connection!

   So our database connection string would be jdbc:mysql://www.myloadtest.com:3306/loadtest (assuming the database is installed on the default port).

4. Test the connection.
   Try running SELECT * FROM message; in your SQL query tool.

Writing the code

1. Java.sql javadocs are here. There are numerous tutorials on the web, and you have my sample code (text, zipped). What more do you need?

Correlation is one of the fundamental LoadRunner scripting skills; and LoadRunner novices are usually not very good at it (which is expected), but people who think they are LoadRunner experts are sometimes not very good at it either.

Mercury has done just about everything it can to remove the need for complicated correlation (correlation rules, the “scan script for correlations” option in VuGen, HTML mode recording, the Click and Script vuser type), but there will always be web applications that will require you to perform manual correlation when scripting.

Mercury’s training material kind of glosses over correlation, and makes it look overly easy – the correlation exercise from the training material never gives any of my students any problems; but if the training material were to include difficult exercises, it would be necessary to spend another day, and more people would be unable to complete the exercises.

I usually tell my students that they will encounter some much more difficult to correlate applications than the Mercury Tours website, and they will need to spend some time improving their manual correlation skills using WDiff .

This exercise should really test your correlation skills. Every problem (or something very similar) has been seen “in the wild” while I have been creating scripts for LoadRunner. The first exercise is the same as the exercise from Mercury’s VuGen 8.1 Scripting for the Web training material.

Correlation exercise coming soon…

Until the, have a go at the previous scripting exercises:

• Scripting Exercise: Credit Card Challenge
• Scripting Exercise: A basic AJAX application
• Scripting Exercise: Unique Usernames