Your Workload Model is probably wrong

First off, the good news: if you are load testing based on a defined Workload Model, you are already waaaay ahead of a lot of performance testers. I am frequently horrified by testers whose approach is…well, just not very scientific. When their test case is “we’ll just throw some load at it and see what happens” or their load test has been defined in a way that is vague and imprecise, like “our test is to run load with x concurrent users”, then the probability of them not knowing what they are doing approaches 100%, and it is clear that [...]

February 16, 2018|

The Top 7 Mistakes in Performance Testing

This is my talk on the Top 7 Mistakes in Performance Testing that I delivered at the first Neotys Performance Advisory Council in November 2017. My presentation can be downloaded from SlideShare, and Neotys will be making all 15 speakers’ slides available soon. [slideshare id=83352854&doc=neotyspac2017-top7mistakesinperformancetestingv1-171204222626] […]

November 17, 2017|

Neotys Performance Advisory Council 2017

Recently, Neotys (the company responsible for NeoLoad) held their first Performance Advisory Council in Scotland. Their idea was to get 15 of the top performance testers together to talk about some of the trends that are influencing the future of the industry, such as IoT, AI, shift-right, and DevOps. Performance testers traveled from the U.K., France, New Zealand, Germany, USA, Australia and India to share their ideas on the the changes they are seeing to load and performance testing. Holding the event at Borthwick Castle (built in 1430) gave us a picturesque (if slightly incongruous) backdrop to our discussions about [...]

November 16, 2017|

Quality Software Australia Conference 2017

I will be speaking at the upcoming Quality Software Australia conference in Melbourne on May 11, 2017. Those who plan to attend the conference can look forward to presentations from local and international thought-leaders on devops/QAops, CI, testing microservices, test automation, and other topics of interest for people who care about software quality. My presentation on Web Performance Disasters will cover some of the more well known IT failures in recent years, along with some ideas for preventing similar disasters from happening in the future. […]

March 24, 2017|

New Swag: MyLoadTest USB sticks

I have just placed an order for a large number of 8GB USB 3 memory sticks with a MyLoadTest logo on them. The USB sticks are available for clients, and for performance testers who help us out by finding bugs, by contributing to the web performance community, or by referring work to MyLoadTest. Don't forget that there are also MyLoadTest mugs and postcards.

March 24, 2017|

mmdrv.exe command line options

If you are running LoadRunner scripts from the command line using mmdrv.exe, then you will know that mmdrv.exe has a large number of command line options. Unfortunately, the options are not really documented apart from what is displayed in the user interface. Read on for some hints… […]

January 7, 2017|

SQL queries for LoadRunner Analysis graphs

LoadRunner Analysis is a powerful tool for understanding exactly what happened during a load test. You can use it to slice and dice your performance test results data, then you can export selected graphs/charts for your Performance Test Summary Report. Some people may want to use a different tool to create charts from their LoadRunner test results. Fortunately, test results data is stored in a database, and there is an SQL query behind each graph/chart you see in LoadRunner Analysis. If you understand the LoadRunner Analysis database tables, you can write your own queries (unfortunately you can’t do this directly [...]

December 8, 2016|

New Relic vs. AppDynamics

In recent years, IT projects seem to have stopped asking “which APM solution should we buy”, and have started asking “should we buy New Relic or AppDynamics?” Given the speed at which these two companies are innovating, the many product comparisons available on the web quickly become outdated. This comparison is a little more “high level”, so hopefully it will help give people some context for further research for several years to come. Please read some (recent) technical comparisons too, and take advantage of the free trial periods to install and try both products. […]

January 26, 2015|

Static code analysis for LoadRunner scripts

The idea of static code analysis has been around since at least the 1970s. These days, some kind of static analysis is usually built into most good quality IDEs (or it is available as a plug-in). Static analysis tools find bugs in software by looking at the program’s source code rather than executing the program. While the compiler will pick up syntax errors, there are thousands of other types of errors that a developer can accidentally create while programming – and, unlike compiler errors, there is nothing to tell you that you have made a mistake unless you are using [...]

December 19, 2014|

Performance Testing in the Cloud

This is my presentation on Performance Testing in the Cloud that Chris De Lorenzo and I delivered at the Iqnite Australia 2014 conference on October 16th. The slides from Chris on the state of performance testing at Tabcorp (one of the world’s largest online betting companies) were excellent. The original presentation can be downloaded from SlideShare. [slideshare id=41836324&doc=performancetestinginthecloud-141120213134-conversion-gate02] […]

October 19, 2014|

Iqnite Australia 2014

This year was the second time that I have spoken at the Iqnite Australia conference. This year, Iqnite Australia was was held from October 16-17 at the Melbourne Convention and Exhibition centre. The conference was a great opportunity to hear from experts in the field of software testing and to meet with other practitioners. Over 200 software testing professionals and vendors came to the conference to discuss the latest industry trends. […]

October 18, 2014|

Performance Testing memes

Image macros – those pictures superimposed with vaguely humorus text – are everywhere on the Internet. I decided that it was time for someone to create some that performance testers could relate to; something we could all print out and pin to our cubicle walls. To get everything started, I have created 8 images. Maybe they will become successful memes and help spread performance testing culture across the world…or maybe they will just generate a chuckle or two. If you create your own performance testing-related image macro, please leave a comment (with a link), or send me an email. I [...]

October 15, 2014|

Stuart has resigned from JDS Australia

Today was officially my last day at JDS Australia. I worked at JDS from November 2nd, 2005 to August 22nd, 2014 (8 years, 10 months). I was employee number 6, and was the first performance tester to be hired. Over the years the company grew to over 60 people in 4 states, and it was great to be involved in helping the business grow. I am looking forward to seeing continued growth, and watching the business continue to help customers.

August 22, 2014|

.gitignore file for LoadRunner

If you are version controlling your VuGen scripts using Git, then you will want to prevent certain files from being added to your Git repository (e.g. replay logs, binary files outputted by the compiler, etc.) Adding a .gitignore file to your repository allows you to specify files or directories that should not be tracked by Git. I have created a LoadRunner-specific .gitignore file to get you started… […]

November 30, 2013|

Iqnite Australia 2013

I am quite excited about the upcoming Iqnite software testing conference which will be held in Sydney in October. The conference programme is full of interesting sessions, and it is always stimulating to be surrounded by a couple of hundred people who are all interested in software testing. I will be speaking at Iqnite in one of the sponsor sessions (thanks HP). I hope you will come along and say hi. […]

September 11, 2013|

Useful Tools for Performance Testers

On August 23rd, 21 JDS performance testers got together for their quarterly breakout session to share tips and tricks with each other. One of the breakout topics was Useful Tools for Performance Testers. This was about useful tools that were not load generation tools (or everyone would have just said “LoadRunner” and it would have been a very short session). I have included everyone’s contributions in this post… If you are a performance tester and have a different tool that you find useful in your job, please add it in the comments below. […]

August 24, 2013|

Should Performance Center and ALM be merged?

Since the release of Performance Center 11.0, the product has been based on the HP Application Lifecycle Management (HP ALM) platform (i.e. Quality Center). This architectural change means that functional testers and performance testers can share a combined testing platform – a single repository for requirements, test cases, and defects – giving full visibility across the functional, non-functional and performance testing areas. It sounds great, but there are pros and cons to combining Performance Center and ALM/Quality Center… […]

August 1, 2013|

HP Discover 2013 (Las Vegas)

This year HP Discover (Las Vegas) was bigger than ever, with more than 11,000 attendees and nearly 700 sessions over 3 days. HP Discover is a great place to learn more about HP Software products, meet some great people, and get a peek at the product roadmaps for the next year. I didn’t make it to the conference this year, as JDS asked that I not submit a presentation abstract. Instead, I have collected all the sessions from the Software (Quality and Performance Validation) track that I thought looked interesting, and that might be interesting for other performance testers. [...]

June 15, 2013|

Beware of treating “Non-critical resource errors as warnings”

One of the most poorly understood VuGen Runtime Settings is the one that allows a performance tester to treat “non-critical resource errors as warnings”. The majority of LoadRunner users I have spoken to do not realise that enabling this setting can hide serious load-related defects in the system under test. LoadRunner automatically detects HTTP 4xx or 5xx responses but, if this runtime setting is enabled, LoadRunner will ignore the HTTP 4xx or 5xx error code when requesting a resource (i.e. the current transaction will not fail). […]

April 2, 2013|

LoadRunner Scripting Standards

Whenever a team of performance testers grows larger than three people, then it is time to assemble the team and create some LoadRunner scripting standards (or guidelines if you prefer to be less strict). Documenting your team’s scripting standards helps junior performance testers avoid common mistakes and follow best practices. Experienced performance testers benefit from having consistency between scripts, so there is less confusion when work is transferred between team members. I have developed a short quiz so you can test your knowledge of some of the VuGen scripting best practices. […]

February 20, 2013|

LoadRunner Password Encoder and lr_decrypt()

If you ever need to disguise a password in a VuGen script, you will no doubt have used the lr_decrypt() function. If you have stopped to think for a second or two, you will have realised “encrypting” the password in your script doesn’t make it more secure in any meaningful way. Anyone with access to the script can decode the password with a single line of code: // As a security measure, LoadRunner password encryption is about as effective as rot13. lr_output_message(“The secret password is: %s”, lr_decrypt(“50de7ec44d2ff7033c2fcb9e5cf7307799d7”)) So, if this feature doesn’t really make anything more secure, why was it [...]

December 29, 2012|

Bad Performance Testing

As a performance testing consultant, I get to see a lot of the work that other performance testers do…and a lot of the time it horrifies me. If performance testing was a licensed profession, like law or medicine, then it would be necessary to revoke the licenses of 90% of testers. Their work is not just “low quality”, it is actually wrong or misleading, or based on such a shaky foundation that any predictions made from the test results are tenuous at best. Bad performance testers frequently treat their testing as a ritual, rather than as a science and they [...]

December 22, 2012|

Performance Testing in a Mobile World

This is my presentation on Mobile Performance Testing that I delivered at SoftEd Fusion 2012 (Sydney) on September 13th. The original PowerPoint file can be downloaded from SlideShare. [slideshare id=14380290&doc=fusion-2012-performance-testing-in-a-mobile-world-by-stuart-moncrieff-120921200026-phpapp02] […]

September 15, 2012|

Checking text in a PDF file with LoadRunner

A friend was performance testing a new SAP system that had a self-service portal where employees could view and download a PDF version of their payslip. He asked me how he could add a check to his LoadRunner script to make sure that the information in the PDF was correct (employee name, pay grade, etc). Manually checking all the PDFs at the end of the run was not a scalable solution (and just checking a sample of them would have missed any intermittent errors). Fortunately, it was a simple matter to write some code in VuGen that would solve his [...]

July 28, 2012|

Errors are bad

It seems unnecessarily obvious to even bother making the statement that “errors are bad”; surely this is an idea that everyone agrees with, like “crime is bad” or “you shouldn’t put your underwear on backwards”. But a lot of performance testers that I have worked with don’t seem to care too much about windows errors. A senior performance tester with more than 15 years experience at some of the largest companies in Australia recently said to me:   “Yeah, as long as the error rate is below 5%, I don’t bother putting it in my report.” Are. You. Kidding? How [...]

July 14, 2012|

Does LoadRunner need Version Control?

Since I wrote about my ideas for VuGen add-ins, a few people have spoken to me about their need for an add-in that would integrate VuGen with their revision control/source code management system. The interesting thing is that when I ask what version control features they are missing, they don’t have any specific features in mind (unless you count “backups”, which has nothing to do with version control). Most fall back on “VuGen scripts are source code, and all source code should be managed in a version control tool”. Well, VuGen 11.50 allows basic version control features (check in/out, version [...]

July 7, 2012|

SoftEd Fusion 2012

I am pleased to let you know that I will be speaking at SoftEd Fusion 2012 in Sydney this September. Fusion is a new conference organised by Software Education, which combines their STANZ testing conference and their Software Development Conference (SDC). Fusion will run over 2 days, with sessions on Agile methodologies, Enterprise Architecture, Cloud testing, business analysis, test automation, performance testing, and project management. There should be some great learning (and networking) opportunities, so this will be one of the best ways you could spend your training budget this year if your job involves creating software at large companies. [...]

June 29, 2012|

What’s new in LoadRunner 11.50?

LoadRunner 11.50 was released on June 5th, during HP Discover 2012 in Las Vegas. This release was originally going to be called LoadRunner 12.00, but this would have put it out of step with the version numbers for ALM/Quality Center and QTP, which might have created some marketing confusion. Significant R&D effort has been spent on VuGen, with only bug fixes (or very minor functionality) for LoadRunner Analysis and the Controller. Those of us who participated in the Beta program have already been playing with the new VuGen for the last 9 months, and should be fairly familiar with these [...]

June 23, 2012|

Extending VuGen 11.5 with custom add-ins

This is my presentation on the exciting potential of VuGen Add-ins that I delivered at HP Discover 2012 (Las Vegas) on June 7th. The original PowerPoint file can be downloaded from SlideShare. [slideshare id=13313138&doc=hp-discover-2012-extending-vugen-with-custom-addins-by-stuart-moncrieff-120613130833-phpapp02] Presentation abstract: HP has redesigned the latest version of VuGen from the ground up with a new, flexible architecture. For the first time, users will be able to add their own features to VuGen with custom add-ins. Join this session and learn the best techniques for developing these add-ins, with a demonstration and dissection of the VuGen Script Validator developed by JDS Australia. Whether you would [...]

June 9, 2012|

“Voodoo Scripting” in VuGen

Every time someone uploads a script to the (web-based version of the) VuGen Validator, I get emailed a copy of the validation report for their script. Needless to say, I see a lot of really bad scripts…and hopefully the script developers learn something from the failing grade their scripts are given by the Validator. One thing that worries me is that lately I have seen a lot of BPM scripts containing the following code… Action() { web_cache_cleanup(); web_cleanup_cookies(); web_set_sockets_option(“CLOSE_KEEPALIVE_CONNECTIONS”, “1”); // script body removed from example. return 0; } […]

May 26, 2012|

Handling consumable parameter data in LoadRunner

Imagine that you are load testing the website for a new loyalty card scheme. Millions of people will be mailed a loyalty card, and they will need to register their details online, along with their card number. Each card number may only be used once. So, every time you run your test, you will consume a large amount of card numbers, and you must be sure to use unregisterd card numbers for each test. […]

May 12, 2012|

Ideas for VuGen Add-ins

You might have already heard that it will be possible to add your own features to VuGen when version 11.5 is released mid-year. I will be talking about this in more detail in my “Extending VuGen 11.5 with Custom Add-ins” talk at HP Discover 2012 in June. In the meantime, I have decided to share some of my add-in ideas. Please feel free to develop these add-ins yourself. After all, ideas can’t be copyrighted, and I would prefer to buy or download someone else’s add-in rather than spend time developing them myself. Here are my ideas… […]

May 5, 2012|

HP Discover 2012 (Las Vegas)

This year HP Discover will run from June 4-7 at The Venetian in Las Vegas. The conference will cover HP’s (and partners’) solutions for software, hardware, services, and networking. The hot buzzwords appear to be: cloud, mobility, and DevOps, with not so much Agile this year and only one mention of Web 2.0. As I am really just interested in topics related to Performance Testing, I have made a shortlist of the sessions relevant for people like me, so you don’t have to wade through the session catalogue (which has more than 800 sessions). There are quite a few interesting [...]

April 28, 2012|

LoadRunner vs. Performance Center

Recently a client asked me “What’s the difference between LoadRunner and Performance Center? They’re both sold by the same company, but they seem to do much the same thing.” I think he could see that I was about to launch into a half-hour lecture on the topic because he quickly added “Just give me the elevator pitch”. I thought for a second and explained “Well, obviously they are both used for performance testing, but Performance Center is the Enterprise version of LoadRunner; so it suits larger companies who need to run performance testing for multiple projects at the same time [...]

November 5, 2011|

Sending SNMP Traps with LoadRunner

Imagine that you need to load test a Network Management System (NMS), which might receive hundreds of SNMP traps every second from tens of thousands of devices on the network. Clearly it is impractical to actually deploy thousands of devices in a test environment, so you need a tool to generate the SNMP traps at the required volume. A quick search for SNMP load testing tools shows that they are all fairly primitive, with poor support for creating SNMP traps with dynamic field values, and no support for integrated system monitoring. There are definitely no tools with the same level [...]

May 1, 2011|

Running remote Unix commands from LoadRunner

If you need to run a Unix command on a remote server with a LoadRunner script (running on Windows), then there is a neat program called plink that you will find very useful. If you use any variety of Unix, you will probably already by using PuTTY; well plink is is distributed with PuTTY, and allows you to run remote Unix commands from your DOS prompt. In this example, I have used it to collect disk I/O metrics from a remote Solaris server. […]

August 24, 2010|

Running command-line programs from LoadRunner

Everyone should already know that you can run a command-line program from a LoadRunner script using the system() function. If you have used this, you would already know that things become a little awkward when need to capture the output of the program. The common solution is to pipe the output of the command to a file (i.e. system(“my_command > output_file.txt”)), and then read the contents of the file. This is clunky and awkward, and not safe when multiple virtual users are writing to the same file. A much better solution is to run all your command-line (DOS) programs using [...]

August 23, 2010|

Geeky Books Available for Loan

I had always wanted to work at a company that had a library of intereresting/educational books I could borrow; so when I started at JDS Australia in late 2005, I brought in a few of my technical books in case anyone else wanted to borrow them. Gradually other people brought in books too, until there was quite a nice collection. I like to think that consultants who have read Death March and The Mythical Man Month will be a little wiser than their brethren who have just flicked through Learn x in 24 Hours. The lives of technical people will [...]

August 15, 2010|

Processing DHCP logs with AWK

On a recent project, I needed to run a performance test where tens of thousands of network devices communicate with a DHCP server. After the test, I wanted to create a pretty graph of the number of DHCP requests hitting the server. Using a combination of AWK and Excel, I had a solution within half an hour… […]

August 10, 2010|

Java Record-Replay with LoadRunner

On increasingly rare occasions, you might be required to load test an application that has a Java-based client. While it’s nice for users to have an application that has a rich GUI, it will probably lead to a world of pain for you as a performance tester. Don’t immediately reach for LoadRunner’s Java Record-Replay virtual user type. It might not be the right solution. This article will guide you through your first steps when selecting the correct vuser type to test an application with a Java-based client… […]

July 29, 2010|

LoadRunner Syntax Highlighting (for your blog or forum)

I like posting LoadRunner code snippets on My Load Test and, judging by the emails and comments that I get, a lot of people find them really useful. To make my code more readable, I have added syntax highlighting that uses the same colours you see in VuGen. If you would like to download a copy of the syntax highlighting code to use on your own blog or forum, or you would like to learn more about how I did it, read on… […]

July 25, 2010|

HP Software Universe 2010 (recap)

I just got back from the US after presenting at HP Software Universe 2010. I will save the content of my presentation for another time, so this is a quick overview of what I saw at the Conference and some of the things I learned from the people who attended. […]

June 29, 2010|

I will be presenting at ANZTB 2010

On Tuesday March 2nd, I will be presenting on “The Non-Functional Requirements Every Project Forgets” at the ANZTB 2010 testing conference in Melbourne, Australia. Please come and say “hi”. […]

February 28, 2010|

What’s new in LoadRunner 9.51?

LoadRunner 9.51 is a patch for for LoadRunner 9.50 which adds some minor functionality (including SiteScope 10.0 compatibility) and, according to the release notes, fixes bugs raised in 48 service requests. Readers interested in the major additions to the most recent major release of LoadRunner should read the Tech Tip on new features in LoadRunner 9.50. […]

July 8, 2009|

How to find out what WordPress plugins a site uses

One of the first steps of any web application security audit is to determine what software is installed, which allows you to search for known vulnerabilities at websites like Bugtraq. The Weblog Tools Collection blog recently had a post asking readers to guess how many WordPress plugins they had installed, with the prize being a pro account on Flickr. I'm not a big fan of guessing games, so I thought I would write a simple VuGen script to find the answer. Using the list of plugins taken from my post on content scraping, I got a list of all the [...]

May 16, 2009|

Content Scraping with VuGen

As I’ve said before, VuGen makes a great content scraping tool for cases when you want a quick and dirty script to save specific data from multiple webpages. In this example, I wanted to create a list of all the WordPress plugins available from http://wordpress.org/extend/plugins/ (currently there are 4,245), and save all the metadata about the plugin: Number of downloads Version number Rating etc… […]

May 10, 2009|

Changing Pacing time while a test is running

You might have noticed that once you have started your load test, it is possible to increase the number of transactions per hour by adding virtual users, but not by changing the pacing time. The VuGen Pacing runtime setting is only read at the start of the test so, to change it, it is necessary to stop the test, then make the change and restart. Fortunately (if you need to), you can write code to control the pacing of your virtual users instead of using the runtime settings. If you write code to pick up a pacing setting from a [...]

February 23, 2009|

Checking open ports with Telnet

When requesting that firewall ports be opened, it is useful to be able to check that they really have been opened using the simplest method possible. Using a telnet client is the perfect way to do this, as it comes standard on Windows and all Unix-like systems. […]

December 11, 2008|

Cross-Site Request Forgery Whitepaper

Princeton researchers released a research paper yesterday which documents Cross-Site Request Forgery vulnerabilities in 4 well known commercial websites (ING Direct, NY Times, Youtube, and Metafilter). It makes for interesting reading. Wikipedia article on Cross-Site Request Forgery Jeff Atwood of Coding Horror has a blog post on CSRF CSRF FAQ at CGI Security

September 30, 2008|

Testing WebSphere MQ with LoadRunner

There are a couple of options for testing a WebSphere MQ-based system using LoadRunner, but this blog post shows my preferred approach: write some simple Java code to put a message on a queue (and get a message from a queue). Note that this approach requires that you have a Java-based LoadRunner license such as the Java Tempate vuser or the Java Record-Replay vuser. You are not likely to need many virtual users, even if you want to generate a lot of messages. If you need to test a message-based system, you might also be interested in reading about testing [...]

August 7, 2008|

Why HP is wrong about the Web Security Lifecycle

In my first post I gave a very brief overview of the web security testing products offered by HP. Unfortunately people's understanding of where the products should fit into the software development lifecycle is still weak. This is even the case inside HP. Here is a current slide from HP Software... The obvious, glaring problem with this diagram is that WebInspect is being promoted as a tool to be used in Production. As someone who has spent a long time working in highly technical areas of testing, I have some huge problems with this, and most of my clients will [...]

June 5, 2008|

Web Macro Recorder does not record

Something that confuses new users to WebInspect is that the Web Macro Recorder will not record any requests to or http://localhost. This can cause much head scratching for someone who just wants to try something out on their local machine. The simple solution to this is to add an entry to your hosts file, and use the alias instead of localhost. So (for those who need me to spell it out for them), to record a macro for the HP WebTours website on my local machine (, I added the following line to my hosts file (C:WINDOWSsystem32driversetchosts)... # HP/Mercury [...]

June 3, 2008|

My car as an analogy for web security

I live in a bad neighbourhood...well, okay, not a really bad neighbourhood but its close proximity to the CBD and its abundance of funky bars, galleries and restaurants is neatly balanced by a high concentration of government housing that seems to go hand-in-hand with junkies panhandling outside the supermarket, groups of people drinking in the street during the day, and the occasional stabbing near the public housing estate. Anyway, living in my neighbourhood means that people keep trying to break into my car. I usually know if they have been successful when I find my car unlocked in the morning [...]

May 17, 2008|

Seven Web Application Security Myths

HTTPS will protect your application. Just because a user's browser displays a lock icon when they visit your website, doesn't mean that your website is secure. HTTP over SSL (HTTPS) only encrypts the traffic between the user and the web server, which prevents snooping of the user's traffic. It does not prevent a user sending malicious requests to the web server. A firewall will protect your application. Firewalls are great; you can lock down every non-essential port on your server. But you can't prevent people from accessing port 80 (or 443) or you will have the worlds least-used web server. If port 80 [...]

May 4, 2008|

Akamai – State of the Internet

Akamai, who provide transparent mirroring of web content for high-traffic websites, see a large sample of Internet traffic. They have started to report on some of the trends they see in their traffic patterns (while some not so subtle points about how good they are). Their first report covers Q1 2008 (January - March). The report is available from http://www.akamai.com/stateoftheinternet/ (registration required). Here are some of the security-related things that I found interesting Approximately 2% of all inter-domain Internet traffic was DDoS traffic (does not include spam, phishing, scans or other malicious traffic). The "Anonymous" DDoS attack on the Church [...]

April 4, 2008|

WebInspect Scan Signatures

WebInspect is definitely not a stealthy tool; and that's fine, because you shouldn't be secretly auditing anyone's website. Here are a few of the signs that WebInspect leaves when doing a crawl and audit of a website. WebInspect Scan Signature: The webinspect scan signature is a request that webinspect sends to the server with the text SCANNED-BY-SPI-DYNAMICS-WEBINSPECT-WWW.SPIDYNAMICS.COM. This will be found in the webserver logs, identifying that a scan has taken place. In the web server logs, the request may look something like this... - - [03/Feb/2008:18:44:08 +1000] "GET /----SCANNED-BY-HP-WEBINSPECT---- HTTP/1.1" 404 -1 "" "Mozilla/4.0 (compatible; MSIE 7.0; Windows [...]

February 4, 2008|

Google Hacking for Penetration Testers (free e-book download)

Super-fast posting mode... Slashdot review of Google Hacking for Penetration Testers The book on the publisher's website http://www.syngress.com/catalog/?pid=3150 Google Hacking database (from the author) http://johnny.ihackstuff.com/ghdb.php Google Hacking on Wikipedia http://en.wikipedia.org/wiki/Google_hacking Author's website http://johnny.ihackstuff.com/ Direct download of the PDF of the book (33MB) (removed) http://www.scribd.com/word/download/319798?extension=pdf Flash-based online reader of the book (removed) http://www.scribd.com/doc/319798/Google-Hacking-for-Penetration-Testers New version of the book to be released sometime soon Update: a new version of the book has been released, and is available from Amazon.

January 2, 2008|

Question: Using standard web vusers to test web services

Subject: HTML return value Hi Stu, you probably know the answer, what I need to know if there is an option in Vugen to run a url and to save the url content to a string, something like XML apis in Vugen. Later I can parse the string to see if I got the right value. There is surely an add-on to web_url to save the return value to a string. and I need to save it as a variable in Vugen and not writing it to a file. Tell me if you know if it achievable. Thanks Txxxx Here [...]

December 28, 2007|

Question: SAPGUI script fails on second iteration

Subject: Help wanted in SAP with LR.. Hey Stuart, I Guess u have got good experience in LR with SAP... I got into a problem in my work place.. hope u can figure it out.. pelase let me know if u have a solution for his.. I created a Script for R/3, ( Order to Cash ) which will have 10 materials in a single transaction.. When i run the script with 1 Iteration, it is successfull. But when i run it with 2 Iterations,, its bouncing from the program.. This happens even in Controller... So please let me know [...]

December 28, 2007|

Question: Monitoring and alerting with VuGen?

I frequently receive emails with LoadRunner-related questions. I try to reply to questions when I have time. Maybe posting my answers to MyLoadTest will be an incentive for me to answer more questions, and maybe it will save other people some time. Subject: Sending email from loadrunner script Hi Stuart, We would like to run a LoadRunner script (HTTP WEB Protocol) which will send an email to the user whenever the transaction response time for web page is more than the specified time, or when the application is down for some reason. Basically we would like to monitor the application [...]

December 28, 2007|

Question: How to record a JDBC application with VuGen?

Subject: LR JDBC Protocol Hi Stuart, I have a Java Swing-based client that communicates directly with a database server using JDBC. What are my options for recording this application with LoadRunner Thanks, Jxxxx Cxxxx. As you probably noticed, there is no native vuser type which will record your application successfully. Fortunately, you still have several options for recording your application. Option #1 - Winsock (a bad option) If you record your application with a Windows Sockets vuser, you will see uninteligible data being sent and received. Just because you can record something with a Winsock vuser doesn't mean it is [...]

December 11, 2007|

WebInspect free download (15-day trial)

For those who want to have a look at what WebInspect can do, there is a free 15-day trial available. Note that you will need to provide a valid email address to receive your trial license key, and you will also need to install SQL Server 2005 or SQL Server 2005 Express Edition (free download from Microsoft [36MB]). There is a test website you can scan for vulnerabilities at http://zero.webappsecurity.com, which is fortunate because this is the only website you can scan with your trial license... Also, you may wish to turn off some of the resident protection software that [...]

December 8, 2007|

HP acquires SPI Dynamics

On June 19, HP announced in a press release that it had signed an agreement to acquire SPI Dynamics, a software company specialising in web security testing tools. The deal was finalised on August 1, and it is only now that theses new tools are reaching the wider pool of technical people (like me) at HP partner companies. Purchasing SPI Dynamics nicely compliments HP's November 2006 purchase of Mercury, who were best known for their defect management, functional test automation and performance testing tools. IBM has a similar portfolio in the software testing space, with the Rational products and their [...]

December 3, 2007|

New book on Performance Testing

I've spent the last year or so writing notes and fleshing out chapters for a book called Performance Testing Web Applications, so imagine my very slight feeling of annoyance when I did a google search for my book title and found that someone had released a very similar book 3 months earlier... J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea have collaborated on a book called Performance Testing Guidance for Web Applications, which is available either as a free download, or in dead-tree format through Amazon. The book is really good, so I highly recommend that you [...]

November 28, 2007|

James Bach talks to MAST

James Bach visited Melbourne this week to teach his Rapid Software Testing course. While he was here, he took the time to talk to about 30 developers and testers from the Melbourne XP Enthusiasts Group (MXPEG) and the Melbourne Association of Software Testers (MAST). James is an accomplished speaker and is passionate about testing. He spent the first few minutes railing against the traditional testing world that is "obsessed with techniquism and artifactism" - blindly following a testing process without thinking (which I guess I talked about a couple of weeks ago in my post on Cargo Cult Testing); and [...]

June 7, 2007|

Cargo Cult Testing (testing as ritual rather than science)

Richard Feynman popularised the idea of the cargo cult in his essay on cargo cult science, which was "science" that followed all the forms of scientific investigation, but lacked real critical scientific thought. The idea was transferred to the world of software courtesy of the Jargon File with its entry on cargo cult programming. The Jargon File defines cargo cult programming as: A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose. A cargo cult programmer will usually explain the extra code as a way of working around some bug encountered [...]

May 15, 2007|

Connecting to a MySQL database with LoadRunner

The Virtual Table Server is great for most situations where your virtual users need a common data pool, but the limitations of the API mean that it is a bad fit in some cases. Say you need to find a row in the data table with a specific value. On a real database you could just do a simple SELECT statement. With VTS, you would have to write code to iterate through every row in the table and check the column value each time. So sometimes you need to use a real database. Interfacing with a real database is not [...]

May 14, 2007|

Scripting Exercise: Correlation Challenge

Correlation is one of the fundamental LoadRunner scripting skills; and LoadRunner novices are usually not very good at it (which is expected), but people who think they are LoadRunner experts are sometimes not very good at it either. Mercury has done just about everything it can to remove the need for complicated correlation (correlation rules, the "scan script for correlations" option in VuGen, HTML mode recording, the Click and Script vuser type), but there will always be web applications that will require you to perform manual correlation when scripting. Mercury's training material kind of glosses over correlation, and makes it [...]

May 14, 2007|

The 10 Commandments of Load Testing

I have made a list of the top ten things load testers frequently fail to do that make me feel like smiting them. Thou shalt know how thy test tool works. The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic. Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application. Thou shalt gather realistic usage data. Garbage in, garbage out. [...]

May 9, 2007|

SOAP over JMS LoadRunner script

This is a quick walkthrough of a LoadRunner script I created to load test a "web" service that communicates with SOAP messages sent over JMS. Note that this script supports the queuing/point-to-point model of JMS (rather than the publish and subscribe model), works with JMS version 1.1 rather than 1.02b (as it uses the BytesMessage.getBodyLength() method among others), and has been tested with Java 1.5 using the Tibco implementation of JMS. The LoadRunner script can be downloaded here, the source code is available here, and a mirror of the JMS 1.1 specification can be downloaded here. People using Tibco's JMS [...]

April 30, 2007|

Java Thread Dump

A Java thread dump is a way of finding out what every thread in the JVM is doing at a particular point in time. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck. You can generate a thread dump under Unix/Linux by running kill -QUIT <pid>, and under Windows by hitting Ctl + Break. A great example of where this would be useful is the well-known Dining Philosophers deadlocking problem. Taking example code from Concurrency: State Models & Java Programs, [...]

April 10, 2007|

Free packet sniffer

Yet another application to add to your performance testing toolkit is Wireshark (previously called Ethereal). A packet sniffer like Wireshark will let you see your network traffic at all the different protocol layers. A naturally curious person might use it to investigate why the lights on their router started flashing all the time just after they installed that really useful piece of software that let them change their mouse cursor. As a performance tester, I use it when I'm trying to figure out what the application I am trying to record with LoadRunner is sending and receiving, and the VuGen [...]

November 9, 2006|

LoadRunner ContentCheck Rules

A LoadRunner feature that has made my life a lot easier has been ContentCheck rules, which are available in the script runtime settings. If you are using a web-based vuser type, you can configure your LoadRunner script to search through all returned pages for strings that match known error messages. Using web_reg_find functions is fine, but when you get an error LoadRunner reports it as "failed to find text" instead of something more descriptive. I will always create rules for any error messages I find during scripting and, if I receive an error while running a scenario, I will add [...]

November 3, 2006|

Free WAN emulator

Mercury has just announced that they will no longer be re-selling the Shunra WAN Emulator. This means that I can let you in on a little secret - you can get most of the WAN emulation features for free by using a simple open-source program. As performance testers, we know that an application operating over the network will perform poorly if there is not enough bandwidth available. We also know that response times for some applications are more affected by latency than others no matter how much bandwidth you have (eg/ an interactive multiplayer game like Quake is playable over [...]

November 3, 2006|

Using open source tools for performance testing (Google video)

I just found something interesting on Google Video; it is a 1 hour Google TechTalk presentation by Goranka Bjedov about Using Open Source Tools for Performance Testing. Goranka has some interesting things to say. She makes the point that there is really no standard terminology in performance testing circles, and goes on to prove this by giving her own definitions of performance, stress, load, scalability, and reliability testing. As an example of reliability testing she notes that "typically, when I was at AT&T, we would run for about a month at a time after everything was done just to find [...]

November 2, 2006|

Hacking into VMWare images

Last week I posted a question asking how I could recover or change the password for a VMware guest operating system (Windows 2000) that I had forgotten the password for. After receiving no useful suggestions, this week I allocated some time to solving the problem. Windows password recovery tools usually consist of a bootable CD image containing a version of Linux that will overwrite the NT password with a known value or will extract the hashed password from the filesystem. To boot your virtual machine from a CD, you must change the boot order in the virtual machine's BIOS. Press [...]

October 19, 2006|

Mercury Certified Instructor Exam and Workshop

The Mercury Certified Instructor qualification is not about having mastery over the Mercury toolset (although if you are sitting for the CI exam, you probably have this anyway); it is about ensuring that you have the ability to effectively deliver training material to a classroom of students. Officially, Mercury training must be delivered by someone who has a CI qualification and the appropriate CPC qualification for the tool that they are delivering the training for. In practice, there aren't enough Certified Instructors yet, so this is not enforced. After wanting to do it for a while, I completed the Mercury [...]

October 19, 2006|

Scripting Exercise: Unique Usernames

Imagine that you have to create a script for the "Create New User Account" business process. There are several constraints that will make your job difficult... The username field has the following properties: valid characters are a-z, A-Z, 0-9 usernames are not case sensitive. "Stuart" is the same as "stuart". usernames must start with a letter, not a number. usernames may be a maximum of 8 characters long. username must not be a duplicate Some other information about your testing: It is unclear when the database in the test environment will be refreshed, but it should happen at least yearly. [...]

October 17, 2006|

LoadRunner Script Runtime Settings

When running a training or mentoring session, people often ask what runtime settings they should use; as if there is a magical list of settings that will always be correct for any testing situation. Obviously you select runtime settings that are appropriate for what you are trying to achieve with your test, but the funny thing is that there are actually a small list of settings that are usually appropriate for most situations. Read on... Whenever I am using a vuser type that allows multiple actions in a single script, I will create a separate action for each business process [...]

October 15, 2006|

LoadRunner Vuser Types

As of LoadRunner 8.1, Feature Pack 3, the following Vuser types are available (note that descriptions are provided by Mercury). Application Deployment Solutions Citrix_ICA Represent the Citrix ICA protocol communication between the Citrix client and server as GUI events. Client/Server DB2 CLI The IBM Call Level SQL Interface to the DB2 family of databases. Domain Name Resolution (DNS) A system which translates domain names into internet addresses. Informix IBM's Informix Database servers. MS SQL Server Microsoft's SQL Server using the Dblib interface. ODBC Open Database Connectivity - a protocol providing a common interface for accessing databases. Oracle (2-Tier) Oracle database [...]

October 8, 2006|

SAP Cheat Sheet for Performance Testers

Performance testers must switch between different technology platforms almost every time they join a new project. Here are a few things you should know so you don't look like a complete n00b on your first SAP project. Transactions and Sessions Unless a restriction has been put in place, users may create multiple sessions from a single workstation. Using the menu, this can be done through System > Create Session or by left-clicking on the top right of the SAP client window and selecting the Create Session option. This menu is also the only place a transaction may be stopped if [...]

October 1, 2006|

Network Sociability Testing

If the application that you are performance testing will be operating over a WAN, it is really important to test whether the reduced bandwidth and higher latency of the network link will increase transaction response times to an unacceptable level. If you are being thorough, you should also run some tests to determine whether your response times will be impacted by other network traffic on the WAN (and vice versa). Let the following anecdotes about testing two different applications serve as examples: The first application was to be used by staff who would be interacting directly with customers. There would [...]

September 24, 2006|

Mercury Virtual Table Server (VTS 2)

The Virtual Table Server (VTS) is an in-memory database that can be used to share data between virtual users. I have found this useful when I need my VUsers to pass data to each other; it's certainly a much neater solution than writing to a file and having each VUser check the value in the file on every iteration. Here is what the documentation has to say about it... VTS Virtual Table Server (version 2.10) Virtual Table Server (VTS) first edition, introduced the capability for LoadRunner virtual users, WinRunner and XRunner, to communicate in real time. Data extraction and communication [...]

September 17, 2006|

Discovering your website usage profile with LogParser

It's always nice to base your load test on real-world usage patterns rather than someone's "best guess". If the system is used internally, Business Analysts are usually quite good at providing this information, but if the system is used by external users (like most high-traffic web applications), then they can typically only tell you "big picture" information; e.g. they know how many orders are created, but not how many searches are made per hour. Basic web log analysis tools provide low-level technical detail that isn't very useful - hits per hour. To be useful for load testing, we need to [...]

September 10, 2006|

Performance Testing Notepad

Performance testers sometimes get hung up on tool-based testing. Admittedly, most of the really interesting problems appear when there are more than a few concurrent users on a multi-user system, but it is important to verify that an application performs acceptably when it is not under load too. Even if response times seem fine during functional testing, the functional test team may not have thought to look for the effect of large datasets on response times. It's not just a case of "do we need to put an index on a table somewhere", but sometimes response times increase exponentially instead [...]

September 3, 2006|

Scripting Exercise: A basic AJAX application

A common question on the LoadRunner support forums is "I've got an application that uses this new thing called AJAX. Will LoadRunner work with my application?" ...and the answer is "yes...but...". The "but" is a big one . An "AJAX" application can mean anything from a simple webpage that updates some fields with information from the server (without reloading the whole page), to a horribly complicated rich GUI interface created with JavaScript. Of course LoadRunner is going to be able to handle anything that is sent over HTTP, but scripting might not be the usual simple Record-Correlate-Playback, and the chances [...]

August 28, 2006|

FreeQueryBuilder – a better SQL tool

Frequently moving between companies means that I sometimes find myself jumping from using Toad one week to Microsoft Query (yuck!) the next. Now that I have had the chance to use it on a couple of different projects, I think that FreeQueryBuilder is a pretty good replacement for most of the SQL tools that I have had to use; and it certainly saves the effort of learning a new interface every time I move projects. The best part of it is that it is shipped as a jar file, so you don’t have to install anything to use it - [...]

July 30, 2006|

Developing a Methodology

Sylvia is a very good cook. She makes a particularly fine roast ham, using a family recipe that begins, unusually, by cutting a section off both ends of the meat. One day, a friend dropped by while she was preparing the dish. "Why are you doing that?" he asked, as Sylvia carefully trimmed that ham, "Because that's the way my mother always does it," said Sylvia. But it got her wondering - why did her mother start that way? A few days later Sylvia was at her mother's house. "When you make that special ham, why do you start by [...]

July 23, 2006|

Performance Requirements and Jedi Mind Tricks

A long time ago on a project far far away, I saw something that gave me food for thought. Following an initial baseline performance test, the project was in a tuning cycle, with some very bright performance engineers doing everything they could to drag the application at least somewhere near the performance targets. The performance requirements they were trying to meet weren't unreasonable. They were building a replacement system that was almost functionally equivalent to the system it was replacing, and the requirements had been taken from a baseline test of the old system. As the tuning progressed, the lead [...]

July 11, 2006|

LoadRunner vs. Lode Runner

Useless trivia time... Back in 1983, Broderbund released a simple (and hugely popular) platform game called Lode Runner. Many years later, Mercury Interactive released a hugely popular load testing tool called LoadRunner. Notice that the animated figures look very similar. Do you think that this is subtle homage, or just a case of all stick figures looking alike?

June 17, 2006|

Mercury Diagnostics for SAP

These are the long-awaited slides from my Mercury World Australia presentation. If you are currently performance testing an SAP implementation with LoadRunner or Performance Center, it is worthwhile evaluating Mercury’s diagnostic tools. […]

May 15, 2006|

Mercury World Australia Presentation

I will be presenting at Mercury World Australia this year. If you are interested in hearing about my experiences using Mercury's diagnostic tools for SAP (both R/3 and NetWeaver), please come and see my presentation at 9am on the 12th of May. I will probably post the PowerPoint slides some time after the presentation. Update: slides are here.

April 16, 2006|

New LoadRunner Web (GUI level ) Vuser

There's a saying in software circles that "writing the first 90 percent of a computer program takes 90 percent of the time... The remaining ten percent also takes 90 percent of the time." A year ago, Mercury started telling people about their new GUI level virtual users for the web. Their promise was an impressive one - no more correlation. Anyone who has scripted for the web knows that correlating a web script can be either incredibly easy or painfully difficult if, for example, the web application constructs HTTP POSTs with an unholy mess of JavaScript. Mercury was promising to [...]

March 13, 2006|

Handling Captchas with a LoadRunner script

A question that regularly comes up in the Mercury Support forum is how to get a LoadRunner script to handle websites that require you to type in the characters from a picture of a funny-looking piece of text before you can perform an action such as creating a new account. "Surely someone has come across this before, and has figured out how to correlate these values?" Well...uh...no they haven't. These are Captchas - they are used to prevent bots from doing nasty things like signing up for thousands of Hotmail accounts or leaving comment spam. If it was possible to [...]

March 6, 2006|

Should you load test low-volume applications?

I don't ever like to feel that I am wasting a client's time by doing unnecessary load tests, but a long time ago I had an experience that altered my definition of what a necessary test is. I had come to this project after the scope had been decided and the Detailed Test Plan had been written. The overall system was made up of four applications; privately I queried why we were bothering to test ApplicationX, which was an exceedingly low volume application. ApplicationX was not business-critical or customer-facing. It was performing acceptably with a production-sized dataset and a small [...]

February 27, 2006|

Performance Testing university course

I have mixed feelings on learning that there is now a university-level subject in Performance Testing. CS 4803 EPR: Performance Engineering - Enterprise Computing Performance Engineering, presented by Brian Wilson at the Georgia Institute of Technology looks like a great subject, and it would be a lot of fun to do what Brian has done and run the same course at a local university. But, at the same time, I realise that doing this would be a clear case of fouling your own nest. Flooding a profitable niche industry with recent university grads who are prepared to work for practically [...]

November 8, 2005|

A useful addition to your LoadRunner function library

Okay, this is a bit of a tease because I haven't actually written this function, but it would be nice if someone out there would (maybe it will be me when I get some more free time). web_reg_save_param() is a useful little function for easily extracting text from the last HTTP response received by the virtual user. You specify left and right boundaries, the occurrence of the text you want to find and a large number of other options which, admittedly, don't give you the power of regular expressions but are a lot easier to use. The only problem is [...]

November 6, 2005|

LoadRunner Licensing Agreement

People don't generally read click-through licenses on the software they install. The only license I've read with a lawyer-like (or tester-like) attention to detail has been the GPL. The rest have just had a cursory check for any unduly onerous conditions...like the right to spy on my computer or the soul of my first-born child or something equally sneaky. The LoadRunner license has a couple of surprises in it. I'm not sure how enforceable the license would be, but it's probably in the best interests of anyone who uses Mercury tools that they are aware of the terms and do [...]

August 20, 2005|

WinRunner Coding Standards – Use Constants

I'm thinking of writing a WinRunner Coding Standards document (as if there weren't enough of them out there already), but with more of a focus on why things should be done a certain way rather than just dictating how code should be written. Here is a sample section. Is anyone interested in reading a document like this? No magic numbers Here's a quick quiz for you. What does the numeric argument to the tl_step function below set the test step status to? tl_step("Check that transaction completed successfully.", 0, "Transaction did not complete successfully."); If you answered "fail", you probably figured [...]

August 19, 2005|

String Manipulation in LoadRunner

I think that the following question from the Mercury Support forum neatly illustrates one of the major problems with LoadRunner. Hi all, I have a financial application that submits the data in $ amounts. When I looked in the extended log the value is in number format like (43251) but during submission it is converted into dollar format $43,251.00. How to convert the string into $ format? Is there any function? Thanks in advance, The majority of people using LoadRunner aren't great programmers, and the ones who are aren't necessarily great programmers in C. Fortunately, the level of skill required [...]

June 15, 2005|

Tips for Load Testing CORBA Applications with LoadRunner

Background I recently finished my first CORBA project using LoadRunner 8.0, and this is a quick brain-dump of some of my impressions. The official documentation does not contain all the information that I needed to know, and I found it hard to track down any people who had worked with the LoadRunner CORBA protocol before, so these notes are largely comprised of information that I wish I had known before I started. If you want to do performance testing of a Java client-server application using CORBA, LoadRunner seems to be the only viable option. CORBA is not supported by Rational [...]

June 13, 2005|

Mercury World Australia 2005

No one is ever going to argue that Mercury don't know how to throw a party. Mercury World Australia was a great opportunity to hear experts talk about the direction our industry is headed and ways that business can increase the value they get from technology but, more importantly, it was also an opportunity to network (or schmooze if you prefer) and enjoy their excellent catering. While I definitely wasn't going to miss the Gala dinner on Thursday, due to client commitments I only ended up seeing one presenter - Paul McLean from RPM Solutions and his talk on "Analysing [...]

May 14, 2005|

What you could do with a TSL parser for WinRunner

Transcribing an existing language to BNF and creating a parser for it isn't a trivial task, but it's not an insurmountable one either. A few years ago I started thinking about what you could do if you developed a parser for Mercury's Test Script Language (TSL) that drives WinRunner, one of their functional test automation tools. TSL Code Reformatter Check test scripts for conformity to project coding standards. Reformat existing test scripts to meet coding standards. Indentation, use of whitespace Complexity of compound statements Check for common errors or misleading code if (i = 1) { initialiseTest(); // this code [...]

May 8, 2005|

Web Server Logs

It used to be that there was a fairly good correlation between "unique visitors" in your web server logs and, well, unique visitors - real people who had actually looked at your web site. These days it seems that I get almost more junk traffic than real page views, with most of it being automated. Here's a list roughly ordered by volume. Referrer spam. Thanks to blogging software that automatically displays a list of the top linking pages on the front page, it is worthwhile for spammers to make hundreds of bogus page requests with spoofed HTTP referer headers (usually [...]

May 4, 2005|

Rational Performance Tester

I just received an evaluation copy of the new version of Rational Performance Tester. IBM Rational have re-written the tool from scratch and based it on the open-source Eclipse framework. The tool has only just come out of beta testing and currently supports web protocols only. Everyone in the industry seems to be keen to know how it stacks up against the competition. First off, my impressions of the old version... Coming from a LoadRunner background, my opinions on the tool were a little mixed. Performance Tester was easy to learn and easy to use but it is definitely missing [...]

May 3, 2005|

Parameters in LoadRunner Java protocol scripts

When working with any kind of LoadRunner Java-based vuser type, any string that appears within the script may be parameterised. This differs significantly from all of the C-based vuser types, where only functions that are labelled as "parameterizable" may be parameterised without explicitly calling lr_eval_string(). This is a simple example of parameterisation with a Java-based script: lr.save_string("jumped", "myParam"); String message = "the quick brown fox <myParam> over the lazy dog"; lr.output_message(message); Rather than trying to do something that is theoretically impossible such as overloading the String class to handle this extra functionality, the LoadRunner developers did the simplest thing that [...]

April 25, 2005|

Sign your name on everything

I recently heard an anecdote that reminded me of a tip from one of my favourite software-related books - The Pragmatic Programmer. In it, the authors advise readers to... Sign Your Work Craftsmen of an earlier age were proud to sign their work. You should be, too. Their emphasis is on taking pride in your work and accepting responsibility for it too. But the anecdote I heard showed that signing your work can be useful advertising too. The consultant telling me the story had just received a call from a company he had worked at many years ago, asking him [...]

April 24, 2005|

Performance Tuning Java Garbage Collection

I recently worked on a J2EE project where pauses due to garbage collection seemed to be causing some problems with response times. As the project was using the 1.4.2 JRE, they had four different garbage collectors to choose from. A short series of performance tests was conducted to examine their behaviour. Using the -verbose:gc JVM flag gave visibility into the behaviour of the garbage collector as it logged important information each time the garbage collector ran. [GC 325407K->83000K(776768K), 0.2300771 secs] [GC 325816K->83372K(776768K), 0.2454258 secs] [Full GC 267628K->83769K(776768K), 1.8479984 secs] Excel was used to create a graph of (roughly) heap used [...]

April 8, 2005|

WDiff – “file no longer available” problem

LoadRunner uses WinDiff to compare recorded scripts while manually correlating. I always wondered why WinDiff would throw an error when I invoked it from LoadRunner. It turns out that this only happens if your script directory contains spaces (like the default LoadRunner script directory does). WinDiff is showing its age and is not too good with long filenames or filenames containing special characters.

March 15, 2005|

Environment Availability

I am yet to work on a project where environmental issues haven't impacted my schedule to some degree. But it seems that I only start to document the outages once they start to seriously impact my schedule. This stops now. From now on I will document every outage (no matter how trivial) before they start to have any impact. Once the cumulative downtime reaches more than a few hours, I will include a link to the outage document as part of my weekly (or daily) status report. Here is a template for my Environment Availability Spreadsheet.

March 14, 2005|

How to write a performance test case

I have decided to release an early draft of this document so that others may provide feedback. Please let me know what you think. Writing test cases for performance testing requires a different mindset to writing functional test cases. Fortunately it is not a difficult mental leap. This article should give you enough information to get you up and running. First, lets set out some background and define some terms that are used in performance testing. Test case - a test case is the same as a use case or business process. Just as with a functional test case, it [...]

February 14, 2005|

Slush File

Here is a list of things I would like to write a short article on. They are either topics I am interested in or something I would like to be able to show to a client as background information that may save me from having to explain concepts related to performance testing. Estimating scripting time for performance testing How to write a Detailed Test Plan for Performance Testing How to write a Test Summary Report Maths for Performance testers (statistical significance, sample sizes, significant figures, queue theory) Preparing for a Performance Test - what the Business can do before the [...]

February 13, 2005|

Collecting performance stats with SNMP

All performance testing tools that I know of are bad at collecting system metrics from Unix-like system (AIX, Solaris, Linux, BSD etc). If you are lucky you might (might!) be able to collect CPU utilisation and maybe one or two memory utilisation counters. Generally you have to create a shell script to run perfstat (or similar) then, when your load test is over, you process the output file and import it into your analysis tool. Yesterday I learnt that you can write whatever data you want to an SNMP MIB so, if your performance test tool is able to gather [...]

January 23, 2005|

How to write a Performance Test Plan

I haven't written a huge number of Detailed Test Plans for performance testing, so I am still interested in reading other people's (non-confidential) documents or templates. If anyone out there has a template, I would love to have a look.

January 18, 2005|

WebScarab – a security test tool for web applications

Sometimes when creating performance testing scripts, you will stumble upon vulnerabilities in the application under test. Maybe you will miss a value you should have correlated before sending to the server, and when you go back and check your script you will find that (hypothetically) the web store is allowing you to purchase every item in their inventory for $9.95. This is the sort of problem that will never show up under functional testing, as the client application should always sent good (validated on the client side) data to the server. Once you expose your application to the world by [...]

January 9, 2005|

Web Site Mangement

I am looking for a no-cost tool (or tools) that will crawl my website and validate my XHTML and CSS, and check all of my links. If I had a real Content Management System(tm), I am sure that it would handle this sort of thing for me but, as I am not and this should be a common problem, there has to be a solution out there somewhere... Update: REL Link Checker Lite looks useful, but I am still looking for a total solution.

January 9, 2005|

Mercury Certified Instructor Course

Next on my professional qualification "to do" list is the Mercury Certified Instructor qualification. From the Mercury Australia website: The CI is an instructional certification intended for individuals who wish to teach official Mercury Interactive courses. This is a workshop-based class that focuses on understanding and using principles of learner-centred education to promote learning. The test at the end of the class will address your ability to: Create a safe learning environment that fosters teacher-student and student-student interactions Motivate participants Use effective learner-centred instructional techniques Demonstrate professionalism through effective presentation skills The course content looks good, and it means that [...]

January 8, 2005|

Unix utilities on Windows

Once you know how to use some of the Unix command line utilities, it is hard to go back to doing things the Microsoft way - where data manipulation seems to always be an awkward cut and paste/find and replace/filter with Excel. I have found that Native Win32 ports of some GNU utilities fills the gap nicely. It is a collection of some of the more useful utilities generally found on Unix, and the 3 MB zip file fits nicely on a USB key. Ideal for the times when you just want to grep through a file and don't want [...]

January 8, 2005|

Analysing Web Server Logs

Following a load test, I often need to perform some additional analysis on the web server logs. It is not practical to use any commercial tools, and the free tools are all aimed at people who want graphs and statistics for entire weeks or months, rather than a few hours of high load. So far, I have usually been forced into a roll-your-own solution. While it is easy to create graphs with Microsoft Excel, it's 65536 row limit makes analysing any sort of non-trivial load futile unless the log files have been filtered before importing them. Even with filtering, it [...]

January 7, 2005|

Combatting Comment Spam

Automated comment spam is a huge problem for anyone using any of the popular blogging programs. For anyone sick of comments about cheap Viagra or fake Rolexes, here are my suggestions. 1. Make it hard to identify your website as a weblog. The web is a target-rich environment; a Google search for mt-comments.cgi gives 3.7 million potential targets from one popular piece of software alone. As with any automated attack, being one of many vulnerable sites is not a protection. Ideally, you want to make it hard for spammers to find your blog without making it hard for everyone else. [...]

January 7, 2005|

LoadRunner CPC Exam

I flew up to Sydney yesterday for the privilege of being (as far as I know) the first non-Mercury employee to sit for the LoadRunner CPC exam in Australia. Feedback from people who had already taken the exam seemed to be almost universally along the lines of "there is a lot to get through. Know the subject thoroughly. You will not have time to look up documentation." Mercury representatives said, "If you could do your LoadRunner CPS exam, then you should have no problems with the CPC." I had previously completed my LoadRunner CPS and had been using LoadRunner for [...]

June 19, 2004|

Performance Creep

I got some interesting feedback from someone in the Risk Management group on one of my Performance Test reports. Like any good Performance Test report, it compared the new version of the application with the previous version. He said that he has seen a lot of cases where, over several versions of a piece of software, transaction times or resource utilisation has increased. The differences between versions are too small to worry about but, cumulatively, they may cause the performance of the software to be quite different to the original version. As a consultant specialising in Load Testing, I generally [...]

June 11, 2004|

Solution of the Week

I just received Solution of the Week on the Mercury Support Forum for a Knowledge Base article I posted in November. The article is in my archives for the curious.

June 8, 2004|

Top Twelve Tips for Running a Beta Test

I was recently asked whether I wanted to be part of the Quick Test Pro 8.0 Beta Program. Unfortunately I had to decline, but it got me thinking about beta testing in general. As a Test Consultant, it is rare to be involved in testing of software that is going to be shrink-wrapped and sold. Most engagements seem to involve the testing of an application that is to be used "in-house", whether that is a custom implementation of an existing application (as is seen with ERP/CRM apps), or an application that has been written from scratch. It's probably possible to [...]

May 26, 2004|

Bug Fix Bingo

There is someone who is talented at marketing working for Gold Coast software testing firm K. J. Ross & Associates. They partnered with Software Engineering Australia to create the Certified Software Test Professional course, ensuring themselves some valuable mindshare and a guiding hand in the careers of future Australian software testers. Another inspired, although not quite as influential, piece of work is Bug Fix Bingo. It's one of those "I wish I'd though of that" ideas which strikes a chord with every software tester. Visit the site and I guarantee you will want to print it out and pin it [...]

March 1, 2004|

QA Certification Programs

I'm looking at some general QA certification programs that are not tool-based certifications. Offerings in Australia are divided into two steams - the ISEB (delivered by several different companies and recognised internationally), and the CSTP (developed and delivered by K. J. Ross & Associates, and recognised in Australia only). ISEB Foundation Certificate in Software Testing Introductory level 3 day course (~$2000 - $3000) Internationally recognised Endorsed by the Australian Computer Society and the Institute of Engineers Australia ISEB Practitioner Certificate in Software Testing Builds on ISEB Foundation Certificate (which is a pre-requisite) 9 day course (~$9000) Not offered by most [...]

February 9, 2004|

Measuring Throughput On IIS

Some changes were made to a web application to reduce the amount of bandwidth used. To determine the impact of the changes, a previous benchmark test was compared with a new test. Both tests measured Bytes Received/sec on IIS, Bytes Sent/sec on IIS and Throughput (in Bytes/sec) from the perspective of the load testing tool. Comparing the tests showed that Bytes Received/sec had been reduced by 1/3 but the Total Throughput had been reduced by 2/3 (Bytes Received/sec was a trivially small value). Where were the missing bytes? Throughput: Displays the amount of throughput (in bytes) on the Web server [...]

February 7, 2004|

Unique Usernames in LoadRunner

LoadRunner Knowledge Base article 5218 "Generating Unique Names", posted by John Sawbridge, gives a function to generate a "unique" string [a-zA-Z] up to 20 characters long. The function accepts a string length argument and always writes the output to a parameter called {rname}. He notes "If you invoke the function successively in the same second, you will get the same name." Clearly this is a weak solution. As the function writes to the same parameter each time, it is not possible to use the function more than once in a LoadRunner script without overwriting your original value and if you [...]

February 2, 2004|

Document Classification Levels and Document Management

I was browsing the project share drive at work today looking for a design document that I needed. Every document that is created should, in theory, be given a classification; Comercial in Confidence, Confidential, and so on, with increasing levels of restrictions. I was amazed at how many documents had been classified as Confidential. The required document controls for this level of classification are pretty onerous. Information must be encrypted at all times when stored and must be encrypted with keylength of x or greated if emailed or faxed. Third parties must sign confidentiality agreements and get the permission of [...]

February 2, 2004|

How To Raise a Defect

Reading some of the problem descriptions posted by people who are supposed to be QA professionals in a technical specialist forum always gives me a chuckle. "Uggg. Test tool no work with Java applets". I love irony. A big part of their job is to find and raise defects. Sometimes it seems that their employers would get better value by sending them on a quest for fire. Just as everyone should read the guidelines before posting to a technical forum, Eric S Raymond's page on How To Ask Smart Questions and Simon Tatham's essay How to Report Bugs Effectively should [...]

February 2, 2004|

Extracting Errors from the LoadRunner Database

This entry appears in the Mercury Interactive Knowledge Base as entry 5012. It was posted by me on 2003-11-30. Description It is easy to view the number and type of errors as they occur during your test from LoadRunner Controller. It is difficult to determine error information with the same level of detail after the test unless the errors are exported to a file before the Controller is closed. Solution All errors are written to a Microsoft Access file in the results directory (output.mdb). Knowledge Base article #4164 provides an executable that extracts this information in the same format as [...]

January 29, 2004|

My Load Test – First Post.

A co-worker mentioned that he found it useful to keep a journal with technical problems he encountered in his job and the solutions he found to solve them. I thought it was a good idea and I have taken it one step further; putting it online in blog format (since that seems to be what all the cool kids are doing these days). Welcome to the world's only blog on Load Testing - let's hope that there isn't a good reason that is the case. I have always enjoyed inflicting my opinions on other people, but I hope that this [...]

January 21, 2004|
Go to Top