My Load Test

I haven’t written a huge number of Detailed Test Plans for performance testing, so I am still interested in reading other people’s (non-confidential) documents or templates. If anyone out there has a template, I would love to have a look.

I flew up to Sydney yesterday for the privilege of being (as far as I know) the first non-Mercury employee to sit for the LoadRunner CPC exam in Australia.

Feedback from people who had already taken the exam seemed to be almost universally along the lines of “there is a lot to get through. Know the subject thoroughly. You will not have time to look up documentation.” Mercury representatives said, “If you could do your LoadRunner CPS exam, then you should have no problems with the CPC.”

I had previously completed my LoadRunner CPS and had been using LoadRunner for a long time but, even so, I found the exam extremely challenging and used every minute of the allocated time.

The LoadRunner 7.6 CPC exam is to be completed in 1 day (8 hours). The pass mark is 70%, and a total of 134 possible points are allocated to the four exam sections as follows:

Section A – Scripting exercises.
60 points (45%)
3 hours.

Section B – LoadRunner Controller + Multiple choice questions.
28 points (21%)
2 hours 30 minutes.

Section C – Multiple choice questions.
36 points (27%)
15 minutes.

Section D – C programming.
10 points (7%)
30 minutes.

As can be seen from the recommended completion times for each section, Mercury’s estimates are definitely on the low side (adding up to only 6 hours, 15 minutes), and the possible scores for each section do not correlate with the suggested times.

I spent 4 hours and 30 minutes on Section A, and 45 minutes on Section D, with the rest of the time split between the two remaining sections.

The questions in Section A build on each other. If you cannot complete the first question, you will not be able to go on with subsequent questions. Having completed the exercises from the CPS comes in handy here, as some of the questions are similar. Ensure that you are familiar with parameterisation and correlation using both recording modes (as far as I can remember, the CPS only used URL mode). Try not to make assumptions about what is required. Follow the instructions to the letter. If you miss something on an earlier question, you will probably not have time to go back, make changes to all your previous scripts and re-run them. Remember that script execution for debugging purposes can be sped up by turning off “Animated Run” and closing the output window. Your script execution will be CPU-bound, and these items have a large overhead.

Section B focuses on creating and executing a scenario. There are no surprises here, but one of the key questions is based on how Mercury thinks a load test should be run, so your own experiences/prejudices may have little value. There is at least 30 minutes of execution time in this section that can be completed in parallel with Section C or D. Just remember that, as the Application Under Test will probably be running on the same machine as the controller, any compilations or searches through Acrobat files will affect the scenario results as they push CPU utilisation to 100%.

Section C is an opportunity to pick up some easy marks. There are 36 possible points for the 32 multiple-choice questions. There are some obscure questions here that you are guaranteed to have never encountered, but all answers can be found by searching through the documentation or experimenting with LoadRunner. As with the CPC, there are some very ambiguous questions, and even a question on the features of a type of graph that unintentionally leaves the name of the graph out of the question. When in doubt, deal with ambiguity by documenting your assumptions about what the question means.

Section D is the easiest section for guaranteed marks. You are given a script that contains compilation, logical and functional errors and must make it run against the Application Under Test. if you have a working knowledge of C, it shouldn’t be hard to track down the compilation errors. Start with the first error and work down the list. Remember that a “not writing pre_cci.ci” output message from the compiler can (non-obviously) push the real error off the top of the debug window. Once the compiler errors are taken care of, you must ensure that the script meets its stated aim. This will not be hard if you can successfully complete Section A.

Apparently the LoadRunner 7.6 CPC exam has a very high failure rate. I am not totally confident of receiving a passing grade myself. All questions were answered and all exercises were completed to the specifications, but my scripts were short on elegant solutions and clearly written code. My fingers are crossed for some generous marking.

Update: I finally received my results, and I passed this exam comfortably.

Update (December/2007) : This entry was written in mid-2004, and is no longer relevant to the new HP LoadRunner certifications. New LoadRunner certification exams are expected to be available in late Feb/March of 2008. I am involved in the review process for these exams and will post more information about where you can take the new certifcation exams soon.

I got some interesting feedback from someone in the Risk Management group on one of my Performance Test reports. Like any good Performance Test report, it compared the new version of the application with the previous version.

He said that he has seen a lot of cases where, over several versions of a piece of software, transaction times or resource utilisation has increased. The differences between versions are too small to worry about but, cumulatively, they may cause the performance of the software to be quite different to the original version.

As a consultant specialising in Load Testing, I generally don’t hang around for the full lifecycle of the software (ie the maintanence phase), so this is not something I had come across before. Perhaps it’s blindingly obvious to everyone else :)

This type of information will always be available in past reports, but whoever is signing off the test report is not likely to think to look outside of the document they are currently reviewing.

Unfortunately it is not always easy to provide a good comparison to an early version. Your mix of test cases may change, or transaction definitions may change with business requirements or as you gain better knowledge of the application. And there is limited usefulness in a comparison unless you are comparing like with like.

It’s always interesting to hear a different perspective…

I just received Solution of the Week on the Mercury Support Forum for a Knowledge Base article I posted in November. The article is in my archives for the curious.

I was recently asked whether I wanted to be part of the Quick Test Pro 8.0 Beta Program. Unfortunately I had to decline, but it got me thinking about beta testing in general.

As a Test Consultant, it is rare to be involved in testing of software that is going to be shrink-wrapped and sold. Most engagements seem to involve the testing of an application that is to be used “in-house”, whether that is a custom implementation of an existing application (as is seen with ERP/CRM apps), or an application that has been written from scratch. It’s probably possible to be involved in QA for an entire career and never be on the vendor side of a beta test.

In-house applications are not generally beta tested as they have clearly defined functionality (and therefore clearly testable functionality, not open-ended like a computer game) and they usually run on a standardised platform, so testing a million different permutations of end-user hardware and software is not an issue. Usability issues are covered by Usability Testing and User Acceptance Testing, and defects that are missed by the test team are hopefully picked up early by real users if the software can be rolled out in staged releases. Besides, managers seem to get a little agitated when their staff time is wasted, especially when there is a dedicated (and budgeted for) test team.

But, after saying all of that, if I was going to be managing a beta test, I would probably pay attention to Joel Spolsky‘s article, Top Twelve Tips for Running a Beta Test.

There is someone who is talented at marketing working for Gold Coast software testing firm K. J. Ross & Associates. They partnered with Software Engineering Australia to create the Certified Software Test Professional course, ensuring themselves some valuable mindshare and a guiding hand in the careers of future Australian software testers.

Another inspired, although not quite as influential, piece of work is Bug Fix Bingo. It’s one of those “I wish I’d though of that” ideas which strikes a chord with every software tester. Visit the site and I guarantee you will want to print it out and pin it to your cubicle.

Bug Fix Bingo is a fun new testing game created by K. J. Ross & Associates. The game is based on the traditional bingo game yet with a software testing twist. Instead of using numbers, participants in Bug Fix Bingo use statements from developers in defect review meetings to mark off squares. Statements such as “It works on my machine” and “Its never done that before”.

Incidentally, developers have a game like this too. They score points every time a QA person tries to raise a defect on functionality that is working as specified.

Update: Software Engineering Australia (SEA) folded in March 2005 after they found that they were not a viable organisation once their government grant ran out. Staff were absorbed by Object Consulting, who now offer CSTP training.

I was browsing the project share drive at work today looking for a design document that I needed. Every document that is created should, in theory, be given a classification; Comercial in Confidence, Confidential, and so on, with increasing levels of restrictions.

I was amazed at how many documents had been classified as Confidential. The required document controls for this level of classification are pretty onerous. Information must be encrypted at all times when stored and must be encrypted with keylength of x or greated if emailed or faxed. Third parties must sign confidentiality agreements and get the permission of the document creator before viewing. The document creator must approve disclosure to all other staff. The document cannot be discussed in public. And so on…

This creates an interesting problem for document management. In a case such as this, where it seems that documents have been classified by the document owner in a fairly arbitary way (many people seem to be applying the CYA rule), the classification system breaks down. All documents get stored in a way that everyone can access so that people can get their jobs done, with the attendant risk that something that actually is important slips through the system.

Some questions to ask:
* What is the average cost per document to properly manage information that has been classified at a certain level?
* Is it worthwhile having a documented process for document classification and managment if it is not audited or enforced?
* What is the cost to the business if a Confidential (or higher level classified) document is leaked? (How long is a piece of string?)
* How does the DoD do it? (They seem to classify everything. If you’ve ever received email from anyone at the Department of Defence, you will see a classification level in the subject header – usually UNRESTRICTED).
* Is document classification, access control and management already a feature of the Enterprise’s CMS (Content Management System)? What is the additional overhead experienced by each user to search for and retrieve useful information with this system?

Addendum: It is a particularly bad sign when supposedly confidential documents are posted on a public website or “hidden” in a place that will still get crawled by web spiders. Google sees all.

Reading some of the problem descriptions posted by people who are supposed to be QA professionals in a technical specialist forum always gives me a chuckle. “Uggg. Test tool no work with Java applets”. I love irony. A big part of their job is to find and raise defects. Sometimes it seems that their employers would get better value by sending them on a quest for fire.

Just as everyone should read the guidelines before posting to a technical forum, Eric S Raymond’s page on How To Ask Smart Questions and
Simon Tatham’s essay How to Report Bugs Effectively should be required reading for any software professional.

When it comes to raising and tracking defects, one of my pet annoyances are people who attach a screenshot of their defect, but do not include a text version of the error message. This means that you cannot do keyword searches effectively. It’s probably worth your time to spend five minutes learning the features of your defect tracking tool before you start using it. My favourite new feature in TestDirector is the System Information tool, which can extract a detailed configuration report for your computer to attach to the defect you are raising (Add Defect > Attach SysInfo).

A co-worker pointed out to me that I tended to raise defects that were too technical, or that provided enough technical information, but not enough “big picture detail”. The developers could understand them, but the defect manager and project managers were having trouble determining if my priority and severity assessments were correct in their daily defect meetings. I changed the way I raised defects after that.

Just like any other business communication…
* Write to your audience
* Make it as complete as possible
* But still try to be concise and on-topic
* Try to anticipate (and answer) obvious follow-up questions.

Addendum: I recently had a string of problems with my virtual hosting company. I raised several problem tickets providing details of exactly what my problems were. From the names on the replies, it looks like the hosting company has outsourced support to India even though the servers are still in the US (the traceroute ends in the United States). Even though the replies were in perfect english, I regret using so many large words in the problem descriptions. No one should have to try to interpret that in their second language. Newspapers write to a 13 year-old reading level, perhaps bug reports should too. It all comes back to knowing your audience.

A co-worker mentioned that he found it useful to keep a journal with technical problems he encountered in his job and the solutions he found to solve them. I thought it was a good idea and I have taken it one step further; putting it online in blog format (since that seems to be what all the cool kids are doing these days). Welcome to the world’s only blog on Load Testing – let’s hope that there isn’t a good reason that is the case.

I have always enjoyed inflicting my opinions on other people, but I hope that this also proves to be a useful resource. Let’s see what transpires over the coming year.