load testing

Your Workload Model is probably wrong

First off, the good news: if you are load testing based on a defined Workload Model, you are already waaaay ahead of a lot of performance testers. I am frequently horrified by testers whose approach is…well, just not very scientific. When their test case is “we’ll just throw some load at it and see what happens” or their load test has been defined in a way that is vague and imprecise, like “our test is to run load with x concurrent users”, then the probability of them not knowing what they are doing approaches 100%, and it is clear that [...]

February 16, 2018|

The Top 7 Mistakes in Performance Testing

This is my talk on the Top 7 Mistakes in Performance Testing that I delivered at the first Neotys Performance Advisory Council in November 2017. My presentation can be downloaded from SlideShare, and Neotys will be making all 15 speakers’ slides available soon. [slideshare id=83352854&doc=neotyspac2017-top7mistakesinperformancetestingv1-171204222626] […]

November 17, 2017|

Neotys Performance Advisory Council 2017

Recently, Neotys (the company responsible for NeoLoad) held their first Performance Advisory Council in Scotland. Their idea was to get 15 of the top performance testers together to talk about some of the trends that are influencing the future of the industry, such as IoT, AI, shift-right, and DevOps. Performance testers traveled from the U.K., France, New Zealand, Germany, USA, Australia and India to share their ideas on the the changes they are seeing to load and performance testing. Holding the event at Borthwick Castle (built in 1430) gave us a picturesque (if slightly incongruous) backdrop to our discussions about [...]

November 16, 2017|

Performance Testing memes

Image macros – those pictures superimposed with vaguely humorus text – are everywhere on the Internet. I decided that it was time for someone to create some that performance testers could relate to; something we could all print out and pin to our cubicle walls. To get everything started, I have created 8 images. Maybe they will become successful memes and help spread performance testing culture across the world…or maybe they will just generate a chuckle or two. If you create your own performance testing-related image macro, please leave a comment (with a link), or send me an email. I [...]

October 15, 2014|

Useful Tools for Performance Testers

On August 23rd, 21 JDS performance testers got together for their quarterly breakout session to share tips and tricks with each other. One of the breakout topics was Useful Tools for Performance Testers. This was about useful tools that were not load generation tools (or everyone would have just said “LoadRunner” and it would have been a very short session). I have included everyone’s contributions in this post… If you are a performance tester and have a different tool that you find useful in your job, please add it in the comments below. […]

August 24, 2013|

Bad Performance Testing

As a performance testing consultant, I get to see a lot of the work that other performance testers do…and a lot of the time it horrifies me. If performance testing was a licensed profession, like law or medicine, then it would be necessary to revoke the licenses of 90% of testers. Their work is not just “low quality”, it is actually wrong or misleading, or based on such a shaky foundation that any predictions made from the test results are tenuous at best. Bad performance testers frequently treat their testing as a ritual, rather than as a science and they [...]

December 22, 2012|

Performance Testing in a Mobile World

This is my presentation on Mobile Performance Testing that I delivered at SoftEd Fusion 2012 (Sydney) on September 13th. The original PowerPoint file can be downloaded from SlideShare. [slideshare id=14380290&doc=fusion-2012-performance-testing-in-a-mobile-world-by-stuart-moncrieff-120921200026-phpapp02] […]

September 15, 2012|

Errors are bad

It seems unnecessarily obvious to even bother making the statement that “errors are bad”; surely this is an idea that everyone agrees with, like “crime is bad” or “you shouldn’t put your underwear on backwards”. But a lot of performance testers that I have worked with don’t seem to care too much about windows errors. A senior performance tester with more than 15 years experience at some of the largest companies in Australia recently said to me:   “Yeah, as long as the error rate is below 5%, I don’t bother putting it in my report.” Are. You. Kidding? How [...]

July 14, 2012|

New book on Performance Testing

I've spent the last year or so writing notes and fleshing out chapters for a book called Performance Testing Web Applications, so imagine my very slight feeling of annoyance when I did a google search for my book title and found that someone had released a very similar book 3 months earlier... J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea have collaborated on a book called Performance Testing Guidance for Web Applications, which is available either as a free download, or in dead-tree format through Amazon. The book is really good, so I highly recommend that you [...]

November 28, 2007|

The 10 Commandments of Load Testing

I have made a list of the top ten things load testers frequently fail to do that make me feel like smiting them. Thou shalt know how thy test tool works. The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic. Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application. Thou shalt gather realistic usage data. Garbage in, garbage out. [...]

May 9, 2007|

Using open source tools for performance testing (Google video)

I just found something interesting on Google Video; it is a 1 hour Google TechTalk presentation by Goranka Bjedov about Using Open Source Tools for Performance Testing. Goranka has some interesting things to say. She makes the point that there is really no standard terminology in performance testing circles, and goes on to prove this by giving her own definitions of performance, stress, load, scalability, and reliability testing. As an example of reliability testing she notes that "typically, when I was at AT&T, we would run for about a month at a time after everything was done just to find [...]

November 2, 2006|

Network Sociability Testing

If the application that you are performance testing will be operating over a WAN, it is really important to test whether the reduced bandwidth and higher latency of the network link will increase transaction response times to an unacceptable level. If you are being thorough, you should also run some tests to determine whether your response times will be impacted by other network traffic on the WAN (and vice versa). Let the following anecdotes about testing two different applications serve as examples: The first application was to be used by staff who would be interacting directly with customers. There would [...]

September 24, 2006|

Discovering your website usage profile with LogParser

It's always nice to base your load test on real-world usage patterns rather than someone's "best guess". If the system is used internally, Business Analysts are usually quite good at providing this information, but if the system is used by external users (like most high-traffic web applications), then they can typically only tell you "big picture" information; e.g. they know how many orders are created, but not how many searches are made per hour. Basic web log analysis tools provide low-level technical detail that isn't very useful - hits per hour. To be useful for load testing, we need to [...]

September 10, 2006|

Rational Performance Tester

I just received an evaluation copy of the new version of Rational Performance Tester. IBM Rational have re-written the tool from scratch and based it on the open-source Eclipse framework. The tool has only just come out of beta testing and currently supports web protocols only. Everyone in the industry seems to be keen to know how it stacks up against the competition. First off, my impressions of the old version... Coming from a LoadRunner background, my opinions on the tool were a little mixed. Performance Tester was easy to learn and easy to use but it is definitely missing [...]

May 3, 2005|

Environment Availability

I am yet to work on a project where environmental issues haven't impacted my schedule to some degree. But it seems that I only start to document the outages once they start to seriously impact my schedule. This stops now. From now on I will document every outage (no matter how trivial) before they start to have any impact. Once the cumulative downtime reaches more than a few hours, I will include a link to the outage document as part of my weekly (or daily) status report. Here is a template for my Environment Availability Spreadsheet.

March 14, 2005|

How to write a performance test case

I have decided to release an early draft of this document so that others may provide feedback. Please let me know what you think. Writing test cases for performance testing requires a different mindset to writing functional test cases. Fortunately it is not a difficult mental leap. This article should give you enough information to get you up and running. First, lets set out some background and define some terms that are used in performance testing. Test case - a test case is the same as a use case or business process. Just as with a functional test case, it [...]

February 14, 2005|

How to write a Performance Test Plan

I haven't written a huge number of Detailed Test Plans for performance testing, so I am still interested in reading other people's (non-confidential) documents or templates. If anyone out there has a template, I would love to have a look.

January 18, 2005|
Go to Top