A cargo cult reproduces a landing strip complete with wooden plane.Richard Feynman popularised the idea of the cargo cult in his essay on cargo cult science, which was “science” that followed all the forms of scientific investigation, but lacked real critical scientific thought. The idea was transferred to the world of software courtesy of the Jargon File with its entry on cargo cult programming.

The Jargon File defines cargo cult programming as:

A style of (incompetent) programming dominated by ritual inclusion of code or program structures that serve no real purpose. A cargo cult programmer will usually explain the extra code as a way of working around some bug encountered in the past, but usually neither the bug nor the reason the code apparently avoided the bug was ever fully understood.

As someone who works in one of the more technical areas of testing, I see the same thing in the testing world too.

Functional testing is pretty straight-forward; ask a test manager why they are testing, they will probably say something like “we are seeing if the business functions work” (really smart test managers will add that they are seeing that the actions that are meant to fail do actually fail too).

But once you add a layer of technology, it becomes a bit harder to grasp. If your functional test regression suite is automated, then the quality of the testing is less easy for most people to see. Were the test cases that were automated good test cases to begin with? After a few months or a year, regression testing becomes a little bit like a ritual; you wave the magical testing tool over a new version of the application and declare it “tested”. Whether it is tested well is an entirely different question…and one which is unlikely to occur to those deeply involved in the ritualistic behaviour.

In the stress/volume/performance/load/reliability testing world things are even worse. Occasionally I visit companies that I have consulted at in the past to see how they are going with their testing. They generally know to run a peak load test for new builds (and platform changes) before applying the change to Prod, but if you ask a question like “have the transaction volumes changed in Production since I wrote the original Detailed Test Plan, and have the LoadRunner scenarios been updated to reflect this?”, you will draw a blank.

Performance testing during the maintenance phase of software frequently seems to degrade to a situation where the testers are more interested in whether the test cases and LoadRunner scripts still run successfully, rather than whether the test cases reflect reality or even cover the point of change that is driving the current round of testing. And a lack of understanding of what each test case is designed to test leads to wasteful re-running of test cases that are not impacted by a change.

Unfortunately I don’t really have any solutions for this besides companies paying me to occasionally come in and review their performance testing activities (which kind of smacks of self-interest).

 

Published On: May 15, 2007Tags:

3 Comments

  1. Stuart Moncrieff May 15, 2007 at 11:26 am - Reply

    A couple of points
    * you see “cargo-cult testing” in the early stages too, not just the maintenance phase. Ever seen a performance tester plan to run a series of test cases without asking “are these relevant to what I am testing?”
    * also, the main constraint to testing during the maintenance phase of an application is that there is not much budget. It is easy to ask for money when you are initially developing software, but much harder to ask for money for testing when the application has been in Prod for a while.

  2. Ben Simo June 9, 2007 at 6:34 am - Reply

    testers are more interested in whether the test cases and LoadRunner scripts still run successfully, rather than whether the test cases reflect reality

    Bingo! I keep hearing that pre-documented repeatable process is more important than adapting to change. 🙁

  3. gerry newby January 25, 2011 at 3:35 pm - Reply

    ok so this is an old and maybe defunct thread but
    what you are observing is true in almost all human experience. that is operating a repeatable process, effective or not, is preferable to actually implementing and effective, efficient, economic (the three “e’s” of management) solution to any current situation. but don’t write the ‘solution’ in tablets of stone, they’ll need amendments too, eventually!@

Leave A Comment