The Case for Regression

How to Prove Software is Ready?

Shipping software is hard, making sure that what is being sent to the end-user works as intended is extremely difficult. Do they use your product exactly as it was tested? Were the needs captured during the initial planning stages correctly? Are there use cases that were not originally obvious? Does the software behave the same under load? Does it work the same the 35th time as the 1st? 

One of the ways as a QA (or tester) group can do this is by running exhaustive regression testing on it. It is possible to moved through every state of some small projects but becomes impossible with anything that moves beyond the trivial size. Generally we are constrained by time and number of personnel available. We also cannot accept quality dropping. So where does that leave us?

We need to select a sub-set of all possible states and run through them in a coherent and communicable way. This is where test cases can come in.

Regression vs. Feature Work

In a regression mode of testing the only thing we are looking for is, did it change? We literally do not care if there are other bugs in the software. Is there a UI element that looks wonky? File it if you have time, but we don't care. The workflow doesn't make sense? Do not care right now, if it is the same as last time, not a regression bug. All of the feedback or subjective opinions should have been expressed when the feature was being worked on. The only question right now is does it work in the way that test case says it should? Ok, then it passes. Move on.

Outcome

So what does this result in? After having written and executed all of these tests what is the hoped for outcome? The goal of writing and running Regression test cases is that the product is in working order and the end-users will not spot issues that block them from using the product. That is the whole goal, it should not take more than a few minutes to determine whether the test passed or failed. There should be no double or triple checking for minor glitches. It either did it or it did not. The only real question here is: can we ship?

Consistency is the Point!

Identifying and checking for a specific set of parameters means that you are narrowing the mental bandwidth of the testers to only need to pick-up very few items. This comes with risks, if you pick the wrong set of parameters it can lead to end-users being unhappy with the product (and maybe rejecting it altogether). So the goal should be that you re-use these parameters from release to release. Especially if the previous releases were successful. The template that we should be using is whatever is currently in Production environment and making the company money. 

Ready to Ship

So now all the Regression test cases have been run, everything is green (or waived). The release now gets a stamp of approval and we can ship. All of the tests should be documented and updated to reflect the current state of the software at the time of shipping. That way they are all ready for next release.