Last week I discussed the fallacy of the minefield analogy misrepresented by some people to suggest regression testing as uninteresting or unlikely to reveal new or important information.  Their premise is that executing the same test is similar to walking in someone’s footsteps through a minefield. While this argument seems logical on the surface, this interpretation of the minefield analogy trivializes the importance of regression testing. However, there may be specific instances where regression testing (especially the execution of redundant, poorly designed, basic scripted test cases that simply reuse hard-coded data or mindlessly follow a prescriptive steps to retest unchanged or unaffected areas of code) may not provide great value to the organization. Some of those situations include:

• retesting scripted, or simple procedural programs
• retesting programs that are relatively static (unchanging),
• retesting programs with no internal or external dependencies
• rapid testing approaches where the test strategy is to simply sample the program behavior in order to provide a quick assessment and find some bugs

So, everything being equal, I would tend to agree that walking through someone else’s footsteps in a minefield may be somewhat redundant. But, there is a huge difference between redundant testing and effective regression testing. Redundant testing has zero probability of revealing new or valuable information. But, well-designed tests are not all equal, and in iterative and agile software development lifecycles some portion of effective regression test suites have some reasonable probability in exposing new information. For example, regression testing is typically useful where:

• the underlying code base is changing (new features, refactoring, or bug fixes)
• poor or incorrectly used revision control processes
• a change in one module affects other modules in software developed using object oriented and procedural programming paradigms
• the complex system has other internal and external dependencies
• the software is regulated by some governing authority or law (Sarbanes-Oxley, FDA, FAA, etc)
• the system is highly critical
• an established baseline is required or important
• legacy code bases where new functionality can unintentionally destabilize older code
• code bases contain lots of ‘spaghetti’ code that is simply patched together
• the results of a well-designed regression test suite helps instill a sense of confidence in the decision makers

In these contexts, an effective regression testing strategy can not only help identify important and potentially destabilizing changes in expected functionality quicker, but can also increase overall confidence in the product; especially in key areas.

Overcoming some common misconceptions of regression testing

One common misconception of regression testing strategies is that all tests are equal and all documented (manual or automated) tests go into the regression test suite. But, the simple fact is that not all tests are equal, and not all tests need to be re-executed repeatedly (every build) throughout the development lifecycle. For example, a test to check for proper tab order on a dialog or property sheet doesn’t need to be re-executed on each new build unless the UI elements have changed on that dialog or property sheet. So, in most situations this type of test is probably a poor candidate for inclusion in a regression test suite. (As a side note, if the UI elements are changing every build, I wouldn’t even test tab order until there was at least some semblance of UI stabilization.) When redundant test cases or tests that don’t provide great value to the overall testing effort are slovenly added to the regression test suite to simply increase code coverage or to artificially inflate the number of tests for ‘feel-good’ generally only results in an overloaded test suite that rapidly grows out of control and becomes a management nightmare.

Another common misconception of regression testing strategies is the suggestion that unit tests are sufficient for retesting code churn. Of course, this is simply a fool-hearty assumption and idealistic. Unit tests are a type of smoke tests performed by developers. The primary purpose of unit testing is to verify a method or function does what it is supposed to do, and may include ancillary negative tests as well. However, unit tests generally do not include comprehensive data coverage, data permutations, combinatorial analysis of parameters, etc. (which is why API testing is important in a well-rounded test strategy). Also, unit tests are usually executed in a “clean-room” environment using stubs or mock-objects, although in some cases unit tests are reran on private builds before the code is checked into the main build tree as a form of low level regression tests. While there have been significant advances in both static and dynamic analysis tools to increase the robustness of unit tests, if unit testing were the answer for effectively evaluating code churn I suspect a lot of testers would be out of a job and testing would simply be a rudimentary process of behavioral-type acceptance testing performed by non-technical ‘end-user-like’ individuals poking about the UI looking for errant anomalies and subjectively evaluating their perception of ‘usefulness’ of a product.

Another common misconception of regression testing strategies is that regression tests are less valuable because of their ineffectiveness in identifying or exposing ‘new’ bugs. This is a myopic view of testing that focuses on the current version of the product, and on ‘bug-finding’ as the main purpose of testing. Of course, value is very subjective, and industry reports suggest that less than 15% of the bugs reported during a product development lifecycle are detected via regression testing. But, if even half that number are critical issues that would cause the company to pay upwards of $100K to release a patch (or even worse, the issue leads to a major calamity) then regression testing (especially highly automated regression testing) is most probably worth the time and effort. Of course, the cost of designing effective regression test cases is quite high and executing those tests requires valuable resources. So, we should not just consider the legal or financial liability issues to the company. We should also consider the product shelf-life as a factor. An effective suite of well-designed regression test cases will not only help provide a baseline assessment of the current version, but a significant portion of the test suite will/should be reused during maintenance or sustained engineering of that product for upwards of 10 years, and some subset of those tests in that suite will also be reused on the next release or version of the product. There are additional benefits to well-designed regression test cases (or any test case for that matter) in that it preserves knowledge and helps eliminate or at least reduce tribal knowledge and hero worship mentalities.

Some effective regression testing strategies

The regression test suite is essentially a set of test cases that provide a baseline measure of expected, or important functionality, and to verify previously fixed bugs do not reoccur. Based on those assertions, test cases included in the regression test suite must be carefully selected to prevent overloading or rerunning unimportant tests during a regression test pass. I generally recommend 4 types of test cases for inclusion in a regression test suite:

1. Test cases designed to verify critical functional attributes and capabilities of the software program
2. Test cases designed to validate baseline functionality/behavior specified in project requirements or user acceptance criteria
3. Test cases designed to verify functional anomalies that are found and fixed during the software development lifecycle
4. Test cases designed to evaluate collateral or dependent areas associated with a code fix

Even with these targeted tests the regression test suite can grow quite large. Within the regression test suite categorize the test cases by the feature or component that test directly targets, and sub-categorized to identify dependent modules or features. Associating test cases in the regression test suite with the primary features or components they are designed to evaluate allow the tester to prioritize the regression test pass to focus initially on the modules in which code churn occurred,  followed by dependent or interrelated modules, and finally the other remaining tests in the suite. Also, prioritize test cases within each category based on criticality. Prioritization of tests is another mechanism we can use to help us better identify an appropriate set of tests based on the constraints of available time and resources.

Finally, automate, automate, automate! In order to gain the full benefit of an effective regression test suite (similar to the build verification/acceptance (BVT/BVA) test suite ) automate as many regression test cases as is reasonable possible. But, I am referring to well-designed automated test cases; not just a bunch of simple, elementary UI script monkeys. While some regression test cases will be very targeted prescriptive tests designed to verify specific functionality, most well-designed test cases include variability in the execution while still evaluating the hypothesis, or purpose of the test case. Also, some test cases in the regression test suite will be executed through the UI layer, but in applications where the functional code is separated from the UI layer some regression test cases are more effectively executed below the UI or through an abstraction layer.

Similar to other testing approaches, regression testing is an organizational investment, and its value to the team must be considered as an investment and not done simply because it seems like the right thing to do. In some cases it may not make much sense to invest heavily in regression testing, but in cases where a missed regression in functionality results in fines, legal costs, or potential loss of hundreds of thousands of dollars in post-production costs I suspect regression testing is a critical component of the overall testing strategy to validate a baseline and instill confidence.

In my studies at university I studied anthropology. Several courses I took surveyed folklore and its relevance in modern society. Many people mistakenly believe that most folklore (folktales, legends, myths, ballads, etc.) are purely fictional and simply fanciful tales. However, folklore is usually based on some grain of truth, or is used to instill societal or religious mores and values. For example, social scientists have found that many ancient civilizations have folklore regarding a massive “flood” in the distant past which wiped out huge populations of people. Did this actually occur? Well, we don’t know for certain, but geological evidence does suggests is that at one time coastal waters did rise significantly. Was this caused by cyclical change in the earth’s temperatures or by a series of earthquakes causing tsunami’s to ravage coastal villages? We don’t know; but the folklore may indicate that at some point many societies suffered a devastating travesty caused by rising waters. Was the story embellished over time…certainly. Another example is the “Cinderella” story. There are over 450 versions of the “Cinderella” story around the world.  The story is about over-coming adversity and oppression, and avoiding self-pity and selfishness. Basically, it is much more than a Disney animation, in traditional folklore it has been passed down through the generations to reinforce societal values.

The first time I read about a minefield analogy was in the context of sampling. Later, Brian Marick used a similar analogy to suggest repeating tests (regression testing) is not likely to reveal new bugs. Marick’s analogy is perpetuated by Bach, Kaner, and others who tend to diminish the value of regression testing (especially automated testing) because we are simply traversing a minefield by following a previously cleared path.

The Marick minefield analogy is simply an alternate perspective of Beizer’s pesticide paradox which states “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. Basically, no single approach to software testing is effective in identifying all categories of defects, and we must use many approaches in software testing and vary our tests. In that context I absolutely agree with the analogy.

However, a basic problem of Marick’s minefield analogy as it is often misrepresented is that it seems to treat the software under test as a static, unchanging field of easily exposed mines.

If you were hired as a consultant to come in a perform a rapid evaluation of a software product using a sampling approach such as exploratory testing, then Marick’s minefield analogy is a wonderful strategy. In that context re-running a test provides no new value and has little probability of exposing new information.  However, for the rest of us who work in iterative software development lifecycles (including agile lifecycles) building complex systems with interdependent components the minefield analogy may not be as useful.

For example, in complex systems with interdependent modules we know that a change in one module can adversely affect other modules that have some dependence on that module. So, a change in one module can impact the functional behavior of other modules. In layman’s terms, activating a mine while traversing one path through the minefield may reactivate an already cleared mine in another part of the minefield, or even plant a new mine in a previously traversed path.

In iterative development lifecycles, the minefield is in constant flux (at least until the code complete stage, but even then the code is changing as issues are being addressed.) In iterative lifecycles features are being added, changed, and possibly removed during the process. Depending on the length of your product lifecycle the changes can be massive. The PDC release of Windows 95 ‘looked’ very different as compared to the final release.  The build verification/acceptance test suite for Windows 95 was a relatively static baseline regression test suite that continued to find ‘regression’ problems up to the final weeks of the project due to code churn.

Also, not all mines are equal! Some mines are quite easy to detect while others are very hard to find which is why systematic probing is still used by professional’s to clear latent minefields. Similarly, an exploratory approach to testing software will easily reveal some bugs very quickly, but without ‘systematic probing’ we could just as easily overlook other types of issues.

There are also different types of mines which may be activated differently, so traversing a minefield with a size 10 boot may not activate the mine, but someone with a size 12 boot, or who weighs more than the previous person may in fact activate the mine. Likewise, traversing the same path through software using different data or applying a more systematic analysis of a path may reveal interesting information or expose anomalies that were not previously discovered. For example, throwing simple ASCII characters at a text input control is not likely to expose any bugs (or restated it is likely to show us a clear path through the minefield). However, when we take that same exact path using Unicode characters, or Unicode surrogate pair characters we are very likely to expose problems not revealed previously.

In part 2 I will discuss regression testing and specific situations where regression testing is very valuable.

There is a lot written about regression testing, and yet there seems to be a lot of confusion about regression testing as well. Just to make sure we are all on the same page, by regression I am referring to the denotation of the word to indicate a relapse to a less perfect or developed state (American Heritage Dictionary). So, the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition. It is important to note the purpose of a regression test suite is not necessarily to expose new defects. The primary purpose of a regression test is to identify changes in behavior from a previously established baseline, which is supported by Beizer’s and Myers’ definitions of regression testing.

However, even on small projects the number of tests required to ensure new builds do not regress or change previous functionality can be quite numerous. So, regression testing demands a strategy in which we limit the number of tests to establish an effective baseline measurement. In IEEE 610 documentation it states regression testing is selective retesting. Thus, the key to an effective regression testing strategy is to design a test suite that provides a high degree of confidence without retesting everything. To limit the number of tests in the regression test suite, we must systematically reduce the number of possible tests. So, we must decide what tests are included in a regression test suite?

Deciding what tests to include in the regression test suite

The most effective regression test suites I have seen include two categories of tests. The first category of tests includes high priority tests for commonly expected functionality (e.g. the 20% of the product that 80% of the customers demand or rely on). The second category of tests includes any functional defects that are found and fixed. Found and fixed functional defects are included because fixed defects do occasionally regress, and if a business decision was made previously to fix a defect then we probably want it fixed before we release the product.

Prioritized feature area/functionality buckets

The tests in the regression test suite should also be partitioned into functional areas and each test in each functional partition or bucket should also be prioritized based on risk assessment criteria.  If the regression test suite is especially large or time is limited, and the regression suite is portioned into functional areas (and those areas are mapped to the project files or modules contain that specific functionality and any dependencies) the regression test pass can execute a limited subset of tests from the regression test suite that strategically target the modules that have changed (and tests for dependent modules as well). Simple directory comparison tools (such as Diff2Dirs), and tools to identify dependencies between modules (such as Depends) are useful in identifying which modules change between builds and to map out dependencies between the modules in each build.

Automate, Automate, Automate

Also, since the regression test suite will ideally be ran on each new build, this is one suite of tests that should be 99.999% automated. Similar to the BVT/BAT test suite the purpose of the regression test suite is not necessarily to expose defects; a regression test suite provides baseline measurement of functionality. Therefore, since these are tests that will be ran several times during the software development lifecycle and are not necessarily designed to expose new defects the ROI for automation is very high. In fact, any test that cannot be automated is suspect for inclusion in the regression test suite.

These are a few ideas to develop a highly successful automation strategy. What other tactics have you found to be successful?