After a 1 year sabbatical from writing the time has come for me to get back to sharing some ideas, and hopefully provide some thought provoking discourses. Both personally and professionally it has been an exciting year with many challenges and new opportunities. Professionally I am now working on solutions to improve the effectiveness and efficiency of our automated tests (more on that later). Another new opportunity for me has been teaching courses on software testing fundamentals at Duy Tan University in Vietnam. I must admit that sitting on the beach in Hoi An and writing this post suites me very well.
Anyway…back to the testing stuff. Earlier this year I started working on a new project at work that I have dubbed “Automation Anarchy.” As a product matures from one version to the next the pool of automated tests also continues to expand as legacy tests are ported forward and additional tests are added for new features. But as the number of automated tests increase the number of helper methods in the automation stack are also increasing.
One problem with this seemingly unending piling on of test code is that the build times and run times of the automation runs continues to creep up. In a world of high volume automation and quick build cycles (2 or 3 a day on branches) we don’t have the luxury of letting our automation churn for hours. The results needed to decide whether we integrate lower branches up into the main branch must percolate to management within 1 to 2 hours after the build.
Another problem with the non-stop growth of test code includes the additional costs to maintain the code base. Changes in product behavior may require changes in the test code. Some of the automated tests will become obsolete. Poorly designed tests, or tests without a clear purpose generally require more maintenance costs. The most costly tests are those that are unreliable. Sometimes they pass, sometimes they don’t complete and throw a test exception during execution. Every time an automated test fails to run or throws a test exception while executing that test case must be investigated. This takes time and valuable resources away from other tasks on the tester’s plate.
Duplicate and/or redundant automated tests are sometimes inadvertently added to a test suite when the tester changes and the intent or purpose of an automated test is unclear or not well documented. Duplicate and redundant tests do not provide any additional or useful information. The primary purpose of testing is to provide the necessary information to enable decision makers make business decisions. Automation can help provide that important information more timely compared to manual testing. But, if redundant or duplicate tests#160; are not providing valuable information then they are wasting time, or even worse these tests can ambiguate test results.
Finally, the proliferation of test code also causes our wonderfully intended automation stack to start to decompose. Parts of the automation stack decay and results in pockets of dead code. Obsolete tests and helper methods linger in the code base. And sometimes as we start to dig deeper we find code clones in the automation stack. Sometimes these clones are benign and the result of a tester copying and pasting a chunk of test code, a helper method, or maybe even an entire class. But, other times some of the code clones is not only copied but then mutated. Code clones and mutations of code clones are like a cancer within the automation stack. Cloned code obfuscates the code base, and also increases maintenance costs.
In the future I will write more about this project and hopefully offer some suggestions to help others prevent automation anarchy from taking over their testing projects.
Automated tests are like dandelions. Carefully cultivated dandelions are used for foods and medicines around the world, and the plant can also be used to help bring nutrients to other plants. But, left unchecked and untended the dandelion is a prolific weed that will take over and ruin an otherwise beautiful garden./p