I.M. Testy

Last month I was travelling throughout Europe. A few days in Switzerland to speak at Swiss Testing Days and visit our communication server team in Zurich. About a week and a half in Denmark teaching at our offices there, a few days in Dublin teaching, and finally a full day meeting with one of our teams in Reading, UK. The only good thing about being on the road that much is getting in more reading time. Two of the books I read while travelling were Agile Testing by Lisa Crispin and Janet Gregory. The other was Clean Code by Robert Martin. I found both books thought provoking and informative. Of course I didn’t agree with everything, but even the statements I didn’t agree with made me think about that topic a bit more. Martin’s book inspired me to improve my testing and coding craft. If nothing else, testers should read the first chapter of Clean Code, those who design and develop automated tests should read the whole book.

But, one statement that has always troubled me that I saw repeated in the Agile Testing book is in regards to exploratory testing as the most effective way to expose important bugs. Every time I hear/read this or one of its many derivatives it makes me ponder. If this is really true then why do we invest so much time and resources in other approaches to software testing. Why do we spend so much time in static and dynamic analysis, reviews, and coverage analysis? Why do we try to prevent bugs when they can (and presumably are) found easier and ‘quicker’ downstream via exploratory testing? Why don’t we simply continue to hire hoards of diverse people to rapidly explore the product just before release and beat out all those important/critical bugs?

I wonder why it seems that we can ‘think critically’ when we have the product to play with, but we can’t seem to apply critical thinking while reviewing documented requirements, or a model? Personally, I am not sure that ‘trying’ something and then deciding what to do based on the outcome or state of the computer, or deciding whether something is a bug after the fact constitutes critical thinking. Perhaps sometimes it is, but I suspect that sometimes it is just guessing. To me, critical thinking involves the ability to foresee potential issues; not simply stumble upon them, or apply an attack from my bag of exploratory tricks to reveal them downstream in the development lifecycle when I have the product to play with. 

I sometimes wonder if exploratory testing really is the holy grail of software testing, or if we have just given up on other approaches because they are too hard. Could it be that our testers are not capable of ‘testing’ without also engaging their fingers? Are our testers untrained or under-trained in our profession? Is the reason why our pre-defined test cases are seemingly ineffective due to the realization that we suck at test design, and learning how to design more robust and ineffective tests is just too hard? Is the reason we can’t think of tests until we get the product in hand perhaps due to our inability to model or is it because we really don’t know that much about the systems we are testing? (You can’t model what you don’t understand and you don’t understand what you can’t model.) Besides, why should we burden testers with the overhead of learning about programming concepts, computer science, and math. You don’t need all that stuff to find bugs.

Now many of you might think that I despise exploratory testing. I am not anti-exploratory testing (whichever form that takes); because we all do it! Show me a tester who doesn’t include some form of exploratory testing in his or her repertoire and I’ll show you a thoughtless simpleton who needs explicit instructions for determining which side of the heal of a loaf of bread to butter. But, I have always thought that we could actually improve the craft of testing. I thought we might play a much bigger role in an organization. I thought we could help reduce overall production costs by being involved earlier and throughout the lifecycle. I thought we could help reduce risk through defect prevention and in-depth analysis. And, I thought we could learn to use tools and techniques to help us design better tests up front, and rely on a team of testers who are capable of thinking while executing a test case.

Yes, it is true that exploratory testing is fun, and we don’t really need to understand all that computer science stuff to find bugs using an exploratory approach. Regardless of what empirical studies on the topic reveal, exploratory testing ‘feels’ so much more productive. Why should we base our profession on evidence when we can get by so easily with emotion? Why should we have to think critically during the design or implementation of software when we can simply muddle our way through some graphical user interface to find bugs? If our approaches to unit testing and component level testing are so shoddy as to simply invoke happy path” tests then of course finding bugs by exploration seems rather logical. But, as developers become more adept at unit can component level testing in flushing out functional bugs earlier we may only need ‘testers’ to explore the product before release for behavioral issues, compatibility problems, and the few remaining obscure functional bugs. Or possibly exploratory testing providing a ‘good enough’ service to our team by finding the bugs that matter to the majority of customers, and perhaps that is ‘good enough.’

Personally, I think we can improve our testing processes. I think we will need to improve the practice of software testing. I think that our job is not simply to find bugs, but a more important objective is to prevent defects (which can help reduce overall product costs). I also don’t view testing as destructive, and I think those who do are doomed to become obsolete. Successful businesses don’t hire folks who just want to ‘destroy’ the products they build, I suspect they want to hire people who can help the company grow and improve. Finding bugs helps me ship a product, but it doesn’t necessarily help me grow the business, reduce production costs, or improve quality long-term.

But maybe we don’t need to change. Maybe our businesses are satisfied with the status quo. Maybe we don’t need to think about cost savings or helping our teams mature their processes, reduce overall risk, and potentially improve quality earlier. Maybe I am too  idealistic thinking that testers can provide greater value to an organization beyond finding bugs at the end of the development cycle.  Maybe our craft has reached the the apex of of our profession and this is as good as it gets.