I.M. Testy

When someone compares agile with “traditional’ development I have no clue what they are talking about. I have never worked in a software company that produced software as if it were a widget on a factory assembly line. I have never worked in an organization where people didn’t talk to each other constantly. I have never worked for a company that didn’t periodically reflect on its own processes and/or procedures to identify areas of improvement. I have never worked for a company that wasn’t capable of adapting to changing to ‘requirements’ when necessary. And, I would say that the majority of people that I have worked with are highly motivated individuals who strive to do the best possible work, and who are capable of adapting, improvising, and over-coming obstacles in order to ship complex world class software.

This is the way it has been for me during my almost 2 decades in this industry. So, when I read books and papers that compare agile to “traditional” approaches I ask myself, “What in the hell do they mean by “traditional” approach?

For example, when I joined the Windows 95 team we had weekly builds, and at least once per week the build had to satisfy the requirements for “self-host” status. Self-host meant that anyone in the company could use that build for their day to day work. Also, many of the self-host builds are released to key strategic industry partners and 3rd party vendors so they could take advantages of new features in their software.

After the PDC release of Windows 95 we made several key changes in the operating system based on feedback from our customers and partners. Also, late in the cycle after beta 2 we had to make a major change in how Windows started because of a quirk in a relatively popular MS DOS based game. Throughout a typical lifecycle features are added, removed, and tweaked based on several factors including customer feedback.

Instead of sprints we generally have milestones. At the end of each milestone many teams have a post-mortem or retrospective to review key objectives, discuss things that went well, things that could have been improved, and  “tune and adjust its behavior” to make sure everyone is aligned and find ways to increase effectiveness.

Most developers were also doing unit testing. (TDD is simply another way of restating Dave Gelperin’s and Bill Hetzel’s famous quote from around 1987 “Test then code.”). Testers and developers talked frequently, and on many occasions the developer I worked closely with would come to my office to debug a hard to reproduce issue. And the majority of testers who were hired in the early 90’s were typically required to have an in-depth technical and often coding background in order to be able to engage in all aspects of the software development lifecycle and perform all the various job roles that might be required from a testing professional.

By 1998 we were doing daily builds. We have well established unit test libraries that not only assist developers in refactoring, but also help prevent build breaks and down-stream regressions. We still do a lot of API (component) level testing, as well as integration and system level testing.

Of course some of our products have longer release lifecycles than others. However, as indicated earlier we deliver “working software” at least once a week internally, and even to some premier customers (especially after the third milestone leading up to the final release of that version. But, some of our teams release every month.

Stepping back and looking at the debates, the books, and conference presentations I realized that I can relate well to the Agile principles. We often use models, principles and other abstractions to describe things. And, when all is said and done Agile principles are simply another way to present concepts that are intended to enable a team of highly motivated people to work together to ship a high quality product that satisfies their customers.

We have different models to generalize our processes to help us describe our typical workflow to others. But in truth there is no single way to build software, and models are simply our abstract expression of how we view the process. In fact, each team chooses processes and procedures that work for them and their particular product in their unique context.

(BTW, I am also really tired of Agile pundits always comparing Agile to waterfall models. Why don’t they compare their understanding of Agile to the V-model, W-model, or especially to spiral or other iterative models of software lifecycles? These are models; how your team chooses to implement a development lifecycle may resemble one of these models or you may adopt things from different models to implement your processes and approaches to software development.

Remember, George Box stated, “All models are wrong, some models are useful.” We should implement the concepts embodied in the model;  we should not implement the model.)

So, to me, Agile is the ability of a team to interact with each other, to adapt to changes, to periodically reassess its productivity in order to strive to become more effective and efficient in delivering software to its customers. Agile (and every other lifecycle models) is not a process; Agile is a mindset.

Last month I was travelling throughout Europe. A few days in Switzerland to speak at Swiss Testing Days and visit our communication server team in Zurich. About a week and a half in Denmark teaching at our offices there, a few days in Dublin teaching, and finally a full day meeting with one of our teams in Reading, UK. The only good thing about being on the road that much is getting in more reading time. Two of the books I read while travelling were Agile Testing by Lisa Crispin and Janet Gregory. The other was Clean Code by Robert Martin. I found both books thought provoking and informative. Of course I didn’t agree with everything, but even the statements I didn’t agree with made me think about that topic a bit more. Martin’s book inspired me to improve my testing and coding craft. If nothing else, testers should read the first chapter of Clean Code, those who design and develop automated tests should read the whole book.

But, one statement that has always troubled me that I saw repeated in the Agile Testing book is in regards to exploratory testing as the most effective way to expose important bugs. Every time I hear/read this or one of its many derivatives it makes me ponder. If this is really true then why do we invest so much time and resources in other approaches to software testing. Why do we spend so much time in static and dynamic analysis, reviews, and coverage analysis? Why do we try to prevent bugs when they can (and presumably are) found easier and ‘quicker’ downstream via exploratory testing? Why don’t we simply continue to hire hoards of diverse people to rapidly explore the product just before release and beat out all those important/critical bugs?

I wonder why it seems that we can ‘think critically’ when we have the product to play with, but we can’t seem to apply critical thinking while reviewing documented requirements, or a model? Personally, I am not sure that ‘trying’ something and then deciding what to do based on the outcome or state of the computer, or deciding whether something is a bug after the fact constitutes critical thinking. Perhaps sometimes it is, but I suspect that sometimes it is just guessing. To me, critical thinking involves the ability to foresee potential issues; not simply stumble upon them, or apply an attack from my bag of exploratory tricks to reveal them downstream in the development lifecycle when I have the product to play with. 

I sometimes wonder if exploratory testing really is the holy grail of software testing, or if we have just given up on other approaches because they are too hard. Could it be that our testers are not capable of ‘testing’ without also engaging their fingers? Are our testers untrained or under-trained in our profession? Is the reason why our pre-defined test cases are seemingly ineffective due to the realization that we suck at test design, and learning how to design more robust and ineffective tests is just too hard? Is the reason we can’t think of tests until we get the product in hand perhaps due to our inability to model or is it because we really don’t know that much about the systems we are testing? (You can’t model what you don’t understand and you don’t understand what you can’t model.) Besides, why should we burden testers with the overhead of learning about programming concepts, computer science, and math. You don’t need all that stuff to find bugs.

Now many of you might think that I despise exploratory testing. I am not anti-exploratory testing (whichever form that takes); because we all do it! Show me a tester who doesn’t include some form of exploratory testing in his or her repertoire and I’ll show you a thoughtless simpleton who needs explicit instructions for determining which side of the heal of a loaf of bread to butter. But, I have always thought that we could actually improve the craft of testing. I thought we might play a much bigger role in an organization. I thought we could help reduce overall production costs by being involved earlier and throughout the lifecycle. I thought we could help reduce risk through defect prevention and in-depth analysis. And, I thought we could learn to use tools and techniques to help us design better tests up front, and rely on a team of testers who are capable of thinking while executing a test case.

Yes, it is true that exploratory testing is fun, and we don’t really need to understand all that computer science stuff to find bugs using an exploratory approach. Regardless of what empirical studies on the topic reveal, exploratory testing ‘feels’ so much more productive. Why should we base our profession on evidence when we can get by so easily with emotion? Why should we have to think critically during the design or implementation of software when we can simply muddle our way through some graphical user interface to find bugs? If our approaches to unit testing and component level testing are so shoddy as to simply invoke happy path” tests then of course finding bugs by exploration seems rather logical. But, as developers become more adept at unit can component level testing in flushing out functional bugs earlier we may only need ‘testers’ to explore the product before release for behavioral issues, compatibility problems, and the few remaining obscure functional bugs. Or possibly exploratory testing providing a ‘good enough’ service to our team by finding the bugs that matter to the majority of customers, and perhaps that is ‘good enough.’

Personally, I think we can improve our testing processes. I think we will need to improve the practice of software testing. I think that our job is not simply to find bugs, but a more important objective is to prevent defects (which can help reduce overall product costs). I also don’t view testing as destructive, and I think those who do are doomed to become obsolete. Successful businesses don’t hire folks who just want to ‘destroy’ the products they build, I suspect they want to hire people who can help the company grow and improve. Finding bugs helps me ship a product, but it doesn’t necessarily help me grow the business, reduce production costs, or improve quality long-term.

But maybe we don’t need to change. Maybe our businesses are satisfied with the status quo. Maybe we don’t need to think about cost savings or helping our teams mature their processes, reduce overall risk, and potentially improve quality earlier. Maybe I am too  idealistic thinking that testers can provide greater value to an organization beyond finding bugs at the end of the development cycle.  Maybe our craft has reached the the apex of of our profession and this is as good as it gets.