I.M. Testy

I remember going camping with my cousins as I was growing up. It was great fun despite sleeping inside a musty smelling canvas tent that retained heat so well it was more like a sauna. But, my father was adamant that the old canvas tent he bought at an Army Surplus store and took 2 men and 4 boys to carry and assemble was much better then those new fangled nylon tents. Nylon ripped too easily he reasoned, but canvas will withstand anything short of a raging bear. I’m not too sure there were too many raging bears roaming the camp-grounds of Maryland, Pennsylvania, and Virginia but I was confident that even if there were I would have stood a better chance inside a canvas tent as compared to those paper thin nylon tents. To this day I remember those camping trips when I smell old canvas, or perhaps it’s dried mold spores embedded in the canvas. Whichever, it takes me back to a time of fun and fond memories.

One of the best parts of the trips were sitting around the camp fire at night listening to my uncle concocting some story intended to scare the wits out of us young boys. You know, the kind of stories about headless Confederate soldiers, or werewolves, vampires, or other such wicked creatures of the night. I think these campfire chats are remnants of man’s tribal roots where the elders tried to scare the hell out of the juvenile hominids to prevent them from wandering off at night. As we got older we realized that these stories were simply fictitious folktales; sort of like successful, value-add GUI automation projects.

Last week I had lunch with a colleague who wanted to talk to me about an automation project on his team that went horribly awry. As he started to tell me his story I thought, “Wait…I heard this tale before. I can tell this story because I’ve heard it so many times over and over again…just like those scary stories I heard around the campfire growing up.” The story goes something like this.

Our team bought a new tool, or built another framework, and taught everyone how to script “black box” test cases. They developed quite a number of automated test scripts, and of course everything was working very well. The scripts were running and managers were happy because the team had a lot of automated test scripts. But, the tests weren’t finding any bugs, so just out of curiosity the managers suggested a bug bash. And sure enough as the testers started exploring the project it didn’t take long for them to fill the database with bugs. The developers were shell shocked, and the managers couldn’t believe it! They couldn’t understand why the automated GUI tests weren’t finding any bugs? And so, in a typical knee-jerk reaction fashion, the managers immediately halted the GUI automation project and required every tester to embark on an exploratory testing adventure in search of bugs. Of course, the managers decided this approach was better than investing in more GUI automation bringing an end to another GUI automation project.

Unfortunately, unlike the scary stories my father and uncles told around the campfires, stories of failed GUI automation are often true, and usually much scarier. Why are they so scary? Because I hear these sorts of stories repeated so often. It seems that we as a discipline rely on tribal knowledge where each generation simply learns through trial and error and the folktales of our elders and thrive more on hero worship of people who are often remarkably good at finding bugs by poking and prodding hour upon hour.

Now, if you haven’t caught on already you will know that I am no big fan of GUI automation. Not because I don’t think it can be useful. In fact, I think GUI automation can provide tremendous value in some situations. But unfortunately much of the automated GUI test cases I see (especially in examples) are poorly designed, simple rudimentary script-lets. Many of these automated tests are nothing more than mindless automated sequences of events contrived because the testers have been told to automate, but are not given strategic vision (why) and little to no tactical direction (what and how).

With little or no direction or goals, or without an in-depth understanding of the system they are testing the biggest problems with GUI automation is that many testers attempt to automate

• functional tests intended to expose computational errors in the business logic layer or in the underlying APIs
• usability tests intended to imitate ‘me’ trying to emulate the scenarios or tasks I think the customer might do

GUI automation is probably the least effective approach for functional testing. This is not to say that GUI automation will not find functional issues in the lower logic layers. I suspect we will always find ‘functional’ bugs (e.g. boundary issues, unhandled exceptions, string parsing errors, calculation problems, etc.) while testing through the UI. But, as indicated in my previous post, well-designed software is usually built in layers and a good many of the ‘functional’ issues we find today can likely be more efficiently found through more robust unit and component (API) levels of testing.

Perhaps even more silly than trying to use GUI automation for low level functional testing is the notion of using GUI automation to emulate a ‘user’ by scripting out prescriptive sequences of actions (often with hard-coded data) that are then played over and over again. Test automation cannot and should not attempt to replicate ‘me.’ I’ve said before the purpose of automation is to provide value to me, to free up my time, to increase my efficiency, and to help me be more effective in my job; automation does not replace me. Let’s face it…we (humans) are much better at evaluating the ease of use of software and whether scenarios that represent target customer segments are intuitive for those customers.

For example, I once had a conversation explaining that GUI automation runs much faster than I can interact with software, and sometimes I make mistakes when typing in something that throws an unexpected message or takes me down a path. My colleague replied, “Well, we can slow down the automation.” Why? Why in the hell would I want to slow down my automated tests? C’mon…we all should know by now that the 100% automation (or automate all tests) mantra is a ridiculous dream and I’ve heard more plausible fantasies from people on acid trips.

So, where does GUI automation add value. In my opinion, GUI automation is probably most effective in testing UI control properties and the event handlers between the UI layer and the API layers. It is also effective in behavioral testing areas such as performance and stress.  And GUI automation is also much more effective in evaluating UI layout issues such as misaligned controls, or clipped or truncated controls on a window as compared to the human eye.

Similar to how we use different techniques to expose different categories of defects, and how we use different approaches to testing depending on the context or test objective test automation is a useful tool in our toolbox. It is certainly not the only tool. We can do some remarkable things with automation, but we must learn where GUI automation adds value and where other approaches testing (automated or manual) might be more effective.

For the past few weeks my test automation class at the University of Washington has been focused on API (application programming interface) testing, or component level testing. Boris Beizer defines component level testing as “an integrated aggregate of one or more units” and that a “component can be anything from a unit to an entire system.”

This seems a bit confusing at first but then we realize that a single method (or function) may be a unit, or a component, or may (although unlikely) be the ‘system.’ A collection of methods wrapped in a library or DLL that interact to meet a functional requirement is a component. Rather than a developer having to call each method individually to achieve some usually repetitive functional outcome from the library that functionality is usually exposed via a call to a single API.

In this situation the students are testing a public API in a single library (DLL) that calls several methods to produce a randomly generated string (outcome) based on parameterized property values. The interface is the single API call (and the property variables) in an automated test, so we might consider the DLL that contains this API as the ‘system’ under test based on Beizer’s definition of component. Also, since students don’t actually see the underlying code, API testing in this context is ‘black-box’ testing.

The debate of who is responsible for API or component testing is tangential to the practice. I promote API testing because even in today’s extreme programming and TDD development lifecycle models we testers are still finding way too many ‘functional’ bugs (as opposed to behavioral bugs) during the integration and system levels of testing.

Also, generally (not all) software is designed and developed in layers similar to this simplified illustration. In well-designed, more easily testable projects the business logic or logical functionality is (should be) contained in classes or libraries (DLLs), and the public methods or APIs in those libraries know nothing about the user interface. and visa versa. End user inputs at the GUI are marshaled to the business logic layer via event handlers and properties (get/set accessors in C#) in different classes.

So, it shouldn’t be a surprise to anyone that certain categories of functional issues are more easily exposed at the API level of testing as opposed to testing through the UI. In fact, sometimes the UI properties and event handler layers in one project may actually mask some bugs in the APIs which aren’t exposed until much later when someone else uses that API in a different application or feature. The value of API testing is that a lot of functional testing can be performed very early in the project cycle, and functional testing can progress while the UI layer is unstable or in flux.

I sometimes think we are stuck testing from the end-user’s perspective. It seems that we often approach testing by trying to expose both behavioral type bugs and functional type bugs by testing completely through the UI. But, if we think of testing in layers the same way many products are developed then I wonder if we could better focus our test designs to target specific categories of functional bugs earlier and concentrate on behavioral issues and end-2-end customer scenarios when we have a more stable UI?