We started with GUI test automation on one of our products with smoke test prep — installing the daily build, configuring the basics, saving time for a manual tester by giving him something to test further with, than having to start configuring from scratch. And of course, we test the configuration process during the automation. As we started off with automating the smoke tests first, this enabled us to address most of the features and their basic use cases. Once we had a steady base, we extended our GUI tests to address performance tests (which you mention): measuring app responsiveness and assisting automated memory leak analysis.

Currently, we are also progressing with the automation of functional system tests that are either manually too laborious, or of lower priority but ones that still counts (because the manual tester would test the main use cases anyway), or plain better suited for automation, like some of the alignment and placement topics that you mentioned. As the app’s still fresh, there is need for weekly regression system test runs, and with limited manual tester resources, GUI test automation provides a respite by adding about 20% to the count of system tests run per week.

In the end of the day, the key to our GUI test automation sustenance has been ensuring testability, delivering results to retain the management’s interest in continuing to support testability, and guiding and tracking the test automation team’s work just as it were a development project.

It is very reassuring for me to read your view of GUI test automation, and those of other readers here. Look forward to more of your posts!

[Bj's Reply] Hi Chai, great example! Thanks for sharing.

[Bj's Reply] Very well said.

The only use of GUI automation I have found of any use is for data entry via the front end system

[Bj's Reply] Thanks John. GUI automation can add value in other ways as well; especially in some types of behavioral or non-functional testing such as performance, stress, etc. The problem comes from trying to automate everything through the GUI. But, I suspect that if our only view of software is the UI then GUI automation is the answser to every problem…sort of like…if the only tool we have is a hammer then everything looks like a nail.

[Bj's Reply] Hi Li, Yes these same problems may occur with any type of automation; although I will say that in my experience timing issues and synchronization problems tend to be a lot less at the API level as compared to the GUI level. Automation is a wonderful tool, but in my opinion it is often misused. If we are looking for increased bug count from our automation we are likely not be be satisfied with the results. The #1 lie about test autoamtion is that it will find more bugs. (Someday I will do a post on my perspective of big automation lies.) I also agree with you that pointy-haired managers who are fixated on simple numbers also tend to lead a team towards project failure because their perspective is one-sided and often biased.

In my experience, your tale of “They couldn’t understand why the automated GUI tests weren’t finding any bugs?” happens just as often when the string “GUI” is omitted.

Tests (automated, GUI, API-level, functional, usability, and otherwise) are only valuable if they are finding the bugs that exist.

[Bj's Reply] Hi Joe, You are quite right that any automation that is poorly designed has a low probability of adding value to the testing effort or exposing any bugs that might exist. But, just like highly skilled testers, even well designed automated tests may not find all the bugs. To me the greatest value-add of test automation is that it can increase test coverage of variables much more efficiently than humans and thus reduce overall risk and increase confidence.

Also, in my experience, another problem I have seen is that new testers will automate a test and then massage the test to make it work while completely overlooking a bug; bascially they are automating around the bug. Certainly while this can happen with any automation approach at any level, in my experience I see this more in GUI automation efforts.

In the end I think the value (or lack thereof) of automation depends heavily on the ability of the tester to design automated tests using a variety of approaches to most effectively expose or reduce risk. Poorly designed automated tests simply adds to poor automated testing.