Originally Published Thursday, March 12, 2009
There are many approaches to test automation from unit testing to system level testing through the GUI. Of course, the most often discussed approach is the automation approach that drives the GUI to perform some action; or GUI automation. This also happens to be the most controversial approach to test automation, and is perhaps the hardest type of automation to design and develop. One reason why an automated GUI test fails or doesn’t achieve its potential is due to a lack of understanding of the "system" by the tester, which in turn leads to a poorly designed test from the outset.
This problem is especially obvious when people who may have specialized business knowledge but lack a in-depth understanding of the systems they are working on (non-technical testers) are asked to ‘automate’ something. The automation in this case is usually in the form of using record/playback tools or perhaps creating a rote script to drive a keyword-driven framework, and the ‘test’ is usually consists of nothing more than merely mimicking some contrived behavior by the ‘tester.’ In fact, this over-simplified view of test automation is sometimes perpetuated by tool vendors. A manager at one tool vendor said, "By automatically capturing the tester’s process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating."
There are 2 fundamental problems with the above quote, and this approach to ‘automation’ (and I use that term loosely in this context). First, while I don’t totally discount the value of record/playback, and in the right context it could very well be the best approach in a specific situation, the general consensus in the industry is that due to the limited capabilities of record/playback type tools this type of automation is simply one level above using a hoard of monkeys trained to repeat a set of sequences using the keyboard and mouse. In fact, some might suggest trained monkeys may be a better alternative because bananas are much cheaper than the costs of licensing a tool and then realizing that you have to hire someone to try to patch together some proprietary script in an often vain attempt to build an automation test suite that is beyond the reasonable capabilities of a record/playback automation approach. Unfortunately, in either case the organization is usually left with a horrible mess that nobody wants to clean up. Secondly, it makes a ridiculous assumption that testers are too stupid to automate complex scenarios or test automation is a brain-dead, non-interesting, zero-value-add activity.
If simply recording or ‘documenting’ rudimentary scripts that essentially repeat a sequence of ‘hard-coded’ steps over and over again is someone’s idea of well-designed test automation then I would agree that automation is a brain-dead activity. When you automate poorly designed tests, you simply get poorly designed automated tests! And, since I don’t mind calling the kettle black, I will say it…recording a set of actions, or documenting a sequence of steps with hard-coded values to feed into a keyword driven architecture is not test automation! Surely it automates tasks, but well designed automated tests are much more powerful than the production of some crude script that automates the actions performed by someone sitting in front of a computer.
Of course, well designed automated tests require highly skilled professional testers with in-depth knowledge of the systems they are working on, as well as proficiency in programming concepts and languages. Similar to how doctors study anatomy, physiology, pharmacology, immunology, biochemistry, etc., professional testers need to constantly study the various systems they are tasked to test. Developing a well designed automated test is very different than simply using a tool to automatically repeat a sequence of actions. Designing robust tests (automated or not) requires not only incredible creativity and problem-solving skills, but an in-depth knowledge of the system and an understanding of how to manipulate the system programmatically.
But, if you buy into the idea that automation is simple and merely rote recording or documenting some sequence of steps performed by some person then you get exactly that; simplistic repetitive automated actions. Simple automation is simply automated simplicity.
5 Comments
Hi BJ,
I am not sure if I would be able to agree with a statement you made in your post:
If simply recording or ‘documenting’ rudimentary scripts that essentially repeat a sequence of ‘hard-coded’ steps over and over again is someone’s idea of well-designed test automation then I would agree that automation is a brain-dead activity.
Either you are wrong, or I have been performing automation testing with QTP incorrectly for some time now.
I have started a discussion on SQAForums:
http://www.sqaforums.com/showflat.php?Cat=0&Number=556980&an=0&page=0#Post556980
If you have a few minutes, would you be able to clarify on what you are intending to explain?
Monday, March 16, 2009 1:14 PM by Gaurav.Pandey
Hi Gaurav,
Since I do not know what you testing, your target customer profile for your product, or what the specific purpose of your tests are then I am not able to say whether or not your use of QTP is incorrect. As I said, I can envision a few specific contexts when record/playback automation is useful and provides some amount of information that is deemed sufficient by the decision makers and thus provides some value to the organization.
However, in my opinion, I would not hire a professional tester to sit down and record a scripted sequence of ‘hard-coded’ actions, or to fill out a spreadsheet or other form that is fed into a keyword-driven automation harness. I personally would consider outsourcing that type of work to a business analyst or a “domain expert” for those sorts of rudimentary tasks before assigning such mundane activities to a highly-skilled professional tester.
Also, when I refer to well designed automation, I am not referring to a rudimentary script that starts, performs a prescribed set of actions, and then makes some pass/fail determination based on one or maybe two outcomes.
To me, well-designed automation requires a procedural programming paradigm. Well-designed automation is able to run on multiple versions of an operating system, reusable on multiple revisions of a product, reusable on multiple langauge versions, easily distributed across different machine configurations, resusable in sustained engineering/maintenance efforts, increases my test coverage (not synonomous with code coverage), and perhaps most importantly, has some probability of providing some new and useful information each time it is executed.
Well designed test automation can also get “impatient” by including polling loops that timeout or count-out when some action/event takes too long. Well designed automation can get ‘aggravated’ by using a background worker to count the number of times an errant pop-up window or message box appears and bails out of the test process and logs the specific reasons for the premature exit. Well-designed automation is smart enough to know how to detect an errant pop-up window or message box appears and can make a logical choice on how to deal with it and either continue the test or abort the test and report the test result as inconclusive. Well-designed automation removes hard-coded constant values used in variable inputs and effectively employs various techniques to use variable inputs to significantly expand data coverage, combinatorial testing, or permutation testing. Well-designed automation is automation that a professional tester develops to control the computer as a tool for his/her advantage; not a tool that essentially makes a slave of the tester to the computer. (There is more…but, that is a post unto itself.)
Of course, there is still the option that I could be wrong. I have been wrong in the past, and I strive to learn from my mistakes. (Of course, well-researched, repeatable scientific evidence could also be used to sway my stated opinion.)
Ultimately this is my opinion (formal expression of professional judgment based on my experiences), and the perceived wrongness or rightness of anyone’s opinion should always be validated by the opinion of several other experienced professionals.
Monday, March 16, 2009 11:20 PM by I.M.Testy
Totally agree…In my experience, record-playback are also a no-go from an abstraction point-of view. We develop UI behavior abstractions that hide away control specifics from the daily grid of the regular test automation developers — record-playback gives a naked view and it just does not cope with the dynamics when daily builds are on the ball.
Tuesday, March 17, 2009 1:17 PM by chai
Great Post!!
I would like to say up front that no test automation tool is a silver bullet. Automation takes time, effort, and commitment from all involved, including an understanding from management about the realities of what automation can and cannot do.
Automation should stand for values like Re-usability, Maintainability and Stability. Automation is not a simple test activity; I wish it is equivalent to code development.
Wednesday, April 22, 2009 5:16 AM by verand
Hi Verand,
Thank you, you also make very good points that are spot on!
And I whole-heartedly agree that it should be on par or better than production code, because it must be ‘bullet-proof’ or without error. Everytime an automated test fails or causes a false negative the team loses confidence in the automated testing.
The industry has made great strides in test automation approaches in the past few years, and we must continue to find ways to improve the reliability, robustness, and the capabilities of our automated testing processes as our software grows in complexity.
Wednesday, April 22, 2009 1:06 PM by I.M.Testy