Thank you, you also make very good points that are spot on!

And I whole-heartedly agree that it should be on par or better than production code, because it must be ‘bullet-proof’ or without error. Everytime an automated test fails or causes a false negative the team loses confidence in the automated testing.

The industry has made great strides in test automation approaches in the past few years, and we must continue to find ways to improve the reliability, robustness, and the capabilities of our automated testing processes as our software grows in complexity.

Wednesday, April 22, 2009 1:06 PM by I.M.Testy

I would like to say up front that no test automation tool is a silver bullet. Automation takes time, effort, and commitment from all involved, including an understanding from management about the realities of what automation can and cannot do.

Automation should stand for values like Re-usability, Maintainability and Stability. Automation is not a simple test activity; I wish it is equivalent to code development.

Wednesday, April 22, 2009 5:16 AM by verand

Tuesday, March 17, 2009 1:17 PM by chai

Since I do not know what you testing, your target customer profile for your product, or what the specific purpose of your tests are then I am not able to say whether or not your use of QTP is incorrect. As I said, I can envision a few specific contexts when record/playback automation is useful and provides some amount of information that is deemed sufficient by the decision makers and thus provides some value to the organization.

However, in my opinion, I would not hire a professional tester to sit down and record a scripted sequence of ‘hard-coded’ actions, or to fill out a spreadsheet or other form that is fed into a keyword-driven automation harness. I personally would consider outsourcing that type of work to a business analyst or a “domain expert” for those sorts of rudimentary tasks before assigning such mundane activities to a highly-skilled professional tester.

Also, when I refer to well designed automation, I am not referring to a rudimentary script that starts, performs a prescribed set of actions, and then makes some pass/fail determination based on one or maybe two outcomes.

To me, well-designed automation requires a procedural programming paradigm. Well-designed automation is able to run on multiple versions of an operating system, reusable on multiple revisions of a product, reusable on multiple langauge versions, easily distributed across different machine configurations, resusable in sustained engineering/maintenance efforts, increases my test coverage (not synonomous with code coverage), and perhaps most importantly, has some probability of providing some new and useful information each time it is executed.

Well designed test automation can also get “impatient” by including polling loops that timeout or count-out when some action/event takes too long. Well designed automation can get ‘aggravated’ by using a background worker to count the number of times an errant pop-up window or message box appears and bails out of the test process and logs the specific reasons for the premature exit. Well-designed automation is smart enough to know how to detect an errant pop-up window or message box appears and can make a logical choice on how to deal with it and either continue the test or abort the test and report the test result as inconclusive. Well-designed automation removes hard-coded constant values used in variable inputs and effectively employs various techniques to use variable inputs to significantly expand data coverage, combinatorial testing, or permutation testing. Well-designed automation is automation that a professional tester develops to control the computer as a tool for his/her advantage; not a tool that essentially makes a slave of the tester to the computer. (There is more…but, that is a post unto itself.)

Of course, there is still the option that I could be wrong. I have been wrong in the past, and I strive to learn from my mistakes. (Of course, well-researched, repeatable scientific evidence could also be used to sway my stated opinion.)

Ultimately this is my opinion (formal expression of professional judgment based on my experiences), and the perceived wrongness or rightness of anyone’s opinion should always be validated by the opinion of several other experienced professionals.

Monday, March 16, 2009 11:20 PM by I.M.Testy

I am not sure if I would be able to agree with a statement you made in your post:

If simply recording or ‘documenting’ rudimentary scripts that essentially repeat a sequence of ‘hard-coded’ steps over and over again is someone’s idea of well-designed test automation then I would agree that automation is a brain-dead activity.

Either you are wrong, or I have been performing automation testing with QTP incorrectly for some time now.

I have started a discussion on SQAForums:

http://www.sqaforums.com/showflat.php?Cat=0&Number=556980&an=0&page=0#Post556980

If you have a few minutes, would you be able to clarify on what you are intending to explain?

Monday, March 16, 2009 1:14 PM by Gaurav.Pandey