“We have the technology, we can rebuild it.” *Play intro Music* “The One Billion Dollar Test Suite!”

This is generally where automation projects become cancelled automation projects. People simply don’t understand that automation, just like everything else, subscribes to that pestering 80/20 rule, where 20% of the test cases account for 80% of ROI.

Unfortunately, GUI automation does NOT provide us with the silver bullet that killed the software quality analyst. In fact, I sometimes think that we’d get a better ROI from an automated butt scratcher than a GUI.

However, if we phase our development properly, like every software development project we work on (HA!!) then we can do the highest ROI work up-front and give management the warm fuzzies while we develop the stuff that shows a little less ROI

Hi Brent, Automated butt scratcher??? Now if you’re talking about an automated tingler head massager http://www.bodytimewellness.com/tingler/a rel=”nofollow”> I’m all in!

I wrote in the past http://www.testingmentor.com/imtesty/2009/11/18/measuring-test-automation-roi/ rel=”nofollow”> and http://www.testingmentor.com/imtesty/2009/11/18/test-automation-roi-part-ii/ rel=”nofollow”> that I think trying to “calculate” automation ROI is a silly wasteful pursuit. It’s akin to the rediculous notion of measuring quality based on the number of bugs found, or some magic code coverage metric.

I design a test case because it has a high potential to provide me with important information that is relevant to the people who make decisions. I design and develop automated tests to provide value to me in the ability to achieve the first objective more efficiently and/or effectively.

Silly measures such as automated:manual test ratios, or percentage of bugs found by automation vs. other means, or raw count of automated tests are meaningless. Bone-headed managers who want to track these sorts of numbers will eventually get the numbers they want, but testers will be focused on doing things just to impact the metric, and not necessarily on providing automation that provides value.

When someone figures out how to objectively measure ‘value’ for me in each uinque context I will be all ears. Until then…

And similar questions arise when we decide to test at levels below the UI.

If automate at the API level only, then we either explicitly or implicitly decide that the UI and API layers are connected appropriately, or at least that the connection doesn’t need to be tested through our automation.

Thinking first, then consciously deciding the appropriate test approaches, is always a good choice.

[Bj's Reply] Hi Joe, I wouldn’t suggest only automating at the API layer, but you are absolutely right that we must explicitly or implicitly decide how to test the “connections” between the UI elements and the event handlers, and the event handlers and the “business logic layer.” I think this brings us back to the “Think, Then Test” concept.

[Bj's Reply] As do I! Automation in of itself isn’t bad, it’s when we (or our managers want us to) automate for the sake of automating. But, to play devil’s advocate, if we as testers don’t really understand what is going on below the UI then I suspect any automated tests we design might also be rather superficial.