Saturday, June 07, 2008 7:35 PM by Eric Hu’s Weblog

I really like your statement “with good automation, you will do more testing (accomplish more) than before.” I couldn’t agree with you more. Well-designed automated tests can provide accurate evaluations of functionality and allow a tester to design and/or execute different tests to increase overall coverage.

Some of those ‘extra’ things in an automation cycle that you mention are also in manual testing. For example setup of unique environments, and of course I there will always be a maintenance cost for formalized test cases (tests that are written).

Of course, there are some breakthoughs in test automation and manual testing to reduce setup costs such as virtual machines. Also, a test automation framework can distribute tests across the wire to any networked computer that matches a particular setup, or recreate that setup from an image (if it exists) much quicker than manual testing. For example, at Microsoft an SDET may have anywhere from 2 to 6 machines in his or her office. However, once thier automated test gets checked in to source control it can be distributed to any computer in that team’s lab (and with approx 120,000 lab machines around the company that automated test has the potential to execute on a lot more hardware and softwaer configurations than can be accomplished via manual testing within the same period.

We are also doing a lot of work with smarter deterministic and declarative type oracles, and also automated test result analysis. I talked a bit about this last year at some conferences. And we are starting to see these areas mature and providing increased value to our testing efforts.

I am well aware of the glorious stories told by automated test tool hucksters. In some cases, these tools provide what the customers need (and that is good), but in some cases I think customers are mislead, or perhaps the customer doesn’t really understand testing, or perhaps some managers are attempting to use a tool to perhaps solve a much larger systemic problem.

Oh, and I also agree with you that any comparison between manual and automated testing is usually vague and meaningless. Manual testing gives us information that an automated test can’t, and there is some information that an automated test can provide more accurately, or more timely as compared to a manual test. We must learn to use both approaches to their greatest benefit rather than view them as competing approaches.

Sunday, April 06, 2008 2:57 PM by I.M.Testy

Here are my reasons…

1. You would do lots of things “extra” in a automation cycle (automation setup, automation script failures/investigation/fix/rerun, analysis of results) that are not presenting in a pure manual testing cycle. Depending upon how good is your automation environment, Application under test and automation code – some times these extra items eat up time and net-net your overall cycle time with automation “might” be higher than pure manual testing time. Not withstanding the fact that any comparison between manual testing cycle and automation cycle are really vague and stupid.

2. What would you do if automation test cycle throws up some interesting bugs and development struggles to fix them … will you take your product to market faster than before?

We need to understand every cycle of testing reveals a set of bugs that need to be addressed so there is some development/testing work to be done after every testing cycle. Will automation reduce or eliminate that time? Not likely – those tasks are out of purview of automation

The fact that there is automation, you can gain some time and “accomplish” more testing but not faster time to market as there are other things to take care of anyway.

I always say “with good automation, you will do more testing (accomplish more) than before”. This has taken many IT managers by surprise. Some told me that they bought automation tool so that they can do less testing and believed the promise given by sales person of the tool vendor/offshore service provider !!! Funny but true

Shrini

Sunday, April 06, 2008 2:21 PM by Shrini

Reducing test cycle time with automation occurs within the overall product cycle. For example, when I wrote the automated BVT for the international versions of Windows 95 it ran approximately 1000 functional tests on 4 different language versions simultaneously within approximately 30 minutes. This automated BVT/BAT test suite saved more 15 hours per build in BVT/BAT time which then provided the test team additional time to design and execute an increased number of tests to expand depth and breadth of coverage.

There is a common myth that test automation will reduce the product development lifecycle. However, this is a foolish assumption for 2 reasons.

First, there are a greater number of all possible tests in any complex system then can be feasibly accomplished within any reasonable timeframe to make exhaustive testing virtually impossible. (The single biggest challenge in testing is to reduce the nearly infinate number of tests to a very small subset of tests that is going to provide the decision makers with the appropriate qualified information that will allow them to better analyze risks and make more informed business decisions.)

The second reason increased test automation typically does not impact the total development lifecycle is a result of Parkinson’s Law; “work expands so as to fill the time available for its completion.” Test automation may indeed reduce specific testing cycles within the overall development lifecycle; however because the potential test burden is so large the overall schedule will unlikely be effected to any large degree.

Saturday, April 05, 2008 5:06 AM by I.M.Testy

What do you say about this?

>>> Under what circumstances it is possible to achieve reduction in test cycle time by automation?

Shrini

Friday, April 04, 2008 11:30 PM by Shrini

Agreed. In my opinion many people make too many incorrect assumptions about the value a good automation effort provides to an organization.

I would also say that automated GUI testing can be useful in identifying certain types of usability issues. For example, missing key mnemonics, tab order, layout errors such as control overlap, truncation, etc. can be highly automated. (So, I guess I am that someone.

But, I also agree there are other aspects of usability testing such as the ‘ease of use’ perspective may never be automated, nor should it be.

Friday, April 04, 2008 5:24 PM by I.M.Testy

I often hear people saying in GUI automation world in IT scenario (where there is always pressure to less of testing hence more automation is pushed as possible way to achieve less testing … read human testing)…

Test automation cuts down test cycle time and enables faster time to market…

Yes it depends on what one mean by cycle time and how good the product is and how good the developers are and so on ….

Under what circumstances it is possible to achieve reduction in test cycle time?

Another things… I look ROI is three parts “Return”, “Investment” and “timeline”.

Articulating “return” is where most of us have challenges in attaching a $ value.

Most of GUI tool vendors claim (in IT scenario) that ROI can be achieved in definite time frame or number of test cycle without looking at release cycles etc … Pl note that it is a not a product scenario (daily build etc) like Microsoft. In IT, the test cycles are decided on the basis of high level business plan and release calendar hence without getting into the specifics of this … it is nearly impossible to predict breakeven and other things.

Unfortunately … such things happen and customers are taken for a ride.

Another perspective that I would like to share about IT scenario is lack know-how and support for building testability features so that automation can be built at a level bellow GUI.

When I talk about testability features … people ask “what”? Same goes with formalized unit testing … proliferation of xUnit frameworks and rigor on unit testing is by and large missing in IT kind of environment. Offshoring and development/testing split between multiple vendors complicates this issue … we reach a situation where no good unit testing can happen …

Now look at skill level, Testers in IT organizations and those in offshore services providers – typically lack skills to code at sub-GUI level. So they happily go with “easy-to-use” tools like QuickTestPro. Very few understand why is bad idea on the long run to code at GUI level.

So where does this take Testing and automation in IT … Do more GUI automation and cut down testing costs … It is kind of going in vicious circle of death …

>>> I do agree that GUI testing is useful in identifying usability type issues.

Let me reiterate here .. when you say GUI testing is useful in identifiying usability type issues … you are referring to human testing not test execution by machine (automation). I will not be surprised if some one states that they can perform usability tests by automation (without human users interacting with the application)

Let us not mix up – “usability testing/issues”, “GUI testing”, “Human testing” and “machine testing”.

Shrini

Friday, April 04, 2008 3:46 PM by Shrini

Short story: everyone involved in the process (test, dev, and PM) concurred, spurious failures fell from (some random number between 10-50% per run) to <1%. Best quote from a tester: “I no longer get a knot in my stomach when I have to do automation investigations.”

I feel justified with this decision and “losing” the UI based tests. After all, if I click the button to “make it green” and it is made gold instead, our dogfood use will hit that. We cna automate the functional test of making it green and gold separately.

John Guin

OneNote Test

Wednesday, April 02, 2008 5:46 PM by JohnGuin

I’m still on the fence with about unit testing. It’s got maintenance problems too.

Could you point me to the evidence – some objective measures of each type of testing? I haven’t seem much in the way of objective measures of unit testing ROI on the web.

Personally, I’d *love* to go with more unit testing, however, I’m afraid that I want to because I like to code, not because it identifies a maximum number of important, fixable problems. I frequently suspect this is why developers like it, in general.

Friday, March 28, 2008 12:54 PM by ian807

Great points regarding the inexperience and poor practices, and that was the intent of my message. I suspect that many people simply attempt to automate from the GUI too soon in the project.

However, I don’t necessarily agree with your comment that [the GUI] “seems to most consistently produce useful infomration about bugs.”

I suspect most testing approaches over the past few years have tended to focus on testing from the GUI; however, there is a lot of evidence that demonstrates the single most effective means of identifying funcitonal and security issues is code reviews or inspections, and well designed unit testing can also consistently produce information about functional problems.

I do agree that GUI testing is useful in identifying usability type issues, but a lot of functional issues could be identified much earlier in the process…of course it all depends on how much an organization is willing to invest in testing.

Friday, March 28, 2008 12:29 PM by I.M.Testy