I.M. Testy

Originally Published Wednesday, August 02, 2006

My respected colleague Keith Stobie recently posted to his blog my response an internal query regarding “unrealistically high” goals for the percentage of tests that are automated. (Since, I have been remiss in my public rants I will reuse Keith’s post. Thanks Keith.) Of course, the unrealistically high percentage is probably near the 100% mark. There has been a lot of talk recently about 100% automation goals especially in the Agile world. So, for all those out there that think 100% test automation is a “reasonable” or even achievable goal I have a suggestion. Learn to just say NO to drugs!

There are some areas of testing where I do see the need and value of approximately 100% of the tests being automated. BVT/BAT is a prime example. When I was the Setup Test Lead on Windows 95 I was responsible for developing the BVT/BAT suite. The suite contained approximately 1000 unique functional tests and ran simultaneously on 4 different language versions within the span of 30 minutes once a week for about 1½ years upon delivery of each new build. But it was only maintenance free for the last 6 months of the product cycle. Still, that’s a pretty good return on investment considering Ed Kit, and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test. Other areas which lend themselves to high percentages of automated tests include performance testing, stress testing, regression testing, etc. Areas of testing in which I need to establish baselines and run consistent tests in order to compare repeatedly predictable results against those baselines towards some ultimate predefined goal are prime candidates for high levels of automation. Scripted test automation does a very poor job of exposing “new” defects post test design, and GUI automation requires enormous maintenance especially if it is created too early in the product cycle when the U/I is still in transition. The bottom line is we need to be smart about what and when to automate, because good reusable test automation is damn expensive.

For my unabridged response to managers or others who throw around the 100% automation mantra see my comments in Keith’s blog. For more information about making smart decisions of what and when to automate I highly recommend the book by Daniel J.Mosley and Bruce A. Posey entitled Just Enough Software Test Automation.

Originally Published Thursday, June 08, 2006

In the past few years there has been a push to use Ruby as a programming language for test automation. Ruby as a programming language has some benefits compared to other scripting languages. However, for test automation or any other serious application development the disadvantages of Ruby certainly outweigh any of the recent sensational hype promoting the language. I may be a bit biased, but I honestly can’t fathom why a tester wanting to learn a programming language would spend time learning Ruby, especially with the ease of use, availability of resources, and broad adoption of C#.

1. Ruby lacks informational resources. A search on Barnes & Noble, or Amazon reveals about 50 or so books on Ruby programming. But, that is barely a drop in the bucket compared to more than 400 books written about C#. These numbers certainly don’t inspire a lot of confidence in Ruby as a broadly accepted programming language in the industry at large. Web searches for available online resources also reflect this tremendous disparity. Sure, the Ruby zealots support the few websites and respond to requests for assistance. But, there are a greater number of C# forums with greater numbers of registered members who frequently participate and provide solutions to questions. Additionally, you won’t find too many universities or community colleges offering courses in Ruby programming. If Ruby is so good then why are there such limited resources? The answer is because there simply isn’t the business demand, or other compelling reasons for adoption.

2. Ruby is not a high demand skill among employers. Take a look at any of the technical job sites such as Dice, Monster, etc. and you will not find a plethora of jobs asking for Ruby programming skills. For example, of the approximately 16,500 software testing jobs on Dice only 65 contain the keyword Ruby as compared to 1,668 job listings containing the keyword C#. That means there are 25 times more employers desiring C# as compared to Ruby. IT Jobs Watch in the UK has an interesting site with lots of statistics relative to software testing positions in the UK. Looking at software testing jobs C# is listed in the top 5 desired programming languages. (Ruby doesn’t even make the top 20 list anywhere on this site.) The job trends on Indeed provides a visual perspective comparing jobs with C#, Ruby, Perl, and VB.NET keywords.

3. Ruby has performance problems. Scripting languages are notoriously slower than compiled languages, but it seems that Ruby is often slower (CPU time) and requires a larger memory footprint as compared to other scripting languages. The Computer Language Shootout site provides some very interesting benchmark results for various languages. One benefit of automation is the decreased time to execute a specific test. Now, you may be thinking that it only takes a few more seconds to execute a test in Ruby as compared to Java or C#. So, let’s arbitrarily say that each automated script in Ruby takes 5 seconds longer to execute as compared to the same automated test in C#. That may not seem like a big deal. But, instead of running one or two automated tests you want to run a test library of 200,000 tests. That’s about 12 hours of additional time needed to run the test automation written in Ruby. Now, I am sure Ruby advocates will discuss the reduced cost of development time, but time to develop an automated test is a one time expense (this does not include sustained maintenance which is a variable cost compounded over time for all automation libraries). Depending on your product’s shelf-life, you may need to rerun your test automation suites for the next 7 to 10 years for sustained engineering and product maintenance.

4. Ruby has not been widely adopted in application development. So, you may ask why this is a weakness for testers writing test automation? The simple fact is that if the development team is programming in say C/C++ or Java, and the test automation is in Ruby you probably won’t get a lot of support from the development team to help reviewing or debugging test automation. Also, it is very likely the developers may not want to install the Ruby interpreter on their Windows machine to use test automation to reproduce a defect, and instead ask for the manual steps. The test libraries the development team creates will require porting to Ruby which increases the cost and effort. Since many developers are familiar with at least the basic syntax of C/C++ and Java it is easier for them to pick up C# syntax and understand automated test code.

5. Ruby is just as cryptic as any other programming language. All programming languages use unique syntax, and users must learn the language’s syntax to code effectively. Now, I am no expert in Ruby but let’s compare a Ruby script to launch Windows calc.exe as compared to a C# program.

————————————————————–

# Launch calc.exe in Ruby

require “win32/guitest”

include win32::GuiTest

run_cmd “calc.exe”

—————————————————————–

// Launch calc.exe in C#

using System;

using System.Diagnostics;

namespace MyNameSpace

{

class MyClass

{

static void main ()

{

Process.Start(“calc.exe”);

}

}

}

—————————————————————–

Obviously there are more lines of code in the C# program as compared to the Ruby script. But, considering the fact the template in Visual Studio auto-generates the framework for a console application (the primary method of writing an automated test case) the only thing I need to add to the .cs file are the ‘using System.Diagnostics’ namespace declaration, and the ‘Process.Start(“calc.exe”);’ statement. Additionally, the IntelliSense feature of the Visual Studio IDE references language elements, and even inserts the selected element into the code. Also, perhaps it is a matter of personal taste, but Process.Start() seems a lot more ‘readable’ than run_cmd.

Ruby activists boast how quickly they can teach Ruby scripting to non-programming testers. I have been teaching C# to testers for more than 3 years. I have been very successful at teaching testers with no previous programming skills to write automated GUI tests in C# that will launch a Windows application, manipulate the application, generate and send data, verify and log results, and clean up the test environment within a day.

There may be some interesting features of Ruby, but don’t get sucked in by all the fanatical propaganda. Ruby has been around for more than 10 years and hasn’t replaced any language or garnered a significant following! The simple fact is that Ruby simply doesn’t offer anything revolutionary, and thus hasn’t compelled the development community to rush to adopt or support it. All programming languages have strengths and weaknesses depending on the application. But, for test automation Ruby is not the best choice as a programming language.

In my (biased) opinion, I think C# is a much better choice, and in a later post I will outline the benefits of C# as a programming language for test automation.

Published Thursday, June 08, 2006 12:55 AM by I.M.Testy

Originally posted Friday, May 19, 2006

We often limit our test automation projects to internal use only. This restricted use of test automation has contributed the typical sub-standard quality of automated test code. If you think production code is poorly documented you haven’t reviewed very much automated test code. But, since the test code is for internal use only we really don’t need to put as much effort into it as we do production code. Or do we?

Unfortunately, the restricted view of test automation as an internal only tool in the testing process has perhaps limited its potential and its overall value. But, as testing matures into a profession testers are going to have a greater impact not only during the product development lifecycle, but testers will also extend their scope of influence directly to the customers. You may ask how or why test automation be used by customers? Read the following scenarios, and think about the possibilities!

A company contracts for a customized software solution based on a specific set of requirements. At the completion of the project the product is handed off to the company who then conducts several suites of tests such as user acceptance tests and integration tests to test compatibility with the existing environment. Additional ‘acceptance’ testing by the company costs them time and uses their valuable IT resources to re-verify the product’s functional capabilities and other quality attributes. Now imagine the software company also provides well-documented automated test suites along with the product that demonstrates the product does what it is supposed to do, and does it in the customer’s environment. The customers ‘acceptance’ testing time is greatly decreased, their overall cost of adoption is decreased, and (just possibly) customer satisfaction is increased.

Another company purchases several Microsoft products such as Windows Server, Exchange Server, and SQL. But, it is often not as simple as installing these platforms and setting the preferred configuration. So, the company either staffs an IT department that rivals an IT team assembled by a Fortune 500 company, or hires a company similar to Avanade to put the pieces together and make them work with the desired configuration settings. Of course, either solution is going to cost the company a lot of money, time, and potentially use internal resources to get things setup correctly. Now suppose Microsoft creates an enterprise Software Test Kit (STK) that contains thousands of automated tests. The IT team identifies the specific configuration settings for its platforms, identifies applications in the corporate environment, and selects other requirements for adoption. Based on the IT department’s criteria the STK will automatically select a suite of tests the company’s IT department can execute to verify their platform configuration and other acceptance tests. Automated test suites in the STK can even be used for non-functional testing such as performance and stress testing by the company based on their configuration settings.

Are these potential uses of test automation by customers? I believe these are only some of the possibilities that exist. However, to realize this potential our test automation must evolve. The quality of our automated tests must rival that of our production code. In fact, test code must be better than production code because there is no room for errors or false positives. If customers use our automated tests to reduce their overall costs of adoption the tests must be bullet-proof or customer confidence decreases and customer satisfaction plummets.

So, what must change in order to increase the overall significance and potential usefulness of test automation beyond its current value? To begin with, automated test cases must be considered a product with a meaningful shelf-life instead of disposable artifact or something we simply pass on to the maintenance or sustained engineering team. The automated tests must be well-planned and well-documented (preferably with XML). The testers who develop the tests must understand the product and be able to develop the automated tests with a modern programming language. (Sorry, but scripting languages, keyword or data-driven approaches, and record/playback are simply too limited in their ability to scope to test automation of this magnitude.) Testers will need to use static analysis tools and perform code reviews to verify the quality of the test code. We will also need an framework that can select and run the appropriate set of tests to satisfy the customer’s needs, but is extensible enough for the customer to easily create and import additional automated tests if necessary.

Perhaps this is all just wishful thinking, or perhaps it is a vision of things to come in the professional world of software testing.

Originally posted Published Monday, May 08, 2006

I overheard some test managers discussing problems with their test automation effort, so I couldn’t refrain from asking the redundant question, “What is your test automation strategy?” They looked at me as if I had just beamed down from another planet and said, “c’mon, you know our strategy is to automate everything!”

It is unfortunately true that some managers drink the proverbial kool-aid and blindly regurgitate the 100% automation mantra or similar incantations such as “no manual testing” popular among agile pundits like Lisa Crispin.

Let me be clear. A goal of 100% automation is not a test strategy; it is a fantasy! Similar to the Disney fairytales where fairy dust causes magical transformations, evil is defeated, the prince marries the maiden, and everyone lives happily ever after forever automating everything is not practical or realistic.

Perhaps the single biggest problem with most test automation efforts is lack of a practical strategy. A practical test automation strategy is one that provides a pragmatic solution to address specific business needs with well-defined, measurable goals based upon realistic expectations.

Business needs drive a lot of the change in any organization, and usually involve cost saving measures, quality improvement, or increased customer satisfaction. A business need for test automation includes reduced testing time. (This doesn’t mean reduced ship cycles; it simply means the time it takes to perform certain tests during the product life cycle can be shortened.) For example, the Build Verification Test (BVT) is a necessary test suite to verify the stability of each new build. Depending on the size and complexity of the product a manual BVT suite can be very time consuming. An automated BVT suite (which should be 100% automated including results validation because it establishes a baseline measure on build stability and the tests remain relatively static over the duration of the development life cycle) can substantially reduce the time spent in this phase of testing especially in iterative build environments where the team is getting daily or even weekly builds. It doesn’t take long to realize the cost savings over the product life cycle.

Test automation strategies must also have realistic expectations. For example, I have never been convinced that finding “new’ bugs is a realistic expectation for test automation. (Yes, it will occasionally find some new bugs, but let’s face it…the majority of the 5 -15% of the bugs exposed by test automation in production environments are regressions.) I have never seen data that suggests increased automation reduces the overall development cycle. Nor will test automation eliminate testers. (This is a false hope imagined only by prima donna developers and bean counting managers scheming of ways to find value in their Masters in Business Mismanagement degrees.) So, what are realistic expectations for test automation? Well, I can reasonably expect test automation to identify stress issues such as mean time to failure (MTTF) and mean time between failures (MTBF). I can reasonably require test automation to establish baseline measures such as BVT suites or regression suites. Test automation is a pragmatic solution for load any type of load testing or other forms of concurrency testing.

Finally, a good test automation strategy must have measurable goals so we clearly understand what success looks like (or identifies where we need to improve). Without goals we are developing automated tests just to say we are automating. Unfortunately, I occasionally see teams with goals of automating n% of existing tests. This really doesn’t make much sense because it doesn’t take into account logical decisions of what tests should be automated (remember, not all tests need to be or should be automated), so some redundant tests or run-once type tests are automated (which may not be the best use of your limited testing resources). Also, the ‘existing’ set of tests is usually a moving target, so that means the goal is a moving target, which means we can never achieve the goal. Goals for test automation should be specific, measurable, achievable, realistic, and timely (SMART). Set short term and long range SMART goals for your test automation effort. For example, a short term goals might be 100% automation of the BVT suite within 1 week after the first build drop. Long term goals might include design elements and processes to transfer automation to sustained engineering or maintenance teams, or 100% language neutral automation that will execute on any localized (or pseudo localized) language version.

Test automation is expensive. Testers have a lot of work to do in a very limited timeframe, so it is important that we use our testing resources effectively. A well defined automation strategy will establish clear goals, set expectations, and provide practical, automated solutions.