I.M. Testy

Originally Published Wednesday, January 10, 2007

There is a lot written about regression testing, and yet there seems to be a lot of confusion about regression testing as well. Just to make sure we are all on the same page, by regression I am referring to the denotation of the word to indicate a relapse to a less perfect or developed state (American Heritage Dictionary). So, the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition. It is important to note the purpose of a regression test suite is not necessarily to expose new defects. The primary purpose of a regression test is to identify changes in behavior from a previously established baseline, which is supported by Beizer’s and Myers’ definitions of regression testing.

However, even on small projects the number of tests required to ensure new builds do not regress or change previous functionality can be quite numerous. So, regression testing demands a strategy in which we limit the number of tests to establish an effective baseline measurement. In IEEE 610 documentation it states regression testing is selective retesting. Thus, the key to an effective regression testing strategy is to design a test suite that provides a high degree of confidence without retesting everything. To limit the number of tests in the regression test suite, we must systematically reduce the number of possible tests. So, we must decide what tests are included in a regression test suite?

Deciding what tests to include in the regression test suite

The most effective regression test suites I have seen include two categories of tests. The first category of tests includes high priority tests for commonly expected functionality (e.g. the 20% of the product that 80% of the customers demand or rely on). The second category of tests includes any functional defects that are found and fixed. Found and fixed functional defects are included because fixed defects do occasionally regress, and if a business decision was made previously to fix a defect then we probably want it fixed before we release the product.

Prioritized feature area/functionality buckets

The tests in the regression test suite should also be partitioned into functional areas and each test in each functional partition or bucket should also be prioritized based on risk assessment criteria.  If the regression test suite is especially large or time is limited, and the regression suite is portioned into functional areas (and those areas are mapped to the project files or modules contain that specific functionality and any dependencies) the regression test pass can execute a limited subset of tests from the regression test suite that strategically target the modules that have changed (and tests for dependent modules as well). Simple directory comparison tools (such as Diff2Dirs), and tools to identify dependencies between modules (such as Depends) are useful in identifying which modules change between builds and to map out dependencies between the modules in each build.

Automate, Automate, Automate

Also, since the regression test suite will ideally be ran on each new build, this is one suite of tests that should be 99.999% automated. Similar to the BVT/BAT test suite the purpose of the regression test suite is not necessarily to expose defects; a regression test suite provides baseline measurement of functionality. Therefore, since these are tests that will be ran several times during the software development lifecycle and are not necessarily designed to expose new defects the ROI for automation is very high. In fact, any test that cannot be automated is suspect for inclusion in the regression test suite.

These are a few ideas to develop a highly successful automation strategy. What other tactics have you found to be successful?

Originally Published Sunday, December 24, 2006

Well, for those of you living outside the Pacific Northwest you are probably unaware of the recent wind storm with winds gusting to 60+ miles per hour that left more than 1 million people on the eastern side of the state without power. The damage was pretty extensive, and since I live in a fairly remote area I was without power for more than 7 days and without the Internet for almost 9 days. I do have a generator, but it hadn’t been used in almost 4 years. Sure, I started it every 6 months for about 15 minutes each time, but after the first full day of operation the generator started doing wierd things. So, during the past week I have become pretty good at fixing generators (mine and my neighbors), tracing electrical systems, troubleshooting furnace problems, splitting a lot of firewood, cutting up fallen trees, and repairing fences.

After the sun set (which is quite early) I had little else to do (other than making sure nobody stole my generator), so between stoking the fire I started developing a DLL for Unicode string generation in automated tests based on the GString utility. While reviewing the data tables I created for the GString utility with the Unicode Handbook I noticed some holes (OK…defects). Some of the boundaries for code ranges that are not assigned to any Unicode script group were incorrect. (That will teach me to use a web page with the listing put together by a web developer rather than using the Unicode handbook.) But, I also found a problem that prevented unassigned code points from being generated even if the Only use assigned code points check box was unchecked.So, the (hopefully final) update to GString is complete, including the GString.DLL! So, along with the massive overhaul of the Unicode data tables, the new GString package available from my personal website also includes a new DLL for anyone needing to generate strings of random Unicode characters in test automation. The GString zip file also includes detailed documentation on the utility and the dll usage. Let me know if you have any questions about the tool or using random string generation in your testing.

Well, now back to (mostly) normal life.

Originally Published Friday, August 25, 2006

I was stuck at home this weekend with house painters pondering how best to spend the day sitting at home, so I decided to actually play around with the Ruby scripting language a bit. I will admit my bias in my previous post about the Ruby scripting language was based on business justifications and personal preferences towards the C family of languages. But, because Ruby is still being pimped by some experts in the industry for test automation, I figured I should understand more about the mechanics of language other than it’s lack luster appeal by many professionals in the software industry.

I didn’t delve very deeply into Ruby. A few chapters of Hal Fulton’s book The Ruby Way and some additional research on the web confirmed my suspicion that Ruby for all purpose software test automation is simply not a good investment in your time given the other alternatives.

International Support Sucks!

If you are stuck working on small projects in English only then this probably won’t bother you too much. But, if you realize the majority of the world does not communicate in English then Ruby’s lack of international support simply sucks. Unless there is some hidden secret in the Ruby society the fact is Ruby can’t do Unicode. Yes, you can store UTF-8 encoded data in 8-bit encoded strings, but many Ruby String methods assume single byte encoding. I have to admit this completely blows my mind. UTF-8 is a popular encoding form of Unicode; however, it is only one of several Unicode transformation formats. So, forget about UTF-16, and definitely forget about testing Unicode surrogate pairs or conformance to the GB18030–2000 Chinese encoding standard. So, what does this mean for testers? String testing is basically limited to a very small subset of characters.

Multiple ways to initialize arrays (and just about anything else)

Hal Fulton lists the following 3 ways to initialize the array class method.

a = Array.[](1,2,3,4)

b = Array[1,2,3,4]

c = [1,2,3,4]

I don’t know why this would be a ‘cool’ thing to do, but I do know that it increases confusion, and potentially adds to the cost overhead of white box testing and code maintenance.

I guess it’s pretty cool that arrays in Ruby contain objects rather than data types, and an array in Ruby can contain integers, strings, etc. I am not quite sure why this is a positive because I can’t think of a situation where I would want to iterate through an array with disparate data types.

Cryptic codes

If Hungarian notation wasn’t bad enough, just stir in some Ruby identifiers into the cryptic pool of sludge. Let’s see the # sign is a comment if it is the first character in a new line or after a statement (which of course doesn’t end with a semi-colon or other signifier), otherwise it is an embedded variable. And what is the deal with variable scope declarations? The @ sign is used to declare instance variables, using @@ signs declares a class variable, and of course the $ sign declares a variable that is global in scope. Hmmm…I am thinking the access modifiers ‘public’ and ‘private’ in C# make a lot more sense, but of course I have to type all those letters instead of just pressing the shift key followed by one or two characters.

Useless keywords

For all you Ruby experts, can someone please explain the purpose of the unless keyword? It seems to me to be another bit of useless confusion. Let’s see if I have this right in Ruby:

if x < 5 then

statement

end

and

unless x < 5 then

statement

end

are identical! What’s the point?

Syntax?

And what’s the deal with ‘downcase’ and ‘upcase’ instead of ‘lowercase’ or ‘uppercase?’ Is downcase really intuitive to anyone other than Ruby zealots?

I could probably go on, but frankly I am convinced that Ruby is not the way.

So, that’s a day in my life that I will never get back. But, all was not lost because in-between chapters of the Ruby book I got to watch the paint dry on my house.

Originally Published Wednesday, August 02, 2006

My respected colleague Keith Stobie recently posted to his blog my response an internal query regarding “unrealistically high” goals for the percentage of tests that are automated. (Since, I have been remiss in my public rants I will reuse Keith’s post. Thanks Keith.) Of course, the unrealistically high percentage is probably near the 100% mark. There has been a lot of talk recently about 100% automation goals especially in the Agile world. So, for all those out there that think 100% test automation is a “reasonable” or even achievable goal I have a suggestion. Learn to just say NO to drugs!

There are some areas of testing where I do see the need and value of approximately 100% of the tests being automated. BVT/BAT is a prime example. When I was the Setup Test Lead on Windows 95 I was responsible for developing the BVT/BAT suite. The suite contained approximately 1000 unique functional tests and ran simultaneously on 4 different language versions within the span of 30 minutes once a week for about 1½ years upon delivery of each new build. But it was only maintenance free for the last 6 months of the product cycle. Still, that’s a pretty good return on investment considering Ed Kit, and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test. Other areas which lend themselves to high percentages of automated tests include performance testing, stress testing, regression testing, etc. Areas of testing in which I need to establish baselines and run consistent tests in order to compare repeatedly predictable results against those baselines towards some ultimate predefined goal are prime candidates for high levels of automation. Scripted test automation does a very poor job of exposing “new” defects post test design, and GUI automation requires enormous maintenance especially if it is created too early in the product cycle when the U/I is still in transition. The bottom line is we need to be smart about what and when to automate, because good reusable test automation is damn expensive.

For my unabridged response to managers or others who throw around the 100% automation mantra see my comments in Keith’s blog. For more information about making smart decisions of what and when to automate I highly recommend the book by Daniel J.Mosley and Bruce A. Posey entitled Just Enough Software Test Automation.

Originally Published Thursday, June 08, 2006

In the past few years there has been a push to use Ruby as a programming language for test automation. Ruby as a programming language has some benefits compared to other scripting languages. However, for test automation or any other serious application development the disadvantages of Ruby certainly outweigh any of the recent sensational hype promoting the language. I may be a bit biased, but I honestly can’t fathom why a tester wanting to learn a programming language would spend time learning Ruby, especially with the ease of use, availability of resources, and broad adoption of C#.

1. Ruby lacks informational resources. A search on Barnes & Noble, or Amazon reveals about 50 or so books on Ruby programming. But, that is barely a drop in the bucket compared to more than 400 books written about C#. These numbers certainly don’t inspire a lot of confidence in Ruby as a broadly accepted programming language in the industry at large. Web searches for available online resources also reflect this tremendous disparity. Sure, the Ruby zealots support the few websites and respond to requests for assistance. But, there are a greater number of C# forums with greater numbers of registered members who frequently participate and provide solutions to questions. Additionally, you won’t find too many universities or community colleges offering courses in Ruby programming. If Ruby is so good then why are there such limited resources? The answer is because there simply isn’t the business demand, or other compelling reasons for adoption.

2. Ruby is not a high demand skill among employers. Take a look at any of the technical job sites such as Dice, Monster, etc. and you will not find a plethora of jobs asking for Ruby programming skills. For example, of the approximately 16,500 software testing jobs on Dice only 65 contain the keyword Ruby as compared to 1,668 job listings containing the keyword C#. That means there are 25 times more employers desiring C# as compared to Ruby. IT Jobs Watch in the UK has an interesting site with lots of statistics relative to software testing positions in the UK. Looking at software testing jobs C# is listed in the top 5 desired programming languages. (Ruby doesn’t even make the top 20 list anywhere on this site.) The job trends on Indeed provides a visual perspective comparing jobs with C#, Ruby, Perl, and VB.NET keywords.

3. Ruby has performance problems. Scripting languages are notoriously slower than compiled languages, but it seems that Ruby is often slower (CPU time) and requires a larger memory footprint as compared to other scripting languages. The Computer Language Shootout site provides some very interesting benchmark results for various languages. One benefit of automation is the decreased time to execute a specific test. Now, you may be thinking that it only takes a few more seconds to execute a test in Ruby as compared to Java or C#. So, let’s arbitrarily say that each automated script in Ruby takes 5 seconds longer to execute as compared to the same automated test in C#. That may not seem like a big deal. But, instead of running one or two automated tests you want to run a test library of 200,000 tests. That’s about 12 hours of additional time needed to run the test automation written in Ruby. Now, I am sure Ruby advocates will discuss the reduced cost of development time, but time to develop an automated test is a one time expense (this does not include sustained maintenance which is a variable cost compounded over time for all automation libraries). Depending on your product’s shelf-life, you may need to rerun your test automation suites for the next 7 to 10 years for sustained engineering and product maintenance.

4. Ruby has not been widely adopted in application development. So, you may ask why this is a weakness for testers writing test automation? The simple fact is that if the development team is programming in say C/C++ or Java, and the test automation is in Ruby you probably won’t get a lot of support from the development team to help reviewing or debugging test automation. Also, it is very likely the developers may not want to install the Ruby interpreter on their Windows machine to use test automation to reproduce a defect, and instead ask for the manual steps. The test libraries the development team creates will require porting to Ruby which increases the cost and effort. Since many developers are familiar with at least the basic syntax of C/C++ and Java it is easier for them to pick up C# syntax and understand automated test code.

5. Ruby is just as cryptic as any other programming language. All programming languages use unique syntax, and users must learn the language’s syntax to code effectively. Now, I am no expert in Ruby but let’s compare a Ruby script to launch Windows calc.exe as compared to a C# program.

————————————————————–

# Launch calc.exe in Ruby

require “win32/guitest”

include win32::GuiTest

run_cmd “calc.exe”

—————————————————————–

// Launch calc.exe in C#

using System;

using System.Diagnostics;

namespace MyNameSpace

{

class MyClass

{

static void main ()

{

Process.Start(“calc.exe”);

}

}

}

—————————————————————–

Obviously there are more lines of code in the C# program as compared to the Ruby script. But, considering the fact the template in Visual Studio auto-generates the framework for a console application (the primary method of writing an automated test case) the only thing I need to add to the .cs file are the ‘using System.Diagnostics’ namespace declaration, and the ‘Process.Start(“calc.exe”);’ statement. Additionally, the IntelliSense feature of the Visual Studio IDE references language elements, and even inserts the selected element into the code. Also, perhaps it is a matter of personal taste, but Process.Start() seems a lot more ‘readable’ than run_cmd.

Ruby activists boast how quickly they can teach Ruby scripting to non-programming testers. I have been teaching C# to testers for more than 3 years. I have been very successful at teaching testers with no previous programming skills to write automated GUI tests in C# that will launch a Windows application, manipulate the application, generate and send data, verify and log results, and clean up the test environment within a day.

There may be some interesting features of Ruby, but don’t get sucked in by all the fanatical propaganda. Ruby has been around for more than 10 years and hasn’t replaced any language or garnered a significant following! The simple fact is that Ruby simply doesn’t offer anything revolutionary, and thus hasn’t compelled the development community to rush to adopt or support it. All programming languages have strengths and weaknesses depending on the application. But, for test automation Ruby is not the best choice as a programming language.

In my (biased) opinion, I think C# is a much better choice, and in a later post I will outline the benefits of C# as a programming language for test automation.

Published Thursday, June 08, 2006 12:55 AM by I.M.Testy

Originally posted Friday, May 19, 2006

We often limit our test automation projects to internal use only. This restricted use of test automation has contributed the typical sub-standard quality of automated test code. If you think production code is poorly documented you haven’t reviewed very much automated test code. But, since the test code is for internal use only we really don’t need to put as much effort into it as we do production code. Or do we?

Unfortunately, the restricted view of test automation as an internal only tool in the testing process has perhaps limited its potential and its overall value. But, as testing matures into a profession testers are going to have a greater impact not only during the product development lifecycle, but testers will also extend their scope of influence directly to the customers. You may ask how or why test automation be used by customers? Read the following scenarios, and think about the possibilities!

A company contracts for a customized software solution based on a specific set of requirements. At the completion of the project the product is handed off to the company who then conducts several suites of tests such as user acceptance tests and integration tests to test compatibility with the existing environment. Additional ‘acceptance’ testing by the company costs them time and uses their valuable IT resources to re-verify the product’s functional capabilities and other quality attributes. Now imagine the software company also provides well-documented automated test suites along with the product that demonstrates the product does what it is supposed to do, and does it in the customer’s environment. The customers ‘acceptance’ testing time is greatly decreased, their overall cost of adoption is decreased, and (just possibly) customer satisfaction is increased.

Another company purchases several Microsoft products such as Windows Server, Exchange Server, and SQL. But, it is often not as simple as installing these platforms and setting the preferred configuration. So, the company either staffs an IT department that rivals an IT team assembled by a Fortune 500 company, or hires a company similar to Avanade to put the pieces together and make them work with the desired configuration settings. Of course, either solution is going to cost the company a lot of money, time, and potentially use internal resources to get things setup correctly. Now suppose Microsoft creates an enterprise Software Test Kit (STK) that contains thousands of automated tests. The IT team identifies the specific configuration settings for its platforms, identifies applications in the corporate environment, and selects other requirements for adoption. Based on the IT department’s criteria the STK will automatically select a suite of tests the company’s IT department can execute to verify their platform configuration and other acceptance tests. Automated test suites in the STK can even be used for non-functional testing such as performance and stress testing by the company based on their configuration settings.

Are these potential uses of test automation by customers? I believe these are only some of the possibilities that exist. However, to realize this potential our test automation must evolve. The quality of our automated tests must rival that of our production code. In fact, test code must be better than production code because there is no room for errors or false positives. If customers use our automated tests to reduce their overall costs of adoption the tests must be bullet-proof or customer confidence decreases and customer satisfaction plummets.

So, what must change in order to increase the overall significance and potential usefulness of test automation beyond its current value? To begin with, automated test cases must be considered a product with a meaningful shelf-life instead of disposable artifact or something we simply pass on to the maintenance or sustained engineering team. The automated tests must be well-planned and well-documented (preferably with XML). The testers who develop the tests must understand the product and be able to develop the automated tests with a modern programming language. (Sorry, but scripting languages, keyword or data-driven approaches, and record/playback are simply too limited in their ability to scope to test automation of this magnitude.) Testers will need to use static analysis tools and perform code reviews to verify the quality of the test code. We will also need an framework that can select and run the appropriate set of tests to satisfy the customer’s needs, but is extensible enough for the customer to easily create and import additional automated tests if necessary.

Perhaps this is all just wishful thinking, or perhaps it is a vision of things to come in the professional world of software testing.