I.M. Testy

Originally Published Tuesday, May 13, 2008

Now, many of you probably know that I am a big fan of computer generated random test data that is a represents a reasonable sample data set from the total population of possible test data. (I refer to this a probabilistic stochastic test data.) So, why would I argue against preserving randomly generated test data?

I just returned from STAREast, where for the second time in a month I heard someone suggest storing randomly generated test data in a file. Many people will site the inability to recreate random test data as a drawback to using randomly generated test data in a test. So, the reason these people suggested storing the random data in a file is so they can easily repeat a test with the same data should some randomly generated test data expose an anomaly. I absolutely concur that if we generate random test data, and that test data exposes a problem we need a way to recreate the data. But, isn’t there a better way than to save random test data in a file?

Saving randomly generated test data to a file creates a test artifact. Depending on how much randomly generated data is generated, this file could become quite large. Also, saving data to a file impacts the performance of an automated test and certainly slows down manual execution of tests. Then consider the number of tests that generate random test data are executed numerous times throughout the lifecycle, and it doesn’t take long until we have countless test artifacts simply storing more static test data that quickly loses its value (especially if no problems were detected). Of course, we can easily delete the files after the test if no anomaly was detected, but I suspect that most testers will delete those files upon the completion of the test if no problems were detected.

So, the question is how can we reproduce computer generated probabilistic stochastic test data if we don’t save that randomly generated data to a file?

Planting Seeds

In computing, a seed is simply an integer value that is used by a random generator as the starting value. If we pass a seed value as an argument to a given random generator then we will consistently get the same random value each and every time. Essentially, a seed allows us to replicate computer generated probabilistic stochastic test data anytime as long as we use the same seed and the same random generator algorithm. So, instead of saving each and every piece of randomly generated test data used in any given test, we can simply log the seed value used by that test in the test results log file.

But, if we use the same seed all the time, then we are simply generating the same data over and over again. And, manually inputting a seed for each test that generates probabilistic stochastic test data is not an ideal situation, especially for automated tests. So, to solve that problem we can randomly generate a seed value that is then passed to the random generator algorithm!  Again, logging the randomly generated seed allows us to accurately reproduce the probabilistic stochastic test data at any later time.

The example below illustrates a simple method in C# that will either generate a random seed or return a user specified seed value.

   1: public static int GetSeedValue(string seedValue)

   2: {

   3:     // check if user specified seed value is passed as an arguement to 

   4:     // the seedValue parameter

   5:     if (seedValue == string.Empty)

   6:     {

   7:         // Create a new random object

   8:         Random randomObject = new Random();

   9:         // Generate a random integer value between 0 and 2,147,483,647

  10:         return randomObject.Next();

  11:     }

  12:     else

  13:     {

  14:         // convert the seedValue to an integer value

  15:         // NOTE: This example method does not include exception handling

  16:         return int.Parse(seedValue);

  17:     }

  18: }

The following example illustrates how to use this method to get a random seed value to generate random strings and numbers that increase the breadth of test data coverage in each subsequent iteration of a test.

   1: static void Main(string[] args)

   2: {

   3:     // These variables declare the range of characters used for the

   4:     // string test data. In this case the strings are composed of upper

   5:     // case ASCII characters 'A' through 'Z'

   6:     char minChar = '\u0041';

   7:     char maxChar = '\u005A';

   8:     

   9:     // This reads the user specified seed value from the console window

  10:     // If no seed value is specified an empty string is passed to the 

  11:     // GetRandomSeed method which will cause it to generate a random 

  12:     // seed value.

  13:     string mySeed = Console.ReadLine();

  14:     

  15:     // Declare a seed variable and initialize it to either the user

  16:     // specified seed or to a computer generated random seed value

  17:     int seed = GetSeedValue(mySeed);

  18:  

  19:     // The seed value should be permenently recorded in the logged

  20:     // results for this test

  21:     Console.WriteLine("The seed value for this test is {0}\n", seed);

  22:  

  23:     // Create a new random object based on the seed

  24:     Random randomGeneratorObject = new Random(seed);

  25:  

  26:     // Generate 10 random strings

  27:     for (int count = 0; count < 10; count++)

  28:     {

  29:         // Declare and initialize a string variable for our test data

  30:         string testString = string.Empty;

  31:         // Generate random length strings between 1 and 10 characters

  32:         for (int length = 0; length < randomGeneratorObject.Next(1, 11); length++)

  33:         {

  34:             // Generate a random character within the defined range and

  35:             // concatenate it to the testString variable until the 

  36:             // random string length has been reached

  37:             testString += Convert.ToChar(randomGeneratorObject.Next(

  38:                 minChar, maxChar + 1)).ToString();

  39:         }

  40:  

  41:         // Write the test string to the console window

  42:         Console.WriteLine("Test String {0}: {1}", count + 1, testString);

  43:     }

  44:  

  45:     Console.WriteLine("\nRandom numbers");

  46:     // Generate 5 random numbers

  47:     for (int numberCount = 0; numberCount < 5; numberCount++)

  48:     {    

  49:         Console.WriteLine("{0} ", randomGeneratorObject.Next());

  50:     }

  51: }

Calling the Main method and passing an integer value between 0 and 2,147,483,647 will generate 10 random length strings composed of random upper case characters between ‘A’ and ‘Z’ and 5 random numbers. If no user specified seed is passed to the Main method then the code will call the GetGenerateSeed method and generate a random seed value for use in the test. Of course, passing the same integer value will produce the same strings and numbers each and every time.

Using probabilistic stochastic test data is valuable because it efficiently increases the breadth of data coverage, and significantly augments ‘typical’ static test data, user-generated test data, or static test data derived from historical failure indicators. But, instead of storing randomly generated test data in a file, it is a best practice to simply record the seed value of each test. With a seed value we can easily recreate the computer generated random test data should any of the random data used in a test exposes an anomaly.

Originally Published Friday, March 28, 2008

It seems that many test automation efforts around the industry tend to focus on GUI automation, or automating functional tests primarily by manipulating GUI objects. In general, GUI automation tends to be a very expensive approach to test automation, and the automation efforts often end in failure or achieve less than satisfactory results.

The majority of automated tests at Microsoft are below the GUI; however, automated tests that manipulate GUI objects are quite useful within specific contexts. Unfortunately, many testers attempt to develop automated GUI tests way too early in the project cycle while the user interface design is still unstable. I guess the assumption is that constantly maintaining automated tests is somehow better then executing manual tests. But, in general, when the UI is in flux it is usually counter-productive and a loss of return to automate GUI level tests too soon in the development lifecycle.

Dan Mosley and Bruce Posey (Just Enough Software Test Automation) suggest that on average an automated test must run approximately 17 times in order to break even. But, this doesn’t imply that we break even if we simply run the same test on a daily build for the next 17 days. The presumption underlying the ROI after 17 runs of a test is that something changed in the build that is covered by that particular test, and so by executing that (regression) test we are providing important information (changes in the build did not destabilize that area) to the team. Steve Rowe also has an excellent blog post on Too Much Test Automation that you should consider reading, and Dustin Andrews also has an excellent blog post on getting great results from test automation with an 8-minute video.

The cost of test automation is never easy to figure out and is certainly not a straight forward comparison of automated time versus manual time. Comparing automation time versus manual execution time is an overly simplistic measure that rarely takes into consideration the design and the development time of the initial tests (or the overall costs of building and enhancing test frameworks or drivers), or the time required to identify false negatives, troubleshoot the cause, fix the problem, verify the fix (manually), and then check the source back in for the next run. (And, there are other tangible costs, and intangible costs such as loss of confidence that must also be considered in any cost model.)

So, in general any automation that requires constant maintenance is usually not cost effective, and the more our test automation throws false negatives the less our management team views automation as a viable resource to provide us with valuable and reliable information for improved risk analysis. Ultimately the decision comes down to how much perceived and measurable value a test has in providing important information for improved risk assessment and quality measurement.

Just because we can automate something, doesn’t always mean we should!

Originally Published Sunday, January 06, 2008

It has been awhile since I have written about test automation, so I thought I would start the new year off with a post about test automation. More specifically, I wanted to start talking about coding guidelines. Just as many development teams have adopted coding standards and guidelines the test team should also adopt a set of guidelines and standards for all testers to follow when developing test automation. Some teams have coding standards and guidelines, but some teams just sort of wing it. Of course, those teams that just sort of wing it and allow developers the freedom to develop their own style of coding soon realize the cost of reviews and maintenance increase. Sloppy coding also increases the probability of subtle defects that may go unnoticed. Coding standards or guidelines are not (or should not) restrict a test developers creativity; but consistent coding standards and guidelines improve the efficiency and effectiveness of code reviews and sustained maintenance (which is often performed by someone other than the person who wrote the original code).

This week I want to start by talking about problems with various coding styles in source files such as indentation and general layout issues that make code more difficult to read. Readability is important for efficient code reviews, and when the code is later maintained by someone other than the person who originally wrote the code to begin with. There are many coding standards and guidelines that a team can and should adopt and I plan to discuss various guidelines over time, but for this week I thought I would introduce some commonly accepted guidelines that improve the readability of code by adopting a consistent layout.

Indentation

• Set the indentation or tab size to 4 characters in width
  I have seen people set the tab size to as small as 2 and in some rare cases to 3 for some bizarre reason. But, 4 characters in width seems to be the standard tab size, and an additional 2 characters per line is really not going to make that much of a difference, and I think it improves readability of the code by clearly indenting blocks. (I really don’t want to get into a long debate about 2 or 4 spaces; the point is that the team should use a consistent indentation size.) 
• Use spaces instead of tabs
  This is often controversial, but spaces are recommended over tabs for a variety of reasons. One reason why one should use spaces instead of tabs is for consistent layout and readability in a variety of editors. Many text editors interpret tab stops differently, and occasionally the editor a reviewer uses to look at source code may be different than the source code editor used by the developer causing the line formatting to change radically. For example, when I need to take a quick look at a .cs file I will often open that file using Notepad instead of launching Visual Studio. Sure, I don’t get the pretty colors and such, but it’s a lot quicker! (Again, I don’t really want to get into a debate about tabs vs. spaces. That subject has been beaten to death on the Internet and there are pundits on both sides of the fence. If file size is a really critical factor than tabs will reduce the overall physical file size in bytes. If readability in different editors is important, or if restricting the use of the 0×09 character code point control code in your source code is important the use of spaces is preferential. Again the important point would be consistency within the team.)

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Tools -> Options… menu item
2. Select Text Editor -> All Languages -> Tabs item in the Options tree control
3. Set the indentation to Smart.
4. Set the tab size and indent size to 4
5. Select Insert spaces (or Keep tabs; whichever standard your team adopts)

Line length

• Do not extend lines beyond 80 characters in length
  Occasionally, developers will simply write a statement that extends well beyond the viewing area of the editor window forcing reviewers to scroll. Or worse yet, when opening a file in Notepad or some other simple text editor the line wraps (if word wrapping is selected) around causing the layout to become quite confusing to the eyes. Word wrapping also becomes a problem when printing source files for review. Either way extending lines beyond 80 columns looks sloppy and makes readability more difficult.

Notice lines 76 and 80 extend well beyond 90 characters. If this source file was printed lines 76, 80, and 83 would wrap to the next lines on the page starting from the left margin impacting the readability of the file.

The column (or character count) per line is usually displayed in the status bar of an IDE, but for Visual Studio (including Visual Studio Express editions) a simple registry setting provides a visual guide in the IDE. The visual guide in the IDE means that I don’t have to look at the status bar to see where my column count is, or go back later and reformat lines after I am finished coding a method or class.

Notice the red dotted line on the right hand side of the code window set at a column width of 80 characters.

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Start -> Run menu item
2. Type regedit in the Open control on the Run dialog and press OK
3. In the Key tree control select HKEY_CURRENT_USER | Software | Microsoft | Visual Studio | [version] | Text Editor
4. Right click in the value pane and select New -> String Value from the pop up menu
5. Rename the new value Guides
6. Right click on the Guides value item and select the Modify… menu item
7. Type RGB(255, 0, 0) 80 in the Value data textbox on the Edit String dialog

I also prefer to display line numbers. This makes it a bit more convenient when doing things such as Finding All References to variables.

In subsequent posts I will discuss additional coding guidelines, but these are just a few ideas to get the discussion started. If you have additional ideas on layout then please let us know!

Originally Published Thursday, August 30, 2007

I never really understood why so many people external to Microsoft seem to be against the Microsoft strategy to increase the amount of automation we rely on to test our products. Test automation has become sine qua non at Microsoft for many valuable reasons.  Although there are some uninformed managers who propose the nonsensical notion of 100% automation, Microsoft as a company does not have some magical goal of 100% automation (whatever that means).

I have come to the conclusion these (mostly) external naysayers ("people who tend to be negative, who have difficulty thinking outside of the box, who go around, consciously or unconsciously, squelching dreams, ideas and visions") of automation simply do not, cannot, or perhaps refuse to comprehend the complexity of the systems or the problem space we deal with at Microsoft or how we design and develop our automated tests (in general, we do not write scripted tests). Or perhaps the anti-automation folks are SOS (stuck on stupid) and believe that any test (including automation) that does not find a bug is not a reasonable test or that executing that test does not provide some perception of reasonable value to an organization.

So, for those of you who are not afraid of being replaced by automation (if you are afraid a machine will take your job, then it probably will), and for those of you who have an open mind and know that real testing is not a simple comparison of scripted vs. exploratory, and that successful testing requires a myriad of techniques, methods, and approaches, then read on and I will discuss some of the reasons why we automate at Microsoft.

• Increasingly complex test matrices. (Whether you like it or not) the Windows operating system proliferates throughout the world. But, even the different Windows operating systems in common use throughout the world are staggering. Although we no longer support Windows 9x (including ME) there is still support for Windows NT 4.0, Windows 2000, Windows Xp, Windows 2003 Server, and Windows Vista. Then add in various combinations of service packs and hot fixes released for each of these, and test environments composed of upgrade scenarios (Windows Xp upgraded to Windows Vista) approximately 30 language versions, 16- bit, 32-bit, and 64 -bit and the number of operating system platforms alone becomes mathematically staggering. But, instead of writing several automated tests for each combination of environment parameters our testers ideally design a single test that detects the OS platform and environments, and develop the test in a way it can decide how to achieve its objective based on platform profiling and other design techniques. (For example, a test for defrag has to take into account the German version of the OS does not contain the defrag utility (long story), or a security test on the French version considers the French version does not contain 128-bit encryption). One automated test instead of an army of button pushers doing the same thing on 5+ operating environments and 30 languages just makes sense!
• Sustained maintenance. Microsoft supports many of its products for 7 to 10 years. So, the more test automation we can hand off to our sustained engineering teams, the less cost the company has to bear in the long term. Do these regression tests find a lot of defects. Perhaps not, but they do provide confidence that changes to the code base introduced by hotfixes and service packs do not adversely impact previous functionality. They also eliminate the burden of having to maintain an army of testers for the entire shelf life of the product.
• Increase breadth of coverage. Automation is distributed across the network, it is not ran on one or two desktops or on a few lab machines. Many product teams have extensive test harnesses that are capable of scheduling tests, configuring environments, then distributing tests to hundreds of machines in a lab, or even on idle machines on the network. This way we not only run test automation on pre-defined environments, but also on work machines and other systems. Test automation is running 24 hours a day and collecting and providing valuable information.
• We don’t rely on automation to find bugs, we use test automation to provide information. Sometimes automated tests can expose unexpected and/or unpredictable behavior, but we know that automated tests do not find a great deal of defects after the design and development phase of that automated test. However, many of our products have daily builds and instead of rerunning a bunch of tests to ensure changes from a previous build have not affected previously tested functionality an automated test is more efficient. The minefield analogy sometimes applied as an argument against regression testing only really makes sense if the test is being executed on a static code base. But, if you test in a real world where you might get daily or even weekly builds of complex systems then you probably realize that executing a set of tests after someone changes the minefield, has a probability that following the same path may expose a defect, and if it doesn’t then it still provides valuable information.
• Increased job satisfaction! This is a big side benefit. I know some people are afraid of automation, and some people may lack the skill to design and develop effective automation, but professional testers realize it is a huge challenge to design and develop effective test automation for complex systems. To quote one tester, "the hardest code I ever wrote was the code needed to test other code."

There are many more justifiable reasons why increased automation simply makes sense at Microsoft, but this should give you some idea of why without automation our jobs would be much more difficult than they already are.

Originally Published Wednesday, August 01, 2007

I am fascinated with the advances computing, and have always approached computing from the perspective of what can this tool do for me to make my life easier. As a professional tester I have a lot more work to do then I can reasonably accomplish in the limited timeframe allotted for most projects. So, well-designed test automation is a great tool that frees up some of my cycles performing mundane tasks that need to be accomplished.

A few months ago I had an email exchange and an industry consultant regarding test automation and emotions that I blogged about and he later talked about at a Star conference. In that post I also tried to illustrate a simple technique called polling for simulating irritation (or frustration) of a task that is taking too long to complete via an automated test.

The consultant I had the email exchange with wrote, "I would want my automation to feel frustration, to recognize it, and to act on those feelings in some way that provides valuable information to the product.  But until we’ve got not only artificial intelligence, but also artificial emotional intelligence, that ain’t gonna happen."

When I design an automated test I often think of the various ways to achieve exactly what it is I am trying to prove or disprove with the test. Then I think of the things that can go wrong (such as race conditions, errant message boxes, tasks taking too long to complete, making simple decisions based on Boolean states, etc.) and design the test in such a way that can logically deal with those situations. So, as I thought about our conversation and automated test design I asked myself, can automation do more than simple mundane tasks? Can automation make decisions or perform tasks based on practical reasoning or simulated emotions?

It seems that some researchers in the Netherlands are unlocking doors with artificial intelligence that may eventually lead to advances in smarter test automation design. Researchers at Utrecht University are hard at work on an emotional robot (a cat none the less) that simulates 22 emotions including "anger, hope, gratification, fear, and joy," used in complex decision making processes. Marvin Minsky stated "…we all have these things called emotions, and people think of them as mysterious additions to rational thinking. My view is that an emotional state is a different way of thinking."

I agree with the researchers, and I "don’t believe that computers can have emotions," and also mostly agree with their statement "that emotions have a certain function in human practical reasoning." (I say mostly because I do know that some emotions express by some people are completely irrational and result in impractical reasoning.) Perhaps AI in test automation this is still a long way off, but I am always looking for ways to improve and become more effective and more efficient. I am always learning and looking for ideas to improve myself and my skills.  So, based on this research I now ask myself, are there cost effective emotional logic patterns to simulate rational reasoning, and I can or should I employ that in the design of some of my automated tests to make them more robust?

Just a thought. Isn’t technology great!

Originally Published Wednesday, July 25, 2007

When getting close to shipping a product and the team discovers a critical defect that must be fixed one of the first questions asked by the management team is, “how long will it take?” From a testing perspective they generally want to know 2 things; what has to be done to reduce perceived risk (or increase their confidence the fix hasn’t introduce new defects), and then they ask how long will it take. Unfortunately, many teams take a swag at the time element and proceed to beat on the product until the magically agreed upon time expires.

I have seen a lot of test case templates, and one field that I often see omitted from the template (or ignored by the test designer) for a test case is duration, or a reasonable approximation of how long it should take an experienced tester to complete a particular test (whether it is a discrete, functional test or a complex user or business scenario). Adding an estimated time for completion to a test then allows us to better approximate (within an hour or so) how long it will take to execute a particular test suite or a portion of that test suite.

For automated tests written in C#, getting the time for a test to complete is now simple with the Stopwatch class introduced in the .NET Framework 3.0. The Stopwatch class is exactly what the name implies; it provides methods and properties to accurately measure the elapsed time between starting the stopwatch instance and stopping the stopwatch instance. The following code snippet provides a simple example of how the Stopwatch instance might be used in a functional or behavioral automated test.

   1: using System;

   2: using System.Diagnostics;

   3: 

   4: namespace TestingMentor.Examples

   5: {

   6:     class StopwatchExample

   7:     {

   8:         static void Main(string[] args)

   9:         {

  10:             // initialize a new stop watch instance

  11:             Stopwatch myTotalTestTime = new Stopwatch();

  12:             // start the stop watch

  13:             myTotalTestTime.Start();

  14:

  15:             // test code goes - simulated by sleep

  16:             System.Threading.Thread.Sleep(3500);

  17: 

  18:             // stop the stop watch

  19:             myTotalTestTime.Stop();

  20: 

  21:             // log the elapsed time to the log file 

  22:             // (simulated by writing to the console window in this example)

  23:             Console.WriteLine("Total test time: " +

  24:                 GetElapsedTestTime(myTotalTestTime.Elapsed));

  25:         }

  26: 

  27:         private static string GetElapsedTestTime(TimeSpan timeSpanObject)

  28:         {

  29:             string time = String.Format("{0:00}:{1:00}:{2:00}:{3:00}",

  30:                 timeSpanObject.Hours, timeSpanObject.Minutes,

  31:                 timeSpanObject.Seconds, timeSpanObject.Milliseconds / 10);

  32:             return time;

  33:         }

  34:     }

  35: }

It is also possible to create several instances, or start and stop several stopwatches in a single test to measure not only the duration of the complete test, but how long it takes to complete a particular task within an automated test. This is extremely valuable for performance testing (a test used to determine the time required to perform and complete a particular task), or stress testing to measure mean time to failure (MTTF) and mean time between failures (MTBF).

Adding a stopwatch to the tests in your automated test suite will better enable you to accurately determine the duration or amount of time required not only to execute a complete test suite such as the build verification test suite or an automated regression test suite, but it also allows us to better calculate the time required to execute a particular subset of tests in a test suite (such as all priority 1 tests, or tests in a particular functional area).

Originally Published Thursday, June 21, 2007

Anyone who has listened to me talk about automated test design knows that I loathe hard-coded strings in test automation. Yes, I know that static test data is important, but it is almost never a good practice to hard-code strings or other test data in the body of your test code (or any code for that matter). 

But, as I started porting some of my demos to run on Windows Vista I realized that I violated my own axiom by hard-coding local paths for test artifacts (test files) in my own test code. Granted this code was for demonstration and training, but that is not an excuse for slovenly written code. Besides, as a general principle we should develop all code following which ever best practices you adhere to because we never really know who is going to critique that code or use it as an example elsewhere. 

Unfortunately, I am not alone in committing this mistake and I occasionally encounter hard coded paths in various automated test projects. Perhaps this occurs because it takes less effort (and lines of code) to hard code a path rather than get the path using a .NET method or Win32 API, or perhaps because we might assume the code will never run on a different machine (both of which are probably bad assumptions). No matter what the reason hard-coded paths are generally not a good practice.

In some automated tests I generate dynamic test data during runtime and create a file to temporarily store that data for use in the test, or save other user files with information that doesn’t necessarily have to persist after the completion of that automated test. (In generating dynamic test data I only have to log or record the seed value used to generate the data.) In one demonstration I created test artifacts (test files) and saved them to the root of the C: drive. (Yes, I know the old PC-98 architecture used in Japan might not have a C: drive because it had a floating drive architecture, but on the PC/AT architecture running the Windows OS environment the default drive is almost always C:.)  Writing to the root directory of the C:\ drive was easy and relatively safe in strictly controlled environments, but it is certainly not the best thing to do (and generally bad from a security perspective). But, fast forward to Windows Vista and my code breaks. Even though I have administrator privileges when I run the demo code and attempt to save a file to the root of C:\ on the Vista platform the system gives me an error message indicating I don’t have permission to save in that location. Unexpected dialog == unexpected behavior == code breaking. File not found by the test oracle in the expected location == false positive. Demonstration falling apart in front of an audience == embarrassment (because it really does no good to say "well, it worked on my other machine!").

Avoiding hard-coded paths for storing temporary test artifacts (e.g. disposable test files generated during the automated test) in automated test cases is easy with C# and the .NET framework. If we need the path to the desktop folder, or program files folder (or other special folders created by the Windows operating system) the GetFolderPath method in the System.Environment class gets the path to "special" system folders. The special folders include folders such as the system folder \windows\system32), the Program Files folder, the Desktop folder, the MyDocuments folder, etc. and are listed on MSDN under the Environment.SpecialFolder enumeration.)

For example, the following snippet returns a path to the current user’s Documents folder ("C:\Users\[logged in account name]\Documents") which is a good place to store temporary test artifacts.

            string desktopFolderPath = Environment.GetFolderPath(
                Environment.SpecialFolder.MyDocuments);

It would probably be better to use a StringBuilder object as illustrated in the following example because a string is an immutable object in C#, and we will probably want to append the path separator and eventually a filename for the test artifact.

            StringBuilder desktopFolderPath = new StringBuilder(
                Environment.GetFolderPath(
                Environment.SpecialFolder.MyDocuments) +
                Path.DirectorySeparatorChar);

This returns a StringBuilder object of "c:\\users\\[logged in account name]\\desktop\\".

Another useful location to temporarily store disposable test artifacts might be the current user’s temporary directory ("C:\\Users\\[logged in account name]\\AppData\\Local\\Temp\\") as illustrated in the following example:

            StringBuilder tempFolderPath = new StringBuilder(
                Path.GetTempPath());

Using the Environment.SpecialFolder enumeration or the .GetTempPath method are easy ways to get the path to a folder to store disposable test artifacts on the local machine without hard-coding a path in the automation code. It is likely this will be reused, so it is a good idea to wrap some methods for inclusion in your test automation framework or library. Also, don’t forget to delete or move any test artifacts from the test machine after the automated test completes. It is generally a best practice to restore a system to its’ pre-test state after your automated test executes.

Originally Published Friday, June 08, 2007

In April I hosted a session at the Software Testing and Performance conference entitled Why Test Automation Fails. I actually tried out a new delivery style for me similar to a town hall format where I attempted to elicit participation from the attendees. Unfortunately, it didn’t work out as well as I had expected, but at one point a consultant who attended the session offered their opinion expressing what they thought automation can’t do.

He stated automation can’t empathize, project, anticipate, recognize, judge, predict, project, evaluate, assess, become resigned, get frustrated, invent, model, resource, collaborate, decide, work around, strategize, charter, teach, learn, appreciate, question, refine, investigate, speculate, suggest, contextualize, explain, elaborate, reframe, refocus, troubleshoot, and THINK.

To this I exclaimed, “I am not sure I want my automation to get frustrated!” At the time I was thinking frustration is typically not a productive emotional trait because it usually manifests itself as “a deep chronic sense or state of insecurity and dissatisfaction arising from unresolved problems or unfulfilled needs” (Merriam-Webster Medical Dictionary). But, while flying home to Seattle that evening I pondered how an emotional condition such as ‘frustration’ might be applied to a tool such as test automation. (Because make no mistake about it; to me test automation is a tool similar to a screwdriver in a toolbox, or a global positioning system on a boat. These tools are designed to be very effective for their purpose. And when test automation is designed and used correctly it also serves a very valuable purpose to those who use it as a tool to increase their productivity.)

So, I sent an email to the person and asked him for the list and restated what I said at the conference.  I wrote, “Like I countered, I am not sure that I want my automation to get frustrated (and honestly I am not quite sure of the point you are trying to make there)…” He responded, “When a tester gets frustrated with some aspect of an application’s behavior (It’s taking too long to respond!  I have to click on this annoying confirmation dialog!  This sequence of steps doesn’t fit with my workflow!  This set of options confuses me!), I can infer that an end-user will too.  An infinitely patient human tester would ill-equipped to identify problems in the application that would annoy or irritate a normal human being.  Automation is infinitely patient (unless we’re programmed it not to be, via a timeout); if it has to wait, it will wait emotionlessly.  We cannot expect the same of our customers.  It’s not that I want automation to get frustrated; I don’t know how to make it do that.  But I do want to get information on how the product feels to humans–so I use human testers for that.”

Other than the fact this person misquoted me in his lightning talk at StarEast 2007, he also acknowledged in his email, “It’s not that I want automation to get frustrated” as well. It is usually not a productive use of one’s time to discuss issues with individual’s who argue against their own points. It’s sort of like someone saying "Elephant’s can’t fly. It’s not that I want elephants to fly, but they can’t…so there!" But, I see in his response that he describes other emotions such as ‘annoyance’ and ‘irritation’ (to rouse to impatience or anger; annoy – American Heritage Dictionary) to describe ‘frustration.’ Now annoyance and irritation are things I can relate to because I sometimes do get annoyed and irritated. So, let’s examine the emotional condition of irritation (because frustration generally implies a negative, and often counter-productive emotion) as described in the quoted email above and determine whether or not ‘irritation’ can be simulated by a computer program.

When I design an automated test I never let the test run “infinitely” because I know that a ‘hang’ or inordinate delay may cause an unexpected reaction in my test execution. (For example, if I am trying to ‘click a button’ on a dialog I must first make sure the dialog with the button instantiates and has focus and the button is not ‘grayed.’) A best practice for designing automated tests is to use polling with a finite timeout. Simply put polling is looping on a conditional statement until either the condition occurs, or the max poll count is reached. If the maximum poll count is reached before the condition occurs the automated test ‘decides’ the application is not in the ‘predicted’ and expected state and a timeout occurs implying that “It’s taking too long to respond” based some predetermined value to simulate irritation (or ‘frustration’).  If the automation becomes ‘irritated’ then I can cause the automation to ‘become resigned’ by designing the automation to log an unexpected error (rather than a failure), and collect information about the machine state and the automation state and log that information for closer examination by a tester.  (I don’t really think about this as programming an artificial emotional response; I think about it from the perspective of good automated test design.  But, in retrospect I guess it is simulating ‘irritation.’)

For example, if I think an application is taking too long to shut down rather than writing a bug report stating something like “The application takes too long to close” I can write a method similar to the following example that actually measures the amount of time it takes for the application under test (AUT) to close (and assuming the method returns true).

   1: public static bool CloseAutProcess(

   2:     Process autProcess, 

   3:     int maxPatience,

   4:     out int levelOfIrritation)

   5: {

   6:     levelOfIrritation = 0;

   7:     autProcess.CloseMainWindow();

   8:     if (!autProcess.HasExited)

   9:     {

  10:         while (!autProcess.HasExited && levelOfIrritation < maxPatience)

  11:         {

  12:             autProcess.WaitForExit(1);

  13:             levelOfIrritation ++;

  14:         }

  15:     }

  16:  

  17:     if (autProcess.HasExited)

  18:     {

  19:         return true;

  20:     }

  21:  

  22:     return false;

  23: }

In this example, the maxPatience variable sets the maximum amount of time (in milliseconds) I am going to wait for the AUT to shutdown before becoming ‘irritated.’ The levelOfIrritation variable accurately measures the time in milliseconds (give or take a few nanoseconds). Now, if the shut down time exceeds reasonable expectations (based on a comparative analysis of similar programs, or stated or implied requirements) I can write an objective bug report and allow management to decide if the ‘level of irritation’ needs to be adjusted based on solid factual information rather than some ambiguous personal patience threshold, or lack thereof. Because at the end of the day, my managers may value my opinion, and sometimes opinions are important deciding factors. But, as a software testing professional my role is to provide reliable, accurate, and detailed information to my teammates and other stakeholders.

(BTW…I can use event handlers to identify the frequency and number of ‘annoying confirmation dialogs” and deal with them. And, with regard to the other things described as ‘frustrating above (“This sequence of steps doesn’t fit with my workflow!  This set of options confuses me!” ) I am thinking these would most certainly be discovered while designing the automated tests (because test automation doesn’t design and develop itself).

So, I am still of the mindset that I don’t want my automation to develop a deep chronic sense or state of insecurity and dissatisfaction (def. – frustration) and I don’t want my automation to have ‘needs.’  And even though I can simulate some degree of artificial ‘emotional’ conditioning in test automation, I don’t want my automation to have emotions any more than I want a hammer or saw or other tool to develop human emotions. Test automation is simply a tool, and it is a tool developed and used by humans (who are emotional) who constantly utilize cognitive processes (including emotional responses) while solving difficult problems or working towards achieving specific goals. The power of a tool is limited only by the skills, ability, and knowledge of the persons designing, developing, and/or using that tool.

I agree that automation can’t THINK (although I am not overly convinced thinking is an emotion). That is why highly skilled people design and develop great test automation, and they don’t simply rely on record/playback or simple rote scripts for automated testing.

• working with data types
• using branches and loops
• basic exception handling

Originally Published Friday, August 25, 2006

I was stuck at home this weekend with house painters pondering how best to spend the day sitting at home, so I decided to actually play around with the Ruby scripting language a bit. I will admit my bias in my previous post about the Ruby scripting language was based on business justifications and personal preferences towards the C family of languages. But, because Ruby is still being pimped by some experts in the industry for test automation, I figured I should understand more about the mechanics of language other than it’s lack luster appeal by many professionals in the software industry.

I didn’t delve very deeply into Ruby. A few chapters of Hal Fulton’s book The Ruby Way and some additional research on the web confirmed my suspicion that Ruby for all purpose software test automation is simply not a good investment in your time given the other alternatives.

International Support Sucks!

If you are stuck working on small projects in English only then this probably won’t bother you too much. But, if you realize the majority of the world does not communicate in English then Ruby’s lack of international support simply sucks. Unless there is some hidden secret in the Ruby society the fact is Ruby can’t do Unicode. Yes, you can store UTF-8 encoded data in 8-bit encoded strings, but many Ruby String methods assume single byte encoding. I have to admit this completely blows my mind. UTF-8 is a popular encoding form of Unicode; however, it is only one of several Unicode transformation formats. So, forget about UTF-16, and definitely forget about testing Unicode surrogate pairs or conformance to the GB18030–2000 Chinese encoding standard. So, what does this mean for testers? String testing is basically limited to a very small subset of characters.

Multiple ways to initialize arrays (and just about anything else)

Hal Fulton lists the following 3 ways to initialize the array class method.

a = Array.[](1,2,3,4)

b = Array[1,2,3,4]

c = [1,2,3,4]

I don’t know why this would be a ‘cool’ thing to do, but I do know that it increases confusion, and potentially adds to the cost overhead of white box testing and code maintenance.

I guess it’s pretty cool that arrays in Ruby contain objects rather than data types, and an array in Ruby can contain integers, strings, etc. I am not quite sure why this is a positive because I can’t think of a situation where I would want to iterate through an array with disparate data types.

Cryptic codes

If Hungarian notation wasn’t bad enough, just stir in some Ruby identifiers into the cryptic pool of sludge. Let’s see the # sign is a comment if it is the first character in a new line or after a statement (which of course doesn’t end with a semi-colon or other signifier), otherwise it is an embedded variable. And what is the deal with variable scope declarations? The @ sign is used to declare instance variables, using @@ signs declares a class variable, and of course the $ sign declares a variable that is global in scope. Hmmm…I am thinking the access modifiers ‘public’ and ‘private’ in C# make a lot more sense, but of course I have to type all those letters instead of just pressing the shift key followed by one or two characters.

Useless keywords

For all you Ruby experts, can someone please explain the purpose of the unless keyword? It seems to me to be another bit of useless confusion. Let’s see if I have this right in Ruby:

if x < 5 then

statement

end

and

unless x < 5 then

statement

end

are identical! What’s the point?

Syntax?

And what’s the deal with ‘downcase’ and ‘upcase’ instead of ‘lowercase’ or ‘uppercase?’ Is downcase really intuitive to anyone other than Ruby zealots?

I could probably go on, but frankly I am convinced that Ruby is not the way.

So, that’s a day in my life that I will never get back. But, all was not lost because in-between chapters of the Ruby book I got to watch the paint dry on my house.