I.M. Testy

Originally Published Tuesday, September 09, 2008

Templates can be useful tools to help increase individual efficiency and team consistency. Yet, I am sometimes quite surprised that some organizations lack standard templates for such daily routines such as bug reporting, status reporting, and even test cases. There are some in the industry who shun templates because they claim they are too restrictive and the person using the template is ‘forced’ to do things one way and not encouraged to think beyond the scope of the template. Of course, any person who has matured to the point of being able to think rationally for themselves and has the cognitive ability to for logical reasoning (as opposed to those who base decisions on pure emotion and volitional impulse) realize this narrow-minded viewpoint is pure nonsense!

A template is simply a model or reference that provides a starting point. Frankly, if a person does not understand that a template simply provides a starting framework from which he must rely on his knowledge, skill, and experience to customize to fit the specific context, and how to use a template to help him accomplish a task more efficiently then I would recommend that person not be allowed beyond the reach of their mother’s apron strings because they are probably a danger to themselves and people around them and they are incapable of rational constructive thinking.

An example of where templates have added value for me is in the test development process. When developing test cases using Visual Studio I frequently find myself adding the references and using directives repeatedly to each new test case. For example, when developing a UI automated test case I need to include references for Windows.Forms, UIAutomationClient, UIAutomationType, UIAutomationClientsideProviders, and UIAutomationProvider. That’s a lot of clicking for each new test case.

Creating a template in Visual Studio is easy and straight forward. In this case, I started with a standard console application project. Next, I added the following references to the project:

• System.Windows.Forms
• UIAutomationClient
• UIAutomationClientsideProviders
• UIAutomationProvider
• UIAutomationTypes

Then I customize the template as illustrated in the code below by adding common namespace directives, including XML comments, and commonly used code snippets.

   1: using System;

   2: using System.IO;                       // For file I/O operations

   3: using System.Text;                     // For StringBuilder and encoder/decoder

   4: using System.Diagnostics;              // Process, stopwatch, trace/debug 

   5: using System.Globalization;            // For Unicode support

   6: using System.Windows.Forms;            // For sendkeys and clipboard

   7: using System.Windows.Automation;       // For UIAutomation

   8:  

   9: namespace CompanyName.ProjectName.FeatureArea

  10: {

  11:     class TestCase

  12:     {

  13:         /// <summary>TEST PURPOSE: (specific functional capability or end-2-end user scenario test is verifying)</summary>

  14:         /// <status>TEST STATUS: (Active, NotInMilestone, Obsolete)</status>

  15:         /// <category>TEST TYPE: (category or test suite bucket such as BVT/BAT, Regression, etc.)</category>

  16:         /// <priority>TEST PRIORITY: (importance of test in relation to critical functionality or requirements)</priority>

  17:         /// <time>TEST TIME: (approximate duration of test)</time>

  18:         /// <steps>TEST STEPS: (outline high level steps or procedures)</steps>

  19:         /// <expected>EXPECTED RESULTS: (specific verification criteria for oracle to determine pass, fail, or indeterminate)</expected>

  20:         /// <dependencies>CONFIGURATION: (required expected settings or dependencies)</dependencies>

  21:         /// <remarks>OTHER INFO:</remarks>

  22:  

  23:         [STAThread]

  24:         static void Main(string[] args)

  25:         {

  26:             // Time the duration of test in order to better predict time to

  27:             // run each test suite

  28:             Stopwatch TestTimer = new Stopwatch();

  29:             TestTimer.Start();

  30:  

  31:             // TODO: Design and develop the code to achieve the specific purpose

  32:             //       of the test. 

  33:  

  34:             // Don't forget to add comments to the code!

  35:  

  36:             // Don't forget to clean up or restore the test bed to pre-test

  37:             // conditions in cases other than end-to-end user scenarios.

  38:  

  39:             TestTimer.Stop();

  40:             // Don't forget to log or record test time

  41:         }

  42:     }

  43: }

I also like to include valuable information about the test case using XML comments to jog my memory or provide some background for anyone who views the code. For example it is a best practice to clearly define the specific purpose of each test so people don’t have to guess what the test is proving/disproving (or at least what it is supposed to be doing). (If someone doesn’t have a clearly defined purpose and goal of what the automated test is going to prove or disprove and they are simply writing automated test just to automate tests than that person is probably a mindless dolt!) The information that adds value to the organization may be subjective, so determine what information is important for the consumers of the completed template. Essentially, in this case, the comments captured in each test should provide value to the management and maintainability of the test and the testing process. If a template tries to collect too much information that is not being consumed or is not important to reduce long term costs of maintenance then that information should be suspect and is probably not adding value to the overall team. A big advantage of Visual Studio and C# is that it allows comments and important information to be easily captured via XML tags documented in a XML file at build time so I don’t have to worry about performing a separate task to document each automated test.

The STAThread attribute is included above the Main method to support the Clipboard class members. Without this attribute set the test will throw an un-handled exception. Finally, I put in any code that I commonly reuse. That doesn’t mean that I have to use it…I can easily remove it if I don’t need it. (That’s the beauty of being able to think logically for one’s self!) But, if it is code that is commonly used then I don’t have to retype it (meaning…it increases my efficiency even by a little bit).

To complete the process I simply select the File –> Export Template… menu item and complete the Export Template Wizards.

I can create several custom templates depending on the context of each type of test. For example, if the automated test case is not going to drive the UI then I don’t need all those additional references in my template. Rather than remove them each time, I can simply create another template for non-UI automated test cases. Now, each time I start a new project I can simply select the appropriate template. The templates are saved in a ZIP file in the Visual Studio\My Exported Templates directory, and those ZIP files can be sent around to other members of the team.

Also, here are some important key concepts for creating standard templates (in general):

• Don’t screw the majority just to make the minority happy! Only include key elements that provide value to the majority of consumers of the template the majority of the time. A common problem with many templates is that they try to include everything for everybody. When templates are over-laden with things that only a handful of people use, or if people have to spend too much time trimming out things the majority of people don’t use then the templates are a bottleneck. Remember, templates can be customized by those few who might need a few additional things occasionally; majority rules!
• Don’t over-document useless tidbits. Notes and additional comments must be useful, but brief. If someone has to spend more time learning how to fill out a template than it takes to fill out the template then that template provides no value in terms of making the process or the team more efficient. If the template is so complicated that it requires a user manual…then it is not a useful template.
• Intuition is important. If the template is not easy to use, if the consumer of the template has to search around for items, or even worse, forces the user to use parts of the template because they are on the template, then the template is not intuitive and using the template becomes a chore rather than a tool to improve efficiency.
• Get buy in! Don’t develop templates in isolation. Talk to various consumers and users of the templates to better define the template and what is captured in the template. For example, if you are developing a template for reporting anomalies in software then developers should be included in that conversation as should program managers and any other consumers of the completed templates.
• Templates are not commandments! Templates provide a starting point; they are not engraved in stone. Templates should be designed to include things that are commonly reused by various users and consumers of the completed template. But, sometimes they can include things that are unnecessary and do not add value in a specific context and in those cases the user of the template should be able to remove those parts of the template or at least ignore them. Conversely, other times additional things may be necessary in a template and the user of the template should be able to easily add those things that provide important value to the purpose of the completed template.

Bottom line, templates are tools to provide a common reference, or central starting point…they are not hand-cuffs designed to constrain rational and logical thought or constrict a person’s creativity. In fact, good templates are tools that enable and support more efficient creativity!

Originally Published Wednesday, July 02, 2008

Occasionally, the execution or outcome of a test depends on the operating system version on which the test is executed. Platform profiling is important because subtle differences in operating system platforms can affect how certain tasks are carried out, inclusion or exclusion of specific features or capabilities, work-arounds, or even changes in deterministic or declarative oracles that determines pass, fail, or indeterminate results. For example, accessing certain features on the Vista platform will trigger the User Account Control (UAC) dialog, whereas this feature does not exist on previous versions of Windows.

It is easy for a tester to select the appropriate OS version when executing tests manually and determine the correct steps or settings and determine the correct results of a test based on their cognitive expectations for an outcome, But, in an automated test system where tests are distributed across the wire to various machine configurations and operating system platforms the test designer must programmatically detect the operating system version in automated tests that need to be profiled for the appropriate platform. Programmatic platform profiling negates the need to write several different tests for a single test case that is dependent on variations in the operating system versions. Detecting the operating system version at runtime allows the automation test designer to handle subtle differences between the operating systems within one one test case.

Detecting the major and minor OS versions with C# is relatively straight forward using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members.

The System.PlatformID enumeration detects the major operating platforms including:

• Win32Windows – versions of Windows 9x, including Windows ME
• Win32NT – versions of Windows 2000, Xp, and Vista
• WinCE – versions of Windows CE
• Unix – versions of the Unix OS

Once the platform is determined, the test designer can drill down further using the Environment.OSVersion.Version.Major and Environment.OSVerions.Version.Minor members to find the specific major and minor operating system version number. If tests are dependent on specific revisions or service packs of the major operating system then the integer or string values can be obtained using the OSVersion.Version.Revision or OSVersion.Version.Build members to determine  specific versions of an operating system. Some versions of Windows used alphabetic characters in the revision number, so in those cases the revision should be converted to and compared using a string object as represented by the constant values for Windows 95 retail, Windows 95 OSR 2.1 and Windows 95 OSR 2.5 and Windows 98 Second Edition. Also, the OperatingSystem.ServicePack property can be used to determine if, and what service pack has been installed on the operating system.

   1: class OSVersionInfo

   2: {

   3:   /// <summary>

   4:   /// Windows NT Kernel Major Version Numbers

   5:   /// </summary>

   6:   enum Win32NTMajorKernelVersion 

   7:   {

   8:     NT4 = 4,

   9:     NT5 = 5,

  10:     NT6 = 6

  11:   }

  12:  

  13:   /// <summary>

  14:   /// Windows NT Kernel Minor Version Numbers

  15:   /// </summary>

  16:   enum Win32NTMinorKernelVersion

  17:   {

  18:     Win2000 = 0,

  19:     WinXp = 1,

  20:     Win2003Server = 2,

  21:     Vista = 0,

  22:     Win7 = 1

  23:   }

  24:  

  25:   /// <summary>

  26:   /// Windows 9x Kernel Minor Version Numbers

  27:   /// </summary>

  28:   enum Win9xMinorKernelVersion

  29:   {

  30:     Win95 = 0,

  31:     Win95Osr2 = 3,

  32:     Win98 = 10,

  33:     WinME = 90

  34:   }

  35:   

  36:   // Service Pack identification strings

  37:   private const string VistaSp1  = "Service Pack 1";

  38:   private const string Win95Sp1     = "950A";

  39:   private const string Win95Osr2   = "950B";

  40:   private const string Win95Osr25   = "950C";

  41:   private const string Win98SE       = "2222A";

  42:  

  43:   /// <summary>

  44:   /// Detects the current version of the operating system useful for

  45:   /// controlling execution branches in automated test cases that are

  46:   /// platform dependent

  47:   /// </summary>

  48:   /// <returns>A string representing the OS version</returns>

  49:   private static string GetOSVersion()

  50:   {

  51:     OperatingSystem osVersionInfo = Environment.OSVersion;

  52:     string osVersion = string.Empty;

  53:     switch (osVersionInfo.Platform)

  54:     {

  55:       case PlatformID.Win32NT:

  56:         switch (osVersionInfo.Version.Major)

  57:         {

  58:           case (int)Win32NTMajorKernelVersion.NT4:

  59:             osVersion = "NT4";

  60:             break;

  61:           case (int)Win32NTMajorKernelVersion.NT5:

  62:             switch (osVersionInfo.Version.Minor)

  63:             {

  64:               case (int)Win32NTMinorKernelVersion.Win2000:

  65:                 osVersion = "Win2K";

  66:                 break;

  67:               case (int)Win32NTMinorKernelVersion.WinXp:

  68:                 osVersion = "WinXp";

  69:                 break;

  70:               case (int)Win32NTMinorKernelVersion.Win2003Server:

  71:                 osVersion = "Win2003Server";

  72:                 break;

  73:             }

  74:  

  75:             break;

  76:           case (int)Win32NTMajorKernelVersion.NT6:

  77:             switch (osVersionInfo.Version.Minor)

  78:             {

  79:               case (int)Win32NTMinorKernelVersion.Vista:

  80:                 if (osVersionInfo.ServicePack == VistaSp1)

  81:                 {

  82:                   osVersion = "Vista SP1";

  83:                 }

  84:                 else

  85:                 {

  86:                   // Windows Vista or WinServer2008

  87:                   osVersion = "Vista";

  88:                 }

  89:  

  90:                 break;

  91:               case (int)Win32NTMinorKernelVersion.Win7:

  92:                 osVersion = "Windows 7";

  93:                 break;

  94:             }

  95:  

  96:             break;

  97:         }

  98:  

  99:         break;

 100:       // Windows 9x Versions

 101:       case PlatformID.Win32Windows:

 102:         switch (osVersionInfo.Version.Minor)

 103:         {

 104:           // Windows 95, OSR 1.0, OSR 2.0

 105:           case (int)Win9xMinorKernelVersion.Win95:

 106:             if (osVersionInfo.Version.Revision.ToString() == Win95Sp1)

 107:             {

 108:               osVersion = "Windows 95 SP1";

 109:             }

 110:             else if (osVersionInfo.Version.Revision.ToString() == Win95Osr2)

 111:             {

 112:               osVersion = "Windows 95 OSR 2.0";

 113:             }

 114:             else

 115:             {

 116:               osVersion = "Win95";

 117:             }

 118:             break;

 119:           // Windows 95 OSR 2.1, OSR 2.5

 120:           case (int)Win9xMinorKernelVersion.Win95Osr2:

 121:             if (osVersionInfo.Version.Revision.ToString() == Win95Osr2)

 122:             {

 123:               osVersion = "Win95_OSR2.1";

 124:             }

 125:             else if (osVersionInfo.Version.Revision.ToString() == Win95Osr25)

 126:             {

 127:               osVersion = "Win95_OSR2.5";

 128:             }

 129:             break;

 130:           // Windows 98 and Windows 98 Second Edition

 131:           case (int)Win9xMinorKernelVersion.Win98:

 132:             if (osVersionInfo.Version.Revision.ToString() == Win98SE)

 133:             {

 134:               osVersion = "Win98SE";

 135:             }

 136:             else

 137:             {

 138:               osVersion = "Win98";

 139:             }

 140:             break;

 141:           // Windows ME

 142:           case (int)Win9xMinorKernelVersion.WinME:

 143:             osVersion = "WinME";

 144:             break;

 145:         }

 146:  

 147:         break;

 148:     }

 149:  

 150:     return osVersion;

 151:   } 

 152: }

 153:  

Currently, C# does not have a property or method to determine the specific edition of an operating system. If a test is dependent on a specific edition of Windows 2000, Windows Xp, or Windows 2003 operating system edition then we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function.

Programmatically detecting the operating system version at runtime enables the test designer to design one test that can execute on a different operating system versions by allowing for specific control flow of a test based on the operating system the test is running. Programmatic platform profiling also negates the need to rewrite the same fundamental test over and over again to handle differences in behavior or expectations of operating system versions. It can also reduce some long term costs of test case maintenance, give the test designer greater control over the execution of an automated test, and increases the reusability and the maintainability of a test case that must be ran on multiple operating system versions.

Quite simply, programmatic platform profiling is a best practice in software test automation when a given test is dependent on the profile of the platform on which it is executing!

Originally Published Tuesday, May 13, 2008

Now, many of you probably know that I am a big fan of computer generated random test data that is a represents a reasonable sample data set from the total population of possible test data. (I refer to this a probabilistic stochastic test data.) So, why would I argue against preserving randomly generated test data?

I just returned from STAREast, where for the second time in a month I heard someone suggest storing randomly generated test data in a file. Many people will site the inability to recreate random test data as a drawback to using randomly generated test data in a test. So, the reason these people suggested storing the random data in a file is so they can easily repeat a test with the same data should some randomly generated test data expose an anomaly. I absolutely concur that if we generate random test data, and that test data exposes a problem we need a way to recreate the data. But, isn’t there a better way than to save random test data in a file?

Saving randomly generated test data to a file creates a test artifact. Depending on how much randomly generated data is generated, this file could become quite large. Also, saving data to a file impacts the performance of an automated test and certainly slows down manual execution of tests. Then consider the number of tests that generate random test data are executed numerous times throughout the lifecycle, and it doesn’t take long until we have countless test artifacts simply storing more static test data that quickly loses its value (especially if no problems were detected). Of course, we can easily delete the files after the test if no anomaly was detected, but I suspect that most testers will delete those files upon the completion of the test if no problems were detected.

So, the question is how can we reproduce computer generated probabilistic stochastic test data if we don’t save that randomly generated data to a file?

Planting Seeds

In computing, a seed is simply an integer value that is used by a random generator as the starting value. If we pass a seed value as an argument to a given random generator then we will consistently get the same random value each and every time. Essentially, a seed allows us to replicate computer generated probabilistic stochastic test data anytime as long as we use the same seed and the same random generator algorithm. So, instead of saving each and every piece of randomly generated test data used in any given test, we can simply log the seed value used by that test in the test results log file.

But, if we use the same seed all the time, then we are simply generating the same data over and over again. And, manually inputting a seed for each test that generates probabilistic stochastic test data is not an ideal situation, especially for automated tests. So, to solve that problem we can randomly generate a seed value that is then passed to the random generator algorithm!  Again, logging the randomly generated seed allows us to accurately reproduce the probabilistic stochastic test data at any later time.

The example below illustrates a simple method in C# that will either generate a random seed or return a user specified seed value.

   1: public static int GetSeedValue(string seedValue)

   2: {

   3:     // check if user specified seed value is passed as an arguement to 

   4:     // the seedValue parameter

   5:     if (seedValue == string.Empty)

   6:     {

   7:         // Create a new random object

   8:         Random randomObject = new Random();

   9:         // Generate a random integer value between 0 and 2,147,483,647

  10:         return randomObject.Next();

  11:     }

  12:     else

  13:     {

  14:         // convert the seedValue to an integer value

  15:         // NOTE: This example method does not include exception handling

  16:         return int.Parse(seedValue);

  17:     }

  18: }

The following example illustrates how to use this method to get a random seed value to generate random strings and numbers that increase the breadth of test data coverage in each subsequent iteration of a test.

   1: static void Main(string[] args)

   2: {

   3:     // These variables declare the range of characters used for the

   4:     // string test data. In this case the strings are composed of upper

   5:     // case ASCII characters 'A' through 'Z'

   6:     char minChar = '\u0041';

   7:     char maxChar = '\u005A';

   8:     

   9:     // This reads the user specified seed value from the console window

  10:     // If no seed value is specified an empty string is passed to the 

  11:     // GetRandomSeed method which will cause it to generate a random 

  12:     // seed value.

  13:     string mySeed = Console.ReadLine();

  14:     

  15:     // Declare a seed variable and initialize it to either the user

  16:     // specified seed or to a computer generated random seed value

  17:     int seed = GetSeedValue(mySeed);

  18:  

  19:     // The seed value should be permenently recorded in the logged

  20:     // results for this test

  21:     Console.WriteLine("The seed value for this test is {0}\n", seed);

  22:  

  23:     // Create a new random object based on the seed

  24:     Random randomGeneratorObject = new Random(seed);

  25:  

  26:     // Generate 10 random strings

  27:     for (int count = 0; count < 10; count++)

  28:     {

  29:         // Declare and initialize a string variable for our test data

  30:         string testString = string.Empty;

  31:         // Generate random length strings between 1 and 10 characters

  32:         for (int length = 0; length < randomGeneratorObject.Next(1, 11); length++)

  33:         {

  34:             // Generate a random character within the defined range and

  35:             // concatenate it to the testString variable until the 

  36:             // random string length has been reached

  37:             testString += Convert.ToChar(randomGeneratorObject.Next(

  38:                 minChar, maxChar + 1)).ToString();

  39:         }

  40:  

  41:         // Write the test string to the console window

  42:         Console.WriteLine("Test String {0}: {1}", count + 1, testString);

  43:     }

  44:  

  45:     Console.WriteLine("\nRandom numbers");

  46:     // Generate 5 random numbers

  47:     for (int numberCount = 0; numberCount < 5; numberCount++)

  48:     {    

  49:         Console.WriteLine("{0} ", randomGeneratorObject.Next());

  50:     }

  51: }

Calling the Main method and passing an integer value between 0 and 2,147,483,647 will generate 10 random length strings composed of random upper case characters between ‘A’ and ‘Z’ and 5 random numbers. If no user specified seed is passed to the Main method then the code will call the GetGenerateSeed method and generate a random seed value for use in the test. Of course, passing the same integer value will produce the same strings and numbers each and every time.

Using probabilistic stochastic test data is valuable because it efficiently increases the breadth of data coverage, and significantly augments ‘typical’ static test data, user-generated test data, or static test data derived from historical failure indicators. But, instead of storing randomly generated test data in a file, it is a best practice to simply record the seed value of each test. With a seed value we can easily recreate the computer generated random test data should any of the random data used in a test exposes an anomaly.

Originally Published Friday, March 28, 2008

It seems that many test automation efforts around the industry tend to focus on GUI automation, or automating functional tests primarily by manipulating GUI objects. In general, GUI automation tends to be a very expensive approach to test automation, and the automation efforts often end in failure or achieve less than satisfactory results.

The majority of automated tests at Microsoft are below the GUI; however, automated tests that manipulate GUI objects are quite useful within specific contexts. Unfortunately, many testers attempt to develop automated GUI tests way too early in the project cycle while the user interface design is still unstable. I guess the assumption is that constantly maintaining automated tests is somehow better then executing manual tests. But, in general, when the UI is in flux it is usually counter-productive and a loss of return to automate GUI level tests too soon in the development lifecycle.

Dan Mosley and Bruce Posey (Just Enough Software Test Automation) suggest that on average an automated test must run approximately 17 times in order to break even. But, this doesn’t imply that we break even if we simply run the same test on a daily build for the next 17 days. The presumption underlying the ROI after 17 runs of a test is that something changed in the build that is covered by that particular test, and so by executing that (regression) test we are providing important information (changes in the build did not destabilize that area) to the team. Steve Rowe also has an excellent blog post on Too Much Test Automation that you should consider reading, and Dustin Andrews also has an excellent blog post on getting great results from test automation with an 8-minute video.

The cost of test automation is never easy to figure out and is certainly not a straight forward comparison of automated time versus manual time. Comparing automation time versus manual execution time is an overly simplistic measure that rarely takes into consideration the design and the development time of the initial tests (or the overall costs of building and enhancing test frameworks or drivers), or the time required to identify false negatives, troubleshoot the cause, fix the problem, verify the fix (manually), and then check the source back in for the next run. (And, there are other tangible costs, and intangible costs such as loss of confidence that must also be considered in any cost model.)

So, in general any automation that requires constant maintenance is usually not cost effective, and the more our test automation throws false negatives the less our management team views automation as a viable resource to provide us with valuable and reliable information for improved risk analysis. Ultimately the decision comes down to how much perceived and measurable value a test has in providing important information for improved risk assessment and quality measurement.

Just because we can automate something, doesn’t always mean we should!

Originally Published Sunday, January 06, 2008

It has been awhile since I have written about test automation, so I thought I would start the new year off with a post about test automation. More specifically, I wanted to start talking about coding guidelines. Just as many development teams have adopted coding standards and guidelines the test team should also adopt a set of guidelines and standards for all testers to follow when developing test automation. Some teams have coding standards and guidelines, but some teams just sort of wing it. Of course, those teams that just sort of wing it and allow developers the freedom to develop their own style of coding soon realize the cost of reviews and maintenance increase. Sloppy coding also increases the probability of subtle defects that may go unnoticed. Coding standards or guidelines are not (or should not) restrict a test developers creativity; but consistent coding standards and guidelines improve the efficiency and effectiveness of code reviews and sustained maintenance (which is often performed by someone other than the person who wrote the original code).

This week I want to start by talking about problems with various coding styles in source files such as indentation and general layout issues that make code more difficult to read. Readability is important for efficient code reviews, and when the code is later maintained by someone other than the person who originally wrote the code to begin with. There are many coding standards and guidelines that a team can and should adopt and I plan to discuss various guidelines over time, but for this week I thought I would introduce some commonly accepted guidelines that improve the readability of code by adopting a consistent layout.

Indentation

• Set the indentation or tab size to 4 characters in width
  I have seen people set the tab size to as small as 2 and in some rare cases to 3 for some bizarre reason. But, 4 characters in width seems to be the standard tab size, and an additional 2 characters per line is really not going to make that much of a difference, and I think it improves readability of the code by clearly indenting blocks. (I really don’t want to get into a long debate about 2 or 4 spaces; the point is that the team should use a consistent indentation size.) 
• Use spaces instead of tabs
  This is often controversial, but spaces are recommended over tabs for a variety of reasons. One reason why one should use spaces instead of tabs is for consistent layout and readability in a variety of editors. Many text editors interpret tab stops differently, and occasionally the editor a reviewer uses to look at source code may be different than the source code editor used by the developer causing the line formatting to change radically. For example, when I need to take a quick look at a .cs file I will often open that file using Notepad instead of launching Visual Studio. Sure, I don’t get the pretty colors and such, but it’s a lot quicker! (Again, I don’t really want to get into a debate about tabs vs. spaces. That subject has been beaten to death on the Internet and there are pundits on both sides of the fence. If file size is a really critical factor than tabs will reduce the overall physical file size in bytes. If readability in different editors is important, or if restricting the use of the 0×09 character code point control code in your source code is important the use of spaces is preferential. Again the important point would be consistency within the team.)

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Tools -> Options… menu item
2. Select Text Editor -> All Languages -> Tabs item in the Options tree control
3. Set the indentation to Smart.
4. Set the tab size and indent size to 4
5. Select Insert spaces (or Keep tabs; whichever standard your team adopts)

Line length

• Do not extend lines beyond 80 characters in length
  Occasionally, developers will simply write a statement that extends well beyond the viewing area of the editor window forcing reviewers to scroll. Or worse yet, when opening a file in Notepad or some other simple text editor the line wraps (if word wrapping is selected) around causing the layout to become quite confusing to the eyes. Word wrapping also becomes a problem when printing source files for review. Either way extending lines beyond 80 columns looks sloppy and makes readability more difficult.

Notice lines 76 and 80 extend well beyond 90 characters. If this source file was printed lines 76, 80, and 83 would wrap to the next lines on the page starting from the left margin impacting the readability of the file.

The column (or character count) per line is usually displayed in the status bar of an IDE, but for Visual Studio (including Visual Studio Express editions) a simple registry setting provides a visual guide in the IDE. The visual guide in the IDE means that I don’t have to look at the status bar to see where my column count is, or go back later and reformat lines after I am finished coding a method or class.

Notice the red dotted line on the right hand side of the code window set at a column width of 80 characters.

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Start -> Run menu item
2. Type regedit in the Open control on the Run dialog and press OK
3. In the Key tree control select HKEY_CURRENT_USER | Software | Microsoft | Visual Studio | [version] | Text Editor
4. Right click in the value pane and select New -> String Value from the pop up menu
5. Rename the new value Guides
6. Right click on the Guides value item and select the Modify… menu item
7. Type RGB(255, 0, 0) 80 in the Value data textbox on the Edit String dialog

I also prefer to display line numbers. This makes it a bit more convenient when doing things such as Finding All References to variables.

In subsequent posts I will discuss additional coding guidelines, but these are just a few ideas to get the discussion started. If you have additional ideas on layout then please let us know!

Originally Published Thursday, August 30, 2007

I never really understood why so many people external to Microsoft seem to be against the Microsoft strategy to increase the amount of automation we rely on to test our products. Test automation has become sine qua non at Microsoft for many valuable reasons.  Although there are some uninformed managers who propose the nonsensical notion of 100% automation, Microsoft as a company does not have some magical goal of 100% automation (whatever that means).

I have come to the conclusion these (mostly) external naysayers ("people who tend to be negative, who have difficulty thinking outside of the box, who go around, consciously or unconsciously, squelching dreams, ideas and visions") of automation simply do not, cannot, or perhaps refuse to comprehend the complexity of the systems or the problem space we deal with at Microsoft or how we design and develop our automated tests (in general, we do not write scripted tests). Or perhaps the anti-automation folks are SOS (stuck on stupid) and believe that any test (including automation) that does not find a bug is not a reasonable test or that executing that test does not provide some perception of reasonable value to an organization.

So, for those of you who are not afraid of being replaced by automation (if you are afraid a machine will take your job, then it probably will), and for those of you who have an open mind and know that real testing is not a simple comparison of scripted vs. exploratory, and that successful testing requires a myriad of techniques, methods, and approaches, then read on and I will discuss some of the reasons why we automate at Microsoft.

• Increasingly complex test matrices. (Whether you like it or not) the Windows operating system proliferates throughout the world. But, even the different Windows operating systems in common use throughout the world are staggering. Although we no longer support Windows 9x (including ME) there is still support for Windows NT 4.0, Windows 2000, Windows Xp, Windows 2003 Server, and Windows Vista. Then add in various combinations of service packs and hot fixes released for each of these, and test environments composed of upgrade scenarios (Windows Xp upgraded to Windows Vista) approximately 30 language versions, 16- bit, 32-bit, and 64 -bit and the number of operating system platforms alone becomes mathematically staggering. But, instead of writing several automated tests for each combination of environment parameters our testers ideally design a single test that detects the OS platform and environments, and develop the test in a way it can decide how to achieve its objective based on platform profiling and other design techniques. (For example, a test for defrag has to take into account the German version of the OS does not contain the defrag utility (long story), or a security test on the French version considers the French version does not contain 128-bit encryption). One automated test instead of an army of button pushers doing the same thing on 5+ operating environments and 30 languages just makes sense!
• Sustained maintenance. Microsoft supports many of its products for 7 to 10 years. So, the more test automation we can hand off to our sustained engineering teams, the less cost the company has to bear in the long term. Do these regression tests find a lot of defects. Perhaps not, but they do provide confidence that changes to the code base introduced by hotfixes and service packs do not adversely impact previous functionality. They also eliminate the burden of having to maintain an army of testers for the entire shelf life of the product.
• Increase breadth of coverage. Automation is distributed across the network, it is not ran on one or two desktops or on a few lab machines. Many product teams have extensive test harnesses that are capable of scheduling tests, configuring environments, then distributing tests to hundreds of machines in a lab, or even on idle machines on the network. This way we not only run test automation on pre-defined environments, but also on work machines and other systems. Test automation is running 24 hours a day and collecting and providing valuable information.
• We don’t rely on automation to find bugs, we use test automation to provide information. Sometimes automated tests can expose unexpected and/or unpredictable behavior, but we know that automated tests do not find a great deal of defects after the design and development phase of that automated test. However, many of our products have daily builds and instead of rerunning a bunch of tests to ensure changes from a previous build have not affected previously tested functionality an automated test is more efficient. The minefield analogy sometimes applied as an argument against regression testing only really makes sense if the test is being executed on a static code base. But, if you test in a real world where you might get daily or even weekly builds of complex systems then you probably realize that executing a set of tests after someone changes the minefield, has a probability that following the same path may expose a defect, and if it doesn’t then it still provides valuable information.
• Increased job satisfaction! This is a big side benefit. I know some people are afraid of automation, and some people may lack the skill to design and develop effective automation, but professional testers realize it is a huge challenge to design and develop effective test automation for complex systems. To quote one tester, "the hardest code I ever wrote was the code needed to test other code."

There are many more justifiable reasons why increased automation simply makes sense at Microsoft, but this should give you some idea of why without automation our jobs would be much more difficult than they already are.

Originally Published Wednesday, August 01, 2007

I am fascinated with the advances computing, and have always approached computing from the perspective of what can this tool do for me to make my life easier. As a professional tester I have a lot more work to do then I can reasonably accomplish in the limited timeframe allotted for most projects. So, well-designed test automation is a great tool that frees up some of my cycles performing mundane tasks that need to be accomplished.

A few months ago I had an email exchange and an industry consultant regarding test automation and emotions that I blogged about and he later talked about at a Star conference. In that post I also tried to illustrate a simple technique called polling for simulating irritation (or frustration) of a task that is taking too long to complete via an automated test.

The consultant I had the email exchange with wrote, "I would want my automation to feel frustration, to recognize it, and to act on those feelings in some way that provides valuable information to the product.  But until we’ve got not only artificial intelligence, but also artificial emotional intelligence, that ain’t gonna happen."

When I design an automated test I often think of the various ways to achieve exactly what it is I am trying to prove or disprove with the test. Then I think of the things that can go wrong (such as race conditions, errant message boxes, tasks taking too long to complete, making simple decisions based on Boolean states, etc.) and design the test in such a way that can logically deal with those situations. So, as I thought about our conversation and automated test design I asked myself, can automation do more than simple mundane tasks? Can automation make decisions or perform tasks based on practical reasoning or simulated emotions?

It seems that some researchers in the Netherlands are unlocking doors with artificial intelligence that may eventually lead to advances in smarter test automation design. Researchers at Utrecht University are hard at work on an emotional robot (a cat none the less) that simulates 22 emotions including "anger, hope, gratification, fear, and joy," used in complex decision making processes. Marvin Minsky stated "…we all have these things called emotions, and people think of them as mysterious additions to rational thinking. My view is that an emotional state is a different way of thinking."

I agree with the researchers, and I "don’t believe that computers can have emotions," and also mostly agree with their statement "that emotions have a certain function in human practical reasoning." (I say mostly because I do know that some emotions express by some people are completely irrational and result in impractical reasoning.) Perhaps AI in test automation this is still a long way off, but I am always looking for ways to improve and become more effective and more efficient. I am always learning and looking for ideas to improve myself and my skills.  So, based on this research I now ask myself, are there cost effective emotional logic patterns to simulate rational reasoning, and I can or should I employ that in the design of some of my automated tests to make them more robust?

Just a thought. Isn’t technology great!

Originally Published Wednesday, July 25, 2007

When getting close to shipping a product and the team discovers a critical defect that must be fixed one of the first questions asked by the management team is, “how long will it take?” From a testing perspective they generally want to know 2 things; what has to be done to reduce perceived risk (or increase their confidence the fix hasn’t introduce new defects), and then they ask how long will it take. Unfortunately, many teams take a swag at the time element and proceed to beat on the product until the magically agreed upon time expires.

I have seen a lot of test case templates, and one field that I often see omitted from the template (or ignored by the test designer) for a test case is duration, or a reasonable approximation of how long it should take an experienced tester to complete a particular test (whether it is a discrete, functional test or a complex user or business scenario). Adding an estimated time for completion to a test then allows us to better approximate (within an hour or so) how long it will take to execute a particular test suite or a portion of that test suite.

For automated tests written in C#, getting the time for a test to complete is now simple with the Stopwatch class introduced in the .NET Framework 3.0. The Stopwatch class is exactly what the name implies; it provides methods and properties to accurately measure the elapsed time between starting the stopwatch instance and stopping the stopwatch instance. The following code snippet provides a simple example of how the Stopwatch instance might be used in a functional or behavioral automated test.

   1: using System;

   2: using System.Diagnostics;

   3: 

   4: namespace TestingMentor.Examples

   5: {

   6:     class StopwatchExample

   7:     {

   8:         static void Main(string[] args)

   9:         {

  10:             // initialize a new stop watch instance

  11:             Stopwatch myTotalTestTime = new Stopwatch();

  12:             // start the stop watch

  13:             myTotalTestTime.Start();

  14:

  15:             // test code goes - simulated by sleep

  16:             System.Threading.Thread.Sleep(3500);

  17: 

  18:             // stop the stop watch

  19:             myTotalTestTime.Stop();

  20: 

  21:             // log the elapsed time to the log file 

  22:             // (simulated by writing to the console window in this example)

  23:             Console.WriteLine("Total test time: " +

  24:                 GetElapsedTestTime(myTotalTestTime.Elapsed));

  25:         }

  26: 

  27:         private static string GetElapsedTestTime(TimeSpan timeSpanObject)

  28:         {

  29:             string time = String.Format("{0:00}:{1:00}:{2:00}:{3:00}",

  30:                 timeSpanObject.Hours, timeSpanObject.Minutes,

  31:                 timeSpanObject.Seconds, timeSpanObject.Milliseconds / 10);

  32:             return time;

  33:         }

  34:     }

  35: }

It is also possible to create several instances, or start and stop several stopwatches in a single test to measure not only the duration of the complete test, but how long it takes to complete a particular task within an automated test. This is extremely valuable for performance testing (a test used to determine the time required to perform and complete a particular task), or stress testing to measure mean time to failure (MTTF) and mean time between failures (MTBF).

Adding a stopwatch to the tests in your automated test suite will better enable you to accurately determine the duration or amount of time required not only to execute a complete test suite such as the build verification test suite or an automated regression test suite, but it also allows us to better calculate the time required to execute a particular subset of tests in a test suite (such as all priority 1 tests, or tests in a particular functional area).

Originally Published Thursday, June 21, 2007

Anyone who has listened to me talk about automated test design knows that I loathe hard-coded strings in test automation. Yes, I know that static test data is important, but it is almost never a good practice to hard-code strings or other test data in the body of your test code (or any code for that matter). 

But, as I started porting some of my demos to run on Windows Vista I realized that I violated my own axiom by hard-coding local paths for test artifacts (test files) in my own test code. Granted this code was for demonstration and training, but that is not an excuse for slovenly written code. Besides, as a general principle we should develop all code following which ever best practices you adhere to because we never really know who is going to critique that code or use it as an example elsewhere. 

Unfortunately, I am not alone in committing this mistake and I occasionally encounter hard coded paths in various automated test projects. Perhaps this occurs because it takes less effort (and lines of code) to hard code a path rather than get the path using a .NET method or Win32 API, or perhaps because we might assume the code will never run on a different machine (both of which are probably bad assumptions). No matter what the reason hard-coded paths are generally not a good practice.

In some automated tests I generate dynamic test data during runtime and create a file to temporarily store that data for use in the test, or save other user files with information that doesn’t necessarily have to persist after the completion of that automated test. (In generating dynamic test data I only have to log or record the seed value used to generate the data.) In one demonstration I created test artifacts (test files) and saved them to the root of the C: drive. (Yes, I know the old PC-98 architecture used in Japan might not have a C: drive because it had a floating drive architecture, but on the PC/AT architecture running the Windows OS environment the default drive is almost always C:.)  Writing to the root directory of the C:\ drive was easy and relatively safe in strictly controlled environments, but it is certainly not the best thing to do (and generally bad from a security perspective). But, fast forward to Windows Vista and my code breaks. Even though I have administrator privileges when I run the demo code and attempt to save a file to the root of C:\ on the Vista platform the system gives me an error message indicating I don’t have permission to save in that location. Unexpected dialog == unexpected behavior == code breaking. File not found by the test oracle in the expected location == false positive. Demonstration falling apart in front of an audience == embarrassment (because it really does no good to say "well, it worked on my other machine!").

Avoiding hard-coded paths for storing temporary test artifacts (e.g. disposable test files generated during the automated test) in automated test cases is easy with C# and the .NET framework. If we need the path to the desktop folder, or program files folder (or other special folders created by the Windows operating system) the GetFolderPath method in the System.Environment class gets the path to "special" system folders. The special folders include folders such as the system folder \windows\system32), the Program Files folder, the Desktop folder, the MyDocuments folder, etc. and are listed on MSDN under the Environment.SpecialFolder enumeration.)

For example, the following snippet returns a path to the current user’s Documents folder ("C:\Users\[logged in account name]\Documents") which is a good place to store temporary test artifacts.

            string desktopFolderPath = Environment.GetFolderPath(
                Environment.SpecialFolder.MyDocuments);

It would probably be better to use a StringBuilder object as illustrated in the following example because a string is an immutable object in C#, and we will probably want to append the path separator and eventually a filename for the test artifact.

            StringBuilder desktopFolderPath = new StringBuilder(
                Environment.GetFolderPath(
                Environment.SpecialFolder.MyDocuments) +
                Path.DirectorySeparatorChar);

This returns a StringBuilder object of "c:\\users\\[logged in account name]\\desktop\\".

Another useful location to temporarily store disposable test artifacts might be the current user’s temporary directory ("C:\\Users\\[logged in account name]\\AppData\\Local\\Temp\\") as illustrated in the following example:

            StringBuilder tempFolderPath = new StringBuilder(
                Path.GetTempPath());

Using the Environment.SpecialFolder enumeration or the .GetTempPath method are easy ways to get the path to a folder to store disposable test artifacts on the local machine without hard-coding a path in the automation code. It is likely this will be reused, so it is a good idea to wrap some methods for inclusion in your test automation framework or library. Also, don’t forget to delete or move any test artifacts from the test machine after the automated test completes. It is generally a best practice to restore a system to its’ pre-test state after your automated test executes.

Originally Published Friday, June 08, 2007

In April I hosted a session at the Software Testing and Performance conference entitled Why Test Automation Fails. I actually tried out a new delivery style for me similar to a town hall format where I attempted to elicit participation from the attendees. Unfortunately, it didn’t work out as well as I had expected, but at one point a consultant who attended the session offered their opinion expressing what they thought automation can’t do.

He stated automation can’t empathize, project, anticipate, recognize, judge, predict, project, evaluate, assess, become resigned, get frustrated, invent, model, resource, collaborate, decide, work around, strategize, charter, teach, learn, appreciate, question, refine, investigate, speculate, suggest, contextualize, explain, elaborate, reframe, refocus, troubleshoot, and THINK.

To this I exclaimed, “I am not sure I want my automation to get frustrated!” At the time I was thinking frustration is typically not a productive emotional trait because it usually manifests itself as “a deep chronic sense or state of insecurity and dissatisfaction arising from unresolved problems or unfulfilled needs” (Merriam-Webster Medical Dictionary). But, while flying home to Seattle that evening I pondered how an emotional condition such as ‘frustration’ might be applied to a tool such as test automation. (Because make no mistake about it; to me test automation is a tool similar to a screwdriver in a toolbox, or a global positioning system on a boat. These tools are designed to be very effective for their purpose. And when test automation is designed and used correctly it also serves a very valuable purpose to those who use it as a tool to increase their productivity.)

So, I sent an email to the person and asked him for the list and restated what I said at the conference.  I wrote, “Like I countered, I am not sure that I want my automation to get frustrated (and honestly I am not quite sure of the point you are trying to make there)…” He responded, “When a tester gets frustrated with some aspect of an application’s behavior (It’s taking too long to respond!  I have to click on this annoying confirmation dialog!  This sequence of steps doesn’t fit with my workflow!  This set of options confuses me!), I can infer that an end-user will too.  An infinitely patient human tester would ill-equipped to identify problems in the application that would annoy or irritate a normal human being.  Automation is infinitely patient (unless we’re programmed it not to be, via a timeout); if it has to wait, it will wait emotionlessly.  We cannot expect the same of our customers.  It’s not that I want automation to get frustrated; I don’t know how to make it do that.  But I do want to get information on how the product feels to humans–so I use human testers for that.”

Other than the fact this person misquoted me in his lightning talk at StarEast 2007, he also acknowledged in his email, “It’s not that I want automation to get frustrated” as well. It is usually not a productive use of one’s time to discuss issues with individual’s who argue against their own points. It’s sort of like someone saying "Elephant’s can’t fly. It’s not that I want elephants to fly, but they can’t…so there!" But, I see in his response that he describes other emotions such as ‘annoyance’ and ‘irritation’ (to rouse to impatience or anger; annoy – American Heritage Dictionary) to describe ‘frustration.’ Now annoyance and irritation are things I can relate to because I sometimes do get annoyed and irritated. So, let’s examine the emotional condition of irritation (because frustration generally implies a negative, and often counter-productive emotion) as described in the quoted email above and determine whether or not ‘irritation’ can be simulated by a computer program.

When I design an automated test I never let the test run “infinitely” because I know that a ‘hang’ or inordinate delay may cause an unexpected reaction in my test execution. (For example, if I am trying to ‘click a button’ on a dialog I must first make sure the dialog with the button instantiates and has focus and the button is not ‘grayed.’) A best practice for designing automated tests is to use polling with a finite timeout. Simply put polling is looping on a conditional statement until either the condition occurs, or the max poll count is reached. If the maximum poll count is reached before the condition occurs the automated test ‘decides’ the application is not in the ‘predicted’ and expected state and a timeout occurs implying that “It’s taking too long to respond” based some predetermined value to simulate irritation (or ‘frustration’).  If the automation becomes ‘irritated’ then I can cause the automation to ‘become resigned’ by designing the automation to log an unexpected error (rather than a failure), and collect information about the machine state and the automation state and log that information for closer examination by a tester.  (I don’t really think about this as programming an artificial emotional response; I think about it from the perspective of good automated test design.  But, in retrospect I guess it is simulating ‘irritation.’)

For example, if I think an application is taking too long to shut down rather than writing a bug report stating something like “The application takes too long to close” I can write a method similar to the following example that actually measures the amount of time it takes for the application under test (AUT) to close (and assuming the method returns true).

   1: public static bool CloseAutProcess(

   2:     Process autProcess, 

   3:     int maxPatience,

   4:     out int levelOfIrritation)

   5: {

   6:     levelOfIrritation = 0;

   7:     autProcess.CloseMainWindow();

   8:     if (!autProcess.HasExited)

   9:     {

  10:         while (!autProcess.HasExited && levelOfIrritation < maxPatience)

  11:         {

  12:             autProcess.WaitForExit(1);

  13:             levelOfIrritation ++;

  14:         }

  15:     }

  16:  

  17:     if (autProcess.HasExited)

  18:     {

  19:         return true;

  20:     }

  21:  

  22:     return false;

  23: }

In this example, the maxPatience variable sets the maximum amount of time (in milliseconds) I am going to wait for the AUT to shutdown before becoming ‘irritated.’ The levelOfIrritation variable accurately measures the time in milliseconds (give or take a few nanoseconds). Now, if the shut down time exceeds reasonable expectations (based on a comparative analysis of similar programs, or stated or implied requirements) I can write an objective bug report and allow management to decide if the ‘level of irritation’ needs to be adjusted based on solid factual information rather than some ambiguous personal patience threshold, or lack thereof. Because at the end of the day, my managers may value my opinion, and sometimes opinions are important deciding factors. But, as a software testing professional my role is to provide reliable, accurate, and detailed information to my teammates and other stakeholders.

(BTW…I can use event handlers to identify the frequency and number of ‘annoying confirmation dialogs” and deal with them. And, with regard to the other things described as ‘frustrating above (“This sequence of steps doesn’t fit with my workflow!  This set of options confuses me!” ) I am thinking these would most certainly be discovered while designing the automated tests (because test automation doesn’t design and develop itself).

So, I am still of the mindset that I don’t want my automation to develop a deep chronic sense or state of insecurity and dissatisfaction (def. – frustration) and I don’t want my automation to have ‘needs.’  And even though I can simulate some degree of artificial ‘emotional’ conditioning in test automation, I don’t want my automation to have emotions any more than I want a hammer or saw or other tool to develop human emotions. Test automation is simply a tool, and it is a tool developed and used by humans (who are emotional) who constantly utilize cognitive processes (including emotional responses) while solving difficult problems or working towards achieving specific goals. The power of a tool is limited only by the skills, ability, and knowledge of the persons designing, developing, and/or using that tool.

I agree that automation can’t THINK (although I am not overly convinced thinking is an emotion). That is why highly skilled people design and develop great test automation, and they don’t simply rely on record/playback or simple rote scripts for automated testing.