I.M. Testy

Originally Published Tuesday, August 25, 2009

I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent points such as:

• Automated software testing “is software development.”
• Automated software testing “and manual testing are intertwined and complement each other.”
• And, “The overall objective of AST (automated software testing) is to design, develop, and deliver an automated test and retest capability that increases testing efficiencies.”

Of course, I was also pleased to read the section on test data generation since I design and develop test data generation tools as a hobby. The authors correctly note that random test data increases flexibility, improve functional testing, and reduce limited in scope and error prone manually produced test data.

There is also a chapter on presenting the business case for an automation project by calculating a return on investment (ROI) measure via various worksheets. I have 2 essential problems with ROI calculations within the context of test automation. First, if the business manager doesn’t understand the value of automation within a complex software project (especially one which will have multiple iterations) they should read a book on managing software development projects. I really think most managers understand that test automation would benefit their business (in most cases). I suspect many managers have experienced less than successful automation projects but don’t understand how to establish a more successful automation effort. I also suspect really bright business managers are not overly impressed with magic beans.

Magic beans pimped by a zealous huckster are the second essential problem with automation ROI calculations. Let’s be honest, the numbers produced by these worksheets or other automation ROI calculators are simply magic beans. Now, why do I make this statement? Because the numbers that are plugged into the calculators or worksheets are ROMA data. I mean really, how many of us can realistically predict the number of atomic tests for any complex project? Also, do all tests take the same amount of time, or will all tests be executed the same number of iterations? Does it take the same amount of time to develop all automated tests, and how does one go about predicting a realistic time for all automated tests to run? And of course, how many of those tests will be automated? (Actually, that answer is easy….the number of automated tests should be 100% of the tests that should be automated.)

Personally, I think test managers should not waste their time trying to convince their business manager of the value of a test automation project; especially with magic beans produced from ROMA data. Instead test managers should start helping their team members think about ROI at the test level itself. In other words, teach your team how to make smart decisions about what tests to automate and what tests should not be automated because they can be more effectively tested via other approaches.

In my next post I will outline some factors that testers, and test managers can use to help decide which tests you might consider automating. Basically, the bottom line here is that an automated test should provide significant value to the tester and the organization, and should help free up the testers time in order to increase the breadth and/or scope of testing.

Originally Published Saturday, August 15, 2009

My daughter made me laugh today when she offered a bit of her philosophy. She told me that her favorite candy is gummy bears “because gummy bears get stuck between your teeth, and then you can dig out a second helping with your tongue.” I never really thought of it that way, but how many of us have not picked at a piece of licorice stuck between our teeth with our tongue (or a toothpick) and savor that last little bit? Ummm….

Perhaps it is my own twisted logic, but as I started writing this post I thought about my daughter’s predilection for gummy bears and somehow made a connection to static test data used in tests. Static test data that is simply reused over and over in a test is similar to that last little bit of licorice we dig out of our teeth. The last bit tastes just like the first bite, and all the other bites between. This may be good for those who like the flavor of licorice, but it is not so good for hard-coded test data in rudimentary test scripts, especially in automated tests.

If you have followed my posts or my personal website then you know that I am a big proponent of probabilistic stochastic test data (statistically unbiased, parameterized randomly generated test data that is representative of the population of possible inputs for a specific variable). The latest addition to my random test data generator toolbox is PseudoName, a random name (pseudonym) generator library for automated testing.

Before designing and developing PseudoName I researched the plethora of available random name generators currently available because I am not a big fan of reinventing the wheel either. In fact, there are many very good online html based random name generators. For example, Fake Name Generator that not only generates a pseudonym, but also generates an address, phone number, etc. essentially creating a fictitious persona. However, while this tool is useful for manual testing it is not so useful for automated tests. The Automated Testing Institute website provides code samples in VBScript and Ruby for generating random names from a built in collection of names stored in an array. These examples are also useful and the collections can certainly be expanded to include a greater variety of names, but they are still limited in scope.

A common problem that I noticed among all available random name generators is the Romanization (representing a written language with the Latin alphabet) of the pseudonym. Basically this means the random names are always represented with ASCII characters. Romanization may be satisfactory for those who only know the letters “A” through “z” or for those whose eyes glaze when the displayed character glyphs are in a foreign language. But, for those of us dealing with modern software or services that supports Unicode and may be adapted (or localized) or used in different locales where it is important to support the native language we soon realize that Romanization using simple ASCII characters is simply not enough for effective globalization testing.

Unlike most random name generators PseudoName generates a random name (pseudonym) from columns of name data in an Excel spreadsheet. The name data in the Excel spreadsheet is stored as Unicode so the characters can be the same as those used in the desired region or locale. For example to generate a random female Chinese name most name generators would produce a string such as “Dongyi Li.” However, PseudoName can randomly generate a name using Chinese characters such as  “冬怡 李.” (Actually, Dongyi Li is not a pseudonym. Dongyi is my friend and she was kind enough to produce the Chinese name list of female, male, and surnames, and also helped me with refactoring the code used in this tool.)

The PseudoName library is simple to use in an automated test. The PseudoName members page also includes simple examples, and the NameInfo properties allow customization of the pseudonym output. If additional properties are necessary to generate reasonably realistic names in different locales please let me know. Also, if there is enough demand I might consider slapping on a GUI.)

The format for the Excel sheet is simple. The first column is female names, the second is male names, and the third is surnames. The names listed in the currently available US and Japanese names data files are the most common names in those countries according to census data. The names in the Chinese data file are the characters used for feminine and masculine names, as well as the most common surnames used in China. (I could really use some help collecting name lists using Unicode character scripts from other countries around the world. If you want to contribute please send me a name list in Excel and I will post it on the tool website for other testers to use.)

Originally Published Saturday, August 01, 2009

When many people think of test automation they envision rudimentary scripts with hard-coded events and data that manipulate user interface objects much the same way a customer might interact with the software to accomplish a pre-defined, robot-like task. Perhaps this is the reason there is a plethora of tools available to business analysts or super-users hired as ‘black-box’ testers to help them record and playback (or list keywords to sequentially step through) some contrived set of steps they think a customer might perform. Sure…it’s cool to watch windows open and close, and the mouse cursor move across the desktop as if by magic. But, for anyone with half-a-brain the visual amazement lasts for for oh….about 1.7 seconds….after that it is mind numbingly boring! Unfortunately, this automation is usually short lived, requires tremendous overhead in terms of maintenance costs, and contributes to the exceedingly high percentage of failed or less than successful automation projects.

I will say that in general I am not a big fan of GUI automation for a litany of reasons, but mostly because it is misused, or applied to tests that are simply more effectively and efficiently performed by a person. However, I also understand the GUI automation does provide value when used in the right context. I am not against GUI automation, but I certainly don’t advocate automating a test just because we think we can, or because we have some bizarre idealistic belief that everything should be automated.

For example, in one situation I spoke with a tester whose manager wanted him to maintain a legacy test designed to detect the correct color of an arrow symbol after an action was performed. If the action completed correctly the arrow was green; and if it was unsuccessful the arrow appeared red. Now, besides the fact that we could have just as easily automated a test to check the HRESULT value, this test could have been executed by a user within a reasonable time, there was little probability of change in this area of the code, and there were no dependencies. However, the manager insisted this GUI test run despite this test which used image comparison as an oracle was notoriously problematic. (This shouldn’t be surprising since many image comparison oracles are notoriously problematic and throw an inordinate number of false positives.)

The tester said the manager claimed by automating this test it would negate a tester from having to execute the test manually thus saving time. What??? This tester was spending hours per week chasing down false positives and tweaking the automation to “make it work” on the daily builds just to make his manager happy. So, although this feature was used repeatedly by hundreds of people dog-fooding the daily build, another few thousand people around the company self-hosting internal releases, and thousands of customers using beta releases this particular manager determined continued tweaking of this test would save some tester’s time!

In another example a tester inquired how to automate a a test to determine IF the order of the slides in a power point presentation had changed between different copies of a .ppt file. Of course, the question was followed by a flurry of responses suggesting creating a base set of images of each slide in the deck, and then using an image comparison tool to identify changes. I responded a bit differently. First, there are several ways to programmatically detect file changes, and if we detect changes in the binary properties we can easily open the Power Point presentation in slide sorter view and take a few seconds (depending on the number of slides) and visually compare it against an original. Sure it is a bit slower than an automated test, but I really suspect it would be more effective and probably even more efficient in the long run. I also wondered how many times this “test” would actually need to be ran during whatever project this person was working on (it wasn’t PowerPoint) in comparison to the hours/days it would take to develop such a test, and the ensuing maintenance nightmare.

These are just 2 examples of the misuse of automated UI testing that I think illustrate a few important points:

• Not all automated UI tests save time!
  Tests that require constant massaging and tweaking because they constantly throw false positives take up a huge amount of a tester’s time in wasted maintenance.
• Sometimes a human is a more efficient oracle than a computer algorithm!
  Sure, just about anything a computer does can be automated to some degree in some fashion, but there really are clearly some tests where it is more prudent and simpler to rely on a tester.
• Don’t rely on automation to emulate your customers!
  Test automation does not effectively emulate a human user. Sure, we have test methods in some of our internal automation frameworks to slow down simulated keystrokes (the actual keys are not being pressed on the keyboard), or simulate multiple or repeated clicks on a control or the mouse, and other tricks that try to emulate various user behaviors; however, test automation is generally poor at detecting behavioral issues such as usability, ease of use, or other customer value type assessments. Rely on the feedback from internal and external customers who are dog-fooding, self-hosting, and beta-testing your product (and act on it).
• Go under the covers!
  I think many testers rely too heavily on UI automation because they think it emulates user behavior (although most things such as populating a text box are simulated via Windows APIs), or perhaps because they don’t know how to dig into the product below the surface of the UI. Whatever the case, think about the specific purpose of the test. If it is easier to check a return value, or call an API to change a setting then go deep…and stop messing around on the surface. (It only complicates the test, wastes valuable machine cycles, reduces reuse across multiple versions, and often leads to long term maintenance costs. (For an example of this see my previous post.)
• Constantly massaging code contributes to false negatives!
  I have seen many cases where a tester designs a a UI automated test, and then tweaks a bit here and there to get it to run. Often times this tweaking contributes to a tests ineffectiveness in exposing problems, and may even hide other problems. Also, some tweaks are geared around synchronization issues (sync’ing the automated test with the system under test) and involve artificially slowing down the automation (usually by stopping or ‘sleeping’ the automated test process for a specific period of time). Other tweaks might hard-code parameters that then make the test fail on a different resolution or non-portable across different environments.
• STOP trying to automate every damn test!
  As I stated before…just because we can automate something doesn’t mean that we should try to automate everything! We need to make rational decisions about what tests to automate, and what is the best approach to automating that test.

It is easy to be lured in by the siren call of UI automation. I write automated tests to free up my time to design and develop more and different tests, and so I don’t have to sit in front of the computer executing redundant tests, or constantly massage code to make it run. Automation is a great tool in the arsenal of competent professionals who understand its capabilities and know how to exploit its potential. But, it is one of many tools in our toolbox; and the best tool is the one sitting on our shoulders. Use it!

Originally Published Thursday, May 14, 2009

Regardless of the personal opinions of a few people, the simple fact is that the demand for software testers who can design and develop effective test automation is increasing. Perhaps one reason for the distain by some folks in the industry is due to limitations of the test automation approach they are most familiar with, and they sometimes assume those limitations apply to all types of test automation. However, not all test automation approaches are equal, and there are advantages and disadvantages for any approach.

At its core an automated test case is software code, and similar to the various approaches used in developing product software there are different programming paradigms used to develop test automation such as:

1. Record and playback automation
2. Keyword or action-word driven automation
3. Scripted automation
4. Procedural automation
5. Model based automation

Record and playback automation

The record and playback paradigm simply records sequences of keyboard and mouse events, auto-magically codifies them usually into some proprietary scripting language which can then be replayed (executed) over and over again. There are usually severe limitations to this type of automation and it tends to be extremely fragile requiring constant massaging (re-recording). Although many record/playback paradigm allows ‘test developers’ to modify the scripted actions to some extent, and possibly even incorporate simple yes/no oracles I think many people view the record/playback paradigm as being slightly more useful than trained monkeys in limited situations.

Keyword or action-word driven automation

Keyword or action-words are simple scripts usually in some tabular format that ‘describe’ a sequence of ‘actions’ for the computer to perform. Of course, the key to keywords is the underlying architecture of the tool that interprets the keywords and executes the sequence of events. The beauty of keyword driven testing is that it hides the actual code, and similar to record and playback can be more easily used by business analysts or ‘user domain experts’ hired to into testing roles to automate something. I do see the benefit of keyword driven testing in some limited contexts (especially for companies who rely on business analysts/user domain experts for testing software), but let’s be real…these people aren’t automating anything…they are simply filling out a form that is then feed into a tool that performs the actions as prescribed by the listed instructions. The keyword form does nothing by itself, and the only thing a ‘tester’ has to think about is using the correct key words to sequentially get from point A to point Z for a ‘test.’

Scripted automation (imperative programming)

The primary difference between keyword and scripted automation is the tester actually develops the test in a programming language rather than filling in a form with abstracted key words that drive some automation engine. However, similar to keywords, scripted automation tends to use rudimentary statements of basic instructions that manipulate the software to perform a pre-determined sequence of events as illustrated below.

   1: def test_b_googlenews

   2: #-------------------------------------------------------------------------

   3: # Test to demonstrate WATIR select from drop-down box functionality

   4: #

   5: #variables

   6: test_site = 'http://news.google.com'

   7: puts '## Beginning of test: google news use drop-down box'

   8: puts '  '

   9: puts 'Step 1: go to the google news site: news.google.com'

  10: $browser.goto(test_site)

  11: puts '  Action: entered ' + test_site + ' in the address bar.'

  12: puts 'Step 2: Select Canada from the Top Stories drop-down list'

  13: $browser.select_list( :index , 1).select("Canada English")

  14: puts '  Action: selected Canada from the drop-down list.'

  15: puts 'Step 3: click the "Go" button'

  16: $browser.button(:caption, "Go").click

  17: puts '  Action: clicked the Go button.'

  18: puts 'Expected Result: '

  19: puts ' - The Google News Canada site should be displayed'

  20: puts 'Actual Result: Check that "Canada" appears on the page by using an assertion'

  21: assert($browser.text.include?("Canada") )

  22: puts '  '

  23: puts '## End of test: google news selection'

  24: end # end of test_googlenews

  25: def test_c_googleradio

Most examples of scripted automation appear as codified versions of a set of steps listed in a less-than-adequately designed manual test case using hard-coded arguments for variables, mindless progression between steps, and simple deterministic oracles. Scripted automation is probably most beneficial for automating specific sub-tasks in "computer assisted testing." However, scripted automation is usually too prescriptive, and rely heavily on nothing going wrong during the execution of the test case.

Procedural automation (procedural programming)

In procedural automation the testers also develops a test by writing a  series of computational steps to achieve a desired purpose. However unlike scripted automation the procedural automation paradigm generally provides better control flow options during the execution of the automated test case, allows for greater complexity in the design, improves reuse and reduces maintenance through modularity, and can employ both deterministic and heuristic oracles.

   1: // Procedural programming example

   2: static void Main(string[] args)

   3: {

   4:   string logResult = string.Empty;

   5:   // Path to the data file passed as a string argument to the test case

   6:   string pictTestData = args[0];

   7:   //Stopwatch to measure test case duration

   8:   Stopwatch sw = new Stopwatch();

   9:   sw.Start();

  10:   // Launch the AUT

  11:   AutomationElement desktop = AutomationElement.RootElement;

  12:   AutomationElement myAutForm = null;

  13:   Process myProc = new Process();

  14:   myProc.StartInfo.FileName = myConstantAutFileName;

  15:   if (myProc.Start())

  16:   {

  17:     // Polling loop to find AUT window by window property

  18:     int pollCount = 0;

  19:     do

  20:     {

  21:       myAutForm = desktop.FindFirst(TreeScope.Children,

  22:         new PropertyCondition(AutomationElement.AutomationIdProperty,

  23:           myConstantAUTPropertyID));

  24:       pollCount++;

  25:       System.Threading.Thread.Sleep(100);

  26:     }

  27:     while (myAutForm == null && pollCount < 50);

  28:  

  29:     if (myAutForm == null)

  30:     {

  31:       throw new Exception("Failed to find dialog");

  32:     }

  33:  

  34:     // Get UI element collection here...

  35:     // Call method to read in test data

  36:     string[] testData = ReadTabDelimitedFile(pictTestData);

  37:     // iterate through each set of test data (data-driven test example)

  38:     foreach (string test in testData)

  39:     {

  40:       // Call method to execute each set of test data and assign the return

  41:       // value to the logResult variable; Oracle is separate method called 

  42:       // from the test method

  43:       LogResultMethod(ExecuteCombinatorialTestMethod(test));

  44:     }

  45:  

  46:     // close AUT and clean-up

  47:     TimeSpan ts1 = sw.Elapsed;

  48:     // log test case duration...

  49:     // Deal with situation if AUT failed to launch

  50:   }

  51: }

Procedural automation can be used for anything from API to GUI automated test cases designed to evaluate functionality (computational logic), non-functional areas such as stress, performance, and security, and also behavioral  . Using a language similar to the programming language removes abstraction layers, and also enables other members of the team (developers) to easily review test cases.

Model based automation

Model based automation is a relatively new automation paradigm, and its complexity is beyond the scope of this single post. Basically, model based automation involves codifying abstracted state machines and state traversals and couples these parts with an automation engine that uses some form of graph traversal logic to drive the system under test between the various state machines identified in the model. In some sense model based automation is similar to exploratory testing because tests are generally not pre-determined or pre-scripted, what constitutes a single test is really hard to describe, and the oracles generally detect errant behavior (or being in an unexpected state). Personally, I think there is tremendous potential in model based automation, but the industry has just begun to scratch the surface of this automation paradigm and it is still largely misunderstood. This automation paradigm requires more complex skill sets of the person designing the test automation such as the ability to abstract important machine states as a model, and encode system behaviors. For more information about model based automation I recommend taking a look at http://research.microsoft.com/en-us/projects/specexplorer.

So, approach which is best?

In my opinion, there may be some limited value in record/playback, keyword, and scripted automation in specific contexts; however, a robust automated test case that will run on multiple environments, multiple languages, and distributed across multiple platforms without rewriting the test for each variation requires well designed tests developed using procedural automation or model based automation approach.

Originally Published Tuesday, April 14, 2009

Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.

Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.

For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!

To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.

In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below

   1: namespace TestingMentor.PInvokeSample

   2: {

   3:   using System;

   4:   using System.Runtime.InteropServices;

   5:  

   6:   /// <summary>

   7:   /// This class contains Native Win32 API functions that are marshalled

   8:   /// over for use in C#

   9:   /// </summary>

  10:   class NativeMethod

  11:   {

  12:     /// <summary>

  13:     /// Sets an item of information in the user override portion of the

  14:     /// current locale. This function does not set the system defaults.

  15:     /// </summary>

  16:     /// <param name="locale">the locale identifier of the locale with the

  17:     /// code page used </param>

  18:     /// <param name="localeType">Type of locale information to set.</param>

  19:     /// <param name="localeData">A null-terminated string containing the

  20:     /// locale information to set</param>

  21:     /// <returns>Returns true if successful; otherwise false</returns>

  22:     [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]

  23:     public static extern bool SetLocaleInfo(

  24:       uint locale,

  25:       uint localeType,

  26:       string localeData);

  27:  

  28:     /// <summary>

  29:     /// Sets an item of information in the user override portion of the

  30:     /// current locale. This function does not set the system defaults.

  31:     /// </summary>

  32:     /// <param name="locale">the locale identifier of the locale with the

  33:     /// code page used </param>

  34:     /// <param name="localeType">Type of locale information to set.</param>

  35:     /// <param name="localeData">An integer value representing the locale

  36:     /// information to set</param>

  37:     /// <returns>Returns true if successful; otherwise false</returns>

  38:     [DllImport("kernel32.dll", SetLastError = true)]

  39:     public static extern bool SetLocaleInfo(

  40:       uint locale,

  41:       uint localeType,

  42:       int localeData);

  43:   }

  44: }

The argument values that we can pass to these functions are enumerated in a separate class similar to the one below

   1: namespace TestingMentor.NlsInfo

   2: {

   3:   /// <summary>

   4:   /// Constant values for SetLocaleInfo API

   5:   /// </summary>

   6:   class NlsConstant

   7:   {

   8:     public enum Locale : uint

   9:     {

  10:       Invariant = 0x007F,

  11:       SystemDefault = 0x0800, // use system default for setlocaleinfo

  12:       UserDefault = 0x0400,

  13:       Neutral = 0x0000,

  14:       CustomDefault = 0X0C00, // Vista and later

  15:       CustomUiDefault = 0x1400, // Vista and later

  16:       CustomUnspecified = 0X1000  // Vista and later

  17:     };

  18:     

  19:     public enum LocaleType : uint

  20:     {

  21:       // VALUE LCDATA TYPES

  22:       CalendarType = 0x00001009,  // type of calendar specifier

  23:       CurrencyDigits = 0x00000019,  // local monetary fractional digits

  24:       CurrencySymbol = 0x0000001B,  // position of positive currency symbol

  25:       FractionalDigits = 0x00000011,  // number of fractional digits

  26:       NativeDigitSubstitution = 0x00001014,  // native digit substitution

  27:       FirstDayOfWeek = 0x0000100C,  // first day of week specifier

  28:       FirstWeekOfYear = 0x0000100D,  // first week of year specifier

  29:       LeadingZeros = 0x00000012,  // leading zeros for decimal

  30:       Measure = 0x0000000D,  // 0 = metric, 1 = US

  31:       NegativeCurrency = 0x0000001C,  // negative currency mode

  32:       NegativeNumber = 0x00001010,  // negative number mode

  33:       PaperSize = 0x0000100A,  // paper size

  34:       TimeFormat = 0x00000023,  // time format specifier

  35:       

  36:       // STRING LCDATA TYPES

  37:       // Valid Unicode characters

  38:       AM = 0X00000028,  // AM designator

  39:       PM = 0x00000029,  // PM designator

  40:       CurrencySymbol = 0x00000014,  // local monetary symbol

  41:       DecimalSeparator = 0x0000000E,  // decimal separator

  42:       DigitGrouping = 0x00000010,  // digit grouping

  43:       ListSeparator = 0x0000000C,  // list item separator

  44:       LongDate = 0x00000020,  // long date format string

  45:       MonetaryDecimalSeparator = 0x00000016,  // monetary decimal separator 

  46:       MonetaryGrouping = 0x00000018,  // monetary grouping 

  47:       MonetaryThousandSeparator = 0x00000017,  // monetary thousand separator

  48:       NativeDigits = 0x00000013,  // native ascii 0-9 

  49:       NegativeSign = 0x00000051,  // negative sign

  50:       PositiveSign = 0x00000050,  // positive sign

  51:       ShortDate = 0x0000001D,  // short date format string

  52:       ThousandSeparator = 0x0000000F,  // thousand separator

  53:       TimeSeparator = 0x0000001E,  // time separator

  54:       TimeFormat = 0x00001003,  // time format string

  55:       YearMonthFormat = 0x00001006   // year month format string

  56:     };

  57:     

  58:     public enum LocaleData : int

  59:     {

  60:       // LOCALE_ICALENDARTYPE VALUES

  61:       Gregorian = 1, // Gregorian (localized)

  62:       GregorianUS = 2, // Gregorian(Always English)

  63:       GregorianMEFrench = 9, // Middle East French

  64:       GregorianArabic = 10,

  65:       GregorianXlitEnglish = 11, // transliterated English

  66:       GregorianXlitFrench = 12, // transliterated French

  67:       Japan = 3,

  68:       Taiwan = 4,

  69:       Korea = 5,

  70:       Hijri = 6,

  71:       Thai = 7,

  72:       Hebrew = 8, 

  73:       Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later 

  74:       

  75:       // LOCALE_ICURRENCY

  76:       PositiveCurrencyPrefixNoSeparation = 0,

  77:       PositiveCurrencySuffixNoSeparation = 1, 

  78:       PositiveCurrencyPrefixSeparation = 2, // one character separation

  79:       PositiveCurrencySuffixSeparation = 3, // one character separation  

  80:       

  81:       // LOCALE_IDIGITSUBSTITUTION

  82:       DigitSubstitutionContextBased = 0,

  83:       DigitSubstitutionNone = 1, // use this setting for full unicode support

  84:       DigitSubstitutionNative = 2, // uses digits based on national conventions

  85:                                    // according to LOCALE_SNATIVEDIGITS

  86:       

  87:       //LOCALE_IFIRSTDAYOFWEEK 

  88:       Monday = 0, // LOCALE_SDAYNAME1 

  89:       Tuesday = 1, // LOCALE_SDAYNAME2

  90:       Wednesday = 2, // LOCALE_SDAYNAME3

  91:       Thursday = 3, // LOCALE_SDAYNAME4 

  92:       Friday = 4, // LOCALE_SDAYNAME5

  93:       Saturday = 5, // LOCALE_SDAYNAME6

  94:       Sunday = 6, // LOCALE_SDAYNAME7

  95:       

  96:       // LOCALE_IFRISTWEEKOFYEAR

  97:       FirstDay = 0, // Week containing 1/1 even if single day

  98:       FirstFullWeek = 1, // first full week following 1/1

  99:       FirstWeek = 2, // first week with at least 4 days after 1/1

 100:       

 101:       // LOCALE_ILZERO

 102:       NoLeadingZero = 0,  // .975 119:   

 103:       LeadingZero = 1,     // 0.975 

 104:       

 105:       // LOCALE_IMEASURE 

 106:       Metric = 0, 

 107:       US = 1, 

 108:       

 109:       // LOCALE_INEGCURR

 110:       ParenthesisSymbolNumber = 0,   // ($1.1)

 111:       NegativeSignSymbolNumber = 1,  // -$1.1 

 112:       SymbolNegativeSignNumber = 2,  // $-1.1

 113:       SymbolNumberNegativeSign = 3,  // $1.1- 

 114:       ParenthesisNumberSymbol = 4,   // (1.1$)

 115:       NegativeSignNumberSymbol = 5,  // -1.1$

 116:       NumberNegativeSignSymbol = 6,  // 1.1-$

 117:       NumberSymbolNegativeSign = 7,  // 1.1$-

 118:       NegativeSignNumberSpaceSymbol = 8,  // -1.1 $

 119:       NegativeSignSymbolSpaceNumber = 9,  // -$ 1.1

 120:       NumberSpaceSymbolNegativeSign = 10,  // 1.1 $- 

 121:       SymbolSpaceNumberNegativeSign = 11,  // $ 1.1-

 122:       SymbolSpaceNegativeSignNumber = 12,  // $ -1.1

 123:       NumberNegativeSignSpaceSymbol = 13,  // 1.1- $ 

 124:       ParenthesisSymbolSpaceNumber = 14,  // ($ 1.1) 

 125:       ParenthesisNumberSpaceSymbol = 15,  // (1.1 $) 

 126:       

 127:       // LOCALE_INEGNUMBER

 128:       Parenthesis = 0,  // (1) 

 129:       NegativeSignNumber = 1,  // -1 

 130:       NegativeSignSpaceNumber = 2, // - 1

 131:       NumberNegativeSign = 3,  // 1- 

 132:       NumberSpaceNegativeSign = 4,  // 1 - 

 133:       

 134:       // LOCALE_PAPERSIZE

 135:       USLetter = 1, 

 136:       USLegal = 5,

 137:       A3 = 8,

 138:       A4 = 9,

 139:      

 140:       // LOCALE_ITIME

 141:       FormatAM_PM = 0,

 142:       Format24Hour = 1

 143:     };

 144:   } 

 145: }

You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-­­dd. The automated oracle verified the year, month and day  values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.

I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals.  Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and  keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.

Originally Published Thursday, March 12, 2009

There are many approaches to test automation from unit testing to system level testing through the GUI. Of course, the most often discussed approach is the automation approach that drives the GUI to perform some action; or GUI automation. This also happens to be the most controversial approach to test automation, and is perhaps the hardest type of automation to design and develop. One reason why an automated GUI test fails or doesn’t achieve its potential is due to a lack of understanding of the "system" by the tester, which in turn leads to a poorly designed test from the outset.

This problem is especially obvious when people who may have specialized business knowledge but lack a in-depth understanding of the systems they are working on (non-technical testers) are asked to ‘automate’ something. The automation in this case is usually in the form of using record/playback tools or perhaps creating a rote script to drive a keyword-driven framework, and the ‘test’ is usually consists of nothing more than merely mimicking some contrived behavior by the ‘tester.’ In fact, this over-simplified view of test automation is sometimes perpetuated by tool vendors. A manager at one tool vendor said, "By automatically capturing the tester’s process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating."

There are 2 fundamental problems with the above quote, and this approach to ‘automation’ (and I use that term loosely in this context). First, while I don’t totally discount the value of record/playback, and in the right context it could very well be the best approach in a specific situation, the general consensus in the industry is that due to the limited capabilities of record/playback type tools this type of automation is simply one level above using a hoard of monkeys trained to repeat a set of sequences using the keyboard and mouse. In fact, some might suggest trained monkeys may be a better alternative because bananas are much cheaper than the costs of licensing a tool and then realizing that you have to hire someone to try to patch together some proprietary script in an often vain attempt to build an automation test suite that is beyond the reasonable capabilities of a record/playback automation approach. Unfortunately, in either case the organization is usually left with a horrible mess that nobody wants to clean up. Secondly, it makes a ridiculous assumption that testers are too stupid to automate complex scenarios or test automation is a brain-dead, non-interesting, zero-value-add activity.

If simply recording or ‘documenting’ rudimentary scripts that essentially repeat a sequence of ‘hard-coded’ steps over and over again is someone’s idea of well-designed test automation then I would agree that automation is a brain-dead activity. When you automate poorly designed tests, you simply get poorly designed automated tests! And, since I don’t mind calling the kettle black, I will say it…recording a set of actions, or documenting a sequence of steps with hard-coded values to feed into a keyword driven architecture is not test automation! Surely it automates tasks, but well designed automated tests are much more powerful than the production of some crude script that automates the actions performed by someone sitting in front of a computer.

Of course, well designed automated tests require highly skilled professional testers with in-depth knowledge of the systems they are working on, as well as proficiency in programming concepts and languages. Similar to how doctors study anatomy, physiology, pharmacology, immunology, biochemistry, etc., professional testers need to constantly study the various systems they are tasked to test. Developing a well designed automated test is very different than simply using a tool to automatically repeat a sequence of actions. Designing robust tests (automated or not) requires not only incredible creativity and problem-solving skills, but an in-depth knowledge of the system and an understanding of how to manipulate the system programmatically.

But, if you buy into the idea that automation is simple and merely rote recording or documenting some sequence of steps performed by some person then you get exactly that; simplistic repetitive automated actions. Simple automation is simply automated simplicity.

Originally Published Sunday, January 04, 2009

I am generally not a big fan of static data in test automation, but being a pragmatic person, I know there are clearly times when using data-driven testing is just plain common-sense. For example, data-driven testing is an effective automation approach when designing ‘black-box’ tests for testing an API.

Data-driven testing is a common approach to test automation where static test data is passed to application parameters and the expected result (which is usually also read from static data) is compared against the actual result. This automation approach is effective when the actual result compared to some expected result can be resolved as a Boolean outcome. In other words, if the actual and expected results match the outcome is true and the test passes; otherwise the outcome is false and the test fails. (Of course, if something occurs during the test where there is no actual result then that particular test is usually logged as indeterminate.)

Of course, the key to effective data-driven testing is the data! If we don’t identify the most appropriate data to use in the test then the test case may have holes and we might overlook important information or miss critical anomalies. If we have too much redundant data then we may be simply running unnecessary tests (yes, even with test automation redundant testing is not an efficient use of resources).

Let’s say we had to test an API method such as:

public bool IsValidNetBiosName(string name)

where the return value is true if the string argument passed to the name parameter is a valid NetBIOS name on the Windows operating environment; otherwise it returns false.

With a data-driven testing approach we could use a simple CSV file that contained the string arguments and the expected result for each string passed to the name parameter. A partial sample of the CSV data file would be:

a,true
validname,true
validnamexxxxxx,true
invalidnamexxxxx,false
invalidnamexxxxxxxxxxxxxxxxxxxxxxx,false
,false
null,false,
xx\x,false
xx/xx,false
x:x,false
xxx*xx,false
x?xx,false
xxxx",false
;,false
xxx|xxx,false

(NOTE: null is a special case in which we need to convert the string “null” to a null in the test code, and the test above null is an empty string. An empty string and null are two different things and both must be tested in this case.)

Next, we need to read in the CSV file into our automated test, and perhaps the easiest way I found to read in a text or CSV file in C# is with the File.ReadAllLines method. The ReadAllLines method opens a text file, reads each line of text as an element in a string array, and then closes the file. Once we have a array of all lines in our data file, we simply need to parse each element in the string array into test data and/or expected result, and then compare the actual result against the expected result as illustrated in this example.

   1: // Read each line in the entire CSV file into a string array

   2: string[] testDataArray = System.IO.File.ReadAllLines("myTestData.csv");

   3:  

   4: // Iterate through each element in the test data file

   5: foreach(string test in testDataArray)

   6: {

   7:     // Split each element in each line into an array where the elements are the

   8:     // test data and the expected result

   9:     string[] testElement = test.Split(',');

  10:     string testData = testElement[0];

  11:     string expectedResult = testElement[1];

  12:     

  13:     // Special case for passing a null to the API parameter

  14:     if (string.Equals(testData, "null", 

  15:         StringComparison.OrdinalIgnoreCase))

  16:     {

  17:         if (string.Equals(api.IsValidNetBiosName(null).ToString(), expectedResult, 

  18:             StringComparison.OrdinalIgnoreCase))

  19:         {

  20:             result = "Pass";

  21:         }        

  22:         else

  23:        {

  24:             result = "Fail";

  25:        }

  26:     }

  27:  

  28:      // Compare the return value against the expected result

  29:     else if(string.Equals(api.IsValidNetBiosName(testData).ToString(), expectedResult,

  30:         StringComparison.OrdinalIgnoreCase))

  31:     {

  32:         result = "Pass";

  33:     }

  34:     else

  35:     {

  36:         result = "Fail";

  37:     }

  38: }

This is rather simple example, but data-driven testing is effective for unit testing, API testing, and can even be used in automated GUI testing (although data-driven automation may only have limited applicability in GUI automation). I am a firm believer in the KISS principle when it comes to developing automated tests, and the ReadAllLines method is perhaps the easiest and most efficient way to read in data file for data-driven development. Of course, data-driven testing doesn’t solve all problems. Chan Chaiyochlarb has a good post on some pitfalls to watch out for. But, in the right context, data-driven testing can be one approach used in automated testing.

Originally Published Tuesday, December 02, 2008

There are occasionally times during an automated test needs to create a temporary file during the execution of that test. The problem is that often this file is left behind on the system, or even worse stored in some obscure directory on a server. I say worse because those files will be discovered by someone approximately 9.25 months from the time they were created, and that person will spend about 6.5 hours trying to figure out who they belong to before realizing they are no longer needed and can blow them away.

The test designer must decide whether or not to leave test artifacts once created. My general rule of thumb is if the test run is a  system level (or dirty room) test then it is probably ok to leave any artifacts from previous tests lying about on the local machine. But, if the test run is a integration or component level (or primarily clean room) test, then in order to preserve the clean room environment (in other words, I want to eliminate or control the number of unknown environmental variables) then I probably want to remove any test artifacts that are created during the execution of a test.

There are many ways to create a file and delete a file in an automated test. But, one of the easiest ways is to use File class members in C# to create a file that will automatically delete itself when the automated test ends. The FileOptions enumeration in File.Create method provides additional options for FileStream objects including a member that will automatically delete the file when the thread that created it closes.

FileStream fs = File.Create(path, bufferSize, FileOptions.DeleteOnClose)

Of course if you wanted to design in the option to delete the file at the end of a test based on some Boolean decision you could simply create the file using WriteAllText(), WriteAllLines() or WriteAllBytes() methods.

Another problem with temporary test files is that the test designer frequently hard-codes a clever name such at “test.txt” to name the temporary file in the test case itself. Then of course, the next test fires off and that test also has a hard-coded file name which also happens to be “text.txt” that will either overwrite the first file, or in the worse case append to it. Again, the tester can easily create a random string generator to generate random strings for use as file names, or use another method that is already built into C#.

Another useful method for creating temporary file names is found in the Path class. The Path class member Path.GetRandomFileName() will generate a random file name. This random file name also includes a random extension which may not be desired, so you can use the Path.ChangeExtension method to automatically generate a randomly named file and change the extension to the desired extension.

string filename = Path.ChangeExtension(GetRandomFileNamePath.GetRandomFileName(), "txt");

Now, we can create a temporary test file at a desired location (say the My Documents folder) on the local machine with a random name and that file will be automatically deleted at the end of the test.

FileStream fs = File.Create(Path.Combine(

    Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments),

    Path.ChangeExtension(Path.GetRandomFileName(), "txt")),

    bufferSize, FileOptions.DeleteOnClose);

Originally Published Tuesday, September 09, 2008

Templates can be useful tools to help increase individual efficiency and team consistency. Yet, I am sometimes quite surprised that some organizations lack standard templates for such daily routines such as bug reporting, status reporting, and even test cases. There are some in the industry who shun templates because they claim they are too restrictive and the person using the template is ‘forced’ to do things one way and not encouraged to think beyond the scope of the template. Of course, any person who has matured to the point of being able to think rationally for themselves and has the cognitive ability to for logical reasoning (as opposed to those who base decisions on pure emotion and volitional impulse) realize this narrow-minded viewpoint is pure nonsense!

A template is simply a model or reference that provides a starting point. Frankly, if a person does not understand that a template simply provides a starting framework from which he must rely on his knowledge, skill, and experience to customize to fit the specific context, and how to use a template to help him accomplish a task more efficiently then I would recommend that person not be allowed beyond the reach of their mother’s apron strings because they are probably a danger to themselves and people around them and they are incapable of rational constructive thinking.

An example of where templates have added value for me is in the test development process. When developing test cases using Visual Studio I frequently find myself adding the references and using directives repeatedly to each new test case. For example, when developing a UI automated test case I need to include references for Windows.Forms, UIAutomationClient, UIAutomationType, UIAutomationClientsideProviders, and UIAutomationProvider. That’s a lot of clicking for each new test case.

Creating a template in Visual Studio is easy and straight forward. In this case, I started with a standard console application project. Next, I added the following references to the project:

• System.Windows.Forms
• UIAutomationClient
• UIAutomationClientsideProviders
• UIAutomationProvider
• UIAutomationTypes

Then I customize the template as illustrated in the code below by adding common namespace directives, including XML comments, and commonly used code snippets.

   1: using System;

   2: using System.IO;                       // For file I/O operations

   3: using System.Text;                     // For StringBuilder and encoder/decoder

   4: using System.Diagnostics;              // Process, stopwatch, trace/debug 

   5: using System.Globalization;            // For Unicode support

   6: using System.Windows.Forms;            // For sendkeys and clipboard

   7: using System.Windows.Automation;       // For UIAutomation

   8:  

   9: namespace CompanyName.ProjectName.FeatureArea

  10: {

  11:     class TestCase

  12:     {

  13:         /// <summary>TEST PURPOSE: (specific functional capability or end-2-end user scenario test is verifying)</summary>

  14:         /// <status>TEST STATUS: (Active, NotInMilestone, Obsolete)</status>

  15:         /// <category>TEST TYPE: (category or test suite bucket such as BVT/BAT, Regression, etc.)</category>

  16:         /// <priority>TEST PRIORITY: (importance of test in relation to critical functionality or requirements)</priority>

  17:         /// <time>TEST TIME: (approximate duration of test)</time>

  18:         /// <steps>TEST STEPS: (outline high level steps or procedures)</steps>

  19:         /// <expected>EXPECTED RESULTS: (specific verification criteria for oracle to determine pass, fail, or indeterminate)</expected>

  20:         /// <dependencies>CONFIGURATION: (required expected settings or dependencies)</dependencies>

  21:         /// <remarks>OTHER INFO:</remarks>

  22:  

  23:         [STAThread]

  24:         static void Main(string[] args)

  25:         {

  26:             // Time the duration of test in order to better predict time to

  27:             // run each test suite

  28:             Stopwatch TestTimer = new Stopwatch();

  29:             TestTimer.Start();

  30:  

  31:             // TODO: Design and develop the code to achieve the specific purpose

  32:             //       of the test. 

  33:  

  34:             // Don't forget to add comments to the code!

  35:  

  36:             // Don't forget to clean up or restore the test bed to pre-test

  37:             // conditions in cases other than end-to-end user scenarios.

  38:  

  39:             TestTimer.Stop();

  40:             // Don't forget to log or record test time

  41:         }

  42:     }

  43: }

I also like to include valuable information about the test case using XML comments to jog my memory or provide some background for anyone who views the code. For example it is a best practice to clearly define the specific purpose of each test so people don’t have to guess what the test is proving/disproving (or at least what it is supposed to be doing). (If someone doesn’t have a clearly defined purpose and goal of what the automated test is going to prove or disprove and they are simply writing automated test just to automate tests than that person is probably a mindless dolt!) The information that adds value to the organization may be subjective, so determine what information is important for the consumers of the completed template. Essentially, in this case, the comments captured in each test should provide value to the management and maintainability of the test and the testing process. If a template tries to collect too much information that is not being consumed or is not important to reduce long term costs of maintenance then that information should be suspect and is probably not adding value to the overall team. A big advantage of Visual Studio and C# is that it allows comments and important information to be easily captured via XML tags documented in a XML file at build time so I don’t have to worry about performing a separate task to document each automated test.

The STAThread attribute is included above the Main method to support the Clipboard class members. Without this attribute set the test will throw an un-handled exception. Finally, I put in any code that I commonly reuse. That doesn’t mean that I have to use it…I can easily remove it if I don’t need it. (That’s the beauty of being able to think logically for one’s self!) But, if it is code that is commonly used then I don’t have to retype it (meaning…it increases my efficiency even by a little bit).

To complete the process I simply select the File –> Export Template… menu item and complete the Export Template Wizards.

I can create several custom templates depending on the context of each type of test. For example, if the automated test case is not going to drive the UI then I don’t need all those additional references in my template. Rather than remove them each time, I can simply create another template for non-UI automated test cases. Now, each time I start a new project I can simply select the appropriate template. The templates are saved in a ZIP file in the Visual Studio\My Exported Templates directory, and those ZIP files can be sent around to other members of the team.

Also, here are some important key concepts for creating standard templates (in general):

• Don’t screw the majority just to make the minority happy! Only include key elements that provide value to the majority of consumers of the template the majority of the time. A common problem with many templates is that they try to include everything for everybody. When templates are over-laden with things that only a handful of people use, or if people have to spend too much time trimming out things the majority of people don’t use then the templates are a bottleneck. Remember, templates can be customized by those few who might need a few additional things occasionally; majority rules!
• Don’t over-document useless tidbits. Notes and additional comments must be useful, but brief. If someone has to spend more time learning how to fill out a template than it takes to fill out the template then that template provides no value in terms of making the process or the team more efficient. If the template is so complicated that it requires a user manual…then it is not a useful template.
• Intuition is important. If the template is not easy to use, if the consumer of the template has to search around for items, or even worse, forces the user to use parts of the template because they are on the template, then the template is not intuitive and using the template becomes a chore rather than a tool to improve efficiency.
• Get buy in! Don’t develop templates in isolation. Talk to various consumers and users of the templates to better define the template and what is captured in the template. For example, if you are developing a template for reporting anomalies in software then developers should be included in that conversation as should program managers and any other consumers of the completed templates.
• Templates are not commandments! Templates provide a starting point; they are not engraved in stone. Templates should be designed to include things that are commonly reused by various users and consumers of the completed template. But, sometimes they can include things that are unnecessary and do not add value in a specific context and in those cases the user of the template should be able to remove those parts of the template or at least ignore them. Conversely, other times additional things may be necessary in a template and the user of the template should be able to easily add those things that provide important value to the purpose of the completed template.

Bottom line, templates are tools to provide a common reference, or central starting point…they are not hand-cuffs designed to constrain rational and logical thought or constrict a person’s creativity. In fact, good templates are tools that enable and support more efficient creativity!

Originally Published Wednesday, July 02, 2008

Occasionally, the execution or outcome of a test depends on the operating system version on which the test is executed. Platform profiling is important because subtle differences in operating system platforms can affect how certain tasks are carried out, inclusion or exclusion of specific features or capabilities, work-arounds, or even changes in deterministic or declarative oracles that determines pass, fail, or indeterminate results. For example, accessing certain features on the Vista platform will trigger the User Account Control (UAC) dialog, whereas this feature does not exist on previous versions of Windows.

It is easy for a tester to select the appropriate OS version when executing tests manually and determine the correct steps or settings and determine the correct results of a test based on their cognitive expectations for an outcome, But, in an automated test system where tests are distributed across the wire to various machine configurations and operating system platforms the test designer must programmatically detect the operating system version in automated tests that need to be profiled for the appropriate platform. Programmatic platform profiling negates the need to write several different tests for a single test case that is dependent on variations in the operating system versions. Detecting the operating system version at runtime allows the automation test designer to handle subtle differences between the operating systems within one one test case.

Detecting the major and minor OS versions with C# is relatively straight forward using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members.

The System.PlatformID enumeration detects the major operating platforms including:

• Win32Windows – versions of Windows 9x, including Windows ME
• Win32NT – versions of Windows 2000, Xp, and Vista
• WinCE – versions of Windows CE
• Unix – versions of the Unix OS

Once the platform is determined, the test designer can drill down further using the Environment.OSVersion.Version.Major and Environment.OSVerions.Version.Minor members to find the specific major and minor operating system version number. If tests are dependent on specific revisions or service packs of the major operating system then the integer or string values can be obtained using the OSVersion.Version.Revision or OSVersion.Version.Build members to determine  specific versions of an operating system. Some versions of Windows used alphabetic characters in the revision number, so in those cases the revision should be converted to and compared using a string object as represented by the constant values for Windows 95 retail, Windows 95 OSR 2.1 and Windows 95 OSR 2.5 and Windows 98 Second Edition. Also, the OperatingSystem.ServicePack property can be used to determine if, and what service pack has been installed on the operating system.

   1: class OSVersionInfo

   2: {

   3:   /// <summary>

   4:   /// Windows NT Kernel Major Version Numbers

   5:   /// </summary>

   6:   enum Win32NTMajorKernelVersion 

   7:   {

   8:     NT4 = 4,

   9:     NT5 = 5,

  10:     NT6 = 6

  11:   }

  12:  

  13:   /// <summary>

  14:   /// Windows NT Kernel Minor Version Numbers

  15:   /// </summary>

  16:   enum Win32NTMinorKernelVersion

  17:   {

  18:     Win2000 = 0,

  19:     WinXp = 1,

  20:     Win2003Server = 2,

  21:     Vista = 0,

  22:     Win7 = 1

  23:   }

  24:  

  25:   /// <summary>

  26:   /// Windows 9x Kernel Minor Version Numbers

  27:   /// </summary>

  28:   enum Win9xMinorKernelVersion

  29:   {

  30:     Win95 = 0,

  31:     Win95Osr2 = 3,

  32:     Win98 = 10,

  33:     WinME = 90

  34:   }

  35:   

  36:   // Service Pack identification strings

  37:   private const string VistaSp1  = "Service Pack 1";

  38:   private const string Win95Sp1     = "950A";

  39:   private const string Win95Osr2   = "950B";

  40:   private const string Win95Osr25   = "950C";

  41:   private const string Win98SE       = "2222A";

  42:  

  43:   /// <summary>

  44:   /// Detects the current version of the operating system useful for

  45:   /// controlling execution branches in automated test cases that are

  46:   /// platform dependent

  47:   /// </summary>

  48:   /// <returns>A string representing the OS version</returns>

  49:   private static string GetOSVersion()

  50:   {

  51:     OperatingSystem osVersionInfo = Environment.OSVersion;

  52:     string osVersion = string.Empty;

  53:     switch (osVersionInfo.Platform)

  54:     {

  55:       case PlatformID.Win32NT:

  56:         switch (osVersionInfo.Version.Major)

  57:         {

  58:           case (int)Win32NTMajorKernelVersion.NT4:

  59:             osVersion = "NT4";

  60:             break;

  61:           case (int)Win32NTMajorKernelVersion.NT5:

  62:             switch (osVersionInfo.Version.Minor)

  63:             {

  64:               case (int)Win32NTMinorKernelVersion.Win2000:

  65:                 osVersion = "Win2K";

  66:                 break;

  67:               case (int)Win32NTMinorKernelVersion.WinXp:

  68:                 osVersion = "WinXp";

  69:                 break;

  70:               case (int)Win32NTMinorKernelVersion.Win2003Server:

  71:                 osVersion = "Win2003Server";

  72:                 break;

  73:             }

  74:  

  75:             break;

  76:           case (int)Win32NTMajorKernelVersion.NT6:

  77:             switch (osVersionInfo.Version.Minor)

  78:             {

  79:               case (int)Win32NTMinorKernelVersion.Vista:

  80:                 if (osVersionInfo.ServicePack == VistaSp1)

  81:                 {

  82:                   osVersion = "Vista SP1";

  83:                 }

  84:                 else

  85:                 {

  86:                   // Windows Vista or WinServer2008

  87:                   osVersion = "Vista";

  88:                 }

  89:  

  90:                 break;

  91:               case (int)Win32NTMinorKernelVersion.Win7:

  92:                 osVersion = "Windows 7";

  93:                 break;

  94:             }

  95:  

  96:             break;

  97:         }

  98:  

  99:         break;

 100:       // Windows 9x Versions

 101:       case PlatformID.Win32Windows:

 102:         switch (osVersionInfo.Version.Minor)

 103:         {

 104:           // Windows 95, OSR 1.0, OSR 2.0

 105:           case (int)Win9xMinorKernelVersion.Win95:

 106:             if (osVersionInfo.Version.Revision.ToString() == Win95Sp1)

 107:             {

 108:               osVersion = "Windows 95 SP1";

 109:             }

 110:             else if (osVersionInfo.Version.Revision.ToString() == Win95Osr2)

 111:             {

 112:               osVersion = "Windows 95 OSR 2.0";

 113:             }

 114:             else

 115:             {

 116:               osVersion = "Win95";

 117:             }

 118:             break;

 119:           // Windows 95 OSR 2.1, OSR 2.5

 120:           case (int)Win9xMinorKernelVersion.Win95Osr2:

 121:             if (osVersionInfo.Version.Revision.ToString() == Win95Osr2)

 122:             {

 123:               osVersion = "Win95_OSR2.1";

 124:             }

 125:             else if (osVersionInfo.Version.Revision.ToString() == Win95Osr25)

 126:             {

 127:               osVersion = "Win95_OSR2.5";

 128:             }

 129:             break;

 130:           // Windows 98 and Windows 98 Second Edition

 131:           case (int)Win9xMinorKernelVersion.Win98:

 132:             if (osVersionInfo.Version.Revision.ToString() == Win98SE)

 133:             {

 134:               osVersion = "Win98SE";

 135:             }

 136:             else

 137:             {

 138:               osVersion = "Win98";

 139:             }

 140:             break;

 141:           // Windows ME

 142:           case (int)Win9xMinorKernelVersion.WinME:

 143:             osVersion = "WinME";

 144:             break;

 145:         }

 146:  

 147:         break;

 148:     }

 149:  

 150:     return osVersion;

 151:   } 

 152: }

 153:  

Currently, C# does not have a property or method to determine the specific edition of an operating system. If a test is dependent on a specific edition of Windows 2000, Windows Xp, or Windows 2003 operating system edition then we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function.

Programmatically detecting the operating system version at runtime enables the test designer to design one test that can execute on a different operating system versions by allowing for specific control flow of a test based on the operating system the test is running. Programmatic platform profiling also negates the need to rewrite the same fundamental test over and over again to handle differences in behavior or expectations of operating system versions. It can also reduce some long term costs of test case maintenance, give the test designer greater control over the execution of an automated test, and increases the reusability and the maintainability of a test case that must be ran on multiple operating system versions.

Quite simply, programmatic platform profiling is a best practice in software test automation when a given test is dependent on the profile of the platform on which it is executing!