Also this week the newest edition of the Automated Software Testing magazine published an article I wrote entitled UI Automation Beneath the Presentation Layer. It discusses some tips on how to use reflection for automating applications developed using managed code. If you have questions or comments about my article please let me know. Your feedback is appreciated.

I have always respected the work of Dion Johnson, and I was quite honored when he asked me to write an article for the magazine. The magazine and the Automated Testing Institute website provide a wonderful resource to people interested in software test automation. I really look forward to working with Dion and the Automated Testing Institute more closely in the future to provide whatever help I can to this monumental task he has undertaken. I also hope there are others out there who will share some of their ideas as well. I think the days of Not Invented Here (NIH) and I’ve Got A Secret (IGAS) syndrome are becoming issues of the past. We still have too much reinvention and not enough reuse of test code. I think that we have well established there is no single approach to testing that is effective in all contexts. We might approach problems from different perspectives, but I also think that all testers are passionate and have very similar goals of personal and professional improvement. To mature our discipline, and increase our effectiveness in our roles we should collaborate more and pool our resources, and the Automated Testing Institute is a growing community of professionals.

Technorati Tags: Automation,Reflection
Windows Live Tags: Automation,Reflection
WordPress Tags: Automation,Reflection

Many people might also say that globalization testers also need to know that different locales (places) around the world use different formats for date and time (national conventions). For example, in the United States the default long date format is Thursday, June 03, 2010 but in Germany it is Donnerstag, 3. Juni 2010. A tester doesn’t have to ‘read’ German to see the abstract date format has changed from dddd, MMMM dd, yyyy to dddd, d. MMMM yyyy.

Testing for support of these different national conventions used around the world is referred to as basic international sufficiency testing. I suspect the reason why some people might assume basic international sufficiency testing these different national conventions is the domain of the globalization tester is because the national conventions are set by default on the different localized versions of a software product so that’s when they are tested. But, this reasoning is absurd!

First, not all products are “localized” into all languages or ‘locales.” So, who tests the Canadian long date format of MMMM-dd-yy, or the Georgian (Georgia) long date format of yyyy ‘წლის’ dd MM, dddd? Also, Vista and later versions of Windows allow the user to ‘customize’ the date and time “format pictures” to use different separator symbols and orderings.

Secondly, way too many bugs such as hard-coded date formats are found way too late in the testing cycle (because localized versions tend to lag US English language version). And of course, we all know the cost of finding bugs later in the lifecycle are more costly to correct.

So, we must ask if there is a way for basic international sufficiency testing to be ‘pushed upstream?’ And of course the answer is yes. The easiest way is to host a “globalization bug bash” early in the cycle. (A “bug bash” is a day where testers are given some basic training on attack patterns, fault models, etc., in a general focus area and then spend a day exploring different areas of the product trying to flush out bugs in a competition style format.) Another way is to assign each tester a different locale (preferably one that is not associated with a localized language version) and have them set their test and self-host environments to that locale during their testing.

This is easily accomplished on Windows test environments by having testers launch the Regional and Language control panel applet (the short cut is Start –> Run, then type “intl.cpl” without the quotes, and press the OK button).

This just tests for a basic level of international sufficiency, and any good tester would want to explore their project’s capability to support the more than 150 different locale national conventions at a deeper level. This is especially true if your product is going to be used by customers around the world (including Canada). But, of course, we don’t want to run the same tests on all 150+ locales supported by the operating system.

The national convention settings for a particular locale are stored in a data type called the LCID, and when we change our locale (Format on the latest Regional and Language control panel applet) through the user interface we are actually calling various National Language Support (NLS) APIs. A “world-wide” application should use the universal NLS APIs and data available via the operating system.

One way to test our application’s ability to correctly use the national convention data supplied by the operating system is to set customized conventions. For example, did you know the Windows 7 operating system allows a digit grouping symbol to be a string of up to 3 characters? Or the Negative sign symbol can be a string of up to 4 characters.

Although having testers change their default locale (Format) on their test environment and self-host machines is a good first step in basic international sufficiency testing, we also want to see if our application can process a negative value of “!NEG7” instead of just “–7,” and any textboxes correctly display the customized negative sign symbol (especially at the upper extreme boundary of the textbox size property.

To customize the national convention settings we simply click the Advanced settings… button on the Formats property sheet of the Region and Language control panel applet which instantiates a new dialog with 4 property sheets for Numbers, Currency, Time, and Date.

Solution for Test Automation

That’s all well and fine for basic testing, or testing a “few” customized values, but if we wanted to test the permutations for each convention, or the combination of different conventions on numbers, currency, time, or date formats the number of tests is astronomical. Typically, testers writing an automated test would try to navigate the user interface of the Regional and Language control panel applet and the Customize Format property sheets in order to set custom conventions.

In the past I provided some code snippets for changing the convention settings on the Customize Format property sheets on versions of Windows pre-Vista. Earlier this year I also provided code snippets for customizing the date format picture and the time format picture.

That’s all well and good, but I recently released a new test automation library called GlobalTester for test developers to use in their automated test scripts. The GlobalTester library provides testers methods to set custom national conventions for the current user without having to navigate the user interface of the Region and Language options control panel applet. These national conventions include number formats, currency formats, date formats, time formats, and also current location.

The following example illustrates how we might design a test script to customize the date format for a test and reset the date format to its original setting (restoring the test environment to pre-test conditions). (Usage documentation for the GlobalTester library is on the Testing Mentor website.)

   1: namespace CustomizeDateSettingsExampleScript

   2: {

   3:   using System;

   4:   using System.Globalization;

   5:   using TestingMentor.TestTool.GlobalTester;

   6:  

   7:   class MyTest

   8:   {

   9:     static void Main()

  10:     {

  11:       try

  12:       {

  13:         CustomDateFormat time = new CustomDateFormat();

  14:  

  15:         string defaultLongDateFormat = 

  16:           CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern;

  17:         string newLongDateFormat = "MMM - d (yyyy) gg";

  18:  

  19:         if (time.ChangeLongDateFormat(newLongDateFormat))

  20:         {

  21:           // Launch AUT

  22:           // Execute test - (e.g. AUT implements long date string)

  23:           // Oracle - (e.g. compare long date format against customized pattern)

  24:  

  25:           // Reset test platform to original configuration

  26:           time.ChangeLongDateFormat(defaultLongDateFormat);

  27:         }

  28:         else

  29:         {

  30:           // Date format not changed; test not executed (e.g. invalid 

  31:           // day, month, year, and era format pictures)

  32:         }

  33:       }

  34:  

  35:       catch(ArgumentOutOfRange e)

  36:       {

  37:         // Test script failure - (e.g. long date format string argument out of range)

  38:       }

  39:  

  40:       finally

  41:       { 

  42:         // Log test results

  43:       }

  44:     } 

  45:   }

  46: }

Time and date information is commonly pulled from the operating system by many developers for use in headers or footers on documents, default file names, printing, and other places time/date stamps are useful or important. To ensure our products are “world-ready” we should modify the formats to validate whether our product supports various national conventions used in different regions (locales) around the world. In the previous post I illustrated how to programmatically customize the date formats on a Windows environment for including some basic globalization tests in your test automation. This week let’s look at how we can programmatically change both the short time and long time formats.

We will again need the 2 Win32 API functions SetLocaleInfo() and PostMessage() that we marshaled over into the NativeMethods class. Since that code doesn’t change I won’t repeat it here you can simply refer to the code snippet in the previous post. In this situation we need to set the lcType in SetLocaleInfo() to the LOCALE_STIMEFORMAT constant. Then we can pass a null-terminated string to the lcData variable in the SetLocaleInfo() function. MSDN explains “The maximum number of characters allowed for this string is 80, including a terminating null character. The string can consist of a combination of hour, minute, and second format pictures.”

Once again, to simplify that a bit I wrote some more wrapper methods to change the time format. Also, since we will be calling SetLocaleInfo()  and PostMessage() a lot for customizing date, time, and other national conventions I created a wrapper method called UpdateLocaleInformation() to remove redundancy.

   1: namespace TestingMentor.TestTool.GlobalTester

   2: {

   3:   using System;

   4:  

   5:   public enum TimeFormatType

   6:   {

   7:     LongTimeFormat = 0x00001003,

   8:     ShortTimeFormat = 0x00000079

   9:   }

  10:  

  11:   public class CustomTimeFormat

  12:   {

  13:     private int timeFormatType = (int)TimeFormatType.ShortTimeFormat;

  14:     private string timeFormatPicture = string.Empty;

  15:  

  16:     public int SetTimeFormatType

  17:     {

  18:       set

  19:       {

  20:         if (value == (int)TimeFormatType.ShortTimeFormat ||

  21:           value == (int)TimeFormatType.LongTimeFormat)

  22:         {

  23:           this.timeFormatType = value;

  24:         }

  25:         else

  26:         {

  27:           throw new ArgumentOutOfRangeException("TimeFormatType invalid");

  28:         }

  29:       }

  30:     }

  31:   

  32:     public string SetTimeFormatPicture

  33:     {

  34:       set { this.timeFormatPicture = value; }

  35:     }

  36:  

  37:     public bool ChangeTimeFormat()

  38:     {

  39:       return UpdateLocaleInformation(

  40:         this.timeFormatType,

  41:         this.timeFormatPicture);

  42:     }

  43:  

  44:     private bool UpdateLocaleInformation(int localeType, string localeData)

  45:     {

  46:       bool success = false;

  47:       if (NativeMethods.SetLocaleInfo(

  48:         NativeMethods.SystemDefaultLocale,

  49:         localeType,

  50:         localeData))

  51:       {

  52:         NativeMethods.PostMessage(

  53:           NativeMethods.BroadcastMessage,

  54:           NativeMethods.SettingChangeMessage,

  55:           IntPtr.Zero,

  56:           IntPtr.Zero);

  57:         success = true;

  58:       }

  59:  

  60:       return success;

  61:     }

  62:   }

  63: }

Once again, we simply have to set the SetTimeFormatType property to either the Short time or Long time format, provide the format picture by setting the SetTimeFormatPicture property, and then call ChangeTimeFormat(). The sample below illustrates how to change the short time format with different time separators and a reverse order.

   1: static void Main(string[] args)

   2: {

   3:   CustomTimeFormat time = new CustomTimeFormat();

   4:   time.SetTimeFormatType = (int)TimeFormatType.ShortTimeFormat;

   5:   time.SetTimeFormatPicture = "ss'mm,hh - tt";

   6:   if (time.ChangeTimeFormat())

   7:   {

   8:     Console.WriteLine("Success");

   9:   }

  10: }

Now, we can also customize the AM/PM designator as well. To change the AM/PM designator we need to add a few more properties and another wrapper method. In this case, I’ve added the SetAmPmDesignator property, the SetAmPmString property, and the ChangeAmPmDesignator() method.

   1: public enum AmPmDesignator

   2: {

   3:   AM = 0x00000028,

   4:   PM = 0x00000029

   5: }

   6:  

   7: public class CustomTimeFormat

   8: {

   9:   public int SetAmPmDesignator

  10:   {

  11:     set

  12:     {

  13:       if (value == (int)AmPmDesignator.AM || value == (int)AmPmDesignator.PM)

  14:       {

  15:         this.designatorForAmPm = value;

  16:       }

  17:       else

  18:       {

  19:         throw new ArgumentOutOfRangeException("AmPmDesignator invalid.");

  20:       }

  21:     }

  22:   }

  23: }

  24: public string SetAmPmString

  25: {

  26:   set { this.timeDesignator = value; }

  27: }

  28:  

  29: public bool ChangeAmPmDesignator()

  30: {

  31:   return UpdateLocaleInformation(

  32:     this.designatorForAmPm,

  33:     this.timeDesignator);

  34: }

The code snippet below illustrates how to change the AM designator from “AM” to “In the morning.”

   1: static void Main(string[] args)

   2: {

   3:   CustomTimeFormat time = new CustomTimeFormat();

   4:   time.SetAmPmDesignator = AmPmDesignator.AM;

   5:   time.SetAmPmString = "In the morning.";

   6:   if (time.ChangeAmPmDesignator())

   7:   {

   8:     Console.WriteLine("Success");

   9:   }

  10: }

Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle. Time and date are perhaps the most visible national conventions used in many different ways in our applications. We should test the common (equivalent) conventions used in various regions around the world, and customizing these settings helps ensure the developer is properly calling NLS APIs and not using custom functions.

Also, check out the beta release of the GlobalTester automation library that has this functionality and more, and let me know what you think.

I sometimes find code that is way over-engineered. In some cases it may be necessary for making the code more robust. But, over-engineered code may also result from ignorance of more efficient algorithms or design patterns. And, sometimes overly complex algorithms are a result of developers trying to craft something that obfuscates the simplicity of the solution and fool others into believing the complexity of the solution is somehow better than a more simple solution.

I sometimes see this in test code. I am a firm believer in robust, ‘bullet-proof’ test code because each time an automated test case throws a false positive or ‘breaks’ the team loses confidence in the automation project, and it takes our time to ‘massage’ the test back to health. But, there seems to be 2 extremes in test automation. Simplistic prescriptive scripted tests with a bunch of hard-coded ‘test-data,’ or overly complex test code that is virtually undecipherable by anyone other than the original developer that renders any downstream maintenance virtually impossible. I have seen many instances where complete libraries of automation was scrapped and re-developed simply because it was easier to re-write the code rather than trying to wade through the quagmire of complexity.

In a recent example, some of my students submitted their test automation projects with a library that contained a method to generate a random string. When I first looked at their method for generating a random string I was a bit perplexed. Besides the fact that the method only produced a grand total of 26 upper case characters, the code was much more complicated than necessary. Despite having showed them how to use the Babel random string generator they choose to make their own, so I asked them how they came up with this solution and they said “they searched on the Internet. ” Others in the class indicated they also used the same method in their projects. So, I when I got home I did a quick search and found the code sample.

   1: private static string RandomString(int size)

   2: {

   3:   StringBuilder builder = new StringBuilder();

   4:   Random random = new Random();

   5:   

   6:   char ch;

   7:   for (int i = 0; i < size; i++)

   8:   {

   9:     ch = Convert.ToChar(

  10:       Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));

  11:     builder.Append(ch);

  12:   }

  13:  

  14:   return builder.ToString();

  15: }

OK…for a moment let’s overlook the fact that if we make 2 consecutive calls to this method we will get the same identical random string of characters (which the author of this code also discovered), and let’s assume that the range of characters is limited to upper case A through Z, and let’s also ignore the fact that we are not seeding our Random generator for reproducibility of the randomly generated output from this method.

Let’s focus on the statement in lines 9 and 10. Why in the world would we generate a number between 0.0 and 1.0, multiply it by 26 and add 65, and then use Math.Floor to return the largest integer less then or equal to the resultant double, then convert that double to a type Int32, and then convert the Int32 to a type char? Now I ask you, can we make this any more complicated?

Now, I am not critiquing the style of the code or its inherent limitations, but this solution is an example of over-engineering. As I have said before, complexity cultivates chaos. At first I thought maybe this approach might give me a better distribution of characters, but when I tested this hypothesis it simply didn’t pan out. So, the way the random character is generated is simply too complex. An easier, more readable, and effective alternative might be to simply generate a random integer within the allowable range and cast that int to a type char as illustrated below in line 10. (Yes, I realize the limitations with this approach if trying to generate surrogate pair characters above U+FFFF.)

   1: private const int minCharacter = 65;

   2: private const int maxCharacter = 91;

   3: private static string SimpleRandomString(int size)

   4: {

   5:   StringBuilder sb = new StringBuilder();

   6:   System.Threading.Thread.Sleep(1);

   7:   Random r = new Random();

   8:   for (int i = 0; i < size; i++)

   9:   {

  10:     sb.Append((char)r.Next(minCharacter, maxCharacter + 1));

  11:   }

  12:  

  13:   return sb.ToString();

  14: }

BTW…the Sleep() in line 6 is a easy solution to preventing identical return values for consecutive calls. But, a more effective solution would be to pass in a seed value as a parameter to the method and use that to seed the new Random generator as illustrated below. Different seeds not only guarantee randomness in the resultant string, but also allow for repeatability if necessary as long as the seed value is preserved.

   1: private static string SimpleRandomString(int size, int seed)

   2: {

   3:   StringBuilder sb = new StringBuilder();

   4:   Random r = new Random(seed);

   5:   ...

   6: }

But, I digress. This post isn’t about random generation or style…it’s about complexity. Overly complex code tends to harbor errors that might go undetected (at least initially). Also, maintenance of complex code not only becomes more problematic, it often leads to costly re-writes down the road.

If we consider that greater complexity may increase the likelihood of error, then we don’t want our development partners to unnecessarily over-engineered algorithms. Also, overly complex solutions often require overly complex testing. This not only leads to increased testing costs, but it also increases the probability of an important test being missed or overlooked. Think testability!

Finally, with regard to test automation we should consider that our automated tests might be reused by other teams, and they certainly will be used during maintenance or sustained engineering efforts well after the product has released. And, in some cases, the people maintaining the product might not be the same people who shipped the product. Well-written automated tests are not just reviewable by someone other than the author of the code, but they should also be easily maintainable. Otherwise, we might end up paying double for that automated test case. Ouch!

GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.

However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’)  But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’

There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?

First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.

So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”

The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.

Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:

and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.

But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”

“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”

I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.

Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.

This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of  new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)

Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”

Also sucking up some of my time over the past couple of weeks were ‘reorgs’ at work. Change at Microsoft is a constant. Most people get used to it after awhile; others still freak out even with the slightest change. For me it mostly means shifting some priorities based on the new General Manager’s strategic vision, acclimating to a new manager and letting him know what I am working on, and generally making sure the day to day business on our team keeps moving forward during the transition.

Just like life at Microsoft and technology in general the Windows operating system continuously goes through changes. New versions, new service packs, Ultimate, Home, Server versions, etc. Sometimes it is hard to keep up with all the changes. And from a test automation perspective it is sometimes important to know which operating system version the test is running on. In some cases control flow in the automated test case may need to branch in order for the test to execute on different operating system versions. Branching in an automated test based on the operating system version eliminates the need to write separate test cases for variances in operating system versions.

And, certainly if our test matrix includes multiple product versions (Home, Ultimate, Professional, etc) and our automated test exposes a bug then we certainly want to collect information about the operating system version the test was running on. This is especially important if the automated test fails on one machine, but passes on another. Sometimes the cause of the failure may be a slight difference in the machine configuration or the operating system version.

Almost 2 years ago I described a way to get the operating system version information using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members in this blog post. But, I also mention some limitations such as detecting the specific edition or product type of a Windows Version. Another limitation is the difficulty in detecting whether the operating system version is Windows 7 or Windows 2008 Server R2.

I also mentioned that in order to identify a particular version  and/or edition of Windows we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function. But, to use these Win32 APIs we need to use platform invocation services (P/Invoke) to marshal native code into our managed test code.

The code snippet below illustrates the required Win32 function marshaled using the DLLImport attribute in C#, constant values, wrapper methods to get common operating system information, and public properties to get the operating system version, any installed service pack information, and the operating system edition for Windows Vista, Windows Server 2008, and Windows 7.

   1: // <copyright file = VersionInfo.cs" company = "Testing Mentor">

   2: // Copyright © 2010 All Rights Reserved. Test developers can simply copy and

   3: // paste the code into their code, but may not reproduce or publish the code

   4: // snippets on any web site, online service, or distribute as source on any

   5: // media without express written permission. </copyright>

   6:  

   7: namespace TestingMentor.Snippet.OperatingSystemVersionInfo

   8: {

   9:   using System;

  10:   using System.Runtime.InteropServices;

  11:  

  12:   public class WindowsVersionInfo

  13:   {

  14:     public string GetOSVersion

  15:     {

  16:       get { return this.GetOSVersionInfo(); }

  17:     }

  18:  

  19:     public string GetServicePack

  20:     {

  21:       get { return this.GetServicePackInfo(); }

  22:     }

  23:  

  24:     public string GetProductType

  25:     {

  26:       get { return this.GetProductTypeInfo(); }

  27:     }

  28:  

  29:     private string GetOSVersionInfo()

  30:     {

  31:       string version = "Unsupported Version";

  32:  

  33:       NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();

  34:       osvi.VersionInfoSize = 

  35:         Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

  36:       NativeMethods.GetVersionEx(ref osvi);

  37:  

  38:       if (OsviConstant.SupportedPlatform == osvi.PlatformId &&

  39:         osvi.MajorVersion > 4)

  40:       {

  41:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  42:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows2000)

  43:         {

  44:           version = "Windows 2000";

  45:         }

  46:  

  47:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  48:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsXP)

  49:         {

  50:           version = "Windows XP";

  51:         }

  52:  

  53:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  54:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsServer2003)

  55:         {

  56:           if (osvi.ProductType == (byte)OsviConstant.WorkStation)

  57:           {

  58:             version = "Windows XP Professional x64";

  59:           }

  60:           else

  61:           {

  62:             version = "Windows Server 2003";

  63:             if (NativeMethods.GetSystemMetrics(OsviConstant.ServerR2) != 0)

  64:             {

  65:               version += " R2";

  66:             }

  67:           }

  68:         }

  69:  

  70:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&

  71:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsVista)

  72:         {

  73:           if (osvi.ProductType ==

  74:             (byte)OsviConstant.WorkStation)

  75:           {

  76:             version = "Windows Vista";

  77:           }

  78:           else

  79:           {

  80:             version = "Windows Server 2008";

  81:           }

  82:         }

  83:  

  84:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&

  85:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows7)

  86:         {

  87:           if (osvi.ProductType == (byte)OsviConstant.WorkStation)

  88:           {

  89:             version = "Windows 7";

  90:           }

  91:           else

  92:           {

  93:             version = "Windows Server 2008 R2";

  94:           }

  95:         }

  96:       }

  97:  

  98:       return version;

  99:     }

 100:  

 101:     private string GetServicePackInfo()

 102:     {

 103:       NativeMethods.OSVersionInfoEx versionInfo = new NativeMethods.OSVersionInfoEx();

 104:       versionInfo.VersionInfoSize = Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

 105:       NativeMethods.GetVersionEx(ref versionInfo);

 106:       return versionInfo.CSDVersion; 

 107:     }

 108:  

 109:     private string GetProductTypeInfo()

 110:     {

 111:       string product = String.Empty;

 112:  

 113:       NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();

 114:       osvi.VersionInfoSize =

 115:         Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

 116:       NativeMethods.GetVersionEx(ref osvi);

 117:  

 118:       if (osvi.MajorVersion > 5)

 119:       {

 120:         uint productType = 0;

 121:  

 122:         NativeMethods.GetProductInfo(

 123:           osvi.MajorVersion,

 124:           osvi.MinorVersion,

 125:           osvi.ServicePackMajor,

 126:           osvi.ServicePackMinor,

 127:           ref productType);

 128:  

 129:         switch (productType)

 130:         {

 131:           case (uint)OsviConstant.ProductInfo.Business:

 132:             product = "Business Edition";

 133:             break;

 134:           case (uint)OsviConstant.ProductInfo.BusinessN:

 135:             product = "Business N Edition";

 136:             break;

 137:           case (uint)OsviConstant.ProductInfo.ClusterServer:

 138:             product = "HPC Edition";

 139:             break;

 140:           case (uint)OsviConstant.ProductInfo.DatacenterServer:

 141:             product = "Server Datacenter (Full)";

 142:             break;

 143:           case (uint)OsviConstant.ProductInfo.DatacenterServerCore:

 144:             product = "Server Datacenter (Core)";

 145:             break;

 146:           case (uint)OsviConstant.ProductInfo.DataCenterServerCoreV:

 147:             product = "Server Datacenter without Hyper-V (Core)";

 148:             break;

 149:           case (uint)OsviConstant.ProductInfo.DataCenterServerV:

 150:             product = "Server Datacenter without Hyper-V (Full)";

 151:             break;

 152:           case (uint)OsviConstant.ProductInfo.Enterprise:

 153:             product = "Enterprise Edition";

 154:             break;

 155:           case (uint)OsviConstant.ProductInfo.EnterpriseE:

 156:             product = "Enterprise E Edition";

 157:             break;

 158:           case (uint)OsviConstant.ProductInfo.EnterpriseN:

 159:             product = "Enterprise N Edition";

 160:             break;

 161:           case (uint)OsviConstant.ProductInfo.EnterpriseServer:

 162:             product = "Server Enterprise (Full)";

 163:             break;

 164:           case (uint)OsviConstant.ProductInfo.EnterpriseServerCore:

 165:             product = "Server Enterprise (Core)";

 166:             break;

 167:           case (uint)OsviConstant.ProductInfo.EnterpriseServerCoreV:

 168:             product = "Server Enterprise without Hyper-V (Core)";

 169:             break;

 170:           case (uint)OsviConstant.ProductInfo.EnterpriseServerIA64:

 171:             product = "Server Enterprise for Itanium-based Systems";

 172:             break;

 173:           case (uint)OsviConstant.ProductInfo.EnterpriseServerV:

 174:             product = "Server Enterprise without Hyper-V (Full)";

 175:             break;

 176:           case (uint)OsviConstant.ProductInfo.HomeBasic:

 177:             product = "Home Basic Edition";

 178:             break;

 179:           case (uint)OsviConstant.ProductInfo.HomeBasicE:

 180:             product = "Home Basic E Edition";

 181:             break;

 182:           case (uint)OsviConstant.ProductInfo.HomeBasicN:

 183:             product = "Home Basic N Edition";

 184:             break;

 185:           case (uint)OsviConstant.ProductInfo.HomePremium:

 186:             product = "Home Premium Edition";

 187:             break;

 188:           case (uint)OsviConstant.ProductInfo.HomePremiumE:

 189:             product = "Home Premium E Edition";

 190:             break;

 191:           case (uint)OsviConstant.ProductInfo.HomePremiumN:

 192:             product = "Home Premium N Edition";

 193:             break;

 194:           case (uint)OsviConstant.ProductInfo.HomeServer:

 195:             product = "Home Server Edition";

 196:             break;

 197:           case (uint)OsviConstant.ProductInfo.HyperV:

 198:             product = "Microsoft Hyper-V Server";

 199:             break;

 200:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerManagement:

 201:             product = "Windows Essential Business Server Management Server";

 202:             break;

 203:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerMessaging:

 204:             product = "Windows Essential Business Server Messaging Server";

 205:             break;

 206:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerSecurity:

 207:             product = "Windows Essential Business Server Security Server";

 208:             break;

 209:           case (uint)OsviConstant.ProductInfo.Professional:

 210:             product = "Professional Edition";

 211:             break;

 212:           case (uint)OsviConstant.ProductInfo.ProfessionalE:

 213:             product = "Professional E Edition";

 214:             break;

 215:           case (uint)OsviConstant.ProductInfo.ProfessionalN:

 216:             product = "Professional N Edition";

 217:             break;

 218:           case (uint)OsviConstant.ProductInfo.ServerForSmallBusiness:

 219:             product = "Windows Server 2008 for Windows Essential Server Solutions";

 220:             break;

 221:           case (uint)OsviConstant.ProductInfo.ServerForSmallBusinessV:

 222:             product = "Windows Server 2008 without Hyper-V for Windows Essential Server Solutions";

 223:             break;

 224:           case (uint)OsviConstant.ProductInfo.ServerFoundation:

 225:             product = "Server Foundation";

 226:             break;

 227:           case (uint)OsviConstant.ProductInfo.SmallBusinessServer:

 228:             product = "Windows Small Business Server";

 229:             break;

 230:           case (uint)OsviConstant.ProductInfo.SmallBusinessServerPremium:

 231:             product = "Windows Small Busines Server Premium";

 232:             break;

 233:           case (uint)OsviConstant.ProductInfo.StandardServer:

 234:             product = "Server Standard (Full)";

 235:             break;

 236:           case (uint)OsviConstant.ProductInfo.StandardServerCore:

 237:             product = "Server Standard (Core)";

 238:             break;

 239:           case (uint)OsviConstant.ProductInfo.StandardServerCoreV:

 240:             product = "Server Standard without Hyper-V (Core)";

 241:             break;

 242:           case (uint)OsviConstant.ProductInfo.StandardServerV:

 243:             product = "Server Standard without Hyper-V (Full)";

 244:             break;

 245:           case (uint)OsviConstant.ProductInfo.Starter:

 246:             product = "Starter Edition";

 247:             break;

 248:           case (uint)OsviConstant.ProductInfo.StarterE:

 249:             product = "Starter E Edition";

 250:             break;

 251:           case (uint)OsviConstant.ProductInfo.StarterN:

 252:             product = "Starter N Edition";

 253:             break;

 254:           case (uint)OsviConstant.ProductInfo.StorageEnterpriseServer:

 255:             product = "Storage Server Enterprise";

 256:             break;

 257:           case (uint)OsviConstant.ProductInfo.StorageExpressServer:

 258:             product = "Storage Server Express";

 259:             break;

 260:           case (uint)OsviConstant.ProductInfo.StorageStandardServer:

 261:             product = "Storage Server Standard";

 262:             break;

 263:           case (uint)OsviConstant.ProductInfo.StorageWorkgroupServer:

 264:             product = "Storage Server Workgroup";

 265:             break;

 266:           case (uint)OsviConstant.ProductInfo.Ultimate:

 267:             product = "Ultimate Edition";

 268:             break;

 269:           case (uint)OsviConstant.ProductInfo.UltimateE:

 270:             product = "Ultimate E Edition";

 271:             break;

 272:           case (uint)OsviConstant.ProductInfo.UltimateN:

 273:             product = "Ulitmate N Edition";

 274:             break;

 275:           case (uint)OsviConstant.ProductInfo.Undefined:

 276:             product = "Unknown Product";

 277:             break;

 278:           case (uint)OsviConstant.ProductInfo.Unlicensed:

 279:             product = "Unlicensed or Expired";

 280:             break;

 281:           case (uint)OsviConstant.ProductInfo.WebServer:

 282:             product = "Web Server (Full)";

 283:             break;

 284:           case (uint)OsviConstant.ProductInfo.WebServerCore:

 285:             product = "Web Server (Core)";

 286:             break;

 287:         }

 288:       }

 289:  

 290:       return product;

 291:     }

 292:   }

 293:  

 294: // ****************************************************************************

 295: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE

 296: // ****************************************************************************

 297:   

 298:   internal class OsviConstant

 299:   {

 300:     internal const int SupportedPlatform = 2;

 301:     internal const int ServerR2 = 89;

 302:     internal const int WorkStation = 0x00000001;

 303:  

 304:     private OsviConstant()

 305:     {

 306:     }

 307:  

 308:     internal enum MajorVersion

 309:     {

 310:       NT5 = 5,

 311:       NT6 = 6

 312:     }

 313:  

 314:     internal enum MinorVersion

 315:     {

 316:       Windows2000 = 0,

 317:       WindowsXP = 1,

 318:       WindowsServer2003 = 2,

 319:       WindowsVista = 0,

 320:       Windows7 = 1

 321:     }

 322:  

 323:     internal enum ProductInfo : uint

 324:     {

 325:       Business = 0x00000006,

 326:       BusinessN = 0x00000010,

 327:       ClusterServer = 0x00000012,

 328:       DatacenterServer = 0x00000008,

 329:       DatacenterServerCore = 0x0000000C,

 330:       DataCenterServerCoreV = 0x00000027,

 331:       DataCenterServerV = 0x00000025,

 332:       Enterprise = 0x00000004,

 333:       EnterpriseE = 0x00000046,

 334:       EnterpriseN = 0x0000001B,

 335:       EnterpriseServer = 0x0000000A,

 336:       EnterpriseServerCore = 0x0000000E,

 337:       EnterpriseServerCoreV = 0x00000029,

 338:       EnterpriseServerIA64 = 0x0000000F,

 339:       EnterpriseServerV = 0x00000026,

 340:       HomeBasic = 0x00000002,

 341:       HomeBasicE = 0x00000043,

 342:       HomeBasicN = 0x00000005,

 343:       HomePremium = 0x00000003,

 344:       HomePremiumE = 0x00000044,

 345:       HomePremiumN = 0x0000001A,

 346:       HyperV = 0x0000002A,

 347:       MediumBusinessServerManagement = 0x0000001E,

 348:       MediumBusinessServerSecurity = 0x0000001F,

 349:       MediumBusinessServerMessaging = 0x00000020,

 350:       Professional = 0x00000030,

 351:       ProfessionalE = 0x00000045,

 352:       ProfessionalN = 0x00000031,

 353:       ServerForSmallBusiness = 0x00000018,

 354:       ServerForSmallBusinessV = 0x00000023,

 355:       ServerFoundation = 0x00000021,

 356:       SmallBusinessServer = 0x00000009,

 357:       StandardServer = 0x00000007,

 358:       StandardServerCore = 0x0000000D,

 359:       StandardServerCoreV = 0x00000028,

 360:       StandardServerV = 0x00000024,

 361:       Starter = 0x0000000B,

 362:       StarterE = 0x00000042,

 363:       StarterN = 0x0000002F,

 364:       StorageEnterpriseServer = 0x00000017,

 365:       StorageExpressServer = 0x00000014,

 366:       StorageStandardServer = 0x00000015,

 367:       StorageWorkgroupServer = 0x00000016,

 368:       Undefined = 0x00000000,

 369:       Ultimate = 0x00000001,

 370:       UltimateE = 0x00000047,

 371:       UltimateN = 0x0000001C,

 372:       WebServer = 0x00000011,

 373:       WebServerCore = 0x0000001D,

 374:       Unlicensed = 0xABCDABCD,

 375:       HomeServer = 0x00000013,

 376:       SmallBusinessServerPremium = 0x00000019,

 377:     }

 378:   }

 379:  

 380: // ****************************************************************************

 381: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE

 382: // ****************************************************************************

 383:  

 384:   internal class NativeMethods

 385:   {

 386:     private NativeMethods()

 387:     {

 388:     }

 389:  

 390:     [DllImport("kernel32")]

 391:     [return: MarshalAs(UnmanagedType.Bool)]

 392:     internal static extern bool GetVersionEx(ref OSVersionInfoEx osvi);

 393:  

 394:     [DllImport("kernel32.dll")]

 395:     [return: MarshalAs(UnmanagedType.Bool)]

 396:     internal static extern bool GetProductInfo(

 397:       int osMajorVersion,

 398:       int osMinorVersion,

 399:       int spMajorVersion,

 400:       int spMinorVersion,

 401:       ref uint type);

 402:  

 403:     [DllImport("kernel32.dll")]

 404:     internal static extern int GetSystemMetrics(

 405:       int index);

 406:  

 407:     [StructLayout(LayoutKind.Sequential)]

 408:     internal struct OSVersionInfoEx

 409:     {

 410:       public int VersionInfoSize;

 411:       public int MajorVersion;

 412:       public int MinorVersion;

 413:       public int BuildNumber;

 414:       public int PlatformId;

 415:       [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]

 416:       public string CSDVersion;

 417:       public Int16 ServicePackMajor;

 418:       public Int16 ServicePackMinor;

 419:       public Int16 SuiteMask;

 420:       public byte ProductType;

 421:       public byte Reserved;

 422:     }

 423:   }

 424: }

Now, some of my readers have indicated that these code snippets are not very useful because they can’t copy them directly and put them to use. So, to help resolve that issue I have created a new section on my web site called the Code Snippet Library. This snippet is posted there along with fully annotated (mostly FxCopy and StyleCopy compliant) file available for download or to copy for inclusion in your automated test cases, or compiled into a dynamic link library (DLL).

This example doesn’t differentiate between 32-bit and 64-bit Windows operating systems, but that is not really difficult to add, and if I get enough requests I will certainly add that into the pot. If the operating system version is no longer supported by Microsoft the GetOsVersion property will return "Unsupported Version." If no Service Packs are installed the GetServicePack property will return an empty string. If you need to detect an unsupported operating system version use the example here.

One of the best parts of the trips were sitting around the camp fire at night listening to my uncle concocting some story intended to scare the wits out of us young boys. You know, the kind of stories about headless Confederate soldiers, or werewolves, vampires, or other such wicked creatures of the night. I think these campfire chats are remnants of man’s tribal roots where the elders tried to scare the hell out of the juvenile hominids to prevent them from wandering off at night. As we got older we realized that these stories were simply fictitious folktales; sort of like successful, value-add GUI automation projects.

Last week I had lunch with a colleague who wanted to talk to me about an automation project on his team that went horribly awry. As he started to tell me his story I thought, “Wait…I heard this tale before. I can tell this story because I’ve heard it so many times over and over again…just like those scary stories I heard around the campfire growing up.” The story goes something like this.

Our team bought a new tool, or built another framework, and taught everyone how to script “black box” test cases. They developed quite a number of automated test scripts, and of course everything was working very well. The scripts were running and managers were happy because the team had a lot of automated test scripts. But, the tests weren’t finding any bugs, so just out of curiosity the managers suggested a bug bash. And sure enough as the testers started exploring the project it didn’t take long for them to fill the database with bugs. The developers were shell shocked, and the managers couldn’t believe it! They couldn’t understand why the automated GUI tests weren’t finding any bugs? And so, in a typical knee-jerk reaction fashion, the managers immediately halted the GUI automation project and required every tester to embark on an exploratory testing adventure in search of bugs. Of course, the managers decided this approach was better than investing in more GUI automation bringing an end to another GUI automation project.

Unfortunately, unlike the scary stories my father and uncles told around the campfires, stories of failed GUI automation are often true, and usually much scarier. Why are they so scary? Because I hear these sorts of stories repeated so often. It seems that we as a discipline rely on tribal knowledge where each generation simply learns through trial and error and the folktales of our elders and thrive more on hero worship of people who are often remarkably good at finding bugs by poking and prodding hour upon hour.

Now, if you haven’t caught on already you will know that I am no big fan of GUI automation. Not because I don’t think it can be useful. In fact, I think GUI automation can provide tremendous value in some situations. But unfortunately much of the automated GUI test cases I see (especially in examples) are poorly designed, simple rudimentary script-lets. Many of these automated tests are nothing more than mindless automated sequences of events contrived because the testers have been told to automate, but are not given strategic vision (why) and little to no tactical direction (what and how).

With little or no direction or goals, or without an in-depth understanding of the system they are testing the biggest problems with GUI automation is that many testers attempt to automate

• functional tests intended to expose computational errors in the business logic layer or in the underlying APIs
• usability tests intended to imitate ‘me’ trying to emulate the scenarios or tasks I think the customer might do

GUI automation is probably the least effective approach for functional testing. This is not to say that GUI automation will not find functional issues in the lower logic layers. I suspect we will always find ‘functional’ bugs (e.g. boundary issues, unhandled exceptions, string parsing errors, calculation problems, etc.) while testing through the UI. But, as indicated in my previous post, well-designed software is usually built in layers and a good many of the ‘functional’ issues we find today can likely be more efficiently found through more robust unit and component (API) levels of testing.

Perhaps even more silly than trying to use GUI automation for low level functional testing is the notion of using GUI automation to emulate a ‘user’ by scripting out prescriptive sequences of actions (often with hard-coded data) that are then played over and over again. Test automation cannot and should not attempt to replicate ‘me.’ I’ve said before the purpose of automation is to provide value to me, to free up my time, to increase my efficiency, and to help me be more effective in my job; automation does not replace me. Let’s face it…we (humans) are much better at evaluating the ease of use of software and whether scenarios that represent target customer segments are intuitive for those customers.

For example, I once had a conversation explaining that GUI automation runs much faster than I can interact with software, and sometimes I make mistakes when typing in something that throws an unexpected message or takes me down a path. My colleague replied, “Well, we can slow down the automation.” Why? Why in the hell would I want to slow down my automated tests? C’mon…we all should know by now that the 100% automation (or automate all tests) mantra is a ridiculous dream and I’ve heard more plausible fantasies from people on acid trips.

So, where does GUI automation add value. In my opinion, GUI automation is probably most effective in testing UI control properties and the event handlers between the UI layer and the API layers. It is also effective in behavioral testing areas such as performance and stress.  And GUI automation is also much more effective in evaluating UI layout issues such as misaligned controls, or clipped or truncated controls on a window as compared to the human eye.

Similar to how we use different techniques to expose different categories of defects, and how we use different approaches to testing depending on the context or test objective test automation is a useful tool in our toolbox. It is certainly not the only tool. We can do some remarkable things with automation, but we must learn where GUI automation adds value and where other approaches testing (automated or manual) might be more effective.

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of  3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

   1: // <copyright file="simpletestcase.cs" company="TestingMentor"> 

   2:  // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3:  // </copyright> 

   4: 

   5: namespace TestingMentor.Sample

   6: {

   7:   using System;

   8:   using System.IO;

   9:   using TestingMentor.Simulation;

  10: 

  11:   class TestCase

  12:   {

  13:     static void Main(string[] args)

  14:     {

  15:       int testCounter = 0;

  16:       // Read in an array of strings representing the test data. 

  17:       // Of course this would likely come from a static test data file

  18:       // on a server or copied to a folder on the local machine

  19:       string[] testData = new string[]

  20:       { "90,Z",

  21:         "24798,惞",

  22:         "0,null",

  23:         "65536,Error",

  24:         "-1,Error",

  25:         "1.5,",

  26:         "xyz,xyz"

  27:       };

  28: 

  29:       // Loop through each test data string

  30:       foreach (string test in testData)

  31:       {

  32:         testCounter++;

  33:         // This nested try/catch block catches invalid test data

  34:         // but allow additonal tests in the testData array

  35:         try

  36:         {

  37:           // Parse each string into the test data and expected result

  38:           string[] testElement = test.Split(',');

  39:           string expectedResult = testElement[1];

  40:           string actualResult = String.Empty;

  41: 

  42:           // Convert the string to a type int value

  43:           int value = int.Parse(testElement[0]);

  44:

  45:           // We need a way to handle int values 0 through 32 which are 

  46:           // control characters, this is an example of how to deal with 

  47:           // a int value of 0 which is a null character

  48:           if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))

  49:           {

  50:             expectedResult = '\0'.ToString();

  51:           }

  52: 

  53:           // This nested try/catch block tests catches exceptions thrown by 

  54:           // the method under test. If the method under test throws an 

  55:           // exception we certainly want to test for that case!

  56:           try

  57:           {

  58:             // Call the API method under test 

  59:             char result = Converter.ConvertValueToUnicodeChar(value);

  60:             actualResult = result.ToString();

  61:           }

  62: 

  63:           catch (ArgumentOutOfRangeException)

  64:           {

  65:             actualResult = "Error";

  66:           }

  67: 

  68:           catch (Exception)

  69:           {

  70:             // if this happens this is a failure because the documentation

  71:             // states that this method will only throw an 

  72:             // ArgumentOutOfRangeException.

  73:             actualResult = "Non-specific or unexpected error thrown";

  74:           }

  75: 

  76:           // Call a simple oracle and log results

  77:           if (String.Equals(actualResult, expectedResult))

  78:           {

  79:             // log pass

  80:             Console.WriteLine("{0} Pass", testCounter);

  81:           }

  82:           else

  83:           {

  84:             // log fail...of course log as much detail as possible

  85:             Console.WriteLine("{0} Fail", testCounter);

  86:           }

  87:         }

  88: 

  89:         catch (FormatException)

  90:         {

  91:           // log the test data for this test as incorrect, test is skipped

  92:           Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);

  93:         }

  94:       }

  95:     }

  96:   }

  97: }

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

One of the most popular random test data generators that I have developed so far has been a tool called CCMaker to generate random valid and invalid credit card numbers. (Sometimes I wonder why that is, but I don’t dwell on it for too long and I haven’t been interrogated by the FBI lately.) Testing forms that require a credit card has always been risky business because you certainly don’t want to use your own card. Often times developers will include a check on web forms or client apps to do a high level verification of a credit card number before sending all the data across the wire to be validated. This early or high level verification prevents flooding the pipe with bad data. So, one test we can do prior to testing the end-to-end scenario is to test to see if and how the developer is validating credit cards numbers prior to submission.

As far as data goes, generating credit card numbers are fairly simple. There is a bank ID number (BIN), there is a number of digits between 12 and 19 depending on the card type, and there is a checksum. So, if we know the valid BINs for each issuing bank, the valid number of digits for each card type, and how to calculate the checksum we can generate valid credit card numbers. (Of course this is a bit oversimplified because many credit and debit card companies are issued multiple BINs and use varying number lengths.)

Testing for invalid credit card numbers should include using numbers that look close to being correct in some way but are slightly altered. For example for the 3 defined equivalent partitions (BIN, length, checksum) there are seven possible invalid combinations (2³ – 1) we could test.

1. Valid BIN, invalid length and valid checksum
2. Valid BIN, valid length, and invalid checksum
3. Valid BIN, invalid length, and invalid checksum
4. Invalid BIN, valid length and valid checksum
5. Invalid BIN, invalid length, and valid checksum
6. Invalid BIN, valid length, and invalid checksum
7. Invalid BIN, checksum and length

This doesn’t mean I run 7 tests and call it good because there are numerous invalid lengths and invalid BINs for the different card types. A common mistake when using an equivalent partition testing approach is to simply plug in values for each combination listed above and call it good. The problem is that there are several hundred BINs and 8 different valid lengths. For example, for just the Discover card there are 829 valid BIN numbers, and for the Maestro cards there are 56 combinations of BINs and card lengths ranging from 12 to 19 numbers in length. This doesn’t include the permutations of the other numbers that compose the entire card number.

The question every tester must ask him or herself every day when designing tests is how many tests do I need to have any reasonable sense of confidence that risk is minimal and the perception of quality is high. Of course, there is no single right answer here and not magic formula, but since we can’t possibly execute every possible positive or negative test we should at least understand that ultimately testing is sampling.

For example, one strategy for positive testing might be to test every valid BIN for every valid card length for any given credit card. For example for American Express I would want to test at least one number with a BIN of 34 and a card length of 15 that satisfies the checksum requirement, and at least one number with a BIN of 37 and a card length of 15 that also satisfies the checksum requirement. For a card type of Visa I would need a minimum of 2 tests in which the BIN is 4, the checksum requirement is satisfied, and one has a card length of  13 numbers and the other has a card length of 16 numbers.

That probably sounds like quite a bit of testing, and tests which most likely would not produce an error (unless of course the BIN is miss identified (e.g. instead of checking for a BIN of 5020 the BIN is incorrectly assigned as 5002), or if a valid BIN is not recognized as valid because it is omitted from a list or enumeration of valid BINs for that credit card). Certainly testing of this magnitude would be expensive if done manually. But when automated using a random test data generator and a data-driven automation approach to set the random generator properties comprehensive testing becomes a much more reasonable proposition and can significantly increase overall confidence.

This is where my CCMaker 3.1 test data generator can help by randomly generating both valid and invalid credit card numbers. The updated CCMaker test automation library has just been posted to my web site with documentation and examples. If you have any questions, or find any issues with the new library please let me know.

A significant percentage of static test data is stored in tabular comma delimited or tab-delimited formats and saved in Excel spreadsheets. Reading in comma or tab-delimited static test data into an automated test is pretty straight forward and there are numerous examples in many programming languages illustrating how to read in these types of test data repositories. Reading in rows of data is the foundation of data-driven automation and definitely has its place in any automation project.

I am a big proponent of stochastic (random) test data generation that is customized to the context, but I also know that sometimes static test data is useful for establishing baselines and more exact emulation of ‘real-world’ customer-like inputs. But, if the automated test is simply passing the same variable arguments to the same input parameters in the same order over and over again the value of subsequent iterations of that automated test using that static data set diminishes rather quickly. So how can we more effectively utilize static test data in our automated tests?

One possible solution is to randomly select an argument from a collection of static variables that is passed to the specific input parameter. The advantage of this approach is that it effectively increases the test data permutations in each iteration of the test case. For example, let’s consider 2 input parameters; one for a given name and one for a surname. In a traditional data-driven approach in which the static test data is read in by rows our test data file might be similar to:

Bob,Smith
John,Johnson
Roger,Williams
Steve,Abbot

This static data file would give us 4 sets of test data, but each time the test data is read into the test case the given and surnames are always the same.

However, if we read in the given names and surnames into 2 collections, and then randomly select a given name and surname from the appropriate collection to pass to the respective parameter we effectively have 16 possible combinations of static test data to work with. An advantage of this approach is that our ‘collections’ of given names and surnames can contain differing numbers of elements (in which case the number of possible combinations of test data is the Cartesian product of the number of elements in each collection).

Of course there are many ways to accomplish this. For example, one approach is to continue to use a comma or tab-delimited file format and list given names in one row and surnames in a second row. Another approach is to list the given names and surnames in columns in a spreadsheet and read in each column into a collection of some sort. The latter is the approach I used in developing my PseudoName test data generator tool. I chose this approach for 2 reasons; first an Excel spreadsheet is a simple yet powerful file format for storing static test data, and secondly because lists of test data are sometimes better represented in columns rather than rows.

The following code shows one way to read in test data by columns from an Excel spreadsheet.

   1: // <copyright file="datareader.cs" company="TestingMentor"> 

   2: // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3: // </copyright> 

   4:   

   5: namespace TestingMentor.TestTool 

   6: { 

   7:   using System; 

   8:   using System.Collections; 

   9:   using System.Globalization; 

  10:   using System.Runtime.InteropServices; 

  11:   using System.Threading; 

  12:   using Excel = Microsoft.Office.Interop.Excel; 

  13:   

  14:   /// <summary> 

  15:   /// This class contains methods for reading test data from Excel spreadsheets 

  16:   /// </summary> 

  17:   public class TestDataReader 

  18:   { 

  19:     /// <summary> 

  20:     /// This method reads all the data elements in the specified number of 

  21:     /// columns in the specified Excel spreadsheet containing the test data 

  22:     /// and copies the data into a multi-dimensional array 

  23:     /// </summary> 

  24:     /// <param name="dataFileName">The filename containing the test data</param> 

  25:     /// <param name="columnCount">The number of columns in the Excel 

  26:     /// spreadsheet to read</param> 

  27:     /// <returns>A multi-dimensional array containing the data eleements for 

  28:     /// each column </returns> 

  29:     public static string[][] ExcelColumnReader(string dataFileName, uint columnCount) 

  30:     { 

  31:       CultureInfo originalCulture = null; 

  32:       Excel.Application excelApp = null; 

  33:       Excel.Workbook excelWorkbook = null; 

  34:       Excel.Worksheet excelActiveWorksheet = null; 

  35:       string[][] testData = new string[columnCount][]; 

  36:   

  37:       originalCulture = Thread.CurrentThread.CurrentCulture; 

  38:       Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US"); 

  39:   

  40:       excelApp = new Excel.Application(); 

  41:       excelWorkbook = excelApp.Workbooks.Open( 

  42:         dataFileName, 

  43:         0, 

  44:         false, 

  45:         5, 

  46:         String.Empty, 

  47:         String.Empty, 

  48:         false, 

  49:         Type.Missing, 

  50:         String.Empty, 

  51:         true, 

  52:         false, 

  53:         0, 

  54:         true, 

  55:         false, 

  56:         false); 

  57:       excelActiveWorksheet = (Excel.Worksheet)excelWorkbook.ActiveSheet; 

  58:   

  59:       for (int i = 0; i < columnCount; i++) 

  60:       { 

  61:         // Start at column 1 

  62:         object columnIndex = i + 1; 

  63:   

  64:         // Row 1 is the column title; test data starts on Row 2 

  65:         object rowIndex = 2; 

  66:         ArrayList tempCollection = new ArrayList(); 

  67:         while ( 

  68:           ((Excel.Range) 

  69:           excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2 != null) 

  70:         { 

  71:           tempCollection.Add( 

  72:             ((Excel.Range) 

  73:             excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2); 

  74:           rowIndex = (int)rowIndex + 1; 

  75:         } 

  76:   

  77:         testData[i] = new string[tempCollection.Count]; 

  78:         testData[i] = (string[])tempCollection.ToArray(typeof(string)); 

  79:       } 

  80:   

  81:       // Clean up 

  82:       excelWorkbook.Close(false, Type.Missing, Type.Missing); 

  83:       excelWorkbook = null; 

  84:       excelApp.Quit(); 

  85:       excelApp = null; 

  86:   

  87:       // Garbage collection is not pretty, but necessary to release Excel proc 

  88:       System.GC.Collect(); 

  89:       System.GC.WaitForPendingFinalizers(); 

  90:   

  91:       if (originalCulture != null) 

  92:       { 

  93:         Thread.CurrentThread.CurrentCulture = originalCulture; 

  94:       } 

  95:   

  96:       return testData; 

  97:     } 

  98:   } 

  99: } 

I must tell you that performance can be an issue especially if the columns contain a lot of data. For example, to read in approximately 700 elements of test data in 3 separate columns took slightly less than 1 second, and reading in 1800 elements in 3 columns required just over 4 seconds. Unfortunately, I didn’t compare total byte counts, but it is pretty obvious the greater the number of test data elements being read the longer the read operation will take and you certainly will have to take the read time into consideration in your automated test case.

Reading static test data line by line from a data file while looping through a data-driven automated test case is a useful test design approach in some situations, this is another useful approach that will allow the test designer to randomize the combinations of static test data values applied to multiple input parameters in multiple iterations of an automated test case.