Archive for the ‘Test Automation’ Category
Globalization Testing: Customizing Time Formats
Time is a commodity in short supply. I have been juggling a lot lately and there never seems to be enough time to do everything I need to do, and even less time to do the things I want to do. (Blogging falls under the want to do category.) I wish sometimes I could slow down the hands of time, but that is beyond my control. What is within my control is changing the time format displayed on the computer. And if I need to do that in an automated test to increase the robustness of my test to include globalization, then I can programmatically change the time format without having to manipulate the Region and Language settings control panel applet.
Time and date information is commonly pulled from the operating system by many developers for use in headers or footers on documents, default file names, printing, and other places time/date stamps are useful or important. To ensure our products are “world-ready” we should modify the formats to validate whether our product supports various national conventions used in different regions (locales) around the world. In the previous post I illustrated how to programmatically customize the date formats on a Windows environment for including some basic globalization tests in your test automation. This week let’s look at how we can programmatically change both the short time and long time formats.
We will again need the 2 Win32 API functions SetLocaleInfo() and PostMessage() that we marshaled over into the NativeMethods class. Since that code doesn’t change I won’t repeat it here you can simply refer to the code snippet in the previous post. In this situation we need to set the lcType in SetLocaleInfo() to the LOCALE_STIMEFORMAT constant. Then we can pass a null-terminated string to the lcData variable in the SetLocaleInfo() function. MSDN explains “The maximum number of characters allowed for this string is 80, including a terminating null character. The string can consist of a combination of hour, minute, and second format pictures.”
Once again, to simplify that a bit I wrote some more wrapper methods to change the time format. Also, since we will be calling SetLocaleInfo() and PostMessage() a lot for customizing date, time, and other national conventions I created a wrapper method called UpdateLocaleInformation() to remove redundancy.
1: namespace TestingMentor.TestTool.GlobalTester
2: {
3: using System;
4:
5: public enum TimeFormatType
6: {
7: LongTimeFormat = 0x00001003,
8: ShortTimeFormat = 0x00000079
9: }
10:
11: public class CustomTimeFormat
12: {
13: private int timeFormatType = (int)TimeFormatType.ShortTimeFormat;
14: private string timeFormatPicture = string.Empty;
15:
16: public int SetTimeFormatType
17: {
18: set
19: {
20: if (value == (int)TimeFormatType.ShortTimeFormat ||
21: value == (int)TimeFormatType.LongTimeFormat)
22: {
23: this.timeFormatType = value;
24: }
25: else
26: {
27: throw new ArgumentOutOfRangeException("TimeFormatType invalid");
28: }
29: }
30: }
31:
32: public string SetTimeFormatPicture
33: {
34: set { this.timeFormatPicture = value; }
35: }
36:
37: public bool ChangeTimeFormat()
38: {
39: return UpdateLocaleInformation(
40: this.timeFormatType,
41: this.timeFormatPicture);
42: }
43:
44: private bool UpdateLocaleInformation(int localeType, string localeData)
45: {
46: bool success = false;
47: if (NativeMethods.SetLocaleInfo(
48: NativeMethods.SystemDefaultLocale,
49: localeType,
50: localeData))
51: {
52: NativeMethods.PostMessage(
53: NativeMethods.BroadcastMessage,
54: NativeMethods.SettingChangeMessage,
55: IntPtr.Zero,
56: IntPtr.Zero);
57: success = true;
58: }
59:
60: return success;
61: }
62: }
63: }
Once again, we simply have to set the SetTimeFormatType property to either the Short time or Long time format, provide the format picture by setting the SetTimeFormatPicture property, and then call ChangeTimeFormat(). The sample below illustrates how to change the short time format with different time separators and a reverse order.
1: static void Main(string[] args)
2: {
3: CustomTimeFormat time = new CustomTimeFormat();
4: time.SetTimeFormatType = (int)TimeFormatType.ShortTimeFormat;
5: time.SetTimeFormatPicture = "ss'mm,hh - tt";
6: if (time.ChangeTimeFormat())
7: {
8: Console.WriteLine("Success");
9: }
10: }
Now, we can also customize the AM/PM designator as well. To change the AM/PM designator we need to add a few more properties and another wrapper method. In this case, I’ve added the SetAmPmDesignator property, the SetAmPmString property, and the ChangeAmPmDesignator() method.
1: public enum AmPmDesignator
2: {
3: AM = 0x00000028,
4: PM = 0x00000029
5: }
6:
7: public class CustomTimeFormat
8: {
9: public int SetAmPmDesignator
10: {
11: set
12: {
13: if (value == (int)AmPmDesignator.AM || value == (int)AmPmDesignator.PM)
14: {
15: this.designatorForAmPm = value;
16: }
17: else
18: {
19: throw new ArgumentOutOfRangeException("AmPmDesignator invalid.");
20: }
21: }
22: }
23: }
24: public string SetAmPmString
25: {
26: set { this.timeDesignator = value; }
27: }
28:
29: public bool ChangeAmPmDesignator()
30: {
31: return UpdateLocaleInformation(
32: this.designatorForAmPm,
33: this.timeDesignator);
34: }
The code snippet below illustrates how to change the AM designator from “AM” to “In the morning.”
1: static void Main(string[] args)
2: {
3: CustomTimeFormat time = new CustomTimeFormat();
4: time.SetAmPmDesignator = AmPmDesignator.AM;
5: time.SetAmPmString = "In the morning.";
6: if (time.ChangeAmPmDesignator())
7: {
8: Console.WriteLine("Success");
9: }
10: }
Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle. Time and date are perhaps the most visible national conventions used in many different ways in our applications. We should test the common (equivalent) conventions used in various regions around the world, and customizing these settings helps ensure the developer is properly calling NLS APIs and not using custom functions.
Also, check out the beta release of the GlobalTester automation library that has this functionality and more, and let me know what you think.
Globalization Testing: Customizing the Date Format
The ability of our software products to function correctly in a global environment is becoming more and more important. Our software should support national conventions used by the various locales around the globe. For example, in some regions of the world the period character is used as the number group separator and the comma is used as the decimal symbol (radix). European calendars generally start on Monday rather than Sunday which is customary in the United States. Era based calendars are still in common use in Japan and Korea, date formats and order, and time formats also vary by region or locale. As testers we need to test our software to ensure our customers around the world can use the national conventions they are accustomed to, and not force them down a US-centric, one-size-fits-all format or standard.
There are several settings that we can modify and customize for more robust globalization testing such as number, currency, time and date formats. Modifying these settings can help us test that our application is globalized to use National Language System (NLS) APIs provided by the system.Although a user would change these settings using the Regional Options user interface property sheets, if the purpose of our test is not to emulate user interaction, then modifying the custom regional settings for globalization testing programmatically is more efficient.
Last year I talked about how to programmatically make changes to the settings in the Region and Language control panel applet when doing globalization testing. Unfortunately, the code sample provided in the previous post was appropriate for versions of Windows XP and earlier. For versions of Windows Vista and later things have changed a bit. Also, the previous sample tried to be a one-size fits all and relied on the test developer to set the appropriate lcType constants and lcData argument variables required by the Win32 function SetLocaleInfo().
This time, I decided to simplify things a bit and wrapped some methods to call the appropriate Win32 API functions and properties to set lcType and lcData values to make it easier to incorporate into automated tests. I also separated the various advanced custom formats for Region and Language options into separate classes. Of course, I have a beta version of an automation library (DLL) called GlobalTest.DLL on my website that testers can use in their automated test cases, but this week let’s look at the class for setting custom date formats.
Making these changes programmatically still requires the Win32 SetLocaleInfo() function. MSDN also states this function modifies the specified values for all applications, so to prevent potential issues in other applications running on the system we should also broadcast the WM_SETTINGCHANGE message. To broadcast the WM_SETTINGCHANGE message we will also need the Win32 PostMessage() function. Since we are Process Invocation (PInvoke) to call these unmanaged functions we should put them in a separate class that I’ve called NativeMethods. I also included all necessary constant values required by these methods in the NativeMethods class also as illustrated below.
1: namespace TestingMentor.TestTool.GlobalTester
2: {
3: using System;
4: using System.Runtime.InteropServices;
5:
6: internal sealed class NativeMethods
7: {
8: internal const int SystemDefaultLocale = (int)0x00000800;
9: internal const int BroadcastMessage = (int)0x0000FFFF;
10: internal const int SettingChangeMessage = (int)0x0000001A;
11:
12: private NativeMethods() { }
13:
14: [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
15: [return: MarshalAs(UnmanagedType.Bool)]
16: internal static extern bool SetLocaleInfo(
17: int locale,
18: int localeType,
19: string localeData);
20:
21: [DllImport("user32.dll", SetLastError = true)]
22: [return: MarshalAs(UnmanagedType.Bool)]
23: internal static extern bool PostMessage(
24: int handle,
25: int message,
26: IntPtr wParam,
27: IntPtr lParam);
28: }
29: }
The class for the custom wrapper method is TestingMentor.TestTool.GlobalTester.SetDateFormat. There is a public enumeration for the short date and long date constants. One of these values must be assigned to the SetDateType property. The other property that must be set is the SetDateFormatPicture. The big change in the SetLocaleInfo() function is that the lcData type is a null-terminated string that MSDN refers to as a format picture. Current versions of Windows allow users to customize the order of the month, day and year, the format for each, and even allow different separators between the date elements. The format picture enables the user to select various format types in different orders for either the short date or the long date. See MSDN’s Month, Day, Year and Era Format Pictures for the various supported format types.
1: namespace TestingMentor.TestTool.GlobalTester
2: {
3: using System;
4:
5: public enum DateFormatType
6: {
7: ShortDate = 0x0000001F,
8: LongDate = 0x00000020
9: }
10:
11: public class CustomDateFormat
12: {
13: private string dateFormatPicture = string.Empty;
14: private int dateType = (int)DateFormatType.ShortDate;
15:
16: public string SetDateFormatPicture
17: {
18: set { this.dateFormatPicture = value; }
19: }
20:
21: public int SetDateType
22: {
23: set
24: {
25: if (value == (int)DateFormatType.ShortDate ||
26: value == (int)DateFormatType.LongDate)
27: {
28: this.dateType = value;
29: }
30: else
31: {
32: throw new ArgumentOutOfRangeException("Invalid DateType");
33: }
34: }
35: }
36:
37: public bool ChangeDateFormat()
38: {
39: bool success = false;
40: if (NativeMethods.SetLocaleInof(
41: NativeMethods.SystemDefaultLocale,
42: this.dateType,
43: this.dateFormatPicture))
44: {
45: NativeMethods.PostMessage(
46: NativeMethods.BroadcastMessage,
47: NativeMethods.SettingChangeMessage,
48: IntPtr.Zero,
49: IntPtr.Zero);
50: }
51:
52: return success;
53: }
54: }
55: }
Once the SetDateType and SetDateFormatPicture properties are assigned we simply have to call ChangeDateFormat() method to change the settings and broadcast the message to the system. The code snippet below illustrates how a tester would change the default long date format in an automated test to determine globalization support in the application under test. Customizing the date format is useful if the application under test uses a date string in any way. For example, if the application includes a function to insert a date string in an edit control, or if the date is printed as a header or footer in a document, or if a date string is appended to a record.
1: using TestingMentor.TestTool.GlobalTester;
2: ...
3: static void Main(string[] args)
4: {
5: CustomDateFormat date = new CustomDateFormat();
6: date.SetDateType = (int)DateType.LongDate;
7: date.SetDateFormatPicture = "[ dd % MM | yyyy ]";
8: if (date.ChangeDateFormat())
9: {
10: Console.WriteLine("Long date was changed");
11: }
12: }
Programmatically changing the date format is an easy way testers can customize date formats in their automated tests without having to manipulate the controls on Region and Language property sheet. Also note, that since the format picture is a string the order of the supported date format types is now controlled by the arrangement in the string, and the separator characters can be different between the day and month and the month and year as illustrated in the example above.
Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle.
Next week I will discuss customizing the time format. Also, check out the beta release of the GlobalTester automation library that has this functionality and more and let me know what you think.
Complex != Better
I know all about over-engineering. I previously wrote about a barn my father designed and built using telephone poles and oak planks and pallets for the stall walls. From a structural perspective this barn would have stood virtually anything mother nature could have thrown at it. And if someday people wanted to tear down that barn they would surely curse the builders because the job would be much harder then they expect.
I sometimes find code that is way over-engineered. In some cases it may be necessary for making the code more robust. But, over-engineered code may also result from ignorance of more efficient algorithms or design patterns. And, sometimes overly complex algorithms are a result of developers trying to craft something that obfuscates the simplicity of the solution and fool others into believing the complexity of the solution is somehow better than a more simple solution.
I sometimes see this in test code. I am a firm believer in robust, ‘bullet-proof’ test code because each time an automated test case throws a false positive or ‘breaks’ the team loses confidence in the automation project, and it takes our time to ‘massage’ the test back to health. But, there seems to be 2 extremes in test automation. Simplistic prescriptive scripted tests with a bunch of hard-coded ‘test-data,’ or overly complex test code that is virtually undecipherable by anyone other than the original developer that renders any downstream maintenance virtually impossible. I have seen many instances where complete libraries of automation was scrapped and re-developed simply because it was easier to re-write the code rather than trying to wade through the quagmire of complexity.
In a recent example, some of my students submitted their test automation projects with a library that contained a method to generate a random string. When I first looked at their method for generating a random string I was a bit perplexed. Besides the fact that the method only produced a grand total of 26 upper case characters, the code was much more complicated than necessary. Despite having showed them how to use the Babel random string generator they choose to make their own, so I asked them how they came up with this solution and they said “they searched on the Internet. ” Others in the class indicated they also used the same method in their projects. So, I when I got home I did a quick search and found the code sample.
1: private static string RandomString(int size)
2: {
3: StringBuilder builder = new StringBuilder();
4: Random random = new Random();
5:
6: char ch;
7: for (int i = 0; i < size; i++)
8: {
9: ch = Convert.ToChar(
10: Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));
11: builder.Append(ch);
12: }
13:
14: return builder.ToString();
15: }
OK…for a moment let’s overlook the fact that if we make 2 consecutive calls to this method we will get the same identical random string of characters (which the author of this code also discovered), and let’s assume that the range of characters is limited to upper case A through Z, and let’s also ignore the fact that we are not seeding our Random generator for reproducibility of the randomly generated output from this method.
Let’s focus on the statement in lines 9 and 10. Why in the world would we generate a number between 0.0 and 1.0, multiply it by 26 and add 65, and then use Math.Floor to return the largest integer less then or equal to the resultant double, then convert that double to a type Int32, and then convert the Int32 to a type char? Now I ask you, can we make this any more complicated?
Now, I am not critiquing the style of the code or its inherent limitations, but this solution is an example of over-engineering. As I have said before, complexity cultivates chaos. At first I thought maybe this approach might give me a better distribution of characters, but when I tested this hypothesis it simply didn’t pan out. So, the way the random character is generated is simply too complex. An easier, more readable, and effective alternative might be to simply generate a random integer within the allowable range and cast that int to a type char as illustrated below in line 10. (Yes, I realize the limitations with this approach if trying to generate surrogate pair characters above U+FFFF.)
1: private const int minCharacter = 65;
2: private const int maxCharacter = 91;
3: private static string SimpleRandomString(int size)
4: {
5: StringBuilder sb = new StringBuilder();
6: System.Threading.Thread.Sleep(1);
7: Random r = new Random();
8: for (int i = 0; i < size; i++)
9: {
10: sb.Append((char)r.Next(minCharacter, maxCharacter + 1));
11: }
12:
13: return sb.ToString();
14: }
BTW…the Sleep() in line 6 is a easy solution to preventing identical return values for consecutive calls. But, a more effective solution would be to pass in a seed value as a parameter to the method and use that to seed the new Random generator as illustrated below. Different seeds not only guarantee randomness in the resultant string, but also allow for repeatability if necessary as long as the seed value is preserved.
1: private static string SimpleRandomString(int size, int seed)
2: {
3: StringBuilder sb = new StringBuilder();
4: Random r = new Random(seed);
5: ...
6: }
But, I digress. This post isn’t about random generation or style…it’s about complexity. Overly complex code tends to harbor errors that might go undetected (at least initially). Also, maintenance of complex code not only becomes more problematic, it often leads to costly re-writes down the road.
If we consider that greater complexity may increase the likelihood of error, then we don’t want our development partners to unnecessarily over-engineered algorithms. Also, overly complex solutions often require overly complex testing. This not only leads to increased testing costs, but it also increases the probability of an important test being missed or overlooked. Think testability!
Finally, with regard to test automation we should consider that our automated tests might be reused by other teams, and they certainly will be used during maintenance or sustained engineering efforts well after the product has released. And, in some cases, the people maintaining the product might not be the same people who shipped the product. Well-written automated tests are not just reviewable by someone other than the author of the code, but they should also be easily maintainable. Otherwise, we might end up paying double for that automated test case. Ouch!
Do I Really Need To Automate This Test?
For the past 2 weeks my students in my automation course at University of Washington have been tasked with designing automated test cases through the GUI for a shareware program. In my opinion, GUI automation is the least effective approach for testing the functional or business logic of a program (assuming a well designed architecture where the business logic code is separate from the form (GUI) and the event hander (GUI object behavior) code). However, if a tester doesn’t have access to the underlying APIs used in the application under test (AUT) and is given just a compiled application (‘GUI application’) to test then functional testing through the GUI may be the only alternative.
GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.
However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’) But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’
There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?
First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.
So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”
The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.
Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:
and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.
But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”
“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”
I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.
Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.
This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)
Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”
Programmatically Detecting The Operating System Version (Part II)
Time is a commodity in short supply! It has been more than 2 weeks since my last post. I have not been sitting idle, but really haven’t had a lot of free time to write. In preparation for my up-coming trip to Zurich, Switzerland to give a workshop and keynote at Swiss Testing Day I was interviewed by Marco van der Spek for TESTNIEUWS.NL. The interview provides a bit more background about me and some of my perspectives of Microsoft and software testing.
Also sucking up some of my time over the past couple of weeks were ‘reorgs’ at work. Change at Microsoft is a constant. Most people get used to it after awhile; others still freak out even with the slightest change. For me it mostly means shifting some priorities based on the new General Manager’s strategic vision, acclimating to a new manager and letting him know what I am working on, and generally making sure the day to day business on our team keeps moving forward during the transition.
Just like life at Microsoft and technology in general the Windows operating system continuously goes through changes. New versions, new service packs, Ultimate, Home, Server versions, etc. Sometimes it is hard to keep up with all the changes. And from a test automation perspective it is sometimes important to know which operating system version the test is running on. In some cases control flow in the automated test case may need to branch in order for the test to execute on different operating system versions. Branching in an automated test based on the operating system version eliminates the need to write separate test cases for variances in operating system versions.
And, certainly if our test matrix includes multiple product versions (Home, Ultimate, Professional, etc) and our automated test exposes a bug then we certainly want to collect information about the operating system version the test was running on. This is especially important if the automated test fails on one machine, but passes on another. Sometimes the cause of the failure may be a slight difference in the machine configuration or the operating system version.
Almost 2 years ago I described a way to get the operating system version information using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members in this blog post. But, I also mention some limitations such as detecting the specific edition or product type of a Windows Version. Another limitation is the difficulty in detecting whether the operating system version is Windows 7 or Windows 2008 Server R2.
I also mentioned that in order to identify a particular version and/or edition of Windows we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function. But, to use these Win32 APIs we need to use platform invocation services (P/Invoke) to marshal native code into our managed test code.
The code snippet below illustrates the required Win32 function marshaled using the DLLImport attribute in C#, constant values, wrapper methods to get common operating system information, and public properties to get the operating system version, any installed service pack information, and the operating system edition for Windows Vista, Windows Server 2008, and Windows 7.
1: // <copyright file = VersionInfo.cs" company = "Testing Mentor">
2: // Copyright © 2010 All Rights Reserved. Test developers can simply copy and
3: // paste the code into their code, but may not reproduce or publish the code
4: // snippets on any web site, online service, or distribute as source on any
5: // media without express written permission. </copyright>
6:
7: namespace TestingMentor.Snippet.OperatingSystemVersionInfo
8: {
9: using System;
10: using System.Runtime.InteropServices;
11:
12: public class WindowsVersionInfo
13: {
14: public string GetOSVersion
15: {
16: get { return this.GetOSVersionInfo(); }
17: }
18:
19: public string GetServicePack
20: {
21: get { return this.GetServicePackInfo(); }
22: }
23:
24: public string GetProductType
25: {
26: get { return this.GetProductTypeInfo(); }
27: }
28:
29: private string GetOSVersionInfo()
30: {
31: string version = "Unsupported Version";
32:
33: NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();
34: osvi.VersionInfoSize =
35: Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
36: NativeMethods.GetVersionEx(ref osvi);
37:
38: if (OsviConstant.SupportedPlatform == osvi.PlatformId &&
39: osvi.MajorVersion > 4)
40: {
41: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
42: osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows2000)
43: {
44: version = "Windows 2000";
45: }
46:
47: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
48: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsXP)
49: {
50: version = "Windows XP";
51: }
52:
53: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
54: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsServer2003)
55: {
56: if (osvi.ProductType == (byte)OsviConstant.WorkStation)
57: {
58: version = "Windows XP Professional x64";
59: }
60: else
61: {
62: version = "Windows Server 2003";
63: if (NativeMethods.GetSystemMetrics(OsviConstant.ServerR2) != 0)
64: {
65: version += " R2";
66: }
67: }
68: }
69:
70: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&
71: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsVista)
72: {
73: if (osvi.ProductType ==
74: (byte)OsviConstant.WorkStation)
75: {
76: version = "Windows Vista";
77: }
78: else
79: {
80: version = "Windows Server 2008";
81: }
82: }
83:
84: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&
85: osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows7)
86: {
87: if (osvi.ProductType == (byte)OsviConstant.WorkStation)
88: {
89: version = "Windows 7";
90: }
91: else
92: {
93: version = "Windows Server 2008 R2";
94: }
95: }
96: }
97:
98: return version;
99: }
100:
101: private string GetServicePackInfo()
102: {
103: NativeMethods.OSVersionInfoEx versionInfo = new NativeMethods.OSVersionInfoEx();
104: versionInfo.VersionInfoSize = Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
105: NativeMethods.GetVersionEx(ref versionInfo);
106: return versionInfo.CSDVersion;
107: }
108:
109: private string GetProductTypeInfo()
110: {
111: string product = String.Empty;
112:
113: NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();
114: osvi.VersionInfoSize =
115: Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
116: NativeMethods.GetVersionEx(ref osvi);
117:
118: if (osvi.MajorVersion > 5)
119: {
120: uint productType = 0;
121:
122: NativeMethods.GetProductInfo(
123: osvi.MajorVersion,
124: osvi.MinorVersion,
125: osvi.ServicePackMajor,
126: osvi.ServicePackMinor,
127: ref productType);
128:
129: switch (productType)
130: {
131: case (uint)OsviConstant.ProductInfo.Business:
132: product = "Business Edition";
133: break;
134: case (uint)OsviConstant.ProductInfo.BusinessN:
135: product = "Business N Edition";
136: break;
137: case (uint)OsviConstant.ProductInfo.ClusterServer:
138: product = "HPC Edition";
139: break;
140: case (uint)OsviConstant.ProductInfo.DatacenterServer:
141: product = "Server Datacenter (Full)";
142: break;
143: case (uint)OsviConstant.ProductInfo.DatacenterServerCore:
144: product = "Server Datacenter (Core)";
145: break;
146: case (uint)OsviConstant.ProductInfo.DataCenterServerCoreV:
147: product = "Server Datacenter without Hyper-V (Core)";
148: break;
149: case (uint)OsviConstant.ProductInfo.DataCenterServerV:
150: product = "Server Datacenter without Hyper-V (Full)";
151: break;
152: case (uint)OsviConstant.ProductInfo.Enterprise:
153: product = "Enterprise Edition";
154: break;
155: case (uint)OsviConstant.ProductInfo.EnterpriseE:
156: product = "Enterprise E Edition";
157: break;
158: case (uint)OsviConstant.ProductInfo.EnterpriseN:
159: product = "Enterprise N Edition";
160: break;
161: case (uint)OsviConstant.ProductInfo.EnterpriseServer:
162: product = "Server Enterprise (Full)";
163: break;
164: case (uint)OsviConstant.ProductInfo.EnterpriseServerCore:
165: product = "Server Enterprise (Core)";
166: break;
167: case (uint)OsviConstant.ProductInfo.EnterpriseServerCoreV:
168: product = "Server Enterprise without Hyper-V (Core)";
169: break;
170: case (uint)OsviConstant.ProductInfo.EnterpriseServerIA64:
171: product = "Server Enterprise for Itanium-based Systems";
172: break;
173: case (uint)OsviConstant.ProductInfo.EnterpriseServerV:
174: product = "Server Enterprise without Hyper-V (Full)";
175: break;
176: case (uint)OsviConstant.ProductInfo.HomeBasic:
177: product = "Home Basic Edition";
178: break;
179: case (uint)OsviConstant.ProductInfo.HomeBasicE:
180: product = "Home Basic E Edition";
181: break;
182: case (uint)OsviConstant.ProductInfo.HomeBasicN:
183: product = "Home Basic N Edition";
184: break;
185: case (uint)OsviConstant.ProductInfo.HomePremium:
186: product = "Home Premium Edition";
187: break;
188: case (uint)OsviConstant.ProductInfo.HomePremiumE:
189: product = "Home Premium E Edition";
190: break;
191: case (uint)OsviConstant.ProductInfo.HomePremiumN:
192: product = "Home Premium N Edition";
193: break;
194: case (uint)OsviConstant.ProductInfo.HomeServer:
195: product = "Home Server Edition";
196: break;
197: case (uint)OsviConstant.ProductInfo.HyperV:
198: product = "Microsoft Hyper-V Server";
199: break;
200: case (uint)OsviConstant.ProductInfo.MediumBusinessServerManagement:
201: product = "Windows Essential Business Server Management Server";
202: break;
203: case (uint)OsviConstant.ProductInfo.MediumBusinessServerMessaging:
204: product = "Windows Essential Business Server Messaging Server";
205: break;
206: case (uint)OsviConstant.ProductInfo.MediumBusinessServerSecurity:
207: product = "Windows Essential Business Server Security Server";
208: break;
209: case (uint)OsviConstant.ProductInfo.Professional:
210: product = "Professional Edition";
211: break;
212: case (uint)OsviConstant.ProductInfo.ProfessionalE:
213: product = "Professional E Edition";
214: break;
215: case (uint)OsviConstant.ProductInfo.ProfessionalN:
216: product = "Professional N Edition";
217: break;
218: case (uint)OsviConstant.ProductInfo.ServerForSmallBusiness:
219: product = "Windows Server 2008 for Windows Essential Server Solutions";
220: break;
221: case (uint)OsviConstant.ProductInfo.ServerForSmallBusinessV:
222: product = "Windows Server 2008 without Hyper-V for Windows Essential Server Solutions";
223: break;
224: case (uint)OsviConstant.ProductInfo.ServerFoundation:
225: product = "Server Foundation";
226: break;
227: case (uint)OsviConstant.ProductInfo.SmallBusinessServer:
228: product = "Windows Small Business Server";
229: break;
230: case (uint)OsviConstant.ProductInfo.SmallBusinessServerPremium:
231: product = "Windows Small Busines Server Premium";
232: break;
233: case (uint)OsviConstant.ProductInfo.StandardServer:
234: product = "Server Standard (Full)";
235: break;
236: case (uint)OsviConstant.ProductInfo.StandardServerCore:
237: product = "Server Standard (Core)";
238: break;
239: case (uint)OsviConstant.ProductInfo.StandardServerCoreV:
240: product = "Server Standard without Hyper-V (Core)";
241: break;
242: case (uint)OsviConstant.ProductInfo.StandardServerV:
243: product = "Server Standard without Hyper-V (Full)";
244: break;
245: case (uint)OsviConstant.ProductInfo.Starter:
246: product = "Starter Edition";
247: break;
248: case (uint)OsviConstant.ProductInfo.StarterE:
249: product = "Starter E Edition";
250: break;
251: case (uint)OsviConstant.ProductInfo.StarterN:
252: product = "Starter N Edition";
253: break;
254: case (uint)OsviConstant.ProductInfo.StorageEnterpriseServer:
255: product = "Storage Server Enterprise";
256: break;
257: case (uint)OsviConstant.ProductInfo.StorageExpressServer:
258: product = "Storage Server Express";
259: break;
260: case (uint)OsviConstant.ProductInfo.StorageStandardServer:
261: product = "Storage Server Standard";
262: break;
263: case (uint)OsviConstant.ProductInfo.StorageWorkgroupServer:
264: product = "Storage Server Workgroup";
265: break;
266: case (uint)OsviConstant.ProductInfo.Ultimate:
267: product = "Ultimate Edition";
268: break;
269: case (uint)OsviConstant.ProductInfo.UltimateE:
270: product = "Ultimate E Edition";
271: break;
272: case (uint)OsviConstant.ProductInfo.UltimateN:
273: product = "Ulitmate N Edition";
274: break;
275: case (uint)OsviConstant.ProductInfo.Undefined:
276: product = "Unknown Product";
277: break;
278: case (uint)OsviConstant.ProductInfo.Unlicensed:
279: product = "Unlicensed or Expired";
280: break;
281: case (uint)OsviConstant.ProductInfo.WebServer:
282: product = "Web Server (Full)";
283: break;
284: case (uint)OsviConstant.ProductInfo.WebServerCore:
285: product = "Web Server (Core)";
286: break;
287: }
288: }
289:
290: return product;
291: }
292: }
293:
294: // ****************************************************************************
295: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE
296: // ****************************************************************************
297:
298: internal class OsviConstant
299: {
300: internal const int SupportedPlatform = 2;
301: internal const int ServerR2 = 89;
302: internal const int WorkStation = 0x00000001;
303:
304: private OsviConstant()
305: {
306: }
307:
308: internal enum MajorVersion
309: {
310: NT5 = 5,
311: NT6 = 6
312: }
313:
314: internal enum MinorVersion
315: {
316: Windows2000 = 0,
317: WindowsXP = 1,
318: WindowsServer2003 = 2,
319: WindowsVista = 0,
320: Windows7 = 1
321: }
322:
323: internal enum ProductInfo : uint
324: {
325: Business = 0x00000006,
326: BusinessN = 0x00000010,
327: ClusterServer = 0x00000012,
328: DatacenterServer = 0x00000008,
329: DatacenterServerCore = 0x0000000C,
330: DataCenterServerCoreV = 0x00000027,
331: DataCenterServerV = 0x00000025,
332: Enterprise = 0x00000004,
333: EnterpriseE = 0x00000046,
334: EnterpriseN = 0x0000001B,
335: EnterpriseServer = 0x0000000A,
336: EnterpriseServerCore = 0x0000000E,
337: EnterpriseServerCoreV = 0x00000029,
338: EnterpriseServerIA64 = 0x0000000F,
339: EnterpriseServerV = 0x00000026,
340: HomeBasic = 0x00000002,
341: HomeBasicE = 0x00000043,
342: HomeBasicN = 0x00000005,
343: HomePremium = 0x00000003,
344: HomePremiumE = 0x00000044,
345: HomePremiumN = 0x0000001A,
346: HyperV = 0x0000002A,
347: MediumBusinessServerManagement = 0x0000001E,
348: MediumBusinessServerSecurity = 0x0000001F,
349: MediumBusinessServerMessaging = 0x00000020,
350: Professional = 0x00000030,
351: ProfessionalE = 0x00000045,
352: ProfessionalN = 0x00000031,
353: ServerForSmallBusiness = 0x00000018,
354: ServerForSmallBusinessV = 0x00000023,
355: ServerFoundation = 0x00000021,
356: SmallBusinessServer = 0x00000009,
357: StandardServer = 0x00000007,
358: StandardServerCore = 0x0000000D,
359: StandardServerCoreV = 0x00000028,
360: StandardServerV = 0x00000024,
361: Starter = 0x0000000B,
362: StarterE = 0x00000042,
363: StarterN = 0x0000002F,
364: StorageEnterpriseServer = 0x00000017,
365: StorageExpressServer = 0x00000014,
366: StorageStandardServer = 0x00000015,
367: StorageWorkgroupServer = 0x00000016,
368: Undefined = 0x00000000,
369: Ultimate = 0x00000001,
370: UltimateE = 0x00000047,
371: UltimateN = 0x0000001C,
372: WebServer = 0x00000011,
373: WebServerCore = 0x0000001D,
374: Unlicensed = 0xABCDABCD,
375: HomeServer = 0x00000013,
376: SmallBusinessServerPremium = 0x00000019,
377: }
378: }
379:
380: // ****************************************************************************
381: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE
382: // ****************************************************************************
383:
384: internal class NativeMethods
385: {
386: private NativeMethods()
387: {
388: }
389:
390: [DllImport("kernel32")]
391: [return: MarshalAs(UnmanagedType.Bool)]
392: internal static extern bool GetVersionEx(ref OSVersionInfoEx osvi);
393:
394: [DllImport("kernel32.dll")]
395: [return: MarshalAs(UnmanagedType.Bool)]
396: internal static extern bool GetProductInfo(
397: int osMajorVersion,
398: int osMinorVersion,
399: int spMajorVersion,
400: int spMinorVersion,
401: ref uint type);
402:
403: [DllImport("kernel32.dll")]
404: internal static extern int GetSystemMetrics(
405: int index);
406:
407: [StructLayout(LayoutKind.Sequential)]
408: internal struct OSVersionInfoEx
409: {
410: public int VersionInfoSize;
411: public int MajorVersion;
412: public int MinorVersion;
413: public int BuildNumber;
414: public int PlatformId;
415: [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]
416: public string CSDVersion;
417: public Int16 ServicePackMajor;
418: public Int16 ServicePackMinor;
419: public Int16 SuiteMask;
420: public byte ProductType;
421: public byte Reserved;
422: }
423: }
424: }
Now, some of my readers have indicated that these code snippets are not very useful because they can’t copy them directly and put them to use. So, to help resolve that issue I have created a new section on my web site called the Code Snippet Library. This snippet is posted there along with fully annotated (mostly FxCopy and StyleCopy compliant) file available for download or to copy for inclusion in your automated test cases, or compiled into a dynamic link library (DLL).
This example doesn’t differentiate between 32-bit and 64-bit Windows operating systems, but that is not really difficult to add, and if I get enough requests I will certainly add that into the pot. If the operating system version is no longer supported by Microsoft the GetOsVersion property will return "Unsupported Version." If no Service Packs are installed the GetServicePack property will return an empty string. If you need to detect an unsupported operating system version use the example here.
Scary Stories and GUI Automation
I remember going camping with my cousins as I was growing up. It was great fun despite sleeping inside a musty smelling canvas tent that retained heat so well it was more like a sauna. But, my father was adamant that the old canvas tent he bought at an Army Surplus store and took 2 men and 4 boys to carry and assemble was much better then those new fangled nylon tents. Nylon ripped too easily he reasoned, but canvas will withstand anything short of a raging bear. I’m not too sure there were too many raging bears roaming the camp-grounds of Maryland, Pennsylvania, and Virginia but I was confident that even if there were I would have stood a better chance inside a canvas tent as compared to those paper thin nylon tents. To this day I remember those camping trips when I smell old canvas, or perhaps it’s dried mold spores embedded in the canvas. Whichever, it takes me back to a time of fun and fond memories.
One of the best parts of the trips were sitting around the camp fire at night listening to my uncle concocting some story intended to scare the wits out of us young boys. You know, the kind of stories about headless Confederate soldiers, or werewolves, vampires, or other such wicked creatures of the night. I think these campfire chats are remnants of man’s tribal roots where the elders tried to scare the hell out of the juvenile hominids to prevent them from wandering off at night. As we got older we realized that these stories were simply fictitious folktales; sort of like successful, value-add GUI automation projects.
Last week I had lunch with a colleague who wanted to talk to me about an automation project on his team that went horribly awry. As he started to tell me his story I thought, “Wait…I heard this tale before. I can tell this story because I’ve heard it so many times over and over again…just like those scary stories I heard around the campfire growing up.” The story goes something like this.
Our team bought a new tool, or built another framework, and taught everyone how to script “black box” test cases. They developed quite a number of automated test scripts, and of course everything was working very well. The scripts were running and managers were happy because the team had a lot of automated test scripts. But, the tests weren’t finding any bugs, so just out of curiosity the managers suggested a bug bash. And sure enough as the testers started exploring the project it didn’t take long for them to fill the database with bugs. The developers were shell shocked, and the managers couldn’t believe it! They couldn’t understand why the automated GUI tests weren’t finding any bugs? And so, in a typical knee-jerk reaction fashion, the managers immediately halted the GUI automation project and required every tester to embark on an exploratory testing adventure in search of bugs. Of course, the managers decided this approach was better than investing in more GUI automation bringing an end to another GUI automation project.
Unfortunately, unlike the scary stories my father and uncles told around the campfires, stories of failed GUI automation are often true, and usually much scarier. Why are they so scary? Because I hear these sorts of stories repeated so often. It seems that we as a discipline rely on tribal knowledge where each generation simply learns through trial and error and the folktales of our elders and thrive more on hero worship of people who are often remarkably good at finding bugs by poking and prodding hour upon hour.
Now, if you haven’t caught on already you will know that I am no big fan of GUI automation. Not because I don’t think it can be useful. In fact, I think GUI automation can provide tremendous value in some situations. But unfortunately much of the automated GUI test cases I see (especially in examples) are poorly designed, simple rudimentary script-lets. Many of these automated tests are nothing more than mindless automated sequences of events contrived because the testers have been told to automate, but are not given strategic vision (why) and little to no tactical direction (what and how).
With little or no direction or goals, or without an in-depth understanding of the system they are testing the biggest problems with GUI automation is that many testers attempt to automate
- functional tests intended to expose computational errors in the business logic layer or in the underlying APIs
- usability tests intended to imitate ‘me’ trying to emulate the scenarios or tasks I think the customer might do
GUI automation is probably the least effective approach for functional testing. This is not to say that GUI automation will not find functional issues in the lower logic layers. I suspect we will always find ‘functional’ bugs (e.g. boundary issues, unhandled exceptions, string parsing errors, calculation problems, etc.) while testing through the UI. But, as indicated in my previous post, well-designed software is usually built in layers and a good many of the ‘functional’ issues we find today can likely be more efficiently found through more robust unit and component (API) levels of testing.
Perhaps even more silly than trying to use GUI automation for low level functional testing is the notion of using GUI automation to emulate a ‘user’ by scripting out prescriptive sequences of actions (often with hard-coded data) that are then played over and over again. Test automation cannot and should not attempt to replicate ‘me.’ I’ve said before the purpose of automation is to provide value to me, to free up my time, to increase my efficiency, and to help me be more effective in my job; automation does not replace me. Let’s face it…we (humans) are much better at evaluating the ease of use of software and whether scenarios that represent target customer segments are intuitive for those customers.
For example, I once had a conversation explaining that GUI automation runs much faster than I can interact with software, and sometimes I make mistakes when typing in something that throws an unexpected message or takes me down a path. My colleague replied, “Well, we can slow down the automation.” Why? Why in the hell would I want to slow down my automated tests? C’mon…we all should know by now that the 100% automation (or automate all tests) mantra is a ridiculous dream and I’ve heard more plausible fantasies from people on acid trips.
So, where does GUI automation add value. In my opinion, GUI automation is probably most effective in testing UI control properties and the event handlers between the UI layer and the API layers. It is also effective in behavioral testing areas such as performance and stress. And GUI automation is also much more effective in evaluating UI layout issues such as misaligned controls, or clipped or truncated controls on a window as compared to the human eye.
Similar to how we use different techniques to expose different categories of defects, and how we use different approaches to testing depending on the context or test objective test automation is a useful tool in our toolbox. It is certainly not the only tool. We can do some remarkable things with automation, but we must learn where GUI automation adds value and where other approaches testing (automated or manual) might be more effective.
API Testing – Thinking Differently About the Problem
Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.
A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).
So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.
Anyway, the final project after the first 10 weeks of the course is to design automated tests of 3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.
I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.
Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.
For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.
1: // <copyright file="simpletestcase.cs" company="TestingMentor">
2: // Copyright © 2009 by Bj Rollison. All rights reserved.
3: // </copyright>
4:
5: namespace TestingMentor.Sample
6: {
7: using System;
8: using System.IO;
9: using TestingMentor.Simulation;
10:
11: class TestCase
12: {
13: static void Main(string[] args)
14: {
15: int testCounter = 0;
16: // Read in an array of strings representing the test data.
17: // Of course this would likely come from a static test data file
18: // on a server or copied to a folder on the local machine
19: string[] testData = new string[]
20: { "90,Z",
21: "24798,惞",
22: "0,null",
23: "65536,Error",
24: "-1,Error",
25: "1.5,",
26: "xyz,xyz"
27: };
28:
29: // Loop through each test data string
30: foreach (string test in testData)
31: {
32: testCounter++;
33: // This nested try/catch block catches invalid test data
34: // but allow additonal tests in the testData array
35: try
36: {
37: // Parse each string into the test data and expected result
38: string[] testElement = test.Split(',');
39: string expectedResult = testElement[1];
40: string actualResult = String.Empty;
41:
42: // Convert the string to a type int value
43: int value = int.Parse(testElement[0]);
44:
45: // We need a way to handle int values 0 through 32 which are
46: // control characters, this is an example of how to deal with
47: // a int value of 0 which is a null character
48: if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))
49: {
50: expectedResult = '\0'.ToString();
51: }
52:
53: // This nested try/catch block tests catches exceptions thrown by
54: // the method under test. If the method under test throws an
55: // exception we certainly want to test for that case!
56: try
57: {
58: // Call the API method under test
59: char result = Converter.ConvertValueToUnicodeChar(value);
60: actualResult = result.ToString();
61: }
62:
63: catch (ArgumentOutOfRangeException)
64: {
65: actualResult = "Error";
66: }
67:
68: catch (Exception)
69: {
70: // if this happens this is a failure because the documentation
71: // states that this method will only throw an
72: // ArgumentOutOfRangeException.
73: actualResult = "Non-specific or unexpected error thrown";
74: }
75:
76: // Call a simple oracle and log results
77: if (String.Equals(actualResult, expectedResult))
78: {
79: // log pass
80: Console.WriteLine("{0} Pass", testCounter);
81: }
82: else
83: {
84: // log fail...of course log as much detail as possible
85: Console.WriteLine("{0} Fail", testCounter);
86: }
87: }
88:
89: catch (FormatException)
90: {
91: // log the test data for this test as incorrect, test is skipped
92: Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);
93: }
94: }
95: }
96: }
97: }
Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).
Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)
The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?
Random Test Data – Credit Card Numbers
Things are winding down for the year. The Christmas lights are up on the house, my gardens are tilled and mulched for next spring, people are disappearing from the office like there is a plague, it hasn’t snowed in a while which means the mountains are mostly ice (I dislike skiing on ice), the next GSHL ice hockey league doesn’t start for awhile and pick up games are few and far between (I suck at hockey but it is fun). So, what to do? Oh…I forgot Christmas shopping. I hate Christmas shopping! So, I have spent the past few idle nights refactoring the automation libraries for some of my test data generation tools after my daughter goes to bed.
One of the most popular random test data generators that I have developed so far has been a tool called CCMaker to generate random valid and invalid credit card numbers. (Sometimes I wonder why that is, but I don’t dwell on it for too long and I haven’t been interrogated by the FBI lately.) Testing forms that require a credit card has always been risky business because you certainly don’t want to use your own card. Often times developers will include a check on web forms or client apps to do a high level verification of a credit card number before sending all the data across the wire to be validated. This early or high level verification prevents flooding the pipe with bad data. So, one test we can do prior to testing the end-to-end scenario is to test to see if and how the developer is validating credit cards numbers prior to submission.
As far as data goes, generating credit card numbers are fairly simple. There is a bank ID number (BIN), there is a number of digits between 12 and 19 depending on the card type, and there is a checksum. So, if we know the valid BINs for each issuing bank, the valid number of digits for each card type, and how to calculate the checksum we can generate valid credit card numbers. (Of course this is a bit oversimplified because many credit and debit card companies are issued multiple BINs and use varying number lengths.)
Testing for invalid credit card numbers should include using numbers that look close to being correct in some way but are slightly altered. For example for the 3 defined equivalent partitions (BIN, length, checksum) there are seven possible invalid combinations (23 – 1) we could test.
- Valid BIN, invalid length and valid checksum
- Valid BIN, valid length, and invalid checksum
- Valid BIN, invalid length, and invalid checksum
- Invalid BIN, valid length and valid checksum
- Invalid BIN, invalid length, and valid checksum
- Invalid BIN, valid length, and invalid checksum
- Invalid BIN, checksum and length
This doesn’t mean I run 7 tests and call it good because there are numerous invalid lengths and invalid BINs for the different card types. A common mistake when using an equivalent partition testing approach is to simply plug in values for each combination listed above and call it good. The problem is that there are several hundred BINs and 8 different valid lengths. For example, for just the Discover card there are 829 valid BIN numbers, and for the Maestro cards there are 56 combinations of BINs and card lengths ranging from 12 to 19 numbers in length. This doesn’t include the permutations of the other numbers that compose the entire card number.
The question every tester must ask him or herself every day when designing tests is how many tests do I need to have any reasonable sense of confidence that risk is minimal and the perception of quality is high. Of course, there is no single right answer here and not magic formula, but since we can’t possibly execute every possible positive or negative test we should at least understand that ultimately testing is sampling.
For example, one strategy for positive testing might be to test every valid BIN for every valid card length for any given credit card. For example for American Express I would want to test at least one number with a BIN of 34 and a card length of 15 that satisfies the checksum requirement, and at least one number with a BIN of 37 and a card length of 15 that also satisfies the checksum requirement. For a card type of Visa I would need a minimum of 2 tests in which the BIN is 4, the checksum requirement is satisfied, and one has a card length of 13 numbers and the other has a card length of 16 numbers.
That probably sounds like quite a bit of testing, and tests which most likely would not produce an error (unless of course the BIN is miss identified (e.g. instead of checking for a BIN of 5020 the BIN is incorrectly assigned as 5002), or if a valid BIN is not recognized as valid because it is omitted from a list or enumeration of valid BINs for that credit card). Certainly testing of this magnitude would be expensive if done manually. But when automated using a random test data generator and a data-driven automation approach to set the random generator properties comprehensive testing becomes a much more reasonable proposition and can significantly increase overall confidence.
This is where my CCMaker 3.1 test data generator can help by randomly generating both valid and invalid credit card numbers. The updated CCMaker test automation library has just been posted to my web site with documentation and examples. If you have any questions, or find any issues with the new library please let me know.
Randomizing Static Test Data in Automated Tests
Originally Published Sunday, October 11, 2009
A significant percentage of static test data is stored in tabular comma delimited or tab-delimited formats and saved in Excel spreadsheets. Reading in comma or tab-delimited static test data into an automated test is pretty straight forward and there are numerous examples in many programming languages illustrating how to read in these types of test data repositories. Reading in rows of data is the foundation of data-driven automation and definitely has its place in any automation project.
I am a big proponent of stochastic (random) test data generation that is customized to the context, but I also know that sometimes static test data is useful for establishing baselines and more exact emulation of ‘real-world’ customer-like inputs. But, if the automated test is simply passing the same variable arguments to the same input parameters in the same order over and over again the value of subsequent iterations of that automated test using that static data set diminishes rather quickly. So how can we more effectively utilize static test data in our automated tests?
One possible solution is to randomly select an argument from a collection of static variables that is passed to the specific input parameter. The advantage of this approach is that it effectively increases the test data permutations in each iteration of the test case. For example, let’s consider 2 input parameters; one for a given name and one for a surname. In a traditional data-driven approach in which the static test data is read in by rows our test data file might be similar to:
Bob,Smith
John,Johnson
Roger,Williams
Steve,Abbot
This static data file would give us 4 sets of test data, but each time the test data is read into the test case the given and surnames are always the same.
However, if we read in the given names and surnames into 2 collections, and then randomly select a given name and surname from the appropriate collection to pass to the respective parameter we effectively have 16 possible combinations of static test data to work with. An advantage of this approach is that our ‘collections’ of given names and surnames can contain differing numbers of elements (in which case the number of possible combinations of test data is the Cartesian product of the number of elements in each collection).
Of course there are many ways to accomplish this. For example, one approach is to continue to use a comma or tab-delimited file format and list given names in one row and surnames in a second row. Another approach is to list the given names and surnames in columns in a spreadsheet and read in each column into a collection of some sort. The latter is the approach I used in developing my PseudoName test data generator tool. I chose this approach for 2 reasons; first an Excel spreadsheet is a simple yet powerful file format for storing static test data, and secondly because lists of test data are sometimes better represented in columns rather than rows.
The following code shows one way to read in test data by columns from an Excel spreadsheet.
1: // <copyright file="datareader.cs" company="TestingMentor">
2: // Copyright © 2009 by Bj Rollison. All rights reserved.
3: // </copyright>
4:
5: namespace TestingMentor.TestTool
6: {
7: using System;
8: using System.Collections;
9: using System.Globalization;
10: using System.Runtime.InteropServices;
11: using System.Threading;
12: using Excel = Microsoft.Office.Interop.Excel;
13:
14: /// <summary>
15: /// This class contains methods for reading test data from Excel spreadsheets
16: /// </summary>
17: public class TestDataReader
18: {
19: /// <summary>
20: /// This method reads all the data elements in the specified number of
21: /// columns in the specified Excel spreadsheet containing the test data
22: /// and copies the data into a multi-dimensional array
23: /// </summary>
24: /// <param name="dataFileName">The filename containing the test data</param>
25: /// <param name="columnCount">The number of columns in the Excel
26: /// spreadsheet to read</param>
27: /// <returns>A multi-dimensional array containing the data eleements for
28: /// each column </returns>
29: public static string[][] ExcelColumnReader(string dataFileName, uint columnCount)
30: {
31: CultureInfo originalCulture = null;
32: Excel.Application excelApp = null;
33: Excel.Workbook excelWorkbook = null;
34: Excel.Worksheet excelActiveWorksheet = null;
35: string[][] testData = new string[columnCount][];
36:
37: originalCulture = Thread.CurrentThread.CurrentCulture;
38: Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");
39:
40: excelApp = new Excel.Application();
41: excelWorkbook = excelApp.Workbooks.Open(
42: dataFileName,
43: 0,
44: false,
45: 5,
46: String.Empty,
47: String.Empty,
48: false,
49: Type.Missing,
50: String.Empty,
51: true,
52: false,
53: 0,
54: true,
55: false,
56: false);
57: excelActiveWorksheet = (Excel.Worksheet)excelWorkbook.ActiveSheet;
58:
59: for (int i = 0; i < columnCount; i++)
60: {
61: // Start at column 1
62: object columnIndex = i + 1;
63:
64: // Row 1 is the column title; test data starts on Row 2
65: object rowIndex = 2;
66: ArrayList tempCollection = new ArrayList();
67: while (
68: ((Excel.Range)
69: excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2 != null)
70: {
71: tempCollection.Add(
72: ((Excel.Range)
73: excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2);
74: rowIndex = (int)rowIndex + 1;
75: }
76:
77: testData[i] = new string[tempCollection.Count];
78: testData[i] = (string[])tempCollection.ToArray(typeof(string));
79: }
80:
81: // Clean up
82: excelWorkbook.Close(false, Type.Missing, Type.Missing);
83: excelWorkbook = null;
84: excelApp.Quit();
85: excelApp = null;
86:
87: // Garbage collection is not pretty, but necessary to release Excel proc
88: System.GC.Collect();
89: System.GC.WaitForPendingFinalizers();
90:
91: if (originalCulture != null)
92: {
93: Thread.CurrentThread.CurrentCulture = originalCulture;
94: }
95:
96: return testData;
97: }
98: }
99: }
I must tell you that performance can be an issue especially if the columns contain a lot of data. For example, to read in approximately 700 elements of test data in 3 separate columns took slightly less than 1 second, and reading in 1800 elements in 3 columns required just over 4 seconds. Unfortunately, I didn’t compare total byte counts, but it is pretty obvious the greater the number of test data elements being read the longer the read operation will take and you certainly will have to take the read time into consideration in your automated test case.
Reading static test data line by line from a data file while looping through a data-driven automated test case is a useful test design approach in some situations, this is another useful approach that will allow the test designer to randomize the combinations of static test data values applied to multiple input parameters in multiple iterations of an automated test case.
Test Automation ROI (Part II)
Originally Published Wednesday, September 02, 2009
Last week I talked about the silliness of wasting time calculating the return on investment (ROI) of an automation effort on any non-trivial software project; especially if it has an extended shelf-life. As my friend Joe Strazzre commented, “If you need an ROI analysis to convince business management that test automation is a good thing when used intelligently, than you have already lost.”
But, management might need to be educated on the limitations of record/playback, rudimentary hard-coded scripts and keyword driven automation efforts because these it is often more appealing for bean counters to invest in low cost tools and continue to rely on non-coding bug finders or domain experts to script out ‘tests’ which do nothing more than repeat some rote set of steps over and over again. But, as E.Dustin, T. Garrett, and B. Gauf wrote in Implementing Automated Software Testing any serious software automation effort “is software development.” Well designed automated tests requires highly skilled, technically competent, extremely creative, analytical testers capable of designing and developing automated tests using procedural programming paradigms.
We should still apply ROI concepts in test automation, but at a much lower level. Essentially, each tester must evaluate the return on investment of any test before automating it. The most fundamental purpose of an automated test effort is to provide some perceived value to the tester, the testing effort, and the organization. As a tester, the primary reason I automate a test is to:
- Free up my time,
- Re-verify baseline assessments (BVT/BAT, regression, acceptance test suites)
- Increase test coverage (via increased breadth of data or state variability),
- Accomplish tasks that can’t easily be achieved via manual testing approaches.
For example, the build verification and build acceptance test suites are baseline tests that must be ran on each new build; these tests should be 99.9% automated because they free up my time to design other tests. Tests that evaluate a non-trivial number of combinations or permutations are generally good candidates for automation because they increase test coverage. Performance, stress, load, and concurrency tests should be heavily automated because they are difficult to conduct manually.
It is important to note that I am not simply referring to UI type automation. A significant amount of “functional tests” designed to evaluate the computational logic of methods or functions can be automated below the UI layer in software architectures using OOP and procedural paradigms where the business and computational logic is separate from the UI layer.
There are many papers that discuss specific factors to take into consideration when deciding what tests to automate. Unfortunately, there is no single cookie-cutter approach in deciding what tests to automate. Different projects have different requirements and expectations, and, of course, not all tests are equal. One of the best papers I’ve read on deciding what tests to automate is When Should a Test Be Automated by Brian Marick. I like the simplicity in his 3 key questions:
- How much more will this test cost to automate versus running it manually?
Some people think that automating a test reduces costs because it eliminates the tester from manually executing that test. Unfortunately, this is not always the case. As i talked about in a previous post, visual comparative oracles are notoriously error prone requiring the testers to constantly massage the test code and manually verify results anyway. Sometimes paying a vendor to run a test periodically is cheaper than paying an SDET to tweak the test every build. But, if the population of potential test data is large, or combinatorial testing of non-trivial features then automating that test case is probably a good investment. - What is the potential lifetime of this automated test?
How many times will this test be re-ran during the development cycle and in maintenance or sustained engineering efforts? Can this test be reused in the next iteration of the product? - Does the automated test have some probability of exposing new issues?
Although I don’t necessarily agree with this question because many automated tests may not expose new issues, but they still provide value to the overall testing effort. For example, I wouldn’t expect tests in my regression test suite to expose new defects because if they do there was a regression in the product. So, I would rephrase this question to ask, “Does this automated test have some probability of exposing new issues, providing additional information that increases confidence, or increases test coverage?”
A few other questions I ask myself when I am deciding whether to automate a particular test include:
- What exactly is being evaluated?
This is perhaps the first question I ask myself. If the test is evaluating functional or non-functional (stress, perf, security, etc.) capabilities then automation may be worthwhile. But, behavioral tests such as usability tests and content testing are generally not good candidates for automated testing. - What is the reliability of automating this test?
I don’t want to have to constantly massage a test in order to get it to run. So, what is the probability this test will throw a lot of false positives or false negatives? How much tweaking will this test require due to code or UI instability? - What are the oracles for this test and can they be automated?
I don’t want to sit in front of a computer and watch software run software. Also, there is a difference between an automated test and a macro (A single, user-defined command that is part of an application and executes a series of commands). There are different types of oracles, and the professional test designer needs to also design the most effective oracle for the test. By the way…if the most effective oracle is a human reviewing the results then that test should probably not be automated using current technologies.
For each test I consider these questions in deciding whether to automate that test. For some tests, I may ask additional questions depending on the context and the business needs of my organization. I don’t use a cookie-cutter template, or try to fill out some spreadsheet to do a cost comparison based on dollar amounts. It’s hard to put a price on value. Instead, I ask myself a few key questions to help me decide if automating a test is worth it to me, my team, and the organization. Is automating a particular test the right thing to do or am I automating something because it’s challenging, or to increase some magical percentage of automated tests compared to all currently defined tests. The key message here is not to blindly automate everything; use your brain and make smart decisions about whether each test should be automated and being able to explain how automating that test benefits the testing effort.