Archive for the ‘Test Automation’ tag
Test Automation: Beyond Rudimentary Script-lets
When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.
Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.
The purpose of the test
Think about abstract big picture rather than a myopic singular purpose
In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.
The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.
As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.
Automation perspective
Think about programming the computer rather than trying to program human interactions
In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.
In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.
But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.
Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.
Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully
Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.
Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.
Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.
The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.
Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.
1: namespace HelloTest
2: {
3: using System;
4: using System.Diagnostics;
5: using System.IO;
6: using System.Runtime.InteropServices;
7: using System.Windows.Automation;
8: using TestingMentor.TestTool.Babel;
9:
10: internal class Constant
11: {
12: internal const string AutName = "hello.exe";
13: internal const string AutFormTitle = "HelloForm";
14: internal const string AutNameTextBox = "textboxFirstName";
15: internal const string AutOKButton = "buttonOK";
16: internal const string AutClearButton = "buttonClear";
17: internal const int FirstNameTextboxLength = 25;
18: internal const int MaxPollCount = 50;
19: }
20:
21: public class NativeMethod
22: {
23: [DllImport("user32.dll")]
24: public static extern IntPtr GetForegroundWindow();
25: }
26:
27: class SampleHelloTest
28: {
29: static void Main(string[] args)
30: {
31: try
32: {
33: Stopwatch timer = new Stopwatch();
34: timer.Start();
35:
36: int testIterationCount = 25;
37:
38: // Declare a new process and automation elements
39: Process autProcess = new Process();
40: AutomationElement desktopPath = AutomationElement.RootElement;
41: AutomationElement autForm = null;
42: AutomationElement nameTextbox;
43: AutomationElement clearButton;
44: AutomationElement okButton;
45:
46: // Launch the AUT
47: autProcess = LaunchApplication(
48: Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
49: Constant.AutName),
50: desktopPath,
51: Constant.MaxPollCount,
52: ref autForm);
53:
54: // Get the UI automation element object on main form
55: GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);
56:
57: // Verify the textbox control is 'visible'
58: if (nameTextbox != null &&
59: (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))
60: {
61: // iterate through a specified number of test data variants
62: for (int testCount = 0; testCount < testIterationCount; testCount++)
63: {
64:
65: int seedValue;
66: string testData = GetRandomTestDataVariant(out seedValue);
67:
68: // Put test string into textbox
69: SetTextboxText(nameTextbox, testData);
70:
71: // Emulate clicking the OK button to instantiate the message box
72: PushButton(okButton);
73:
74: // Oracle to compare actual vs expected result and log results/important information
75: string actualResult = string.Empty;
76: if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))
77: {
78: // Log Test Results as appropriate
79: Console.WriteLine(testData);
80: }
81: else
82: {
83: Console.BackgroundColor = ConsoleColor.Red;
84: Console.WriteLine(
85: "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",
86: testData, actualResult, seedValue);
87: }
88:
89: // Emulate clicking the clear button
90: PushButton(clearButton);
91: }
92: }
93: else
94: {
95: Console.WriteLine("TEST FAILURE: Textbox control not found");
96: }
97:
98: // Clean up the test environment
99: CloseAutProcess(autProcess);
100:
101: // Log total elapsed time and any additional important info
102: timer.Stop();
103: Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);
104: }
105: catch (Exception e)
106: {
107: Console.WriteLine(e.ToString());
108: }
109: }
110:
111: #region HELPER METHODS
112: /// <summary>
113: /// Launches AUT process; or throws exception if process fails to launch
114: /// </summary>
115: internal static Process LaunchApplication(
116: string autPathAndFilename,
117: AutomationElement desktop,
118: int maxPollCount,
119: ref AutomationElement myAutForm)
120: {
121: Process myProc = new Process();
122: myProc.StartInfo.FileName = autPathAndFilename;
123: if (myProc.Start())
124: {
125: // Find the MyFontForm window
126: int pollCount = 0;
127: do
128: {
129: myAutForm = desktop.FindFirst(
130: TreeScope.Descendants, new PropertyCondition(
131: AutomationElement.AutomationIdProperty, Constant.AutFormTitle));
132: pollCount++;
133: System.Threading.Thread.Sleep(100);
134: }
135: while (myAutForm == null && pollCount < maxPollCount);
136:
137: if (myAutForm == null)
138: {
139: throw new ArgumentNullException("Failed to find AUT form");
140: }
141: }
142: return myProc;
143: }
144:
145: /// <summary>
146: /// Get UI automation element objects on main form by property name
147: /// </summary>
148: private static void GetUIAutomationElements(
149: AutomationElement autFormName,
150: out AutomationElement nameTextbox,
151: out AutomationElement okButton,
152: out AutomationElement clearButton)
153: {
154: nameTextbox = autFormName.FindFirst(
155: TreeScope.Descendants, new PropertyCondition(
156: AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));
157: okButton = autFormName.FindFirst(
158: TreeScope.Descendants, new PropertyCondition(
159: AutomationElement.AutomationIdProperty, Constant.AutOKButton));
160: clearButton = autFormName.FindFirst(
161: TreeScope.Descendants, new PropertyCondition(
162: AutomationElement.AutomationIdProperty, Constant.AutClearButton));
163: }
164:
165: /// <summary>
166: /// Generate a pseudo-random Unicode string using the Babel.dll automation library
167: /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm
168: /// </summary>
169: private static string GetRandomTestDataVariant(out int seedValue)
170: {
171: string randomString = string.Empty;
172: do
173: {
174: Random prng = new Random();
175: seedValue = prng.Next();
176: StringGenerator sg = new StringGenerator();
177: sg.Info.Seed = seedValue;
178: sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;
179: sg.Info.RandomizeCharacterCount = true;
180: sg.Info.AllowSurrogatePairCharacters = false;
181: randomString = sg.Polyglot();
182: }
183: while (randomString.Contains("\0"));
184:
185: return randomString;
186: }
187:
188: /// <summary>
189: /// Generic method that enters the test data into the a specified textbox
190: /// Throws exception if the length of the testData argument exceeds the size
191: /// property of the textbox control, or textbox control in not enabled
192: /// </summary>
193: private static void SetTextboxText(AutomationElement inputTextbox, string testData)
194: {
195: try
196: {
197: ValuePattern input =
198: (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);
199: input.SetValue(testData);
200: }
201: catch (ElementNotEnabledException)
202: {
203: throw new ElementNotEnabledException("Textbox is not enabled.");
204: }
205: catch (InvalidOperationException)
206: {
207: throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");
208: }
209: }
210:
211: /// <summary>
212: /// Oracle that compares the randomly generated string against the variable
213: /// result in the message box text
214: /// </summary>
215: private static bool ValidateMessageBoxStringOracle(
216: string testData, int maxPollCount, out string actualResult)
217: {
218: bool result = false;
219: IntPtr msgBoxHandle = IntPtr.Zero;
220: AutomationElement messageBox;
221: int pollCount = 0;
222: actualResult = string.Empty;
223: try
224: {
225: do
226: {
227: msgBoxHandle = NativeMethod.GetForegroundWindow();
228: messageBox = AutomationElement.FromHandle(msgBoxHandle);
229: string className =
230: (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);
231: System.Threading.Thread.Sleep(100);
232: pollCount++;
233: }
234: while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);
235:
236: if (messageBox != null)
237: {
238: AutomationElement textbox = messageBox.FindFirst(
239: TreeScope.Descendants, new PropertyCondition(
240: AutomationElement.AutomationIdProperty, "65535"));
241: string messageBoxText =
242: (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);
243: actualResult =
244: messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);
245:
246: if (actualResult.Equals(testData))
247: {
248: result = true;
249: }
250:
251: AutomationElement messageboxOKButton = messageBox.FindFirst(
252: TreeScope.Descendants, new PropertyCondition(
253: AutomationElement.ClassNameProperty, "Button"));
254: PushButton(messageboxOKButton);
255: }
256: else
257: {
258: actualResult = "Messagebox not found.";
259: }
260:
261: return result;
262: }
263: catch (ArgumentOutOfRangeException)
264: {
265: throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");
266: }
267: catch (NullReferenceException)
268: {
269: throw new NullReferenceException("Messagebox unexpectedly became null");
270: }
271: }
272:
273: /// <summary>
274: /// Generic method to push a button control
275: /// Throws exception if button control is not enabled, or buttonName is null
276: /// </summary>
277: private static void PushButton(AutomationElement buttonName)
278: {
279: try
280: {
281: InvokePattern pushButton =
282: (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);
283: pushButton.Invoke();
284: }
285: catch (ElementNotEnabledException)
286: {
287: throw new ElementNotEnabledException("Button is not enabled.");
288: }
289: catch (InvalidOperationException)
290: {
291: throw new InvalidOperationException("Button not found.");
292: }
293: }
294:
295: /// <summary>
296: /// Generic method to close an AUT process
297: /// </summary>
298: private static void CloseAutProcess(Process autProcess)
299: {
300: try
301: {
302: autProcess.CloseMainWindow();
303: autProcess.WaitForExit(500);
304: if (!autProcess.HasExited)
305: {
306: autProcess.Kill();
307: }
308: }
309: catch (InvalidOperationException)
310: {
311: throw new InvalidOperationException("Specified process not found.");
312: }
313: }
314: #endregion HELPER METHODS
315: }
316: }
Automating Screen Captures
When I first started at Microsoft I worked on the Windows 95 international test team. Not only did we focus on globalization testing, but we also did a lot of the localization testing for the East Asian language versions (Japanese, Korean, Simplified Chinese, and Traditional Chinese). Localization is the adaptation of software to a particular target market. Translation of resource strings is one of the most visible parts of the localization process, and in those days part of our testing effort was spent on translation validation (e.g. checking the strings for appropriate translation). (In retrospect, I now think it is a huge waste of time and resources to use professional testers to validate the translation “quality.”) During our localization testing cycles it was common practice to take screen shots of issues we found during localization testing. These screen shots often help put things in context for the localization engineers, and helped them troubleshoot the issue. Of course, there were other times when we took screen shots to put other anomalies in context.
I am generally not a big fan of just attaching screen images to bug reports carte blanche. Joe Strazzere has an excellent post describing why a screen shot doesn’t always add value in bug reports. But, I also know that there are times when screen shots of the desktop can be of value. When we are testing using pre-defined tests or exploratory approaches we are physically there to see anomalies as they occur. But, (hopefully) nobody is babysitting the machines our automated GUI test scripts are running on. So, when an unexpected anomaly occurs with automated GUI test scripts is might sometimes be beneficial to capture the desktop state as an image. This screen capture might provide some clues as to the state of the desktop at the time of unexpected automated test case failures resulting in an indeterminate automated test script result.
There are several 3rd party tools that some testers use to capture the desktop image and save it to a file. Automated test frameworks should also have methods that test developers can call to capture screen shots at important points during the execution of an automated test script, or when an anomaly occurs. But, if not, here is a simply method that will take a snapshot of the desktop and save a jpeg file of the desktop state.
1: using System.Drawing;
2: using System.Drawing.Imaging;
3: using System.Windows.Forms;
4:
5: ...
6:
7: public static void GetDesktopImage(string imageFilePath)
8: {
9: // Declare a structure for the width and height of the desktop
10: Rectangle rect = Screen.GetBounds(Point.Empty);
11:
12: // Declare a new instance of the bitmap class
13: Bitmap image = new Bitmap(rect.Width, rect.Height);
14:
15: // Capture the desktop
16: using (Graphics desktopImage = Graphics.FromImage(image))
17: {
18: desktopImage.CopyFromScreen(Point.Empty, Point.Empty, rect.Size);
19: }
20:
21: // Save the image to the specified path & filename
22: image.Save(imageFilePath, ImageFormat.Jpeg);
23: }
I would not capture tons of images during a test run; they just aren’t that valuable. And, I do not advocate capturing images to use as oracles…they are just too unreliable in my opinion. But, there are times when a screen capture of the desktop might add value and provide some context for the tester or the developer in troubleshooting issues.
UI Automation Beneath the Presentation Layer using .NET’s Reflection
This week I am in Bad Homburg, Germany where I gave a keynote at the Testing & Finance conference. It was really great to see some very dear friends and meet new colleagues from around the world. Bad Homburg is about 30 minutes by train from Frankfurt, and it is a quaint little town with beautiful parks and Schloss Homburg which was built around 1680 and was the summer residence of the German Kaiser.
Also this week the newest edition of the Automated Software Testing magazine published an article I wrote entitled UI Automation Beneath the Presentation Layer. It discusses some tips on how to use reflection for automating applications developed using managed code. If you have questions or comments about my article please let me know. Your feedback is appreciated.
I have always respected the work of Dion Johnson, and I was quite honored when he asked me to write an article for the magazine. The magazine and the Automated Testing Institute website provide a wonderful resource to people interested in software test automation. I really look forward to working with Dion and the Automated Testing Institute more closely in the future to provide whatever help I can to this monumental task he has undertaken. I also hope there are others out there who will share some of their ideas as well. I think the days of Not Invented Here (NIH) and I’ve Got A Secret (IGAS) syndrome are becoming issues of the past. We still have too much reinvention and not enough reuse of test code. I think that we have well established there is no single approach to testing that is effective in all contexts. We might approach problems from different perspectives, but I also think that all testers are passionate and have very similar goals of personal and professional improvement. To mature our discipline, and increase our effectiveness in our roles we should collaborate more and pool our resources, and the Automated Testing Institute is a growing community of professionals.
Technorati Tags: Automation,Reflection
Windows Live Tags: Automation,Reflection
WordPress Tags: Automation,Reflection
Globalization Testing: Basic International Sufficiency
I started my career at Microsoft in 1994 working on the Windows 95 International Test team. Globalization testing is a unique specialty in software testing just like performance, security, and other specific areas of testing. Globalization testing doesn’t necessarily require a tester to be bi-lingual, or be from a country other than the United States. A good globalization tester has an in-depth understanding of such things as character encoding types and issues associated with the different types, character mapping and conversion issues, data manipulation by the application, operating system, and network protocols.
Many people might also say that globalization testers also need to know that different locales (places) around the world use different formats for date and time (national conventions). For example, in the United States the default long date format is Thursday, June 03, 2010 but in Germany it is Donnerstag, 3. Juni 2010. A tester doesn’t have to ‘read’ German to see the abstract date format has changed from dddd, MMMM dd, yyyy to dddd, d. MMMM yyyy.
Testing for support of these different national conventions used around the world is referred to as basic international sufficiency testing. I suspect the reason why some people might assume basic international sufficiency testing these different national conventions is the domain of the globalization tester is because the national conventions are set by default on the different localized versions of a software product so that’s when they are tested. But, this reasoning is absurd!
First, not all products are “localized” into all languages or ‘locales.” So, who tests the Canadian long date format of MMMM-dd-yy, or the Georgian (Georgia) long date format of yyyy ‘წლის’ dd MM, dddd? Also, Vista and later versions of Windows allow the user to ‘customize’ the date and time “format pictures” to use different separator symbols and orderings.
Secondly, way too many bugs such as hard-coded date formats are found way too late in the testing cycle (because localized versions tend to lag US English language version). And of course, we all know the cost of finding bugs later in the lifecycle are more costly to correct.
So, we must ask if there is a way for basic international sufficiency testing to be ‘pushed upstream?’ And of course the answer is yes. The easiest way is to host a “globalization bug bash” early in the cycle. (A “bug bash” is a day where testers are given some basic training on attack patterns, fault models, etc., in a general focus area and then spend a day exploring different areas of the product trying to flush out bugs in a competition style format.) Another way is to assign each tester a different locale (preferably one that is not associated with a localized language version) and have them set their test and self-host environments to that locale during their testing.
This is easily accomplished on Windows test environments by having testers launch the Regional and Language control panel applet (the short cut is Start –> Run, then type “intl.cpl” without the quotes, and press the OK button).
This just tests for a basic level of international sufficiency, and any good tester would want to explore their project’s capability to support the more than 150 different locale national conventions at a deeper level. This is especially true if your product is going to be used by customers around the world (including Canada). But, of course, we don’t want to run the same tests on all 150+ locales supported by the operating system.
The national convention settings for a particular locale are stored in a data type called the LCID, and when we change our locale (Format on the latest Regional and Language control panel applet) through the user interface we are actually calling various National Language Support (NLS) APIs. A “world-wide” application should use the universal NLS APIs and data available via the operating system.
One way to test our application’s ability to correctly use the national convention data supplied by the operating system is to set customized conventions. For example, did you know the Windows 7 operating system allows a digit grouping symbol to be a string of up to 3 characters? Or the Negative sign symbol can be a string of up to 4 characters.
Although having testers change their default locale (Format) on their test environment and self-host machines is a good first step in basic international sufficiency testing, we also want to see if our application can process a negative value of “!NEG7” instead of just “–7,” and any textboxes correctly display the customized negative sign symbol (especially at the upper extreme boundary of the textbox size property.
To customize the national convention settings we simply click the Advanced settings… button on the Formats property sheet of the Region and Language control panel applet which instantiates a new dialog with 4 property sheets for Numbers, Currency, Time, and Date.
Solution for Test Automation
That’s all well and fine for basic testing, or testing a “few” customized values, but if we wanted to test the permutations for each convention, or the combination of different conventions on numbers, currency, time, or date formats the number of tests is astronomical. Typically, testers writing an automated test would try to navigate the user interface of the Regional and Language control panel applet and the Customize Format property sheets in order to set custom conventions.
In the past I provided some code snippets for changing the convention settings on the Customize Format property sheets on versions of Windows pre-Vista. Earlier this year I also provided code snippets for customizing the date format picture and the time format picture.
That’s all well and good, but I recently released a new test automation library called GlobalTester for test developers to use in their automated test scripts. The GlobalTester library provides testers methods to set custom national conventions for the current user without having to navigate the user interface of the Region and Language options control panel applet. These national conventions include number formats, currency formats, date formats, time formats, and also current location.
The following example illustrates how we might design a test script to customize the date format for a test and reset the date format to its original setting (restoring the test environment to pre-test conditions). (Usage documentation for the GlobalTester library is on the Testing Mentor website.)
1: namespace CustomizeDateSettingsExampleScript
2: {
3: using System;
4: using System.Globalization;
5: using TestingMentor.TestTool.GlobalTester;
6:
7: class MyTest
8: {
9: static void Main()
10: {
11: try
12: {
13: CustomDateFormat time = new CustomDateFormat();
14:
15: string defaultLongDateFormat =
16: CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern;
17: string newLongDateFormat = "MMM - d (yyyy) gg";
18:
19: if (time.ChangeLongDateFormat(newLongDateFormat))
20: {
21: // Launch AUT
22: // Execute test - (e.g. AUT implements long date string)
23: // Oracle - (e.g. compare long date format against customized pattern)
24:
25: // Reset test platform to original configuration
26: time.ChangeLongDateFormat(defaultLongDateFormat);
27: }
28: else
29: {
30: // Date format not changed; test not executed (e.g. invalid
31: // day, month, year, and era format pictures)
32: }
33: }
34:
35: catch(ArgumentOutOfRange e)
36: {
37: // Test script failure - (e.g. long date format string argument out of range)
38: }
39:
40: finally
41: {
42: // Log test results
43: }
44: }
45: }
46: }
Globalization Testing: Customizing Time Formats
Time is a commodity in short supply. I have been juggling a lot lately and there never seems to be enough time to do everything I need to do, and even less time to do the things I want to do. (Blogging falls under the want to do category.) I wish sometimes I could slow down the hands of time, but that is beyond my control. What is within my control is changing the time format displayed on the computer. And if I need to do that in an automated test to increase the robustness of my test to include globalization, then I can programmatically change the time format without having to manipulate the Region and Language settings control panel applet.
Time and date information is commonly pulled from the operating system by many developers for use in headers or footers on documents, default file names, printing, and other places time/date stamps are useful or important. To ensure our products are “world-ready” we should modify the formats to validate whether our product supports various national conventions used in different regions (locales) around the world. In the previous post I illustrated how to programmatically customize the date formats on a Windows environment for including some basic globalization tests in your test automation. This week let’s look at how we can programmatically change both the short time and long time formats.
We will again need the 2 Win32 API functions SetLocaleInfo() and PostMessage() that we marshaled over into the NativeMethods class. Since that code doesn’t change I won’t repeat it here you can simply refer to the code snippet in the previous post. In this situation we need to set the lcType in SetLocaleInfo() to the LOCALE_STIMEFORMAT constant. Then we can pass a null-terminated string to the lcData variable in the SetLocaleInfo() function. MSDN explains “The maximum number of characters allowed for this string is 80, including a terminating null character. The string can consist of a combination of hour, minute, and second format pictures.”
Once again, to simplify that a bit I wrote some more wrapper methods to change the time format. Also, since we will be calling SetLocaleInfo() and PostMessage() a lot for customizing date, time, and other national conventions I created a wrapper method called UpdateLocaleInformation() to remove redundancy.
1: namespace TestingMentor.TestTool.GlobalTester
2: {
3: using System;
4:
5: public enum TimeFormatType
6: {
7: LongTimeFormat = 0x00001003,
8: ShortTimeFormat = 0x00000079
9: }
10:
11: public class CustomTimeFormat
12: {
13: private int timeFormatType = (int)TimeFormatType.ShortTimeFormat;
14: private string timeFormatPicture = string.Empty;
15:
16: public int SetTimeFormatType
17: {
18: set
19: {
20: if (value == (int)TimeFormatType.ShortTimeFormat ||
21: value == (int)TimeFormatType.LongTimeFormat)
22: {
23: this.timeFormatType = value;
24: }
25: else
26: {
27: throw new ArgumentOutOfRangeException("TimeFormatType invalid");
28: }
29: }
30: }
31:
32: public string SetTimeFormatPicture
33: {
34: set { this.timeFormatPicture = value; }
35: }
36:
37: public bool ChangeTimeFormat()
38: {
39: return UpdateLocaleInformation(
40: this.timeFormatType,
41: this.timeFormatPicture);
42: }
43:
44: private bool UpdateLocaleInformation(int localeType, string localeData)
45: {
46: bool success = false;
47: if (NativeMethods.SetLocaleInfo(
48: NativeMethods.SystemDefaultLocale,
49: localeType,
50: localeData))
51: {
52: NativeMethods.PostMessage(
53: NativeMethods.BroadcastMessage,
54: NativeMethods.SettingChangeMessage,
55: IntPtr.Zero,
56: IntPtr.Zero);
57: success = true;
58: }
59:
60: return success;
61: }
62: }
63: }
Once again, we simply have to set the SetTimeFormatType property to either the Short time or Long time format, provide the format picture by setting the SetTimeFormatPicture property, and then call ChangeTimeFormat(). The sample below illustrates how to change the short time format with different time separators and a reverse order.
1: static void Main(string[] args)
2: {
3: CustomTimeFormat time = new CustomTimeFormat();
4: time.SetTimeFormatType = (int)TimeFormatType.ShortTimeFormat;
5: time.SetTimeFormatPicture = "ss'mm,hh - tt";
6: if (time.ChangeTimeFormat())
7: {
8: Console.WriteLine("Success");
9: }
10: }
Now, we can also customize the AM/PM designator as well. To change the AM/PM designator we need to add a few more properties and another wrapper method. In this case, I’ve added the SetAmPmDesignator property, the SetAmPmString property, and the ChangeAmPmDesignator() method.
1: public enum AmPmDesignator
2: {
3: AM = 0x00000028,
4: PM = 0x00000029
5: }
6:
7: public class CustomTimeFormat
8: {
9: public int SetAmPmDesignator
10: {
11: set
12: {
13: if (value == (int)AmPmDesignator.AM || value == (int)AmPmDesignator.PM)
14: {
15: this.designatorForAmPm = value;
16: }
17: else
18: {
19: throw new ArgumentOutOfRangeException("AmPmDesignator invalid.");
20: }
21: }
22: }
23: }
24: public string SetAmPmString
25: {
26: set { this.timeDesignator = value; }
27: }
28:
29: public bool ChangeAmPmDesignator()
30: {
31: return UpdateLocaleInformation(
32: this.designatorForAmPm,
33: this.timeDesignator);
34: }
The code snippet below illustrates how to change the AM designator from “AM” to “In the morning.”
1: static void Main(string[] args)
2: {
3: CustomTimeFormat time = new CustomTimeFormat();
4: time.SetAmPmDesignator = AmPmDesignator.AM;
5: time.SetAmPmString = "In the morning.";
6: if (time.ChangeAmPmDesignator())
7: {
8: Console.WriteLine("Success");
9: }
10: }
Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle. Time and date are perhaps the most visible national conventions used in many different ways in our applications. We should test the common (equivalent) conventions used in various regions around the world, and customizing these settings helps ensure the developer is properly calling NLS APIs and not using custom functions.
Also, check out the beta release of the GlobalTester automation library that has this functionality and more, and let me know what you think.
Complex != Better
I know all about over-engineering. I previously wrote about a barn my father designed and built using telephone poles and oak planks and pallets for the stall walls. From a structural perspective this barn would have stood virtually anything mother nature could have thrown at it. And if someday people wanted to tear down that barn they would surely curse the builders because the job would be much harder then they expect.
I sometimes find code that is way over-engineered. In some cases it may be necessary for making the code more robust. But, over-engineered code may also result from ignorance of more efficient algorithms or design patterns. And, sometimes overly complex algorithms are a result of developers trying to craft something that obfuscates the simplicity of the solution and fool others into believing the complexity of the solution is somehow better than a more simple solution.
I sometimes see this in test code. I am a firm believer in robust, ‘bullet-proof’ test code because each time an automated test case throws a false positive or ‘breaks’ the team loses confidence in the automation project, and it takes our time to ‘massage’ the test back to health. But, there seems to be 2 extremes in test automation. Simplistic prescriptive scripted tests with a bunch of hard-coded ‘test-data,’ or overly complex test code that is virtually undecipherable by anyone other than the original developer that renders any downstream maintenance virtually impossible. I have seen many instances where complete libraries of automation was scrapped and re-developed simply because it was easier to re-write the code rather than trying to wade through the quagmire of complexity.
In a recent example, some of my students submitted their test automation projects with a library that contained a method to generate a random string. When I first looked at their method for generating a random string I was a bit perplexed. Besides the fact that the method only produced a grand total of 26 upper case characters, the code was much more complicated than necessary. Despite having showed them how to use the Babel random string generator they choose to make their own, so I asked them how they came up with this solution and they said “they searched on the Internet. ” Others in the class indicated they also used the same method in their projects. So, I when I got home I did a quick search and found the code sample.
1: private static string RandomString(int size)
2: {
3: StringBuilder builder = new StringBuilder();
4: Random random = new Random();
5:
6: char ch;
7: for (int i = 0; i < size; i++)
8: {
9: ch = Convert.ToChar(
10: Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));
11: builder.Append(ch);
12: }
13:
14: return builder.ToString();
15: }
OK…for a moment let’s overlook the fact that if we make 2 consecutive calls to this method we will get the same identical random string of characters (which the author of this code also discovered), and let’s assume that the range of characters is limited to upper case A through Z, and let’s also ignore the fact that we are not seeding our Random generator for reproducibility of the randomly generated output from this method.
Let’s focus on the statement in lines 9 and 10. Why in the world would we generate a number between 0.0 and 1.0, multiply it by 26 and add 65, and then use Math.Floor to return the largest integer less then or equal to the resultant double, then convert that double to a type Int32, and then convert the Int32 to a type char? Now I ask you, can we make this any more complicated?
Now, I am not critiquing the style of the code or its inherent limitations, but this solution is an example of over-engineering. As I have said before, complexity cultivates chaos. At first I thought maybe this approach might give me a better distribution of characters, but when I tested this hypothesis it simply didn’t pan out. So, the way the random character is generated is simply too complex. An easier, more readable, and effective alternative might be to simply generate a random integer within the allowable range and cast that int to a type char as illustrated below in line 10. (Yes, I realize the limitations with this approach if trying to generate surrogate pair characters above U+FFFF.)
1: private const int minCharacter = 65;
2: private const int maxCharacter = 91;
3: private static string SimpleRandomString(int size)
4: {
5: StringBuilder sb = new StringBuilder();
6: System.Threading.Thread.Sleep(1);
7: Random r = new Random();
8: for (int i = 0; i < size; i++)
9: {
10: sb.Append((char)r.Next(minCharacter, maxCharacter + 1));
11: }
12:
13: return sb.ToString();
14: }
BTW…the Sleep() in line 6 is a easy solution to preventing identical return values for consecutive calls. But, a more effective solution would be to pass in a seed value as a parameter to the method and use that to seed the new Random generator as illustrated below. Different seeds not only guarantee randomness in the resultant string, but also allow for repeatability if necessary as long as the seed value is preserved.
1: private static string SimpleRandomString(int size, int seed)
2: {
3: StringBuilder sb = new StringBuilder();
4: Random r = new Random(seed);
5: ...
6: }
But, I digress. This post isn’t about random generation or style…it’s about complexity. Overly complex code tends to harbor errors that might go undetected (at least initially). Also, maintenance of complex code not only becomes more problematic, it often leads to costly re-writes down the road.
If we consider that greater complexity may increase the likelihood of error, then we don’t want our development partners to unnecessarily over-engineered algorithms. Also, overly complex solutions often require overly complex testing. This not only leads to increased testing costs, but it also increases the probability of an important test being missed or overlooked. Think testability!
Finally, with regard to test automation we should consider that our automated tests might be reused by other teams, and they certainly will be used during maintenance or sustained engineering efforts well after the product has released. And, in some cases, the people maintaining the product might not be the same people who shipped the product. Well-written automated tests are not just reviewable by someone other than the author of the code, but they should also be easily maintainable. Otherwise, we might end up paying double for that automated test case. Ouch!
Do I Really Need To Automate This Test?
For the past 2 weeks my students in my automation course at University of Washington have been tasked with designing automated test cases through the GUI for a shareware program. In my opinion, GUI automation is the least effective approach for testing the functional or business logic of a program (assuming a well designed architecture where the business logic code is separate from the form (GUI) and the event hander (GUI object behavior) code). However, if a tester doesn’t have access to the underlying APIs used in the application under test (AUT) and is given just a compiled application (‘GUI application’) to test then functional testing through the GUI may be the only alternative.
GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.
However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’) But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’
There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?
First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.
So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”
The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.
Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:
and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.
But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”
“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”
I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.
Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.
This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)
Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”
Programmatically Detecting The Operating System Version (Part II)
Time is a commodity in short supply! It has been more than 2 weeks since my last post. I have not been sitting idle, but really haven’t had a lot of free time to write. In preparation for my up-coming trip to Zurich, Switzerland to give a workshop and keynote at Swiss Testing Day I was interviewed by Marco van der Spek for TESTNIEUWS.NL. The interview provides a bit more background about me and some of my perspectives of Microsoft and software testing.
Also sucking up some of my time over the past couple of weeks were ‘reorgs’ at work. Change at Microsoft is a constant. Most people get used to it after awhile; others still freak out even with the slightest change. For me it mostly means shifting some priorities based on the new General Manager’s strategic vision, acclimating to a new manager and letting him know what I am working on, and generally making sure the day to day business on our team keeps moving forward during the transition.
Just like life at Microsoft and technology in general the Windows operating system continuously goes through changes. New versions, new service packs, Ultimate, Home, Server versions, etc. Sometimes it is hard to keep up with all the changes. And from a test automation perspective it is sometimes important to know which operating system version the test is running on. In some cases control flow in the automated test case may need to branch in order for the test to execute on different operating system versions. Branching in an automated test based on the operating system version eliminates the need to write separate test cases for variances in operating system versions.
And, certainly if our test matrix includes multiple product versions (Home, Ultimate, Professional, etc) and our automated test exposes a bug then we certainly want to collect information about the operating system version the test was running on. This is especially important if the automated test fails on one machine, but passes on another. Sometimes the cause of the failure may be a slight difference in the machine configuration or the operating system version.
Almost 2 years ago I described a way to get the operating system version information using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members in this blog post. But, I also mention some limitations such as detecting the specific edition or product type of a Windows Version. Another limitation is the difficulty in detecting whether the operating system version is Windows 7 or Windows 2008 Server R2.
I also mentioned that in order to identify a particular version and/or edition of Windows we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function. But, to use these Win32 APIs we need to use platform invocation services (P/Invoke) to marshal native code into our managed test code.
The code snippet below illustrates the required Win32 function marshaled using the DLLImport attribute in C#, constant values, wrapper methods to get common operating system information, and public properties to get the operating system version, any installed service pack information, and the operating system edition for Windows Vista, Windows Server 2008, and Windows 7.
1: // <copyright file = VersionInfo.cs" company = "Testing Mentor">
2: // Copyright © 2010 All Rights Reserved. Test developers can simply copy and
3: // paste the code into their code, but may not reproduce or publish the code
4: // snippets on any web site, online service, or distribute as source on any
5: // media without express written permission. </copyright>
6:
7: namespace TestingMentor.Snippet.OperatingSystemVersionInfo
8: {
9: using System;
10: using System.Runtime.InteropServices;
11:
12: public class WindowsVersionInfo
13: {
14: public string GetOSVersion
15: {
16: get { return this.GetOSVersionInfo(); }
17: }
18:
19: public string GetServicePack
20: {
21: get { return this.GetServicePackInfo(); }
22: }
23:
24: public string GetProductType
25: {
26: get { return this.GetProductTypeInfo(); }
27: }
28:
29: private string GetOSVersionInfo()
30: {
31: string version = "Unsupported Version";
32:
33: NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();
34: osvi.VersionInfoSize =
35: Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
36: NativeMethods.GetVersionEx(ref osvi);
37:
38: if (OsviConstant.SupportedPlatform == osvi.PlatformId &&
39: osvi.MajorVersion > 4)
40: {
41: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
42: osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows2000)
43: {
44: version = "Windows 2000";
45: }
46:
47: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
48: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsXP)
49: {
50: version = "Windows XP";
51: }
52:
53: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&
54: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsServer2003)
55: {
56: if (osvi.ProductType == (byte)OsviConstant.WorkStation)
57: {
58: version = "Windows XP Professional x64";
59: }
60: else
61: {
62: version = "Windows Server 2003";
63: if (NativeMethods.GetSystemMetrics(OsviConstant.ServerR2) != 0)
64: {
65: version += " R2";
66: }
67: }
68: }
69:
70: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&
71: osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsVista)
72: {
73: if (osvi.ProductType ==
74: (byte)OsviConstant.WorkStation)
75: {
76: version = "Windows Vista";
77: }
78: else
79: {
80: version = "Windows Server 2008";
81: }
82: }
83:
84: if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&
85: osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows7)
86: {
87: if (osvi.ProductType == (byte)OsviConstant.WorkStation)
88: {
89: version = "Windows 7";
90: }
91: else
92: {
93: version = "Windows Server 2008 R2";
94: }
95: }
96: }
97:
98: return version;
99: }
100:
101: private string GetServicePackInfo()
102: {
103: NativeMethods.OSVersionInfoEx versionInfo = new NativeMethods.OSVersionInfoEx();
104: versionInfo.VersionInfoSize = Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
105: NativeMethods.GetVersionEx(ref versionInfo);
106: return versionInfo.CSDVersion;
107: }
108:
109: private string GetProductTypeInfo()
110: {
111: string product = String.Empty;
112:
113: NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();
114: osvi.VersionInfoSize =
115: Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));
116: NativeMethods.GetVersionEx(ref osvi);
117:
118: if (osvi.MajorVersion > 5)
119: {
120: uint productType = 0;
121:
122: NativeMethods.GetProductInfo(
123: osvi.MajorVersion,
124: osvi.MinorVersion,
125: osvi.ServicePackMajor,
126: osvi.ServicePackMinor,
127: ref productType);
128:
129: switch (productType)
130: {
131: case (uint)OsviConstant.ProductInfo.Business:
132: product = "Business Edition";
133: break;
134: case (uint)OsviConstant.ProductInfo.BusinessN:
135: product = "Business N Edition";
136: break;
137: case (uint)OsviConstant.ProductInfo.ClusterServer:
138: product = "HPC Edition";
139: break;
140: case (uint)OsviConstant.ProductInfo.DatacenterServer:
141: product = "Server Datacenter (Full)";
142: break;
143: case (uint)OsviConstant.ProductInfo.DatacenterServerCore:
144: product = "Server Datacenter (Core)";
145: break;
146: case (uint)OsviConstant.ProductInfo.DataCenterServerCoreV:
147: product = "Server Datacenter without Hyper-V (Core)";
148: break;
149: case (uint)OsviConstant.ProductInfo.DataCenterServerV:
150: product = "Server Datacenter without Hyper-V (Full)";
151: break;
152: case (uint)OsviConstant.ProductInfo.Enterprise:
153: product = "Enterprise Edition";
154: break;
155: case (uint)OsviConstant.ProductInfo.EnterpriseE:
156: product = "Enterprise E Edition";
157: break;
158: case (uint)OsviConstant.ProductInfo.EnterpriseN:
159: product = "Enterprise N Edition";
160: break;
161: case (uint)OsviConstant.ProductInfo.EnterpriseServer:
162: product = "Server Enterprise (Full)";
163: break;
164: case (uint)OsviConstant.ProductInfo.EnterpriseServerCore:
165: product = "Server Enterprise (Core)";
166: break;
167: case (uint)OsviConstant.ProductInfo.EnterpriseServerCoreV:
168: product = "Server Enterprise without Hyper-V (Core)";
169: break;
170: case (uint)OsviConstant.ProductInfo.EnterpriseServerIA64:
171: product = "Server Enterprise for Itanium-based Systems";
172: break;
173: case (uint)OsviConstant.ProductInfo.EnterpriseServerV:
174: product = "Server Enterprise without Hyper-V (Full)";
175: break;
176: case (uint)OsviConstant.ProductInfo.HomeBasic:
177: product = "Home Basic Edition";
178: break;
179: case (uint)OsviConstant.ProductInfo.HomeBasicE:
180: product = "Home Basic E Edition";
181: break;
182: case (uint)OsviConstant.ProductInfo.HomeBasicN:
183: product = "Home Basic N Edition";
184: break;
185: case (uint)OsviConstant.ProductInfo.HomePremium:
186: product = "Home Premium Edition";
187: break;
188: case (uint)OsviConstant.ProductInfo.HomePremiumE:
189: product = "Home Premium E Edition";
190: break;
191: case (uint)OsviConstant.ProductInfo.HomePremiumN:
192: product = "Home Premium N Edition";
193: break;
194: case (uint)OsviConstant.ProductInfo.HomeServer:
195: product = "Home Server Edition";
196: break;
197: case (uint)OsviConstant.ProductInfo.HyperV:
198: product = "Microsoft Hyper-V Server";
199: break;
200: case (uint)OsviConstant.ProductInfo.MediumBusinessServerManagement:
201: product = "Windows Essential Business Server Management Server";
202: break;
203: case (uint)OsviConstant.ProductInfo.MediumBusinessServerMessaging:
204: product = "Windows Essential Business Server Messaging Server";
205: break;
206: case (uint)OsviConstant.ProductInfo.MediumBusinessServerSecurity:
207: product = "Windows Essential Business Server Security Server";
208: break;
209: case (uint)OsviConstant.ProductInfo.Professional:
210: product = "Professional Edition";
211: break;
212: case (uint)OsviConstant.ProductInfo.ProfessionalE:
213: product = "Professional E Edition";
214: break;
215: case (uint)OsviConstant.ProductInfo.ProfessionalN:
216: product = "Professional N Edition";
217: break;
218: case (uint)OsviConstant.ProductInfo.ServerForSmallBusiness:
219: product = "Windows Server 2008 for Windows Essential Server Solutions";
220: break;
221: case (uint)OsviConstant.ProductInfo.ServerForSmallBusinessV:
222: product = "Windows Server 2008 without Hyper-V for Windows Essential Server Solutions";
223: break;
224: case (uint)OsviConstant.ProductInfo.ServerFoundation:
225: product = "Server Foundation";
226: break;
227: case (uint)OsviConstant.ProductInfo.SmallBusinessServer:
228: product = "Windows Small Business Server";
229: break;
230: case (uint)OsviConstant.ProductInfo.SmallBusinessServerPremium:
231: product = "Windows Small Busines Server Premium";
232: break;
233: case (uint)OsviConstant.ProductInfo.StandardServer:
234: product = "Server Standard (Full)";
235: break;
236: case (uint)OsviConstant.ProductInfo.StandardServerCore:
237: product = "Server Standard (Core)";
238: break;
239: case (uint)OsviConstant.ProductInfo.StandardServerCoreV:
240: product = "Server Standard without Hyper-V (Core)";
241: break;
242: case (uint)OsviConstant.ProductInfo.StandardServerV:
243: product = "Server Standard without Hyper-V (Full)";
244: break;
245: case (uint)OsviConstant.ProductInfo.Starter:
246: product = "Starter Edition";
247: break;
248: case (uint)OsviConstant.ProductInfo.StarterE:
249: product = "Starter E Edition";
250: break;
251: case (uint)OsviConstant.ProductInfo.StarterN:
252: product = "Starter N Edition";
253: break;
254: case (uint)OsviConstant.ProductInfo.StorageEnterpriseServer:
255: product = "Storage Server Enterprise";
256: break;
257: case (uint)OsviConstant.ProductInfo.StorageExpressServer:
258: product = "Storage Server Express";
259: break;
260: case (uint)OsviConstant.ProductInfo.StorageStandardServer:
261: product = "Storage Server Standard";
262: break;
263: case (uint)OsviConstant.ProductInfo.StorageWorkgroupServer:
264: product = "Storage Server Workgroup";
265: break;
266: case (uint)OsviConstant.ProductInfo.Ultimate:
267: product = "Ultimate Edition";
268: break;
269: case (uint)OsviConstant.ProductInfo.UltimateE:
270: product = "Ultimate E Edition";
271: break;
272: case (uint)OsviConstant.ProductInfo.UltimateN:
273: product = "Ulitmate N Edition";
274: break;
275: case (uint)OsviConstant.ProductInfo.Undefined:
276: product = "Unknown Product";
277: break;
278: case (uint)OsviConstant.ProductInfo.Unlicensed:
279: product = "Unlicensed or Expired";
280: break;
281: case (uint)OsviConstant.ProductInfo.WebServer:
282: product = "Web Server (Full)";
283: break;
284: case (uint)OsviConstant.ProductInfo.WebServerCore:
285: product = "Web Server (Core)";
286: break;
287: }
288: }
289:
290: return product;
291: }
292: }
293:
294: // ****************************************************************************
295: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE
296: // ****************************************************************************
297:
298: internal class OsviConstant
299: {
300: internal const int SupportedPlatform = 2;
301: internal const int ServerR2 = 89;
302: internal const int WorkStation = 0x00000001;
303:
304: private OsviConstant()
305: {
306: }
307:
308: internal enum MajorVersion
309: {
310: NT5 = 5,
311: NT6 = 6
312: }
313:
314: internal enum MinorVersion
315: {
316: Windows2000 = 0,
317: WindowsXP = 1,
318: WindowsServer2003 = 2,
319: WindowsVista = 0,
320: Windows7 = 1
321: }
322:
323: internal enum ProductInfo : uint
324: {
325: Business = 0x00000006,
326: BusinessN = 0x00000010,
327: ClusterServer = 0x00000012,
328: DatacenterServer = 0x00000008,
329: DatacenterServerCore = 0x0000000C,
330: DataCenterServerCoreV = 0x00000027,
331: DataCenterServerV = 0x00000025,
332: Enterprise = 0x00000004,
333: EnterpriseE = 0x00000046,
334: EnterpriseN = 0x0000001B,
335: EnterpriseServer = 0x0000000A,
336: EnterpriseServerCore = 0x0000000E,
337: EnterpriseServerCoreV = 0x00000029,
338: EnterpriseServerIA64 = 0x0000000F,
339: EnterpriseServerV = 0x00000026,
340: HomeBasic = 0x00000002,
341: HomeBasicE = 0x00000043,
342: HomeBasicN = 0x00000005,
343: HomePremium = 0x00000003,
344: HomePremiumE = 0x00000044,
345: HomePremiumN = 0x0000001A,
346: HyperV = 0x0000002A,
347: MediumBusinessServerManagement = 0x0000001E,
348: MediumBusinessServerSecurity = 0x0000001F,
349: MediumBusinessServerMessaging = 0x00000020,
350: Professional = 0x00000030,
351: ProfessionalE = 0x00000045,
352: ProfessionalN = 0x00000031,
353: ServerForSmallBusiness = 0x00000018,
354: ServerForSmallBusinessV = 0x00000023,
355: ServerFoundation = 0x00000021,
356: SmallBusinessServer = 0x00000009,
357: StandardServer = 0x00000007,
358: StandardServerCore = 0x0000000D,
359: StandardServerCoreV = 0x00000028,
360: StandardServerV = 0x00000024,
361: Starter = 0x0000000B,
362: StarterE = 0x00000042,
363: StarterN = 0x0000002F,
364: StorageEnterpriseServer = 0x00000017,
365: StorageExpressServer = 0x00000014,
366: StorageStandardServer = 0x00000015,
367: StorageWorkgroupServer = 0x00000016,
368: Undefined = 0x00000000,
369: Ultimate = 0x00000001,
370: UltimateE = 0x00000047,
371: UltimateN = 0x0000001C,
372: WebServer = 0x00000011,
373: WebServerCore = 0x0000001D,
374: Unlicensed = 0xABCDABCD,
375: HomeServer = 0x00000013,
376: SmallBusinessServerPremium = 0x00000019,
377: }
378: }
379:
380: // ****************************************************************************
381: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE
382: // ****************************************************************************
383:
384: internal class NativeMethods
385: {
386: private NativeMethods()
387: {
388: }
389:
390: [DllImport("kernel32")]
391: [return: MarshalAs(UnmanagedType.Bool)]
392: internal static extern bool GetVersionEx(ref OSVersionInfoEx osvi);
393:
394: [DllImport("kernel32.dll")]
395: [return: MarshalAs(UnmanagedType.Bool)]
396: internal static extern bool GetProductInfo(
397: int osMajorVersion,
398: int osMinorVersion,
399: int spMajorVersion,
400: int spMinorVersion,
401: ref uint type);
402:
403: [DllImport("kernel32.dll")]
404: internal static extern int GetSystemMetrics(
405: int index);
406:
407: [StructLayout(LayoutKind.Sequential)]
408: internal struct OSVersionInfoEx
409: {
410: public int VersionInfoSize;
411: public int MajorVersion;
412: public int MinorVersion;
413: public int BuildNumber;
414: public int PlatformId;
415: [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]
416: public string CSDVersion;
417: public Int16 ServicePackMajor;
418: public Int16 ServicePackMinor;
419: public Int16 SuiteMask;
420: public byte ProductType;
421: public byte Reserved;
422: }
423: }
424: }
Now, some of my readers have indicated that these code snippets are not very useful because they can’t copy them directly and put them to use. So, to help resolve that issue I have created a new section on my web site called the Code Snippet Library. This snippet is posted there along with fully annotated (mostly FxCopy and StyleCopy compliant) file available for download or to copy for inclusion in your automated test cases, or compiled into a dynamic link library (DLL).
This example doesn’t differentiate between 32-bit and 64-bit Windows operating systems, but that is not really difficult to add, and if I get enough requests I will certainly add that into the pot. If the operating system version is no longer supported by Microsoft the GetOsVersion property will return "Unsupported Version." If no Service Packs are installed the GetServicePack property will return an empty string. If you need to detect an unsupported operating system version use the example here.
Scary Stories and GUI Automation
I remember going camping with my cousins as I was growing up. It was great fun despite sleeping inside a musty smelling canvas tent that retained heat so well it was more like a sauna. But, my father was adamant that the old canvas tent he bought at an Army Surplus store and took 2 men and 4 boys to carry and assemble was much better then those new fangled nylon tents. Nylon ripped too easily he reasoned, but canvas will withstand anything short of a raging bear. I’m not too sure there were too many raging bears roaming the camp-grounds of Maryland, Pennsylvania, and Virginia but I was confident that even if there were I would have stood a better chance inside a canvas tent as compared to those paper thin nylon tents. To this day I remember those camping trips when I smell old canvas, or perhaps it’s dried mold spores embedded in the canvas. Whichever, it takes me back to a time of fun and fond memories.
One of the best parts of the trips were sitting around the camp fire at night listening to my uncle concocting some story intended to scare the wits out of us young boys. You know, the kind of stories about headless Confederate soldiers, or werewolves, vampires, or other such wicked creatures of the night. I think these campfire chats are remnants of man’s tribal roots where the elders tried to scare the hell out of the juvenile hominids to prevent them from wandering off at night. As we got older we realized that these stories were simply fictitious folktales; sort of like successful, value-add GUI automation projects.
Last week I had lunch with a colleague who wanted to talk to me about an automation project on his team that went horribly awry. As he started to tell me his story I thought, “Wait…I heard this tale before. I can tell this story because I’ve heard it so many times over and over again…just like those scary stories I heard around the campfire growing up.” The story goes something like this.
Our team bought a new tool, or built another framework, and taught everyone how to script “black box” test cases. They developed quite a number of automated test scripts, and of course everything was working very well. The scripts were running and managers were happy because the team had a lot of automated test scripts. But, the tests weren’t finding any bugs, so just out of curiosity the managers suggested a bug bash. And sure enough as the testers started exploring the project it didn’t take long for them to fill the database with bugs. The developers were shell shocked, and the managers couldn’t believe it! They couldn’t understand why the automated GUI tests weren’t finding any bugs? And so, in a typical knee-jerk reaction fashion, the managers immediately halted the GUI automation project and required every tester to embark on an exploratory testing adventure in search of bugs. Of course, the managers decided this approach was better than investing in more GUI automation bringing an end to another GUI automation project.
Unfortunately, unlike the scary stories my father and uncles told around the campfires, stories of failed GUI automation are often true, and usually much scarier. Why are they so scary? Because I hear these sorts of stories repeated so often. It seems that we as a discipline rely on tribal knowledge where each generation simply learns through trial and error and the folktales of our elders and thrive more on hero worship of people who are often remarkably good at finding bugs by poking and prodding hour upon hour.
Now, if you haven’t caught on already you will know that I am no big fan of GUI automation. Not because I don’t think it can be useful. In fact, I think GUI automation can provide tremendous value in some situations. But unfortunately much of the automated GUI test cases I see (especially in examples) are poorly designed, simple rudimentary script-lets. Many of these automated tests are nothing more than mindless automated sequences of events contrived because the testers have been told to automate, but are not given strategic vision (why) and little to no tactical direction (what and how).
With little or no direction or goals, or without an in-depth understanding of the system they are testing the biggest problems with GUI automation is that many testers attempt to automate
- functional tests intended to expose computational errors in the business logic layer or in the underlying APIs
- usability tests intended to imitate ‘me’ trying to emulate the scenarios or tasks I think the customer might do
GUI automation is probably the least effective approach for functional testing. This is not to say that GUI automation will not find functional issues in the lower logic layers. I suspect we will always find ‘functional’ bugs (e.g. boundary issues, unhandled exceptions, string parsing errors, calculation problems, etc.) while testing through the UI. But, as indicated in my previous post, well-designed software is usually built in layers and a good many of the ‘functional’ issues we find today can likely be more efficiently found through more robust unit and component (API) levels of testing.
Perhaps even more silly than trying to use GUI automation for low level functional testing is the notion of using GUI automation to emulate a ‘user’ by scripting out prescriptive sequences of actions (often with hard-coded data) that are then played over and over again. Test automation cannot and should not attempt to replicate ‘me.’ I’ve said before the purpose of automation is to provide value to me, to free up my time, to increase my efficiency, and to help me be more effective in my job; automation does not replace me. Let’s face it…we (humans) are much better at evaluating the ease of use of software and whether scenarios that represent target customer segments are intuitive for those customers.
For example, I once had a conversation explaining that GUI automation runs much faster than I can interact with software, and sometimes I make mistakes when typing in something that throws an unexpected message or takes me down a path. My colleague replied, “Well, we can slow down the automation.” Why? Why in the hell would I want to slow down my automated tests? C’mon…we all should know by now that the 100% automation (or automate all tests) mantra is a ridiculous dream and I’ve heard more plausible fantasies from people on acid trips.
So, where does GUI automation add value. In my opinion, GUI automation is probably most effective in testing UI control properties and the event handlers between the UI layer and the API layers. It is also effective in behavioral testing areas such as performance and stress. And GUI automation is also much more effective in evaluating UI layout issues such as misaligned controls, or clipped or truncated controls on a window as compared to the human eye.
Similar to how we use different techniques to expose different categories of defects, and how we use different approaches to testing depending on the context or test objective test automation is a useful tool in our toolbox. It is certainly not the only tool. We can do some remarkable things with automation, but we must learn where GUI automation adds value and where other approaches testing (automated or manual) might be more effective.
API Testing – Thinking Differently About the Problem
Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.
A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).
So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.
Anyway, the final project after the first 10 weeks of the course is to design automated tests of 3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.
I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.
Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.
For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.
1: // <copyright file="simpletestcase.cs" company="TestingMentor">
2: // Copyright © 2009 by Bj Rollison. All rights reserved.
3: // </copyright>
4:
5: namespace TestingMentor.Sample
6: {
7: using System;
8: using System.IO;
9: using TestingMentor.Simulation;
10:
11: class TestCase
12: {
13: static void Main(string[] args)
14: {
15: int testCounter = 0;
16: // Read in an array of strings representing the test data.
17: // Of course this would likely come from a static test data file
18: // on a server or copied to a folder on the local machine
19: string[] testData = new string[]
20: { "90,Z",
21: "24798,惞",
22: "0,null",
23: "65536,Error",
24: "-1,Error",
25: "1.5,",
26: "xyz,xyz"
27: };
28:
29: // Loop through each test data string
30: foreach (string test in testData)
31: {
32: testCounter++;
33: // This nested try/catch block catches invalid test data
34: // but allow additonal tests in the testData array
35: try
36: {
37: // Parse each string into the test data and expected result
38: string[] testElement = test.Split(',');
39: string expectedResult = testElement[1];
40: string actualResult = String.Empty;
41:
42: // Convert the string to a type int value
43: int value = int.Parse(testElement[0]);
44:
45: // We need a way to handle int values 0 through 32 which are
46: // control characters, this is an example of how to deal with
47: // a int value of 0 which is a null character
48: if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))
49: {
50: expectedResult = '\0'.ToString();
51: }
52:
53: // This nested try/catch block tests catches exceptions thrown by
54: // the method under test. If the method under test throws an
55: // exception we certainly want to test for that case!
56: try
57: {
58: // Call the API method under test
59: char result = Converter.ConvertValueToUnicodeChar(value);
60: actualResult = result.ToString();
61: }
62:
63: catch (ArgumentOutOfRangeException)
64: {
65: actualResult = "Error";
66: }
67:
68: catch (Exception)
69: {
70: // if this happens this is a failure because the documentation
71: // states that this method will only throw an
72: // ArgumentOutOfRangeException.
73: actualResult = "Non-specific or unexpected error thrown";
74: }
75:
76: // Call a simple oracle and log results
77: if (String.Equals(actualResult, expectedResult))
78: {
79: // log pass
80: Console.WriteLine("{0} Pass", testCounter);
81: }
82: else
83: {
84: // log fail...of course log as much detail as possible
85: Console.WriteLine("{0} Fail", testCounter);
86: }
87: }
88:
89: catch (FormatException)
90: {
91: // log the test data for this test as incorrect, test is skipped
92: Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);
93: }
94: }
95: }
96: }
97: }
Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).
Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)
The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?