Archive for the ‘Test Automation’ Category
Dealing with locale/language specific static test data
It has been sometime since my last post. This seems to happen every so often lately; not because I don’t have anything to write about but mostly due to having too many irons in the fire so to speak and juggling hot irons is never fun and one is always going to drop. Also about this time every year I go sailing in the San Juan Islands or the Gulf Islands of British Columbia. This year I went to the San Juan Islands, and spent a few days incommunicado anchored in Shallow Bay on Sucia Island. Sucia. Echo Bay is a great anchorage with sandy beaches (unusual for the PNW), and the famous China Caves to explore.
Another place I have been known to explore from time to time is the Stack Exchange Software Quality Assurance and Testing forum. There are many interesting questions and a great variety of responses that offer a wealth of information or provide different perspectives. Recently a question was posed about how to read in static test data for a specific locale or language. Many regular readers know that I am a strong proponent of pseudo-random test data generation in conjunction with automated testing to increase the variability of test data used in each test iteration and generally improve test coverage. But I also understand the value of static test data in providing a solid baseline, and in some cases enabling access to specific test data in different locales or languages.
For example, suppose I am testing a text editor application and I want to read in a text file in the appropriate language based on the operating system current users locale settings. In this situation, I could save a text file containing strings or sentences for each target language or locale dialect. Each file would get a unique name based on the 3 letter ISO-639-2 language name (the complete list is at http://www.loc.gov/standards/iso639-2/php/code_list.php), prepended it to a common filename that describes the contents and the appropriate extension. For example,
- ENG[TestData].txt would be English
- ZHO[TestData].txt would be Chinese
- DEU[TestData].txt would be German
To get the appropriate text file auto-magically read in to the test at runtime the only thing we would need to do is to get the current user locale using the CultureInfo class Three Letter ISO Language Name property in C#.
1: string testDataFileName = "testdata.txt";
2:
3: CultureInfo ci = CultureInfo.CurrentCulture;
4:
5: // Path to server location where static files exist
6: string path = Path.GetFullPath(
7: Environment.GetFolderPath(Environment.SpecialFolder.Desktop));
8:
9: // Read file contents
10: using (StreamReader readFile =
11: new StreamReader(Path.Combine(
12: path, string.Concat(
13: ci.ThreeLetterISOLanguageName, testDataFileName))))
14: {
15: //parse test data and do test stuff
16: }
Notice we concatenate the filename (and extension) and the 3-letter ISO language name in line 13 and then combine that with the path to the file location and read the file contents using StreamReader.
But, we might need more specialization depending on what we are testing. For example, if we were testing a spell checker for US versus Great Britain (and Canada), or testing simplified Chinese and also traditional Chinese. In this case the ISO 639-2 specification does not delineate between simplified Chinese and traditional Chinese or US English and British English. In this case we could “make up” a 3-letter designation such as GBR for Great Britain, or CHT for Chinese (traditional).
Or, perhaps a better solution would be to use the Locale Identifiers (LCID) used by Windows to identify specific locales (rather than languages). The solution is identical to the above except instead of calling the ThreeLetterISOLanguageName property we call the LCID property as illustrated below.
1: string testDataFileName = "testdata.txt";
2:
3: CultureInfo ci = CultureInfo.CurrentCulture;
4:
5: // Path to server location where static files exist
6: string path = Path.GetFullPath(
7: Environment.GetFolderPath(Environment.SpecialFolder.Desktop));
8:
9: // Read file contents
10: using (StreamReader readFile =
11: new StreamReader(Path.Combine(
12: path, string.Concat(
13: ci.LCID, testDataFileName))))
14: {
15: //parse test data and do test stuff
16: }
Of course, now we would need to name our static file names with the appropriate LCID decimal number such as
- 1028testdata.txt would be traditional Chinese used in Taiwan, and
- 2052testdata.txt would be for simplified Chinese used in PRC
Personally, I prefer getting the LCID as it provides greater control and more specificity. But the down side of using LCIDs is that if you may end up having multiple files that contain the same contents. For example, although Singapore, Malaysia, and PRC all use simplified Chinese there are 3 different LCIDs.
There are other properties that allow you to get the culture info for the current user in Windows, and the right property to use ultimately depends on your specific needs. But, CultureInfo class members can easily be used to manage localized static data files or even manage control flow through an automated test that has specific dependencies on a language or a locale setting.
Test Automation: Beyond Rudimentary Script-lets
When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.
Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.
The purpose of the test
Think about abstract big picture rather than a myopic singular purpose
In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.
The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.
As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.
Automation perspective
Think about programming the computer rather than trying to program human interactions
In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.
In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.
But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.
Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.
Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully
Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.
Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.
Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.
The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.
Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.
1: namespace HelloTest
2: {
3: using System;
4: using System.Diagnostics;
5: using System.IO;
6: using System.Runtime.InteropServices;
7: using System.Windows.Automation;
8: using TestingMentor.TestTool.Babel;
9:
10: internal class Constant
11: {
12: internal const string AutName = "hello.exe";
13: internal const string AutFormTitle = "HelloForm";
14: internal const string AutNameTextBox = "textboxFirstName";
15: internal const string AutOKButton = "buttonOK";
16: internal const string AutClearButton = "buttonClear";
17: internal const int FirstNameTextboxLength = 25;
18: internal const int MaxPollCount = 50;
19: }
20:
21: public class NativeMethod
22: {
23: [DllImport("user32.dll")]
24: public static extern IntPtr GetForegroundWindow();
25: }
26:
27: class SampleHelloTest
28: {
29: static void Main(string[] args)
30: {
31: try
32: {
33: Stopwatch timer = new Stopwatch();
34: timer.Start();
35:
36: int testIterationCount = 25;
37:
38: // Declare a new process and automation elements
39: Process autProcess = new Process();
40: AutomationElement desktopPath = AutomationElement.RootElement;
41: AutomationElement autForm = null;
42: AutomationElement nameTextbox;
43: AutomationElement clearButton;
44: AutomationElement okButton;
45:
46: // Launch the AUT
47: autProcess = LaunchApplication(
48: Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
49: Constant.AutName),
50: desktopPath,
51: Constant.MaxPollCount,
52: ref autForm);
53:
54: // Get the UI automation element object on main form
55: GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);
56:
57: // Verify the textbox control is 'visible'
58: if (nameTextbox != null &&
59: (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))
60: {
61: // iterate through a specified number of test data variants
62: for (int testCount = 0; testCount < testIterationCount; testCount++)
63: {
64:
65: int seedValue;
66: string testData = GetRandomTestDataVariant(out seedValue);
67:
68: // Put test string into textbox
69: SetTextboxText(nameTextbox, testData);
70:
71: // Emulate clicking the OK button to instantiate the message box
72: PushButton(okButton);
73:
74: // Oracle to compare actual vs expected result and log results/important information
75: string actualResult = string.Empty;
76: if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))
77: {
78: // Log Test Results as appropriate
79: Console.WriteLine(testData);
80: }
81: else
82: {
83: Console.BackgroundColor = ConsoleColor.Red;
84: Console.WriteLine(
85: "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",
86: testData, actualResult, seedValue);
87: }
88:
89: // Emulate clicking the clear button
90: PushButton(clearButton);
91: }
92: }
93: else
94: {
95: Console.WriteLine("TEST FAILURE: Textbox control not found");
96: }
97:
98: // Clean up the test environment
99: CloseAutProcess(autProcess);
100:
101: // Log total elapsed time and any additional important info
102: timer.Stop();
103: Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);
104: }
105: catch (Exception e)
106: {
107: Console.WriteLine(e.ToString());
108: }
109: }
110:
111: #region HELPER METHODS
112: /// <summary>
113: /// Launches AUT process; or throws exception if process fails to launch
114: /// </summary>
115: internal static Process LaunchApplication(
116: string autPathAndFilename,
117: AutomationElement desktop,
118: int maxPollCount,
119: ref AutomationElement myAutForm)
120: {
121: Process myProc = new Process();
122: myProc.StartInfo.FileName = autPathAndFilename;
123: if (myProc.Start())
124: {
125: // Find the MyFontForm window
126: int pollCount = 0;
127: do
128: {
129: myAutForm = desktop.FindFirst(
130: TreeScope.Descendants, new PropertyCondition(
131: AutomationElement.AutomationIdProperty, Constant.AutFormTitle));
132: pollCount++;
133: System.Threading.Thread.Sleep(100);
134: }
135: while (myAutForm == null && pollCount < maxPollCount);
136:
137: if (myAutForm == null)
138: {
139: throw new ArgumentNullException("Failed to find AUT form");
140: }
141: }
142: return myProc;
143: }
144:
145: /// <summary>
146: /// Get UI automation element objects on main form by property name
147: /// </summary>
148: private static void GetUIAutomationElements(
149: AutomationElement autFormName,
150: out AutomationElement nameTextbox,
151: out AutomationElement okButton,
152: out AutomationElement clearButton)
153: {
154: nameTextbox = autFormName.FindFirst(
155: TreeScope.Descendants, new PropertyCondition(
156: AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));
157: okButton = autFormName.FindFirst(
158: TreeScope.Descendants, new PropertyCondition(
159: AutomationElement.AutomationIdProperty, Constant.AutOKButton));
160: clearButton = autFormName.FindFirst(
161: TreeScope.Descendants, new PropertyCondition(
162: AutomationElement.AutomationIdProperty, Constant.AutClearButton));
163: }
164:
165: /// <summary>
166: /// Generate a pseudo-random Unicode string using the Babel.dll automation library
167: /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm
168: /// </summary>
169: private static string GetRandomTestDataVariant(out int seedValue)
170: {
171: string randomString = string.Empty;
172: do
173: {
174: Random prng = new Random();
175: seedValue = prng.Next();
176: StringGenerator sg = new StringGenerator();
177: sg.Info.Seed = seedValue;
178: sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;
179: sg.Info.RandomizeCharacterCount = true;
180: sg.Info.AllowSurrogatePairCharacters = false;
181: randomString = sg.Polyglot();
182: }
183: while (randomString.Contains("\0"));
184:
185: return randomString;
186: }
187:
188: /// <summary>
189: /// Generic method that enters the test data into the a specified textbox
190: /// Throws exception if the length of the testData argument exceeds the size
191: /// property of the textbox control, or textbox control in not enabled
192: /// </summary>
193: private static void SetTextboxText(AutomationElement inputTextbox, string testData)
194: {
195: try
196: {
197: ValuePattern input =
198: (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);
199: input.SetValue(testData);
200: }
201: catch (ElementNotEnabledException)
202: {
203: throw new ElementNotEnabledException("Textbox is not enabled.");
204: }
205: catch (InvalidOperationException)
206: {
207: throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");
208: }
209: }
210:
211: /// <summary>
212: /// Oracle that compares the randomly generated string against the variable
213: /// result in the message box text
214: /// </summary>
215: private static bool ValidateMessageBoxStringOracle(
216: string testData, int maxPollCount, out string actualResult)
217: {
218: bool result = false;
219: IntPtr msgBoxHandle = IntPtr.Zero;
220: AutomationElement messageBox;
221: int pollCount = 0;
222: actualResult = string.Empty;
223: try
224: {
225: do
226: {
227: msgBoxHandle = NativeMethod.GetForegroundWindow();
228: messageBox = AutomationElement.FromHandle(msgBoxHandle);
229: string className =
230: (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);
231: System.Threading.Thread.Sleep(100);
232: pollCount++;
233: }
234: while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);
235:
236: if (messageBox != null)
237: {
238: AutomationElement textbox = messageBox.FindFirst(
239: TreeScope.Descendants, new PropertyCondition(
240: AutomationElement.AutomationIdProperty, "65535"));
241: string messageBoxText =
242: (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);
243: actualResult =
244: messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);
245:
246: if (actualResult.Equals(testData))
247: {
248: result = true;
249: }
250:
251: AutomationElement messageboxOKButton = messageBox.FindFirst(
252: TreeScope.Descendants, new PropertyCondition(
253: AutomationElement.ClassNameProperty, "Button"));
254: PushButton(messageboxOKButton);
255: }
256: else
257: {
258: actualResult = "Messagebox not found.";
259: }
260:
261: return result;
262: }
263: catch (ArgumentOutOfRangeException)
264: {
265: throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");
266: }
267: catch (NullReferenceException)
268: {
269: throw new NullReferenceException("Messagebox unexpectedly became null");
270: }
271: }
272:
273: /// <summary>
274: /// Generic method to push a button control
275: /// Throws exception if button control is not enabled, or buttonName is null
276: /// </summary>
277: private static void PushButton(AutomationElement buttonName)
278: {
279: try
280: {
281: InvokePattern pushButton =
282: (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);
283: pushButton.Invoke();
284: }
285: catch (ElementNotEnabledException)
286: {
287: throw new ElementNotEnabledException("Button is not enabled.");
288: }
289: catch (InvalidOperationException)
290: {
291: throw new InvalidOperationException("Button not found.");
292: }
293: }
294:
295: /// <summary>
296: /// Generic method to close an AUT process
297: /// </summary>
298: private static void CloseAutProcess(Process autProcess)
299: {
300: try
301: {
302: autProcess.CloseMainWindow();
303: autProcess.WaitForExit(500);
304: if (!autProcess.HasExited)
305: {
306: autProcess.Kill();
307: }
308: }
309: catch (InvalidOperationException)
310: {
311: throw new InvalidOperationException("Specified process not found.");
312: }
313: }
314: #endregion HELPER METHODS
315: }
316: }
Automation isn’t bad; bad automation is bad!
As I write this I am flying at 38,000 feet or so across the Artic Circle. I am on my way to the Netherlands where I have been invited to keynote the TestNet conference. I am very excited to attend this conference and to meet new friends and connect with old ones. But, the guy beside me is hacking his head off (God, I hope he is not contagious). There is an elderly gentleman sitting in front of me constantly readjusting his chair up and back, up and back. And, I think the woman behind me has a bladder problem because about every 15 minutes she slams her tray up into my back, and jerks violently on the back of my chair as she lifts herself up. Fortunately on this flight there are no crying children within earshot (as of yet anyway). Since I don’t sleep well on airplanes without being in a drug induced stupor I thought I would hammer some thoughts on automated testing.
I am still surprised how often I hear people disparage automated testing. It’s almost like a rampant phobia similar to fear of water, fear of heights, or fear of the unknown. What leads people to proclaim disingenuous tripe that I sometimes read on blogs, or discussion forums? For example, the other day I read “automation will not find any new bugs.” Well, I guess that is the experience of some folks. But of course we could say that any inexperienced, untrained, or non-thinking person who uses a tool is likely to misuse the tool and get hurt, do damage, or come to a false conclusion that the tool doesn’t work. Automated tests are not bad; stupid automated tests are bad. If your test automation is not providing value don’t blame the tool…maybe a little introspection is in order.
All Automation Is Not Equal
The first problem in almost any discussion about automation is the lack of context. Automated tests are different and when done well can be effective at helping us identify some types of functional issues. An automated unit test suite can be a powerful tool for developers during refactoring. Of course some bugs escape unit testing because unit tests are not intended to be comprehensive. I have long been a proponent of functional testing at the API layer and this is where my team plays today. I see first hand the value of our automated tests in the early identification of potentially critical functional issues during our continuous integration process. While this level of testing is effective in helping prevent build breaks, it is not comprehensive and does not test the behavior of the application through the user interface. And then, comes GUI automation.
Why (Most) GUI Automation Sucks!
Although I am not a big fan of GUI automation I have done it, I teach it, and I know that it can be of value in many ways. Last week I was discussing reasons for intermittent failures in test automation with a colleague. Of course, a large number of intermittent failures occur in GUI automation for a variety of reasons (that’s a different post somewhere in the future). But, as I was reviewing the test code something dawned on me…the tests sucked!
It seems that when many people sit down to automate a GUI test (using just about any test automation paradigm) the automation is about a brain dead as a zombie. I suspect that when folks sit down to automate a GUI test the first thing they do is step through the UI. Then the tester automates each step until he/she arrives at some final state and validates the results with an equally brain dead oracle. In other words a lot of GUI automation is comprised of simplistic scripts that attempts to emulate human behavior patterns rather than being a well-designed test that capitalizes on the power of the computer as a tool.
Let’s look at a simple example; the “Hello” application. The application takes a first name string then displays a message box that reads “Hello [name]!”
Here is an example of a brain dead automated test that we might find if we asked someone to automate this “feature.”
1: namespace BraindDeadAutomatedGuiTest
2: {
3: class Program
4: {
5: [STAThread]
6: static void Main(string[] args)
7: {
8: // Launch the AUT
9: Process testAut = new Process();
10: testAut.StartInfo.FileName = "Hello.exe";
11: testAut.Start();
12:
13: // Hard coded sleep to give AUT time to instantiate
14: System.Threading.Thread.Sleep(1000);
15:
16: // Use key mnemonic to set focus to textbox
17: // Often this is skipped because people assume this textbox has focus
18: SendKeys.SendWait("%f");
19:
20: // Enter hard-coded test data
21: SendKeys.SendWait("Boob");
22:
23: // Tab to the OK key and press
24: SendKeys.SendWait("{Tab}");
25: SendKeys.SendWait("{ENTER}");
26:
27: // Another hard coded sleep to make sure messagebox appears
28: System.Threading.Thread.Sleep(1000);
29:
30: // Now comes the brain dead oracle
31: Clipboard.Clear();
32:
33: // Capture the messagebox text
34: SendKeys.SendWait("^(+c)");
35:
36: // Now get the text from the clipboard and see if it contains the
37: // hard-coded test data; often I see people compare strings
38: // verbatim which is a recipe for false positives & maintenance
39: string actualResult = Clipboard.GetText(TextDataFormat.Text);
40: if (actualResult.Contains("Boob"))
41: {
42: Console.WriteLine("Pass");
43: }
44: else
45: {
46: Console.WriteLine("Fail");
47: }
48:
49: // Clean up - kill the messagebox then close the AUT
50: SendKeys.SendWait("{ESC}");
51: testAut.CloseMainWindow();
52: }
53: }
54: }
Now, you might be saying that this is an absurd example, Really? I don’t think so. I see this sort of automation all the time in examples on the web and at conferences. For example, how often have you ever seen an example along the lines of:
- Launch IE (or some other browser)
- Navigate to “http://google.com”
- Enter “Ruby” in test box
- Press Search button
- Verify “Ruby” appears in results
Now, if you really expect this type of automation to add value then you should really consider a new line of work…perhaps assembly line work.
Well, we are about 1 1/2 hours out from Amsterdam and breakfast is about to be served, so I am going to wrap this up now. Yes, I learned a long time ago not to complain if I don’t at least offer a solution (whining is never appreciated). So, later this week, perhaps on the trip home on Wednesday, I will suggest one possible solution to automate even this simple app that could provide value by increasing test coverage and potentially exposing some errant behavior (bugs).
(Oh…and by the way, while writing this a little dirt merchant in pajamas and a pacifier sticking out of his pie-hole started doing laps through the cabin banging off the sides of the chairs…but at least he wasn’t crying!)
Bugs that automated tests aren’t good at finding
In response to my last post, Shrini suggested, “You should probably do a post on types of bugs that unit testing (or developer testing at any level) would’nt catch. That would be a fitting reply to all those who swear by automated unit testing/API Testing.”
I am a big proponent of well-designed automated tests, and I have written a lot about the value of automation as an effective tool in the development process. But, I also know that automated tests find a relatively low number of bugs throughout a product lifecycle. If you think the primary goal of an automated testing effort is to find lots of bugs, or the same types of bugs as a person then you probably don’t know very much about test automation. Personally, I don’t think of automated testing as a proxy for the tester or as a bug finding solution. Sure, some automated tests can help find bugs, but more importantly it is one approach I might use for
- defect prevention (esp. computational logic problems),
- earlier identification of key integration issues,
- potential degradation of critical areas (battery, performance, memory),
- efficient execution of redundant ‘checks’ (if necessary or for confidence)
- more effective/precise ‘oracles’ as compared to humans
- cost reduction in long term sustained engineering
In my new role at Microsoft I am leading a team of great SDETs that tests primarily at the foundation (API) level. Everyday I see first hand the value that automated regression testing at the unit and API levels of testing provides to the overall production lifecycle (because we build everyday). While the tests we run ultimately affects the customer’s experience, it also helps reduce our overall production costs and drives certain aspects of ‘quality’ upstream. This is especially important in large scale, enterprise systems where a build breaks or integration failures can be costly and lead to unnecessary delays.
But, this is just one level of testing, and I realize that most testers are testing at the ‘system’ level of testing and rely heavily on GUI automated tests. I have never been a big fan of GUI automated tests; especially GUI regression testing. This is not to suggest that all GUI automation is bad, and there are several situations where GUI automated tests can be especially valuable to a test team. However, there are a few situations where I think automating tests that manipulate the GUI is mostly a complete waste of time such as attempting to emulate the behavior of a customer “scenario,” or trying to verify the ‘correctness’ of what the customer “sees” (e.g. visual verification).
I once told a colleague that the computer is really bad at emulating “me.” I said, “for example, sometimes when I type I hit the wrong key, or I lay my finger on a key for too long and that stupid sticky keys message box appears, and sometimes my hand position on my laptop causes my insertion point to jump to some random point in the text body and I have to reset it to the correct location. You can’t automate the unpredictability of me typing!” He said, “Sure I can!” I said, “Ok..I know that we can automate randomness or errant behaviors to some extent, but why would we?”
I sometimes think that in our zeal to “automate everything” we forget that our products often get a lot of “face time” through self-hosting, product partners, beta releases, and other strategies that are intended to get feedback on unanticipated or escaped functional issues, behavioral issues in how people might use the features in different ways (scenarios), and of course the “look” or visual anomalies that might occur while using the product.
As an example, the other day I was searching for a new program for my students to practice their GUI automation skills (yes, while I generally dislike GUI automation it is still a good skill to have). I came across text editor application called XINT. Within minutes of downloading the application and exploring the features I found a bug with the feature that inserts a URL into the text body.
As a simple example I was going to show how we can automatically go through the menu structures to make sure there are no changes, and that the menu items trigger the appropriate events (e.g. displaying a dialog). I was going to develop an automated test demo that systematically marched through the menu structure and validated the expected event triggered by that menu item. An automated GUI test such as this could provide a high level ‘check’ much more quickly than could be performed by a tester. Also, it automates a redundant ‘check’ of the application under test that we might want to perform on each new build to potentially ‘look’ for changes. I wouldn’t expect this automated test to find a lot of bugs, but it would clue us in very quickly to any changes in the menu structure and any anomalies with basic functional expectations triggered by those menu items. Quick and simple.
So, we get to this menu item and programmatically simulating the menu item “click” via a SendMessage() call to the appropriate menu item the ‘correct’ dialog appeared. OK so far. We are not testing the “insert a URL” functionality; my high level ‘check’ is simply checking that the correct event occurred (in this case a dialog appeared). So, now I am going to send a message to “click” the Cancel button and my expected result is for this dialog to go away and focus will return back to the application under test (XINT). But, in this case the URL insertion dialog is repainted with the sample text removed, and a new label. (It gets even better because clicking Cancel again forces the sample text into the text edit control. You have no choice at this point…you selected to insert a URL…so you are going to get whether you like it or not!) But, there is a bug, and my automation could have found it.
But, a little more exploration and I discover another anomaly that almost defies logic, and that automation would certainly not have found. When I pressed the ALT key, I noticed something odd. The fricking Cancel button disappears! However, as far as the ‘system’ is concerned the handle to this control is still there and I can programmatically send a message to that button control. But, to me the user…it’s gone!
This is just one example of the types of issues that automation is not especially suited for, and even trying to automate a test for these types of issues is not just an effort in futility, I would say it is damn near insanity. Of course there are other types of issues that automation is not especially efficient or effective in detecting such as ease of use, consistency in layout or behavior, general “look and feel,” and most importantly customer scenarios or user stories.
Bottom line, use automation for things that computers are really good at such as computational logic and redundant ‘checks’ that we might want to do after each new build. And, use humans to test for the things that humans are really good at which often happen to be the things that will delight your customer if you get it right, or the things that will piss them off if you screw it up!
Test Automation: Checking for Bit-ness
I am almost through my first week in my new job here at Microsoft and trying to get my head wrapped around everything. It is really exciting to work on new technologies and in a feature area that helps users stay connected with friends and family in amazing ways. For now, let’s just say that I feel like I am treading water which is not unusual when changing jobs within Microsoft. But, of course, that is a good thing. If I felt like I could’ve easily stepped into this job I would not feel challenged, I would not open my mind to learn new things, and I would likely be bored after a few months. Yes…I am a bit overwhelmed in a very exciting way!
Of course, I am still teaching classes at the University of Washington Extension, and am currently teaching a class in test automation design. A few sessions ago we hit an issue with some canned examples failing to run properly on 64-bit operating systems. This problem appeared for 2 reasons:
- I developed the lab examples 2 years ago before I upgraded my home system to a 64-bit machine and have only ran them in the classroom environment since
- I hard-coded the Program Files folder environment path to C:\Program Files.
Of course, it is a bit embarrassing as an teacher when your samples do not run (it’s even worse when they don’t even build, but that is a different story). In a training situation it is often easier to control the environment to minimize problems and allow learners to (initially) focus on the main topic or skill being presented. But, the downside is that many of us don’t work in ‘controlled’ environments in the real world. And while I am adamantly opposed to hard-coded strings in code, I find that sometimes I ignore my own advice in samples or demo code (and it often comes back to haunt me).
The specific problem was that on a 64-bit operating system the application is installed to the Program Files (x86) folder instead of simply the Program Files folder and the application failed to launch when we attempted to start the process. The solution is actually quite simple in this case because we can actually call the ProgramFiles member of the Environment.SpecialFolder enumeration. For example, a hard-coded path on a default installation
string autPath = @"c:\program files\[autFolder]\[autName.exe]";
failed on a 64-bit system, so a better approach to detect the ‘correct’ program files directory might be
string autDefaultFolderName = "[aut_folder_name]"; string autExecutableName = "[filename.exe]"; string autPath = System.IO.Path.Combine( Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles), autDefaultFolderName, autExecutableName);
The ProgramFiles member of the Environment.SpecialFolder will return Program File on a 32-bit machine, and Program Files (x86) on a 64-bit machine.Of course, this works fine for the default installation folder, but if for some reason you need to run your automation on installations of an application under test in a non-default directory then you would need to qualify the path in other ways.
Also, this is just one difference between a 32 and 64-bit operating system environment, but there are other situations where we want to detect bit-ness so we can write a single test rather than one test for 32 bit operating system environments and another for 64-bit operating system environments. Fortunately, the .NET 4.0 framework makes it really easy to detect whether a machine is 64-bit or not with the Environment.Is64BitOperatingSystem property. For example, you can control flow for 64 vs 32 bit with a simple decision such as:
if (Environment.Is64BitOperatingSystem) { // TODO 64-bit stuff } else { // TODO 32-bit stuff }
But, if you are running the .NET 3.5 framework or earlier it becomes a bit more complicated to detect 64-bit operating systems. The following method can help determine whether or not your operating system environment is 64-bit
public static bool Is64BitOperatingSystem() { if (!string.Equals(Environment.GetEnvironmentVariable( "PROCESSOR_ARCHITECTURE"), "x86", StringComparison.OrdinalIgnoreCase) || !String.IsNullOrEmpty(Environment.GetEnvironmentVariable( "PROCESSOR_ARCHITEW6432"))) { return true; } return false; }
There are also Win32 API functions we can P-Invoke such as GetSystemInfo,
So, rather than having separate automated tests for 32 and 64 bit operating systems a better test design approach might be to simply detect the bit-ness and auto-magically set variables or redirect the control flow path through your test for the appropriate operating system environment.
Generating Random Dates
Last week was mostly a blur for me. Three hockey games (1 game with my regular team, 1 substituting on another team, and 1 in the over 40 league). By Thursday I was pretty wiped out, and with all the holiday stuff going on I completely blew off updating my blog.
Since my daughter was about 6 months old her and I have always had a daddy & daughter day at least once per week where I focus 100% on her and we do what she wants to do. About every 2 weeks she wants to go ice skating, and she completely surprised me by saying that she wanted to go to a free hockey clinic for kids for our our daddy & daughter day on Sunday. Castle Ice hosts a free hockey clinic for kids once per month. Although my daughter understands all the rules and enjoys watching games (she has become quite a Sid the Kid fan), I know that the odds of her ever taking up the sport are somewhere between 0 and .00001. But, she said she had a great time pushing the puck around on the ice and doing some of the drills with the other kids, and I was sure proud of her for giving it a try.
Ok…enough about my excuses for not posting last week, and time to move on from combinatorial testing and get back to something else I am interested in…random test data generation.
Every once in awhile a discussion comes up about how to generate random dates. There are several examples on the web to generate random dates. But many of them hard code locale specific formats such as month/day/year or day/month/year. Some allow you to specify a separator character such as a dash (-) or a forward slash (/) or other character. Many of these generators are limited to out-dated formatting options compared to the formats that are available (see MSDN Day, Month, Year and Era format picture).
In an earlier post I discussed how to we can programmatically change the date format to incorporate globalization testing or international sufficiency testing in our automated tests. I also created an automation library called GlobalTester to help testers change other settings to push international sufficiency testing upstream, and I talk about that in a previous post as well.
So, instead of generating dates based on some hard-coded format (that may or may not be the same format used by the current user locale settings), or using some hard-coded separator character (that may or may not be the character for the current user locale) I decided to create a method that focused on generating random dates and allow the operating system to decide the proper formatting.
An easy way to generate a single random date string within a specified range that is formatted to the current user settings for the date format is illustrated below.
1: public static string RandomDate(DateTime startDate, DateTime endDate)
2: {
3: try
4: {
5: Random prng = new Random();
6: int range = ((TimeSpan)(endDate - startDate)).Days;
7:
8: // Returns a string that follows the current short date formatting options
9: return startDate.AddDays(prng.Next(range)).ToShortDateString();
10:
11: // Returns a string that follows the current long date formatting options
12: // return startDate.AddDays(prng.Next(range)).ToLongDateString();
13: }
14: catch (ArgumentOutOfRangeException)
15: {
16: throw new ArgumentOutOfRangeException(
17: "The end date must be later than the start date");
18: }
19: }
Now, all we have to do is define our start date range and our end date range and call the method. To generate a random date string using the long date format settings for the current user simply change the RandomDate method above to return the value in line 12.
1: static void Main(string[] args)
2: {
3: DateTime startDate = new DateTime(
4: DateTime.Now.Year - 100, DateTime.Now.Month, DateTime.Now.Day);
5: DateTime endDate = new DateTime(
6: DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day);
7:
8: Console.WriteLine(RandomDate(endDate, startDate));
9: }
Because we can use a constructor to pass int values for the year, month and day this approach allows us to set a date range in the past, the future, or we can specifically define a range of dates.
The above method works well if we only need a single random date, but if we need several random dates this method will not produce reasonably random dates. To produce multiple random dates at one time we should use a slightly different approach. A better approach to generate multiple random dates at one time is to use an IEnumerable interface as illustrated.
1: public static IEnumerable<DateTime> RandomDate(
2: DateTime startDate,
3: DateTime endDate)
4: {
5: Random prng = new Random();
6: int range = ((TimeSpan)(endDate - startDate)).Days;
7: while (true)
8: {
9: yield return startDate.AddDays(prng.Next(range));
10: }
11: }
1: static void Main(string[] args)
2: {
3: DateTime startDate = new DateTime(
4: DateTime.Now.Year - 100, DateTime.Now.Month, DateTime.Now.Day);
5: DateTime endDate = new DateTime(
6: DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day);
7:
8: int i = 0;
9: foreach (DateTime date in RandomDay(startDate, endDate))
10: {
11: Console.WriteLine(date.ToLongDateString());
12: Console.WriteLine(date.ToShortDateString());
13: if (++i == 10)
14: {
15: break;
16: }
17: }
18: }
Compared to all the random date generators out there, this is about the simplest implementation available. We don’t have to use a generator tool or library, we can simply include these methods in our libraries or code. If you know of an easier solution, please let us know.
Some additional information can be found at
Automated test case design…the first step
This week I am in Hyderabad, India. It has been some time since I have been here, and it is really nice to get to this part of the world. It’s also nice to finally put a face to an email alias and get to meet new engineers in Microsoft IDC and reconnect with old friends. For those have never been here, India is a an incredibly amazing place with a rich culture. And the food…wow…it’s like a party in my mouth.
I am teaching 2 classes on various approaches and techniques used in software testing. Our course discusses various fault models, provides examples and hands-on practice with patterns of software testing (PoST) to help expose specific categories of functional bugs, structural test design to evaluate control flow to help reduce overall risk, and some internal tools. Of course, many of these concepts are intended to improve the effectiveness of our test designs and also provide a foundation when using an exploratory and even ad hoc (bug bash) testing approaches.
Of course much of the conversation tends to center around incorporating these foundational principles into the automated test designs. To me the act of designing an automated test is much more than recording some sequence of actions to emulate a user, or filling in a spreadsheet with keywords and hard-coded test data. Also, when I talk about automated test scripts I an not simply referring to some script-let that drives the automation to emulate the actions/inputs a user might perform. In fact those that read the blog regularly know that I am not a big fan of GUI driven automation. But, that is a tangent. Much of our automation spans from the API level to GUI level.
Of course, an automated test is code. We need competent test developers to write (nearly) bullet proof code. The code must be essentially bullet proof because every time an automated test throws a false positive our team loses a bit of confidence in the value the automation effort is providing. Of course, getting to this level of automation requires a strong design. Before we begin writing a single line of code for an automated test script we should consider a few key factors.
- Purpose – we need to define the primary objective of the test. Some tests are focused on evaluating a single outcome (micro), and some are designed to look for systemic type issues by emulating end-to-end scenarios (macro). Also, we must decide if this is a positive test that is intended to demonstrate the program functions as expected with specific or generalized valid inputs, or a negative test using invalid inputs that should trigger the appropriate exception handlers or error messages. A well-designed test has a clearly defined purpose with a predictable outcome that we can objectively evaluate against a predetermined expectation. Without a purpose or a predictable outcome we are simply automating events and at best hoping to expose some gross error.
- Oracle – this is perhaps the hardest part of automating a test. If we cannot automate the oracle then we should really consider whether to automate that test. Sure, automation can also be useful to ‘set-up’ the environment to a particular state, but I don’t consider that to be an automated test. In my opinion an automated test must have an oracle that can accurately determine the outcome of that test. The whole purpose of an automated test is to provide value to the testing effort and to free up my time. If I have to literally check the outcome of an automated test it isn’t really freeing up much of my time. There is is little more boring than sitting in front of a machine and watching an automated test. Also, in my opinion automated test oracles should evaluate whether or not the purpose or objective of the test succeeded or failed. Trying to design an oracle to look for systemic level problems is usually not effective other than identifying gross anomalies (crashes, hangs, etc.). But, the test should have checkpoints that are able to catch unexpected failures or inconclusive conditions that cause the test to exit prematurely or otherwise fail to execute as intended.
- Data – many tests from unit level tests to system level tests require test data. During the design phase of a test is a good time to think about what data we need for our tests. For example, does the test require specific ‘real-world’ data, or could we model the data by decomposing the data into equivalent partitions and generate parameterized random test data that is representative of the data required for the test. Many tests require some form of input, and using hard-coded values in an automated test just makes no sense at all.
- Approach – we also need to consider the approach for our test. For example, if our application under test requires input data we need to decide if we should use a data-driven approach with static test data, or will we generate random test data. Also, is this a combinatorial test, a permutation test, a state transition test, or are we looking for single mode faults and targeting individual parameters. The type of issue we are trying to detect or hypothesis we are attempting to validate might impact our approach used in the design. Now is also a good time to think about reuse, dependencies, and specific environment configurations necessary to run the test.
I am sure there are other things to consider. But, without a good design strategy our automated tests are likely to be fragile, error prone, and likely to provide little value in the long run.
Automating Screen Captures
When I first started at Microsoft I worked on the Windows 95 international test team. Not only did we focus on globalization testing, but we also did a lot of the localization testing for the East Asian language versions (Japanese, Korean, Simplified Chinese, and Traditional Chinese). Localization is the adaptation of software to a particular target market. Translation of resource strings is one of the most visible parts of the localization process, and in those days part of our testing effort was spent on translation validation (e.g. checking the strings for appropriate translation). (In retrospect, I now think it is a huge waste of time and resources to use professional testers to validate the translation “quality.”) During our localization testing cycles it was common practice to take screen shots of issues we found during localization testing. These screen shots often help put things in context for the localization engineers, and helped them troubleshoot the issue. Of course, there were other times when we took screen shots to put other anomalies in context.
I am generally not a big fan of just attaching screen images to bug reports carte blanche. Joe Strazzere has an excellent post describing why a screen shot doesn’t always add value in bug reports. But, I also know that there are times when screen shots of the desktop can be of value. When we are testing using pre-defined tests or exploratory approaches we are physically there to see anomalies as they occur. But, (hopefully) nobody is babysitting the machines our automated GUI test scripts are running on. So, when an unexpected anomaly occurs with automated GUI test scripts is might sometimes be beneficial to capture the desktop state as an image. This screen capture might provide some clues as to the state of the desktop at the time of unexpected automated test case failures resulting in an indeterminate automated test script result.
There are several 3rd party tools that some testers use to capture the desktop image and save it to a file. Automated test frameworks should also have methods that test developers can call to capture screen shots at important points during the execution of an automated test script, or when an anomaly occurs. But, if not, here is a simply method that will take a snapshot of the desktop and save a jpeg file of the desktop state.
1: using System.Drawing;
2: using System.Drawing.Imaging;
3: using System.Windows.Forms;
4:
5: ...
6:
7: public static void GetDesktopImage(string imageFilePath)
8: {
9: // Declare a structure for the width and height of the desktop
10: Rectangle rect = Screen.GetBounds(Point.Empty);
11:
12: // Declare a new instance of the bitmap class
13: Bitmap image = new Bitmap(rect.Width, rect.Height);
14:
15: // Capture the desktop
16: using (Graphics desktopImage = Graphics.FromImage(image))
17: {
18: desktopImage.CopyFromScreen(Point.Empty, Point.Empty, rect.Size);
19: }
20:
21: // Save the image to the specified path & filename
22: image.Save(imageFilePath, ImageFormat.Jpeg);
23: }
I would not capture tons of images during a test run; they just aren’t that valuable. And, I do not advocate capturing images to use as oracles…they are just too unreliable in my opinion. But, there are times when a screen capture of the desktop might add value and provide some context for the tester or the developer in troubleshooting issues.
UI Automation Beneath the Presentation Layer using .NET’s Reflection
This week I am in Bad Homburg, Germany where I gave a keynote at the Testing & Finance conference. It was really great to see some very dear friends and meet new colleagues from around the world. Bad Homburg is about 30 minutes by train from Frankfurt, and it is a quaint little town with beautiful parks and Schloss Homburg which was built around 1680 and was the summer residence of the German Kaiser.
Also this week the newest edition of the Automated Software Testing magazine published an article I wrote entitled UI Automation Beneath the Presentation Layer. It discusses some tips on how to use reflection for automating applications developed using managed code. If you have questions or comments about my article please let me know. Your feedback is appreciated.
I have always respected the work of Dion Johnson, and I was quite honored when he asked me to write an article for the magazine. The magazine and the Automated Testing Institute website provide a wonderful resource to people interested in software test automation. I really look forward to working with Dion and the Automated Testing Institute more closely in the future to provide whatever help I can to this monumental task he has undertaken. I also hope there are others out there who will share some of their ideas as well. I think the days of Not Invented Here (NIH) and I’ve Got A Secret (IGAS) syndrome are becoming issues of the past. We still have too much reinvention and not enough reuse of test code. I think that we have well established there is no single approach to testing that is effective in all contexts. We might approach problems from different perspectives, but I also think that all testers are passionate and have very similar goals of personal and professional improvement. To mature our discipline, and increase our effectiveness in our roles we should collaborate more and pool our resources, and the Automated Testing Institute is a growing community of professionals.
Technorati Tags: Automation,Reflection
Windows Live Tags: Automation,Reflection
WordPress Tags: Automation,Reflection
Globalization Testing: Basic International Sufficiency
I started my career at Microsoft in 1994 working on the Windows 95 International Test team. Globalization testing is a unique specialty in software testing just like performance, security, and other specific areas of testing. Globalization testing doesn’t necessarily require a tester to be bi-lingual, or be from a country other than the United States. A good globalization tester has an in-depth understanding of such things as character encoding types and issues associated with the different types, character mapping and conversion issues, data manipulation by the application, operating system, and network protocols.
Many people might also say that globalization testers also need to know that different locales (places) around the world use different formats for date and time (national conventions). For example, in the United States the default long date format is Thursday, June 03, 2010 but in Germany it is Donnerstag, 3. Juni 2010. A tester doesn’t have to ‘read’ German to see the abstract date format has changed from dddd, MMMM dd, yyyy to dddd, d. MMMM yyyy.
Testing for support of these different national conventions used around the world is referred to as basic international sufficiency testing. I suspect the reason why some people might assume basic international sufficiency testing these different national conventions is the domain of the globalization tester is because the national conventions are set by default on the different localized versions of a software product so that’s when they are tested. But, this reasoning is absurd!
First, not all products are “localized” into all languages or ‘locales.” So, who tests the Canadian long date format of MMMM-dd-yy, or the Georgian (Georgia) long date format of yyyy ‘წლის’ dd MM, dddd? Also, Vista and later versions of Windows allow the user to ‘customize’ the date and time “format pictures” to use different separator symbols and orderings.
Secondly, way too many bugs such as hard-coded date formats are found way too late in the testing cycle (because localized versions tend to lag US English language version). And of course, we all know the cost of finding bugs later in the lifecycle are more costly to correct.
So, we must ask if there is a way for basic international sufficiency testing to be ‘pushed upstream?’ And of course the answer is yes. The easiest way is to host a “globalization bug bash” early in the cycle. (A “bug bash” is a day where testers are given some basic training on attack patterns, fault models, etc., in a general focus area and then spend a day exploring different areas of the product trying to flush out bugs in a competition style format.) Another way is to assign each tester a different locale (preferably one that is not associated with a localized language version) and have them set their test and self-host environments to that locale during their testing.
This is easily accomplished on Windows test environments by having testers launch the Regional and Language control panel applet (the short cut is Start –> Run, then type “intl.cpl” without the quotes, and press the OK button).
This just tests for a basic level of international sufficiency, and any good tester would want to explore their project’s capability to support the more than 150 different locale national conventions at a deeper level. This is especially true if your product is going to be used by customers around the world (including Canada). But, of course, we don’t want to run the same tests on all 150+ locales supported by the operating system.
The national convention settings for a particular locale are stored in a data type called the LCID, and when we change our locale (Format on the latest Regional and Language control panel applet) through the user interface we are actually calling various National Language Support (NLS) APIs. A “world-wide” application should use the universal NLS APIs and data available via the operating system.
One way to test our application’s ability to correctly use the national convention data supplied by the operating system is to set customized conventions. For example, did you know the Windows 7 operating system allows a digit grouping symbol to be a string of up to 3 characters? Or the Negative sign symbol can be a string of up to 4 characters.
Although having testers change their default locale (Format) on their test environment and self-host machines is a good first step in basic international sufficiency testing, we also want to see if our application can process a negative value of “!NEG7” instead of just “–7,” and any textboxes correctly display the customized negative sign symbol (especially at the upper extreme boundary of the textbox size property.
To customize the national convention settings we simply click the Advanced settings… button on the Formats property sheet of the Region and Language control panel applet which instantiates a new dialog with 4 property sheets for Numbers, Currency, Time, and Date.
Solution for Test Automation
That’s all well and fine for basic testing, or testing a “few” customized values, but if we wanted to test the permutations for each convention, or the combination of different conventions on numbers, currency, time, or date formats the number of tests is astronomical. Typically, testers writing an automated test would try to navigate the user interface of the Regional and Language control panel applet and the Customize Format property sheets in order to set custom conventions.
In the past I provided some code snippets for changing the convention settings on the Customize Format property sheets on versions of Windows pre-Vista. Earlier this year I also provided code snippets for customizing the date format picture and the time format picture.
That’s all well and good, but I recently released a new test automation library called GlobalTester for test developers to use in their automated test scripts. The GlobalTester library provides testers methods to set custom national conventions for the current user without having to navigate the user interface of the Region and Language options control panel applet. These national conventions include number formats, currency formats, date formats, time formats, and also current location.
The following example illustrates how we might design a test script to customize the date format for a test and reset the date format to its original setting (restoring the test environment to pre-test conditions). (Usage documentation for the GlobalTester library is on the Testing Mentor website.)
1: namespace CustomizeDateSettingsExampleScript
2: {
3: using System;
4: using System.Globalization;
5: using TestingMentor.TestTool.GlobalTester;
6:
7: class MyTest
8: {
9: static void Main()
10: {
11: try
12: {
13: CustomDateFormat time = new CustomDateFormat();
14:
15: string defaultLongDateFormat =
16: CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern;
17: string newLongDateFormat = "MMM - d (yyyy) gg";
18:
19: if (time.ChangeLongDateFormat(newLongDateFormat))
20: {
21: // Launch AUT
22: // Execute test - (e.g. AUT implements long date string)
23: // Oracle - (e.g. compare long date format against customized pattern)
24:
25: // Reset test platform to original configuration
26: time.ChangeLongDateFormat(defaultLongDateFormat);
27: }
28: else
29: {
30: // Date format not changed; test not executed (e.g. invalid
31: // day, month, year, and era format pictures)
32: }
33: }
34:
35: catch(ArgumentOutOfRange e)
36: {
37: // Test script failure - (e.g. long date format string argument out of range)
38: }
39:
40: finally
41: {
42: // Log test results
43: }
44: }
45: }
46: }