Archive for May, 2011
Test Automation: Beyond Rudimentary Script-lets
When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.
Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.
The purpose of the test
Think about abstract big picture rather than a myopic singular purpose
In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.
The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.
As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.
Automation perspective
Think about programming the computer rather than trying to program human interactions
In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.
In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.
But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.
Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.
Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully
Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.
Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.
Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.
The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.
Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.
1: namespace HelloTest
2: {
3: using System;
4: using System.Diagnostics;
5: using System.IO;
6: using System.Runtime.InteropServices;
7: using System.Windows.Automation;
8: using TestingMentor.TestTool.Babel;
9:
10: internal class Constant
11: {
12: internal const string AutName = "hello.exe";
13: internal const string AutFormTitle = "HelloForm";
14: internal const string AutNameTextBox = "textboxFirstName";
15: internal const string AutOKButton = "buttonOK";
16: internal const string AutClearButton = "buttonClear";
17: internal const int FirstNameTextboxLength = 25;
18: internal const int MaxPollCount = 50;
19: }
20:
21: public class NativeMethod
22: {
23: [DllImport("user32.dll")]
24: public static extern IntPtr GetForegroundWindow();
25: }
26:
27: class SampleHelloTest
28: {
29: static void Main(string[] args)
30: {
31: try
32: {
33: Stopwatch timer = new Stopwatch();
34: timer.Start();
35:
36: int testIterationCount = 25;
37:
38: // Declare a new process and automation elements
39: Process autProcess = new Process();
40: AutomationElement desktopPath = AutomationElement.RootElement;
41: AutomationElement autForm = null;
42: AutomationElement nameTextbox;
43: AutomationElement clearButton;
44: AutomationElement okButton;
45:
46: // Launch the AUT
47: autProcess = LaunchApplication(
48: Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
49: Constant.AutName),
50: desktopPath,
51: Constant.MaxPollCount,
52: ref autForm);
53:
54: // Get the UI automation element object on main form
55: GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);
56:
57: // Verify the textbox control is 'visible'
58: if (nameTextbox != null &&
59: (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))
60: {
61: // iterate through a specified number of test data variants
62: for (int testCount = 0; testCount < testIterationCount; testCount++)
63: {
64:
65: int seedValue;
66: string testData = GetRandomTestDataVariant(out seedValue);
67:
68: // Put test string into textbox
69: SetTextboxText(nameTextbox, testData);
70:
71: // Emulate clicking the OK button to instantiate the message box
72: PushButton(okButton);
73:
74: // Oracle to compare actual vs expected result and log results/important information
75: string actualResult = string.Empty;
76: if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))
77: {
78: // Log Test Results as appropriate
79: Console.WriteLine(testData);
80: }
81: else
82: {
83: Console.BackgroundColor = ConsoleColor.Red;
84: Console.WriteLine(
85: "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",
86: testData, actualResult, seedValue);
87: }
88:
89: // Emulate clicking the clear button
90: PushButton(clearButton);
91: }
92: }
93: else
94: {
95: Console.WriteLine("TEST FAILURE: Textbox control not found");
96: }
97:
98: // Clean up the test environment
99: CloseAutProcess(autProcess);
100:
101: // Log total elapsed time and any additional important info
102: timer.Stop();
103: Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);
104: }
105: catch (Exception e)
106: {
107: Console.WriteLine(e.ToString());
108: }
109: }
110:
111: #region HELPER METHODS
112: /// <summary>
113: /// Launches AUT process; or throws exception if process fails to launch
114: /// </summary>
115: internal static Process LaunchApplication(
116: string autPathAndFilename,
117: AutomationElement desktop,
118: int maxPollCount,
119: ref AutomationElement myAutForm)
120: {
121: Process myProc = new Process();
122: myProc.StartInfo.FileName = autPathAndFilename;
123: if (myProc.Start())
124: {
125: // Find the MyFontForm window
126: int pollCount = 0;
127: do
128: {
129: myAutForm = desktop.FindFirst(
130: TreeScope.Descendants, new PropertyCondition(
131: AutomationElement.AutomationIdProperty, Constant.AutFormTitle));
132: pollCount++;
133: System.Threading.Thread.Sleep(100);
134: }
135: while (myAutForm == null && pollCount < maxPollCount);
136:
137: if (myAutForm == null)
138: {
139: throw new ArgumentNullException("Failed to find AUT form");
140: }
141: }
142: return myProc;
143: }
144:
145: /// <summary>
146: /// Get UI automation element objects on main form by property name
147: /// </summary>
148: private static void GetUIAutomationElements(
149: AutomationElement autFormName,
150: out AutomationElement nameTextbox,
151: out AutomationElement okButton,
152: out AutomationElement clearButton)
153: {
154: nameTextbox = autFormName.FindFirst(
155: TreeScope.Descendants, new PropertyCondition(
156: AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));
157: okButton = autFormName.FindFirst(
158: TreeScope.Descendants, new PropertyCondition(
159: AutomationElement.AutomationIdProperty, Constant.AutOKButton));
160: clearButton = autFormName.FindFirst(
161: TreeScope.Descendants, new PropertyCondition(
162: AutomationElement.AutomationIdProperty, Constant.AutClearButton));
163: }
164:
165: /// <summary>
166: /// Generate a pseudo-random Unicode string using the Babel.dll automation library
167: /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm
168: /// </summary>
169: private static string GetRandomTestDataVariant(out int seedValue)
170: {
171: string randomString = string.Empty;
172: do
173: {
174: Random prng = new Random();
175: seedValue = prng.Next();
176: StringGenerator sg = new StringGenerator();
177: sg.Info.Seed = seedValue;
178: sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;
179: sg.Info.RandomizeCharacterCount = true;
180: sg.Info.AllowSurrogatePairCharacters = false;
181: randomString = sg.Polyglot();
182: }
183: while (randomString.Contains("\0"));
184:
185: return randomString;
186: }
187:
188: /// <summary>
189: /// Generic method that enters the test data into the a specified textbox
190: /// Throws exception if the length of the testData argument exceeds the size
191: /// property of the textbox control, or textbox control in not enabled
192: /// </summary>
193: private static void SetTextboxText(AutomationElement inputTextbox, string testData)
194: {
195: try
196: {
197: ValuePattern input =
198: (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);
199: input.SetValue(testData);
200: }
201: catch (ElementNotEnabledException)
202: {
203: throw new ElementNotEnabledException("Textbox is not enabled.");
204: }
205: catch (InvalidOperationException)
206: {
207: throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");
208: }
209: }
210:
211: /// <summary>
212: /// Oracle that compares the randomly generated string against the variable
213: /// result in the message box text
214: /// </summary>
215: private static bool ValidateMessageBoxStringOracle(
216: string testData, int maxPollCount, out string actualResult)
217: {
218: bool result = false;
219: IntPtr msgBoxHandle = IntPtr.Zero;
220: AutomationElement messageBox;
221: int pollCount = 0;
222: actualResult = string.Empty;
223: try
224: {
225: do
226: {
227: msgBoxHandle = NativeMethod.GetForegroundWindow();
228: messageBox = AutomationElement.FromHandle(msgBoxHandle);
229: string className =
230: (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);
231: System.Threading.Thread.Sleep(100);
232: pollCount++;
233: }
234: while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);
235:
236: if (messageBox != null)
237: {
238: AutomationElement textbox = messageBox.FindFirst(
239: TreeScope.Descendants, new PropertyCondition(
240: AutomationElement.AutomationIdProperty, "65535"));
241: string messageBoxText =
242: (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);
243: actualResult =
244: messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);
245:
246: if (actualResult.Equals(testData))
247: {
248: result = true;
249: }
250:
251: AutomationElement messageboxOKButton = messageBox.FindFirst(
252: TreeScope.Descendants, new PropertyCondition(
253: AutomationElement.ClassNameProperty, "Button"));
254: PushButton(messageboxOKButton);
255: }
256: else
257: {
258: actualResult = "Messagebox not found.";
259: }
260:
261: return result;
262: }
263: catch (ArgumentOutOfRangeException)
264: {
265: throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");
266: }
267: catch (NullReferenceException)
268: {
269: throw new NullReferenceException("Messagebox unexpectedly became null");
270: }
271: }
272:
273: /// <summary>
274: /// Generic method to push a button control
275: /// Throws exception if button control is not enabled, or buttonName is null
276: /// </summary>
277: private static void PushButton(AutomationElement buttonName)
278: {
279: try
280: {
281: InvokePattern pushButton =
282: (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);
283: pushButton.Invoke();
284: }
285: catch (ElementNotEnabledException)
286: {
287: throw new ElementNotEnabledException("Button is not enabled.");
288: }
289: catch (InvalidOperationException)
290: {
291: throw new InvalidOperationException("Button not found.");
292: }
293: }
294:
295: /// <summary>
296: /// Generic method to close an AUT process
297: /// </summary>
298: private static void CloseAutProcess(Process autProcess)
299: {
300: try
301: {
302: autProcess.CloseMainWindow();
303: autProcess.WaitForExit(500);
304: if (!autProcess.HasExited)
305: {
306: autProcess.Kill();
307: }
308: }
309: catch (InvalidOperationException)
310: {
311: throw new InvalidOperationException("Specified process not found.");
312: }
313: }
314: #endregion HELPER METHODS
315: }
316: }
Automation isn’t bad; bad automation is bad!
As I write this I am flying at 38,000 feet or so across the Artic Circle. I am on my way to the Netherlands where I have been invited to keynote the TestNet conference. I am very excited to attend this conference and to meet new friends and connect with old ones. But, the guy beside me is hacking his head off (God, I hope he is not contagious). There is an elderly gentleman sitting in front of me constantly readjusting his chair up and back, up and back. And, I think the woman behind me has a bladder problem because about every 15 minutes she slams her tray up into my back, and jerks violently on the back of my chair as she lifts herself up. Fortunately on this flight there are no crying children within earshot (as of yet anyway). Since I don’t sleep well on airplanes without being in a drug induced stupor I thought I would hammer some thoughts on automated testing.
I am still surprised how often I hear people disparage automated testing. It’s almost like a rampant phobia similar to fear of water, fear of heights, or fear of the unknown. What leads people to proclaim disingenuous tripe that I sometimes read on blogs, or discussion forums? For example, the other day I read “automation will not find any new bugs.” Well, I guess that is the experience of some folks. But of course we could say that any inexperienced, untrained, or non-thinking person who uses a tool is likely to misuse the tool and get hurt, do damage, or come to a false conclusion that the tool doesn’t work. Automated tests are not bad; stupid automated tests are bad. If your test automation is not providing value don’t blame the tool…maybe a little introspection is in order.
All Automation Is Not Equal
The first problem in almost any discussion about automation is the lack of context. Automated tests are different and when done well can be effective at helping us identify some types of functional issues. An automated unit test suite can be a powerful tool for developers during refactoring. Of course some bugs escape unit testing because unit tests are not intended to be comprehensive. I have long been a proponent of functional testing at the API layer and this is where my team plays today. I see first hand the value of our automated tests in the early identification of potentially critical functional issues during our continuous integration process. While this level of testing is effective in helping prevent build breaks, it is not comprehensive and does not test the behavior of the application through the user interface. And then, comes GUI automation.
Why (Most) GUI Automation Sucks!
Although I am not a big fan of GUI automation I have done it, I teach it, and I know that it can be of value in many ways. Last week I was discussing reasons for intermittent failures in test automation with a colleague. Of course, a large number of intermittent failures occur in GUI automation for a variety of reasons (that’s a different post somewhere in the future). But, as I was reviewing the test code something dawned on me…the tests sucked!
It seems that when many people sit down to automate a GUI test (using just about any test automation paradigm) the automation is about a brain dead as a zombie. I suspect that when folks sit down to automate a GUI test the first thing they do is step through the UI. Then the tester automates each step until he/she arrives at some final state and validates the results with an equally brain dead oracle. In other words a lot of GUI automation is comprised of simplistic scripts that attempts to emulate human behavior patterns rather than being a well-designed test that capitalizes on the power of the computer as a tool.
Let’s look at a simple example; the “Hello” application. The application takes a first name string then displays a message box that reads “Hello [name]!”
Here is an example of a brain dead automated test that we might find if we asked someone to automate this “feature.”
1: namespace BraindDeadAutomatedGuiTest
2: {
3: class Program
4: {
5: [STAThread]
6: static void Main(string[] args)
7: {
8: // Launch the AUT
9: Process testAut = new Process();
10: testAut.StartInfo.FileName = "Hello.exe";
11: testAut.Start();
12:
13: // Hard coded sleep to give AUT time to instantiate
14: System.Threading.Thread.Sleep(1000);
15:
16: // Use key mnemonic to set focus to textbox
17: // Often this is skipped because people assume this textbox has focus
18: SendKeys.SendWait("%f");
19:
20: // Enter hard-coded test data
21: SendKeys.SendWait("Boob");
22:
23: // Tab to the OK key and press
24: SendKeys.SendWait("{Tab}");
25: SendKeys.SendWait("{ENTER}");
26:
27: // Another hard coded sleep to make sure messagebox appears
28: System.Threading.Thread.Sleep(1000);
29:
30: // Now comes the brain dead oracle
31: Clipboard.Clear();
32:
33: // Capture the messagebox text
34: SendKeys.SendWait("^(+c)");
35:
36: // Now get the text from the clipboard and see if it contains the
37: // hard-coded test data; often I see people compare strings
38: // verbatim which is a recipe for false positives & maintenance
39: string actualResult = Clipboard.GetText(TextDataFormat.Text);
40: if (actualResult.Contains("Boob"))
41: {
42: Console.WriteLine("Pass");
43: }
44: else
45: {
46: Console.WriteLine("Fail");
47: }
48:
49: // Clean up - kill the messagebox then close the AUT
50: SendKeys.SendWait("{ESC}");
51: testAut.CloseMainWindow();
52: }
53: }
54: }
Now, you might be saying that this is an absurd example, Really? I don’t think so. I see this sort of automation all the time in examples on the web and at conferences. For example, how often have you ever seen an example along the lines of:
- Launch IE (or some other browser)
- Navigate to “http://google.com”
- Enter “Ruby” in test box
- Press Search button
- Verify “Ruby” appears in results
Now, if you really expect this type of automation to add value then you should really consider a new line of work…perhaps assembly line work.
Well, we are about 1 1/2 hours out from Amsterdam and breakfast is about to be served, so I am going to wrap this up now. Yes, I learned a long time ago not to complain if I don’t at least offer a solution (whining is never appreciated). So, later this week, perhaps on the trip home on Wednesday, I will suggest one possible solution to automate even this simple app that could provide value by increasing test coverage and potentially exposing some errant behavior (bugs).
(Oh…and by the way, while writing this a little dirt merchant in pajamas and a pacifier sticking out of his pie-hole started doing laps through the cabin banging off the sides of the chairs…but at least he wasn’t crying!)
The Second 30 Days: Getting a handle on the project
Well, it has been 60 days on my new team. It has been a challenge going from an IC back to a management role. Time management is a huge challenge. I also am still ramping up on our feature area (I now know more about social networks then I ever thought I would). By day I find my time split between people management (team meetings, 1:1’s, etc.) and project management (triage meetings, status meetings, external partner sync-ups, etc.) At night I read through the API specs and brush up on my C++. While I come up to speed I am fortunate to lead a great team of highly competent SDETs. Someone once asked me if I were to ever leave the classroom and go back to manage a team in a product group if I would hire people as smart as I was. I replied, “Well, I am not really that smart, so I would want a team of people who are much smarter than me.” And, that’s exactly the team I have!
So, what does my team do? Essentially we are responsible for integrating the social networking functionality onto the Windows Phone devices. The Windows Phone is already the “real Facebook Phone” according to Wired magazine, and also helps you manage feeds from your LinkedIn contacts. The next release will also include Twitter integrated into the People’s hub (peeps) as well so you won’t need to install and switch between multiple apps to read or send updates from your contacts on different social networks. (The photo illustrates key aspects of the Peeps hub for those of you that don’t have a Windows Phone yet).
How we test in our corner of Microsoft
Last year I wrote about testing at the importance of functional testing at the API layer. So it is interesting that 1 year later I am leading a team that does just that. Our feature area includes the functional capabilities of the APIs that integrate social network features into the Windows Phone. 100% of our automated tests are designed to test functional scenarios at the API layer. Like many other teams at Microsoft we have daily builds of our feature branch. In addition to code reviews, our dev team runs their unit tests and a series of critical tests authored by our test team before checking in fixes and updates. Then an automated test suite is fired off in a lab on each nightly build to further validate the code changes did not destabilize key functional areas. Next lower priority automated tests are ran in the lab, as well as stress, performance and other test suites. This is continuous integration continuously.
While the team members are highly competent coders writing tests in C++ and some C#, they are more than developers in testing roles. Although a sister team is responsible for testing the customer experience, we also spend a time exploring our functional area via the user interface using an emulator or a device. This helps us develop customer empathy, but more importantly when our automated tests expose an issue we are better able to explain how that anomaly might impact our customer. We also spend a lot of time eating our own dog food and self hosting on test devices and some of us even flash our personal phones.
So, while I feel I am starting to come up to speed and beginning to contribute to the team I know I still have a lot more to learn. But, most importantly…I am having a blast!