Test Automation: Beyond Rudimentary Script-lets
When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.
Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.
The purpose of the test
Think about abstract big picture rather than a myopic singular purpose
In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.
The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.
As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.
Automation perspective
Think about programming the computer rather than trying to program human interactions
In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.
In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.
But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.
Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.
Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully
Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.
Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.
Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.
The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.
Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.
1: namespace HelloTest
2: {
3: using System;
4: using System.Diagnostics;
5: using System.IO;
6: using System.Runtime.InteropServices;
7: using System.Windows.Automation;
8: using TestingMentor.TestTool.Babel;
9:
10: internal class Constant
11: {
12: internal const string AutName = "hello.exe";
13: internal const string AutFormTitle = "HelloForm";
14: internal const string AutNameTextBox = "textboxFirstName";
15: internal const string AutOKButton = "buttonOK";
16: internal const string AutClearButton = "buttonClear";
17: internal const int FirstNameTextboxLength = 25;
18: internal const int MaxPollCount = 50;
19: }
20:
21: public class NativeMethod
22: {
23: [DllImport("user32.dll")]
24: public static extern IntPtr GetForegroundWindow();
25: }
26:
27: class SampleHelloTest
28: {
29: static void Main(string[] args)
30: {
31: try
32: {
33: Stopwatch timer = new Stopwatch();
34: timer.Start();
35:
36: int testIterationCount = 25;
37:
38: // Declare a new process and automation elements
39: Process autProcess = new Process();
40: AutomationElement desktopPath = AutomationElement.RootElement;
41: AutomationElement autForm = null;
42: AutomationElement nameTextbox;
43: AutomationElement clearButton;
44: AutomationElement okButton;
45:
46: // Launch the AUT
47: autProcess = LaunchApplication(
48: Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
49: Constant.AutName),
50: desktopPath,
51: Constant.MaxPollCount,
52: ref autForm);
53:
54: // Get the UI automation element object on main form
55: GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);
56:
57: // Verify the textbox control is 'visible'
58: if (nameTextbox != null &&
59: (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))
60: {
61: // iterate through a specified number of test data variants
62: for (int testCount = 0; testCount < testIterationCount; testCount++)
63: {
64:
65: int seedValue;
66: string testData = GetRandomTestDataVariant(out seedValue);
67:
68: // Put test string into textbox
69: SetTextboxText(nameTextbox, testData);
70:
71: // Emulate clicking the OK button to instantiate the message box
72: PushButton(okButton);
73:
74: // Oracle to compare actual vs expected result and log results/important information
75: string actualResult = string.Empty;
76: if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))
77: {
78: // Log Test Results as appropriate
79: Console.WriteLine(testData);
80: }
81: else
82: {
83: Console.BackgroundColor = ConsoleColor.Red;
84: Console.WriteLine(
85: "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",
86: testData, actualResult, seedValue);
87: }
88:
89: // Emulate clicking the clear button
90: PushButton(clearButton);
91: }
92: }
93: else
94: {
95: Console.WriteLine("TEST FAILURE: Textbox control not found");
96: }
97:
98: // Clean up the test environment
99: CloseAutProcess(autProcess);
100:
101: // Log total elapsed time and any additional important info
102: timer.Stop();
103: Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);
104: }
105: catch (Exception e)
106: {
107: Console.WriteLine(e.ToString());
108: }
109: }
110:
111: #region HELPER METHODS
112: /// <summary>
113: /// Launches AUT process; or throws exception if process fails to launch
114: /// </summary>
115: internal static Process LaunchApplication(
116: string autPathAndFilename,
117: AutomationElement desktop,
118: int maxPollCount,
119: ref AutomationElement myAutForm)
120: {
121: Process myProc = new Process();
122: myProc.StartInfo.FileName = autPathAndFilename;
123: if (myProc.Start())
124: {
125: // Find the MyFontForm window
126: int pollCount = 0;
127: do
128: {
129: myAutForm = desktop.FindFirst(
130: TreeScope.Descendants, new PropertyCondition(
131: AutomationElement.AutomationIdProperty, Constant.AutFormTitle));
132: pollCount++;
133: System.Threading.Thread.Sleep(100);
134: }
135: while (myAutForm == null && pollCount < maxPollCount);
136:
137: if (myAutForm == null)
138: {
139: throw new ArgumentNullException("Failed to find AUT form");
140: }
141: }
142: return myProc;
143: }
144:
145: /// <summary>
146: /// Get UI automation element objects on main form by property name
147: /// </summary>
148: private static void GetUIAutomationElements(
149: AutomationElement autFormName,
150: out AutomationElement nameTextbox,
151: out AutomationElement okButton,
152: out AutomationElement clearButton)
153: {
154: nameTextbox = autFormName.FindFirst(
155: TreeScope.Descendants, new PropertyCondition(
156: AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));
157: okButton = autFormName.FindFirst(
158: TreeScope.Descendants, new PropertyCondition(
159: AutomationElement.AutomationIdProperty, Constant.AutOKButton));
160: clearButton = autFormName.FindFirst(
161: TreeScope.Descendants, new PropertyCondition(
162: AutomationElement.AutomationIdProperty, Constant.AutClearButton));
163: }
164:
165: /// <summary>
166: /// Generate a pseudo-random Unicode string using the Babel.dll automation library
167: /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm
168: /// </summary>
169: private static string GetRandomTestDataVariant(out int seedValue)
170: {
171: string randomString = string.Empty;
172: do
173: {
174: Random prng = new Random();
175: seedValue = prng.Next();
176: StringGenerator sg = new StringGenerator();
177: sg.Info.Seed = seedValue;
178: sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;
179: sg.Info.RandomizeCharacterCount = true;
180: sg.Info.AllowSurrogatePairCharacters = false;
181: randomString = sg.Polyglot();
182: }
183: while (randomString.Contains("\0"));
184:
185: return randomString;
186: }
187:
188: /// <summary>
189: /// Generic method that enters the test data into the a specified textbox
190: /// Throws exception if the length of the testData argument exceeds the size
191: /// property of the textbox control, or textbox control in not enabled
192: /// </summary>
193: private static void SetTextboxText(AutomationElement inputTextbox, string testData)
194: {
195: try
196: {
197: ValuePattern input =
198: (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);
199: input.SetValue(testData);
200: }
201: catch (ElementNotEnabledException)
202: {
203: throw new ElementNotEnabledException("Textbox is not enabled.");
204: }
205: catch (InvalidOperationException)
206: {
207: throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");
208: }
209: }
210:
211: /// <summary>
212: /// Oracle that compares the randomly generated string against the variable
213: /// result in the message box text
214: /// </summary>
215: private static bool ValidateMessageBoxStringOracle(
216: string testData, int maxPollCount, out string actualResult)
217: {
218: bool result = false;
219: IntPtr msgBoxHandle = IntPtr.Zero;
220: AutomationElement messageBox;
221: int pollCount = 0;
222: actualResult = string.Empty;
223: try
224: {
225: do
226: {
227: msgBoxHandle = NativeMethod.GetForegroundWindow();
228: messageBox = AutomationElement.FromHandle(msgBoxHandle);
229: string className =
230: (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);
231: System.Threading.Thread.Sleep(100);
232: pollCount++;
233: }
234: while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);
235:
236: if (messageBox != null)
237: {
238: AutomationElement textbox = messageBox.FindFirst(
239: TreeScope.Descendants, new PropertyCondition(
240: AutomationElement.AutomationIdProperty, "65535"));
241: string messageBoxText =
242: (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);
243: actualResult =
244: messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);
245:
246: if (actualResult.Equals(testData))
247: {
248: result = true;
249: }
250:
251: AutomationElement messageboxOKButton = messageBox.FindFirst(
252: TreeScope.Descendants, new PropertyCondition(
253: AutomationElement.ClassNameProperty, "Button"));
254: PushButton(messageboxOKButton);
255: }
256: else
257: {
258: actualResult = "Messagebox not found.";
259: }
260:
261: return result;
262: }
263: catch (ArgumentOutOfRangeException)
264: {
265: throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");
266: }
267: catch (NullReferenceException)
268: {
269: throw new NullReferenceException("Messagebox unexpectedly became null");
270: }
271: }
272:
273: /// <summary>
274: /// Generic method to push a button control
275: /// Throws exception if button control is not enabled, or buttonName is null
276: /// </summary>
277: private static void PushButton(AutomationElement buttonName)
278: {
279: try
280: {
281: InvokePattern pushButton =
282: (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);
283: pushButton.Invoke();
284: }
285: catch (ElementNotEnabledException)
286: {
287: throw new ElementNotEnabledException("Button is not enabled.");
288: }
289: catch (InvalidOperationException)
290: {
291: throw new InvalidOperationException("Button not found.");
292: }
293: }
294:
295: /// <summary>
296: /// Generic method to close an AUT process
297: /// </summary>
298: private static void CloseAutProcess(Process autProcess)
299: {
300: try
301: {
302: autProcess.CloseMainWindow();
303: autProcess.WaitForExit(500);
304: if (!autProcess.HasExited)
305: {
306: autProcess.Kill();
307: }
308: }
309: catch (InvalidOperationException)
310: {
311: throw new InvalidOperationException("Specified process not found.");
312: }
313: }
314: #endregion HELPER METHODS
315: }
316: }
Pingback: OpenQuality.ru | Качество программного обеспечения
Hi BJ, (reposting to correct some small typos).
In my new automation framework (Sunscrit) I enable random data generation based on regexps with which one can define what counts as a valid data input. Such regexps are basically “data types” to which objects can belong. This way one can define a UI field to get a different value each time the input operation is performed. Of course it’s possible to request valid or invalid values for negative tests.
BR, Meir
http://www.sunscrit.com
Hi,
Great post Bj!
I liked the concept of random test data generation. In some specific contexts it is more preferred. For example, applications like facebook may not even allow the same comment to be posted more than 5 times and in this case even the data driven approach fails becoz the test suite might need to be run multiple times for several releases. The approach of random test data generation is great as it generates unique data each time and resolves this issue.
This example clearly demonstrates how a modularized robust automation test looks like. It supports re-usability. Thanks again for sharing it.
Regards,
Aruna
http://www.technologyandleadership.com
“The intersection of Technology and Leadership”
A bit of topic, but aren’t you using the codedUI test framework in Visual Studio. Makss the code much easier also default functionality for testdata and ui element selecting.