Test Automation: Look Below the UI for More Effective and Robust UI Automated Test Case Designs
Originally Published Tuesday, April 14, 2009
Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.
Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.
For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!
To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.
In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below
1: namespace TestingMentor.PInvokeSample
2: {
3: using System;
4: using System.Runtime.InteropServices;
5:
6: /// <summary>
7: /// This class contains Native Win32 API functions that are marshalled
8: /// over for use in C#
9: /// </summary>
10: class NativeMethod
11: {
12: /// <summary>
13: /// Sets an item of information in the user override portion of the
14: /// current locale. This function does not set the system defaults.
15: /// </summary>
16: /// <param name="locale">the locale identifier of the locale with the
17: /// code page used </param>
18: /// <param name="localeType">Type of locale information to set.</param>
19: /// <param name="localeData">A null-terminated string containing the
20: /// locale information to set</param>
21: /// <returns>Returns true if successful; otherwise false</returns>
22: [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
23: public static extern bool SetLocaleInfo(
24: uint locale,
25: uint localeType,
26: string localeData);
27:
28: /// <summary>
29: /// Sets an item of information in the user override portion of the
30: /// current locale. This function does not set the system defaults.
31: /// </summary>
32: /// <param name="locale">the locale identifier of the locale with the
33: /// code page used </param>
34: /// <param name="localeType">Type of locale information to set.</param>
35: /// <param name="localeData">An integer value representing the locale
36: /// information to set</param>
37: /// <returns>Returns true if successful; otherwise false</returns>
38: [DllImport("kernel32.dll", SetLastError = true)]
39: public static extern bool SetLocaleInfo(
40: uint locale,
41: uint localeType,
42: int localeData);
43: }
44: }
The argument values that we can pass to these functions are enumerated in a separate class similar to the one below
1: namespace TestingMentor.NlsInfo
2: {
3: /// <summary>
4: /// Constant values for SetLocaleInfo API
5: /// </summary>
6: class NlsConstant
7: {
8: public enum Locale : uint
9: {
10: Invariant = 0x007F,
11: SystemDefault = 0x0800, // use system default for setlocaleinfo
12: UserDefault = 0x0400,
13: Neutral = 0x0000,
14: CustomDefault = 0X0C00, // Vista and later
15: CustomUiDefault = 0x1400, // Vista and later
16: CustomUnspecified = 0X1000 // Vista and later
17: };
18:
19: public enum LocaleType : uint
20: {
21: // VALUE LCDATA TYPES
22: CalendarType = 0x00001009, // type of calendar specifier
23: CurrencyDigits = 0x00000019, // local monetary fractional digits
24: CurrencySymbol = 0x0000001B, // position of positive currency symbol
25: FractionalDigits = 0x00000011, // number of fractional digits
26: NativeDigitSubstitution = 0x00001014, // native digit substitution
27: FirstDayOfWeek = 0x0000100C, // first day of week specifier
28: FirstWeekOfYear = 0x0000100D, // first week of year specifier
29: LeadingZeros = 0x00000012, // leading zeros for decimal
30: Measure = 0x0000000D, // 0 = metric, 1 = US
31: NegativeCurrency = 0x0000001C, // negative currency mode
32: NegativeNumber = 0x00001010, // negative number mode
33: PaperSize = 0x0000100A, // paper size
34: TimeFormat = 0x00000023, // time format specifier
35:
36: // STRING LCDATA TYPES
37: // Valid Unicode characters
38: AM = 0X00000028, // AM designator
39: PM = 0x00000029, // PM designator
40: CurrencySymbol = 0x00000014, // local monetary symbol
41: DecimalSeparator = 0x0000000E, // decimal separator
42: DigitGrouping = 0x00000010, // digit grouping
43: ListSeparator = 0x0000000C, // list item separator
44: LongDate = 0x00000020, // long date format string
45: MonetaryDecimalSeparator = 0x00000016, // monetary decimal separator
46: MonetaryGrouping = 0x00000018, // monetary grouping
47: MonetaryThousandSeparator = 0x00000017, // monetary thousand separator
48: NativeDigits = 0x00000013, // native ascii 0-9
49: NegativeSign = 0x00000051, // negative sign
50: PositiveSign = 0x00000050, // positive sign
51: ShortDate = 0x0000001D, // short date format string
52: ThousandSeparator = 0x0000000F, // thousand separator
53: TimeSeparator = 0x0000001E, // time separator
54: TimeFormat = 0x00001003, // time format string
55: YearMonthFormat = 0x00001006 // year month format string
56: };
57:
58: public enum LocaleData : int
59: {
60: // LOCALE_ICALENDARTYPE VALUES
61: Gregorian = 1, // Gregorian (localized)
62: GregorianUS = 2, // Gregorian(Always English)
63: GregorianMEFrench = 9, // Middle East French
64: GregorianArabic = 10,
65: GregorianXlitEnglish = 11, // transliterated English
66: GregorianXlitFrench = 12, // transliterated French
67: Japan = 3,
68: Taiwan = 4,
69: Korea = 5,
70: Hijri = 6,
71: Thai = 7,
72: Hebrew = 8,
73: Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later
74:
75: // LOCALE_ICURRENCY
76: PositiveCurrencyPrefixNoSeparation = 0,
77: PositiveCurrencySuffixNoSeparation = 1,
78: PositiveCurrencyPrefixSeparation = 2, // one character separation
79: PositiveCurrencySuffixSeparation = 3, // one character separation
80:
81: // LOCALE_IDIGITSUBSTITUTION
82: DigitSubstitutionContextBased = 0,
83: DigitSubstitutionNone = 1, // use this setting for full unicode support
84: DigitSubstitutionNative = 2, // uses digits based on national conventions
85: // according to LOCALE_SNATIVEDIGITS
86:
87: //LOCALE_IFIRSTDAYOFWEEK
88: Monday = 0, // LOCALE_SDAYNAME1
89: Tuesday = 1, // LOCALE_SDAYNAME2
90: Wednesday = 2, // LOCALE_SDAYNAME3
91: Thursday = 3, // LOCALE_SDAYNAME4
92: Friday = 4, // LOCALE_SDAYNAME5
93: Saturday = 5, // LOCALE_SDAYNAME6
94: Sunday = 6, // LOCALE_SDAYNAME7
95:
96: // LOCALE_IFRISTWEEKOFYEAR
97: FirstDay = 0, // Week containing 1/1 even if single day
98: FirstFullWeek = 1, // first full week following 1/1
99: FirstWeek = 2, // first week with at least 4 days after 1/1
100:
101: // LOCALE_ILZERO
102: NoLeadingZero = 0, // .975 119:
103: LeadingZero = 1, // 0.975
104:
105: // LOCALE_IMEASURE
106: Metric = 0,
107: US = 1,
108:
109: // LOCALE_INEGCURR
110: ParenthesisSymbolNumber = 0, // ($1.1)
111: NegativeSignSymbolNumber = 1, // -$1.1
112: SymbolNegativeSignNumber = 2, // $-1.1
113: SymbolNumberNegativeSign = 3, // $1.1-
114: ParenthesisNumberSymbol = 4, // (1.1$)
115: NegativeSignNumberSymbol = 5, // -1.1$
116: NumberNegativeSignSymbol = 6, // 1.1-$
117: NumberSymbolNegativeSign = 7, // 1.1$-
118: NegativeSignNumberSpaceSymbol = 8, // -1.1 $
119: NegativeSignSymbolSpaceNumber = 9, // -$ 1.1
120: NumberSpaceSymbolNegativeSign = 10, // 1.1 $-
121: SymbolSpaceNumberNegativeSign = 11, // $ 1.1-
122: SymbolSpaceNegativeSignNumber = 12, // $ -1.1
123: NumberNegativeSignSpaceSymbol = 13, // 1.1- $
124: ParenthesisSymbolSpaceNumber = 14, // ($ 1.1)
125: ParenthesisNumberSpaceSymbol = 15, // (1.1 $)
126:
127: // LOCALE_INEGNUMBER
128: Parenthesis = 0, // (1)
129: NegativeSignNumber = 1, // -1
130: NegativeSignSpaceNumber = 2, // - 1
131: NumberNegativeSign = 3, // 1-
132: NumberSpaceNegativeSign = 4, // 1 -
133:
134: // LOCALE_PAPERSIZE
135: USLetter = 1,
136: USLegal = 5,
137: A3 = 8,
138: A4 = 9,
139:
140: // LOCALE_ITIME
141: FormatAM_PM = 0,
142: Format24Hour = 1
143: };
144: }
145: }
You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-dd. The automated oracle verified the year, month and day values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.
I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals. Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.