Archive for the ‘Random Test Data Generation’ tag
Test Automation: Beyond Rudimentary Script-lets
When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.
Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.
The purpose of the test
Think about abstract big picture rather than a myopic singular purpose
In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.
The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.
As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.
Automation perspective
Think about programming the computer rather than trying to program human interactions
In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.
In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.
But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.
Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.
Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully
Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.
Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.
Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.
The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.
Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.
1: namespace HelloTest
2: {
3: using System;
4: using System.Diagnostics;
5: using System.IO;
6: using System.Runtime.InteropServices;
7: using System.Windows.Automation;
8: using TestingMentor.TestTool.Babel;
9:
10: internal class Constant
11: {
12: internal const string AutName = "hello.exe";
13: internal const string AutFormTitle = "HelloForm";
14: internal const string AutNameTextBox = "textboxFirstName";
15: internal const string AutOKButton = "buttonOK";
16: internal const string AutClearButton = "buttonClear";
17: internal const int FirstNameTextboxLength = 25;
18: internal const int MaxPollCount = 50;
19: }
20:
21: public class NativeMethod
22: {
23: [DllImport("user32.dll")]
24: public static extern IntPtr GetForegroundWindow();
25: }
26:
27: class SampleHelloTest
28: {
29: static void Main(string[] args)
30: {
31: try
32: {
33: Stopwatch timer = new Stopwatch();
34: timer.Start();
35:
36: int testIterationCount = 25;
37:
38: // Declare a new process and automation elements
39: Process autProcess = new Process();
40: AutomationElement desktopPath = AutomationElement.RootElement;
41: AutomationElement autForm = null;
42: AutomationElement nameTextbox;
43: AutomationElement clearButton;
44: AutomationElement okButton;
45:
46: // Launch the AUT
47: autProcess = LaunchApplication(
48: Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
49: Constant.AutName),
50: desktopPath,
51: Constant.MaxPollCount,
52: ref autForm);
53:
54: // Get the UI automation element object on main form
55: GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);
56:
57: // Verify the textbox control is 'visible'
58: if (nameTextbox != null &&
59: (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))
60: {
61: // iterate through a specified number of test data variants
62: for (int testCount = 0; testCount < testIterationCount; testCount++)
63: {
64:
65: int seedValue;
66: string testData = GetRandomTestDataVariant(out seedValue);
67:
68: // Put test string into textbox
69: SetTextboxText(nameTextbox, testData);
70:
71: // Emulate clicking the OK button to instantiate the message box
72: PushButton(okButton);
73:
74: // Oracle to compare actual vs expected result and log results/important information
75: string actualResult = string.Empty;
76: if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))
77: {
78: // Log Test Results as appropriate
79: Console.WriteLine(testData);
80: }
81: else
82: {
83: Console.BackgroundColor = ConsoleColor.Red;
84: Console.WriteLine(
85: "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",
86: testData, actualResult, seedValue);
87: }
88:
89: // Emulate clicking the clear button
90: PushButton(clearButton);
91: }
92: }
93: else
94: {
95: Console.WriteLine("TEST FAILURE: Textbox control not found");
96: }
97:
98: // Clean up the test environment
99: CloseAutProcess(autProcess);
100:
101: // Log total elapsed time and any additional important info
102: timer.Stop();
103: Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);
104: }
105: catch (Exception e)
106: {
107: Console.WriteLine(e.ToString());
108: }
109: }
110:
111: #region HELPER METHODS
112: /// <summary>
113: /// Launches AUT process; or throws exception if process fails to launch
114: /// </summary>
115: internal static Process LaunchApplication(
116: string autPathAndFilename,
117: AutomationElement desktop,
118: int maxPollCount,
119: ref AutomationElement myAutForm)
120: {
121: Process myProc = new Process();
122: myProc.StartInfo.FileName = autPathAndFilename;
123: if (myProc.Start())
124: {
125: // Find the MyFontForm window
126: int pollCount = 0;
127: do
128: {
129: myAutForm = desktop.FindFirst(
130: TreeScope.Descendants, new PropertyCondition(
131: AutomationElement.AutomationIdProperty, Constant.AutFormTitle));
132: pollCount++;
133: System.Threading.Thread.Sleep(100);
134: }
135: while (myAutForm == null && pollCount < maxPollCount);
136:
137: if (myAutForm == null)
138: {
139: throw new ArgumentNullException("Failed to find AUT form");
140: }
141: }
142: return myProc;
143: }
144:
145: /// <summary>
146: /// Get UI automation element objects on main form by property name
147: /// </summary>
148: private static void GetUIAutomationElements(
149: AutomationElement autFormName,
150: out AutomationElement nameTextbox,
151: out AutomationElement okButton,
152: out AutomationElement clearButton)
153: {
154: nameTextbox = autFormName.FindFirst(
155: TreeScope.Descendants, new PropertyCondition(
156: AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));
157: okButton = autFormName.FindFirst(
158: TreeScope.Descendants, new PropertyCondition(
159: AutomationElement.AutomationIdProperty, Constant.AutOKButton));
160: clearButton = autFormName.FindFirst(
161: TreeScope.Descendants, new PropertyCondition(
162: AutomationElement.AutomationIdProperty, Constant.AutClearButton));
163: }
164:
165: /// <summary>
166: /// Generate a pseudo-random Unicode string using the Babel.dll automation library
167: /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm
168: /// </summary>
169: private static string GetRandomTestDataVariant(out int seedValue)
170: {
171: string randomString = string.Empty;
172: do
173: {
174: Random prng = new Random();
175: seedValue = prng.Next();
176: StringGenerator sg = new StringGenerator();
177: sg.Info.Seed = seedValue;
178: sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;
179: sg.Info.RandomizeCharacterCount = true;
180: sg.Info.AllowSurrogatePairCharacters = false;
181: randomString = sg.Polyglot();
182: }
183: while (randomString.Contains("\0"));
184:
185: return randomString;
186: }
187:
188: /// <summary>
189: /// Generic method that enters the test data into the a specified textbox
190: /// Throws exception if the length of the testData argument exceeds the size
191: /// property of the textbox control, or textbox control in not enabled
192: /// </summary>
193: private static void SetTextboxText(AutomationElement inputTextbox, string testData)
194: {
195: try
196: {
197: ValuePattern input =
198: (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);
199: input.SetValue(testData);
200: }
201: catch (ElementNotEnabledException)
202: {
203: throw new ElementNotEnabledException("Textbox is not enabled.");
204: }
205: catch (InvalidOperationException)
206: {
207: throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");
208: }
209: }
210:
211: /// <summary>
212: /// Oracle that compares the randomly generated string against the variable
213: /// result in the message box text
214: /// </summary>
215: private static bool ValidateMessageBoxStringOracle(
216: string testData, int maxPollCount, out string actualResult)
217: {
218: bool result = false;
219: IntPtr msgBoxHandle = IntPtr.Zero;
220: AutomationElement messageBox;
221: int pollCount = 0;
222: actualResult = string.Empty;
223: try
224: {
225: do
226: {
227: msgBoxHandle = NativeMethod.GetForegroundWindow();
228: messageBox = AutomationElement.FromHandle(msgBoxHandle);
229: string className =
230: (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);
231: System.Threading.Thread.Sleep(100);
232: pollCount++;
233: }
234: while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);
235:
236: if (messageBox != null)
237: {
238: AutomationElement textbox = messageBox.FindFirst(
239: TreeScope.Descendants, new PropertyCondition(
240: AutomationElement.AutomationIdProperty, "65535"));
241: string messageBoxText =
242: (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);
243: actualResult =
244: messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);
245:
246: if (actualResult.Equals(testData))
247: {
248: result = true;
249: }
250:
251: AutomationElement messageboxOKButton = messageBox.FindFirst(
252: TreeScope.Descendants, new PropertyCondition(
253: AutomationElement.ClassNameProperty, "Button"));
254: PushButton(messageboxOKButton);
255: }
256: else
257: {
258: actualResult = "Messagebox not found.";
259: }
260:
261: return result;
262: }
263: catch (ArgumentOutOfRangeException)
264: {
265: throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");
266: }
267: catch (NullReferenceException)
268: {
269: throw new NullReferenceException("Messagebox unexpectedly became null");
270: }
271: }
272:
273: /// <summary>
274: /// Generic method to push a button control
275: /// Throws exception if button control is not enabled, or buttonName is null
276: /// </summary>
277: private static void PushButton(AutomationElement buttonName)
278: {
279: try
280: {
281: InvokePattern pushButton =
282: (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);
283: pushButton.Invoke();
284: }
285: catch (ElementNotEnabledException)
286: {
287: throw new ElementNotEnabledException("Button is not enabled.");
288: }
289: catch (InvalidOperationException)
290: {
291: throw new InvalidOperationException("Button not found.");
292: }
293: }
294:
295: /// <summary>
296: /// Generic method to close an AUT process
297: /// </summary>
298: private static void CloseAutProcess(Process autProcess)
299: {
300: try
301: {
302: autProcess.CloseMainWindow();
303: autProcess.WaitForExit(500);
304: if (!autProcess.HasExited)
305: {
306: autProcess.Kill();
307: }
308: }
309: catch (InvalidOperationException)
310: {
311: throw new InvalidOperationException("Specified process not found.");
312: }
313: }
314: #endregion HELPER METHODS
315: }
316: }
Generating Random Dates
Last week was mostly a blur for me. Three hockey games (1 game with my regular team, 1 substituting on another team, and 1 in the over 40 league). By Thursday I was pretty wiped out, and with all the holiday stuff going on I completely blew off updating my blog.
Since my daughter was about 6 months old her and I have always had a daddy & daughter day at least once per week where I focus 100% on her and we do what she wants to do. About every 2 weeks she wants to go ice skating, and she completely surprised me by saying that she wanted to go to a free hockey clinic for kids for our our daddy & daughter day on Sunday. Castle Ice hosts a free hockey clinic for kids once per month. Although my daughter understands all the rules and enjoys watching games (she has become quite a Sid the Kid fan), I know that the odds of her ever taking up the sport are somewhere between 0 and .00001. But, she said she had a great time pushing the puck around on the ice and doing some of the drills with the other kids, and I was sure proud of her for giving it a try.
Ok…enough about my excuses for not posting last week, and time to move on from combinatorial testing and get back to something else I am interested in…random test data generation.
Every once in awhile a discussion comes up about how to generate random dates. There are several examples on the web to generate random dates. But many of them hard code locale specific formats such as month/day/year or day/month/year. Some allow you to specify a separator character such as a dash (-) or a forward slash (/) or other character. Many of these generators are limited to out-dated formatting options compared to the formats that are available (see MSDN Day, Month, Year and Era format picture).
In an earlier post I discussed how to we can programmatically change the date format to incorporate globalization testing or international sufficiency testing in our automated tests. I also created an automation library called GlobalTester to help testers change other settings to push international sufficiency testing upstream, and I talk about that in a previous post as well.
So, instead of generating dates based on some hard-coded format (that may or may not be the same format used by the current user locale settings), or using some hard-coded separator character (that may or may not be the character for the current user locale) I decided to create a method that focused on generating random dates and allow the operating system to decide the proper formatting.
An easy way to generate a single random date string within a specified range that is formatted to the current user settings for the date format is illustrated below.
1: public static string RandomDate(DateTime startDate, DateTime endDate)
2: {
3: try
4: {
5: Random prng = new Random();
6: int range = ((TimeSpan)(endDate - startDate)).Days;
7:
8: // Returns a string that follows the current short date formatting options
9: return startDate.AddDays(prng.Next(range)).ToShortDateString();
10:
11: // Returns a string that follows the current long date formatting options
12: // return startDate.AddDays(prng.Next(range)).ToLongDateString();
13: }
14: catch (ArgumentOutOfRangeException)
15: {
16: throw new ArgumentOutOfRangeException(
17: "The end date must be later than the start date");
18: }
19: }
Now, all we have to do is define our start date range and our end date range and call the method. To generate a random date string using the long date format settings for the current user simply change the RandomDate method above to return the value in line 12.
1: static void Main(string[] args)
2: {
3: DateTime startDate = new DateTime(
4: DateTime.Now.Year - 100, DateTime.Now.Month, DateTime.Now.Day);
5: DateTime endDate = new DateTime(
6: DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day);
7:
8: Console.WriteLine(RandomDate(endDate, startDate));
9: }
Because we can use a constructor to pass int values for the year, month and day this approach allows us to set a date range in the past, the future, or we can specifically define a range of dates.
The above method works well if we only need a single random date, but if we need several random dates this method will not produce reasonably random dates. To produce multiple random dates at one time we should use a slightly different approach. A better approach to generate multiple random dates at one time is to use an IEnumerable interface as illustrated.
1: public static IEnumerable<DateTime> RandomDate(
2: DateTime startDate,
3: DateTime endDate)
4: {
5: Random prng = new Random();
6: int range = ((TimeSpan)(endDate - startDate)).Days;
7: while (true)
8: {
9: yield return startDate.AddDays(prng.Next(range));
10: }
11: }
1: static void Main(string[] args)
2: {
3: DateTime startDate = new DateTime(
4: DateTime.Now.Year - 100, DateTime.Now.Month, DateTime.Now.Day);
5: DateTime endDate = new DateTime(
6: DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day);
7:
8: int i = 0;
9: foreach (DateTime date in RandomDay(startDate, endDate))
10: {
11: Console.WriteLine(date.ToLongDateString());
12: Console.WriteLine(date.ToShortDateString());
13: if (++i == 10)
14: {
15: break;
16: }
17: }
18: }
Compared to all the random date generators out there, this is about the simplest implementation available. We don’t have to use a generator tool or library, we can simply include these methods in our libraries or code. If you know of an easier solution, please let us know.
Some additional information can be found at
Random Test Data – Credit Card Numbers
Things are winding down for the year. The Christmas lights are up on the house, my gardens are tilled and mulched for next spring, people are disappearing from the office like there is a plague, it hasn’t snowed in a while which means the mountains are mostly ice (I dislike skiing on ice), the next GSHL ice hockey league doesn’t start for awhile and pick up games are few and far between (I suck at hockey but it is fun). So, what to do? Oh…I forgot Christmas shopping. I hate Christmas shopping! So, I have spent the past few idle nights refactoring the automation libraries for some of my test data generation tools after my daughter goes to bed.
One of the most popular random test data generators that I have developed so far has been a tool called CCMaker to generate random valid and invalid credit card numbers. (Sometimes I wonder why that is, but I don’t dwell on it for too long and I haven’t been interrogated by the FBI lately.) Testing forms that require a credit card has always been risky business because you certainly don’t want to use your own card. Often times developers will include a check on web forms or client apps to do a high level verification of a credit card number before sending all the data across the wire to be validated. This early or high level verification prevents flooding the pipe with bad data. So, one test we can do prior to testing the end-to-end scenario is to test to see if and how the developer is validating credit cards numbers prior to submission.
As far as data goes, generating credit card numbers are fairly simple. There is a bank ID number (BIN), there is a number of digits between 12 and 19 depending on the card type, and there is a checksum. So, if we know the valid BINs for each issuing bank, the valid number of digits for each card type, and how to calculate the checksum we can generate valid credit card numbers. (Of course this is a bit oversimplified because many credit and debit card companies are issued multiple BINs and use varying number lengths.)
Testing for invalid credit card numbers should include using numbers that look close to being correct in some way but are slightly altered. For example for the 3 defined equivalent partitions (BIN, length, checksum) there are seven possible invalid combinations (23 – 1) we could test.
- Valid BIN, invalid length and valid checksum
- Valid BIN, valid length, and invalid checksum
- Valid BIN, invalid length, and invalid checksum
- Invalid BIN, valid length and valid checksum
- Invalid BIN, invalid length, and valid checksum
- Invalid BIN, valid length, and invalid checksum
- Invalid BIN, checksum and length
This doesn’t mean I run 7 tests and call it good because there are numerous invalid lengths and invalid BINs for the different card types. A common mistake when using an equivalent partition testing approach is to simply plug in values for each combination listed above and call it good. The problem is that there are several hundred BINs and 8 different valid lengths. For example, for just the Discover card there are 829 valid BIN numbers, and for the Maestro cards there are 56 combinations of BINs and card lengths ranging from 12 to 19 numbers in length. This doesn’t include the permutations of the other numbers that compose the entire card number.
The question every tester must ask him or herself every day when designing tests is how many tests do I need to have any reasonable sense of confidence that risk is minimal and the perception of quality is high. Of course, there is no single right answer here and not magic formula, but since we can’t possibly execute every possible positive or negative test we should at least understand that ultimately testing is sampling.
For example, one strategy for positive testing might be to test every valid BIN for every valid card length for any given credit card. For example for American Express I would want to test at least one number with a BIN of 34 and a card length of 15 that satisfies the checksum requirement, and at least one number with a BIN of 37 and a card length of 15 that also satisfies the checksum requirement. For a card type of Visa I would need a minimum of 2 tests in which the BIN is 4, the checksum requirement is satisfied, and one has a card length of 13 numbers and the other has a card length of 16 numbers.
That probably sounds like quite a bit of testing, and tests which most likely would not produce an error (unless of course the BIN is miss identified (e.g. instead of checking for a BIN of 5020 the BIN is incorrectly assigned as 5002), or if a valid BIN is not recognized as valid because it is omitted from a list or enumeration of valid BINs for that credit card). Certainly testing of this magnitude would be expensive if done manually. But when automated using a random test data generator and a data-driven automation approach to set the random generator properties comprehensive testing becomes a much more reasonable proposition and can significantly increase overall confidence.
This is where my CCMaker 3.1 test data generator can help by randomly generating both valid and invalid credit card numbers. The updated CCMaker test automation library has just been posted to my web site with documentation and examples. If you have any questions, or find any issues with the new library please let me know.
Babel – A ‘New’ Random Unicode String Generator Test Tool
Originally Published Thursday, September 20, 2007
For some time I have wanted to add surrogate pair character support to a tool I developed called GString, and this week I managed to find some time to do that work and more! As I developed the methods for surrogate pair support I rewrote (refactored in developer parlance) some of the previous methods to reduce complexity. And wouldn’t you know it…the simple act of refactoring exposed some otherwise hard to find defects (and one pretty obvious one). I discovered these defects because I had to approach the problem space from a different perspective, and that perspective (working primarily with int types instead of char types) exposed the problems.
So, I decided to retire the GString code base, and I ported what I could into a new tool named Babel (and this is my shameless plug for that tool.) I know it is not ‘customer friendly’ when someone goes and renames a tool, especially when it comes with a library for test automation because now the ‘customer’ has to change their references in order to use the functionality in the new DLL. However, the name Babel seems more fitting in the purpose of this tool to generate random characters across the Unicode spectrum of language scripts; and besides Java also has a class called GString and I didn’t want to cause any confusion. 
The obvious bug fixed in Babel is a problem that occurred when generating character in the ASCII only range. For some bizarre reason I neglected to exclude Japanese half-width katakana characters (and for an even more bizarre reason I failed to find it; which is a really good reason why unit testing only goes so far and we really need a second set of eyes for sufficient testing). One not so obvious defects included exclusion of a range of code points between U+1A20 and U+1AFF instead of U+1B80 and U+1CFF. This was a classic boundary bug! But unless we did a formal code review it is unlikely this one would have never been found.) The other not so obvious defect that has been fixed involved the the programs inability to exclude some valid Unicode code points that have not been assigned a character if the user selected to exclude unassigned code points (again a similar problem to that described above.)
The good news is these are now fixed, and the new Babel tool also includes support for Unicode surrogate pair characters in the range of U+10000 through U+10FFFF as an option. Also, I included a feature to save the output to a text file rather than having to copy and paste. The installation package include a desktop tool, a DLL for test automation, and the user’s guide and can be found at Testing Mentor.
If you encounter any problem using the tool, or if you have any feedback please let me know. Enjoy!
Random Test Data Generation
Originally Published Wednesday, May 30, 20
I am not a big fan of static test data, so this month’s issue of Software Testing and Performance magazine published an article I wrote outlining one approach for generating random string data (although the basic concepts can be used for generating other types of random data).
Unfortunately, it appears that some of the numbers got a little screwed up and the printer did not superscript the exponents correctly so the numbers in the third paragraph are probably looking pretty strange. So, to clarify, the paragraph should read:
Using only the characters ‘A’ – ‘Z’ the total number of possible character combinations using for a filename with an 8-letter filename and a 3-letter extension is 268 + 263, or 208,827,099,728. If we were assigned to test long filenames on a Windows platform using only ASCII characters (see Table 1), the number of possibilities increases because there are 86 possible characters we can use in a valid filename or extension and a maximum filename length is 251 characters with a 3 character extension is 86251 + 863. Trust me, that is one big number.
(NOTE: There have been several assertions regarding the above formula for determining the number of tests, here is the explanation. Essentially, the Windows platform file system treats the base filename and the file extension as 2 separate components and there is no interaction or dependencies between these two components. (For example, we cannot save a filename as CON.txt, but we can save a filename as myFile.CON.) Since there is no dependencies between the base filename component and the extension component they are treated as 2 independent parameters which would mathematically result in 268 + 263, or 208,828,082,152 tests if we elected to test all possible combinations of the base filename component with a nominal valid extension, then test all possible extension component combinations with a nominal valid base filename. One could argue we could combine the 17576 unique 3-character extension combinations with various combinations of the 8-character base filename component to reduce the overall number of tests by 17576; however I choose not to use that approach and instead test each parameter independently. If we mistakenly assumed dependency or inter-relationship between the base filename and extension components of a filename on the Windows platform testing all combinations (or 268 * 263 (or simply 2611) on a Windows OS would result in approximately 3,670,135,659,905,624 redundant tests (if we could do exhaustive testing). This is where in-depth knowledge of the ‘system’ really pays off.)
Of course, the filename length and extension length is variable. Also, 251 characters assumes a base filename component length from the root directory (it does not take into account the MAXPATH constant). So, the total number of combinations using only ASCII characters is much greater because the base filename component length with a ‘default’ 3-letter extension from the root directory is actually 86251 + 86250 + 86249 + 86248 + 86247 … + 861. Then, of course vary the length of extensions, and the total number of combinations increases even further. But, all this is only to provide some scope the magnitude of the testing problem.
Also, the equivalence class table (Table 2) is simplified and does not include reserved device names. For example, Windows will/should prevent a user from saving a filename of LPT1, or COM6, or CON, etc. (The behavior for saving filenames with strings composed of reserved device names is different on Windows Xp and Windows Vista…Vista finally got it right!).
Unfortunately, I did not get a chance to read the edited copy before print, but I think the basic idea comes through and I hope you find value of using intelligent random test data in your testing and would be interested in hearing your feedback.
More on Generating Strings with Random Unicode Characters
Originally Published Sunday, December 24, 2006
Well, for those of you living outside the Pacific Northwest you are probably unaware of the recent wind storm with winds gusting to 60+ miles per hour that left more than 1 million people on the eastern side of the state without power. The damage was pretty extensive, and since I live in a fairly remote area I was without power for more than 7 days and without the Internet for almost 9 days. I do have a generator, but it hadn’t been used in almost 4 years. Sure, I started it every 6 months for about 15 minutes each time, but after the first full day of operation the generator started doing wierd things. So, during the past week I have become pretty good at fixing generators (mine and my neighbors), tracing electrical systems, troubleshooting furnace problems, splitting a lot of firewood, cutting up fallen trees, and repairing fences.
After the sun set (which is quite early) I had little else to do (other than making sure nobody stole my generator), so between stoking the fire I started developing a DLL for Unicode string generation in automated tests based on the GString utility. While reviewing the data tables I created for the GString utility with the Unicode Handbook I noticed some holes (OK…defects). Some of the boundaries for code ranges that are not assigned to any Unicode script group were incorrect. (That will teach me to use a web page with the listing put together by a web developer rather than using the Unicode handbook.) But, I also found a problem that prevented unassigned code points from being generated even if the Only use assigned code points check box was unchecked.So, the (hopefully final) update to GString is complete, including the GString.DLL! So, along with the massive overhaul of the Unicode data tables, the new GString package available from my personal website also includes a new DLL for anyone needing to generate strings of random Unicode characters in test automation. The GString zip file also includes detailed documentation on the utility and the dll usage. Let me know if you have any questions about the tool or using random string generation in your testing.
Well, now back to (mostly) normal life.