I.M. Testy

When I wake up in the morning I like to go out onto my back deck, watch critters scramble back into the wooded area behind my house, feed the koi, and sit down by the pond and enjoy a cup of coffee. I use this time to gather my thoughts about things I want to accomplish that day and run through a mental list of meetings for that day. Sometimes I just think about cutting the grass or weeding the gardens. This morning morning I awoke to a rather cold 36° F with frost on the ground. Who would have thought frost in mid-May? This morning as I drank my coffee I thought it was a good thing that I procrastinated and haven’t moved my plants from the greenhouse into the gardens yet. But, despite the morning chill it is turning out to be a beautiful day here in the PNW.

Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.

The purpose of the test
Think about abstract big picture rather than a myopic singular purpose

In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.

The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.

As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.

Automation perspective
Think about programming the computer rather than trying to program human interactions

In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.

In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.

But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.

Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.

Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully

Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.

Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.

Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.

The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.

Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.

   1:  namespace HelloTest

   2:  {

   3:    using System;

   4:    using System.Diagnostics;

   5:    using System.IO;

   6:    using System.Runtime.InteropServices;

   7:    using System.Windows.Automation;

   8:    using TestingMentor.TestTool.Babel;

   9:  

  10:    internal class Constant

  11:    {

  12:      internal const string AutName = "hello.exe";

  13:      internal const string AutFormTitle = "HelloForm";

  14:      internal const string AutNameTextBox = "textboxFirstName";

  15:      internal const string AutOKButton = "buttonOK";

  16:      internal const string AutClearButton = "buttonClear";

  17:      internal const int FirstNameTextboxLength = 25;

  18:      internal const int MaxPollCount = 50;

  19:    }

  20:  

  21:    public class NativeMethod

  22:    {

  23:      [DllImport("user32.dll")]

  24:      public static extern IntPtr GetForegroundWindow();

  25:    }

  26:  

  27:    class SampleHelloTest

  28:    {

  29:      static void Main(string[] args)

  30:      {

  31:        try

  32:        {

  33:          Stopwatch timer = new Stopwatch();

  34:          timer.Start();

  35:  

  36:          int testIterationCount = 25;

  37:  

  38:          // Declare a new process and automation elements

  39:          Process autProcess = new Process();

  40:          AutomationElement desktopPath = AutomationElement.RootElement;

  41:          AutomationElement autForm = null;

  42:          AutomationElement nameTextbox;

  43:          AutomationElement clearButton;

  44:          AutomationElement okButton;

  45:  

  46:          // Launch the AUT

  47:          autProcess = LaunchApplication(

  48:            Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),

  49:            Constant.AutName),

  50:            desktopPath,

  51:            Constant.MaxPollCount,

  52:            ref autForm);

  53:  

  54:          // Get the UI automation element object on main form

  55:          GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);

  56:  

  57:          // Verify the textbox control is 'visible'

  58:          if (nameTextbox != null &&

  59:            (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))

  60:          {

  61:            // iterate through a specified number of test data variants

  62:            for (int testCount = 0; testCount < testIterationCount; testCount++)

  63:            {

  64:  

  65:              int seedValue;

  66:              string testData = GetRandomTestDataVariant(out seedValue);

  67:  

  68:              // Put test string into textbox

  69:              SetTextboxText(nameTextbox, testData);

  70:  

  71:              // Emulate clicking the OK button to instantiate the message box

  72:              PushButton(okButton);

  73:  

  74:              // Oracle to compare actual vs expected result and log results/important information

  75:              string actualResult = string.Empty;

  76:              if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))

  77:              {

  78:                // Log Test Results as appropriate 

  79:                Console.WriteLine(testData);

  80:              }

  81:              else

  82:              {

  83:                Console.BackgroundColor = ConsoleColor.Red;

  84:                Console.WriteLine(

  85:                  "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",

  86:                  testData, actualResult, seedValue);

  87:              }

  88:  

  89:              // Emulate clicking the clear button

  90:              PushButton(clearButton);

  91:            }

  92:          }

  93:          else

  94:          {

  95:            Console.WriteLine("TEST FAILURE: Textbox control not found");

  96:          }

  97:  

  98:          // Clean up the test environment 

  99:          CloseAutProcess(autProcess);

 100:  

 101:          // Log total elapsed time and any additional important info

 102:          timer.Stop();

 103:          Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);

 104:        }

 105:        catch (Exception e)

 106:        {

 107:          Console.WriteLine(e.ToString());

 108:        }

 109:      }

 110:  

 111:      #region HELPER METHODS

 112:      /// <summary>

 113:      /// Launches AUT process; or throws exception if process fails to launch

 114:      /// </summary>

 115:      internal static Process LaunchApplication(

 116:        string autPathAndFilename,

 117:        AutomationElement desktop,

 118:        int maxPollCount,

 119:        ref AutomationElement myAutForm)

 120:      {

 121:        Process myProc = new Process();

 122:        myProc.StartInfo.FileName = autPathAndFilename;

 123:        if (myProc.Start())

 124:        {

 125:          // Find the MyFontForm window

 126:          int pollCount = 0;

 127:          do

 128:          {

 129:            myAutForm = desktop.FindFirst(

 130:              TreeScope.Descendants, new PropertyCondition(

 131:                AutomationElement.AutomationIdProperty, Constant.AutFormTitle));

 132:            pollCount++;

 133:            System.Threading.Thread.Sleep(100);

 134:          }

 135:          while (myAutForm == null && pollCount < maxPollCount);

 136:  

 137:          if (myAutForm == null)

 138:          {

 139:            throw new ArgumentNullException("Failed to find AUT form");

 140:          }

 141:        }

 142:        return myProc;

 143:      }

 144:  

 145:      /// <summary>

 146:      /// Get UI automation element objects on main form by property name

 147:      /// </summary>

 148:      private static void GetUIAutomationElements(

 149:        AutomationElement autFormName,

 150:        out AutomationElement nameTextbox,

 151:        out AutomationElement okButton,

 152:        out AutomationElement clearButton)

 153:      {

 154:        nameTextbox = autFormName.FindFirst(

 155:          TreeScope.Descendants, new PropertyCondition(

 156:            AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));

 157:        okButton = autFormName.FindFirst(

 158:          TreeScope.Descendants, new PropertyCondition(

 159:            AutomationElement.AutomationIdProperty, Constant.AutOKButton));

 160:        clearButton = autFormName.FindFirst(

 161:          TreeScope.Descendants, new PropertyCondition(

 162:            AutomationElement.AutomationIdProperty, Constant.AutClearButton));

 163:      }

 164:  

 165:      /// <summary>

 166:      /// Generate a pseudo-random Unicode string using the Babel.dll automation library

 167:      /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm

 168:      /// </summary>

 169:      private static string GetRandomTestDataVariant(out int seedValue)

 170:      {

 171:        string randomString = string.Empty;

 172:        do

 173:        {

 174:          Random prng = new Random();

 175:          seedValue = prng.Next();

 176:          StringGenerator sg = new StringGenerator();

 177:          sg.Info.Seed = seedValue;

 178:          sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;

 179:          sg.Info.RandomizeCharacterCount = true;

 180:          sg.Info.AllowSurrogatePairCharacters = false;

 181:          randomString = sg.Polyglot();

 182:        }

 183:        while (randomString.Contains("\0"));

 184:  

 185:        return randomString;

 186:      }

 187:  

 188:      /// <summary>

 189:      /// Generic method that enters the test data into the a specified textbox

 190:      /// Throws exception if the length of the testData argument exceeds the size

 191:      /// property of the textbox control, or textbox control in not enabled

 192:      /// </summary>

 193:      private static void SetTextboxText(AutomationElement inputTextbox, string testData)

 194:      {

 195:        try

 196:        {

 197:          ValuePattern input =

 198:            (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);

 199:          input.SetValue(testData);

 200:        }

 201:        catch (ElementNotEnabledException)

 202:        {

 203:          throw new ElementNotEnabledException("Textbox is not enabled.");

 204:        }

 205:        catch (InvalidOperationException)

 206:        {

 207:          throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");

 208:        }

 209:      }

 210:  

 211:      /// <summary>

 212:      /// Oracle that compares the randomly generated string against the variable

 213:      /// result in the message box text

 214:      /// </summary>

 215:      private static bool ValidateMessageBoxStringOracle(

 216:        string testData, int maxPollCount, out string actualResult)

 217:      {

 218:          bool result = false;

 219:          IntPtr msgBoxHandle = IntPtr.Zero;

 220:          AutomationElement messageBox;

 221:          int pollCount = 0;

 222:          actualResult = string.Empty;

 223:          try

 224:          {

 225:            do

 226:            {

 227:              msgBoxHandle = NativeMethod.GetForegroundWindow();

 228:              messageBox = AutomationElement.FromHandle(msgBoxHandle);

 229:              string className =

 230:                (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);

 231:              System.Threading.Thread.Sleep(100);

 232:              pollCount++;

 233:            }

 234:            while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);

 235:  

 236:            if (messageBox != null)

 237:            {

 238:              AutomationElement textbox = messageBox.FindFirst(

 239:                TreeScope.Descendants, new PropertyCondition(

 240:                  AutomationElement.AutomationIdProperty, "65535"));

 241:              string messageBoxText =

 242:                (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);

 243:              actualResult =

 244:                messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);

 245:  

 246:              if (actualResult.Equals(testData))

 247:              {

 248:                result = true;

 249:              }

 250:  

 251:              AutomationElement messageboxOKButton = messageBox.FindFirst(

 252:                TreeScope.Descendants, new PropertyCondition(

 253:                  AutomationElement.ClassNameProperty, "Button"));

 254:              PushButton(messageboxOKButton);

 255:            }

 256:            else

 257:            {

 258:              actualResult = "Messagebox not found.";

 259:            }

 260:  

 261:            return result;

 262:          }

 263:          catch (ArgumentOutOfRangeException)

 264:          {

 265:            throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");

 266:          }

 267:          catch (NullReferenceException)

 268:          {

 269:            throw new NullReferenceException("Messagebox unexpectedly became null");

 270:          }

 271:      }

 272:  

 273:      /// <summary>

 274:      /// Generic method to push a button control

 275:      /// Throws exception if button control is not enabled, or buttonName is null

 276:      /// </summary>

 277:      private static void PushButton(AutomationElement buttonName)

 278:      {

 279:        try

 280:        {

 281:          InvokePattern pushButton =

 282:            (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);

 283:          pushButton.Invoke();

 284:        }

 285:        catch (ElementNotEnabledException)

 286:        {

 287:          throw new ElementNotEnabledException("Button is not enabled.");

 288:        }

 289:        catch (InvalidOperationException)

 290:        {

 291:          throw new InvalidOperationException("Button not found.");

 292:        }

 293:      }

 294:  

 295:      /// <summary>

 296:      /// Generic method to close an AUT process

 297:      /// </summary>

 298:      private static void CloseAutProcess(Process autProcess)

 299:      {

 300:        try

 301:        {

 302:          autProcess.CloseMainWindow();

 303:          autProcess.WaitForExit(500);

 304:          if (!autProcess.HasExited)

 305:          {

 306:            autProcess.Kill();

 307:          }

 308:        }

 309:        catch (InvalidOperationException)

 310:        {

 311:          throw new InvalidOperationException("Specified process not found.");

 312:        }

 313:      }

 314:      #endregion HELPER METHODS

 315:    }

 316:  }

For the past 2 weeks my students in my automation course at University of Washington have been tasked with designing automated test cases through the GUI for a shareware program. In my opinion, GUI automation is the least effective approach for testing the functional or business logic of a program (assuming a well designed architecture where the business logic code is separate from the form (GUI) and the event hander (GUI object behavior) code). However, if a tester doesn’t have access to the underlying APIs used in the application under test (AUT) and is given just a compiled application (‘GUI application’) to test then functional testing through the GUI may be the only alternative.

GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.

However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’)  But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’

There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?

First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.

So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”

The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.

Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:

and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.

But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”

“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”

I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.

Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.

This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of  new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)

Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”

Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of  3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

   1: // <copyright file="simpletestcase.cs" company="TestingMentor"> 

   2:  // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3:  // </copyright> 

   4: 

   5: namespace TestingMentor.Sample

   6: {

   7:   using System;

   8:   using System.IO;

   9:   using TestingMentor.Simulation;

  10: 

  11:   class TestCase

  12:   {

  13:     static void Main(string[] args)

  14:     {

  15:       int testCounter = 0;

  16:       // Read in an array of strings representing the test data. 

  17:       // Of course this would likely come from a static test data file

  18:       // on a server or copied to a folder on the local machine

  19:       string[] testData = new string[]

  20:       { "90,Z",

  21:         "24798,惞",

  22:         "0,null",

  23:         "65536,Error",

  24:         "-1,Error",

  25:         "1.5,",

  26:         "xyz,xyz"

  27:       };

  28: 

  29:       // Loop through each test data string

  30:       foreach (string test in testData)

  31:       {

  32:         testCounter++;

  33:         // This nested try/catch block catches invalid test data

  34:         // but allow additonal tests in the testData array

  35:         try

  36:         {

  37:           // Parse each string into the test data and expected result

  38:           string[] testElement = test.Split(',');

  39:           string expectedResult = testElement[1];

  40:           string actualResult = String.Empty;

  41: 

  42:           // Convert the string to a type int value

  43:           int value = int.Parse(testElement[0]);

  44:

  45:           // We need a way to handle int values 0 through 32 which are 

  46:           // control characters, this is an example of how to deal with 

  47:           // a int value of 0 which is a null character

  48:           if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))

  49:           {

  50:             expectedResult = '\0'.ToString();

  51:           }

  52: 

  53:           // This nested try/catch block tests catches exceptions thrown by 

  54:           // the method under test. If the method under test throws an 

  55:           // exception we certainly want to test for that case!

  56:           try

  57:           {

  58:             // Call the API method under test 

  59:             char result = Converter.ConvertValueToUnicodeChar(value);

  60:             actualResult = result.ToString();

  61:           }

  62: 

  63:           catch (ArgumentOutOfRangeException)

  64:           {

  65:             actualResult = "Error";

  66:           }

  67: 

  68:           catch (Exception)

  69:           {

  70:             // if this happens this is a failure because the documentation

  71:             // states that this method will only throw an 

  72:             // ArgumentOutOfRangeException.

  73:             actualResult = "Non-specific or unexpected error thrown";

  74:           }

  75: 

  76:           // Call a simple oracle and log results

  77:           if (String.Equals(actualResult, expectedResult))

  78:           {

  79:             // log pass

  80:             Console.WriteLine("{0} Pass", testCounter);

  81:           }

  82:           else

  83:           {

  84:             // log fail...of course log as much detail as possible

  85:             Console.WriteLine("{0} Fail", testCounter);

  86:           }

  87:         }

  88: 

  89:         catch (FormatException)

  90:         {

  91:           // log the test data for this test as incorrect, test is skipped

  92:           Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);

  93:         }

  94:       }

  95:     }

  96:   }

  97: }

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

Did you ever notice that when you ask someone to test something the first thing they do is to start ‘testing?’

I often see this in my classes and I ask the person, “what is the purpose of your test?” Typically the response is, “I’m testing this,” or “I’m trying to find a bug.”

Unfortunately this seems to indicate there is no or very little pre-thought that goes into the act of software testing. To some people, testing appears to be little more than simply pounding away at the keyboard and trying whatever flies into our subconscious mind as we interact with the software and declare a bug when we stumble upon unexpected behavior or see something we might disagree with.

This is why I found it especially interesting in my own research, and the case studies by Juha Itkonen that testers who were trained in formal software testing techniques or patterns there was no significant difference in terms of defect rates or coverage between pre-defined test cases and an exploratory testing approach. This is not to say that one approach to testing is preferred over the other. It is not an either or proposition as I explained in my post on the pesticide paradox, and there are certainly more than 2 approaches to software testing. Testing requires multiple approaches to most effectively aid us in collecting and presenting the appropriate information to the decision makers.

But, I am often puzzled that it seems we can easily think of negative or destructive tests once we have the product in hand, yet when we are designing a set of tests from the requirements the tests simply test the requirements and little else. I wonder why it is that we can think of ‘tests’ while executing other tests, but we can’t think of those same tests before hand. Is there some limitation in our psyche that prevents us from analyzing a problem until we are actually faced with the problem (software in hand)?

I don’t think so, but I suspect there is a mental hurdle in that we sometimes feel more productive when we are interacting with software as opposed to sitting back and analyzing the problem more prior to executing well-designed test cases. (More tests doesn’t equal better testing!)

The bottom line is that if we are given a set of requirements and can only design tests that only test the requirements, then we are probably not thinking critically about how to design test cases.