GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.

However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’)  But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’

There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?

First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.

So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”

The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.

Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:

and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.

But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”

“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”

I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.

Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.

This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of  new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)

Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of  3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

   1: // <copyright file="simpletestcase.cs" company="TestingMentor"> 

   2:  // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3:  // </copyright> 

   4: 

   5: namespace TestingMentor.Sample

   6: {

   7:   using System;

   8:   using System.IO;

   9:   using TestingMentor.Simulation;

  10: 

  11:   class TestCase

  12:   {

  13:     static void Main(string[] args)

  14:     {

  15:       int testCounter = 0;

  16:       // Read in an array of strings representing the test data. 

  17:       // Of course this would likely come from a static test data file

  18:       // on a server or copied to a folder on the local machine

  19:       string[] testData = new string[]

  20:       { "90,Z",

  21:         "24798,惞",

  22:         "0,null",

  23:         "65536,Error",

  24:         "-1,Error",

  25:         "1.5,",

  26:         "xyz,xyz"

  27:       };

  28: 

  29:       // Loop through each test data string

  30:       foreach (string test in testData)

  31:       {

  32:         testCounter++;

  33:         // This nested try/catch block catches invalid test data

  34:         // but allow additonal tests in the testData array

  35:         try

  36:         {

  37:           // Parse each string into the test data and expected result

  38:           string[] testElement = test.Split(',');

  39:           string expectedResult = testElement[1];

  40:           string actualResult = String.Empty;

  41: 

  42:           // Convert the string to a type int value

  43:           int value = int.Parse(testElement[0]);

  44:

  45:           // We need a way to handle int values 0 through 32 which are 

  46:           // control characters, this is an example of how to deal with 

  47:           // a int value of 0 which is a null character

  48:           if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))

  49:           {

  50:             expectedResult = '\0'.ToString();

  51:           }

  52: 

  53:           // This nested try/catch block tests catches exceptions thrown by 

  54:           // the method under test. If the method under test throws an 

  55:           // exception we certainly want to test for that case!

  56:           try

  57:           {

  58:             // Call the API method under test 

  59:             char result = Converter.ConvertValueToUnicodeChar(value);

  60:             actualResult = result.ToString();

  61:           }

  62: 

  63:           catch (ArgumentOutOfRangeException)

  64:           {

  65:             actualResult = "Error";

  66:           }

  67: 

  68:           catch (Exception)

  69:           {

  70:             // if this happens this is a failure because the documentation

  71:             // states that this method will only throw an 

  72:             // ArgumentOutOfRangeException.

  73:             actualResult = "Non-specific or unexpected error thrown";

  74:           }

  75: 

  76:           // Call a simple oracle and log results

  77:           if (String.Equals(actualResult, expectedResult))

  78:           {

  79:             // log pass

  80:             Console.WriteLine("{0} Pass", testCounter);

  81:           }

  82:           else

  83:           {

  84:             // log fail...of course log as much detail as possible

  85:             Console.WriteLine("{0} Fail", testCounter);

  86:           }

  87:         }

  88: 

  89:         catch (FormatException)

  90:         {

  91:           // log the test data for this test as incorrect, test is skipped

  92:           Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);

  93:         }

  94:       }

  95:     }

  96:   }

  97: }

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

I often see this in my classes and I ask the person, “what is the purpose of your test?” Typically the response is, “I’m testing this,” or “I’m trying to find a bug.”

Unfortunately this seems to indicate there is no or very little pre-thought that goes into the act of software testing. To some people, testing appears to be little more than simply pounding away at the keyboard and trying whatever flies into our subconscious mind as we interact with the software and declare a bug when we stumble upon unexpected behavior or see something we might disagree with.

This is why I found it especially interesting in my own research, and the case studies by Juha Itkonen that testers who were trained in formal software testing techniques or patterns there was no significant difference in terms of defect rates or coverage between pre-defined test cases and an exploratory testing approach. This is not to say that one approach to testing is preferred over the other. It is not an either or proposition as I explained in my post on the pesticide paradox, and there are certainly more than 2 approaches to software testing. Testing requires multiple approaches to most effectively aid us in collecting and presenting the appropriate information to the decision makers.

But, I am often puzzled that it seems we can easily think of negative or destructive tests once we have the product in hand, yet when we are designing a set of tests from the requirements the tests simply test the requirements and little else. I wonder why it is that we can think of ‘tests’ while executing other tests, but we can’t think of those same tests before hand. Is there some limitation in our psyche that prevents us from analyzing a problem until we are actually faced with the problem (software in hand)?

I don’t think so, but I suspect there is a mental hurdle in that we sometimes feel more productive when we are interacting with software as opposed to sitting back and analyzing the problem more prior to executing well-designed test cases. (More tests doesn’t equal better testing!)

The bottom line is that if we are given a set of requirements and can only design tests that only test the requirements, then we are probably not thinking critically about how to design test cases.