I.M. Testy

Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of 3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

// <copyright file="simpletestcase.cs" company="TestingMentor"> 
 // Copyright © 2009 by Bj Rollison. All rights reserved. 
 // </copyright> 
 
namespace TestingMentor.Sample
{
  using System;
  using System.IO;
  using TestingMentor.Simulation;
 
  class TestCase
  {
    static void Main(string[] args)
    {
      int testCounter = 0;
      // Read in an array of strings representing the test data. 
      // Of course this would likely come from a static test data file
      // on a server or copied to a folder on the local machine
      string[] testData = new string[]
      { "90,Z",
        "24798,惞",
        "0,null",
        "65536,Error",
        "-1,Error",
        "1.5,",
        "xyz,xyz"
      };
 
      // Loop through each test data string
      foreach (string test in testData)
      {
        testCounter++;
        // This nested try/catch block catches invalid test data
        // but allow additonal tests in the testData array
        try
        {
          // Parse each string into the test data and expected result
          string[] testElement = test.Split(',');
          string expectedResult = testElement[1];
          string actualResult = String.Empty;
 
          // Convert the string to a type int value
          int value = int.Parse(testElement[0]);

          // We need a way to handle int values 0 through 32 which are 
          // control characters, this is an example of how to deal with 
          // a int value of 0 which is a null character
          if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))
          {
            expectedResult = '\0'.ToString();
          }
 
          // This nested try/catch block tests catches exceptions thrown by 
          // the method under test. If the method under test throws an 
          // exception we certainly want to test for that case!
          try
          {
            // Call the API method under test 
            char result = Converter.ConvertValueToUnicodeChar(value);
            actualResult = result.ToString();
          }
 
          catch (ArgumentOutOfRangeException)
          {
            actualResult = "Error";
          }
 
          catch (Exception)
          {
            // if this happens this is a failure because the documentation
            // states that this method will only throw an 
            // ArgumentOutOfRangeException.
            actualResult = "Non-specific or unexpected error thrown";
          }
 
          // Call a simple oracle and log results
          if (String.Equals(actualResult, expectedResult))
          {
            // log pass
            Console.WriteLine("{0} Pass", testCounter);
          }
          else
          {
            // log fail...of course log as much detail as possible
            Console.WriteLine("{0} Fail", testCounter);
          }
        }
 
        catch (FormatException)
        {
          // log the test data for this test as incorrect, test is skipped
          Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);
        }
      }
    }
  }
}

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

API Testing – Thinking Differently About the Problem

One Trackback/Pingback

About This Blog

RSS Links

Recent Comments

Categories

Archives