Since I was young I was involved in team sports. I played little league baseball during my elementary years, in junior high I switched to football, and in high school I played lacrosse. Growing up on the right (east) coast of America a lot of us boys would also play pick-up pond hockey when the rivers and ponds froze over in the winter. Hockey has always been a passion of mine, and I am a big Bruins fan (perhaps because Maryland didn’t have a hockey team), and Bobby Orr was amazing!

There is an ice rink very close to my home, and my daughter started skating at 4 years of age. She has no ambition to be the next Nancy Kerrigan, but she loves to ice skate. She also loves to watch hockey (she is a die hard Penguins fan and her favorite player is Sid The Kid).

The past few weeks I have been involved in a hockey tournament in conjunction with my regular Monarchs schedule, work schedule, and getting the boat ready for the up-coming Leukemia Cup Regatta and summer sailing season. (I’ll use that as an excuse for my long absence from the blogosphere.) In the tournament we won some games, and we lost some games. Eventually our team won the silver medal yesterday afternoon in the final match of the series. One thing all this practice and play has reminded me of is the importance of the team. Whether it is playing hockey or shipping software a team that plays well together is required for success.

Playing well together is more than simply showing up on the ice. Each player has a position, each player is expected to communicate effectively to move the puck around and “read the play,” occasionally players will “block a shot” on goal, and when you are on the ice you give 150% (which is why the shifts (time on ice) are short). Essentially no player can do everything alone, and the weakest link means that others on the team must take up the slack (which eventually leads to mistakes and players getting “out of position”).

I play defenseman, and usually left defense.There are general skills for the game that all players must have, but there are skills and strategies for defense positions that are different than for forwards. And although the 3 forwards and the 2 defensive players must work together as a team to score (or prevent a goal) the responsibilities of each position are different, and each player must play their position.

I have read a lot of analogies between individual sports such as martial arts and the discipline of software testing. These analogies are sometimes good in that they help testers understand why they must constantly learn new things and practice in order to grow in our (or any) professional discipline. Similarly, I could make an analogy between the defensemen on a hockey team and software testers. But, while we might all aspire to be a “testing ninja” (or a Bobby Orr) in our career, we also need a team of other professionals each doing their part to ship a great software product.

When players in the software game are out of position or not playing their best the team is looking for a loss, or at best a very hard earned win (that usually burns players out). For example, program managers who don’t to provide useful customer scenarios or models of potential feature designs, or developers who fail to unit test their code or run unit tests that are only slightly better than “does it compile” put the burden on the test team (the defensive players) to hopefully “score the goal.”

While a defensive player may occasionally score a goal, their primary role (on offense) is to put the puck in front of the opposing net. As defensive players we must be able to read the play (understand our customer scenarios and analyze abstract models of a design and provide critical feedback). Communicate effectively to other defensemen and forwards to get the puck into position, or protect our goal (put the important information in front of those who make the critical business decisions).

At times defensemen need to help the forwards by driving the puck out of the zone, but without the forwards doing their part the defensemen would surely get exhausted very quickly and the outcome would not likely be favorable. Likewise, if the forwards don’t help defend their goal it is likely the opposing team will wear down the defense and score.

So, while we (as testers) sometimes like to think that the success or failure of a software project depends solely on our defensive skills to find bugs and drive in quality, the simple fact is that we need a team of professional to be successful and “quality” depends on the value each player brings to the table. If we don’t start a game with a good strategy and people playing their best in their assigned positions the likelihood of success is marginal at best. A strong defense is critical, but no one ever wins by playing defense.

"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest." – Thomas Reid’s Essays on the Intellectual Powers of Man, 1786

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of  3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

   1: // <copyright file="simpletestcase.cs" company="TestingMentor"> 

   2:  // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3:  // </copyright> 

   4: 

   5: namespace TestingMentor.Sample

   6: {

   7:   using System;

   8:   using System.IO;

   9:   using TestingMentor.Simulation;

  10: 

  11:   class TestCase

  12:   {

  13:     static void Main(string[] args)

  14:     {

  15:       int testCounter = 0;

  16:       // Read in an array of strings representing the test data. 

  17:       // Of course this would likely come from a static test data file

  18:       // on a server or copied to a folder on the local machine

  19:       string[] testData = new string[]

  20:       { "90,Z",

  21:         "24798,惞",

  22:         "0,null",

  23:         "65536,Error",

  24:         "-1,Error",

  25:         "1.5,",

  26:         "xyz,xyz"

  27:       };

  28: 

  29:       // Loop through each test data string

  30:       foreach (string test in testData)

  31:       {

  32:         testCounter++;

  33:         // This nested try/catch block catches invalid test data

  34:         // but allow additonal tests in the testData array

  35:         try

  36:         {

  37:           // Parse each string into the test data and expected result

  38:           string[] testElement = test.Split(',');

  39:           string expectedResult = testElement[1];

  40:           string actualResult = String.Empty;

  41: 

  42:           // Convert the string to a type int value

  43:           int value = int.Parse(testElement[0]);

  44:

  45:           // We need a way to handle int values 0 through 32 which are 

  46:           // control characters, this is an example of how to deal with 

  47:           // a int value of 0 which is a null character

  48:           if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))

  49:           {

  50:             expectedResult = '\0'.ToString();

  51:           }

  52: 

  53:           // This nested try/catch block tests catches exceptions thrown by 

  54:           // the method under test. If the method under test throws an 

  55:           // exception we certainly want to test for that case!

  56:           try

  57:           {

  58:             // Call the API method under test 

  59:             char result = Converter.ConvertValueToUnicodeChar(value);

  60:             actualResult = result.ToString();

  61:           }

  62: 

  63:           catch (ArgumentOutOfRangeException)

  64:           {

  65:             actualResult = "Error";

  66:           }

  67: 

  68:           catch (Exception)

  69:           {

  70:             // if this happens this is a failure because the documentation

  71:             // states that this method will only throw an 

  72:             // ArgumentOutOfRangeException.

  73:             actualResult = "Non-specific or unexpected error thrown";

  74:           }

  75: 

  76:           // Call a simple oracle and log results

  77:           if (String.Equals(actualResult, expectedResult))

  78:           {

  79:             // log pass

  80:             Console.WriteLine("{0} Pass", testCounter);

  81:           }

  82:           else

  83:           {

  84:             // log fail...of course log as much detail as possible

  85:             Console.WriteLine("{0} Fail", testCounter);

  86:           }

  87:         }

  88: 

  89:         catch (FormatException)

  90:         {

  91:           // log the test data for this test as incorrect, test is skipped

  92:           Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);

  93:         }

  94:       }

  95:     }

  96:   }

  97: }

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

I am not a big fan of static test data, so this month’s issue of Software Testing and Performance magazine published an article I wrote outlining one approach for generating random string data (although the basic concepts can be used for generating other types of random data).

Unfortunately, it appears that some of the numbers got a little screwed up and the printer did not superscript the exponents correctly so the numbers in the third paragraph are probably looking pretty strange. So, to clarify, the paragraph should read:

Using only the characters ‘A’ – ‘Z’ the total number of possible character combinations using for a filename with an 8-letter filename and a 3-letter extension is 26⁸ + 26³, or 208,827,099,728. If we were assigned to test long filenames on a Windows platform using only ASCII characters (see Table 1), the number of possibilities increases because there are 86 possible characters we can use in a valid filename or extension and a maximum filename length is 251 characters with a 3 character extension is 86²⁵¹ + 86³. Trust me, that is one big number.

(NOTE: There have been several assertions regarding the above formula for determining the number of tests, here is the explanation. Essentially, the Windows platform file system treats the base filename and the file extension as 2 separate components and there is no interaction or dependencies between these two components. (For example, we cannot save a filename as CON.txt, but we can save a filename as myFile.CON.) Since there is no dependencies between the base filename component and the extension component they are treated as 2 independent parameters which would mathematically result in 26⁸ + 26³, or 208,828,082,152 tests if we elected to test all possible combinations of the base filename component with a nominal valid extension, then test all possible extension component combinations with a nominal valid base filename. One could argue we could combine the 17576 unique 3-character extension combinations with various combinations of the 8-character base filename component to reduce the overall number of tests by 17576; however I choose not to use that approach and instead test each parameter independently. If we mistakenly assumed dependency or inter-relationship between the base filename and extension components of a filename on the Windows platform testing all combinations (or 26⁸ * 26³ (or simply 26¹¹) on a Windows OS would result in approximately 3,670,135,659,905,624 redundant tests (if we could do exhaustive testing). This is where in-depth knowledge of the ‘system’ really pays off.)

Of course, the filename length and extension length is variable. Also, 251 characters assumes a base filename component length from the root directory (it does not take into account the MAXPATH constant). So, the total number of combinations using only ASCII characters is much greater because the base filename component length with a ‘default’ 3-letter extension from the root directory is actually 86²⁵¹ + 86²⁵⁰ + 86²⁴⁹ + 86²⁴⁸ + 86²⁴⁷ … + 86¹. Then, of course vary the length of extensions, and the total number of combinations increases even further. But, all this is only to provide some scope the magnitude of the testing problem.

Also, the equivalence class table (Table 2) is simplified and does not include reserved device names. For example, Windows will/should prevent a user from saving a filename of LPT1, or COM6, or CON, etc. (The behavior for saving filenames with strings composed of reserved device names is different on Windows Xp and Windows Vista…Vista finally got it right!).

Unfortunately, I did not get a chance to read the edited copy before print, but I think the basic idea comes through and I hope you find value of using intelligent random test data in your testing and would be interested in hearing your feedback.

Last week I went to StarWest as a presenter and as a track chair to introduce speakers. Being a track chair is wonderful because you get to interface more closely with other speakers. Anyway…one of the speakers I introduced was Jon Bach. Jon is a good public speaker, and I was pleasantly surprised that he was doing a talk on the allpairs testing technique (also known as pairwise or combinatorial analysis). I wish Jon dedicated a little more time to the specifics of the technique during his talk and was generally more aware of available tools and information  for folks to investigate further, but I think he successfully raised the general awareness and interest in pariwise testing as an effective testing technique among the audience.

Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 2¹¹ or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.

The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.

But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create "mixed orthogonal arrays" where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L₂₅ (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).

The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.

One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.

I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.

Many experts have written articles and devoted chapters of books on the attributes of what constitutes a ‘good’ test case. Unfortunately, most books repeat a common (yet limited) perspective that a good test case has a high probability of finding bugs, and Kaner goes to the extreme by stating “A test that did not reveal a problem was a waste of time.” I pondered this for a while and thought, if I run a test to prove that a feature functions as expected is that really a waste of time? Isn’t it a good thing that testers actually spend some time executing tests that demonstrates the product does what the customer expects it to do? Or do we simply want to restrict the value of testing and reduce our testers to keyboard monkeys who bang away at the keyboard in search of bugs? (I believe limiting the perspective of the purpose of a test in testing literature has only perpetuated the faulty assumption that the purpose of testing is to simply find bugs; a point I previously dismissed). So, what is the purpose of a test?

Jorgensen states in his book, Software Testing: A Craftsman’s Approach that “a test case has two purposes: either to expose an error, or to demonstrate correct execution.” This explanation broadens the purpose of a test case to include both verifying the product meets the requirements, and revealing potential errors or defects. I tend to agree more with Jorgensen in this matter, but also think the purpose of a test case involves even more.

There are several objectives of any effective test whether manually executed or automated. The rationale for test case usually falls into one of seven categories. Each category provides value to the organization in different ways, but they all essentially function to reduce risk and qualify the testing effort. So, a good test provides some measurable value to the organization with the explicit purpose of:

• Verifying conformance to applicable standards and guidelines and customer requirements
• Validating expectations and customer needs
• Increasing control flow coverage
• Increasing logic flow coverage
• Increasing data flow coverage
• Simulating ‘real’ end user scenarios
• Exposing errors or defects