I.M. Testy

Last week I started discussing combinatorial testing. Justin Hunter added some great comments and an link to his blog with several great posts on this subject as well. He also asked to elaborate on the training I designed inside of Microsoft to teach our testers as well as what I discuss in my workshops at conferences. Rather than try to write one post that covered 8 hours of lab based content I will attempt to capture the essence of the training in a series of posts starting with getting started.

First, let me say that combinatorial testing isn’t a sliver bullet. Similar to other techniques and approaches used in software testing it is susceptible to Beizer’s Pesticide Paradox. In other words, it is effective in finding certain categories of bugs when applied smartly in the correct context, but it is not effective in finding all categories of issues.

Combinatorial testing is most useful when testing complex configuration scenarios and/or situations where there are multiple input parameters with numerous variables per parameter that have some change effect on a single output condition or state. For example, it you need to test your application on multiple versions of Windows, and different browser versions, and different protocols and connection speeds then this testing technique will help define a baseline set of test environment configurations. Or, if you are testing an API that has several parameters with multiple arguments values that can be passed to those API parameters then this testing technique will also help testers establish a baseline set of tests. Of course, this can also be applied to input controls on a graphical user interface (GUI) that affect a common output state or condition such as how changes to the settings on a font dialog change the properties of the glyphs in an edit control.

It’s all about modeling

The key to combinatorial testing lies in the testers ability to identify the interdependent input parameters that act on the single output condition being tested, and create an abstract description of the input variables or parameter behavior in a model file that is used by a tool to produce a baseline set of combinatorial tests. In other words the effectiveness of baseline set of combinatorial tests is primarily based on:

• A tester’s ability to identify the appropriate input parameters or configuration parameters
• A tester’s ability to adequately identify the appropriate variables for each input parameter or configuration settings
• A tester’s ability to describe the parameters and the variables or settings in a model of the feature being tested
• The limitations of the tool used to produce a baseline set of combinatorial tests

Models are abstract representations of ideas or real objects. Creating models is not easy and takes a lot of creativity and critical thinking. For example, let’s refer back to the font dialog that I use in my demos. Then let’s look at the two checkbox controls for Bold and Italic. The most simple way to model these two inputs is to simply list each input parameter and the 2 check states for each checkbox.

Bold: check, uncheck
Italic: check, uncheck

This is the example I used in the article because it is easy for a novice to understand quickly. However, another way to represent these 2 inputs is to conceptualize what they are doing individually, but what happens to the output state or condition (the properties of the glyphs in an edit control). In this example when either or both checkboxes are checked or unchecked the style of the displayed glyphs is changing. So, another way to model the style inputs is:

Style: bold, italic, bold/italic, regular

Both models of these input parameters accomplish the same thing. I personally prefer the second abstract model of styles because developing the rules for mutually exclusive variables (I will discuss that in a few weeks) is a bit easier, and it is also easier to modify the model if the types of styles that can be applied increases in the future.

Honestly, I don’t think I can teach someone to model something via a blog, or a book. In a class I can demonstrate different ways to model something and then provide feedback at people work through problem. But, modeling is a skill that takes practice. And one thing I have learned in my experience is that the less we know about the feature being modeled the greater the probability of less than adequate outcome in our testing.

George Box said, “All models are wrong; some models are useful.” In this situation it is clearly our ability as testers to create a model of configuration or input parameters that is useful and provides value to us; otherwise it is simply wrong. It is not necessarily this technique that is broken; it is more likely that our limited understanding of what is being tested or misapplication of the technique that causes us to produce a less than adequate model. In other words, if your model is wrong, the tool output is certainly going to be wrong!