Fall is now upon us here in Seattle. I really like fall; the bright vibrant colors of the changing leaves, the crisp morning and evening air, leaves starting to blanket the lawn, and harvesting the crops from my garden. Of course along with the harvest comes the work of canning the veggies mixing up new batches of jam and conserves. But, it is fun work and it fills the house with delicious aromas that remind me of my boyhood and helping my mom in the kitchen canning the bounty of crops from our garden. Now I try to pass the tradition onto my daughter and we have fun trying different combinations of berries when we make our jam. My favorite is still just plain huckleberry, but our strawberry/blackberry mix is also darn good…probably because we pick the berries right from our backyard. But, not all combinations of berries work well in a recipe. The flavor of some berries overpower or mask the flavor of other berries. Similarly, in software testing not all combinations of input variables that directly impact a single common output work well together and result in a bug.
Pair-wise testing more specifically combinatorial testing, is a functional test technique intended to help testers more effectively expose issues caused by the interaction of 2 or more input variables that directly affect a common output state or condition. In simple situations a tester can often pick out various combinations of inputs based on likely customers settings, historical failure indicators (combinations of inputs that have been problematic in the past) and intuition. However, as the number of input parameters and the number of variables that can be applied to those interdependent input parameters increase in more complex features the potential number of combinations is overwhelming. Of course we still want to focus on common customer inputs, failure indicators and intuition. But, does guessing at other various combinations of inputs really provide us with sufficient confidence in our test coverage? Or, would a more systematic approach to testing combinations of input variables be more effective and more efficient?
There is a lot of empirical research both in academia and industry to suggest that the answer to this last question is…yes! Over the past few years there has been quite a lot written on the topic of pair-wise testing, and I and a few other people have presented at conferences on the topic. In fact, I recently published an article in Better Software magazine on the topic, and also gave a presentation at the recent VistaCon 2010 software testing conference in Vietnam. I have also posted my slides and the demo files (including the source code for a sample data-driven automated test) from the article and the conference presentation on my website.
In the coming weeks I intend to provide more information and tips to help testers think about how to model input parameters and variables for use in a tool to generate a subset of combinatorial tests and overcome some of the limitations of misuses of this best practice. Until then, if you have specific questions or comments please let me know.