Combinatorial Testing
Fall is now upon us here in Seattle. I really like fall; the bright vibrant colors of the changing leaves, the crisp morning and evening air, leaves starting to blanket the lawn, and harvesting the crops from my garden. Of course along with the harvest comes the work of canning the veggies mixing up new batches of jam and conserves. But, it is fun work and it fills the house with delicious aromas that remind me of my boyhood and helping my mom in the kitchen canning the bounty of crops from our garden. Now I try to pass the tradition onto my daughter and we have fun trying different combinations of berries when we make our jam. My favorite is still just plain huckleberry, but our strawberry/blackberry mix is also darn good…probably because we pick the berries right from our backyard. But, not all combinations of berries work well in a recipe. The flavor of some berries overpower or mask the flavor of other berries. Similarly, in software testing not all combinations of input variables that directly impact a single common output work well together and result in a bug.
Pair-wise testing more specifically combinatorial testing, is a functional test technique intended to help testers more effectively expose issues caused by the interaction of 2 or more input variables that directly affect a common output state or condition. In simple situations a tester can often pick out various combinations of inputs based on likely customers settings, historical failure indicators (combinations of inputs that have been problematic in the past) and intuition. However, as the number of input parameters and the number of variables that can be applied to those interdependent input parameters increase in more complex features the potential number of combinations is overwhelming. Of course we still want to focus on common customer inputs, failure indicators and intuition. But, does guessing at other various combinations of inputs really provide us with sufficient confidence in our test coverage? Or, would a more systematic approach to testing combinations of input variables be more effective and more efficient?
There is a lot of empirical research both in academia and industry to suggest that the answer to this last question is…yes! Over the past few years there has been quite a lot written on the topic of pair-wise testing, and I and a few other people have presented at conferences on the topic. In fact, I recently published an article in Better Software magazine on the topic, and also gave a presentation at the recent VistaCon 2010 software testing conference in Vietnam. I have also posted my slides and the demo files (including the source code for a sample data-driven automated test) from the article and the conference presentation on my website.
In the coming weeks I intend to provide more information and tips to help testers think about how to model input parameters and variables for use in a tool to generate a subset of combinatorial tests and overcome some of the limitations of misuses of this best practice. Until then, if you have specific questions or comments please let me know.
I just signed up to your blogs rss feed. Will you post more on this subject?
Great post Bj. I have started experimenting with Combinatorial Testing recently on a project. I have found that you do need in depth knowledge of the inputs and outputs of the system to get a good combination of tests. However the tool I’m using, is a tool called ALLPAIRS by James Bach.
Do you know what the differences between ALLPAIRS and the tool you use?
Will be happy to hear your thoughts and ideas on:
http://xqual.freeforums.org/allowing-test-step-parametrization-t53.html
Is there a way to get notified on new posts?
Thanks,
Kobi
I look forward to reading about this.
Pingback: Tweets that mention I.M. Testy › Combinatorial Testing -- Topsy.com
Bj,
Thank you for posting this article and, especially, for posting your slides from your recent talk in Vietnam. I found them at http://www.testingmentor.com/conferences/Presentations/Rollison%20Combinatorial%20Testing.pdf
I’m a huge proponent of using combinatorial test design techniques and feel they are dramatically under-appreciated by the testing community.
As coincidence would have it, I just published a related blog post on pairwise and combinatorial test design techniques myself just a few minutes ago: http://hexawise.wordpress.com/2010/10/07/a-fun-presentation-on-a-powerful-software-test-design-approach/
I’m looking forward to your upcoming tips about selecting model input parameters and variables for use in test case generators.
A couple comments/questions about your upcoming tips:
1) It is great that you’ll be providing them. Google searches on combinatorial testing and pairwise testing methods turn up two types of information, either (A) barely understandable academic papers packed with complex equations, or (B) three page introductory articles that barely scratch the surface. Practical “How to” guidance from experienced practitioners like yourself are badly needed.
2) I would personally find it useful to hear your views about: (A) common mistakes users make, (B) types of situations where combinatorial testing is especially powerful, and (C) types of situations where it is not a useful approach.
3) In addition, I’d appreciate your elaboration on how you’ve dealt with teaching new testers about modeling techniques, e.g., (a) how much detail to include in plans, (b) how to handle/think about/apply abstractions in modeling, (c) what your experiences have been if you’ve combined exploratory testing with combinatorial testing.
Thanks again for contributing to the community by sharing information about this important topic.
- Justin Hunter
(Disclosure: I’m the founder of Hexawise, a combinatorial test case generating tool).