Combinatorial Testing: Testing Highly Probable Combinations
Autumn is in full swing here in Seattle. I love autumn. The vibrant colors, the smell in the air, the brisk morning air, and the anticipation of snow in the hills. It is also a busy time for me, chopping and stacking wood (it is supposed to be a rough winter here in Seattle), turning over the vegetable gardens, trimming various plants around the house, and getting the generator ready for our almost yearly power outages in my somewhat remote neighborhood. There is something rewarding about this time of year…perhaps it is simply that I made it through another year still mostly sane, and will soon have a new year to look forward to with some exciting changes.
In the previous posts (here and here) on selecting the right values we discussed strategies for potentially improving the likelihood of testing with and increased distribution of input values if the number of possible inputs is very large for a given input parameter, and how the PICT tool can randomize the base set of combinations output by the tool to increase test coverage of the total number of possible combinations. In both situations, these are simply strategies that you can use to improve the effectiveness of your tests by using different values and different combinations that might help improve the probability of selecting the “right” values or combinations to test with.
Weighting important input combinations
The output of most pairwise tools treats all input values equally. Of course we know that not all values or combinations are not used equally by our customers. There are some values or some combinations that are more likely to be used by some customers, and other customers might simply use default or specific settings. Of course, one thing we should do as testers is identify our customer configurations and usage patterns to make sure we are covering highly likely user scenarios.
One way we can increase the likelihood of including frequently used input values in combination with other values is to weight the important or likely input values. For example, we discover that our customers are most likely to use the Arial font, with a font color of black, a nominal size, no style and no effects. So, we can give these values more weight to emphasize their importance. To add weight to an input value we simply enter an integer value within parenthesis after the input value as illustrated in our updated PICT model file below.
# Basic Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsiveStyle: Bold, Italic, BoldItalic, None(10)
Effects: Strike, Underline, StrikeUnderline, None(10)
Colors: Black(10), White, Red, Green, Blue, Yellow
# This does not include abstract size ranges for half-size font sizes (e.g. 1.5 – 1637.5)
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };
The weight values have no absolute value. By weighting Style value None with a value of 10 does not mean that it will be used 10 times more than other values. Also, weighted values may not change the frequency of use because PICT will generate all n-way combinations in the smallest number of tests. When a value is already used in all possible combinations and there is a ‘tie among values for a particular input parameter then weighted input values will have a greater probability of usage in the output.
For example, in the baseline set of tests we can see a difference in our outputs from the PICT tool by comparing an output from a model without weighted input values (the spreadsheet on the left) with an output based on the above model file with weighted input values (the spreadsheet on the right). In this case the number of occurrences of the use of the Black font color increased by 2.
Further analysis of the output revealed the number of instances of the use of Arial font, the None style, and the None effects all increased. But, there was no increase in the selection of the font size.
Weighting doesn’t always mean that a particular value will be used more frequently then other values for that a given input parameter. But, weighting input values is one strategy that may increase the probability of using important values in different combinations.
Covering highly probable combinations
Another strategy for increasing the test coverage different combinations is to create a sub-model of important parameters that are more likely to be used, or more likely to potentially cause an error. For example, let’s say we think that there is a greater likelihood of error between the Effects and the Style input parameters irrespective of the color, font, or size. One thing we can do with our PICT tool is to modify the model file to include a sub-model of those 2 parameters as illustrated below.
# Basic Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsiveStyle: Bold, Italic, BoldItalic, None(10)
Effects: Strike, Underline, StrikeUnderline, None(10)
Colors: Black(10), White, Red, Green, Blue, Yellow
# This does not include abstract size ranges for half-size font sizes (e.g. 1.5 – 1637.5)
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf# Sub-model of Style and Effects input parameters
{ Style, Effects } @ 2# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "BoldItalic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "BoldItalic" };
Sub-models improves the thorough testing of the specified parameters. By adding the above sub-model the number of combination tests increased from approximately 60 to just over 100 for our font dialog example.
Testing Specific Combinations
In some cases, there are specific values or combinations of values that we want to test because they are the default values, and we know that most of our customers don’t change the default settings. Or, maybe we have gathered usage patterns from our customers and identified combinations of values that a significant portion of our target customers use regularly. Or perhaps we know from experience or intuition there are combinations that are more likely to be problematic or are somehow “interesting.”
From a manual test execution perspective including specific combinations is straight forward. But, what if we want to test specific combinations of values once per sprint or milestone (or after each build) in our data-driven automated test. Again, we could simply specify the values in the output produced by the tool. But, what if we wanted to generate different sets of input combinations as illustrated in this post. In this situation it is not very efficient to add the specific combinations for each output; especially when the output can be generated during the runtime of the automated test quite efficiently.
A solution to this situation is to be able to seed the output with specific combinations that must appear in the baseline set of tests. In this case, I can create a tab-delimited file such as the one illustrated to the right.
Now whenever I need my automated test to execute these specific combinations of input values all I need to do is pass an argument to the PICT tool to include the seed file in the output as follows
pict.exe basicmodel.txt > output.xls /e:[path]/myseedfile.txt
The PICT tool will then use the seeded combinations in the baseline set. Even if we generate a random set using the /r:n as discussed in an earlier post these seeded inputs will still be included as long as we pass the /e:[filename] switch to the PICT tool.
So, up to now we have discussed:
- A key to understanding how to use combinatorial or pairwise testing depends largely on our ability to effectively model the input parameters that affect a common output condition
- We can modify the model to deal with mutually exclusive input value combinations using a simple built in syntax for conditional or invariant constraints
- We can increase the likelihood of testing different values using random data generation techniques
- We can increase the test coverage of different combinations by using PICT to randomize the output of the baseline set of tests
- We can increase the potential to use certain values with greater frequency by weighting important values
- We can increase the thoroughness of test coverage between specific input combinations with sub-models
- We can force the tool to include specific input value combinations in a complete n-way output of other combinations using a seed file that specifies those combinations
Next week, I will discuss increasing the order beyond pairwise, and later we still have to discuss negative testing. I hope you find these posts valuable, and if you have any questions please don’t hesitate to ask via the comments or email.
Pingback: Combinatorial testing | Aziel’s tech blog
Pingback: OpenQuality.ru | Качество программного обеспечения
In one of the bullet points you mentioned
‘We can increase the likelihood of testing different values using random data generation techniques’
I am not sure how you could use random data, as most of the blog posts talk only about changing valid input data. Can you pls elaborate the bullet point.
Pingback: Tweets that mention I.M. Testy › Combinatorial Testing: Testing Highly Probable Combinations -- Topsy.com