I.M. Testy

I am finally getting caught up at work after the trip to Korea and Israel. I managed to cut the lawn in the front of my house on Sunday, but still have an acre and a half in the back yard to tend to, rake some leaves, till the garden, tend to the pond, and there is probably some other stuff on the list for this weekend. In the meantime, it is time for another installment of the series on combinatorial testing.

Another reason cited by this paper calling out risks associated with pairwise testing was, “The problem, as we see it, is that the key concept of how program inputs variables interact to create outputs is missing from the pairwise testing discussion. The pairwise testing technique does not even consider outputs of the program (i.e., the definition and application of the pairwise technique is accomplished without any mention of the program’s outputs).” This statement completely surprises me because the foundational principles (heuristic) of this technique or pattern of test is that some errors result from the interaction of input variables adversely affecting a single common output condition or state. So, I wonder how testers can approach a combinatorial testing problem if they don’t know or consider the output condition or state being evaluated? Do testers really simply plug in input variables and wait to see what happens?

Then the authors state,“…it seems to us to become easier for novice testers, and experienced testers alike, to blindly apply it to every combinatorial testing problem, rather than doing the hard work of figuring out how the software’s inputs interact in creating outputs.” I totally agree. But, I would say that people who don’t understand how to use tools properly (novice or otherwise) are likely to get ‘hurt.’ This isn’t necessarily a problem with the technique or tool; this is simply the lack of knowledge or skill of a novice or untrained tester. To effectively use the technique of combinatorial testing and apply test tools in the appropriate context testers must be trained, learn to select the right tools for the job, and be knowledgeable of how the input values impact the output for the feature being tested.

Up to this point we have blindly considered that all combinations of input values for a given output condition are possible. However, this is not always the case. For example, a common testing problem is the increasing matrix of operating systems, service packs, browser versions, flash versions, etc. in configuration or setup testing. Suppose that one of the OS versions is Windows Xp and another is Windows 7, and we want to include IE 6.0, IE 7.0 and IE 8.0 as part of our possible customer configurations. Even if we hacked our way into installing IE 6.0 on Windows 7 it would be a completely unsupported, out of context scenario. The combination of Windows 7 and IE 6.0 are mutually exclusive and so we need to find a way to prevent such combinations from being generated by our tool.

 Validate the output!

In the font dialog example we also have exclusive combinations. The Brush Script MT font can only be Italic or Bold/Italic, and the Monotype Corsive font can only have styles of Bold/Italic or regular (neither bold nor italic selected). But when we examine the baseline set of combinations output by our combinatorial test generation tool we can see there are several invalid combinations.

This is a problem because if we tried to test these combinations our test would indicate a failure because the user cannot force these combinations to occur. But, in this case the failure is not a bug in the feature being tested it is a false positive. The bug is in our test data (values used in the combinations). We must remember…the output from the tool is based on our the tester’s input…it’s only a tool! This why it is critical to review the output from the tool and validate the test combinations.

But, we do not want to simply remove these combinations from our baseline test set, nor do we don’t want to arbitrarily change the the font or the style values because that may change the n-way combinations with other input parameter values.

Applying Conditional Constraints

We don’t want to arbitrarily change our output from the tool, but we need a way to effectively handle the mutually exclusive values for the Monotype Corsive and Brush Script MT fonts and the font styles. To solve this problem, the PICT tool employs a simple scripting language that enables the tester to modify how the output baseline set of combinations are generated. In other words we can ‘program’ the tool to constrain specified parameter values from being used in combinations.

In our font dialog example, we will need to add 2 statements to our model file as illustrated below.

# Basic Model File for MyFontDialog with Conditional Constraints
Font: Arial | Tahoma, BrushScript, MonotypeCorsive
Style: Bold, Italic, BoldItalic, None
Effects: Strike, Underline, StrikeUnderline, None
Colors: Black, White, Red, Green, Blue, Yellow
Size: small, smallH, nominal, nominalH, large, largeH, xLarge, xLargeH xxLarge, xxLargeH

# Conditional constraints prevent mutually exclusive variable settings
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

Revalidate Output

Now we take our modified model file and pass it to the PICT tool. The PICT tool will again generate a set of baseline tests, but this time the test combinations are generated based on the conditional constraints we ‘programmed’ into the model. We need to revalidate the tool’s output to ensure our model is producing a set of combinations for the intended purpose of our test. Our new baseline set of positive test combinations should not generate a false positive for the mutually exclusive font and font style values.

Some may argue that we want to include erroneous behavior to see how the feature responds. That is a perfectly valid argument if are performing manual testing and only limiting our combination tests to a small baseline subset of test. But, if the font dialog behavior allowed me to set the Brush Scritpt MT font to bold only, that is most likely a single-mode fault in the event handler. In other words that error is likely to occur regardless of what the other parameter variables are, and I don’t need to invest in a combinatorial testing approach to identify that issue. Another consideration involves the automation approach. If I am programmatically accessing a control’s window handle to set the state then I can actually force a control into an invalid state which would again result in a false positive because my automated test case is forcing an invalid state.

I will discuss strategies for negative testing of input combinations in a later post. I personally think separating positive tests and negative tests is a good strategy. Positive tests are generally designed to provide confidence while negative tests are usually intended to expose issues. Also, the (automated) oracles tend to be different for negative and positive tests.

In highly complex features the model may have to be ‘tweaked’ several times. However, once the model is complete we won’t have to change it again unless the feature changes. And every time we get a new build we can increase our positive test coverage of combinations that should work by generating numerous test combinations from a single model file using the PICT tool output.

Up to now we have discussed the importance of decomposing the feature based on the output state or condition under test, identifying the variables for each parameter, modeling the variables when appropriate, randomly selecting value from large sets of test data, randomizing the set of combinations tested, and how to prevent false positives by modifying the model using conditional constraints to deal with mutually exclusive variables. In the next post we will discuss how to test with important variable combinations and increase the likelihood of testing with important values.