I.M. Testy

Much of western Washington got its first look of winter starting Monday. This year our winter is supposed to be especially harsh, and it started with a pre-Thanksgiving snow and below freezing temperatures. I woke up yesterday morning with the sun shining down on a beautiful white blanket of snow about 10 inches deep. Since it was a beautiful day, I got my cross country skis out and went for a trek around the neighborhood. I got back home after about 2 hours and went sledding in the backyard with my daughter. Of course, we had to make several snow angels before heading in for some hot chocolate beside the fire.

Up to this point in our discussion of combinatorial testing our models and the resulting tests have focused primarily on valid input values. This is commonly known as positive testing because all valid input combinations should result in a valid output condition or state. Testing different combinations of valid values for multiple input parameters that affect a common output condition or state sometimes exposes unexpected issues, and it also helps improve overall confidence in the feature being tested. Also, since positive testing should result in the behavior or output condition expected by the customer (most customers don’t throw junk inputs to see how many error messages they can find) then designing an automated oracle for positive tests is generally easier than designing automated oracles for negative tests. Basically, an automated oracle for a positive combinatorial test should evaluate whether the output condition or state matches the expected result and are there no unexpected issues such as an error message, an unhandled exception, or other failure.

But, shouldn’t we also test for negative input values as well? That’s a really great question, and I must say at this point I am not overly convinced it is necessary for 2 reasons.

• Most exception handling is the result of single mode errors. In other words, let’s say we have a Windows form that takes 3 integer inputs and the inputs are not validated until the OK or Apply button is clicked. And let’s say that we input the character ‘A’ in each input control. Typically, when the user presses OK or Apply an exception will be thrown (and the application will display an error message) on the first invalid condition detected. Once we throw an exception control flow through the program usually does not look for additional errors or the user would likely get cascading error messages.
• Most errors are not the result of multiple invalid input values in different combinations. For example, even in the case of some web forms that check for multiple input errors the web form usually does not get posted until all input errors on the form are corrected by the user. Even in this case it is unlikely that multiple invalid input values would result in an unexpected error condition.

So we must ask ourselves 3 questions.

• Do we think that an invalid input value in some combination with different valid input values would produce an unexpected result that would not be detected with a test designed to expose single mode errors?
• Do we think that we need to extensively test for different combinations of invalid input values?
• Do we need to rerun theses types of tests throughout the product’s software development lifecycle (SDLC)?

These are important questions to ask because when done properly and in the right context combinatorial testing of inputs is more ‘expensive’ as compared to other approaches to testing. For example, we know that combinatorial testing can find single mode errors but there are more efficient (less ‘expensive’) ways to find single mode bugs. Testing combinations of input values that affect a common output condition involves identifying the output condition or state, identifying the input parameters and variables per parameter, modeling the inputs, reviewing the output from a tool, tweaking the model, and ideally automating a data driven test to test. This is a lot of work (‘expense’) to test whether an invalid input (e.g. a alpha character entered into a textbox that only takes integer values) will produce an appropriate response (e.g. an error message). Another consideration when testing with negative values in our various combinations is that if an unexpected error does occur we will likely have to spend some time investigating whether the anomaly is caused by interaction between different valid input values, or different valid and invalid inputs values, or if this is a single mode fault.

But, if the answer is yes to any of these questions, then we might want to use a combinatorial testing approach. Of course, including invalid input values in our model will produce negative tests in our baseline set of tests that are output from the tool. For example, let’s consider our different expected output conditions for the font dialog simulation.

• If all input values are valid we would expect the characters (actually glyphs) in our edit control to display the appropriate properties as defined by the input values, and no error messages, unhandled exceptions, and the application is not in a ‘not responding’ state.
• If the font color value is any invalid input when the OK or Apply button is clicked no error message is displayed but the font color reverts back to the last known valid font color (or the default font color).
• If the font size is less than 1 or greater than 1638 an error message is displayed, and the size reverts back to last valid value.
• If the font size is a decimal value other than n.5 than the number an error message is displayed (or the number might be rounded to the nearest whole number or n.5 value), and the size reverts back to last valid value.
• If the font color is invalid and the font size is invalid the color will revert to last known valid input and an error message will indicate an invalid font size.

So, let’s look at 2 different scenarios: multiple invalid input values, and single invalid values in combination with valid input values.

Testing multiple invalid input values

The assertion with combinatorial testing is that the interaction of 2 (or more) input values causes an unexpected condition or output. So, if we think that 2 or more invalid input values might result in an unexpected anomaly then we can simply add invalid input values to our model. For example, in our simple font dialog example we would include invalid inputs (in bold) for color and size in our model as illustrated below:

# Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsive
Style: Bold, Italic, BoldItalic, None(10)
Effects: Strike, Underline, StrikeUnderline, None(10)
Colors: Black(10), White, Red, Green, Blue, Yellow, Purple, Orange, randomString, emptyString
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf, emptyString, integerLessThan1, integerGreaterThan1638, floatValueOtherThan.5

# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

The set of combinatorial tests produced by a tool will include both positive and negative tests. For example this model would produce a set of tests that includes combinations such as Color == emptyString and Size == emptyString, and Color == Purple and Size == integerGreaterThan1638 in combination with values for the other input parameters. It would also include tests such as Color == randomString with valid inputs for the other parameters. In this situation we would have to go through the set of tests produced by our tool one by one and identify the expected output condition(s) based on each combination of inputs. This approach might be more practical if we were executing these test combinations manually and we evaluated the outcome of each test. But, this would be a very time consuming process and require several complex oracles if we wanted to automate a data-driven test.

Testing single invalid input values

In some cases we may want to test invalid values in each input parameter individually in combination with valid values in other input parameters to test for specific expected error conditions (e.g. error messages). In this situation we need a way to identify the invalid input values so that they are not used in combination with other invalid values for other input parameters. Fortunately, the PICT tool supports this type of analysis. In our input model we can identify invalid values in our model with the tilde (~) character. The modified model file below now includes the negative values for the size and colors parameters.

# Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsive
Style: Bold, Italic, BoldItalic, None(10)
Effects: Strike, Underline, StrikeUnderline, None(10)
Colors: Black(10), White, Red, Green, Blue, Yellow, ~randomString, ~emptyString
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf, ~emptyString, ~integerLessThan1, ~integerGreaterThan1638, ~floatValueOtherThan.5

# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

The above model file produces a baseline set of combinatorial tests that includes all valid combinations as well as combinations that include 1 invalid value in n-way combinations with other valid values as illustrated in the tab-delimited output file. Notice that only one invalid value occurs in each individual test. This approach makes our oracle problem a bit easier to solve because each test should produce an expected output condition as described above.

But, we still have to identify the expected output for each test in our set of combinatorial tests produced by the PICT tool. Fortunately, an undocumented feature in the PICT tool allows us to specify the expected output or result. To model the expected result we simply include a parameter starting with the dollar sign symbol ($). For example, we would modify our model file to include the “$Result:” parameter and assign expected output conditions to that parameter as illustrated below.

# Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsive
Style: Bold, Italic, BoldItalic, None(10)
Effects: Strike, Underline, StrikeUnderline, None(10)
Colors: Black(10), White, Red, Green, Blue, Yellow, ~randomString, ~emptyString
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf, ~emptyString, ~integerLessThan1, ~integerGreaterThan1638,  ~floatValueOtherThan.5
# Expected Results Parameter
$Result: ErrorMessage, DefaultColor

# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

# Expected Outputs
if [Colors] in {"randomString", "emptyString"} then [$Result] = "DefaultColor";
if [Size] in {"emptyString", "integerLessThan1", "integerGreaterThan1638", "floatValueOtherThan.5" } then [$Result] =
      "ErrorMessage";

Now our the output from the PICT tool also includes a column for the expected output conditions as illustrated. Notice that in this case that if the expected result is a valid outcome (the font properties match the input values) the Result column has a question mark symbol (?). If there are multiple expected output conditions the ‘valid’ output conditions will be listed with the question mark character as in this example.

If the expected result is binary (e.g. Error or NoError) then we can use one statement in our model file such as:

if [param] = “InvalidCondition” then [$Result] = “Error” else [$Result] = “NoError”;

The value of the $Result parameter is that we can use the $Result values as flags in our automated tests to switch between different automated oracles to help validate the specified expected result.

This approach models both valid and invalid input values and the PICT tool produces both positive and negative tests in the the output set of combination tests. Using the $Result parameter as a flag is an effective solution to switch between automated oracles and allows us to design a single automated data-driven test. Even with this approach I ask myself whether an invalid input in combination with valid inputs would likely cause an unexpected error, or would this likely be a single mode failure. But, if I really don’t know how invalid inputs are validated before being passed to the appropriate function then including invalid values in our model could also increase our overall test coverage and improve our confidence or potentially expose some really random errors!

It is a rainy day in Seattle. It has been a busy week at work. I spent 2 days in training then spent 2 days training. Monday was mostly a blur with the exception of 1 memorable meeting. It is also pretty cold and they are predicting snow this weekend in the higher elevations (which includes my home). Fortunately I have 3 chords of wood cut and stacked (hopefully more than enough), the backup generator is tuned and ready (for those pesky power outages), and most of the outside work is done. Also, it could mean an early ski season…YEAH!

Over the past few weeks we have been discussing combinatorial testing. I have tried to focus on how to overcome some obstacles and common mistakes made by inexperienced or untrained testers, and also how we can use this technique and an effective tool to help us significantly improve test coverage. So far I have covered:

• How to effectively model of the input parameters that affect a common output condition
• Modifying the model to deal with mutually exclusive input value combinations
• Using random data generation to increase test coverage
• Automatically generate different combinatorial sets of tests
• Add weights to important values in our model file
• Include sub-models within a model to increase test coverage
• Automatically seed the output with a set of combinations that must be tested

Up to this point we have been looking at a rather simple feature with a limited number input parameters. But, I mentioned previously that combinatorial testing is really demonstrates its full potential when we are faced with a highly complex feature that has numerous input parameters that affect a common output condition. Also, our outputs so far have been based on all variable combinations for every pair of parameters, or pairwise analysis.

Pairwise or 2-way analysis is actually pretty effective because the combinatorial fault model suggests that most errors that result from the interaction of input variables occurs between the simple interaction of 2 inputs. Also, many studies demonstrate that a pairwise or 2-way analysis of input values is very effective in exposing a high percentage of multi-modal (and single mode) defects. But, although pairwise testing is effective in revealing more than 50% of the multi-modal errors there could still be bugs lurking that are caused by more complex interactions of input variables. In a multiyear study of various types of software products Richard Kuhn and Raghu Kacker found that some additional multi-modal bugs were exposed when 3 or more input values interacted as illustrated in the graph below.

R.Kuhn and R. Kacker, “Combinatorial Methods for Cybersecurity Testing”, 2009

Fortunately, in order to increase the order of n-way combinations being tested we don’t have to change our model. But, we do need a tool capable of performing a (2 + n)-way analysis of input parameters, and PICT (and other tools) can do just that. Passing the /o:n switch to the PICT command line (where n is the order up to the maximum number of parameters) will cause PICT to generate a n-way analysis of the input values.

Now, you may be asking why don’t we just start with 3-way or 4-way analysis? The simple answer is cost. Each increment in the n-way order causes an approximate quadratic increase in the number of test combinations as illustrated in the table below. This particular feature had 421,200 possible combinations. We can also see an incremental increase in the number of blocks of code exercised, but this may not always be the case. (Also, remember code coverage does not correlate to quality, but it may help improve confidence and reduce overall risk.)

Even if our test is automated we might not want to immediately do a 3-way or higher analysis of value interactions. (Automation != free testing.) The strategy we use is to start with 2-way interactions, then generate different 2-way sets until no more bugs are found. Then generate an output for 3-way interactions to look for more complex issues. If no bugs are found with 3-way interactions we might jump to 4-way or not depending on the complexity and criticality of the feature, and the confidence of the tester. 

The bottom line is that sometimes pairwise or 2-way analysis of values for interdependent input parameters is not sufficient to find bugs cause by more complex interactions. To help resolve this problem testers need a tool that can generate n-way combinations beyond the a basic pairwise analysis.

In the final installment in this series I will discuss negative testing.

Autumn is in full swing here in Seattle. I love autumn. The vibrant colors, the smell in the air, the brisk morning air, and the anticipation of snow in the hills. It is also a busy time for me, chopping and stacking wood (it is supposed to be a rough winter here in Seattle), turning over the vegetable gardens, trimming various plants around the house, and getting the generator ready for our almost yearly power outages in my somewhat remote neighborhood. There is something rewarding about this time of year…perhaps it is simply that I made it through another year still mostly sane, and will soon have a new year to look forward to with some exciting changes.

In the previous posts (here and here) on selecting the right values we discussed strategies for potentially improving the likelihood of testing with and increased distribution of input values if the number of possible inputs is very large for a given input parameter, and how the PICT tool can randomize the base set of combinations output by the tool to increase test coverage of the total number of possible combinations. In both situations, these are simply strategies that you can use to improve the effectiveness of your tests by using different values and different combinations that might help improve the probability of selecting the “right” values or combinations to test with.

Weighting important input combinations

The output of most pairwise tools treats all input values equally. Of course we know that not all values or combinations are not used equally by our customers. There are some values or some combinations that are more likely to be used by some customers, and other customers might simply use default or specific settings. Of course, one thing we should do as testers is identify our customer configurations and usage patterns to make sure we are covering highly likely user scenarios.

One way we can increase the likelihood of including frequently used input values in combination with other values is to weight the important or likely input values. For example, we discover that our customers are most likely to use the Arial font, with a font color of black, a nominal size, no style and no effects. So, we can give these values more weight to emphasize their importance. To add weight to an input value we simply enter an integer value within parenthesis after the input value as illustrated in our updated PICT model file below.

# Basic Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsive

Style: Bold, Italic, BoldItalic, None(10)

Effects: Strike, Underline, StrikeUnderline, None(10)

Colors: Black(10), White, Red, Green, Blue, Yellow

# This does not include abstract size ranges for half-size font sizes (e.g. 1.5 – 1637.5)
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf

# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

The weight values have no absolute value. By weighting Style value None with a value of 10 does not mean that it will be used 10 times more than other values. Also, weighted values may not change the frequency of use because PICT will generate all n-way combinations in the smallest number of tests. When a value is already used in all possible combinations and there is a ‘tie among values for a particular input parameter then weighted input values will have a greater probability of usage in the output.

For example, in the baseline set of tests we can see a difference in our outputs from the PICT tool by comparing an output from a model without weighted input values (the spreadsheet on the left) with an output based on the above model file with weighted input values (the spreadsheet on the right). In this case the number of occurrences of the use of the Black font color increased by 2.

Further analysis of the output revealed the number of instances of the use of Arial font, the None style, and the None effects all increased. But, there was no increase in the selection of the font size.

Weighting doesn’t always mean that a particular value will be used more frequently then other values for that a given input parameter. But, weighting input values is one strategy that may increase the probability of using important values in different combinations.

Covering highly probable combinations

Another strategy for increasing the test coverage different combinations is to create a sub-model of important parameters that are more likely to be used, or more likely to potentially cause an error. For example, let’s say we think that there is a greater likelihood of error between the Effects and the Style input parameters irrespective of the color, font, or size. One thing we can do with our PICT tool is to modify the model file to include a sub-model of those 2 parameters as illustrated below.

# Basic Model File for MyFontDialog
Font: Arial(50), Tahoma, BrushScript, MonotypeCorsive

Style: Bold, Italic, BoldItalic, None(10)

Effects: Strike, Underline, StrikeUnderline, None(10)

Colors: Black(10), White, Red, Green, Blue, Yellow

# This does not include abstract size ranges for half-size font sizes (e.g. 1.5 – 1637.5)
Size: small, smallHalf, nominal(10), nominalHalf, large, largeHalf, xLarge, xLargeHalf, xxLarge, xxLargeHalf

# Sub-model of Style and Effects input parameters
{ Style, Effects } @ 2

# Conditional constraints necessary to prevent mutually exclusive variable settings
# See previous post for dealing with mutually exclusive variables
if [Font] = "BrushScript" then [Style] in { "Italic", "BoldItalic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "BoldItalic" };

Sub-models improves the thorough testing of the specified parameters. By adding the above sub-model the number of combination tests increased from approximately 60 to just over 100 for our font dialog example.

Testing Specific Combinations

In some cases, there are specific values or combinations of values that we want to test because they are the default values, and we know that most of our customers don’t change the default settings. Or, maybe we have gathered usage patterns from our customers and identified combinations of values that a significant portion of our target customers use regularly. Or perhaps we know from experience or intuition there are combinations that are more likely to be problematic or are somehow “interesting.”

From a manual test execution perspective including specific combinations is straight forward. But, what if we want to test specific combinations of values once per sprint or milestone (or after each build) in our data-driven automated test. Again, we could simply specify the values in the output produced by the tool. But, what if we wanted to generate different sets of input combinations as illustrated in this post. In this situation it is not very efficient to add the specific combinations for each output; especially when the output can be generated during the runtime of the automated test quite efficiently.

A solution to this situation is to be able to seed the output with specific combinations that must appear in the baseline set of tests. In this case, I can create a tab-delimited file such as the one illustrated to the right.

Now whenever I need my automated test to execute these specific combinations of input values all I need to do is pass an argument to the PICT tool to include the seed file in the output as follows

pict.exe basicmodel.txt > output.xls /e:[path]/myseedfile.txt

The PICT tool will then use the seeded combinations in the baseline set. Even if we generate a random set using the /r:n as discussed in an earlier post these seeded inputs will still be included as long as we pass the /e:[filename] switch to the PICT tool.

So, up to now we have discussed:

• A key to understanding how to use combinatorial or pairwise testing depends largely on our ability to effectively model the input parameters that affect a common output condition
• We can modify the model to deal with mutually exclusive input value combinations using a simple built in syntax for conditional or invariant constraints
• We can increase the likelihood of testing different values using random data generation techniques
• We can increase the test coverage of different combinations by using PICT to randomize the output of the baseline set of tests
• We can increase the potential to use certain values with greater frequency by weighting important values
• We can increase the thoroughness of test coverage between specific input combinations with sub-models
• We can force the tool to include specific input value combinations in a complete n-way output of other combinations using a seed file that specifies those combinations

Next week, I will discuss increasing the order beyond pairwise, and later we still have to discuss negative testing. I hope you find these posts valuable, and if you have any questions please don’t hesitate to ask via the comments or email.

I am finally getting caught up at work after the trip to Korea and Israel. I managed to cut the lawn in the front of my house on Sunday, but still have an acre and a half in the back yard to tend to, rake some leaves, till the garden, tend to the pond, and there is probably some other stuff on the list for this weekend. In the meantime, it is time for another installment of the series on combinatorial testing.

Another reason cited by this paper calling out risks associated with pairwise testing was, “The problem, as we see it, is that the key concept of how program inputs variables interact to create outputs is missing from the pairwise testing discussion. The pairwise testing technique does not even consider outputs of the program (i.e., the definition and application of the pairwise technique is accomplished without any mention of the program’s outputs).” This statement completely surprises me because the foundational principles (heuristic) of this technique or pattern of test is that some errors result from the interaction of input variables adversely affecting a single common output condition or state. So, I wonder how testers can approach a combinatorial testing problem if they don’t know or consider the output condition or state being evaluated? Do testers really simply plug in input variables and wait to see what happens?

Then the authors state,“…it seems to us to become easier for novice testers, and experienced testers alike, to blindly apply it to every combinatorial testing problem, rather than doing the hard work of figuring out how the software’s inputs interact in creating outputs.” I totally agree. But, I would say that people who don’t understand how to use tools properly (novice or otherwise) are likely to get ‘hurt.’ This isn’t necessarily a problem with the technique or tool; this is simply the lack of knowledge or skill of a novice or untrained tester. To effectively use the technique of combinatorial testing and apply test tools in the appropriate context testers must be trained, learn to select the right tools for the job, and be knowledgeable of how the input values impact the output for the feature being tested.

Up to this point we have blindly considered that all combinations of input values for a given output condition are possible. However, this is not always the case. For example, a common testing problem is the increasing matrix of operating systems, service packs, browser versions, flash versions, etc. in configuration or setup testing. Suppose that one of the OS versions is Windows Xp and another is Windows 7, and we want to include IE 6.0, IE 7.0 and IE 8.0 as part of our possible customer configurations. Even if we hacked our way into installing IE 6.0 on Windows 7 it would be a completely unsupported, out of context scenario. The combination of Windows 7 and IE 6.0 are mutually exclusive and so we need to find a way to prevent such combinations from being generated by our tool.

 Validate the output!

In the font dialog example we also have exclusive combinations. The Brush Script MT font can only be Italic or Bold/Italic, and the Monotype Corsive font can only have styles of Bold/Italic or regular (neither bold nor italic selected). But when we examine the baseline set of combinations output by our combinatorial test generation tool we can see there are several invalid combinations.

This is a problem because if we tried to test these combinations our test would indicate a failure because the user cannot force these combinations to occur. But, in this case the failure is not a bug in the feature being tested it is a false positive. The bug is in our test data (values used in the combinations). We must remember…the output from the tool is based on our the tester’s input…it’s only a tool! This why it is critical to review the output from the tool and validate the test combinations.

But, we do not want to simply remove these combinations from our baseline test set, nor do we don’t want to arbitrarily change the the font or the style values because that may change the n-way combinations with other input parameter values.

Applying Conditional Constraints

We don’t want to arbitrarily change our output from the tool, but we need a way to effectively handle the mutually exclusive values for the Monotype Corsive and Brush Script MT fonts and the font styles. To solve this problem, the PICT tool employs a simple scripting language that enables the tester to modify how the output baseline set of combinations are generated. In other words we can ‘program’ the tool to constrain specified parameter values from being used in combinations.

In our font dialog example, we will need to add 2 statements to our model file as illustrated below.

# Basic Model File for MyFontDialog with Conditional Constraints
Font: Arial | Tahoma, BrushScript, MonotypeCorsive
Style: Bold, Italic, BoldItalic, None
Effects: Strike, Underline, StrikeUnderline, None
Colors: Black, White, Red, Green, Blue, Yellow
Size: small, smallH, nominal, nominalH, large, largeH, xLarge, xLargeH xxLarge, xxLargeH

# Conditional constraints prevent mutually exclusive variable settings
if [Font] = "BrushScript" then [Style] in { "Italic", "Bold/Italic" };
if [Font] = "MonotypeCorsive" then [Style] in { "None", "Bold/Italic" };

Revalidate Output

Now we take our modified model file and pass it to the PICT tool. The PICT tool will again generate a set of baseline tests, but this time the test combinations are generated based on the conditional constraints we ‘programmed’ into the model. We need to revalidate the tool’s output to ensure our model is producing a set of combinations for the intended purpose of our test. Our new baseline set of positive test combinations should not generate a false positive for the mutually exclusive font and font style values.

Some may argue that we want to include erroneous behavior to see how the feature responds. That is a perfectly valid argument if are performing manual testing and only limiting our combination tests to a small baseline subset of test. But, if the font dialog behavior allowed me to set the Brush Scritpt MT font to bold only, that is most likely a single-mode fault in the event handler. In other words that error is likely to occur regardless of what the other parameter variables are, and I don’t need to invest in a combinatorial testing approach to identify that issue. Another consideration involves the automation approach. If I am programmatically accessing a control’s window handle to set the state then I can actually force a control into an invalid state which would again result in a false positive because my automated test case is forcing an invalid state.

I will discuss strategies for negative testing of input combinations in a later post. I personally think separating positive tests and negative tests is a good strategy. Positive tests are generally designed to provide confidence while negative tests are usually intended to expose issues. Also, the (automated) oracles tend to be different for negative and positive tests.

In highly complex features the model may have to be ‘tweaked’ several times. However, once the model is complete we won’t have to change it again unless the feature changes. And every time we get a new build we can increase our positive test coverage of combinations that should work by generating numerous test combinations from a single model file using the PICT tool output.

Up to now we have discussed the importance of decomposing the feature based on the output state or condition under test, identifying the variables for each parameter, modeling the variables when appropriate, randomly selecting value from large sets of test data, randomizing the set of combinations tested, and how to prevent false positives by modifying the model using conditional constraints to deal with mutually exclusive variables. In the next post we will discuss how to test with important variable combinations and increase the likelihood of testing with important values.