I.M. Testy

Originally Published Sunday, September 30, 2007

I have been teaching formal testing techniques for several years at Microsoft and University of Washington Extension. Techniques are systematic procedures to help solve a complex problem. A technique does not find all types problems; techniques are generally very good at finding very specific classes of defects. But, the usefulness or effectiveness of any particular technique relies on the in-depth system and domain knowledge of the tester, and in the tester’s skill and ability to apply the correct technique to a problem space.

One of those techniques that I teach is equivalence class partitioning (ECP). On the surface ECP appears rather simple, but the effectiveness in the application of this technique relies on the tester’s ability to decompose input and output variables into discrete subsets of valid and invalid classes.

Over-generalization of the data reduces the number of subsets in a specific class and increases the likelihood of under-testing. Hyper-decomposition of the variables may increase the number of subsets in a specific class without providing additional benefit (although it may potentially increase the number of tests). Based on this observation I have proposed the following hypothesis or Bj’s Theory of Equivalence Class Partitioning Data Decomposition: Over-generalization of data reduces the baseline tests and increases the probability of missing errors or generating false positives. Hyper-analysis of data increases the probability of redundancy and reduces the overall effectiveness of each test.

ECP is not some rote activity, but in order for it to be most effective the tester requires a lot of in-depth knowledge about the overall system (hardware, operating system, environment, etc), the domain (client program, programming language used to develop the application, API interfaces, etc.), and the data sets. Limited knowledge or inability to adequately analyze the data based on the system and domain spaces in one or more of these areas is an impediment to the effective application of this technique. I suspect those who fail to perform an in-depth analysis of the system, domain, and data interaction leads some people disregard this technique, or become hyper-critical of its usefulness.

In this months issue of Software Test & Performance magazine I discuss the theory and application of the equivalence class partitioning technique as a viable and extremely useful tool used by professional testers. The magazine is free to download, and the article goes into great depth explaining when and how to apply the technique of equivalence class partitioning. Let me know if you have any questions, and I would appreciate your comments or feedback.

Originally Published Friday, September 14, 2007

I have never been really good at math. Sure I understand basic formulas, but I rely on a calculator when I run out of fingers and toes. I am envious of people who can look at a hexadecimal or octal value and convert it to an integer value in their heads without a second thought. But, I do know that when I multiply  1,073,741,824 * 1,073,741,824 the result should not equal 0! No, this isn’t some perverse bug in Calculator (calc.exe), but this sort of magic math sometimes manifests itself in the outputs of various calculations. (And there is a lot of calculating going on in even simple software programs.)

Familiarity with data types and specifically the physical linear boundaries of data types can help expose defects or explain unexpected errors encountered while testing, especially when evaluating output variables. For example, the physical linear boundary of a type int is -2,147,483,648 through +2,147,483,647 because this data type is a signed 32-bit (2³²) integer value. A value greater than +2,147,483,647  input into a control that only accepts a signed integer value will throw an overflow exception error. But, what happens when we have 2 input parameters that accept type int and the result of the calculation is greater than the maximum bounded value of the data type? Well, the answer to that questions depends on the data type used to store the output. For example, we should expect the result of the variable z below to be  4,294,967,294.

            int x = 2147483647;
            int y = 2147483647;
            int z = x + y;

However, the result is actually -2! How did that happen? Since the maximum value of a signed int is 2,147,483,647, when y is added to x the number simply wraps itself within the physical bounds of the data type. If we added 1 to 2,147,483,647 the int value would equal -2,147,483,648. (In the above example, the number keeps counting positively to eventually equal -2). Hopefully, it is easy to understand why wrapping can be a bad thing! I am thinking some billionaire would be pretty upset if his/her bank balance was $2,147,483,647 and he/she made a deposit of let’s say $10,000 on Monday, then wrote a check for $5 on Wednesday and the check bounced because of a negative balance of -2,147,473,649!

Historical failure indicators suggest traditional boundary type defects have a high rate of occurance any time complex calculations act on linear bounded input or output values. This doesn’t imply that "most bugs occur at boundaries," (in fact that is quite a foolish notion). It simply means that empirical evidence has consistently proven traditional boundary defects are likely to occur at the edges of bounded linear variables, especially involving the output variables of complex calculations.

A real life example of a wrapping type bug can be found in Word 2007; although it is not quite as obvious as the above example. In the below example, the string is indented 5.9" to the left.

Figure 1.

If we increase the size of the indentation to 6.0" the last character of the string wraps to the next line in the document, and this ‘wrapping’ is also displayed in the Preview window as illustrated below.Figure 2.

If we increase the indentation using the numeric up/down control buttons to 6.1" or larger we get an error message indicating the indent size is too large even though the Preview window illustrates the ‘g’ character wrapping to the next line! (In fact, if we input a size of 6.001" or larger we get the error message.)

 Figure 3.

Hmm…OK…so it will wrap one character but not 2 or more? That’s a little puzzling!!! (And yes, I know that I can append additional characters to the end of the string after wrapping in Figure 2., but that is not really relevant to the problem at hand.) Puzzling for sure, but at least the program stopped us at some point it calculated to be "too large,"and didn’t continue to wrap to a negative value as it did in Word 2003!

But, one thing I really dislike about software is when it does something, and then gives me an error message that makes it seem like I did something wrong. For example, if we reset the indentation back to 6.0" as illustrated in Figure 1, then highlight the string, and then click the Bullets button in the tool bar the string wraps to a vertical column of characters as illustrated in Figure 4. Next, we bring up the context menu, select Paragraph menu item and notice the software set the Left indentation to 6.15". So, although we couldn’t manually input any number greater than 6.001" I am thinking if the software did it own its own, then it must be OK…right? But, now we push the OK button on the Paragraph dialog and we get an error message that implies we did something wrong! How can that be…we didn’t change the indentation value, the software set the incorrect value? (Also notice the Preview pane doesn’t even look close to what is happening…I guess that’s what one would call WYSIAWYG or what you see is almost what you get.)

Figure 4.

I understand that word wrapping and word breaking algorithms are really hard especially given the complexity of the interactions between font types, kerning, bullets and numbering styles, paper width, margins, and numerous other variables. This is just used as an example of a traditional output type boundary defect caused by incorrect calculations. This example is used here to illustrate how this category of defects can manifest various symptoms due to a single root cause, and to also provide insight to you, the professional tester, as to how to think about and expose these types of traditional boundary errors.

Originally Published Wednesday, September 05, 2007

The traditional concept of boundary testing was established as a systematic procedure to more effectively and more efficiently identify a particular category of defects. Historically, boundary value analysis has focused on bounded physical (countable) linear input and/or output variables of independent parameters, and is especially useful in programming languages that are not strongly typed (such as C). The value of using the boundary value analysis technique is early identification of:

• typographical errors of specific bounded values (especially artificial constraints on a data type)
• improper use of relational operators around a physical bounded variable
• errors in implicit or explicit casting between data types

The application of boundary value analysis is not easy, because identifying boundary conditions from the user interface is extremely difficult. Also, if the tester lacks an understanding of programming concepts such as data types, casting, and looping structures then identifying boundaries becomes a mysterious art rather than a professional practice.

Casting (or conversion) between data types occurs rather frequently in software. For example, a calculator may only accept integer values as inputs, but the output of a division operation may result in a decimal value. In this case the operands are converted or parsed from strings to integer values using int.Parse(operand), or Convert.ToInt32(operand) in C#. Now, in order for the division operation to display decimal values if the result is not a whole number the operands must be cast to doubles (or floats) as in the example below.

   1: private double DivideOperation(int operand1, int operand2)

   2: {

   3:     //              the operands are explicitly cast from

   4:     //              integer types to double types

   5:     double result = (double)operand1 / (double)operand2;

   6:     return result;

   7: }

OK…so that is the basics of casting types, now let’s examine where casting can get us into trouble by exceeding physical bounded parameters. And to find these common errors we don’t have to go too far. In fact, the Attributes feature of MSPaint program provides us with plenty of examples of boundary issues.

Notice the default units for Width and Height parameters is Pixels. Pixels are in hole numbers. The input parameters Width and Height will accept a decimal value; but if the units selected is Pixels the decimal will be rounded to the nearest integer value. However, selecting either the inches or centimeters radio buttons for Units will allow decimal inputs. This is a pretty good clue that there is some pretty serious casting of these input parameters going on below the GUI. Now, we should know that inches are larger than centimeters, and centimeters are larger than pixels in size. We should also know that the maximum allowable input in the width or height parameter is limited to 5 characters, or a maximum valid input boundary of 99999.

Now there is nothing particularly unique or interesting about 99999, but we know that Pixels defaults to integer values, and inches and centimeters use decimal values, and that inches is bigger than pixels. And now, the magic math. If 29875 inches = 75882.50 centimeters, then 29876 inches should equal 75885.04 centimeters. (It doesn’t matter at this point if we click the OK button on the dialog and get an error message that reads "Please enter no more than 5 characters." even though we didn’t enter those 8 characters the software did that all by itself. We might not like the fact the software does what it tells the user he/she specifically can’t do, but (although many will argue this point) the program flooding its own edit control with more than 5 characters and then displaying an error message is really superficial and generally uninteresting.) But, it is a good thing we don’t use the Attributes feature of MSPaint as a conversion calculator because when we select inches as the units and enter 29876 in either the width or height parameters, then select the Cm radio button the value converts to 0!

What is this magical math? How can this happen when we aren’t even close to the maximum allowable input of 99999 inches? The value of 75885.04 is not some magical, unique or special value in computing. To understand what is happening we have to back up and convert from inches to pixels for a clearer picture. Select the inches radio button in the Unit groupbox, and enter the value 29875 for either the width or height parameters. Now select the Pixels radio button. Notice the value is 2147474. Now, testers know the upper physical bound of a signed 32-bit integer is 2 billion 147 million and some change (2,147,483,647 to be exact), but this value isn’t close. But, what if the last 3 characters are truncated by the control? Even in that case 29875 * 71.882 (pixels/inch) = 2147474.75 and the most significant digits above 1000 is still a value below the range of a signed 32-bit integer value (-2,147,483,648 ~ +2,147,483,647). But, when we enter 29876 inches  and then select the pixels radio button the value jumps to 4294967 (which are the most significant digits above the one thousands place of an unsigned 32-bit integer value (0 ~ 4294967296). This is because 29876 * 71.882 (pixels/inch) =  2147546.632 (and again looking at the significant values above the one thousands place) exceeds the upper physical bound of a signed 32-bit integer (2,147,483,648) resulting in an implicit cast to an unsigned 32-bit integer.

Now, simply setting the Units radio button to inches, entering 99999 for either the width or height parameters, and then selecting the Cm radio button, or selecting the Pixel radio button and then back to inches or Cm would reveal an error. However, a professional tester has enough working knowledge of the underlying system (including basic programming concepts) to not only know how and in which situations this type of defect can occur, but they can also use their cognitive abilities (having a basis in or reducible to empirical factual knowledge [of basic programming concepts in this case]) to perform conscious intellectual acts (cognition) to analyze features and identify areas where this type of problem might occur, and can then apply logical and rational tests that have a greater probability of exposing a defect (if one exists), or providing other valuable information regarding the valid operation or correctness of that feature.

There are other bugs in this relatively simple feature regarding data types, and perhaps I shall expose those at a later time!

Originally Published Wednesday, March 07, 2007

Over the past 3 days I have learned more about Mp3 file encoding and decoding than I have since the technology was introduced. I don’t spend time downloading files from the Internet to burn CD’s, I don’t own an iPod or Mp3 player, or a digital video recorder. So, prior to this I haven’t really paid attention to this technology and was quite ignorant of the various tools available and their capabilities. But, I must say it is pretty fascinating from a technology standpoint even though I am not an audiophile or videophile.

I still disagree with Pradeep’s assertion regarding boundary testing and the notion of no fixed boundaries, but respect Pradeep’s expertise in the area of Mp3 technology. An Aussie gentleman by the name of Dean Harding pointed out my incorrect assumption regarding bitrate encodings and explained the LAME encoder does allow a freeformat option in "expert" mode to produce a fixed bitrate in one kilo bit increments between 8 kb/s and 640 kb/s. (Thanks for serving up the pie Dean.) However, of 30+ common decoders I only discovered 4 decoders supported freeformatted Mp3 files even if the encoded bitrate is less than 320 kb/s. Only one decoder (WinAmp MAD) is capable of decoding files above 600 kb/s.

So, (other than me having to eat a big helping of humble pie) where does that leave us in the specific debate about boundary testing, and Pradeep’s question "As a tester have you ever seen a boundary?" To that, I shall adamantly reply "yes" there are specific boundary conditions in software. Some are easy to find, some are not so easy. A tester’s ability to correctly identify a boundary value are heavily influenced by his/her in-depth domain and ‘system’ knowledge. For example, using the knowledge of Mp3 encodings I have learned over the past 3 days let’s go back and review what tests I would design based on Pradeep’s original description of the audio decoder that played an Mp3 file within the range of 24 kb/s to 196 kb/s.

Since 196 kb/s is not a standard Mp3 encoding supported by ISO standards let’s assume the Mp3 player used either a Cdex, LAME, I3dec, or WinAmp MAD decoder. Using this as a reference, and some recently acquired domain knowledge I would design a set of initial tests using the following sample test data (files encoded with the specified criteria).

1. 23, 24, 25 kb/s – Specified minimum value and minimum -1, and minimum +1 values to analyze relational operators used to artificially constrain the encoding range to a low end of 24 kb/s.

2. 195, 196, 197 kb/s – Specified maximum value and maximum -1 and maximum + 1 values to analyze to analyze relational operators used to artificially constrain the encoding range to a high end of 196 kb/s.

3. 16 kb/s – this is the next ISO standard encoding bitrate below the specified minimum, so although the decoder does not support a file encoded at 23 kb/s (min -1 value) I would still want to check at the next lower standard value.

4. 224 kb/s – this is the next ISO standard encoding bitrate above the specified minimum (same reason as explained above.)

5. 32, 40, 48, 56, 64, 80, 96, 112, 128, 144 (see #6), 160, 176, 192 kb/s – these are the typical ISO standard Mp3 encodings within the specified range, so we should assume that all these must work properly because there is a high probability of decoding files using these bitrates. Since there are not many of them test each one.

6. 143, 144, 145 kb/s because the 144 kb/s bitrate seems to be an interesting value that "sticks out" more than others, and so I may also want to analyze the values around that particular value for any other anomalies

7. Generate several randomly encoded files in the following ranges (between 24 kb/s and 127 kb/s) (between 128 and 143 kb/s), and (between 145 kb/s and 196 kb/s) to gain confidence the decoder can decode non-standard encoded files within the specified range without having to test all 174 or so possibilities

These are not the only tests I would execute; however, they would be the first set of tests I would design and execute to make sure the code at least does what it is supposed to do. Any failure in the above cases means our basic program functionality has some serious flaws. Once I established the program does what it is supposed to do (including handling expected errors gracefully), then I would begin exploring other possibilities including rigorous falsification/negative testing.

Pradeep indicated a file encoded at 96 and 128 kb/s crashed the system. These are not boundaries conditions (unless the developer did something totally unreasonable), and unfortunately since we can only assume files encoded above and below 96 and 128 kb/s played correctly we will never really know the cause of this problem (unless Pradeep did some root cause analysis and will share those findings). However, a failure with 128 kb/s is really a red flag to me because this happens to be the most prevalent bitrate for encoding Mp3 files. As a tester I would really want to know why unit testing or build acceptance testing, etc. didn’t at least hit the most probable encoding format (the happy path) before throwing crap code over the wall for Pradeep to test.

I hope the reader takes away a few lessons from all this (besides the obvious one of not going off half cocked especially if you lack expertise in the specific context (e.g. Mp3 encodings)). For example,

• In-depth knowledge of the domain space (including the data set, how the data is encoded, and how the data set can and cannot be manipulated in code both correctly and incorrectly), industry standards, and how the domain space interacts with the system are critical for greater test effectiveness
• The less we know about the domain, the data set, the system interaction the less effective are the application of specific techniques to identify specific classes of defects
• Boundary testing is simply one technique (systematic procedure to solve one type of a complex problem). The boundary value analysis technique is designed to identify a specific class of defects involving incorrectly specified constant values, incorrect use of data types, or casting between data types, artificially constraining data types, and incorrect usage of relational operators. It is not effective to identify other classes of defects.
• Boundary conditions simply do not exist at the extreme physical ranges, there could also be multiple boundary conditions within the overall range. (A good example is the Unicode repertoire. The Unicode BMP spans from U+0000 to U+FFFF, but within this range there are several important boundaries one must take into consideration when using Unicode data depending on the purpose of the test and the application under test (e.g. Private use area, surrogate pair area, etc).)
• Understanding how to decompose the data set into equivalence class partition subsets exposes boundary conditions we might not otherwise consider
• There is a great deal of detail in the code that can expose interesting information for a tester
• When talking technology, be as specific and precise as possible to avoid ambiguity
• And perhaps most importantly, one technique or one approach to testing is not sufficient. As testers we must gather and learn to use a great variety of skills and knowledge and approach the problem from multiple perspectives to be most effective in our roles.

OK…now time to get some really bitter coffee to wash down that humble pie

Originally Published Tuesday, March 06, 2007

My previous post refuting a conjecture by Pradeep Soundarajan suggesting there are no boundary values in software was a bit harshly worded, and to him and the readers I apologize. Occasionally I get a little overzealous. I am sure Pradeep is a great guy, and I must say his reply to me on his blog was rather cordial given the situation. As I told Pradeep, email and blogs are a poor medium for expressing emotion. MichaelB cautioned me about this before, but my Type A personality sometimes takes over, and it is something I need to work on.

Anyway, although my critique of Pradeep’s conjecture was pretty ruthless, the analysis was accurate and the  boundary values Pradeep suggested for testing an Mp3 file are in fact not possible or probable even using the tools he referenced. I am not an expert on Mp3 encodings or decoding technology, but i did know that Mp3 files used standard encoding bit rate formats. A little investigation quickly revealed the bit rates for audio encoding are based on multiples of 8, and the first 32-bits of an Mp3 file contain header information including 4 bits to specify the bit rate index (Layer 1, Layer 2, or Layer 3) and the bit rates as outlined below. (Thanks to Wikipedia some of the specific data I used in my initial rebuttal was incorrect, and I put a single strikethrough that part of the sentence.)

Layer 1 – 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448
Layer 2 – 32, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, 384
Layer 3 – 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320

In his attempt to contest my argument Pradeep said, “Pooh! I am not sure why you don’t know that bit rates can be 33, 41, 57 or any number you want to generate. I recommend you to go through some multimedia test content generation tools like ffmpeg which gives a multimedia tester an edge to generate test content of his choice.”

So, not being an expert I took Pradeep’s suggestion “For those who don’t know tools like ffmpeg, it is impossible. I suggest you explore the boundary of your education on multimedia.” and went home and increased my understanding of Mp3 encodings and the ffmpeg toolset. As I read through some of the API references I found an interesting struct (illustrated below) for bit rate constants. (Now, I am thinking to myself…this is a clue! I am also thinking…hmmm…there might be some real boundary values here!)

   1: static const int sBitRates[2][3][15] =

   2: {

   3:   00085 { {  0, 32, 64, 96,128,160,192,224,256,288,320,352,384,416,448},

   4:   00086   {  0, 32, 48, 56, 64, 80, 96,112,128,160,192,224,256,320,384},

   5:   00087   {  0, 32, 40, 48, 56, 64, 80, 96,112,128,160,192,224,256,320}

   6:   00088

   7:         },

   8:   00089 { {  0, 32, 48, 56, 64, 80, 96,112,128,144,160,176,192,224,256},

   9:   00090   {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160},

  10:   00091   {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160}

  11:   00092   },

  12: // etc...

I also followed up by asking a co-worker who frequently works with ffmpeg to try to encode a Mp3 file with a bit rate of 57 kb/s. Interestingly enough, when he issued the command line parameters we got an error message indicating “Invalid Value.” (I can get a snapshot of the command window, but I really don’t think that is necessary.)

I am still not an expert on Mp3 encodings, but I am fairly certain that Mp3 file decoding algorithms are standardized across the industry. So, let’s just assume for a moment that we can encode a Mp3 file at 57 kb/s, and that file fails to play. Does it really matter? No, because industry hardware simply doesn’t support that encoding, and as long as the Mp3 player didn’t burst into flames there is no business case that would compel someone to try to make it work (at this time)? (I am not suggesting that we only test only “real-world” scenarios here, but I am suggesting that in-depth domain and system knowledge goes a long way in increasing the efficiency and effectiveness of our testing (and can lead to better identification boundary values)).

Now, perhaps I am still missing something, and as I expressed previously I am not an expert on Mp3 encodings, or the use of ffmpeg or other tools to encode Mp3 files. So, I have asked Pradeep to share his knowledge with me in this area and teach me how to encode an Mp3 file with a bit rate of 57 kb/s using a commonly used tool such as ffmpeg (there is no doubt someone can write a customized algorithm to do this), and to also let me know of a commercially available Mp3 player that will decode and play that file. (Because if it can be done I would like to learn how simply because I love to learn new things.)

Many people do assume that boundary testing is quite simple. The actual execution of boundary tests are in fact rather simple; however, discerning the boundary values in any complex software is not as simple as looking at some minimum and maximum values and trying one value below and above each boundary condition. Boundary testing is a systematic procedure to solve a specific type of complex problem (specifically the incorrect usage of data types or constant values, artificially constrained data types, and relational operators). Boundary value analysis doesn’t solve all problems, it is not the holy grail, and its efficacy relies on the testers ability to understand and decompose the data set effectively.The less the tester knows about the data and how the data is used by the program, the less effective they will be in the application of this technique.

I did not intend my previous post to be construed as a personal attack against Pradeep; I am sure he is a bright guy. But, I am challenging his assertion on boundary testing on its technical merit. I hope he replies here (or on his blog) with an example of how to encode an Mp3 file at 57 kb/s, and I will make sure it is posted (or linked) here because I am certainly curious. (I don’t really like the taste of humble pie, but I will eat it from time to time if it helps me learn.)

Originally Published Wednesday, October 25, 2006

Last week I went to StarWest as a presenter and as a track chair to introduce speakers. Being a track chair is wonderful because you get to interface more closely with other speakers. Anyway…one of the speakers I introduced was Jon Bach. Jon is a good public speaker, and I was pleasantly surprised that he was doing a talk on the allpairs testing technique (also known as pairwise or combinatorial analysis). I wish Jon dedicated a little more time to the specifics of the technique during his talk and was generally more aware of available tools and information  for folks to investigate further, but I think he successfully raised the general awareness and interest in pariwise testing as an effective testing technique among the audience.

Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 2¹¹ or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.

The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.

But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create "mixed orthogonal arrays" where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L₂₅ (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).

The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.

One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.

I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.

Originally Published Friday, August 11, 2006

Last year an ‘expert’ tester and consultant claimed test techniques were gimmicks. I don’t claim to be an expert tester, but I do consider myself to be a professional tester constantly striving to improve my skills and broaden my knowledge to make me a better tester and promote the discipline I have chosen as a career. So, I have learned that test techniques are not gimmicks, but are in fact effective systematic procedures to prove (or disprove) that specific attributes or capabilities of the software function as expected and/or conform to any applicable standards or guidelines. Test techniques also help focus testers towards designing highly effective test cases that target specific functional areas of the product, and also establish a solid foundation from which additional testing can progress.

Techniques are similar to tools in a professional’s toolbox. The professional tester knows what techniques to use in specific situations, and how to use them most effectively. Functional testing techniques such as boundary value analysis (boundary testing), equivalence class partitioning, combinatorial analysis (pair-wise testing), or state transition testing each serve very different purposes. These techniques systematically evaluate specific functional attributes of a software project, but they are not the only ‘tools’ a professional tester uses to ‘test’ software.

The greatest value of test techniques is their potential to instill a high level of confidence in testing coverage of specific functional areas with a fewer number of tests, and thus reduce the amount of redundant testing. However, the value or effectiveness of using any test technique ultimately depends on the individual tester’s ‘system’ and domain knowledge, and the ability to apply the applicable technique to the correct situation.

Over the next few weeks I plan to discuss how to use some of the various testing techniques and demonstrate why these are valuable tools for the professional tester.