I.M. Testy

Originally Published Wednesday, February 27, 2008

This morning I installed Vista SP1 onto my laptop. I was pretty excited about this release of Vista SP1 because it includes some pretty significant performance enhancements. But, as I was preparing to teach an internal course I came across a new boundary issue. I thought, how fitting this comes as I prepare to teach another class on systematic testing techniques (including boundary value analysis) that I find yet another classic boundary issue in Explorer’s list view (albeit the boundary condition is not readily apparent) . Interestingly enough, the previous boundary issue I found appears to be fixed with Vista’s SP1; however, I previously did not run across this issue which is also connected with how listview repaints itself after an event.

To reproduce this issue, open a folder with several dozen files in Explorer and select Views -> List and resize the Explorer window so there are several columns of files.

Select a file in the list and press the up or down arrow key so a dotted line appears around the file name

Resize the Explorer window so that the highlighted area is about 1 pixel away from the right most edge of the Explorer window as in the example below

Press the down (or up depending on which file you have selected) arrow key, and notice how the file highlight jumps to the next file in the list as expected.

Next, resize the the Explorer window so the dotted line touches the inner boundary of the window as in the example below

Now, press the down (or up depending on which file you have selected) and…

Notice, in the image above that just to the left of the frogger.def file the right most edge of the file highlight is visible, and also notice the file image has changed, and the scroll bar has jumped over to the next column.

Press the down (or up) arrow key again, and….

Seeing this behavior the first time made me think I was losing my mind! (OK…I probably am, but that is a completely different topic). And yes, reducing the size of the window beyond where the maximum boundary of the highlight window will also cause the same behavior, but the issue occurs when the width of the file highlight window >= the maximum x boundary size of the Explorer listview window.

The listview in Windows Vista has some very cool features, and overall I like Vista. What I dislike are some of the simple anomalies that could be easily exposed via more detailed, systematic testing approaches and analysis of the system under test rather than simply assuming more bodies banging on keyboards for some arbitrary period of time equates to better testing.

So, some things to consider when boundary testing…

• Not all boundary conditions are easily identified from the GUI by numbers, but all boundaries conditions have linear physical values that can be measured at some level
• Boundaries can change, similar to way the boundaries changed around Kosovo. Somebody moved the lines. The same thing is true in software, a boundary condition may change with human interaction, but a boundary condition is a linear physical range between a minimum and a maximum value (in this case the x coordinates of the Explorer changed with human interaction, but once focus moved away from the Explorer window the size of x established a fixed boundary (at least until a user again resized the window or the window is closed).
• Boundary conditions for one parameter are usually independent. For example, resizing the y-axis in the above scenario has no impact on this defect (unless of course the y-axis is large enough to accommodate all files in the listview without having to scroll horizontally). Boundary testing is based on the single fault assumption theory which states that a boundary issue is most likely to occur with independent parameters where the boundary variables for one parameter are analyzed while holding other parameters to nominal values. (Note: if we suspect that parameters are dependent or semi-coupled, then we should also perform combinatorial analysis testing.)
• There is a difference between boundary conditions (at least momentarily fixed, linear, measures) and threshold values. Threshold values can be altered by various influences. For example, in performance testing the point of degradation in performance can often be changed by several external influences such as increasing physical memory, cleaning and defragmenting the hard disk, modifying the software, etc.
• A detailed analysis of the system under test will reveal issues that other approaches to testing do not expose…that’s the pesticide paradox!

Originally Published Sunday, December 09, 2007

I just left Stockholm after spending a week there. That was my second visit to Stockholm and it is truly a remarkable city. I spoke at EuroStar which is the largest software testing conference in Europe, and had the opportunity to meet some old friends and colleagues as well as chat with many other speakers and delegates throughout the week. My friend Steve Allott invited me to meet with the senior managers of Know IT, which is the largest IT consulting company in Sweden where I discussed controlling test environments and techniques to generate real-world and random test data. This next week I will be in Denmark where I will be teaching at the Microsoft campus in Vedbaek.

While at an after-event gathering (a really great party thrown my the wonderful folks at Know IT AB) one evening last week I met a gentleman whom I had previously challenged regarding his comparison of the agile development model to the waterfall model. His response at the time was simply that most people in the industry were only familiar with the waterfall model, so that’s what he used as a comparison. At the gathering he approached and chided me by asking if I remembered giving him such a hard time in Dusseldorf. After some light-hearted banter I told him I only gave him such a ‘hard time’ because often we only see concepts compared to a completely opposite concept or approach. What I had asked at the conference was for him to compare agile approaches to other iterative development approaches. This evening he was better prepared for that conversation and he was also able to discuss the advantages and disadvantages of the agile approach in relation to various other approaches, as well as state when other approaches may be more relevant as compared to an agile approach. (Yes, believe it or not, some people still understand that agile is not, and should not be, the only way!)

Often when I hear people talk about exploratory testing (just to clarify again, I think everyone does and should do explorative type testing) it is often compared with scripted testing. But, I often ask myself "Is the testing problem really so simple that it can be solved from either an exploratory or scripted testing approach? Or, are there other approaches, methods, and techniques that I should consider when attempting to prove or disprove a hypothesis of a complex system?"

When I think of scripted testing I think of a test case (often poorly written) that is so prescriptive in its set of instructions that it leaves no ability on the part of the tester to deviate while still proving or disproving the hypothesis or purpose of the test. For example, a scripted test to me is exemplified by the following set of prescriptive steps:

1. Click Start button and select Run… menu item
2. Enter "iexplore" in the Open textbox and press the OK button
3. Enter http://www.google.com in the address bar and press Enter
4. Press ALT + q and enter "pickaxe" in the search field
5. Click the Google Search button
6. Verify the search results contain "Programming Ruby"

An automated version of this test in Ruby might look something like the following:

require ‘watir’
include Watir
test_site = ‘http://www.google.com’
ie = IE.new
ie.goto(test_site)
ie.text_field(:name, "q").set("pickaxe")
ie.button(:name, "btnG").click
if ie.contains_text("Programming Ruby")
   puts "Test Passed.
else
   puts "Test Failed!
end

Now, there may be times when I need a prescriptive set of steps in a scripted test to validate specific requirements, or I need absolute control over what is being tested and how it is being tested. There are other times I want to use an explorative approach to test and test design, such as when I am not familiar with a new undocumented or under-documented feature, or when I want to evaluate the system in ways which I have not previously considered.

But, I also know that between the spectrum of explorative type testing and purely scripted testing there are a wide variety of approaches to testing and test design that are extremely beneficial to the professional tester. When I talk about techniques or methods or approaches commonly used in software testing I always discuss the specific context in which they are useful and situations in which they are not applicable. I discuss strengths and weaknesses, and also the different types of information that can be revealed beyond simply the discovery of potential defects.

From my point of view, professional testing is perhaps the most challenging discipline in the software industry. The complexity of the software, and the virtual infinite number of tests that are possible surely require us to approach the situation from multiple approaches and not simply from an explorative approach or a scripted approach. It is certainly easy to bucket things for simplicity (e.g. if it isn’t exploratory it must be scripted). However, when we view a problem from only two, or one of two directions we are certainly more likely to miss important information (other than simply defects) that may be critical for our managers to better analyze risk and make better business decisions. A professional tester not only realizes that there are multiple approaches to a specific situation, but also understands that each approach has advantages and disadvantages and knows how to leverage those advantages in order to gather the appropriate information.

So, let’s break out of only viewing testing within the context of testing approaches (exploratory vs, scripted), and start understanding the contexts of the software under test (the specific set of facts or circumstances surrounding a particular event or situation) and let’s really learn about all the various approaches, techniques, and methods that are available to the professional tester and better understand when and how to best apply those ‘tools of our trade’ in the pursuit of our profession.

Originally Published Thursday, November 29, 2007

In the last post we decomposed the set of characters in the ANSI character set into valid and invalid class subsets for use in a base filename component on the Windows Xp operating system. The second part of the testing technique of equivalence class partitioning is to then use this information in tests that will adequately evaluate the functional capabilities of the base filename parameter used by COMDLG32.DLL in Windows Xp.

(Note: It is appropriate here to differentiate between a test case and a test. A test case is the hypothesis or assertion that I am either trying to prove or disprove. A test is the process or procedure I am using to prove or disprove my hypothesis. In other words, a test case in this example might be "Validate a base filename can be saved using valid characters in the ANSI 1252 character set." In order to prove or disprove this hypothesis, I will execute multiple tests using the various subsets outlined in the previous post, and as described below. If any test fails, the test case fails.)

There are basically 2 procedures for defining the tests that use the data from an equivalence class table. Paul Jorgensen describes a procedure for robust equivalence class testing in which each valid class subset is tested individually. Glenford Myers suggests a procedure in which valid class subsets are combined in a test until all valid class subsets have been covered. (Jorgensen refers to this as weak equivalence class testing.) Both authors agree that subsets in the invalid subsets must be tested in a way that only one parameter is invalid and any other parameters use valid variables. Using Myers’ approach (or what Jorgensen describes as weak equivalence class testing) the tests for the base filename parameter are illustrated in the table below.

[Edited 11/30] I apparently made too many assumptions about conducting these tests, so I must clarify some of those assumptions.

• Any time a key is pressed there is essentially a state change. The primary state change we are concerned with for this particular functional test is whether or not a file is saved to the file system. However, in cases of invalid input there are other state changes that may be interesting. In most well designed Windows applications an erroneous input in an input control will be highlighted (usually after an error message is displayed). This is indeed an expected state change. The only time no noticeable state change occurs (there is still a state change because the machine is processing WM_Key messages) is with Test #5. But, another noticeable state change (the one primarily identified in the table below) is the state change to the list view control.

• There are 3 ways to effectively affect the Save button. One way is to press enter, another is to press the Alt + s key mnemonic, and thirdly to left mouse click the button control (including simulating any of those actions via automation). In some cases the visible state changes may vary; however, the purpose of these tests it so verify the existence of a file name in the file system for valid cases, and the non-existence and no abhorrent side effects in the case of members of the invalid class subsets. So, regardless of the visible state changes, the tester can use any of these procedures to affect the Save button.

• Application of techniques (systematic procedures) at a granular level is very different than simply trying things to see what happens, exploratory testing, guessing, or wild speculation. I made the assumption that readers are familiar with designing and executing atomic functional tests at a granular level in which certain variables are controlled. For example, once we enter an invalid string in the file name text box and press the save button the Save As… dialog is now dirty; meaning that the next input could produce an error but I will not know if the error is a result of an error with the base filename parameter, or with the Save As… dialog state. Equivalence class partitioning is a low level functional test of a single parameter, so in order to test the base filename parameter the tester (or automated test) should minimally close and instantiate the Save As… dialog on each test.

Reducing tests while increasing breadth of coverage

The total number of possible tests of valid string combinations for just the base filename parameter using only characters within the ANSI 1252 character set is 214²⁵¹ + 214²⁵⁰ + 214²⁴⁹ + …214² + 214¹. This number of tests, of course, is a virtual impossibility, so by employing the equivalence class partitioning testing technique we are able to systematically produce a minimum baseline set of tests that has a high probability of both proving or disproving our hypothesis or test purpose, as well as providing great variability in the test data to increase breadth of data coverage. The minimum possible number of valid tests determined by combining at least one element from each valid class subset is only 4 tests. But, let’s look at each valid test a little more closely.

Test #3 is probably a red-herring! This is only an invalid filename on Windows NT 4.0 and below. So, if your application is ported from that time frame, and you are using a custom function for your file save functionality rather than Windows APIs then you might consider running this test once. If if passes, you can probably forget running this again ever again on that product. Test #1 evaluates the literal strings in valid subset V1 can be listed in an array or enumeration and one element can be selected at random throughout the development lifecycle, or each literal string can be tested once and the probability of failure in a later build is most likely less than .001%. Test #4 is also a test that probably doesn’t require a great deal of retesting of various combinations of elements from subsets V2, V3, V4, V5 , & V7. Elements from the valid class subsets described in Test #2 are most interesting and this is the test that we will probably want to automate and run repeated throughout the development lifecycle because it provides great breadth of coverage. Remember this is the minimum number of valid tests. What isn’t covered in this set is common or ‘real-world’ data sets which we would certainly want to include. Additionally, Test #2 relies on at least one element from each indicated subset. We might also want to consider additional tests that focus on subsets V4 and V5 only. Also, we might also consider testing valid class subset V6 as a special case if we suspected a private function excluded code point values that were not assigned character glyphs. However, if these 4 valid tests pass the probability of failure of any combination of these data sets used in this parameter is minimal. Random selection of elements for Test #2 (and possibly Test #4) may slightly increase the probability of exposing a defect in the base filename parsing routine. Tests #5 through #26 are tests for invalid filenames, or in the case of Test #19 and #20 where the expected result is to truncate the leading or trailing space character.

This of course only analyzes (or tests) the base filename parameter and assumes a nominal valid 3-letter extension, valid filenames do not preexist on the file system, and within the specific context described in the first post. If the context changes, (e.g. this example does not apply to Unix platforms, or Windows Vista, or other parameters) then this set of tests (assuming we would include at least Tests #2, and #5 through #26 as part of our regression suite ran on each new build) would provide a high level of confidence in the functional capability of this specific parameter.

Next, we would also decompose the data for the file extension parameter (which is different than the base filename parameter) because in the File name textbox we can enter either the base filename or the base filename and an extension, and once we verify the functionality of the base filename component, we can then proceed to the next step in analyzing the functionality of the arguments passed to the File name textbox parameter which we shall examine at a later date.

It is important to note the this and any other techniques are simply systematic procedures designed to help us wrap our minds around complex problems. They are not the only approach to testing (only a fool would advocate a single approach to testing), but when used in conjunction with various other techniques, methods, and approaches EQ can help to establish a strong foundation of confidence in low level operations. Of course, as has been previously stated, the limiting factor of this and other functional techniques is the knowledge of the professional tester to think critically and rationally analyze the "overall system" within the appropriate context.

Originally Published Thursday, November 15, 2007

Again, I am remiss in my postings…too many irons in the fire these days. Two weeks ago, I posted a challenge to decompose a set of character data (The ANSI Latin 1 Character Set) into valid and invalid equivalence class subsets in order to test the base filename parameter of a filename passed to COMDLG32.DLL on the Windows Xp platform from the user interface using the File Save As… dialog of Notepad.

As illustrated below the filename on a Windows platform is composed of two separate parameters. Although the file name parameter of the Save As… dialog will accept a base filename, a base filename with an extension, or a path with a filename with or without an extension, the purpose of the challenge was to decompose the limited set of characters into equivalence class subsets for the base filename component only (the part outlined with green). (Of course, complete testing will include testing with and without extensions, but let’s first focus on building a foundation of tests to adequately evaluate the base filename parameter first, then we can expand our tests from there to include extensions.)

As suggested in the earlier post, in order to adequately decompose this set of data within the defined, real world context (and not in alternate philosophical alternate universes) a professional tester would need to understand programming concepts, file naming conventions on a Windows platform, Windows Xp file system, basic default character encoding on the Windows Xp operating system (Unicode), some historical knowledge of the FAT file system, and even a bit of knowledge of the PC/AT architecture. The following is a table illustrating how I would decompose the data set into equivalence class subsets.

Valid Class Subsets

• V₁ – escape sequence literal strings
        (STX, SOT, ETX, EOT, ENQ, ACK, BEL,
        BS, HT, LF, VT, FF, CR, SO, SI, DLE,
        DC1, DC2, DC3, DC4, NAK, SYN, ETB,
        CAN, EM, SUB, ESC, FS, GS, RS, US,
        DEL)
• V₂ – space character (0×20) (but not as
         only, first, or last character in the 
         base file name)
• V₃ – period character (0x2E) (but not as
         only character in the base file name)
• V₄ – ASCII characters
         punctuation (0×21, 0×23 – 0×29, 0x2B – 0x2D, 0x3B, 0x3D, 0×40, 0x5B, 0x5D, - 
         0×60, 0x7B, 0x7D, 0x7E),
         numbers (0×30 – 0×39)
         alpha (0×41 – 0x5A, 0×61 – 0x7A)
• V₅ – Ox80 through 0xFF
• V₆ – 0×81, 0x8D, 0x8F, 0×90, 0x9D
• V₇ – Component length between 1 – 251
         characters (assuming a default 3-
         letter extension and a maximum path
         length of 260 characters)
• V₈ – Literal string CLOCK$ (NT 4.0 code 
         base)
• V₉ – a valid string with a reserved
           character 0×22 as the first and
           last character in the string

Invalid Class Subsets

• I₁ – control codes 
       (Ctrl + @, Ctrl + B, Ctrl + C, Ctrl + ], Ctrl + N, 
       etc.)
• I₂ – escape sequence literal string NUL
• I₃ – Tab character
• I₄ – reserved words
        (LPT1 – LPT4, COM1 – COM4, CON, PRN, AUX,
        etc.)
• I₅ – reserved words
         (LPT5 – LPT9, COM5 – COM9)
• I₆ – reserved characters (/ : < > | )
         (0x2F, 0x3A, 0x3C, 0x3E, 0x7C) by
        themselves or as part of a string of 
        characters
• I₇ – reserved character 0×22 as the only
        character or > 2 characters in the string
• I₈ – a string composed of > 1 reserved character
        0x5C
• I₉ – a string containing only 2 reserved
         characters 0×22
• I₁₀ – period character (0x2E) as only
         character in a string
• I₁₁ – two period characters (0x2E) as only
         characters in a string
• I₁₂ – > 2 period characters (0x2E) as only
         characters in a string
• I₁₃ – reserved character 0x5C as the only 
           character in the string
• I₁₄ – space character (0×20) as only character in 
         a string
• I₁₅ – space character (0×20) as first character in
          a string
• I₁₆ – space character (0×20) as last character in a
          string
• I₁₇ – reserved characters (* ?)  (0x2A, 0x3F)
• I₁₈ – a string of valid characters that contains at least one reserved characters (* ?)  (0x2A, 0x3F)
• I₁₉ – a string of valid characters that contains at
          least one reserved character 0x5C but not
          in the first position
• I₂₀ – string > 251 characters
• I₂₁ – character case sensitivity
• I₂₂ – empty

Discussion of valid equivalence class subsets

• Valid subset V₁ is composed of the literal strings for control characters (or escape sequences) between 0×01 and 0x1F, and including 0x7F. The literal strings for control characters may cause problems under various configurations or unique situations. The book How to Break Software: A Practical Guide to Testing goes into great detail explaining various fault models for these various character values. The literal strings in this subset should be tested as the base filename component and possibly in a separate test as an extension component. However, on the Windows platform the probability of one particular string in this subset behaving or being handled differently than any of the others is very low negating the need to test every string in this subset; although the overhead to test all would be minimal and once complete would not likely require repeated testing of all literal strings in this subset during a project cycle.
• Valid subset V₂ provides guidance on the use of the space character in valid filenames. On the Windows operating system a space character (0×20) is allowed in a base filename, but is not permitted as the only character as a file name. Typical behavior on the Windows platform also truncates the space character if it is used as the first character of a base filename or the last character of a base filename. However, if the extension is appended to the base filename in the Filename edit control on the Save or Save As… dialog a space character can be the last character in the base filename. Also note that a space character by itself or as the first character in a filename is acceptable on a UNIX based operating system. Also, although we can force the Windows platform to save a file name with only a space character by typing “ .txt” (including the quotes) in the Filename edit control on the Save/Save As… dialog this practice is not typical of reasonable Windows users’ expectations.
• Valid subset V₃ is the period character (0x2E) which is allowed in a base filename, but it is not a valid filename if it is the only character in the base filename (see Invalid subset for the period character).
• Valid subset V₄ is composed of ‘printable’ ASCII characters that are valid ASCII characters in a Windows filename. The subset includes punctuation characters, numeric characters, and alpha characters. We could also decompose this subset further into additional subsets including valid punctuation characters, numbers, upper case, and lower case characters if we wanted to ensure that we test at least one element from the superset at least once.
• Valid subset V₅ is the set of character code points between 0×80 and 0xFF.
• Valid subset V₆ is a superset of subset V₅ and are separated only because they are code points that do not have character glyphs assigned to those code point values. These would be interesting especially if we needed to test filenames for backwards compatibility on Windows 9x platforms.
• Valid subset V₇ is the minimum and maximum component length assuming the filename is saved in the root directory (C:\). 
• Valid subset V₈ is a probably a red-herring. On the NT 4 platform the string CLOCK$ was a reserved word. On an older application first created for the Windows NT 4 platform that does not use the system Save/Save As dialog we might want to test this string just to make sure the developer did not hard code the string in an error handling routine.
• Valid subset V₉ is an interesting case because this invalid reserved character (0×22) is handled differently when used in first and last character positions of a base filename. When used in the first and last positions of a base filename the characters are truncated and if the remaining string is valid the filename is saved. If only one 0×22 character is used, or if two or more 0×22 characters are used in a string other than the first and last character positions the result will be an error message.

Discussion of invalid equivalence class subsets

• Invalid subset I₁ consists of the control code inputs for escape sequences in the range of 0×01 through 0x1F, and also includes 0x7F. Pressing the control key (CTRL) and any of the control codes keys will cause a system beep.
• Invalid subset I₂ is the literal string “nul”. Nul is a reserved word but could be processed differently than other reserved words on the Windows platform because it is also used in many coding languages as a character for string termination.
• Invalid subset I₃ is the tab character which can be copied and pasted into the Filename textbox control. Pasting a tab into the and pressing the save button will generate an error message.
• The invalid subset I₄ includes literal strings for reserved device names on the PC/AT machine and the Windows platform. Using any string in this subset result in an error message indicating the filename is a reserved device name.
• Invalid subset I₅ also includes reserved device names for LPT5 – LPT9 and COM5 – COM9. However these must be separated into a unique subset because using these specific device names as the base filename on the Windows Xp operating system result in an error message indicating the filename is invalid.
• Invalid subsets I₆, I₇, and I₈, include reserved characters on a Windows platform. When characters in this subset are used by themselves or in any position in a string of characters the result is an error message indicating the above file name is invalid.
• Invalid subsets I₉, I₁₀, I₁₃, also include reserved characters and the space and period characters. When these subsets are tested as defined no error message displayed and focus is restored to the File name control on the Save/Save As… dialog.
• Invalid subsets I₁₁, I₁₂, also include the reserved character (0x2E) as 2 characters in the string and greater than 2 characters in a string. The state machine changes are different.
• Invalid subsets I₁₅ and I₁₆ define the space character when used in the first or last character position of a string. These are placed in the invalid class because Windows normal behavior is to truncate a leading or trailing space character in a file name. If the leading or trailing space character was not truncated and saved as part of the file name on a Windows platform that would constitute a defect.
• Invalid subset I₁₇ and I₁₈ contains two additional reserved characters; the asterisk and the question mark (0x2A and 0x3F respectively). If these characters are used by themselves or as a character in a string of other valid characters a file will not be saved, and no error message will occur. However, the state of the Save/Save As… dialog does change. If the default file type is .txt and there are text files displayed in the Folder View control on the Save As… dialog the files with the .txt extension will be removed after the Save button is depressed. If the default file type is All files then all files will be removed from the Folder View control on the Save As… dialog after the Save button is depressed.
• Invalid subset I₁₉ is a string of valid characters which contains at least backslash character except as the lead character in the string. (Of course, this assumes the string is random and the position of the backslash character in the string is not in a position which would resolve to a valid path.) The backslash character is a reserved character for use as a path delimiter in the file system. An error message will appear indicating the path is invalid.
• Invalid subset I₂₀ tests for extremely long base file name lengths of greater than 252 characters. Note that an interesting anomaly occurs with string lengths. A base file name string length which tests the boundaries of 252 or 253 valid characters will cause an error message to display indicating the file name is invalid. However, a base file name string length of 254 or 255 characters will actually get saved as file name but is not associated with any file type. Any base file name string longer than 255 characters again instantiates an error message.
• Invalid subset I₂₁ describes the tests for case sensitivity. The Windows platform does not consider character case of characters that have an upper case and a lower case representation. For example, a file name with a lower case Latin character ‘a’ is considered the same as a file name with the upper case Latin character ‘A’.
• Invalid subset I₂₂ is, of course, an empty string

Of course, this is a partial list of the complete data set since the filename on a Windows Xp operating system can be any valid Unicode value of which there are several thousand character code points, including surrogate pair characters.

The first and by far the most complex step in the application of the functional technique of equivalence class partitioning is data decomposition. This requires an incredible amount of knowledge about the system. Data decomposition is an exercise in modeling data. The less one understands the data set, or the system under test the greater the probability of missing something. Next week we will analyze the equivalence class subsets to define are baseline set of tests to evaluate the base filename component.

Originally Published Wednesday, October 31, 2007

Wow…where does the time go? I was remiss last week in posting, and it has been a month since I posted about equivalence class partitioning. So, let’s get back to it shall we?

Equivalence class partitioning (ECP) is a functional testing technique useful in either black box or white box test design. A technique is a systematic approach to help solve a complex problem. Techniques are not silver-bullets, but they are a logical and analytical approach to problem solving that heavily draws upon the tester’s cognitive abilities (having a basis in or reducible to empirical factual knowledge) as opposed to random guessing (or little men running about inside one’s head triggering turbid thoughts). Contrary to popular misconceptions the application of ECP is not a rote, brain-dead activity. The ECP technique requires in-depth knowledge of the data set (data type, encoding method, etc), the programming language used in the implementation, the algorithm structure, the operating environment, protocols, and even the hardware platform may impact how the data for a particular parameter might be decomposed. The effectiveness of the application of this technique solely lies in the testers ability to adequately decompose the data set for a given parameter into subsets in which any element from a specific subset would produce the same result as any other element from that subset.

Essentially the ECP technique analyzes input or output variable data for each specific parameter and decomposes the data into discrete valid and invalid class subsets. See my article in Software Testing & Performance magazine for a more in-depth understanding of the equivalence class partitioning technique. The article uses a simple next date program as an example because working with integer type data within a limited range is usually a good introduction to this complex technique.

A frequent reader of this blog suggested he often sees this technique applied to simple parameters limited to number type inputs, but would like to understand how this is applied to a more complex parameter that takes a string input. Interestingly enough, one of the exercises I designed in our internal training (and also used in the workshop I present regularly at the Software Testing and Performance conferences) deals specifically with string input. Let me outline the exercise.

Your goal is to adequately decompose the set of data (the ANSI Latin 1 character set) into valid and invalid equivalent class subsets to evaluate the behavior of the base filename parameter passed to COMDLG32.DLL’s file save functionality from a user’s perspective (GUI) on the Windows Xp operating system with an NTFS file system on a PC/AT platform.

NOTES:

• On the Windows environment the base filename parameter is separate and not interdependent on the extension parameter. (Some frequent readers may remember I explained this point previously to an individual who mistakenly assumed the base filename parameter and extension parameter were interdependent on the Windows environment.) So, remember, the purpose of this exercise is to focus initially on the base filename parameter (don’t worry about the extension parameter…yet).
• To expose the user perspective of the file save functionality of COMDLG32 we will use Notepad.exe, and select the File -> Save As… menu item to instantiate the Save As… dialog.
• Characters greater than 0x7F can be entered using the numeric input method by holding down the ALT key and pressing number keypad keys equivalent to the characters decimal value. For example to enter the € character press and hold the ALT key and press the 0 1 2 8 keys on the numeric keypad. Release the ALT key and the character is displayed. (You can also use my Babel tool to generate specific characters by inputting the Unicode values for the characters in the Custom range groupbox and copying them into the filename control.)
• Not all characters can be entered using the numeric input method. For example, control sequences cannot be directly entered from the keyboard using this method (or by holding down the control key).
• In order to succeed, one must understand
  • Common programming concepts (especially for the C family of languages)
  • Windows File naming conventions
  • Windows Xp, file I/O APIs
  • Windows Xp NTFS file system
  • Knowledge of basic character encoding
  • Historical knowledge of FAT file systems

Give it a try…next week I will offer a solution that you can compare your own results against.

Originally Published Friday, October 19, 2007

A few weeks ago a read an interesting post on SQA Forums about exploratory testing. It was interesting not because there was anything ‘new’ to learn about ‘exploratory’ testing; but because it offered a compelling counter-argument to ad hoc testing. It is also a good read because it differentiates exploratory testing from ad hoc testing in a surprising revelation.

I, perhaps like many of you, have always assumed ad hoc testing to imply an unplanned, random approach to testing. In our internal defect tracking system at Microsoft two entries for the How Found category included "Ad hoc (directed)" and "Ad hoc (undirected)" and I once thought…’How can random testing be directed?’ Even Lee Copeland writes in his excellent book A Practitioner’s Guide to Software Test Design,  "…ad hoc testing which (by my definition, although others may disagree) often denotes sloppy, careless, unfocused, random and unskilled testing."

But, a Moderator on the SQA Forums site pointed out the actual definition of ad hoc (as an adjective – because the context of its use is important and in this context ad hoc describes the type of testing being performed) is "concerned or dealing with a specific subject, purpose, or end" and "improvised and often impromptu."

In sharp contrast to it’s commonly assumed connotation, the denotative implication of ‘ad hoc testing’ might be better interpreted as "improvised testing dealing with a specific subject or a clear purpose without previous preparation."

Now let’s contrast the denotative implication of ad hoc (or a bug bash) testing to exploratory testing. James Bach defines exploratory testing as "simultaneous learning, test design, and test execution." So, I interpret this to mean that a tester thinks about what to do next based on his/her cognitive abilities to apply their knowledge of testing techniques or methods as well as the domain and system space, and then performs or executes an action (or test) while learning about the capabilities or attributes of the application under test (without having a specific goal or purpose in mind other than perhaps the hope of finding a defect.)

When comparing the bastardized connotative assumption of ad hoc testing with exploratory testing I agree with the SQAForums moderator mentioned above who also stated that "…trying to equate exploratory testing to ad hoc testing is incorrect" and that "They [ad hoc testing and exploratory testing] are really on the opposite ends of the spectrum." But I also don’t think that exploratory testing is equitable to the denotative inference of ad hoc testing. Unless of course exploratory testing is largely improvised, doesn’t require previous preparation and does have a purpose or deal with a specific subject; in which case exploratory testing would literally be the same as ad hoc testing (based on the denotation of that adjectival phrase in the context of software testing)?

(BTW…antonyms of the word improvised include planned and predetermined, and we all know that exploratory testing is not planned or predetermined.)

Now, I don’t think for a moment that this revelation about the denotation of the word ad hoc is going to change the connotative implication of the phrase ad hoc testing. I also don’t think some people will ever change their opinion about exploratory testing as the magic snake oil or holy grail of testing. But this post in SQAForums made me think about the ridiculousness of the whole ad hoc/exploratory testing debate a bit. So, to expunge the emotion and religious rebuttals between ad hoc and exploratory testing, and to perhaps expose some common ground among testers I suggest we simply start using the phrase improvised testing to refer to any testing performed without a pre-defined test case with the intent of learning and/or evaluating the attributes and capabilities of a software project and exposing defects. Improvised testing…yeah…that’s it! No, wait. On second thought the word improvised may also carry negative connotations, so maybe we can call it extemporaneous testing, or maybe autoschediastical testing. Now, that sounds pretty damn impressive and sexy! Way better sounding than just simply saying testing. Who’s with me?

(Oh..we do (and have for many years) perform ‘directed ad hoc testing’ all the time at Microsoft…but we lovingly refer to it as a bug bash! A bug bash is when we ‘direct’ the testing effort to focus on a specific feature area or type of testing such as security or globalization for a given period of time. A bug bash involves executing improvised tests with a goal or purpose of exposing defects in a feature area or defects of a particular type or classification often with minimal preparation.)

Originally Published Monday, October 08, 2007

I previously discussed various types of defects exposed via application of the boundary value analysis testing technique including a repaint problem, a casting problem, and a wrapping problem. While the minimum and maximum physical linear boundaries of a parameter are often easy to identify, it is surprisingly more difficult to identify boundary conditions within the minimum and maximum range especially if the tester does not adequately decompose the data. This week I will discuss another boundary defect that is often hidden below the user interface, but could be exposed using boundary value analysis testing at the unit level.

Loops are common structures in software, and (depending on the programming language) are susceptible to boundary defects. Boundary value analysis of a loop structure involves (at a minimum) bypassing the loop, iterating through the loop one time, iterating through the loop 2 times, iterating through the loop the maximum number of times and one minus the maximum number of times, and finally trying to exceed the maximum number of iterations through a loop structure by one time. (This is the min -1, min, min +1, max -1, max, and max +1 analysis of the boundary conditions.)

For example, the following method counts the number of characters in a string (actually, it counts the number of Unicode character code points in a string). To boundary test this method we would need to bypass the loop once by passing an argument of an empty string (minimum -1), then a string of one character "a" (minimum), and a string of 2 characters "ab" (minimum +1). Next, we would test the maximum range with a string of 2,147,483,646 characters (maximum -1), 2,147,483,647 characters (maximum) and 2,147,483,648 characters (maximum +1). The ToCharArray method will copy a maximum number of Unicode characters from a string to a character array equal to a signed 32-bit int type. So, passing this method a string of 1,073,741,824 Unicode surrogate pair characters the actual number of Unicode characters will be 2,147,483,648 which will throw the out of range exception. (This happens to be a common error. Many developers assume one character == one byte or one UTF-16 Unicode code point value.)

   1: private static int GetCharacterCount(string myString)

   2: {

   3:     try

   4:     {

   5:         char[] cArray = myString.ToCharArray();

   6:         int index = 0;

   7:         while (index < cArray.Length)

   8:         {

   9:             index++;

  10:         }

  11:  

  12:         return index;

  13:     }

  14:  

  15:     catch (ArgumentOutOfRangeException)

  16:     {

  17:         throw;

  18:     }

  19: }

However, it is sometimes difficult to identify the boundaries of looping structures unless the tester is familiar with the programming language and/or data types. In the above example, if the tester is not aware of Unicode encoding patterns (especially surrogate pair encoding) and simply tests the physical extreme boundary conditions using only ASCII characters the method will appear to return the correct number of characters in a string up to and including the maximum length of 2,147,483,647 characters. But, passing a string of 2,147,483,647 characters in which even one character in that string is a surrogate pair will cause the ToCharArray method to throw the out of range exception.

Occasionally it may be difficult to even identify looping structures, especially when designing tests from only a black box test design approach. For example, in Window Xp a known defect appeared to allow a device name (LPT1, CON, etc.) as the base file name if the extension was appended to the base filename component in the Filename edit control (I’ll talk more about this defect  later.) A Windows Xp patch attempted to correct this defect; however classic boundary analysis testing easily revealed the defect was only partially fixed as illustrated in the steps below.

1. Launch Notepad
2. Select the File -> Save menu item.
3. In the Filename edit control on the Save dialog enter "LPT1.txt" (without the quotes).
4. Press the Save button
5. Press the Yes button on the error dialog that states the file already exists and asks "Do you want to replace it?’ As illustrated below,

If the patch/update is applied to the system Notepad will return an error message indicating it cannot create the file as illustrated below.

1. Next, press the OK button on the error dialog
2. Repeat steps 2 through 5 above.

Now, notice instead of an error message the Window title of Notepad has changed from Untitled – Notepad to LPT1 – Notepad. But, this is a reserved device name, so how can we save a file named LPT1.txt to the Windows file system? The answer is we cannot! Although the application title reads LPT1 – Notepad a file named LPT1.txt does not exist on the file system. This essentially constitutes a data loss defect because it appears to the user that they saved a file named LPT1.txt. (Yes, I am aware of the other bugs associated with reserved device names as well and shall write about them in the future.)

Now, some of you may ask how I knew to test for a looping structure with the Save dialog? Quite simply; I use a technique I refer to as the deja vu heuristic anytime I encounter an error dialog. (A heuristic is a commonsense rule (or set of rules) intended to increase the probability of solving some problem.)  Anytime I encounter an error dialog I repeat exactly the same set of steps to make sure I get the same exact error dialog. Error handling routines often employ loops and are often prone to defects especially if some variable is initialized inside the loop structure. The deja vu heuristic is designed to analyze the minimum boundary of an error handling routine that employs a loop. The minimum – 1 value is not executing the error path, the minimum boundary condition is executing the path that instantiates the error dialog, and the minimum +1 value is repeating the same steps to execute the same path (or not in case of a defect).  In fact, anytime I execute the same exact steps of an error path and get a different result the underlying architecture of the code is suspect and bound to contain one or more defects.

Looping structures are another common cause of boundary class defects, and this is clearly a case where visibility into the code and in-depth knowledge of data types is advantageous for the professional tester.

Originally Published Sunday, September 30, 2007

I have been teaching formal testing techniques for several years at Microsoft and University of Washington Extension. Techniques are systematic procedures to help solve a complex problem. A technique does not find all types problems; techniques are generally very good at finding very specific classes of defects. But, the usefulness or effectiveness of any particular technique relies on the in-depth system and domain knowledge of the tester, and in the tester’s skill and ability to apply the correct technique to a problem space.

One of those techniques that I teach is equivalence class partitioning (ECP). On the surface ECP appears rather simple, but the effectiveness in the application of this technique relies on the tester’s ability to decompose input and output variables into discrete subsets of valid and invalid classes.

Over-generalization of the data reduces the number of subsets in a specific class and increases the likelihood of under-testing. Hyper-decomposition of the variables may increase the number of subsets in a specific class without providing additional benefit (although it may potentially increase the number of tests). Based on this observation I have proposed the following hypothesis or Bj’s Theory of Equivalence Class Partitioning Data Decomposition: Over-generalization of data reduces the baseline tests and increases the probability of missing errors or generating false positives. Hyper-analysis of data increases the probability of redundancy and reduces the overall effectiveness of each test.

ECP is not some rote activity, but in order for it to be most effective the tester requires a lot of in-depth knowledge about the overall system (hardware, operating system, environment, etc), the domain (client program, programming language used to develop the application, API interfaces, etc.), and the data sets. Limited knowledge or inability to adequately analyze the data based on the system and domain spaces in one or more of these areas is an impediment to the effective application of this technique. I suspect those who fail to perform an in-depth analysis of the system, domain, and data interaction leads some people disregard this technique, or become hyper-critical of its usefulness.

In this months issue of Software Test & Performance magazine I discuss the theory and application of the equivalence class partitioning technique as a viable and extremely useful tool used by professional testers. The magazine is free to download, and the article goes into great depth explaining when and how to apply the technique of equivalence class partitioning. Let me know if you have any questions, and I would appreciate your comments or feedback.

Originally Published Friday, September 14, 2007

I have never been really good at math. Sure I understand basic formulas, but I rely on a calculator when I run out of fingers and toes. I am envious of people who can look at a hexadecimal or octal value and convert it to an integer value in their heads without a second thought. But, I do know that when I multiply  1,073,741,824 * 1,073,741,824 the result should not equal 0! No, this isn’t some perverse bug in Calculator (calc.exe), but this sort of magic math sometimes manifests itself in the outputs of various calculations. (And there is a lot of calculating going on in even simple software programs.)

Familiarity with data types and specifically the physical linear boundaries of data types can help expose defects or explain unexpected errors encountered while testing, especially when evaluating output variables. For example, the physical linear boundary of a type int is -2,147,483,648 through +2,147,483,647 because this data type is a signed 32-bit (2³²) integer value. A value greater than +2,147,483,647  input into a control that only accepts a signed integer value will throw an overflow exception error. But, what happens when we have 2 input parameters that accept type int and the result of the calculation is greater than the maximum bounded value of the data type? Well, the answer to that questions depends on the data type used to store the output. For example, we should expect the result of the variable z below to be  4,294,967,294.

            int x = 2147483647;
            int y = 2147483647;
            int z = x + y;

However, the result is actually -2! How did that happen? Since the maximum value of a signed int is 2,147,483,647, when y is added to x the number simply wraps itself within the physical bounds of the data type. If we added 1 to 2,147,483,647 the int value would equal -2,147,483,648. (In the above example, the number keeps counting positively to eventually equal -2). Hopefully, it is easy to understand why wrapping can be a bad thing! I am thinking some billionaire would be pretty upset if his/her bank balance was $2,147,483,647 and he/she made a deposit of let’s say $10,000 on Monday, then wrote a check for $5 on Wednesday and the check bounced because of a negative balance of -2,147,473,649!

Historical failure indicators suggest traditional boundary type defects have a high rate of occurance any time complex calculations act on linear bounded input or output values. This doesn’t imply that "most bugs occur at boundaries," (in fact that is quite a foolish notion). It simply means that empirical evidence has consistently proven traditional boundary defects are likely to occur at the edges of bounded linear variables, especially involving the output variables of complex calculations.

A real life example of a wrapping type bug can be found in Word 2007; although it is not quite as obvious as the above example. In the below example, the string is indented 5.9" to the left.

Figure 1.

If we increase the size of the indentation to 6.0" the last character of the string wraps to the next line in the document, and this ‘wrapping’ is also displayed in the Preview window as illustrated below.Figure 2.

If we increase the indentation using the numeric up/down control buttons to 6.1" or larger we get an error message indicating the indent size is too large even though the Preview window illustrates the ‘g’ character wrapping to the next line! (In fact, if we input a size of 6.001" or larger we get the error message.)

 Figure 3.

Hmm…OK…so it will wrap one character but not 2 or more? That’s a little puzzling!!! (And yes, I know that I can append additional characters to the end of the string after wrapping in Figure 2., but that is not really relevant to the problem at hand.) Puzzling for sure, but at least the program stopped us at some point it calculated to be "too large,"and didn’t continue to wrap to a negative value as it did in Word 2003!

But, one thing I really dislike about software is when it does something, and then gives me an error message that makes it seem like I did something wrong. For example, if we reset the indentation back to 6.0" as illustrated in Figure 1, then highlight the string, and then click the Bullets button in the tool bar the string wraps to a vertical column of characters as illustrated in Figure 4. Next, we bring up the context menu, select Paragraph menu item and notice the software set the Left indentation to 6.15". So, although we couldn’t manually input any number greater than 6.001" I am thinking if the software did it own its own, then it must be OK…right? But, now we push the OK button on the Paragraph dialog and we get an error message that implies we did something wrong! How can that be…we didn’t change the indentation value, the software set the incorrect value? (Also notice the Preview pane doesn’t even look close to what is happening…I guess that’s what one would call WYSIAWYG or what you see is almost what you get.)

Figure 4.

I understand that word wrapping and word breaking algorithms are really hard especially given the complexity of the interactions between font types, kerning, bullets and numbering styles, paper width, margins, and numerous other variables. This is just used as an example of a traditional output type boundary defect caused by incorrect calculations. This example is used here to illustrate how this category of defects can manifest various symptoms due to a single root cause, and to also provide insight to you, the professional tester, as to how to think about and expose these types of traditional boundary errors.

Originally Published Wednesday, September 05, 2007

The traditional concept of boundary testing was established as a systematic procedure to more effectively and more efficiently identify a particular category of defects. Historically, boundary value analysis has focused on bounded physical (countable) linear input and/or output variables of independent parameters, and is especially useful in programming languages that are not strongly typed (such as C). The value of using the boundary value analysis technique is early identification of:

• typographical errors of specific bounded values (especially artificial constraints on a data type)
• improper use of relational operators around a physical bounded variable
• errors in implicit or explicit casting between data types

The application of boundary value analysis is not easy, because identifying boundary conditions from the user interface is extremely difficult. Also, if the tester lacks an understanding of programming concepts such as data types, casting, and looping structures then identifying boundaries becomes a mysterious art rather than a professional practice.

Casting (or conversion) between data types occurs rather frequently in software. For example, a calculator may only accept integer values as inputs, but the output of a division operation may result in a decimal value. In this case the operands are converted or parsed from strings to integer values using int.Parse(operand), or Convert.ToInt32(operand) in C#. Now, in order for the division operation to display decimal values if the result is not a whole number the operands must be cast to doubles (or floats) as in the example below.

   1: private double DivideOperation(int operand1, int operand2)

   2: {

   3:     //              the operands are explicitly cast from

   4:     //              integer types to double types

   5:     double result = (double)operand1 / (double)operand2;

   6:     return result;

   7: }

OK…so that is the basics of casting types, now let’s examine where casting can get us into trouble by exceeding physical bounded parameters. And to find these common errors we don’t have to go too far. In fact, the Attributes feature of MSPaint program provides us with plenty of examples of boundary issues.

Notice the default units for Width and Height parameters is Pixels. Pixels are in hole numbers. The input parameters Width and Height will accept a decimal value; but if the units selected is Pixels the decimal will be rounded to the nearest integer value. However, selecting either the inches or centimeters radio buttons for Units will allow decimal inputs. This is a pretty good clue that there is some pretty serious casting of these input parameters going on below the GUI. Now, we should know that inches are larger than centimeters, and centimeters are larger than pixels in size. We should also know that the maximum allowable input in the width or height parameter is limited to 5 characters, or a maximum valid input boundary of 99999.

Now there is nothing particularly unique or interesting about 99999, but we know that Pixels defaults to integer values, and inches and centimeters use decimal values, and that inches is bigger than pixels. And now, the magic math. If 29875 inches = 75882.50 centimeters, then 29876 inches should equal 75885.04 centimeters. (It doesn’t matter at this point if we click the OK button on the dialog and get an error message that reads "Please enter no more than 5 characters." even though we didn’t enter those 8 characters the software did that all by itself. We might not like the fact the software does what it tells the user he/she specifically can’t do, but (although many will argue this point) the program flooding its own edit control with more than 5 characters and then displaying an error message is really superficial and generally uninteresting.) But, it is a good thing we don’t use the Attributes feature of MSPaint as a conversion calculator because when we select inches as the units and enter 29876 in either the width or height parameters, then select the Cm radio button the value converts to 0!

What is this magical math? How can this happen when we aren’t even close to the maximum allowable input of 99999 inches? The value of 75885.04 is not some magical, unique or special value in computing. To understand what is happening we have to back up and convert from inches to pixels for a clearer picture. Select the inches radio button in the Unit groupbox, and enter the value 29875 for either the width or height parameters. Now select the Pixels radio button. Notice the value is 2147474. Now, testers know the upper physical bound of a signed 32-bit integer is 2 billion 147 million and some change (2,147,483,647 to be exact), but this value isn’t close. But, what if the last 3 characters are truncated by the control? Even in that case 29875 * 71.882 (pixels/inch) = 2147474.75 and the most significant digits above 1000 is still a value below the range of a signed 32-bit integer value (-2,147,483,648 ~ +2,147,483,647). But, when we enter 29876 inches  and then select the pixels radio button the value jumps to 4294967 (which are the most significant digits above the one thousands place of an unsigned 32-bit integer value (0 ~ 4294967296). This is because 29876 * 71.882 (pixels/inch) =  2147546.632 (and again looking at the significant values above the one thousands place) exceeds the upper physical bound of a signed 32-bit integer (2,147,483,648) resulting in an implicit cast to an unsigned 32-bit integer.

Now, simply setting the Units radio button to inches, entering 99999 for either the width or height parameters, and then selecting the Cm radio button, or selecting the Pixel radio button and then back to inches or Cm would reveal an error. However, a professional tester has enough working knowledge of the underlying system (including basic programming concepts) to not only know how and in which situations this type of defect can occur, but they can also use their cognitive abilities (having a basis in or reducible to empirical factual knowledge [of basic programming concepts in this case]) to perform conscious intellectual acts (cognition) to analyze features and identify areas where this type of problem might occur, and can then apply logical and rational tests that have a greater probability of exposing a defect (if one exists), or providing other valuable information regarding the valid operation or correctness of that feature.

There are other bugs in this relatively simple feature regarding data types, and perhaps I shall expose those at a later time!