Perhaps an even more important question is why do we find so many boundary type bugs via exploratory testing when they can and should be caught earlier? Why don’t we find these types of bugs in our unit testing? Why don’t we find these types of bugs by more systematically testing the software? Maybe we do find them, and those who make the decisions to fix these types of bugs just don’t care if they are fixed because there is no severe negative impact to the user. Maybe someone just wants to give me fodder for my blog!

This week I wanted to compare the range of allowable font sizes for a simulation program I developed as an example for a magazine article that I am working on. I knew that Office applications allow a font size within the range of 1 – 1638. I thought that range might be too large for my purposes, and since I knew that Windows Notepad included a font dialog I decided to check the allowable range of font sizes in Notepad.

The first thing I discovered was that the combobox control allows up to 5 characters! Really? Someone decided it is a good idea to allow users to enter 5 characters?

OK, I’ll play along. Maybe if I put in a size of 99999 and press the OK button on the dialog I will get an error message, or at least Notepad defaults to the last ‘valid’ selected font size. That might seem reasonable. But is that what happens? NO! Instead of doing something reasonable (e.g. error message, default font size) the font changes to a size of 1 (yes that is a font size 1 in the upper left corner in the image below).

I am sure that defaulting to a font size of 1 makes sense when the allowable size value overflows! Really…someone thought that was a good idea? Now I wanted to see what magical boundary value the developer decided was an acceptable font size. Since the combobox size property allowed 5 characters I immediately tried 65535. No, that also resulted in the overflow and displayed the text in a font size of 1. Next I tried 32767. Wait…32767 didn’t display the string in Notepad’s edit control at a font size of 1. Now, I am thinking the developer is using a data type of signed short for the font size variable. So, I enter 32768 expecting the value to overflow and display my string as a size 1 font again. But, no…that doesn’t happen.

Now, when I am design boundary tests I generally rely on 2 heuristics for identifying boundary values for input or output parameters.

1. Values at the extreme edges of a physical range of values
2. Values at the edges of equivalence partitions of physical values

So, in these situations I ask myself “What sort of demented developer debauchery have I now found myself?” I can’t think of any other obvious edge values that might apply, so out of curiosity I quickly narrow down the magical value to 39321. I then ask myself, “OK…even if there were a display capable of rendering or a printer capable of printing a font of this size, what is so unique about 39321?” In hexadecimal it is 0×9999, and in binary it is 1001100110011001. OK…nothing obviously special here, but I am certain the implementation details are much more complex then a simple range of values and at this point I really don’t care because this bug just doesn’t make sense.

Maybe it’s not supposed to make sense! Maybe nobody really cares about these types of bugs!

(BTW…somebody please take the Thesaurus away from the developer…’Oblique?’ Are you serious…why not just be consistent and use the word ‘Italic?’)

Anyway, one of the tool’s properties is a character array of invalid characters for the specific email address system under test. Although the guidelines for email addresses are outlined in RFC 5322 and RFC 2821 many companies can place greater restrictions on the characters that are allowed for the local address component of an email address (the local address is the part before the ‘@’ character).

For example, Yahoo only allows a local address to be between 4 and 32 characters, the first character must be a letter, and only letters, numbers, underscores and only 1 period character. The Google mail local address is between 6 and 30 characters, and only allows letters, numbers, and (multiple) period characters. Hotmail and Live mail allow local address name lengths between 6 and 64 characters (64 is the maximum allowable size according to RFC 5322), and can only contain letters, numbers, periods, hyphens, and underscores.

Even from these few examples we can see a couple of things. First, although we are testing email addresses there is not a universal set of equivalent partitions that works in all contexts. We need to partition the test data into equivalent class subsets based on the specific domain we are testing. For example, the invalid class subset of characters for a Google local address includes the underscore character, but both Yahoo and Hotmail allow the underscore as a valid character in an email local address. (But, I will talk next week about the equivalent partitioning of this data…for now let’s get back to boundary testing!)

Back to my story – as I was exploring each email providers requirements in order to determine how to partition the data I discovered a interesting problem with Yahoo. Remember, the maximum length of the local address for a Yahoo account is 32 characters.

And, the textbox control property on the web page is set to only allow a maximum input of 32 characters to prevent the user from inputting more than 32 characters. Copying a string longer than 32 characters into that textbox simply truncates the string after the 32nd character.

But, when I bump up against the maximum allowable length with some test strings the underlying program that generates suggested alternative local address names will actually produce a local address of 35 characters in length!

Now, if the software message tells me I can’t do something (like have a local address name of more than 32 characters and then the software generates a local address name of 35 characters for me…well, I am the sort of fellow who will push that button!

And sure enough it looks like I can use it. But wait. Only one more button to push and…

What do you mean “Sorry, this appears to be an invalid Yahoo ID?” You generated an invalid local address for me! Why would Yahoo mail torment me so?

I am thinking in the developers mind the user story went sort of like;

User: “I would like this.”

System: “No you can’t have that, but you can have this.”

User: “OK”

System: “No, you can’t have that either.”

It’s funny this came up this week because I was talking with a group of senior SDETs about defect prevention versus defect detection and how 99.999% of boundary issues can be found at the unit level or API level of testing well before the UI is slapped onto the functional layer.

Testing the functional layer more thoroughly or a code review would most likely have revealed this ‘magic’ number was inconsistent. Or by forcing the algorithm that generates suggested local addresses to test boundary conditions would have much sooner exposed this problem.

Now I don’t know Yahoo’s development and testing practices, and unfortunately it’s not uncommon to overlook bugs similar to this. But, I suspect that if developer rely on testers to find all their bugs, and testers primarily rely on testing through the user interface to find bugs then we are always going to find boundary bugs post release (and that’s a good thing because it gives me something to blog about).

At a recent conference a speaker posed a problem in which a field accepted a string of characters with a maximum of 32,768 bytes, then asked the audience what values they would use for boundary testing. Immediately some of the attendees unleashed a flurry of silly wild ass guesses (SWAG) such as “32,000,” “64,000,” and, of course, what attempt at guessing would be complete without someone yelling out “how about a really large string!” One person asked whether it was bytes or characters? A reasonable question, but the speaker than began talking about double byte characters (DBCS). (Double byte is, in technological time, a relatively antiquated character encoding technology since most modern operating systems process data as Unicode.)

So, while some folks in the audience continued to shout out various SWAGs, I was still pondering why anyone in their right mind would artificially constrain a user input to such a seemingly ridiculous magic number within the context of computer processing and programming languages. Programming languages allow specific ranges of numeric input. Most strongly typed languages such as the C family of languages have explicit built in or intrinsic data types that include signed and unsigned ranges. For example, an unsigned short is 2^16 or 0 through 65,535, and a signed short is also 2^16 but the range is -32,768 through +32,787. Since the speaker didn’t indicate what programming language was used in this magical field, the only logical conclusion a professional tester can rationally deduce is that 32,768 is a magic number, or in other words a “hard-coded” constant value embedded somewhere in the code.

Asking questions is important! But, asking a bunch of contextually-free questions or throwing out random guesses is usually not the most efficient or productive use of one’s time. Asking specific rational questions or making logical assertions based on knowledge and understanding is important, and is generally more productive; especially when testing the boundary conditions of input or output values in software. Boundary testing is a technique that focuses on linear input or output values that are fixed, or fixed-in-time and used for various computations or Boolean decisions (branching) within the software. Similar to most testing techniques boundary testing focuses on exposing one category of issues based on a very specific fault model, and is an extremely efficient systematic approach to effectively expose that particular category of issues. In particular boundary testing is useful in identifying problems with:

• improperly used relational operators
• incorrectly assigned constant values
• and computational errors that might cause an intrinsic data type to either overflow or wrap especially when casting or converting between data types (proper identification of the data type and knowledge of the minimum and maximum ranges is critical)

I previously wrote about approaches to help the tester identify potential boundary conditions, and how to design tests to adequately analyze those specific boundary values. As I previously stated, boundary testing involves the systematic analysis of a specific value. For example, a long file name on the Windows platform (both the base file name and the extension) should not exceed 255 characters. For file types that use a default 3-character extension the most interesting boundary values are 1 character (minimum base file name length) 251 characters (maximum base file name length assuming a standard 3-character extension), and 255 characters (with or without an extension to test what occurs with a base file name equal in length to the maximum base file name with a standard 3-character extension. (Of course, if the default extension is 1-character, or 2-characters, or 4-characters, etc., than the maximum base file name without extension needs to be recalculated.) Now, let’s see why specific values are important and critical to accurately analyze boundaries.

On Windows Xp I used Notepad to test file name boundaries with a default 3-character extension. Of course the minimum -1 value is an empty string, and minimum and minimum +1 is saving a file with a 1-character and 2-character file name respectively. Next I entered a base file name of 250-characters (maximum -1) and 251-characters (maximum allowed assuming a default 3-character extension) and these file names were saved to the system with the default extension. Then I entered a 252-character file name and I got the expected error message indicating the file name is too long. But, what about my boundary of 255 characters maximum. (IMPORTANT – boundary values are not just at the edges of the extreme ranges of values, but there could be sub or supra boundary values within a range of values that may occur at the edges of equivalent class ranges, or specific values in special or unique equivalence class subsets.) So, I wondered what would happen if I entered just a base file name of 255 characters (which is the maximum length of a file name assuming an extension is also part of that file name)? Interestingly enough, on Windows Xp the operating system saved a file with 255-characters, but it did not have any extension which means that there was no application associated with the file. The same occurred with a 254-character base file name, and when I tried the maximum +1 of the overall complete file name range I was again presented with the same message I got with a 252 character base file name indicating the file name was too long.

Fortunately, the above issue was fixed in Windows Vista. But, as sometimes occurs in complex systems one fix occasionally leads to a different (but related)issue in the same functional area which is why regression testing is typically an effective testing strategy. So, when I ran my ‘regression tests’ on Windows Vista I quickly discovered the system would not save a file with only a base file name of any number of characters greater than 252-characters via Notepad. But, as I ran the specific boundary tests I realized something very important! When I entered a base file name of 252-characters I received the following error message.

And when I attempted the test with a base file name composed of any number of characters greater than 252 I received the following error message.

Now, those of you who are paying attention realize these 2 messages are different. Of course, in either case a file is not saved to the system which is what I expect; however, there is a strange anomaly here. Although one might notice the first message prepends the drive letter to the string of 252 characters, and the second message does not. But, the important question doesn’t really have anything to do with the message text per se, in this case the professional tester tester should ask, “why is there an apparent conditional branch in the code that shunts control flow one way for a base file name of 252 characters and a different path for a base file name greater than 252 characters?”

Of course, if we just guessed, or tested ‘a really large string of characters, we might have never exposed this anomaly which occurs only at the maximum + 1 length of a base file name (assuming a default 3 character extension). Interestingly enough, if a highly skilled, technically savvy tester had designed white box tests for decision testing or path analysis then I suspect he or she could have very easily found this anomaly with even greater efficiency and exposed it earlier in the cycle.

The point here is that boundary testing is simply not random guessing, wild speculation, or simple parlor tricks. The technique of boundary value analysis requires in-depth knowledge of what the system is doing behind the user interface, and careful analysis of system and data to accurately determine the specific boundary conditions and a rigorous analysis of linear values immediately above and below each identified specific boundary value. Testers must be able to properly identify the specific and interesting boundary values based on in-depth knowledge of the system, an understanding of what is happening beneath the user interface, and experience.  Then we can perform a more systematic analysis of any identified boundary conditions and potentially increase our probability of identifying real anomalies caused by this specific fault model. Boundary value analysis is a prime example of where good enough is simply not good enough in our discipline…we must be technically spot on!

Last week I attended the Software Testing and Performance conference in Boston. I presented a workshop on Systematic Testing Techniques, as well as a talk on random test data generation, and combinatorial analysis. One way I continue to learn about our profession and increase my own knowledge is by going to conferences to hear different points of view from practitioners from around the world. So, I also attended several talks during the conference, but there was one talk in particular that was particularly entertaining (and I don’t mean that in a good way).

When I listen to other testers sometimes I hear something that is new to me and I desire to learn more about it. Sometimes I hear something prophetic that makes me think, contemplate alternatives, or reflect more deeply on my own personal perspectives. Sometimes I hear something revolutionary that causes me to reevaluate my position. And, sometimes I hear something so irrational I almost barf up a lung!

In this case the speaker opened his talk with an attack on a quote from the ISTQB foundation syllabus used to describe boundary testing which states, "Behavior at the edge of each equivalence partition is more likely to be incorrect…" Now I know the speaker a bit, and I know he disdains the ISTQB and other certification organizations, but what surprised me was his initial rebuttal by emphatically stating, "equivalence class partitions are figments of our imaginations!"

These days I usually just try to shake off wild and baseless comments as bombastic bloviations used to generate controversy. But, in this case what caught my attention was when the speaker later said that he and another well-known person defined boundaries as "a dividing point between two otherwise contiguous regions of behavior; or a principle or mechanism by which things are classified into different sets." What!? I couldn’t believe what I heard, so I had to stop reading email and look up at the presentation. As I visually processed the words I thought my head was going to explode from the so seemingly obvious contradiction.

Now, I am not a linguistic expert, but I am pretty sure that "otherwise contiguous regions of behavior" and "classifying things into different sets" are just overly simplistic ways of describing equivalence class partitions. But, I could be wrong. So, I began thinking that since most people start learning about sets in elementary schools they probably understand the foundation of equivalence class partitioning is set theory which basically states "a set is an aggregate, class, or collection of objects," and the collection of objects or ‘classification of things’ in different sets is based on an equivalent relation between the elements in each set. The application of equivalence class partitions in our profession is elegantly explained by Lee Copeland in his excellent book A Practitioner’s Guide to Software Test Design by stating "An equivalence class consists of a set of data that is treated the same by the module or that should produce the same result." Equivalence class partitioning is also discussed in-depth in books by noted experts in the industry such as Beizer, Binder, Myers, Jorgensen, Perry, and Marick just to name a few.

In fact, the concept of sets and equivalence almost seems instinctive in most humans and is generally expressed at a young age. I remember my young daughter at age 2 or so separating beads by color into "different sets" on the carpet. The red beads in one group, blue in another, and so on. She was diligent to make sure the different sets of beads did not touch as she put them into the appropriate piles. If a pile of beads got to close to another pile she would run the edge of her hand between the "contiguous regions" to clearly delineate the "dividing point."  When I asked her to get me a red bead, she would randomly grab one from the pile, because all the red beads were…red, and there were no significant differences among the red beads (elements) in the set she created that were relevant in that context of that game.

Perhaps the majority of the industry’s experts are wrong and I wasted my time reading books on software testing principles and practices because this person is right and equivalence class partitioning is really only a figment of our imagination.

However, on the other hand, although I certainly have never claimed to be an expert, I am still pretty darn sure the underlying foundation of computers and computer software is somewhat influenced by mathematical principles, and as a tester I might be able to use those same principles to help me design effective tests that might help me better evaluate discrete functional capabilities and attributes of software components and more efficiently expose certain categories or patterns of errors.

But, why should we get mired down and confused with facts (especially all that boring math stuff) when it is much easier to appeal to some peoples’ emotions. So,forget everything you just read…and if anyone asks why testing is so hard just tell them testing is an art with no practical foundation in logic because software is…well, it’s just magic!

Too often many tester’s mistakenly assume that functional techniques such as equivalence class partitioning, boundary value analysis, combinatorial analysis, etc. are simply "black-box" testing techniques. I suspect this rather narrow perspective of functional testing techniques is due to a lack of in-depth understanding of testing throughout the product lifecycle, and confusion between test design and test execution. I also suspect this incorrect assumption is perpetuated by ‘testers’ whose only approach to testing is interacting with the software via the user interface with the intent to find bugs. Of course when a person can only ‘test’ via the user interface then everything is a "black-box."

When I teach software testing courses I reinforce the concepts of black, white, and gray boxes as perspectives for test design; not for test execution. Using these concepts as test design approaches designing tests from black box test design makes no assumptions about the code and focuses on inputs and outputs from the end user interface, white box test design explicitly uses the code to help the tester design more effective tests, and gray box principles use an in-depth understanding of the entire system to design effective tests within the context of the software solution.

Functional testing techniques were derived years ago by industry experts and the founding fathers of software testing to provide systematic procedures that help identify specific categories of problems based on fault models that are commonly encountered during the development process. Certainly, functional techniques such as combinatorial analysis are very effective when applied from the end-user interface; however, we all know the fault model which combinatorial testing is effective in identifying involves semi-coupled or directly dependent unordered parameters. So, in order for this functional technique to be applied correctly the tester must have an in-depth understanding of the domain space and know which unordered parameters are semi-coupled or directly dependent. Boundary testing is very effective in evaluating how specific linear variable conditions of independent parameters are handled and is useful for identifying problems with overflowing intrinsic data type ranges, casting between data types, incorrect usage of relational operators, etc. Effective boundary testing means the tester must have some knowledge of data types and programming concepts and insight into the code is helpful in order to design the most effective tests.

Many of the functional techniques we apply at the end-user interface actually require a solid understanding of the underlying system to be most effective. So, designing tests using many functional testing techniques actually approaches test design from a gray-box perspective. These functional testing techniques can also be effectively applied during unit testing to drive quality upstream and prevent problems from getting into the product. Unit tests, of course, that include comprehensive boundary value analysis are tests designed from a white box perspective.

This doesn’t mean that functional testing techniques shouldn’t be executed from the end-user interface. But, the purpose of functional testing techniques is to help testers design tests from multiple perspectives in order to perform a more in-depth, systematic analysis of software, and potentially expose very specific categories of problems based on empirical fault models. The execution of those tests may be applied either at the code level or from the end-user interface. But, effective application of functional techniques requires in-depth system level knowledge and cognitive skills that extends below the end-user interface.

Testers must embrace and extend the role of testing beyond the simple ability to find bugs at the end-user interface. Testers must start testing earlier as a partner in the process rather than an adversary of the development team. Testers must become proficient in various approaches necessary to provide a wide variety of information to the key decision makers throughout the development lifecycle instead of at the end of it. Quite simply, professional testers need to break beyond the "black-box "barrier!

This morning I installed Vista SP1 onto my laptop. I was pretty excited about this release of Vista SP1 because it includes some pretty significant performance enhancements. But, as I was preparing to teach an internal course I came across a new boundary issue. I thought, how fitting this comes as I prepare to teach another class on systematic testing techniques (including boundary value analysis) that I find yet another classic boundary issue in Explorer’s list view (albeit the boundary condition is not readily apparent) . Interestingly enough, the previous boundary issue I found appears to be fixed with Vista’s SP1; however, I previously did not run across this issue which is also connected with how listview repaints itself after an event.

To reproduce this issue, open a folder with several dozen files in Explorer and select Views -> List and resize the Explorer window so there are several columns of files.

Select a file in the list and press the up or down arrow key so a dotted line appears around the file name

Resize the Explorer window so that the highlighted area is about 1 pixel away from the right most edge of the Explorer window as in the example below

Press the down (or up depending on which file you have selected) arrow key, and notice how the file highlight jumps to the next file in the list as expected.

Next, resize the the Explorer window so the dotted line touches the inner boundary of the window as in the example below

Now, press the down (or up depending on which file you have selected) and…

Notice, in the image above that just to the left of the frogger.def file the right most edge of the file highlight is visible, and also notice the file image has changed, and the scroll bar has jumped over to the next column.

Press the down (or up) arrow key again, and….

Seeing this behavior the first time made me think I was losing my mind! (OK…I probably am, but that is a completely different topic). And yes, reducing the size of the window beyond where the maximum boundary of the highlight window will also cause the same behavior, but the issue occurs when the width of the file highlight window >= the maximum x boundary size of the Explorer listview window.

The listview in Windows Vista has some very cool features, and overall I like Vista. What I dislike are some of the simple anomalies that could be easily exposed via more detailed, systematic testing approaches and analysis of the system under test rather than simply assuming more bodies banging on keyboards for some arbitrary period of time equates to better testing.

So, some things to consider when boundary testing…

• Not all boundary conditions are easily identified from the GUI by numbers, but all boundaries conditions have linear physical values that can be measured at some level
• Boundaries can change, similar to way the boundaries changed around Kosovo. Somebody moved the lines. The same thing is true in software, a boundary condition may change with human interaction, but a boundary condition is a linear physical range between a minimum and a maximum value (in this case the x coordinates of the Explorer changed with human interaction, but once focus moved away from the Explorer window the size of x established a fixed boundary (at least until a user again resized the window or the window is closed).
• Boundary conditions for one parameter are usually independent. For example, resizing the y-axis in the above scenario has no impact on this defect (unless of course the y-axis is large enough to accommodate all files in the listview without having to scroll horizontally). Boundary testing is based on the single fault assumption theory which states that a boundary issue is most likely to occur with independent parameters where the boundary variables for one parameter are analyzed while holding other parameters to nominal values. (Note: if we suspect that parameters are dependent or semi-coupled, then we should also perform combinatorial analysis testing.)
• There is a difference between boundary conditions (at least momentarily fixed, linear, measures) and threshold values. Threshold values can be altered by various influences. For example, in performance testing the point of degradation in performance can often be changed by several external influences such as increasing physical memory, cleaning and defragmenting the hard disk, modifying the software, etc.
• A detailed analysis of the system under test will reveal issues that other approaches to testing do not expose…that’s the pesticide paradox!

In the last post we decomposed the set of characters in the ANSI character set into valid and invalid class subsets for use in a base filename component on the Windows Xp operating system. The second part of the testing technique of equivalence class partitioning is to then use this information in tests that will adequately evaluate the functional capabilities of the base filename parameter used by COMDLG32.DLL in Windows Xp.

(Note: It is appropriate here to differentiate between a test case and a test. A test case is the hypothesis or assertion that I am either trying to prove or disprove. A test is the process or procedure I am using to prove or disprove my hypothesis. In other words, a test case in this example might be "Validate a base filename can be saved using valid characters in the ANSI 1252 character set." In order to prove or disprove this hypothesis, I will execute multiple tests using the various subsets outlined in the previous post, and as described below. If any test fails, the test case fails.)

There are basically 2 procedures for defining the tests that use the data from an equivalence class table. Paul Jorgensen describes a procedure for robust equivalence class testing in which each valid class subset is tested individually. Glenford Myers suggests a procedure in which valid class subsets are combined in a test until all valid class subsets have been covered. (Jorgensen refers to this as weak equivalence class testing.) Both authors agree that subsets in the invalid subsets must be tested in a way that only one parameter is invalid and any other parameters use valid variables. Using Myers’ approach (or what Jorgensen describes as weak equivalence class testing) the tests for the base filename parameter are illustrated in the table below.

[Edited 11/30] I apparently made too many assumptions about conducting these tests, so I must clarify some of those assumptions.

• Any time a key is pressed there is essentially a state change. The primary state change we are concerned with for this particular functional test is whether or not a file is saved to the file system. However, in cases of invalid input there are other state changes that may be interesting. In most well designed Windows applications an erroneous input in an input control will be highlighted (usually after an error message is displayed). This is indeed an expected state change. The only time no noticeable state change occurs (there is still a state change because the machine is processing WM_Key messages) is with Test #5. But, another noticeable state change (the one primarily identified in the table below) is the state change to the list view control.

• There are 3 ways to effectively affect the Save button. One way is to press enter, another is to press the Alt + s key mnemonic, and thirdly to left mouse click the button control (including simulating any of those actions via automation). In some cases the visible state changes may vary; however, the purpose of these tests it so verify the existence of a file name in the file system for valid cases, and the non-existence and no abhorrent side effects in the case of members of the invalid class subsets. So, regardless of the visible state changes, the tester can use any of these procedures to affect the Save button.

• Application of techniques (systematic procedures) at a granular level is very different than simply trying things to see what happens, exploratory testing, guessing, or wild speculation. I made the assumption that readers are familiar with designing and executing atomic functional tests at a granular level in which certain variables are controlled. For example, once we enter an invalid string in the file name text box and press the save button the Save As… dialog is now dirty; meaning that the next input could produce an error but I will not know if the error is a result of an error with the base filename parameter, or with the Save As… dialog state. Equivalence class partitioning is a low level functional test of a single parameter, so in order to test the base filename parameter the tester (or automated test) should minimally close and instantiate the Save As… dialog on each test.

Reducing tests while increasing breadth of coverage

The total number of possible tests of valid string combinations for just the base filename parameter using only characters within the ANSI 1252 character set is 214²⁵¹ + 214²⁵⁰ + 214²⁴⁹ + …214² + 214¹. This number of tests, of course, is a virtual impossibility, so by employing the equivalence class partitioning testing technique we are able to systematically produce a minimum baseline set of tests that has a high probability of both proving or disproving our hypothesis or test purpose, as well as providing great variability in the test data to increase breadth of data coverage. The minimum possible number of valid tests determined by combining at least one element from each valid class subset is only 4 tests. But, let’s look at each valid test a little more closely.

Test #3 is probably a red-herring! This is only an invalid filename on Windows NT 4.0 and below. So, if your application is ported from that time frame, and you are using a custom function for your file save functionality rather than Windows APIs then you might consider running this test once. If if passes, you can probably forget running this again ever again on that product. Test #1 evaluates the literal strings in valid subset V1 can be listed in an array or enumeration and one element can be selected at random throughout the development lifecycle, or each literal string can be tested once and the probability of failure in a later build is most likely less than .001%. Test #4 is also a test that probably doesn’t require a great deal of retesting of various combinations of elements from subsets V2, V3, V4, V5 , & V7. Elements from the valid class subsets described in Test #2 are most interesting and this is the test that we will probably want to automate and run repeated throughout the development lifecycle because it provides great breadth of coverage. Remember this is the minimum number of valid tests. What isn’t covered in this set is common or ‘real-world’ data sets which we would certainly want to include. Additionally, Test #2 relies on at least one element from each indicated subset. We might also want to consider additional tests that focus on subsets V4 and V5 only. Also, we might also consider testing valid class subset V6 as a special case if we suspected a private function excluded code point values that were not assigned character glyphs. However, if these 4 valid tests pass the probability of failure of any combination of these data sets used in this parameter is minimal. Random selection of elements for Test #2 (and possibly Test #4) may slightly increase the probability of exposing a defect in the base filename parsing routine. Tests #5 through #26 are tests for invalid filenames, or in the case of Test #19 and #20 where the expected result is to truncate the leading or trailing space character.

This of course only analyzes (or tests) the base filename parameter and assumes a nominal valid 3-letter extension, valid filenames do not preexist on the file system, and within the specific context described in the first post. If the context changes, (e.g. this example does not apply to Unix platforms, or Windows Vista, or other parameters) then this set of tests (assuming we would include at least Tests #2, and #5 through #26 as part of our regression suite ran on each new build) would provide a high level of confidence in the functional capability of this specific parameter.

Next, we would also decompose the data for the file extension parameter (which is different than the base filename parameter) because in the File name textbox we can enter either the base filename or the base filename and an extension, and once we verify the functionality of the base filename component, we can then proceed to the next step in analyzing the functionality of the arguments passed to the File name textbox parameter which we shall examine at a later date.

It is important to note the this and any other techniques are simply systematic procedures designed to help us wrap our minds around complex problems. They are not the only approach to testing (only a fool would advocate a single approach to testing), but when used in conjunction with various other techniques, methods, and approaches EQ can help to establish a strong foundation of confidence in low level operations. Of course, as has been previously stated, the limiting factor of this and other functional techniques is the knowledge of the professional tester to think critically and rationally analyze the "overall system" within the appropriate context.

Again, I am remiss in my postings…too many irons in the fire these days. Two weeks ago, I posted a challenge to decompose a set of character data (The ANSI Latin 1 Character Set) into valid and invalid equivalence class subsets in order to test the base filename parameter of a filename passed to COMDLG32.DLL on the Windows Xp platform from the user interface using the File Save As… dialog of Notepad.

As illustrated below the filename on a Windows platform is composed of two separate parameters. Although the file name parameter of the Save As… dialog will accept a base filename, a base filename with an extension, or a path with a filename with or without an extension, the purpose of the challenge was to decompose the limited set of characters into equivalence class subsets for the base filename component only (the part outlined with green). (Of course, complete testing will include testing with and without extensions, but let’s first focus on building a foundation of tests to adequately evaluate the base filename parameter first, then we can expand our tests from there to include extensions.)

As suggested in the earlier post, in order to adequately decompose this set of data within the defined, real world context (and not in alternate philosophical alternate universes) a professional tester would need to understand programming concepts, file naming conventions on a Windows platform, Windows Xp file system, basic default character encoding on the Windows Xp operating system (Unicode), some historical knowledge of the FAT file system, and even a bit of knowledge of the PC/AT architecture. The following is a table illustrating how I would decompose the data set into equivalence class subsets.

Valid Class Subsets

• V₁ – escape sequence literal strings
        (STX, SOT, ETX, EOT, ENQ, ACK, BEL,
        BS, HT, LF, VT, FF, CR, SO, SI, DLE,
        DC1, DC2, DC3, DC4, NAK, SYN, ETB,
        CAN, EM, SUB, ESC, FS, GS, RS, US,
        DEL)
• V₂ – space character (0×20) (but not as
         only, first, or last character in the 
         base file name)
• V₃ – period character (0×2E) (but not as
         only character in the base file name)
• V₄ – ASCII characters
         punctuation (0×21, 0×23 – 0×29, 0×2B – 0×2D, 0×3B, 0×3D, 0×40, 0×5B, 0×5D, - 
         0×60, 0×7B, 0×7D, 0×7E),
         numbers (0×30 – 0×39)
         alpha (0×41 – 0×5A, 0×61 – 0×7A)
• V₅ – Ox80 through 0xFF
• V₆ – 0×81, 0×8D, 0×8F, 0×90, 0×9D
• V₇ – Component length between 1 – 251
         characters (assuming a default 3-
         letter extension and a maximum path
         length of 260 characters)
• V₈ – Literal string CLOCK$ (NT 4.0 code 
         base)
• V₉ – a valid string with a reserved
           character 0×22 as the first and
           last character in the string

Invalid Class Subsets

• I₁ – control codes 
       (Ctrl + @, Ctrl + B, Ctrl + C, Ctrl + ], Ctrl + N, 
       etc.)
• I₂ – escape sequence literal string NUL
• I₃ – Tab character
• I₄ – reserved words
        (LPT1 – LPT4, COM1 – COM4, CON, PRN, AUX,
        etc.)
• I₅ – reserved words
         (LPT5 – LPT9, COM5 – COM9)
• I₆ – reserved characters (/ : < > | )
         (0×2F, 0×3A, 0×3C, 0×3E, 0×7C) by
        themselves or as part of a string of 
        characters
• I₇ – reserved character 0×22 as the only
        character or > 2 characters in the string
• I₈ – a string composed of > 1 reserved character
        0×5C
• I₉ – a string containing only 2 reserved
         characters 0×22
• I₁₀ – period character (0×2E) as only
         character in a string
• I₁₁ – two period characters (0×2E) as only
         characters in a string
• I₁₂ – > 2 period characters (0×2E) as only
         characters in a string
• I₁₃ – reserved character 0×5C as the only 
           character in the string
• I₁₄ – space character (0×20) as only character in 
         a string
• I₁₅ – space character (0×20) as first character in
          a string
• I₁₆ – space character (0×20) as last character in a
          string
• I₁₇ – reserved characters (* ?)  (0×2A, 0×3F)
• I₁₈ – a string of valid characters that contains at least one reserved characters (* ?)  (0×2A, 0×3F)
• I₁₉ – a string of valid characters that contains at
          least one reserved character 0×5C but not
          in the first position
• I₂₀ – string > 251 characters
• I₂₁ – character case sensitivity
• I₂₂ – empty

Discussion of valid equivalence class subsets

• Valid subset V₁ is composed of the literal strings for control characters (or escape sequences) between 0×01 and 0×1F, and including 0×7F. The literal strings for control characters may cause problems under various configurations or unique situations. The book How to Break Software: A Practical Guide to Testing goes into great detail explaining various fault models for these various character values. The literal strings in this subset should be tested as the base filename component and possibly in a separate test as an extension component. However, on the Windows platform the probability of one particular string in this subset behaving or being handled differently than any of the others is very low negating the need to test every string in this subset; although the overhead to test all would be minimal and once complete would not likely require repeated testing of all literal strings in this subset during a project cycle.
• Valid subset V₂ provides guidance on the use of the space character in valid filenames. On the Windows operating system a space character (0×20) is allowed in a base filename, but is not permitted as the only character as a file name. Typical behavior on the Windows platform also truncates the space character if it is used as the first character of a base filename or the last character of a base filename. However, if the extension is appended to the base filename in the Filename edit control on the Save or Save As… dialog a space character can be the last character in the base filename. Also note that a space character by itself or as the first character in a filename is acceptable on a UNIX based operating system. Also, although we can force the Windows platform to save a file name with only a space character by typing “ .txt” (including the quotes) in the Filename edit control on the Save/Save As… dialog this practice is not typical of reasonable Windows users’ expectations.
• Valid subset V₃ is the period character (0×2E) which is allowed in a base filename, but it is not a valid filename if it is the only character in the base filename (see Invalid subset for the period character).
• Valid subset V₄ is composed of ‘printable’ ASCII characters that are valid ASCII characters in a Windows filename. The subset includes punctuation characters, numeric characters, and alpha characters. We could also decompose this subset further into additional subsets including valid punctuation characters, numbers, upper case, and lower case characters if we wanted to ensure that we test at least one element from the superset at least once.
• Valid subset V₅ is the set of character code points between 0×80 and 0xFF.
• Valid subset V₆ is a superset of subset V₅ and are separated only because they are code points that do not have character glyphs assigned to those code point values. These would be interesting especially if we needed to test filenames for backwards compatibility on Windows 9x platforms.
• Valid subset V₇ is the minimum and maximum component length assuming the filename is saved in the root directory (C:\). 
• Valid subset V₈ is a probably a red-herring. On the NT 4 platform the string CLOCK$ was a reserved word. On an older application first created for the Windows NT 4 platform that does not use the system Save/Save As dialog we might want to test this string just to make sure the developer did not hard code the string in an error handling routine.
• Valid subset V₉ is an interesting case because this invalid reserved character (0×22) is handled differently when used in first and last character positions of a base filename. When used in the first and last positions of a base filename the characters are truncated and if the remaining string is valid the filename is saved. If only one 0×22 character is used, or if two or more 0×22 characters are used in a string other than the first and last character positions the result will be an error message.

Discussion of invalid equivalence class subsets

• Invalid subset I₁ consists of the control code inputs for escape sequences in the range of 0×01 through 0×1F, and also includes 0×7F. Pressing the control key (CTRL) and any of the control codes keys will cause a system beep.
• Invalid subset I₂ is the literal string “nul”. Nul is a reserved word but could be processed differently than other reserved words on the Windows platform because it is also used in many coding languages as a character for string termination.
• Invalid subset I₃ is the tab character which can be copied and pasted into the Filename textbox control. Pasting a tab into the and pressing the save button will generate an error message.
• The invalid subset I₄ includes literal strings for reserved device names on the PC/AT machine and the Windows platform. Using any string in this subset result in an error message indicating the filename is a reserved device name.
• Invalid subset I₅ also includes reserved device names for LPT5 – LPT9 and COM5 – COM9. However these must be separated into a unique subset because using these specific device names as the base filename on the Windows Xp operating system result in an error message indicating the filename is invalid.
• Invalid subsets I₆, I₇, and I₈, include reserved characters on a Windows platform. When characters in this subset are used by themselves or in any position in a string of characters the result is an error message indicating the above file name is invalid.
• Invalid subsets I₉, I₁₀, I₁₃, also include reserved characters and the space and period characters. When these subsets are tested as defined no error message displayed and focus is restored to the File name control on the Save/Save As… dialog.
• Invalid subsets I₁₁, I₁₂, also include the reserved character (0×2E) as 2 characters in the string and greater than 2 characters in a string. The state machine changes are different.
• Invalid subsets I₁₅ and I₁₆ define the space character when used in the first or last character position of a string. These are placed in the invalid class because Windows normal behavior is to truncate a leading or trailing space character in a file name. If the leading or trailing space character was not truncated and saved as part of the file name on a Windows platform that would constitute a defect.
• Invalid subset I₁₇ and I₁₈ contains two additional reserved characters; the asterisk and the question mark (0×2A and 0×3F respectively). If these characters are used by themselves or as a character in a string of other valid characters a file will not be saved, and no error message will occur. However, the state of the Save/Save As… dialog does change. If the default file type is .txt and there are text files displayed in the Folder View control on the Save As… dialog the files with the .txt extension will be removed after the Save button is depressed. If the default file type is All files then all files will be removed from the Folder View control on the Save As… dialog after the Save button is depressed.
• Invalid subset I₁₉ is a string of valid characters which contains at least backslash character except as the lead character in the string. (Of course, this assumes the string is random and the position of the backslash character in the string is not in a position which would resolve to a valid path.) The backslash character is a reserved character for use as a path delimiter in the file system. An error message will appear indicating the path is invalid.
• Invalid subset I₂₀ tests for extremely long base file name lengths of greater than 252 characters. Note that an interesting anomaly occurs with string lengths. A base file name string length which tests the boundaries of 252 or 253 valid characters will cause an error message to display indicating the file name is invalid. However, a base file name string length of 254 or 255 characters will actually get saved as file name but is not associated with any file type. Any base file name string longer than 255 characters again instantiates an error message.
• Invalid subset I₂₁ describes the tests for case sensitivity. The Windows platform does not consider character case of characters that have an upper case and a lower case representation. For example, a file name with a lower case Latin character ‘a’ is considered the same as a file name with the upper case Latin character ‘A’.
• Invalid subset I₂₂ is, of course, an empty string

Of course, this is a partial list of the complete data set since the filename on a Windows Xp operating system can be any valid Unicode value of which there are several thousand character code points, including surrogate pair characters.

The first and by far the most complex step in the application of the functional technique of equivalence class partitioning is data decomposition. This requires an incredible amount of knowledge about the system. Data decomposition is an exercise in modeling data. The less one understands the data set, or the system under test the greater the probability of missing something. Next week we will analyze the equivalence class subsets to define are baseline set of tests to evaluate the base filename component.

Wow…where does the time go? I was remiss last week in posting, and it has been a month since I posted about equivalence class partitioning. So, let’s get back to it shall we?

Equivalence class partitioning (ECP) is a functional testing technique useful in either black box or white box test design. A technique is a systematic approach to help solve a complex problem. Techniques are not silver-bullets, but they are a logical and analytical approach to problem solving that heavily draws upon the tester’s cognitive abilities (having a basis in or reducible to empirical factual knowledge) as opposed to random guessing (or little men running about inside one’s head triggering turbid thoughts). Contrary to popular misconceptions the application of ECP is not a rote, brain-dead activity. The ECP technique requires in-depth knowledge of the data set (data type, encoding method, etc), the programming language used in the implementation, the algorithm structure, the operating environment, protocols, and even the hardware platform may impact how the data for a particular parameter might be decomposed. The effectiveness of the application of this technique solely lies in the testers ability to adequately decompose the data set for a given parameter into subsets in which any element from a specific subset would produce the same result as any other element from that subset.

Essentially the ECP technique analyzes input or output variable data for each specific parameter and decomposes the data into discrete valid and invalid class subsets. See my article in Software Testing & Performance magazine for a more in-depth understanding of the equivalence class partitioning technique. The article uses a simple next date program as an example because working with integer type data within a limited range is usually a good introduction to this complex technique.

A frequent reader of this blog suggested he often sees this technique applied to simple parameters limited to number type inputs, but would like to understand how this is applied to a more complex parameter that takes a string input. Interestingly enough, one of the exercises I designed in our internal training (and also used in the workshop I present regularly at the Software Testing and Performance conferences) deals specifically with string input. Let me outline the exercise.

Your goal is to adequately decompose the set of data (the ANSI Latin 1 character set) into valid and invalid equivalent class subsets to evaluate the behavior of the base filename parameter passed to COMDLG32.DLL’s file save functionality from a user’s perspective (GUI) on the Windows Xp operating system with an NTFS file system on a PC/AT platform.

NOTES:

• On the Windows environment the base filename parameter is separate and not interdependent on the extension parameter. (Some frequent readers may remember I explained this point previously to an individual who mistakenly assumed the base filename parameter and extension parameter were interdependent on the Windows environment.) So, remember, the purpose of this exercise is to focus initially on the base filename parameter (don’t worry about the extension parameter…yet).
• To expose the user perspective of the file save functionality of COMDLG32 we will use Notepad.exe, and select the File -> Save As… menu item to instantiate the Save As… dialog.
• Characters greater than 0×7F can be entered using the numeric input method by holding down the ALT key and pressing number keypad keys equivalent to the characters decimal value. For example to enter the € character press and hold the ALT key and press the 0 1 2 8 keys on the numeric keypad. Release the ALT key and the character is displayed. (You can also use my Babel tool to generate specific characters by inputting the Unicode values for the characters in the Custom range groupbox and copying them into the filename control.)
• Not all characters can be entered using the numeric input method. For example, control sequences cannot be directly entered from the keyboard using this method (or by holding down the control key).
• In order to succeed, one must understand
  • Common programming concepts (especially for the C family of languages)
  • Windows File naming conventions
  • Windows Xp, file I/O APIs
  • Windows Xp NTFS file system
  • Knowledge of basic character encoding
  • Historical knowledge of FAT file systems

Give it a try…next week I will offer a solution that you can compare your own results against.

I previously discussed various types of defects exposed via application of the boundary value analysis testing technique including a repaint problem, a casting problem, and a wrapping problem. While the minimum and maximum physical linear boundaries of a parameter are often easy to identify, it is surprisingly more difficult to identify boundary conditions within the minimum and maximum range especially if the tester does not adequately decompose the data. This week I will discuss another boundary defect that is often hidden below the user interface, but could be exposed using boundary value analysis testing at the unit level.

Loops are common structures in software, and (depending on the programming language) are susceptible to boundary defects. Boundary value analysis of a loop structure involves (at a minimum) bypassing the loop, iterating through the loop one time, iterating through the loop 2 times, iterating through the loop the maximum number of times and one minus the maximum number of times, and finally trying to exceed the maximum number of iterations through a loop structure by one time. (This is the min -1, min, min +1, max -1, max, and max +1 analysis of the boundary conditions.)

For example, the following method counts the number of characters in a string (actually, it counts the number of Unicode character code points in a string). To boundary test this method we would need to bypass the loop once by passing an argument of an empty string (minimum -1), then a string of one character "a" (minimum), and a string of 2 characters "ab" (minimum +1). Next, we would test the maximum range with a string of 2,147,483,646 characters (maximum -1), 2,147,483,647 characters (maximum) and 2,147,483,648 characters (maximum +1). The ToCharArray method will copy a maximum number of Unicode characters from a string to a character array equal to a signed 32-bit int type. So, passing this method a string of 1,073,741,824 Unicode surrogate pair characters the actual number of Unicode characters will be 2,147,483,648 which will throw the out of range exception. (This happens to be a common error. Many developers assume one character == one byte or one UTF-16 Unicode code point value.)

   1: private static int GetCharacterCount(string myString)

   2: {

   3:     try

   4:     {

   5:         char[] cArray = myString.ToCharArray();

   6:         int index = 0;

   7:         while (index < cArray.Length)

   8:         {

   9:             index++;

  10:         }

  11:  

  12:         return index;

  13:     }

  14:  

  15:     catch (ArgumentOutOfRangeException)

  16:     {

  17:         throw;

  18:     }

  19: }

However, it is sometimes difficult to identify the boundaries of looping structures unless the tester is familiar with the programming language and/or data types. In the above example, if the tester is not aware of Unicode encoding patterns (especially surrogate pair encoding) and simply tests the physical extreme boundary conditions using only ASCII characters the method will appear to return the correct number of characters in a string up to and including the maximum length of 2,147,483,647 characters. But, passing a string of 2,147,483,647 characters in which even one character in that string is a surrogate pair will cause the ToCharArray method to throw the out of range exception.

Occasionally it may be difficult to even identify looping structures, especially when designing tests from only a black box test design approach. For example, in Window Xp a known defect appeared to allow a device name (LPT1, CON, etc.) as the base file name if the extension was appended to the base filename component in the Filename edit control (I’ll talk more about this defect  later.) A Windows Xp patch attempted to correct this defect; however classic boundary analysis testing easily revealed the defect was only partially fixed as illustrated in the steps below.

1. Launch Notepad
2. Select the File -> Save menu item.
3. In the Filename edit control on the Save dialog enter "LPT1.txt" (without the quotes).
4. Press the Save button
5. Press the Yes button on the error dialog that states the file already exists and asks "Do you want to replace it?’ As illustrated below,

If the patch/update is applied to the system Notepad will return an error message indicating it cannot create the file as illustrated below.

1. Next, press the OK button on the error dialog
2. Repeat steps 2 through 5 above.

Now, notice instead of an error message the Window title of Notepad has changed from Untitled – Notepad to LPT1 – Notepad. But, this is a reserved device name, so how can we save a file named LPT1.txt to the Windows file system? The answer is we cannot! Although the application title reads LPT1 – Notepad a file named LPT1.txt does not exist on the file system. This essentially constitutes a data loss defect because it appears to the user that they saved a file named LPT1.txt. (Yes, I am aware of the other bugs associated with reserved device names as well and shall write about them in the future.)

Now, some of you may ask how I knew to test for a looping structure with the Save dialog? Quite simply; I use a technique I refer to as the deja vu heuristic anytime I encounter an error dialog. (A heuristic is a commonsense rule (or set of rules) intended to increase the probability of solving some problem.)  Anytime I encounter an error dialog I repeat exactly the same set of steps to make sure I get the same exact error dialog. Error handling routines often employ loops and are often prone to defects especially if some variable is initialized inside the loop structure. The deja vu heuristic is designed to analyze the minimum boundary of an error handling routine that employs a loop. The minimum – 1 value is not executing the error path, the minimum boundary condition is executing the path that instantiates the error dialog, and the minimum +1 value is repeating the same steps to execute the same path (or not in case of a defect).  In fact, anytime I execute the same exact steps of an error path and get a different result the underlying architecture of the code is suspect and bound to contain one or more defects.

Looping structures are another common cause of boundary class defects, and this is clearly a case where visibility into the code and in-depth knowledge of data types is advantageous for the professional tester.