Skip to content

Boundary Bugs…like shooting fish in a barrel

If there is a bug at a boundary that doesn’t lead to an unhandled exception or security exploit should we care?

Perhaps an even more important question is why do we find so many boundary type bugs via exploratory testing when they can and should be caught earlier? Why don’t we find these types of bugs in our unit testing? Why don’t we find these types of bugs by more systematically testing the software? Maybe we do find them, and those who make the decisions to fix these types of bugs just don’t care if they are fixed because there is no severe negative impact to the user. Maybe someone just wants to give me fodder for my blog!

This week I wanted to compare the range of allowable font sizes for a simulation program I developed as an example for a magazine article that I am working on. I knew that Office applications allow a font size within the range of 1 – 1638. I thought that range might be too large for my purposes, and since I knew that Windows Notepad included a font dialog I decided to check the allowable range of font sizes in Notepad.

The first thing I discovered was that the combobox control allows up to 5 characters! Really? Someone decided it is a good idea to allow users to enter 5 characters? notepad 1

OK, I’ll play along. Maybe if I put in a size of 99999 and press the OK button on the dialog I will get an error message, or at least Notepad defaults to the last ‘valid’ selected font size. That might seem reasonable. But is that what happens? NO! Instead of doing something reasonable (e.g. error message, default font size) the font changes to a size of 1 (yes that is a font size 1 in the upper left corner in the image below).

Notepad 2

I am sure that defaulting to a font size of 1 makes sense when the allowable size value overflows! Really…someone thought that was a good idea? Now I wanted to see what magical boundary value the developer decided was an acceptable font size. Since the combobox size property allowed 5 characters I immediately tried 65535. No, that also resulted in the overflow and displayed the text in a font size of 1. Next I tried 32767. Wait…32767 didn’t display the string in Notepad’s edit control at a font size of 1. Now, I am thinking the developer is using a data type of signed short for the font size variable. So, I enter 32768 expecting the value to overflow and display my string as a size 1 font again. But, no…that doesn’t happen.

Now, when I am design boundary tests I generally rely on 2 heuristics for identifying boundary values for input or output parameters.

  1. Values at the extreme edges of a physical range of values
  2. Values at the edges of equivalence partitions of physical values

So, in these situations I ask myself “What sort of demented developer debauchery have I now found myself?” I can’t think of any other obvious edge values that might apply, so out of curiosity I quickly narrow down the magical value to 39321. I then ask myself, “OK…even if there were a display capable of rendering or a printer capable of printing a font of this size, what is so unique about 39321?” In hexadecimal it is 0×9999, and in binary it is 1001100110011001. OK…nothing obviously special here, but I am certain the implementation details are much more complex then a simple range of values and at this point I really don’t care because this bug just doesn’t make sense.

Maybe it’s not supposed to make sense! Maybe nobody really cares about these types of bugs!

(BTW…somebody please take the Thesaurus away from the developer…’Oblique?’ Are you serious…why not just be consistent and use the word ‘Italic?’)

This was written by Bj Rollison. Posted on Wednesday, March 24, 2010, at 11:58 PM. Filed under General Testing Topics, Testing Practices. Tagged , . Bookmark the permalink. Follow comments here with the RSS feed. Comments are closed, but you can leave a trackback.

6 Comments

  1. Joe Strazzere wrote:

    “what is so unique about 39321?” In hexadecimal it is 0×9999″

    Isn’t “9999″ significant? That looks like some sort of boundary to me.

    I’m imagining a developer, intent on a 4-digit limit…

    [Bj's Reply] Hi Joe, if we add 0×1 hex to 0×9999 the value is 0×999A still 4 (hex) values. (I will be glad to get back to Redmond to take a look at what is actually going on here.)

    Thursday, March 25, 2010 at 3:09 AM | Permalink
  2. Mark Waite wrote:

    I wonder if one of the reasons boundary bugs are not detected in unit testing is due to my natural tendency to unit test for base functionality rather than boundaries? For example, most of my unit tests are focused on verifying that the “happy path” works as I expected while I was developing.

    I also wonder if the PEX tool from Microsoft Research (or something like AgitarOne on Java) should be added to developers’ “unit test arsenal” to detect and explore boundary cases in unit tests.

    [Bj's Reply] Hi Mark, Yes, I suspect that many unit tests are focused on “happy path.” In my opinion we should all be thinking about defect prevention and driving quality upstream. If I can find these types of bugs earlier in the process without significant overhead in my unit or component levels of testing then we know the overall cost of correcting the error is less.

    Also, to paraphrase A. Hunt and D. Thomas, ‘developers don’t get paid to write buggy code.’ I think most devs (at least the ones I know) try their best to check-in high ‘quality,’ defect free code. (Of course there are ‘hacks’ who might be satisfied writing crap code and chucking it over the wall in hopes the some poor schmuck ‘tester’ will whack out their bugs.)

    Also, you’re quite right; PEX is a very useful tool for .NET developers to include in our unit testing that might preclude these types of issues.

    Thursday, March 25, 2010 at 7:22 AM | Permalink
  3. lixiong wrote:

    set the size to 888 and you will see nothing…

    [Bj's Reply] Hi Lixiong, just because we don’t see it doesn’t mean it isn’t there. :-)

    Thursday, March 25, 2010 at 8:39 AM | Permalink
  4. Devon Smith wrote:

    I agree with Mark- QA’s role is to spend time thinking about all the ways things could break- QA is destructive. Our strength is thinking in adifferent way than running unit tests to see if it “works” … we want to see if it works or doesn’t work correctly in any scenario.

    [Bj's Reply] I would respectfully suggest that a view of testing as destructive is antiquated and certainly quite limited in scope of the discipline. Boris Beizer suggested ‘the primary objective of testing is defect prevention, and the secondary objective is defect detection.’ If we can find bugs earlier in the cycle then we should! Driving quality upstream was a goal of testing more than 2 decades ago and it is a goal of agile testing/development today.

    I agree that testing at the system level is important and is required to evaluate how all the pieces play together. But, if I can find these types of issues sooner in the development process that frees up my time to focus on other scenarios at the system level, significantly improves overall code quality, and may even prevent latent issues that may not be exposed via system testing.

    Thursday, March 25, 2010 at 11:02 AM | Permalink
  5. May wrote:

    Just a short side remark. Maybe the developer does not use a Thesaurus at all. Because oblique and italic are not the same thing. At least not if you know a lot about fonts (which I don’t). Have a look at the Wikipedia entry (http://en.wikipedia.org/wiki/Oblique_type).
    Still you might be right though. Because in the German version of notepad it is italic. Or the translator did not know about the difference either? Maybe someone can decide by looking at the text created with notepad. I am not sure but it actually might look as if oblique is right and italic would be wrong.

    Ok enough of this lesson about things you nether wanted to know. :)

    [Bj's Reply] Hi May, this was just tongue-in-cheek. Actually, whether the right slanting style is oblique or italic depends on the font selected. For example, the Lucida Console font style is oblique, but the Arial font style is italic. These styles are properties of the fonts, but I am not sure that most Notepad users (I use notepad a lot still) really care about the difference though. :-)

    But, I appreciate you pointing this out.

    Thursday, March 25, 2010 at 11:20 PM | Permalink
  6. Sherry Chupka wrote:

    great article, I thought the exact same thing the other day when I was trying to decide how much longer I want to stay in testing and what I should do next. I’m still undecided. Some days I wonder what I’m doing wrong and then I realize all software is plagued with the same type of problems. I’ll never understand why boundary bugs are classified as low priority though.

    [Bj's Reply] Hi Sherry, I also occasionally wonder if I am making any sort of positive impact, and I look around me and realize that yes the industry and discipline is changing in some very cool and exciting ways. Not necessarily because of me, but it is changing in ways that I believe in and promote. But, of course, change takes time and as I have gotten older I have become much more patient and as long as I see positive change I think I’ll hang around.

    Software will always be plagued by bugs, but when it is constantly reinfected by the same type of bug then we should find different solutions to prevent the problem. One of the cool things out of MSR recently has been PEX. (see http://research.microsoft.com/en-us/projects/pex/). Now we just have to figure out how to get devs to use it! :-)

    Also, leaders in the industry need to help change the perception of testing. Thinking about testing as a destructive activity is destructive to the discipline. Managers don’t see ‘breaking’ the product as adding value. They do understand they want to identify issues before release, but if we are constantly trapped in a break & fix mindset then the business is not growing. Software testers should find ways to add value to the product lifecycle such as driving quality upstream and partnering with devs to prevent low level issues from creeping into the product. That frees up our time to really think about the end-2-end stuff.

    Tuesday, March 30, 2010 at 5:30 AM | Permalink