I.M. Testy

Originally Published Tuesday, March 06, 2007

My previous post refuting a conjecture by Pradeep Soundarajan suggesting there are no boundary values in software was a bit harshly worded, and to him and the readers I apologize. Occasionally I get a little overzealous. I am sure Pradeep is a great guy, and I must say his reply to me on his blog was rather cordial given the situation. As I told Pradeep, email and blogs are a poor medium for expressing emotion. MichaelB cautioned me about this before, but my Type A personality sometimes takes over, and it is something I need to work on.

Anyway, although my critique of Pradeep’s conjecture was pretty ruthless, the analysis was accurate and the  boundary values Pradeep suggested for testing an Mp3 file are in fact not possible or probable even using the tools he referenced. I am not an expert on Mp3 encodings or decoding technology, but i did know that Mp3 files used standard encoding bit rate formats. A little investigation quickly revealed the bit rates for audio encoding are based on multiples of 8, and the first 32-bits of an Mp3 file contain header information including 4 bits to specify the bit rate index (Layer 1, Layer 2, or Layer 3) and the bit rates as outlined below. (Thanks to Wikipedia some of the specific data I used in my initial rebuttal was incorrect, and I put a single strikethrough that part of the sentence.)

Layer 1 – 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448
Layer 2 – 32, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, 384
Layer 3 – 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320

In his attempt to contest my argument Pradeep said, “Pooh! I am not sure why you don’t know that bit rates can be 33, 41, 57 or any number you want to generate. I recommend you to go through some multimedia test content generation tools like ffmpeg which gives a multimedia tester an edge to generate test content of his choice.”

So, not being an expert I took Pradeep’s suggestion “For those who don’t know tools like ffmpeg, it is impossible. I suggest you explore the boundary of your education on multimedia.” and went home and increased my understanding of Mp3 encodings and the ffmpeg toolset. As I read through some of the API references I found an interesting struct (illustrated below) for bit rate constants. (Now, I am thinking to myself…this is a clue! I am also thinking…hmmm…there might be some real boundary values here!)

   1: static const int sBitRates[2][3][15] =

   2: {

   3:   00085 { {  0, 32, 64, 96,128,160,192,224,256,288,320,352,384,416,448},

   4:   00086   {  0, 32, 48, 56, 64, 80, 96,112,128,160,192,224,256,320,384},

   5:   00087   {  0, 32, 40, 48, 56, 64, 80, 96,112,128,160,192,224,256,320}

   6:   00088

   7:         },

   8:   00089 { {  0, 32, 48, 56, 64, 80, 96,112,128,144,160,176,192,224,256},

   9:   00090   {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160},

  10:   00091   {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160}

  11:   00092   },

  12: // etc...

I also followed up by asking a co-worker who frequently works with ffmpeg to try to encode a Mp3 file with a bit rate of 57 kb/s. Interestingly enough, when he issued the command line parameters we got an error message indicating “Invalid Value.” (I can get a snapshot of the command window, but I really don’t think that is necessary.)

I am still not an expert on Mp3 encodings, but I am fairly certain that Mp3 file decoding algorithms are standardized across the industry. So, let’s just assume for a moment that we can encode a Mp3 file at 57 kb/s, and that file fails to play. Does it really matter? No, because industry hardware simply doesn’t support that encoding, and as long as the Mp3 player didn’t burst into flames there is no business case that would compel someone to try to make it work (at this time)? (I am not suggesting that we only test only “real-world” scenarios here, but I am suggesting that in-depth domain and system knowledge goes a long way in increasing the efficiency and effectiveness of our testing (and can lead to better identification boundary values)).

Now, perhaps I am still missing something, and as I expressed previously I am not an expert on Mp3 encodings, or the use of ffmpeg or other tools to encode Mp3 files. So, I have asked Pradeep to share his knowledge with me in this area and teach me how to encode an Mp3 file with a bit rate of 57 kb/s using a commonly used tool such as ffmpeg (there is no doubt someone can write a customized algorithm to do this), and to also let me know of a commercially available Mp3 player that will decode and play that file. (Because if it can be done I would like to learn how simply because I love to learn new things.)

Many people do assume that boundary testing is quite simple. The actual execution of boundary tests are in fact rather simple; however, discerning the boundary values in any complex software is not as simple as looking at some minimum and maximum values and trying one value below and above each boundary condition. Boundary testing is a systematic procedure to solve a specific type of complex problem (specifically the incorrect usage of data types or constant values, artificially constrained data types, and relational operators). Boundary value analysis doesn’t solve all problems, it is not the holy grail, and its efficacy relies on the testers ability to understand and decompose the data set effectively.The less the tester knows about the data and how the data is used by the program, the less effective they will be in the application of this technique.

I did not intend my previous post to be construed as a personal attack against Pradeep; I am sure he is a bright guy. But, I am challenging his assertion on boundary testing on its technical merit. I hope he replies here (or on his blog) with an example of how to encode an Mp3 file at 57 kb/s, and I will make sure it is posted (or linked) here because I am certainly curious. (I don’t really like the taste of humble pie, but I will eat it from time to time if it helps me learn.)

Originally Published Saturday, March 03, 2007

My friend and teammate Alan knows how passionate I get about certain things on occasion, so he threw me a bone the other day regarding a blog post on boundary testing, or should I say a rather a poor attempt to discredit boundary testing and boundary value analysis as a valuable testing technique. The author of the post made an attempt to diminish the value of boundary testing by stating that he has “never seen a boundary!” and asked, “As a tester have you ever seen a boundary?” Maybe that is a trick question, but I am positive the answer is YES! Let me give you a clue…32767 is a boundary value, 65535, is another one, and there are many more. Also, if I artificially constrain an int in a predicate statement using a relational operator such as if (intValue <= 0) then there is another boundary condition that I would certainly want to analyze. (That is what ISTQB, Myers, et. el. mean by Test boundary conditions on, below and above the edges of input and output equivalence classes.)

Unfortunately, the author’s “conjecture that boundary is not static in software” is simply folly, and demonstrates a lack of understanding of how to identify boundary values, or how to adequately analyze boundary values, and simply assumes that boundary testing is as simple as testing at the extremes ranges. In the post the author describes his ‘experience’ boundary testing an audio decoder that will playback MP3 files encoded between 23 kb/s and 196 kb/s. His assumption is that boundary testing would simply entail testing MP3 files at 22 kb/s, 23 kb/s, and 24 kb/s, and 195 kb/s, 196 kb/s, and 197 kb/s encoded bit rates. Not only is this a bad assumption, it is technically impossible.

The bit rates for an MP3 file using mpeg2.0 encoding are 8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160 kb/s, and the bit rates using mpeg-1 layer 3 are 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256 and 320 kb/s. Modern MP3 players are capable of decoding files encoded with variable bit rates; however, with the exception of LAME encoder, files encoded with variable bit rates adhere to the bit rate encoding standards established by ISO/IEC for Layer I, II, and III bit rate indexes (and even the LAME encoder does not allow bit rate increments of 1 kb/s).

Therefore, since no encoding exists for 23 kb/s, 25 kb/s, 195 kb/s, and 197 kb/s encodings these are simply impossible and suggesting tests at these values indicate a lack of domain knowledge and appears to be simple guessing at what to test. Boundary analysis implies testing on, below and above the boundary condition using actual values, not something arbitrary or made up. If the requirements indicated support for mp3 files using a minimum bit rate of 24 kb/s the min – 1 value is 16 kb/s (not 23) and the min + 1 value is 32 kb/s. (Personally, given the limited number of bit rate encodings I probably would have tested a file encoded for each standard bit rate within each bit rate index (Layer I, II, and II), and also random samples of VBR encoded files within ranges specified by the requirements. And yes, include analyzing files encoded at bit rates just above and just below the stated requirement boundaries.)

The blogger also attempts to draw a parallel between the definition of planets in our solar system and boundary values in software. But, even here the author is misinformed. The analogy to planets illustrates antiquated thought based mostly on speculation and controversy. If by chance you are interested in facts with regards to planet classification look here. (BTW…Pluto is no longer classified as a planet, so once again the assumption that Pluto is a “boundary” of some sort is due to inaccurate or imprecise information or simple guessing. ) Fortunately, I don’t think too many of us have to deal with developers who bicker over the size of a 32 bit integer value the same way astronomers argue about planet classification.

Now, of course, if developers simply changed data types or relational operators randomly throughout the code on a weekly basis, then I am all bought into the argument that there are no boundary values and boundary value analysis is not a valuable testing technique. But, on this planet (the one in the “reality” universe) where most developers are not morons who constantly change data types or relational operators, or constants then it is quite possible to identify boundary conditions and then carefully analyze the ‘possible’ values immediately above and below that boundary condition. And that is a good thing because historical analysis indicates that more than a handful of defects occur at or near boundary values.

But, testers must be aware that boundary values don’t always exist only at the extreme ranges of data types or other variables. Occasionally, there are boundary values/conditions within the minimum and maximum physical ranges of a variable. That’s specifically why ISTBQ suggests boundary testing as boundary conditions on, immediately below and immediately above the edges of input and output equivalence class subsets. Of course, that assumes the person knows how to decompose data into equivalence class subsets. This is where in-depth technical, system, and professional knowledge and skills separate the professional testers from the amateur guessers.

This is a good example why, for those who are really interested in being a professional tester and want to pursue software testing as a career, we should read a few more books on software testing, and a few less on epistemology, cognitive psychology, and metaphysics. (I am not saying these topics are not interesting or important. But, I suspect that most testers can already think for themselves, learn and understand abstract and complex thoughts and apply logic and reason.)