But enough about me. My last 2 posts have been discussing code coverage analysis. The primary purpose of using code coverage tools as either a developer or as a tester is not to try to obtain some magical ROMA number. The biggest value of measuring code coverage is to help us analyze untested areas of code and make informed decisions of whether or not we need to design additional tests to increase test coverage and help reduce exposure to risk.

The last post illustrated how we might use code coverage results to help us design additional tests we might have missed during the execution of any pre-defined tests (automated or manual) and additional exploratory testing efforts. But remember, the goal is simply not to design tests in order to get the tool to report 100% code coverage. In fact, in just about any complex system executing 100% of the statements in code may not be feasible or provide any practical value. This is generally referred to as unreachable code.

For example, let’s look at this (albeit antiquated) code snippet.

The code coverage tool is indicating that this conditional statement has been exercised to its true outcome, but not it’s false outcome. This was a common approach used in 16-bit applications to prevent multiple instances of the same application on a single machine. However, in the 32-bit world hPrevInstance always returns null, which means there is no practical way to make this conditional statement return false.

This is a bit of an obscure example, but is used to illustrate why a greater understanding of the programming language used by the development team would help testers from banging their heads against the wall ‘trying’ different things until someone realizes we could never make this conditional statement return false. By analyzing this section of the code coverage results we might suggest refactoring for a Win32/64 environment, or at least be able to explain why this conditional will not return false. (Remember…it’s all about information.)

Another example of unreachable code is sometimes caused by coding style or possibly unnecessary code. For example, the following lines are also in the WinMain() function that is called when the ‘user’ launches the application.

In this situation when the application initially starts and calls the WinMain() function these 2 conditional statements in WinMain() determine whether the Frog and Car bitmaps are true. Since we just launched the application and the bitmaps have not yet been loaded by any other calls to LoadBitmap() then the conditional statements in lines 104 and 109 will never go true, and lines 105 and 110 will never be executed. Again, following an analysis of the section of the code we can provide information regarding why we can’t design a test to cause these conditional statements to return true without fault injection or code mutation. Additional information that we might provide based on our analysis of the code coverage results may be a suggestion to refactor this code to improve testability.

A similar example of unreachable code is a common coding style involving switch statements where developers included a case statement for each possible value, and also included a default statement. For example in the last post we saw how we saw this code chunk which is essentially the menu structure.

When the menu item is selected the submenu displays the submenu items Start and Exit. When the submenu is displayed the only actions possible is to select the Start submenu item (line 270), or the Exit submenu item (line 274). Without fault injection there is no practical way to execute the default statement in line 277. Again, this may be another example where refactoring could improve testability because if the default statement is removed control flow would simply pass out of the switch block.

However, this is not always the case with switch statements. Here is an example in which a modal message box is displayed and draws 2 buttons; a yes button to restart the game, and a no button to quit the game. But notice that the default case statement (line 295) has not been executed.

In this situation if we launch the game, move the frog to get hit by a car this modal message box will display showing the ‘end user’ 2 possible buttons to press (in this case the ESC key or the Close control button in the upper right corner of the modal dialog are not possible options). However, if we put the game into a state where this modal dialog is displayed and then kill the application process using Windows Task Manager control flow will pass to line 295 as the process terminates.

Of course, it may not be practical or reasonable to terminate the application process from every possible machine state. Also, this simply increases the costs of testing without adding any real practical value. Providing this information to the decision makers along with suggestions to refactor and improve testability to reduce overall testing costs is another way code coverage analysis can be a valuable tool in a tester’s toolbox.

Another example of hard to reach code. In this case the conditional statement is if the RegisterClass() function fails then we want to return false.

The RegisterClass() function is also called within the WinMain() function when the application initially launches. So, while analyzing the code coverage results the question we ask ourselves is, “Without fault injection can we make the conditional statement in line 88 return true, and if so how?”

Well, we can. All we have to do is launch about 450 instances of this application to cause line 88 return true. Now, we have to ask ourselves, “What value does this test provide?” Especially since the code design should only allow 1 instance of the application (although it fails to do that because it is a 16-bit app running on a 32-bit environment and that is the nature of hPrevInstance as explained earlier.

From a testing perspective the primary goal of code coverage is not to achieve some magic number; the objective of code coverage is to analyze code coverage results in order to

• Improve test coverage
• Reduce overall risk
• Potentially increase testability of the project

The code coverage number is not really useful information to anyone. It is the analysis of the code coverage results that can help us decide whether we need to design additional tests, identify areas of the code that can’t be executed without even more expensive testing such as fault injection and/or code mutation, or refactor the code to improve testability (which often increases the code coverage measure).

But, this is not to suggest that we should employ code coverage and analyze the results for all software projects. Analyzing code coverage results and designing additional tests from a white box perspective, or refactoring code are all additional expenses for any project. For each project we (or our managers) must decide whether the cost is worth the improved coverage and potentially a reduction in overall risk.

Another way to look at it…it is our responsibility as testers to provide valued information to the decision makers. If the only information we are providing is that we achieved 80% code coverage then we really aren’t doing an effective job. Yes, many managers are number focused; however, the valuable information is in the rest of the story about the 20% that has not been executed.

[NOTE: This was written last week but due to a glitch did not get automatically posted before I left on a boat trip where I disconnected from the world. How refreshing…but more about that later.]

Well, we got a buy in the quarter final play-offs, and we won our semi-final game against the Sockeyes (2 – 0) last Tuesday. Unfortunately, while celebrating with my teammates I was literally knocked to the floor by a kidney stone in my bladder. After a trip to the hospital I am now taking a battery of pain-killers until I pass the stone. Unfortunately, this put me on the injured roster and knocked me out of the final game and until I get an OK from the doctor. Fortunately, I am not prone to kidney stones and this shall pass (literally). I have only had one kidney stone in the past and that was about 20 years ago. For those of you who have not experienced a kidney stone, trust me on this…be very, very glad and I hope you never experience this malady.

Last week I attempted to to illustrate how we might achieve high levels of code coverage (structural control flow) but potentially overlook critical tests, especially from a ‘black-box’ testing approach. The bottom line message…high code coverage does not necessarily equal good test coverage. In reality it is really unlikely to get 100% measured code coverage of an reasonably complex application under test. Unfortunately this often begs the question, “What is the right amount of code coverage for [my] application?” To which I have heard several leads/managers reply, “Our goal is 80% code coverage?” Really? C’mon…that’s just plain ROMA data. Setting arbitrary goals for code coverage is about as pointless as tits on a boar hog. The real answer is that we simply don’t know what measure of code coverage is the ideal level for any product.

Also, for those who have read this article will know that regardless of the testing approach at some point the effectiveness of our tests to hit untested areas of code diminishes. While more time and effort may increase data flow coverage and expose issues, it is unlikely to increase control flow (code) coverage. Remember, just because you exercise a line of code doesn’t mean you found all the bugs, but you have 0% probability of finding any bugs in any untested code.

Fortunately, the majority of customers will traverse the same code paths covered by many of our tests. Also, if your team/organization does robust unit testing then there is a good probability that unit tests at least provided some minimal level of code coverage. (NOTE: While I highly recommend that unit tests and coverage results should be transparent to the test team, I do not recommend using unit tests as part of the battery of tests designed by the test team and executed against the whole build to measure code coverage.) So, there are a couple of questions we have to ask ourselves.

“Does the untested code present significant risk to our customers?”

“Do we need to reduce exposure to risk in the untested areas of code?”

“What is the most efficient way to effectively evaluate the untested code?”

As I wrote last week, code coverage is not about the number. Code coverage is about analyzing the results and potentially designing additional functional tests, or at least being able to explain why areas of the code are untested. If we determine that it is important for our business to better understand the untested code, or improve overall confidence and reduce potential risk then we should use a tool to measure code coverage. But, again it is not about simply measuring code coverage and reporting some magical metric.

Code coverage analysis is the most efficient method to help testers evaluate untested code. Code coverage analysis basically involves the tester reviewing untested code reported by the code coverage tool and determining why some code was not exercised, and possibly design additional tests to exercise the previously untested code. (Remember I also wrote last week the future of professional testing is about analyzing information and designing tests…so, here we go!)

Missing tests

For several years I’ve used the triangle simulation to help set a ‘test effectiveness’ baseline for new testers who had never been formally trained in different test techniques, patterns, or approaches. After a few years of analyzing the results we found that there was about a 70 to 75% probability of tests exercising true branch of the first conditional expression in a compound predicate statement in a key method in the program. There was about a 20 to 25% chance of tests exercising the true branch of the second conditional expression, and there was less than a 10% probability of tests exercising the third conditional expression in the predicate statement. When I found these results I could hardly believe it, so I changed the third conditional expression to inject a bug and sure enough the results held true; in any class of 20 people on average only 1 or 2 people found the bug in the software.

From a black box approach let’s say our tests used the following values for sides A, B, and C respectively:

1. 1, 2, 3 – an error message indicating the values would not produce a triangle
2. 2, 1, 3 – an error message indicating the values would not produce a triangle
3. 4, 5, 6 – scalene triangle
4. 2, 1, 2 – isosceles triangle
5. 5, 5, 5 – equilateral triangle

In this case our code coverage tool would report our coverage is less than 100%. As we drill down we see that the IsValidTriangle() method illustrated below is not completely covered. So, (assuming the arguments values passed to the parameters in this method are all validated to be greater than 0) we analyze the code below in our coverage tool and realize that we need a test to evaluate the third conditional expression to true (e.g. 1, 3, 2 for sides A, B, and C respectively).

   1: internal bool IsValidTriangle(int sideA, int sideB, int sideC)

   2: {

   3:   bool result = true;

   4:   if ((sideA + sideB <= sideC) || (sideB + sideC <= sideA) || (sideA + sideC <= sideB))

   5:   {

   6:     result = false;

   7:   }

   8:  

   9:   return result;

  10: }

So, the refitting on the boat is almost complete and she is ready for a nice long trip to the Gulf Islands starting next week. The Monarch’s Div 7A team finished the regular summer hockey season in first place and playoffs are this week. Winter season doesn’t start until October so I have a month and a half to recuperate. And, I have contemplated a few things to help nurture my professional and personal growth that I will proactively begin working on…beginning with a vacation to the Gulf Islands next week.

But, for now I want to talk more about code coverage. Not the number! Forget the measure! The code coverage percentage is simply a magic metric for pointy-haired managers and other thoughtless chowderheads who like to wave it around as if it actually means something. Look…as a manager I really don’t give a rat’s butt about what percentage of coverage was achieved by testing. If you tell me that your testing achieved 80% code coverage I will probably say, “That’s great! Now tell me about the 20% of the code that you didn’t test!” As a manager what I want to know is:

• Did we run the right tests? Did the test suites we ran give us the information that we need to make good business decisions?
• What is my potential exposure to risk in the areas of untested code and how do/can we reduce that risk?

I have said for years now that the future of professional testing is not simply about beating on a product and finding bugs. The future of testing lies in our ability to design effective tests and critically analyze results in order to provide better information to our primary customers (the decision makers). Code coverage is about analyzing the results and potentially designing additional functional tests, or at least being able to explain why areas of the code are untested.

Did we run the right tests?

The code coverage metric simply measures the number of statements in the code have been exercised by monitoring control flow through the code. Control flow through code is sequential until it hits some type of branch such as an if statement, a for loop, or an exception. For example, when we call the CalculateMonthlyMortgage method below control flows sequentially from the statement in line 6 to the statement in lines 7 and 8 and finally to the return statement in line 9.

   1: public static double CalculateMonthlyMortgage(

   2:   double principle,

   3:   double annualInterest,

   4:   int months)

   5: {

   6:   double monthlyInterest = (0.01 * annualInterest) / 12;

   7:   double monthlyPayment = principle * monthlyInterest /

   8:     (1 - Math.Pow((1 + monthlyInterest), (-1) * months));

   9:   return monthlyPayment;

  10:  }

So, by passing in a value of 250000 for the principle, 4.5 for the annual interest rate, and 360 for the months parameter the method would return an expected value of 1266.71327456471 and we would measure 100% coverage of this method. That’s a happy path test. Yes, it does what we think it is supposed to do. But, what happens when we pass in a value of 0 to the annualInterest parameter? Control flows through the method as described above giving us 100% code coverage, but the return value is NaN, or Not a Number. Similarly, if the value for the months parameter is 0 we get 100% code coverage and the return value is Infinity. Of course, negative values for any of the parameters would also produce undesirable output results.

This example illustrates sequential control flow through a method that contains simple statements. When branching conditions occur in software the control flow through the method becomes a bit more complicated as does the testing required. The following snippet counts the number of characters in a string. But, to prevent a null reference exception we use a predicate or conditional statement in line 4 that branches control flow from line 4 to line 12 if the input argument is null, and from line 4 to the loop structure starting in line 6 if the string is not null. If the input argument is an empty string control will flow from line 6 to line 10 bypassing the inner block that increments the count variable. But, if the input argument is a string of at least 1 character control flows from line 6 to line 8 and loops back to line 6 until all characters in the string are counted. (This previous post shows how you can create a model or control flow graph to map control flow through an algorithm.)

   1: public static int CharacterCounter(string input)

   2: {

   3:   int count = 0;

   4:   if (input != null)

   5:   {

   6:     foreach (char c in input)

   7:     {

   8:       count++;

   9:     }

  10:   }

  11:  

  12:    return count;

  13: }      

These are just 2 simple examples designed to illustrate how we can get high levels of code coverage yet still have bugs lurking in our software. Code coverage tools tell us whether we exercised code; it doesn’t tell us if we ran the right set of tests to expose potential issues.

Next week let’s continue this with some thoughts on analyzing code coverage results to help reduce risk.

Of course, I am a bit of a hard head (even back then) and one day I started playing with the wood lathe while my grandfather was upstairs. Everything seemed to be going pretty well until I pushed the chisel in too far too fast and the wood split and went flying. One piece shattered the overhead light and the other piece ricocheted off the back of my hand leaving an nice gash. I shut off the machine and ran upstairs. After my grandmother cleaned and wrapped my hand, my grandfather made me go back downstairs and clean up the mess and stood over me with a stern look of disapproval making sure I wiped up my blood trail. After that incident, I heeded my grandfather’s advice, at least in his basement shop.

Anyway, with the recent discussions of code coverage around the testing blogosphere I started thinking about what was really being discussed. The discussions (as is the case with most discussions about code coverage) were not actually about the application code coverage as a tool, but more about the code coverage metric. And more specifically the discussions were about how not to assume a high measure of code coverage implies something is well tested. Interestingly enough, 2 years ago I wrote a post illustrating how the metric can be gamed and how the code coverage measure tells us nothing about quality or test effectiveness, but also alluded to how it might be used more effectively.

I thought that how the metric is sometimes misused is mostly self-evident, but then I realized that almost every time testers start talking about code coverage the discussion tends to focus on the metric. This may seem a bit harsh, but if a person’s only contribution to a conversation about code coverage is about how the metric doesn’t relate to quality or testing effectiveness then that person should not be allowed to play with hammers, and employing more complex tools such a wheel-barrows are well beyond that person’s comprehension.

Only thinking of code coverage as a means to get some magic number is akin to thinking “how many nails can I pound with this hammer. The metric itself is mostly irrelevant; and it is completely irrelevant if you don’t know how to interpret it in a way that helps you as a tester. Think about it this way; if we told our managers “our tests achieved 80% code coverage” some of our managers would be elated. (Of course IMHO, these types of managers are metric morons.) But, what do you think these same pointy headed number zombies would say if we told them “we ran our tests and we only missed testing 20% of the code.” I suspect they would start pacing back and forth in the room mumbling “We must run more tests, we must run more tests.”

When we stop thinking of code coverage as a simply measure where our only use of the tool is to try and achieve some magical number then perhaps we can start thinking about how to actually use code coverage as an effective tool to help us design tests (in under-tested or untested areas of the code), reduce potential risk, and possibly even drive quality upstream.

For example, one of my mentees is currently working on a project that uses just in time code coverage as a tool to evaluate how tests exercise changed code and downstream dependencies prior to checking code changes (e.g. bug fixes) back into the main tree. The initial pushback by some members of the team (including some pointy headed managers) was “code coverage doesn’t tell us about product quality” or “its too hard to achieve 80% code coverage” (although no such goal had been mentioned), and my personal favorite, “it’s too difficult to get everyone to measure coverage.” I reminded my mentee that the project is not about achieving some magic number, and in fact, it’s really not even about measuring at all. It’s about using the tool to discover information and to help us design additional functional tests at the API or component level that we might otherwise overlook to help prevent downstream regressions. In a nutshell, its about using code coverage as a defect prevention tool in this case.

Bottom line, code coverage is a tool! If you don’t know how to use it to improve your testing, well…

Teaching my daughter about bullet seating depth.

One of my hobbies is shooting CMP matches and long range precision shooting. Besides lots of practice perfecting the techniques a big part of precision shooting depends on the ammunition and studying the ballistic patterns of various loads. All precision shooters custom load their ammunition and it is not as simple as simply reading a reloading manual. Slight variations of .001” of an inch in seating depth of a bullet or .1 grain of powder may determine whether the group of shots at a target 600 yards away is 1” MOA or 6” MOA. So, getting the ammunition to match the rifle requires continually analyzing your shots, making slight adjustments to the load, and repeating; in computer jargon we might call that refactoring. Reloading for precision is a continually optimizing process until we find the optimal load. Similarly, one of the things we do in the Engineering Excellence group at Microsoft is to continually analyze our internal processes and practices to see how we can help our business groups constantly improve and optimize towards their target. One of the big things on our plate these days is testability.

In Testing Object-Oriented Systems: Models, Patterns, and Tools, a book I consider one of the most important books on software testing practices, the author Robert Binder defines testability as “The relative ease or difficulty of producing and executing an economically feasible test suite to determine whether the [system under test ] SUT (i) conforms to stated requirements and specifications, and (ii) exhibits an acceptably low probability of failure.” This and several definitions of testability floating around on the web and all generally agree that testability generally involves

1.) The ease with which the SUT can be tested
2.) The cost of testing is reasonable

So, as the testability increases the ease with which our tests can determine whether the SUT satisfies implicit and explicit requirements and has a lower chance of failure at reduced testing costs. This all sounds nice, but unfortunately testability cannot be directly measured; testability is a qualitative measure. Although we can’t accurately measure testability we can sometimes do small things to improve the characteristics of testability and help reduce testing costs by reducing the number of tests required to determine whether the SUT satisfies the stated requirements and also has a low chance of failure, or finding ways to test more efficiently through better designs.

In last week’s post I referred a pseudo code example that was written to illustrate how bugs could linger in code despite a high measure of code coverage. Of course we should realize that pseudo code is generally a far cry from the real implementation of the code. Pseudo code is simply a model, and there are many ways to implement that model. The advantage of a model is that we can often test a model earlier to identify potential issues before a single line of code is written. In this particular pseudo code sample, there were a couple of things that stood out that could likely impact the testability of an implementation of the pseudo code model. So, the neurons in my brain starting firing with lots of testing related questions.

So, let’s use that example to discuss potential testability issues. The sample was based on a requirement that stated “Student ID’ are seven digit numbers between one million and 6 million inclusive.” The function is relatively simple in that it takes a string type passed to the sid parameter, and returns a Boolean true or false to the calling function depending on whether the string satisfies the internal Boolean conditions it is being compared against. But this function also calls 2 other functions; the length () function, and the number () function. From the function names I would think the length () function provides a numeric value that represents the number of characters in the string passed to the sid parameter. I am also betting the number () function returns a numeric value (it converts the string variable to a numeric type such as an integer. The pseudo code example was

function validate_studentid(string sid) return
TRUEFALSE
BEGIN
  STATIC TRUEFALSE isOk;
  isOk = true;

  if ((length(sid) is not 7) then
    isOk = False;

  if (number(sid) <= 1000000 or number(sid) > 6000000 then
     isOk = False;

  return isOk;

END

One of the reasons that we hire testers with a programming background at Microsoft is that they can help the developer identify potential issues, reduce the probability of failure, and improve testability by stepping through the code during peer reviews, or while designing additional tests to cover un-tested or under-tested areas of the code that are exposed by code coverage analysis. So, when I come across a code sample, I generally step through it to

• See if it will work as intended (basic unit test)
• See if there are any potential obvious errors in logic
• Identify tests necessary for branch or conditional coverage (because developers are usually only concerned with block coverage)
• Identify argument values for negative testing that might expose undesirable results (bugs)

So, in this pseudo code example, once I got to the second conditional clause (if (number (sid)) <= 1000000 or number (sid) > 6000000 then) the little cranks in my brain began to turn. I thought to myself, why are we checking the length of the string? I mean, if the number can only be between 1,000,000 and 6,000,000 then it seems to me that checking the length of the string is simply redundant.

If we remove the first conditional clause (if ((length(sid) is not 7) then) then we actually reduce the number of tests to 3 instead of 4 assuming short-circuiting since short-circuiting compound Boolean expressions is one of several code optimization techniques. (By the way, the first caveat example in Wikipedia on short-circuiting where a function used as a Boolean conditional also “performs some required operation regardless of whether the first conditional evaluates true or false” is simply poor architectural design and is very, very likely to be problematic.) The 3 tests for condition (and basis path) coverage to exercise the true and false outcome of every single Boolean conditional expression are listed in the table below.

Of course, even testing several samples from the equivalent partitions may not expose the bug in this code because the bug in this code is a typical boundary error. (In a previous post I explained the basic fault model that caused many boundary issues. In a nutshell, boundary bugs are generally caused by incorrect relational operators or magic numbers in code.) Without recognizing that we also need to test the boundaries (999999, 1000000, 1000001, and 5999999, 6000000, 6000001) also we could easily overlook the error in the pseudo code.

Another thing that caught my attention was the lack of exception handling. Some people may not consider including exception handling in pseudo code and take it as a given. But, as a tester when I don’t exception handling in pseudo code in a review then I need to start asking questions so I can better design tests to exercise the exception handling control flow paths that directly impact code coverage measures. Another reason this is an important consideration is because results of code coverage analysis indicates that exception handlers are generally under-tested. It seems we are really good at finding unhandled exceptions with our negative tests (which is really good), but we do not seem to be as thorough in testing the logical code paths of exception handlers. This is especially true for predicate statement with multiple Boolean sub-expressions might trigger an exception. We tend to test one of the conditionals, and the other conditionals expressions in that statement are often under-tested.

So, we can surmise the number () function must be converting the string parameter (the sid variable) to a numeric type and returning a type of number because the conditional clause is comparing it to magic numbers (1000000 and 6000000). But if we entered a string that contained non-numeric characters my initial thought was that the number () function would throw an exception that is unhandled by the validate_studentID () function.

Then I thought a bit more, and considered that the number () function might swallow the exception and return a 0 or even a -1. Now, there are some arguments in favor of swallowing exceptions, but in general it is not a good idea. In this case, it is probably a bad idea because one of the primary purposes of a separate function is reusability. If the number () is reused in some other code, or other part of the code where we need to convert a string to a numeric type regardless of the range (within the range of the data type being converted to), I would suspect we would want to throw an exception, and then rethrow the exception in the calling function. Of course, this is where the rubber hits the road, and a professional tester needs to dig in and start asking some hard questions as to how the developer is going to handle this situation. If the number () function is not going to be reused, then most modern programming languages include a function call that will easily convert the string to a numeric type and do it more efficiently as compared to calling a separate function. And may in that case we could swallow the exception in the validate_studentID () function and simply return false as illustrated in the C# code below.

   1: try

   2: {

   3:     if (int.Parse(sid) < minValue || int.Parse(sid) > maxValue)

   4:     {

   5:         isOk = false;

   6:     }

   7: }

   8: catch (FormatException)

   9: {

  10:     isOk = false;

  11: }

  12: catch (OverflowException)

  13: {

  14:     isOk = false;

  15: }

With the push to drive quality upstream, reduce costs (especially testing costs), and improve testability I envision that many testers will be working alongside our development counterparts to help them prevent defects from getting into the product code base, and improve the maintainability of the code. This doesn’t mean that testers will become developers or visa versa; it simply means that testers are (generally) experts in designing tests, and developers are experts in designing solutions that adhere to requirements. Rather than an adversarial relationship, I suspect in the future developers and testers will have a more symbiotic relationship to improve the intrinsic quality of our code bases.

The bottom line of all this is that in teams where testers are designing white box tests for improved code coverage (control flow testing), or where testers are engaged in design reviews or peer reviews of code prior to check in, I hope this gives you some things to think about.

I read a lot of articles, white papers, and books. I like most of what I read, even if I disagree with some of the points being made. I can’t remember ever reading an article on software testing that ever made me angry. I was not angry because of the message of the article. In fact, I think the point the authors are trying to make is valid and I agree with them on their fundamental point. Unfortunately, the article is filled with technical inaccuracies the end message was almost lost.

I spent the last 10 years studying various techniques, methods, and approaches in software testing. I teach more than 500 testers a year on structural testing techniques, and am now working with a team in the Windows division to implement a new tool for just in time code coverage analysis at the component level that allows us to see how our tests exercise code paths in changed code and the dependent modules. I also discuss structural testing in chapter 5 of our book How We Test Software At Microsoft. I don’t really consider myself to be an expert in the subject, but I might know a thing or two about it. So, let’s Reconsider Code Coverage!

In August 2007 I wrote an informative blog post on the potential misuse of the code coverage measure. But code coverage measures are used by some companies as one of many ways to help them reduce risk. And, let me be very clear here, there is no correlation between code coverage and quality, and code coverage measures don’t tell us “how well” the code was tested. The code coverage measure simply measures what code has been executed, and more importantly what code has not been executed. The value of measuring code coverage is not in producing some “magic number,” but that it helps testers investigate untested or under-tested areas of the product and design additional tests (generally using structural testing techniques) to improve coverage and reduce overall risk.

Just because you execute a line of code doesn’t mean a bug doesn’t still exist, but if you don’t execute a line of code you have 0 probability of finding a bug if one exists!

Also it is important to note there are several ways to measure code coverage. Different tools employ different measures and sometimes different tools measure the same type of coverage differently. Also, I discovered that even the same tool can measure the same code differently depending on how it is compiled (debug, retail, etc.) and previously wrote about my study. Some of the basic ways to measure code coverage (not test coverage) include:

• Function coverage measures the percentage of functions or methods in a class or application that are called at runtime.
• Statement coverage measures the percentage of executable statements exercised at runtime.
• Block coverage measures the percentage of each sequence of non-branching statements that are executed at runtime. Block coverage subsumes statement coverage.
• Decision or branch coverage measures the percentage of both Boolean (not binary) outcomes (true and false) of simple conditional expressions at runtime. If a predicate statement has more than one conditional sub-expression decision (or branch) coverage treats that predicate statement as one conditional clause. Decision coverage subsumes block coverage.
• Condition coverage measures the percentage of both Boolean outcomes of each conditional sub-expressions that are separated by logical and or logical or in compound predicate statements. Condition coverage subsumes decision coverage.
• Basis path coverage measures the number of linearly independent paths through a program. Basis path coverage is based on McCabe’s cyclomatic complexity research.
• Path coverage measures every possible path from the entry to the return statement (or exception) or exit of every method. Unfortunately path testing is usually impossible due to the sheer number of path combinations, and the inability to execute constrained path combinations.

Clearly there are different measures of code coverage, and certain types of measures subsume other measures. So, now that we have a handle on the different types of code coverage measures, let’s look at testing some code. We will use the same pseudo code used in the aforementioned article which is based upon the following requirement.

“Student ID’ are seven digit numbers between one million and 6 million inclusive.”

The authors provided the following pseudo code example for a function to meet this requirement.

function validate_studentid(string sid) return
TRUEFALSE
BEGIN
  STATIC TRUEFALSE isOk;
  isOk = true;

  if ((length(sid) is not 7) then
    isOk = False;

  if (number(sid) <= 1000000 or number(sid) > 6000000 then
     isOk = False;

  return isOk;

END

So, other than the fact that there is no reason to ‘test’ the length of the sid variable before evaluating it to see if it is within the allowable range (removing this first conditional improves performance and also improves testability of the code), and that if the call to the number() function fails to convert the string to a number for a valid Boolean comparison it will throw an unhandled exception, let’s look at path testing of this simple example by starting with control flow diagrams of each possible path (assuming the call to the number() function does not throw an unhandled exception by passing this message a string of characters such as “foo” rather than a string of digits).

Control flow diagram for validate_studentID() function pseudo-code

(Edited 11/25: After thinking about this a bit more, if the number() function returned a 0 (zero) if the input was incorrectly formatted, then the number() function would not throw an exception, and the control flow path would be identical to the first test in the table below).

Because we are doing path coverage testing and not decision testing, we actually have to separate each Boolean conditional sub-expression in the second compound predicate statement if (number(sid) <= 1000000 or number(sid) > 600000. The example in the article treated both sub-expressions in the compound predicate statement as a single Boolean expression which would be synonymous with decision coverage. Path coverage actually treats each sub-expression as if there were 2 single Boolean conditions such as

if (number(sid) <= 1000000
  isOk = False;

if number(sid) > 600000
  isOk = False;

The table below illustrates the tests required for testing control flow through this function for path coverage (again assuming we are going to ignore the unhandled exception in the code that would occur by passing in a string such as “foo.”)

The first test would be a value less than 7 digits, and would cause all Boolean conditional expressions to evaluate as true which will set the isOk variable to false (3 times), and we correctly return the expected result of false (or invalid ID). The second test is a number greater than 6,000,000 (but less than the maximum value that would result in an unhandled overflow exception hopefully being thrown by the number() function). In this case the 3rd conditional expression (if (number(sid) > 6000000) would evaluate as true and the function would return false. The 3rd path is buggy. In this pseudo code example, the only possible way to exercise the true outcome of the Boolean condition if (number(sid) <= 1000000 is to use the value of 1,000,000; any other value larger or smaller will cause this Boolean condition to evaluate as false. In this case we expect the function to return true, but it in fact will return false. Finally, any number greater than 1000001 and less than or equal to 6000000 will return a true result indicating a valid student ID.

The article also suggest that structural testing misses other problems. But, when we look at these issues, they actually have nothing to do with structural testing of the function; in other words they are completely out of context of the problem being discussed.

For example, the assert is the requirement is incorrect and should have read 6,999,999 (which I believe is a typo and should be 5,999,999) because of confusion over the word “inclusive.” Inclusive means “including the stated limit or extremes in consideration or account,” but in computing inclusive means “the predicate holds for all elements of an increasing sequence then it holds for their least upper bound.” I disagree with this assumption because I suspect the analyst writing the spec is basing the inclusive range on the common definition, and not a definition based on domain theory.

The article questions what would occur with incorrectly formatted numbers such as 123 456 789 or 123,456,789. So, beside the point that these values are not within the valid range of student id numbers, the answer to the question would actually lie in how the number() function being called handles improperly formatted numbers (e.g throwing a format exception, which again is unhandled in our validate_studentid() function), or how an event handler that sits between the UI and the function might deal with invalid or incorrectly formatted inputs.

The next question concerned resizing of the input window or the screen (assuming desktop resolution) and repainting the window or form and its affect on code coverage of the validate_studentid() function. Well, I am going out on a limb here and I am going to say…”what are you talking about?” I am not quite sure how to phrase this, but let me try…resizing or repainting a window has 0 effect on the structural control flow of the validate_studentid() function. (Of course, I could be wrong, and the length() function number() function might have some code that mysteriously interacts with the repainting libraries and how it determines the length of a string or whether a string is a valid number.)

Bugs in external libraries are part of the business. Hopefully those external libraries are well tested or at least documented especially if our development team wrote them. Personally, I have not encountered any public functions or APIs which use wild ass random numbers such as 5.8 million as boundary values, but that’s not to say it couldn’t happen. And of course, if these external functions throw exceptions (as they should based on what they are probably doing), we should have exception handler code in our function to deal with any exceptions thrown from external libraries or function calls.

Based on incorrect path analysis, and out-of-context questions that have nothing to do with control flow through the validate_studentid() function the article suggests that path testing is not a magic potion, but I am not too sure that anyone actually believes it is. And so, the article suggests that “input combinatorics coverage” might work better. Hmm…now I have been teaching combinatorial testing for over 10 years and have read some interesting papers on the effectiveness of combinatorics on statistical testing and code coverage, and I must say I pretty sure you need more than one input parameter in combinatorial testing!

Finally, I don’t agree that code coverage measures tell us “how well the developers have tested their code.” The code coverage measure only tells us what percentage of the code has been executed in a particular way, and more importantly it tells us how what percentage of code has been untested. We must determine whether we need to investigate that area to reduce risk. Of course, many code coverage tools provide a “heat map” that helps us and developers identify untested code, and that is where we shift from the simple act of measuring coverage to the testing method of code coverage analysis in order to design new tests that effectively exercise previously untested code if that level of coverage is important to reduce overall risk.

My intent here is not to ridicule the authors of the article. In fact, I agree with their summation that testers should not believe high code coverage numbers mean “well tested.” (Again see my blog post from Aug 2007.) Unfortunately, the path to the point was fraught with inaccuracies and tangents that I almost never made it to the end.

There are many books and white papers on this subject in the ACM and IEEE libraries. Books by Boris Beizer, Robert Binder, and others go into great detail on structural testing. McCabe’s papers linked to in this post are an excellent resources.

OK…I feel better now. I need to clean up the blood, take a sedative, and go to sleep.

In the book How We Test Software at Microsoft I discuss structural testing techniques. Structural testing techniques are systematic procedures designed to analyze and evaluate control flow through a program. These are classic white box test design techniques, although my friend and respected colleague Alan Richardson states in his review of the book that he also employs similar techniques on models and I have to agree with him on that point.

Also, Peter M. sent me mail pointing out a reasonably obvious bug in the code chunks on pages 118 and 119. Both functions are declared as static void, but each has a return statement. Somehow this oversight made it through the review process, but of course a return statement in a function declared as static void would cause a compiler error. (Thanks for discovering that bug Peter and letting us know so we can fix it for the 2nd edition!)

Peter also asked for further clarification of how blocks are counted, and why a test that evaluated both conditional clauses in the compound expression as true in the below example (and on page 119) results in 85.71% coverage. Unfortunately, the answer for that is not simple.

Some surprising details…

   1: public static int BlockExample1(bool cond_1, bool cond_2)

   2: {

   3:   int x = 0, y = 0, z = 0;

   4:   if (cond_1 && cond_2)

   5:   {

   6:     x = 1;

   7:     y = 2;

   8:     z = 3;

   9:   }

  10:  

  11:   return x + y + z;

  12: }

The above code can be re-written as:

   1: public static int BlockExample2(bool cond_1, bool cond_2)

   2: {

   3:   int x = 0,

   4:   y = 0,

   5:   z = 0;

   6:   if (cond_1)

   7:   {

   8:     if (cond_2)

   9:     { 

  10:       x = 1;

  11:       y = 2;

  12:       z = 3;

  13:     } 

  14:   }

  15:  

  16:   return x + y + z;

  17: }

First, a ‘basic block’ is defined as a set of contiguous executable statements with no logical branches which seems pretty straight forward. So, based on our definition of basic blocks it appears there are 4 blocks of contiguous statements. However, the conditional clauses on line 4 and line 6 in the BlockExample2 method introduce logical branches which theoretically introduce 2 implicit blocks (e.g. one block when control flow follows the true path, and another block when control flow follows the false path). So, that is essentially how the 6 blocks are determined. But, that’s not the end of the story.

If we pass a Boolean true to both cond_1 and cond_2 conditional clauses the block coverage measure in BlockExample1 results in 85.71% coverage; however, the block coverage measure for BlockExample2 actually results in 100% coverage as illustrated below.

What? How can this be? Both BlockExample1 and BlockExample2 are syntactically identical. Well, to understand this we would really need to dig deeper into compilers and coverage tools. That is well beyond the boundaries of this blog, but the IL does provide some insight.

The MSIL for BlockExample1 is on the left and BlockExample2 is on the right. Now, I don’t want to do a deep dive into MSIL, but  those who are really observant can see that for some reason the Visual Studio compiler evaluated a branch in BlockExample1 to false (instruction IL_0008), and then instruction IL_000c compares the 2 values for equality and instruction IL_0015 appears to evaluate the optimized compound conditional expression to true. Compare that to BlockExample2 MSIL which shows the first comparison of 2 values occurs at IL_0009 and the branch is evaluated as true (IL_000f) and the second comparison of 2 values occurs at IL_0014 and again evaluates to true at instruction IL_001a.

But wait…it gets even more confusing. We typically measure structural coverage using the debug build. So, imagine my surprise when I recompiled the code using the retail build settings and again passed true arguments to the cond_1 and cond_2 parameters for BlockExample1 and BlockExample2 and the coverage tool in Visual Studio indicated these methods now only had 4 blocks, and the block coverage measure for both methods was 100% as illustrated below.

Also, interestingly enough the compiler optimized the code so both methods had identical MSIL op code instructions as illustrated below.Steve Carroll (a senior developer in Visual Studio) wrote we "shouldn’t be too concerned if you can’t exactly identify where all the blocks are.  When you turn the optimizer on your binary, block counts are fairly unpredictable. Don’t worry though, the source line coloring will almost always lead you to the parts of the code that you need to worry about targeting to get your coverage stats up."

I agree with Steve when he states block counts are unpredictable when the code is optimized (and different tools that measure block coverage may provide different results). However, I only partially with his statement that source line coloring leading us to parts of the code we need to test. Maybe it will, maybe it won’t. But, professional testers performing an in-depth analysis of code coverage results will help us identify important parts of the code that require further investigation and testing.

So, what does it all mean?

Block testing is useful for unit testing and designing white box tests for switch statements and exception handlers (based on how we can track control flow through source code using a debugger as opposed to through the IL Disassembler). But, as I stated in How We Test Software at Microsoft block testing is the weakest form of structural testing. But, it does provide a different perspective as compared to other structural approaches or techniques and is useful when used by a professional tester in the right context.

But, the important point here is that just as we wouldn’t rely on only one tool to tune the carburetor on an automobile, we certainly would rely on only one technique or approach for designing structural tests; and we certainly wouldn’t only rely on structural testing as a single approach to testing. This example further reinforces another important point that I make in the book; code coverage is not directly related to quality. Any professional tester can clearly see that although we are able to achieve high levels of coverage with one test, these methods are not at all well tested.

Only a fool would use code coverage metrics to derive some measure of quality, or suggest the implication that high coverage measures equal greater quality. In truth, the value of code coverage is in its ability to help professional testers identify areas of the code that have not been previously exercised and to design tests to evaluate those areas of the code more effectively to help reduce overall risk.

If we don’t execute an area of code then we have zero probability of exposing errors in that code if they exist. However, just because we do execute a code statement doesn’t mean we expose all potential errors. But, it at least increases the probability from 0% and helps reduce risk.

Code coverage is a frequently sought after measure in software testing. Code coverage is an important metric and it should not be ignored; however, as a measure it must not be abused or over-rated, nor should we attempt to correlate code coverage as a direct measure of quality. While many teams strive for higher percentages of code coverage at the system level (which is good), the code coverage metric simply tells us if statements, block of statements, or conditional expressions have been exercised. Low measures of code coverage may sometimes result from software complexity and lack of testability or from testing ineffectiveness, but are generally indicative of a software project in peril (with regards to risk). Higher percentages of code coverage certainly help reduce perceived overall risk, but the code coverage measure by itself doesn’t necessarily tell us HOW it was exercised, and it doesn’t provide useful information about the areas of the code that have not been exercised other than what percentage of the code is at 100% at risk. (If we don’t test it; we can’t qualitatively say anything about it, so risk must be assumed to be 100%.)

Let’s examine the following simple example to explain this search algorithm to better understand how increased measures of code coverage provide less valuable information regarding testing effectiveness. This algorithm searches for a particular character in a string of characters and returns the index position of the character if found; otherwise it returns 0.

   1: private static int CharSrch(string s, char c)

   2: {

   3:   int i = 0;

   4:   int retVal = 0;

   5:   char[] cArray = s.ToCharArray();

   6:  

   7:   while ((i < cArray.Length) && (cArray[i] != c))

   8:   {

   9:     i++;

  10:   } 

  11:  

  12:   if (i < (cArray.Length))

  13:   {

  14:     retVal = i;

  15:   }

  16:  

  17:   return retVal;

  18: }

Using Visual Studio Team System to measure block coverage and executing a test in which s = "" and c = ‘c’ the code coverage measure is only 72.73% for the CharSrch method as illustrated in the figure below. In this example it is easy to understand why the relatively low code coverage measure is giving us valuable information (perceived risk is great, overall confidence is low). Clearly we have more testing to do!

Again using Visual Studio Team System to measure block coverage and executing a test in which the search string is "abc" and the character to search for is ‘c’ the code coverage measure jumps up to 90.91% for the CharSrch method as illustrated in the figure below. Using just the code coverage measure as an indication of test effectiveness we might feel much more confident and perceive our exposure to risk is greatly reduced, and the algorithm is doing the right thing! But, we are still not at 100% (which is easy for this example), so we need just one more test to achieve that magic number.

A third test in which the search string is "a" and the character to search for is ‘c’ we see the resultant code coverage is again 90.91% as illustrated in the figure below. By merging the code coverage results in Visual Studio Team System we can achieve 100% block coverage by merging the results of Test 1 ( s = "a" and c = ‘c”) and Test 2 (s = "abc" and c = ‘c’) . If unit tests were written in such a way as to check for an output of retVal == 0 for Test 1, and an output of retVal != 0 for Test 2, then both tests pass. Overall, my perceived risk is relatively low, and my confidence is relatively high as compared to the first test based on the code coverage measure. But, did we miss something?

Although the percentage of block coverage is relatively high (OK…100% is the max), the information provided by the measure itself is actually less valuable because it may have actually failed to detect the defect in which the CharSrch method returns a value of 0 if the character is not found, and also returns a value of 0 if the search character is the first character in the string.

This simple example is not meant to discount the overall value of code coverage as a software metric. However, as professional testers we must realize that high levels of code coverage do not directly relate to quality, and code coverage is only an indirect indication of test effectiveness. From my perspective, the most important measure with regards to code coverage is not how much has been exercised, but the percentage of code that has been unexercised by our testing. That is the purpose of code coverage analysis (which is a great segue for a follow up blog post).