I.M. Testy

It has been awhile since my last post, and I apologize for the few folks who follow my rants. Over the last few months I have been busy working on my boat (she is almost 30 years old now and needs some major refitting). Since starting to play hockey again I have been spending a lot of time at the gym and on the ice and trying to keep my body from getting too battered in the games (for I too am older now…let’s just say more than 30). I have also done a bit of  soul searching on the past year or so reflecting on the high points and the low points as well.

So, the refitting on the boat is almost complete and she is ready for a nice long trip to the Gulf Islands starting next week. The Monarch’s Div 7A team finished the regular summer hockey season in first place and playoffs are this week. Winter season doesn’t start until October so I have a month and a half to recuperate. And, I have contemplated a few things to help nurture my professional and personal growth that I will proactively begin working on…beginning with a vacation to the Gulf Islands next week.

But, for now I want to talk more about code coverage. Not the number! Forget the measure! The code coverage percentage is simply a magic metric for pointy-haired managers and other thoughtless chowderheads who like to wave it around as if it actually means something. Look…as a manager I really don’t give a rat’s butt about what percentage of coverage was achieved by testing. If you tell me that your testing achieved 80% code coverage I will probably say, “That’s great! Now tell me about the 20% of the code that you didn’t test!” As a manager what I want to know is:

• Did we run the right tests? Did the test suites we ran give us the information that we need to make good business decisions?
• What is my potential exposure to risk in the areas of untested code and how do/can we reduce that risk?

I have said for years now that the future of professional testing is not simply about beating on a product and finding bugs. The future of testing lies in our ability to design effective tests and critically analyze results in order to provide better information to our primary customers (the decision makers). Code coverage is about analyzing the results and potentially designing additional functional tests, or at least being able to explain why areas of the code are untested.

Did we run the right tests?

The code coverage metric simply measures the number of statements in the code have been exercised by monitoring control flow through the code. Control flow through code is sequential until it hits some type of branch such as an if statement, a for loop, or an exception. For example, when we call the CalculateMonthlyMortgage method below control flows sequentially from the statement in line 6 to the statement in lines 7 and 8 and finally to the return statement in line 9.

   1: public static double CalculateMonthlyMortgage(

   2:   double principle,

   3:   double annualInterest,

   4:   int months)

   5: {

   6:   double monthlyInterest = (0.01 * annualInterest) / 12;

   7:   double monthlyPayment = principle * monthlyInterest /

   8:     (1 - Math.Pow((1 + monthlyInterest), (-1) * months));

   9:   return monthlyPayment;

  10:  }

So, by passing in a value of 250000 for the principle, 4.5 for the annual interest rate, and 360 for the months parameter the method would return an expected value of 1266.71327456471 and we would measure 100% coverage of this method. That’s a happy path test. Yes, it does what we think it is supposed to do. But, what happens when we pass in a value of 0 to the annualInterest parameter? Control flows through the method as described above giving us 100% code coverage, but the return value is NaN, or Not a Number. Similarly, if the value for the months parameter is 0 we get 100% code coverage and the return value is Infinity. Of course, negative values for any of the parameters would also produce undesirable output results.

This example illustrates sequential control flow through a method that contains simple statements. When branching conditions occur in software the control flow through the method becomes a bit more complicated as does the testing required. The following snippet counts the number of characters in a string. But, to prevent a null reference exception we use a predicate or conditional statement in line 4 that branches control flow from line 4 to line 12 if the input argument is null, and from line 4 to the loop structure starting in line 6 if the string is not null. If the input argument is an empty string control will flow from line 6 to line 10 bypassing the inner block that increments the count variable. But, if the input argument is a string of at least 1 character control flows from line 6 to line 8 and loops back to line 6 until all characters in the string are counted. (This previous post shows how you can create a model or control flow graph to map control flow through an algorithm.)

   1: public static int CharacterCounter(string input)

   2: {

   3:   int count = 0;

   4:   if (input != null)

   5:   {

   6:     foreach (char c in input)

   7:     {

   8:       count++;

   9:     }

  10:   }

  11:  

  12:    return count;

  13: }      

These are just 2 simple examples designed to illustrate how we can get high levels of code coverage yet still have bugs lurking in our software. Code coverage tools tell us whether we exercised code; it doesn’t tell us if we ran the right set of tests to expose potential issues.

Next week let’s continue this with some thoughts on analyzing code coverage results to help reduce risk.