This is the way it has been for me during my almost 2 decades in this industry. So, when I read books and papers that compare agile to “traditional” approaches I ask myself, “What in the hell do they mean by “traditional” approach?

For example, when I joined the Windows 95 team we had weekly builds, and at least once per week the build had to satisfy the requirements for “self-host” status. Self-host meant that anyone in the company could use that build for their day to day work. Also, many of the self-host builds are released to key strategic industry partners and 3rd party vendors so they could take advantages of new features in their software.

After the PDC release of Windows 95 we made several key changes in the operating system based on feedback from our customers and partners. Also, late in the cycle after beta 2 we had to make a major change in how Windows started because of a quirk in a relatively popular MS DOS based game. Throughout a typical lifecycle features are added, removed, and tweaked based on several factors including customer feedback.

Instead of sprints we generally have milestones. At the end of each milestone many teams have a post-mortem or retrospective to review key objectives, discuss things that went well, things that could have been improved, and  “tune and adjust its behavior” to make sure everyone is aligned and find ways to increase effectiveness.

Most developers were also doing unit testing. (TDD is simply another way of restating Dave Gelperin’s and Bill Hetzel’s famous quote from around 1987 “Test then code.”). Testers and developers talked frequently, and on many occasions the developer I worked closely with would come to my office to debug a hard to reproduce issue. And the majority of testers who were hired in the early 90’s were typically required to have an in-depth technical and often coding background in order to be able to engage in all aspects of the software development lifecycle and perform all the various job roles that might be required from a testing professional.

By 1998 we were doing daily builds. We have well established unit test libraries that not only assist developers in refactoring, but also help prevent build breaks and down-stream regressions. We still do a lot of API (component) level testing, as well as integration and system level testing.

Of course some of our products have longer release lifecycles than others. However, as indicated earlier we deliver “working software” at least once a week internally, and even to some premier customers (especially after the third milestone leading up to the final release of that version. But, some of our teams release every month.

Stepping back and looking at the debates, the books, and conference presentations I realized that I can relate well to the Agile principles. We often use models, principles and other abstractions to describe things. And, when all is said and done Agile principles are simply another way to present concepts that are intended to enable a team of highly motivated people to work together to ship a high quality product that satisfies their customers.

We have different models to generalize our processes to help us describe our typical workflow to others. But in truth there is no single way to build software, and models are simply our abstract expression of how we view the process. In fact, each team chooses processes and procedures that work for them and their particular product in their unique context.

(BTW, I am also really tired of Agile pundits always comparing Agile to waterfall models. Why don’t they compare their understanding of Agile to the V-model, W-model, or especially to spiral or other iterative models of software lifecycles? These are models; how your team chooses to implement a development lifecycle may resemble one of these models or you may adopt things from different models to implement your processes and approaches to software development.

Remember, George Box stated, “All models are wrong, some models are useful.” We should implement the concepts embodied in the model;  we should not implement the model.)

So, to me, Agile is the ability of a team to interact with each other, to adapt to changes, to periodically reassess its productivity in order to strive to become more effective and efficient in delivering software to its customers. Agile (and every other lifecycle models) is not a process; Agile is a mindset.

Since I was young I was involved in team sports. I played little league baseball during my elementary years, in junior high I switched to football, and in high school I played lacrosse. Growing up on the right (east) coast of America a lot of us boys would also play pick-up pond hockey when the rivers and ponds froze over in the winter. Hockey has always been a passion of mine, and I am a big Bruins fan (perhaps because Maryland didn’t have a hockey team), and Bobby Orr was amazing!

There is an ice rink very close to my home, and my daughter started skating at 4 years of age. She has no ambition to be the next Nancy Kerrigan, but she loves to ice skate. She also loves to watch hockey (she is a die hard Penguins fan and her favorite player is Sid The Kid).

The past few weeks I have been involved in a hockey tournament in conjunction with my regular Monarchs schedule, work schedule, and getting the boat ready for the up-coming Leukemia Cup Regatta and summer sailing season. (I’ll use that as an excuse for my long absence from the blogosphere.) In the tournament we won some games, and we lost some games. Eventually our team won the silver medal yesterday afternoon in the final match of the series. One thing all this practice and play has reminded me of is the importance of the team. Whether it is playing hockey or shipping software a team that plays well together is required for success.

Playing well together is more than simply showing up on the ice. Each player has a position, each player is expected to communicate effectively to move the puck around and “read the play,” occasionally players will “block a shot” on goal, and when you are on the ice you give 150% (which is why the shifts (time on ice) are short). Essentially no player can do everything alone, and the weakest link means that others on the team must take up the slack (which eventually leads to mistakes and players getting “out of position”).

I play defenseman, and usually left defense.There are general skills for the game that all players must have, but there are skills and strategies for defense positions that are different than for forwards. And although the 3 forwards and the 2 defensive players must work together as a team to score (or prevent a goal) the responsibilities of each position are different, and each player must play their position.

I have read a lot of analogies between individual sports such as martial arts and the discipline of software testing. These analogies are sometimes good in that they help testers understand why they must constantly learn new things and practice in order to grow in our (or any) professional discipline. Similarly, I could make an analogy between the defensemen on a hockey team and software testers. But, while we might all aspire to be a “testing ninja” (or a Bobby Orr) in our career, we also need a team of other professionals each doing their part to ship a great software product.

When players in the software game are out of position or not playing their best the team is looking for a loss, or at best a very hard earned win (that usually burns players out). For example, program managers who don’t to provide useful customer scenarios or models of potential feature designs, or developers who fail to unit test their code or run unit tests that are only slightly better than “does it compile” put the burden on the test team (the defensive players) to hopefully “score the goal.”

While a defensive player may occasionally score a goal, their primary role (on offense) is to put the puck in front of the opposing net. As defensive players we must be able to read the play (understand our customer scenarios and analyze abstract models of a design and provide critical feedback). Communicate effectively to other defensemen and forwards to get the puck into position, or protect our goal (put the important information in front of those who make the critical business decisions).

At times defensemen need to help the forwards by driving the puck out of the zone, but without the forwards doing their part the defensemen would surely get exhausted very quickly and the outcome would not likely be favorable. Likewise, if the forwards don’t help defend their goal it is likely the opposing team will wear down the defense and score.

So, while we (as testers) sometimes like to think that the success or failure of a software project depends solely on our defensive skills to find bugs and drive in quality, the simple fact is that we need a team of professional to be successful and “quality” depends on the value each player brings to the table. If we don’t start a game with a good strategy and people playing their best in their assigned positions the likelihood of success is marginal at best. A strong defense is critical, but no one ever wins by playing defense.

"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest." – Thomas Reid’s Essays on the Intellectual Powers of Man, 1786

But, one statement that has always troubled me that I saw repeated in the Agile Testing book is in regards to exploratory testing as the most effective way to expose important bugs. Every time I hear/read this or one of its many derivatives it makes me ponder. If this is really true then why do we invest so much time and resources in other approaches to software testing. Why do we spend so much time in static and dynamic analysis, reviews, and coverage analysis? Why do we try to prevent bugs when they can (and presumably are) found easier and ‘quicker’ downstream via exploratory testing? Why don’t we simply continue to hire hoards of diverse people to rapidly explore the product just before release and beat out all those important/critical bugs?

I wonder why it seems that we can ‘think critically’ when we have the product to play with, but we can’t seem to apply critical thinking while reviewing documented requirements, or a model? Personally, I am not sure that ‘trying’ something and then deciding what to do based on the outcome or state of the computer, or deciding whether something is a bug after the fact constitutes critical thinking. Perhaps sometimes it is, but I suspect that sometimes it is just guessing. To me, critical thinking involves the ability to foresee potential issues; not simply stumble upon them, or apply an attack from my bag of exploratory tricks to reveal them downstream in the development lifecycle when I have the product to play with. 

I sometimes wonder if exploratory testing really is the holy grail of software testing, or if we have just given up on other approaches because they are too hard. Could it be that our testers are not capable of ‘testing’ without also engaging their fingers? Are our testers untrained or under-trained in our profession? Is the reason why our pre-defined test cases are seemingly ineffective due to the realization that we suck at test design, and learning how to design more robust and ineffective tests is just too hard? Is the reason we can’t think of tests until we get the product in hand perhaps due to our inability to model or is it because we really don’t know that much about the systems we are testing? (You can’t model what you don’t understand and you don’t understand what you can’t model.) Besides, why should we burden testers with the overhead of learning about programming concepts, computer science, and math. You don’t need all that stuff to find bugs.

Now many of you might think that I despise exploratory testing. I am not anti-exploratory testing (whichever form that takes); because we all do it! Show me a tester who doesn’t include some form of exploratory testing in his or her repertoire and I’ll show you a thoughtless simpleton who needs explicit instructions for determining which side of the heal of a loaf of bread to butter. But, I have always thought that we could actually improve the craft of testing. I thought we might play a much bigger role in an organization. I thought we could help reduce overall production costs by being involved earlier and throughout the lifecycle. I thought we could help reduce risk through defect prevention and in-depth analysis. And, I thought we could learn to use tools and techniques to help us design better tests up front, and rely on a team of testers who are capable of thinking while executing a test case.

Yes, it is true that exploratory testing is fun, and we don’t really need to understand all that computer science stuff to find bugs using an exploratory approach. Regardless of what empirical studies on the topic reveal, exploratory testing ‘feels’ so much more productive. Why should we base our profession on evidence when we can get by so easily with emotion? Why should we have to think critically during the design or implementation of software when we can simply muddle our way through some graphical user interface to find bugs? If our approaches to unit testing and component level testing are so shoddy as to simply invoke happy path” tests then of course finding bugs by exploration seems rather logical. But, as developers become more adept at unit can component level testing in flushing out functional bugs earlier we may only need ‘testers’ to explore the product before release for behavioral issues, compatibility problems, and the few remaining obscure functional bugs. Or possibly exploratory testing providing a ‘good enough’ service to our team by finding the bugs that matter to the majority of customers, and perhaps that is ‘good enough.’

Personally, I think we can improve our testing processes. I think we will need to improve the practice of software testing. I think that our job is not simply to find bugs, but a more important objective is to prevent defects (which can help reduce overall product costs). I also don’t view testing as destructive, and I think those who do are doomed to become obsolete. Successful businesses don’t hire folks who just want to ‘destroy’ the products they build, I suspect they want to hire people who can help the company grow and improve. Finding bugs helps me ship a product, but it doesn’t necessarily help me grow the business, reduce production costs, or improve quality long-term.

But maybe we don’t need to change. Maybe our businesses are satisfied with the status quo. Maybe we don’t need to think about cost savings or helping our teams mature their processes, reduce overall risk, and potentially improve quality earlier. Maybe I am too  idealistic thinking that testers can provide greater value to an organization beyond finding bugs at the end of the development cycle.  Maybe our craft has reached the the apex of of our profession and this is as good as it gets.

I sometimes find code that is way over-engineered. In some cases it may be necessary for making the code more robust. But, over-engineered code may also result from ignorance of more efficient algorithms or design patterns. And, sometimes overly complex algorithms are a result of developers trying to craft something that obfuscates the simplicity of the solution and fool others into believing the complexity of the solution is somehow better than a more simple solution.

I sometimes see this in test code. I am a firm believer in robust, ‘bullet-proof’ test code because each time an automated test case throws a false positive or ‘breaks’ the team loses confidence in the automation project, and it takes our time to ‘massage’ the test back to health. But, there seems to be 2 extremes in test automation. Simplistic prescriptive scripted tests with a bunch of hard-coded ‘test-data,’ or overly complex test code that is virtually undecipherable by anyone other than the original developer that renders any downstream maintenance virtually impossible. I have seen many instances where complete libraries of automation was scrapped and re-developed simply because it was easier to re-write the code rather than trying to wade through the quagmire of complexity.

In a recent example, some of my students submitted their test automation projects with a library that contained a method to generate a random string. When I first looked at their method for generating a random string I was a bit perplexed. Besides the fact that the method only produced a grand total of 26 upper case characters, the code was much more complicated than necessary. Despite having showed them how to use the Babel random string generator they choose to make their own, so I asked them how they came up with this solution and they said “they searched on the Internet. ” Others in the class indicated they also used the same method in their projects. So, I when I got home I did a quick search and found the code sample.

   1: private static string RandomString(int size)

   2: {

   3:   StringBuilder builder = new StringBuilder();

   4:   Random random = new Random();

   5:   

   6:   char ch;

   7:   for (int i = 0; i < size; i++)

   8:   {

   9:     ch = Convert.ToChar(

  10:       Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));

  11:     builder.Append(ch);

  12:   }

  13:  

  14:   return builder.ToString();

  15: }

OK…for a moment let’s overlook the fact that if we make 2 consecutive calls to this method we will get the same identical random string of characters (which the author of this code also discovered), and let’s assume that the range of characters is limited to upper case A through Z, and let’s also ignore the fact that we are not seeding our Random generator for reproducibility of the randomly generated output from this method.

Let’s focus on the statement in lines 9 and 10. Why in the world would we generate a number between 0.0 and 1.0, multiply it by 26 and add 65, and then use Math.Floor to return the largest integer less then or equal to the resultant double, then convert that double to a type Int32, and then convert the Int32 to a type char? Now I ask you, can we make this any more complicated?

Now, I am not critiquing the style of the code or its inherent limitations, but this solution is an example of over-engineering. As I have said before, complexity cultivates chaos. At first I thought maybe this approach might give me a better distribution of characters, but when I tested this hypothesis it simply didn’t pan out. So, the way the random character is generated is simply too complex. An easier, more readable, and effective alternative might be to simply generate a random integer within the allowable range and cast that int to a type char as illustrated below in line 10. (Yes, I realize the limitations with this approach if trying to generate surrogate pair characters above U+FFFF.)

   1: private const int minCharacter = 65;

   2: private const int maxCharacter = 91;

   3: private static string SimpleRandomString(int size)

   4: {

   5:   StringBuilder sb = new StringBuilder();

   6:   System.Threading.Thread.Sleep(1);

   7:   Random r = new Random();

   8:   for (int i = 0; i < size; i++)

   9:   {

  10:     sb.Append((char)r.Next(minCharacter, maxCharacter + 1));

  11:   }

  12:  

  13:   return sb.ToString();

  14: }

BTW…the Sleep() in line 6 is a easy solution to preventing identical return values for consecutive calls. But, a more effective solution would be to pass in a seed value as a parameter to the method and use that to seed the new Random generator as illustrated below. Different seeds not only guarantee randomness in the resultant string, but also allow for repeatability if necessary as long as the seed value is preserved.

   1: private static string SimpleRandomString(int size, int seed)

   2: {

   3:   StringBuilder sb = new StringBuilder();

   4:   Random r = new Random(seed);

   5:   ...

   6: }

But, I digress. This post isn’t about random generation or style…it’s about complexity. Overly complex code tends to harbor errors that might go undetected (at least initially). Also, maintenance of complex code not only becomes more problematic, it often leads to costly re-writes down the road.

If we consider that greater complexity may increase the likelihood of error, then we don’t want our development partners to unnecessarily over-engineered algorithms. Also, overly complex solutions often require overly complex testing. This not only leads to increased testing costs, but it also increases the probability of an important test being missed or overlooked. Think testability!

Finally, with regard to test automation we should consider that our automated tests might be reused by other teams, and they certainly will be used during maintenance or sustained engineering efforts well after the product has released. And, in some cases, the people maintaining the product might not be the same people who shipped the product. Well-written automated tests are not just reviewable by someone other than the author of the code, but they should also be easily maintainable. Otherwise, we might end up paying double for that automated test case. Ouch!

Perhaps an even more important question is why do we find so many boundary type bugs via exploratory testing when they can and should be caught earlier? Why don’t we find these types of bugs in our unit testing? Why don’t we find these types of bugs by more systematically testing the software? Maybe we do find them, and those who make the decisions to fix these types of bugs just don’t care if they are fixed because there is no severe negative impact to the user. Maybe someone just wants to give me fodder for my blog!

This week I wanted to compare the range of allowable font sizes for a simulation program I developed as an example for a magazine article that I am working on. I knew that Office applications allow a font size within the range of 1 – 1638. I thought that range might be too large for my purposes, and since I knew that Windows Notepad included a font dialog I decided to check the allowable range of font sizes in Notepad.

The first thing I discovered was that the combobox control allows up to 5 characters! Really? Someone decided it is a good idea to allow users to enter 5 characters?

OK, I’ll play along. Maybe if I put in a size of 99999 and press the OK button on the dialog I will get an error message, or at least Notepad defaults to the last ‘valid’ selected font size. That might seem reasonable. But is that what happens? NO! Instead of doing something reasonable (e.g. error message, default font size) the font changes to a size of 1 (yes that is a font size 1 in the upper left corner in the image below).

I am sure that defaulting to a font size of 1 makes sense when the allowable size value overflows! Really…someone thought that was a good idea? Now I wanted to see what magical boundary value the developer decided was an acceptable font size. Since the combobox size property allowed 5 characters I immediately tried 65535. No, that also resulted in the overflow and displayed the text in a font size of 1. Next I tried 32767. Wait…32767 didn’t display the string in Notepad’s edit control at a font size of 1. Now, I am thinking the developer is using a data type of signed short for the font size variable. So, I enter 32768 expecting the value to overflow and display my string as a size 1 font again. But, no…that doesn’t happen.

Now, when I am design boundary tests I generally rely on 2 heuristics for identifying boundary values for input or output parameters.

1. Values at the extreme edges of a physical range of values
2. Values at the edges of equivalence partitions of physical values

So, in these situations I ask myself “What sort of demented developer debauchery have I now found myself?” I can’t think of any other obvious edge values that might apply, so out of curiosity I quickly narrow down the magical value to 39321. I then ask myself, “OK…even if there were a display capable of rendering or a printer capable of printing a font of this size, what is so unique about 39321?” In hexadecimal it is 0×9999, and in binary it is 1001100110011001. OK…nothing obviously special here, but I am certain the implementation details are much more complex then a simple range of values and at this point I really don’t care because this bug just doesn’t make sense.

Maybe it’s not supposed to make sense! Maybe nobody really cares about these types of bugs!

(BTW…somebody please take the Thesaurus away from the developer…’Oblique?’ Are you serious…why not just be consistent and use the word ‘Italic?’)

At the executive dinner and during my keynote I discussed various challenges in software engineering that directly impact testers. One of those challenges we need to get our heads wrapped around is software measures. By software measures I am referring to objects in software engineering mapped to various scales in the mathematical world. Although we sometimes also use biased qualitative measures, such as “too slow,” if we are to be taken with any degree of credibility we have to define what to slow is and set a reasonable goal for ‘acceptable’ based on customer values.

As testers we expend a lot of cycles collecting buckets full of metrics. We spend time producing fancy charts, and spend countless hours ‘looking’ at the data as if it were some type of oracle that would speak to us and tell us what we wanted to know. In the best case we convince ourselves that the numbers are telling us what we want the numbers to tell us. In the worse case the decision makers do not even consider the measures, or we don’t analyze the data in an attempt to identify ways to improve some of our engineering processes and practices. In the end, all the fancy charts are taken off the walls only to be shredded and we start over.

We often get caught up in tracking mostly useless data such as bug count and code coverage. What in the world does bug count or code coverage tell us (or the decision makers) about quality? Nothing; absolutely nothing! Some people want to believe that finding a lot of bugs or have high levels of code coverage means better quality, but that is sort of like believing that you’ll find a pot of gold and a leprechaun at the end of every rainbow. So, why do we measure bug counts and code coverage? Simple…because they are easy to measure!

Good metrics are hard to define mostly because we don’t always have clear goals, or we use a scatter-gun approach to setting a bunch of disparate wishful goals (goals that we hope we can achieve, but nobody is accountable if we don’t). I personally advocate the Goal/Question/Metric paradigm by Victor Basili. But, the biggest problem I have in using this approach is in establishing meaningful goals! People are generally good with coming up with superfluous objectives such as 100% automation or 80% code coverage. But, when you ask those people why they want 100% automation or 80% code coverage they retort only with a bunch of hand-waving and philosophical arguments. It seems we sometimes have difficulty expressing the ‘why’ of setting certain goals. Of course the answer in most cases is to ‘get better’ or ‘improve’ something! But, why? What is the business value?

Once we establish clear goals the next step is to understand the variables that we can manipulate to help us achieve those goals. Then we must decide on which ones we want to change that we think will have the biggest bang for the buck. Finally, we figure out which measures will let us know whether we are progressing towards our goal. (This usually isn’t a single point of measurement.)

At one time I naively believed that there was a core set of metrics that all teams should be collecting all the time that we could put into a ‘dashboard’ and compare across teams. In retrospect that was really a bone-headed notion. Identifying these measures is not easy, and there is no cookie-cutter approach. Each project team needs to decide on their specific goals that may increase customer value or impact business costs. Testers should ask themselves, “why are we measuring this?” “What actions will be taken as a result of these measures?” And, “if there is no actionable objective associated with this measure, then why am I spending time measuring this?”

At times is seems we are locked in a vicious cycle of relearning things via tribal knowledge, and we make decisions based mostly on ‘gut-feel’ and emotion. We collect a bunch of measures and display them similar to how the ancient Chinese used the mystical ‘dragon bones’ as oracles. But, if we are interested in being able to articulate business impact (either positive or negative) in a professional manner then we must be able to find ways to measure the things that are really important and actionable, and spend less time collecting numbers for wall decorations. At the end of the day someone is going to ask, “How do we know?” And trust me on this…really great managers will eat you alive if you answer with “well, we think…” or “we feel…” or try to evaluate success on some other subjective measure.

GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.

However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’)  But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’

There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?

First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.

So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”

The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.

Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:

and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.

But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”

“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”

I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.

Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.

This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of  new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)

Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”

This seems a bit confusing at first but then we realize that a single method (or function) may be a unit, or a component, or may (although unlikely) be the ‘system.’ A collection of methods wrapped in a library or DLL that interact to meet a functional requirement is a component. Rather than a developer having to call each method individually to achieve some usually repetitive functional outcome from the library that functionality is usually exposed via a call to a single API.

In this situation the students are testing a public API in a single library (DLL) that calls several methods to produce a randomly generated string (outcome) based on parameterized property values. The interface is the single API call (and the property variables) in an automated test, so we might consider the DLL that contains this API as the ‘system’ under test based on Beizer’s definition of component. Also, since students don’t actually see the underlying code, API testing in this context is ‘black-box’ testing.

The debate of who is responsible for API or component testing is tangential to the practice. I promote API testing because even in today’s extreme programming and TDD development lifecycle models we testers are still finding way too many ‘functional’ bugs (as opposed to behavioral bugs) during the integration and system levels of testing.

Also, generally (not all) software is designed and developed in layers similar to this simplified illustration. In well-designed, more easily testable projects the business logic or logical functionality is (should be) contained in classes or libraries (DLLs), and the public methods or APIs in those libraries know nothing about the user interface. and visa versa. End user inputs at the GUI are marshaled to the business logic layer via event handlers and properties (get/set accessors in C#) in different classes.

So, it shouldn’t be a surprise to anyone that certain categories of functional issues are more easily exposed at the API level of testing as opposed to testing through the UI. In fact, sometimes the UI properties and event handler layers in one project may actually mask some bugs in the APIs which aren’t exposed until much later when someone else uses that API in a different application or feature. The value of API testing is that a lot of functional testing can be performed very early in the project cycle, and functional testing can progress while the UI layer is unstable or in flux.

I sometimes think we are stuck testing from the end-user’s perspective. It seems that we often approach testing by trying to expose both behavioral type bugs and functional type bugs by testing completely through the UI. But, if we think of testing in layers the same way many products are developed then I wonder if we could better focus our test designs to target specific categories of functional bugs earlier and concentrate on behavioral issues and end-2-end customer scenarios when we have a more stable UI?

Of course, I am a bit of a hard head (even back then) and one day I started playing with the wood lathe while my grandfather was upstairs. Everything seemed to be going pretty well until I pushed the chisel in too far too fast and the wood split and went flying. One piece shattered the overhead light and the other piece ricocheted off the back of my hand leaving an nice gash. I shut off the machine and ran upstairs. After my grandmother cleaned and wrapped my hand, my grandfather made me go back downstairs and clean up the mess and stood over me with a stern look of disapproval making sure I wiped up my blood trail. After that incident, I heeded my grandfather’s advice, at least in his basement shop.

Anyway, with the recent discussions of code coverage around the testing blogosphere I started thinking about what was really being discussed. The discussions (as is the case with most discussions about code coverage) were not actually about the application code coverage as a tool, but more about the code coverage metric. And more specifically the discussions were about how not to assume a high measure of code coverage implies something is well tested. Interestingly enough, 2 years ago I wrote a post illustrating how the metric can be gamed and how the code coverage measure tells us nothing about quality or test effectiveness, but also alluded to how it might be used more effectively.

I thought that how the metric is sometimes misused is mostly self-evident, but then I realized that almost every time testers start talking about code coverage the discussion tends to focus on the metric. This may seem a bit harsh, but if a person’s only contribution to a conversation about code coverage is about how the metric doesn’t relate to quality or testing effectiveness then that person should not be allowed to play with hammers, and employing more complex tools such a wheel-barrows are well beyond that person’s comprehension.

Only thinking of code coverage as a means to get some magic number is akin to thinking “how many nails can I pound with this hammer. The metric itself is mostly irrelevant; and it is completely irrelevant if you don’t know how to interpret it in a way that helps you as a tester. Think about it this way; if we told our managers “our tests achieved 80% code coverage” some of our managers would be elated. (Of course IMHO, these types of managers are metric morons.) But, what do you think these same pointy headed number zombies would say if we told them “we ran our tests and we only missed testing 20% of the code.” I suspect they would start pacing back and forth in the room mumbling “We must run more tests, we must run more tests.”

When we stop thinking of code coverage as a simply measure where our only use of the tool is to try and achieve some magical number then perhaps we can start thinking about how to actually use code coverage as an effective tool to help us design tests (in under-tested or untested areas of the code), reduce potential risk, and possibly even drive quality upstream.

For example, one of my mentees is currently working on a project that uses just in time code coverage as a tool to evaluate how tests exercise changed code and downstream dependencies prior to checking code changes (e.g. bug fixes) back into the main tree. The initial pushback by some members of the team (including some pointy headed managers) was “code coverage doesn’t tell us about product quality” or “its too hard to achieve 80% code coverage” (although no such goal had been mentioned), and my personal favorite, “it’s too difficult to get everyone to measure coverage.” I reminded my mentee that the project is not about achieving some magic number, and in fact, it’s really not even about measuring at all. It’s about using the tool to discover information and to help us design additional functional tests at the API or component level that we might otherwise overlook to help prevent downstream regressions. In a nutshell, its about using code coverage as a defect prevention tool in this case.

Bottom line, code coverage is a tool! If you don’t know how to use it to improve your testing, well…

Anyway, one of the tool’s properties is a character array of invalid characters for the specific email address system under test. Although the guidelines for email addresses are outlined in RFC 5322 and RFC 2821 many companies can place greater restrictions on the characters that are allowed for the local address component of an email address (the local address is the part before the ‘@’ character).

For example, Yahoo only allows a local address to be between 4 and 32 characters, the first character must be a letter, and only letters, numbers, underscores and only 1 period character. The Google mail local address is between 6 and 30 characters, and only allows letters, numbers, and (multiple) period characters. Hotmail and Live mail allow local address name lengths between 6 and 64 characters (64 is the maximum allowable size according to RFC 5322), and can only contain letters, numbers, periods, hyphens, and underscores.

Even from these few examples we can see a couple of things. First, although we are testing email addresses there is not a universal set of equivalent partitions that works in all contexts. We need to partition the test data into equivalent class subsets based on the specific domain we are testing. For example, the invalid class subset of characters for a Google local address includes the underscore character, but both Yahoo and Hotmail allow the underscore as a valid character in an email local address. (But, I will talk next week about the equivalent partitioning of this data…for now let’s get back to boundary testing!)

Back to my story – as I was exploring each email providers requirements in order to determine how to partition the data I discovered a interesting problem with Yahoo. Remember, the maximum length of the local address for a Yahoo account is 32 characters.

And, the textbox control property on the web page is set to only allow a maximum input of 32 characters to prevent the user from inputting more than 32 characters. Copying a string longer than 32 characters into that textbox simply truncates the string after the 32nd character.

But, when I bump up against the maximum allowable length with some test strings the underlying program that generates suggested alternative local address names will actually produce a local address of 35 characters in length!

Now, if the software message tells me I can’t do something (like have a local address name of more than 32 characters and then the software generates a local address name of 35 characters for me…well, I am the sort of fellow who will push that button!

And sure enough it looks like I can use it. But wait. Only one more button to push and…

What do you mean “Sorry, this appears to be an invalid Yahoo ID?” You generated an invalid local address for me! Why would Yahoo mail torment me so?

I am thinking in the developers mind the user story went sort of like;

User: “I would like this.”

System: “No you can’t have that, but you can have this.”

User: “OK”

System: “No, you can’t have that either.”

It’s funny this came up this week because I was talking with a group of senior SDETs about defect prevention versus defect detection and how 99.999% of boundary issues can be found at the unit level or API level of testing well before the UI is slapped onto the functional layer.

Testing the functional layer more thoroughly or a code review would most likely have revealed this ‘magic’ number was inconsistent. Or by forcing the algorithm that generates suggested local addresses to test boundary conditions would have much sooner exposed this problem.

Now I don’t know Yahoo’s development and testing practices, and unfortunately it’s not uncommon to overlook bugs similar to this. But, I suspect that if developer rely on testers to find all their bugs, and testers primarily rely on testing through the user interface to find bugs then we are always going to find boundary bugs post release (and that’s a good thing because it gives me something to blog about).