I.M. Testy

For the past few weeks my test automation class at the University of Washington has been focused on API (application programming interface) testing, or component level testing. Boris Beizer defines component level testing as “an integrated aggregate of one or more units” and that a “component can be anything from a unit to an entire system.”

This seems a bit confusing at first but then we realize that a single method (or function) may be a unit, or a component, or may (although unlikely) be the ‘system.’ A collection of methods wrapped in a library or DLL that interact to meet a functional requirement is a component. Rather than a developer having to call each method individually to achieve some usually repetitive functional outcome from the library that functionality is usually exposed via a call to a single API.

In this situation the students are testing a public API in a single library (DLL) that calls several methods to produce a randomly generated string (outcome) based on parameterized property values. The interface is the single API call (and the property variables) in an automated test, so we might consider the DLL that contains this API as the ‘system’ under test based on Beizer’s definition of component. Also, since students don’t actually see the underlying code, API testing in this context is ‘black-box’ testing.

The debate of who is responsible for API or component testing is tangential to the practice. I promote API testing because even in today’s extreme programming and TDD development lifecycle models we testers are still finding way too many ‘functional’ bugs (as opposed to behavioral bugs) during the integration and system levels of testing.

Also, generally (not all) software is designed and developed in layers similar to this simplified illustration. In well-designed, more easily testable projects the business logic or logical functionality is (should be) contained in classes or libraries (DLLs), and the public methods or APIs in those libraries know nothing about the user interface. and visa versa. End user inputs at the GUI are marshaled to the business logic layer via event handlers and properties (get/set accessors in C#) in different classes.

So, it shouldn’t be a surprise to anyone that certain categories of functional issues are more easily exposed at the API level of testing as opposed to testing through the UI. In fact, sometimes the UI properties and event handler layers in one project may actually mask some bugs in the APIs which aren’t exposed until much later when someone else uses that API in a different application or feature. The value of API testing is that a lot of functional testing can be performed very early in the project cycle, and functional testing can progress while the UI layer is unstable or in flux.

I sometimes think we are stuck testing from the end-user’s perspective. It seems that we often approach testing by trying to expose both behavioral type bugs and functional type bugs by testing completely through the UI. But, if we think of testing in layers the same way many products are developed then I wonder if we could better focus our test designs to target specific categories of functional bugs earlier and concentrate on behavioral issues and end-2-end customer scenarios when we have a more stable UI?

When I was growing up I would sometimes go down into my grandfather’s basement. He had amassed a variety of tools during his lifetime and he was an excellent wood craftsman. I wasn’t allowed to touch any of the power tools, because his rule was, “if you don’t know how to use a tool properly then you shouldn’t play with it.”

Of course, I am a bit of a hard head (even back then) and one day I started playing with the wood lathe while my grandfather was upstairs. Everything seemed to be going pretty well until I pushed the chisel in too far too fast and the wood split and went flying. One piece shattered the overhead light and the other piece ricocheted off the back of my hand leaving an nice gash. I shut off the machine and ran upstairs. After my grandmother cleaned and wrapped my hand, my grandfather made me go back downstairs and clean up the mess and stood over me with a stern look of disapproval making sure I wiped up my blood trail. After that incident, I heeded my grandfather’s advice, at least in his basement shop.

Anyway, with the recent discussions of code coverage around the testing blogosphere I started thinking about what was really being discussed. The discussions (as is the case with most discussions about code coverage) were not actually about the application code coverage as a tool, but more about the code coverage metric. And more specifically the discussions were about how not to assume a high measure of code coverage implies something is well tested. Interestingly enough, 2 years ago I wrote a post illustrating how the metric can be gamed and how the code coverage measure tells us nothing about quality or test effectiveness, but also alluded to how it might be used more effectively.

I thought that how the metric is sometimes misused is mostly self-evident, but then I realized that almost every time testers start talking about code coverage the discussion tends to focus on the metric. This may seem a bit harsh, but if a person’s only contribution to a conversation about code coverage is about how the metric doesn’t relate to quality or testing effectiveness then that person should not be allowed to play with hammers, and employing more complex tools such a wheel-barrows are well beyond that person’s comprehension.

Only thinking of code coverage as a means to get some magic number is akin to thinking “how many nails can I pound with this hammer. The metric itself is mostly irrelevant; and it is completely irrelevant if you don’t know how to interpret it in a way that helps you as a tester. Think about it this way; if we told our managers “our tests achieved 80% code coverage” some of our managers would be elated. (Of course IMHO, these types of managers are metric morons.) But, what do you think these same pointy headed number zombies would say if we told them “we ran our tests and we only missed testing 20% of the code.” I suspect they would start pacing back and forth in the room mumbling “We must run more tests, we must run more tests.”

When we stop thinking of code coverage as a simply measure where our only use of the tool is to try and achieve some magical number then perhaps we can start thinking about how to actually use code coverage as an effective tool to help us design tests (in under-tested or untested areas of the code), reduce potential risk, and possibly even drive quality upstream.

For example, one of my mentees is currently working on a project that uses just in time code coverage as a tool to evaluate how tests exercise changed code and downstream dependencies prior to checking code changes (e.g. bug fixes) back into the main tree. The initial pushback by some members of the team (including some pointy headed managers) was “code coverage doesn’t tell us about product quality” or “its too hard to achieve 80% code coverage” (although no such goal had been mentioned), and my personal favorite, “it’s too difficult to get everyone to measure coverage.” I reminded my mentee that the project is not about achieving some magic number, and in fact, it’s really not even about measuring at all. It’s about using the tool to discover information and to help us design additional functional tests at the API or component level that we might otherwise overlook to help prevent downstream regressions. In a nutshell, its about using code coverage as a defect prevention tool in this case.

Bottom line, code coverage is a tool! If you don’t know how to use it to improve your testing, well…

This past weekend I was working on a new test tool library for generating random email addresses; specifically the local address segment of an email address. I know, there are already a lot of email address generators available and this could be construed as reinventing the wheel. But I wanted to give my students in my test automation course at the University of Washington something to test at the API level. So why not have them test a test tool and learn a bit more about API level testing and how to use combinatorial analysis of the input property values to drive a data-driven automated test case. Also, having them test it means that I don’t have too!

Anyway, one of the tool’s properties is a character array of invalid characters for the specific email address system under test. Although the guidelines for email addresses are outlined in RFC 5322 and RFC 2821 many companies can place greater restrictions on the characters that are allowed for the local address component of an email address (the local address is the part before the ‘@’ character).

For example, Yahoo only allows a local address to be between 4 and 32 characters, the first character must be a letter, and only letters, numbers, underscores and only 1 period character. The Google mail local address is between 6 and 30 characters, and only allows letters, numbers, and (multiple) period characters. Hotmail and Live mail allow local address name lengths between 6 and 64 characters (64 is the maximum allowable size according to RFC 5322), and can only contain letters, numbers, periods, hyphens, and underscores.

Even from these few examples we can see a couple of things. First, although we are testing email addresses there is not a universal set of equivalent partitions that works in all contexts. We need to partition the test data into equivalent class subsets based on the specific domain we are testing. For example, the invalid class subset of characters for a Google local address includes the underscore character, but both Yahoo and Hotmail allow the underscore as a valid character in an email local address. (But, I will talk next week about the equivalent partitioning of this data…for now let’s get back to boundary testing!)

Back to my story – as I was exploring each email providers requirements in order to determine how to partition the data I discovered a interesting problem with Yahoo. Remember, the maximum length of the local address for a Yahoo account is 32 characters.

And, the textbox control property on the web page is set to only allow a maximum input of 32 characters to prevent the user from inputting more than 32 characters. Copying a string longer than 32 characters into that textbox simply truncates the string after the 32nd character.

But, when I bump up against the maximum allowable length with some test strings the underlying program that generates suggested alternative local address names will actually produce a local address of 35 characters in length!

Now, if the software message tells me I can’t do something (like have a local address name of more than 32 characters and then the software generates a local address name of 35 characters for me…well, I am the sort of fellow who will push that button!

And sure enough it looks like I can use it. But wait. Only one more button to push and…

What do you mean “Sorry, this appears to be an invalid Yahoo ID?” You generated an invalid local address for me! Why would Yahoo mail torment me so?

I am thinking in the developers mind the user story went sort of like;

User: “I would like this.”

System: “No you can’t have that, but you can have this.”

User: “OK”

System: “No, you can’t have that either.”

It’s funny this came up this week because I was talking with a group of senior SDETs about defect prevention versus defect detection and how 99.999% of boundary issues can be found at the unit level or API level of testing well before the UI is slapped onto the functional layer.

Testing the functional layer more thoroughly or a code review would most likely have revealed this ‘magic’ number was inconsistent. Or by forcing the algorithm that generates suggested local addresses to test boundary conditions would have much sooner exposed this problem.

Now I don’t know Yahoo’s development and testing practices, and unfortunately it’s not uncommon to overlook bugs similar to this. But, I suspect that if developer rely on testers to find all their bugs, and testers primarily rely on testing through the user interface to find bugs then we are always going to find boundary bugs post release (and that’s a good thing because it gives me something to blog about).

Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.

A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).

So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.

Anyway, the final project after the first 10 weeks of the course is to design automated tests of  3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.

I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.

Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.

For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.

   1: // <copyright file="simpletestcase.cs" company="TestingMentor"> 

   2:  // Copyright © 2009 by Bj Rollison. All rights reserved. 

   3:  // </copyright> 

   4: 

   5: namespace TestingMentor.Sample

   6: {

   7:   using System;

   8:   using System.IO;

   9:   using TestingMentor.Simulation;

  10: 

  11:   class TestCase

  12:   {

  13:     static void Main(string[] args)

  14:     {

  15:       int testCounter = 0;

  16:       // Read in an array of strings representing the test data. 

  17:       // Of course this would likely come from a static test data file

  18:       // on a server or copied to a folder on the local machine

  19:       string[] testData = new string[]

  20:       { "90,Z",

  21:         "24798,惞",

  22:         "0,null",

  23:         "65536,Error",

  24:         "-1,Error",

  25:         "1.5,",

  26:         "xyz,xyz"

  27:       };

  28: 

  29:       // Loop through each test data string

  30:       foreach (string test in testData)

  31:       {

  32:         testCounter++;

  33:         // This nested try/catch block catches invalid test data

  34:         // but allow additonal tests in the testData array

  35:         try

  36:         {

  37:           // Parse each string into the test data and expected result

  38:           string[] testElement = test.Split(',');

  39:           string expectedResult = testElement[1];

  40:           string actualResult = String.Empty;

  41: 

  42:           // Convert the string to a type int value

  43:           int value = int.Parse(testElement[0]);

  44:

  45:           // We need a way to handle int values 0 through 32 which are 

  46:           // control characters, this is an example of how to deal with 

  47:           // a int value of 0 which is a null character

  48:           if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))

  49:           {

  50:             expectedResult = '\0'.ToString();

  51:           }

  52: 

  53:           // This nested try/catch block tests catches exceptions thrown by 

  54:           // the method under test. If the method under test throws an 

  55:           // exception we certainly want to test for that case!

  56:           try

  57:           {

  58:             // Call the API method under test 

  59:             char result = Converter.ConvertValueToUnicodeChar(value);

  60:             actualResult = result.ToString();

  61:           }

  62: 

  63:           catch (ArgumentOutOfRangeException)

  64:           {

  65:             actualResult = "Error";

  66:           }

  67: 

  68:           catch (Exception)

  69:           {

  70:             // if this happens this is a failure because the documentation

  71:             // states that this method will only throw an 

  72:             // ArgumentOutOfRangeException.

  73:             actualResult = "Non-specific or unexpected error thrown";

  74:           }

  75: 

  76:           // Call a simple oracle and log results

  77:           if (String.Equals(actualResult, expectedResult))

  78:           {

  79:             // log pass

  80:             Console.WriteLine("{0} Pass", testCounter);

  81:           }

  82:           else

  83:           {

  84:             // log fail...of course log as much detail as possible

  85:             Console.WriteLine("{0} Fail", testCounter);

  86:           }

  87:         }

  88: 

  89:         catch (FormatException)

  90:         {

  91:           // log the test data for this test as incorrect, test is skipped

  92:           Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);

  93:         }

  94:       }

  95:     }

  96:   }

  97: }

Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).

Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)

The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?

Did you ever notice that when you ask someone to test something the first thing they do is to start ‘testing?’

I often see this in my classes and I ask the person, “what is the purpose of your test?” Typically the response is, “I’m testing this,” or “I’m trying to find a bug.”

Unfortunately this seems to indicate there is no or very little pre-thought that goes into the act of software testing. To some people, testing appears to be little more than simply pounding away at the keyboard and trying whatever flies into our subconscious mind as we interact with the software and declare a bug when we stumble upon unexpected behavior or see something we might disagree with.

This is why I found it especially interesting in my own research, and the case studies by Juha Itkonen that testers who were trained in formal software testing techniques or patterns there was no significant difference in terms of defect rates or coverage between pre-defined test cases and an exploratory testing approach. This is not to say that one approach to testing is preferred over the other. It is not an either or proposition as I explained in my post on the pesticide paradox, and there are certainly more than 2 approaches to software testing. Testing requires multiple approaches to most effectively aid us in collecting and presenting the appropriate information to the decision makers.

But, I am often puzzled that it seems we can easily think of negative or destructive tests once we have the product in hand, yet when we are designing a set of tests from the requirements the tests simply test the requirements and little else. I wonder why it is that we can think of ‘tests’ while executing other tests, but we can’t think of those same tests before hand. Is there some limitation in our psyche that prevents us from analyzing a problem until we are actually faced with the problem (software in hand)?

I don’t think so, but I suspect there is a mental hurdle in that we sometimes feel more productive when we are interacting with software as opposed to sitting back and analyzing the problem more prior to executing well-designed test cases. (More tests doesn’t equal better testing!)

The bottom line is that if we are given a set of requirements and can only design tests that only test the requirements, then we are probably not thinking critically about how to design test cases.

Some of you know that I am an organic gardener. I do this not only because I like the fresh taste of vegetables and fruits that are not tainted with chemicals, but my daughter loves to eat things right off the vine or stem as we are harvesting our bounty and I am perhaps overly protective of my daughter. Here in the Pacific Northwest slugs are a particular problem, and I must use several different techniques and approaches to try to ward off or destroy these nasty, slimy creatures throughout the year because no single approach is 100% effective. Yes it is unfortunately true, even with my diligent efforts and myriad of tactics a few slugs still get by into the raised gardens. This sure seems a lot like software testing and how some tests find some bugs and miss others, and how iterative builds seem to allow new bugs to continually creep in. But, this really isn’t a revelation.

In 1983, in his seminal book Software Testing Techniques, Boris Beizer compared the diminishing effects of insecticides on boll weevils destroying cotton fields to the decreasing effectiveness of testing methods in exposing defects in software. This became well known as the Pesticide Paradox. The Pesticide Paradox states “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”

Now, unlike insects (or pests), software doesn’t magically build up an immunity to bugs. What happens is that we design and execute tests using one approach that exposes some set of issues. But, then the number of issues being exposed by that approach starts to diminish, yet there are still some ‘residue of subtler bugs’ that haven’t been detected by that set of tests or testing approach. Also similar to how the ladybugs in my garden that eat aphids and mites have no effect on slugs, not all approaches or techniques we might use in our testing is effective in detecting or preventing all types of bugs.

For the past 10 years, I have been teaching various software testing practices and approaches for solving complex problems at Microsoft. One cool aspect of my job is that I get to experiment a lot in a reasonably controlled environment with diverse groups of people. We often design these studies to better understand the benefits and limitations of various testing approaches and methods in exposing different categories of defects to better understand how each approach can be used more effectively within the appropriate context. Of course, it should be of no surprise to anyone that the pesticide paradox holds true not just in the classroom, but also in practice.

I often explain Beizer’s paradox by stating, “there is no single approach or method used in software testing that is completely effective in exposing all bugs, and some approaches or methods are more effective in exposing different types or categories of bugs.”

At Microsoft there is no “one size fits all” solution, and the Engineering Excellence group doesn’t dictate how to test or what testing methods can or can’t be used. But, through a series of problem solving exercises in our new SDET training program each tester experiences the benefits and limitations of various approaches and techniques. Based on their experiences in the classroom and in ‘real life’ they also learn the most effective strategy for testing is not to rely too heavily on a single approach, and to use a variety of test design principles and patterns throughout the product lifecycle. But, that’s just how we roll.

Software Testing Techniques 2nd Eidition

Originally Published Thursday, October 01, 2009

Software is knowledge. Software is the intangible product crafted by a team of people who have pooled their intellectual knowledge to help solve a complex problem and add value to those who use that software. So how does a tester contribute to the intellectual knowledge pool?

I guess we could say that finding and reporting bugs during the software development lifecycle (SDLC) is important knowledge because it helps identify many anomalies prior to release. But, the mere act of finding and reporting bugs is transient knowledge. Reporting bugs in the system does not add any long term or persistent value to the intellectual knowledge pool of a software company. Perhaps even worse, finding the same type of issue repeatedly actually stagnates the intellectual knowledge pool because the team is focused on fixing the same problem over and over again. Of course, finding really interesting and important bugs requires a lot of knowledge and creativity. But, once the bug is fixed the value that bug may have provided in the intellectual knowledge pool evaporates; especially if there is no shared learning experience that occurs as a result of that fixed issue.

One way software testers can significantly contribute to the intellectual knowledge pool is through defect prevention instead of defect detection. Simply put, if we expand our vision of the role of the tester to include problem solving instead of just problem finding we can open up new challenges, provide overall greater value to our business, and further advance the discipline of software testing. For example, if we were to identify a particular area or category of defects and identify the root causes for that type of problem then we can implement various strategies or best practices to prevent those types of issues from being injected into the product design from the onset, or at least develop testing patterns or tools to help the team identify many of that category of issues sooner in the SDLC. Understanding why certain categories of problems occur and providing best practice solutions within the appropriate context is intellectual knowledge that can be shared with existing and new team members, and can persist to help prevent certain types of problems from recurring in the future. This is intellectual capital in the knowledge pool. Testing tool and test patterns that can be shared and taught to others that help identify certain types of issues sooner can reduce testing costs. This is also intellectual capital in the knowledge pool.

If I am constantly burdened with finding the same types of problems over and over again, then my contribution to the SDLC and the knowledge pool is essentially limited to the bugs I find, and the value of those bugs often depreciates rapidly. Basically, I am simply identifying problems; I am not contributing to solving the problems.

Of course, I don’t think testers will ever work themselves out of a job and we will always be in the business of identify issues during the SDLC. But, if I solve one type of problem, then I can move on to face new and more difficult challenges. By solving one problem I get that job off my plate, and I can then move onto the next job. Organizational maturity and professional growth occurs through solving increasingly complex problems, not by continually dealing with the same problem.

I think the role of a professional tester is growing beyond that of simple problem identification, and many of us are exploring the more challenging aspects of problem solving. Finding ways to prevent defects or identify issues earlier, and essentially drive quality upstream are exciting challenges that will advance the practice of software testing and increase the value of our contributions to the intellectual knowledge pool and advance the profession of software testing.

Originally Published Thursday, September 24, 2009

The past 2 weeks have been a bit rough. While in Israel I began to feel a bit congested. By the time I hit Nürmberg, Germany for 12th International Conference on Quality Engineering in Software Technology I was injecting nose-juice (nasal decongestant) about every 2-3 hours and couldn’t sleep through the night. Fortunately I didn’t speak until Friday, so Monday morning I visited a local Apotheke (Pharmacy), described the symptoms, and was presented with some medicinal remedies by the pharmacist. By Wednesday I was much worse, so again tried another pharmacy and was given a different batch of drugs. By Friday morning I was struggling, but managed to present my talk on probabilistic stochastic test data generation using parameterized equivalent partitions and genetic algorithms (which I will discuss in a future post). Unfortunately, I had to cancel another engagement and reschedule my flight home for Saturday. Once home I went to my doctor and was quickly diagnosed with a bacterial infection in my nasal cavities.

Now, I am not telling you this story to seek your sympathy, but to illustrate a point. I had convinced myself that I simply had a slight cold that I could treat with over-the-counter remedies, and perhaps due to my own stubborn nature I refused advice from my friends in Germany to see a physician. In the end, I realized I was simply treating the symptoms and ignoring the root cause of the real problem. So, I sometimes wonder if we are too focused on treating the symptoms of buggy software by focusing our testing efforts on bug detection rather than addressing the real problem and thinking more about bug prevention.

In my opinion, one of the most significant ways we can directly impact quality of the product and the effectiveness of our teams is not by trying to beat the bugs out of the product after the designers and developers have spent days/weeks injecting bugs into the product, but through partnering with the PMs and developers earlier in the lifecycle to prevent issues from ever getting into the product to begin with. If we continue to think of testing as an after-the-fact process than we might never advance our discipline, and perhaps even worse, we might relegate the role of testers to nothing more than bug-finders.

Defect prevention doesn’t negate or eliminate the need for system level testing, but it could certainly change the role of testers throughout any product lifecycle. Rather than perpetuating an adversarial  “don’t trust the developer” attitude I envision testers and developers working in a more symbiotic relationship (Доверяй, но проверяй – Trust, but verify). For example, I think many readers would agree that developers are responsible for unit testing, but I wonder how many testers are proactively engaging their development partners and suggesting ways to improve the effectiveness of their unit tests (without adding significant additional overhead), or participating in code inspections. And, how many testers are engaged in design reviews and prototyping with program managers and designers in an effort to prevent sub-optimal designs which often leads to a tremendous amount of rework.

The ability to move quality upstream through defect prevention requires different skills and capabilities, but also opens up new and greater challenges for software testers.

“Bug prevention is testing’s first goal.” – B. Beizer

Originally Published Saturday, September 12, 2009

I have spent the last week in Israel teaching our new SDET course in Herzillya and our Senior SDET course in Haifa. I also did a lot of listening and discussing various issues relating to software testing and the maturation of our discipline; not just here in Israel, but around the world both inside and outside of Microsoft. Now I am sitting at LaLa Land after a relaxing day of sailing in the Med, and reflecting on the past week’s discussions.

One of the topics we discussed was best practices, and that  seems appropriate to write about since the concept of “best practices” was recently discussed (again) in an article in the Software Test and Performance magazine by Eddie Correia. Eddie argues “…the notion of “best practices” is not useful. Best for whom? And for what kind of testing?” Actually, this is just a repetition of the same old fustian melodramatic hyperbole of the “context-driven” posse.

Perhaps the philosophical “questioning” of best practices may be interesting for folks who like to run around quoting Aristotle and Plato. However, from a pragmatic point of view this is a rather benign debate for anyone capable of thinking for themselves.

In reality, many different professions recognize the concept of best practices. For example, a best practice in preventative medicine is to rinse a minor abrasion and apply a topical antiseptic ointment. A best practice in plumbing is to wrap Teflon tape in the direction of the threads when fitting pipes. “Eliminating distractions in the operational area” is listed as one of the best practices by the FAA for airfield safety. Do these “best practices” apply in all situations? No, they don’t.

So, why do so many professionals recognize the concept of “best practices.” Because they understand that best practices provide guidelines that are generally more effective in the appropriate context as compared to other approaches. They understand that “best practices” are not a rules or rigid standards that must be followed in all circumstances, but “best practices” are general solutions to common problems that can be shared among professionals who might face similar situations. The professionals who understand “best practice” concepts are usually well-trained on other comparable practices for the type of problem they are facing, and know when to apply the best practice within the appropriate situation. They understand that “best practices” don’t simply apply to 1 or 2 limited situations, but have been proven to be generally effective for that particular type of problem.

But, most importantly these professionals (who recognize “best practices”) are extremely  knowledgeable about their field and can “act with appropriate judgment” (that’s sapience for the CD crowd), and conversely know when to approach a problem using a different solution.

Fortunately, the argument against “best practices” only stems from a few people who are seemingly more interested in stirring up portentous philosophical debate rather than earnestly discussing the practical advancement of the profession of software testing beyond mysticism and emotionally charged rhetoric. And, that argument really seems to boil down to a rather condescending and incorrect viewpoint that best practices are merely steadfast rules and requirements that must be followed in all situations. I say condescending because this point of view seems to suggest that testers are incapable of analyzing a problem and logically rationalizing the benefits and limitations of various approaches to problems in different situations to reach appropriate decisions on their own.

Personally, I think professionals in the discipline of software testing are highly intelligent, and are quite capable of making smart decisions, and can “act with appropriate judgment” in a wide variety of contextual situations. I also think discussions of best practices are enriched with case studies outlining situations where they may not apply and the alternative approaches that were more effective in those situations. And, I think “best practices” provide a common reference for professionals in that field that can be shared and further developed, and perhaps even give rise to new “best practices” for varying situations.

So, for those of you who believe there is a “one-size fits all” solution that can be applied in every situation I recommend that you don’t subscribe to the concept of best practices. (I would also recommend these people are well supervised and constantly monitored.) But, for the vast majority of professionals in the practice of software testing I suspect you understand the notion of “best practices” is quite useful for pragmatic discussions for advancing the intellectual knowledge pool of our profession and maturing our discipline.

Originally Published Thursday, September 10, 2009

I am really happy to announce that our book has been released in China and available on the Chinese Amazon site! This was really a monumental effort driven by my friend and colleague Kelly Zhang.

We look forward to the feedback from the Chinese testing community, and we hope this provides our Chinese friends  with some additional perspectives on software testing (or at least some interesting stories).