When we report a bug our hope is that bug is fixed. But, of course we know that isn’t always the case which is why there are usually several alternative resolutions developers, project managers, or managers may choose for resolving a bug such as postponed, won’t fix, and by design. It is unfortunately quite common to see a tester metaphorically explode into passionate fits of outrage when one of their bugs is resolved as postponed, won’t fix, or by design. It is unfortunate because these tantrums often involve the tester hurling personal insults (e.g. “How can the developer be so stupid not to fix this bug"?”), decrying product quality (e.g. “If we don’t fix this bug this product will totally suck!”), and playing the whiny customer card (e.g. “We will loose customers if we don’t fix this bug.”). Yes, in my early years I was also guilty of these sorts of irrational outbursts of hyperbole when a bug that I thought was important was resolved not fixed. But, of course, I quickly learned that such sophistical speculations rarely resulted in the bug being fixed, and mostly lessened my credibility with developers and managers.

The other day I was speaking with a tester who was a bit miffed because the developer had resolved a few of her bugs as by design and won’t fix and she asked how she could ‘fight’ these resolutions. “Well,” I began, “Getting people to change their minds usually involves negotiation and the logical presentation of facts in a non-judgmental approach. Sometimes you will succeed, and sometimes you will not succeed. As testers surely we want all our bugs to be fixed; however, from a practical standpoint that may not always be the case especially if the bug is subjective.” I previously wrote about 10 common problems with bug reporting, but, in this case I proceeded to discuss a few strategies I use to advocate bugs.

Make it easy for the developer to fix the bug

As a minimum a tester must provide a description of the problem, the environmental conditions in which the problem occurred (if localized to a specific environment), the shortest number of exact steps to reproduce the bug, and the actual results versus the expected results. Occasionally a screen shot may be beneficial, but mostly if there is a contrasting example. But, I will also point the developer to my test; especially if it is automated. Providing the developer an automated mechanism to reproduce a problem reduces a lot of overhead. Of course, in this case I am talking about an automated test case that runs in a few seconds, or an automated script that even assists the developer reproduce the problem quickly.

Provide specific contradictions to specified and/or implied requirements or standards

Of course, if the product design or functionality deviates from stated requirements pointing this out in a non-confrontational way is a no-brainer. The key here is our argument must be non-confrontational because sometimes we may misinterpret the requirements, and sometimes the requirements may change without us being aware of those changes. There are also occasionally deviations from implied requirements such a UI design guidelines as a result of the introduction of new technologies, or changes in how customers use the product based on usability studies. Other implied standards include competing products or previous versions of the product. In any case, when arguing for a bug fix based on specified or implied requirements I recommend using a compare and contrast type of approach to better illustrate the problem as I perceive it.

Provide concrete examples of customer impact

This is really important! Providing a real world scenario that clearly illustrates not only how this bug will manifest itself to the customer, but also providing corroborating evidence from customers presents a strong case in favor of a bug fix. There are several useful repositories of customer feedback testers can use to bolster their point of view such as newsgroups, popular blogs, trade journal reviews of past or similar products, at Microsoft we also have Watson and SQM data, and product support reports. Using ‘real-world’ constructive feedback is often more meaningful than an internal mutiny by a portion of the test team.

Know your primary target customer profile

Testers often like to think we are representative of our customers. However, this may not always be the case. (It has always puzzled me as to why testers seem to think they have some greater affinity to the end user customer as compared to others on the product team.) Yes, it is important that testers understand who the primary target customer is for the current project or release and that is why many teams have detailed personas of primary, secondary, and sometimes even tertiary customer audiences. Of course, if we are in the commercial software business we want our customer base to be as large as possible. But, as the number of customers increase so does the diversity of value, and as they say…you can never please everyone! So, when defending your position to fix a particular bug it is always better to frame the discussion from the point of view of the primary customer persona as compared to your own personal bias.

Use your brain, not your emotions

Passion has long been an admired trait in software testers. However, unbridled passion fraught with antagonistic accusations can be detrimental to a successful bug resolution (and sometimes even a career). Some bugs obviously need to be fixed, while others may be more dependent on several mitigating (and competing) factors such as where you are in the software lifecycle, business impact, primary customer impact, risk, etc. I think it is largely agreed that perhaps the primary role of testers is to provide information, but that means we must also gather the pertinent information and represent that information logically within the relevant context to the management team (or decision makers). Remember…reckless rants rarely render reasonable results!

The ‘quality’ of defect reports is something testers and managers talk about quite frequently. But quality is usually vaguely defined and is always interpreted differently depending on the context or perceptions of the individuals discussing quality. We all have our own perspectives on what constitutes a high quality defect report, but do we really understand the quality our customers (the developers we work with) expect in a bug report?

It’s funny that many testers always talk about being advocates for the customer, but always seem to fail to consider their internal teammates as customers of specific testing deliverables such as defect reports. So, in order to give ‘quality’ some context I asked 600 developers to identify their top 10 pain points regarding defect reports. The results are aggregated in the top 10 list below. By understanding the customer pain points we can reframe our discussion around specific customer issues and then put the quality of defect reports in the correct context to improve our customer’s satisfaction.

1. Incomplete repro steps

Over 95% of the respondents indicated the steps to reproduce errors as the most significant problem with defect reports. Defect reports are submitted by people other than testers, but since this was the number one complaint among developers it seems to indicate a serious problem. If developers cannot reproduce defects accurately, the cost of that defect increases because the report bounces back and forth between the developer and the tester wasting valuable time. Also, a defect that can’t be immediately reproduced will usually go to the bottom of the pile, and may get overlooked (especially if testers are diligent if following up on the defects they report), or worse become hidden or much harder to reproduce.

2. Email type discussions in the report

If defect reports are too verbose the critical details the developer is looking for is lost in all the words! Defect reports are technical documents. Defect reports should contain enough factual information to get an issue resolved and briefly describe the customer impact. I would recommend that every tester (or anyone who works in software development) at least read a book or take a class on technical writing. A defect report should contain a brief (objective) description of the problem, the specific steps to reproduce the problem, the actual results, the expected results (see #7 below), and a customer impact statement. Some defect reports also include additional notes from troubleshooting, debug information, etc.

3. Lack of detail and investigation

Defects manifest themselves in many ways. When a tester uncovers a defect it is their responsibility to troubleshoot the problem and ascertain other possible paths or ways a customer might encounter the defect. For example, if a defect is encountered when testing a web-lication running in Internet Explorer 7.0, does the same problem occur on Internet Explorer 6.1, or FireFox? Troubleshooting helps to isolate (or at least narrow down) the probable cause which in turn can effect a quicker fix by the development team.

4. Bug morphing

Bug morphing occurs when the original defect is fixed, but during regression testing of the original defect the tester discovers another problem which the tester assumes is either associated with that bug, or caused by the fix for the original problem. So, instead of creating a new bug report the tester amends the bug report to describe a different problem. A defect regression only occurs if the same exact steps cause the same exact problem, or if the tester can prove the root cause of the 2 defects are identical at the code level.

5. Missing data files

This was a surprising issue. In some cases files or test data is mentioned in the report but is not provided and in other instances the files are linked to a share that is inaccessible. If you have a specific test file used to reproduce the defect then provide that file to the development team.

6. Environment/configuration information

Occasionally, defects occur only under certain conditions or configurations. When testers encounter a defect they should not only attempt to reproduce the problem, but they should attempt to reproduce the problem on a different machine running a different configuration. If the problem is reproducible on two or three different machines the probability of the developer not being able to reproduce the defect is greatly reduced. In cases where the defect can only be reproduced on one machine, the tester must troubleshoot the problem to determine the differences between the systems. Veritest Analyzer 2.0 and Filemon and RegMon tools on SysInternals.com are great tools to help testers analyze differences in test systems.

7. Expected Results

Software testing is similar to scientific experimentation. Experimentation typically doesn’t involve mixing chemicals willy-nilly, or just doing something just to see what happens. Experiments are most often performed in controlled environments, using pre-formulated data or materials and have predicted outcomes. Similarly, software testing is conducted in various controlled environments (unit, component, integration, system), using pre-formulated data (both valid and invalid) and one of the most important aspects of testing is to measure the actual outcome of a test against the expected results (and of course to report those results). Even in situations where there is a lack of documentation testers should write a statement of the expected results and provide justification for those expectations. This can help the developer better understand the issue and possibly make a more informed decision on how to resolve the problem.

8. Duplicates

This is a difficult issue to address, and there is no way to eliminate this from occurring altogether in large software projects. However, there are perhaps ways to minimize this issue by addressing some of the issues above. I have occasionally seen testers literally throw temper tantrums when their defect report was resolved as a duplicate of a defect report submitted after the original report. There are several reasons why this might occur. Remember, defects manifest themselves in different ways. Developers are concerned with the root cause of the problem, not necessarily all the potential way or paths to hit the same problem. If a tester discovers multiple paths to the same problem and records each as a separate defect all but one will be resolved as a duplicate. Also, if several defect reports are submitted on the same issue it is likely the developer will use the defect report that is easiest to interpret and provides the most useful information. As a tester, I wouldn’t spend a great deal of time worrying about trivial issues such as this. (If you have a lot of dups, it is probably not a dev conspiracy against you.)

9. No Debug Information

It amazes me how many testers execute tests on a test platform that is incapable of capturing critical failures with a simple post-mortem debugger such as WinDbg. Obviously not all defects result in an access violation, but when an access violation or other type of exception occurs it is important that testers capture information that will help developers troubleshoot the problem. This is especially important for random or intermittent exceptions that are hard to reproduce. A debugger can be used to create a dump file which is essentially a snap shot of the system’s memory at the time of the failure. Developers can use a dump file to help isolate the problem, even if the exception is hard to reproduce. I highly recommend that testers talk with their development team to determine exactly what types of information testers should include in their defect reports when reporting access violations.

10. Multiple bugs in one report

I was also a bit surprised about this complaint. I would think that testers would not group a lot of bugs together (because we all want higher bug counts), but I guess the developers are saying otherwise. I have usually seen this when testers perform ad hoc or exploratory testing and through a series of steps come across different bugs, but report the different symptoms they encountered in one defect report. I think (hope) this problem is limited to more inexperienced testers.

These issues may or may not apply to your testing organization. But, when I showed this list to several test managers at Microsoft they were quite surprised. Those test managers were interested in providing direction to their teams on writing higher quality defect reports but never considered consulting with the development team, and their proposals missed some of the pain points developers in their group were complaining about. While many consider testers as the advocates for our “external” customers, perhaps it is our product team members (developers, managers, etc) who are really the key customers of the tangible artifacts produced by the testing team.