People sometimes assume the entire message is captured in the first sentence; especially in a lengthy paper. This is why outrageous statements simply fuel the ignorance of the misinformed.

- Bj –
Sunday, October 01, 2006 3:48 PM by I.M.Testy

You seem to miss the point that this is a single example on a list of many possible motivations for testing. Sometimes we might test with the goal of maximizing the bug count. Moreover, I don’t think he’s talking at that point about artificially or misleadingly inflating the bug count; I think he’s making a distinction between, given the same amount of time finding a few really super-important bugs, investigating and reporting them in great detail; and finding a whole bunch of bugs (with distinct root causes) irrespective of their relative importance, and investigating and reporting them in somewhat less detail than in the first case.

Note that in some contexts, management does indeed value the fallacious idea that good testing equals the number of bugs discovered. There is much education to be done there, for sure. In my view, counting bugs (or test cases) is a practical guarantee that someone will be misled.

—Michael B.
Saturday, September 30, 2006 3:37 AM by Michael Bolton

Yes, I realize that Kaner has changed his view (and more recently he also changed his view of tester’s need for programming skills). The use of the quote was not to disparage Cem. My comment is meant to point out the fact that a common misconception regarding testing as simply a bug-finding activity was sometimes perpetuated by leading industry experts and I thought the quote from Testing Computer Software 2nd. Ed. epitomized this limited view of the role of software testing.

I design a test case based on the intended purpose of proving or disproving consistency with established facts that are known or can be determined). A test case may comprise several ‘tests’ which support the test case.

For example, if I am testing an edit control’s ability to handle strings composed of valid Unicode characters, then I would iterate through an array, or enumeration, or data file of static strings containing valid characters derived from failure analysis models, then I would auto-generate strings of random length and random valid characters for subsequent iterations of that test case. (NOTE: The explicit purpose here is to validate proper handling of valid Unicode strings > minLength and < maxLength. I am not including boundary testing, invalid characters, or a myriad of other tests that I would also run.)

If one of the tests fail, then that test case fails. If some of the tests fail, then that test case fails. (If they all fail, the project is probably in big trouble!)

For example, let’s assume the edit control isn’t Unicode enabled. Some of the ‘tests’ might pass (those with characters limited to the underlying supported ANSI character set), but the vast majority of the ‘tests’ will fail. Let’s say there are 250 unique static strings we are going to use in our test case, and 225 strings contain characters that are outside the range of supported ANSI characters for that test environment. Do I write 225 bugs; one for every failure?

Absolutely not! That is simply a misrepresentation and artificial inflation of the number of defects. The root cause is that the edit control is not Unicode enabled. We don’t need to tell the developer the control is not Unicode enabled 225 times (because if we do the developer will justifiably resolve 224 of the bugs as a duplicates! (Assuming the same root cause.)) (BTW..this is why I disagree with Kaner’s assertion that we test to “maximize bug count.”)

A bug can manifest itself in many ways. A professional tester will troubleshoot the problem to isolate the cause and identify different paths, variations, or symptoms so the defect report provides more detailed information and enable the developer to effect a better fix. Flooding the developer with several bugs all pointing to a single root cause only serves to artificially inflate the defect tracking database (again reinforcing the fallacious idea that testing is valued by the number of bugs discovered), and discredits professional testers who understand their overall value to an organization is not determined simply by the number of bugs they write.

Thursday, September 21, 2006 6:07 PM by I.M.Testy

Kaner has rescinded his view that “a test that did not reveal a problem was a waste of time” since the book was published, and has said so. See “The Ongoing Revolution in Software Testing”, http://www.kaner.com/pdfs/TheOngoingRevolution.pdf

That paper also has a few more potential purposes for a test that you might like to add to your list.

What’s the difference between a single test that has 30,000 variations and 30,000 tests that are very similar to one another?

The thing that links them is reification error–the notion that a test case, an idea, is a countable thing, rather than a concept.

Cheers,

—Michael B.

Wednesday, September 20, 2006 11:59 AM by Michael Bolton

Actually, I am referring to a quote in the book “Testing Computer Software 2nd ed.”

As I stated previously, I absolutely concur one of the primary objectives of of testing (and thus a test case) is to provide information.

The pesticide paradox applies primarily to completely scripted non-regression type tests that don’t allow the person executing the test to deviate from the instructions (or steps). The paradox doesn’t apply to non-functional test cases. For example, a performance test will run over and over many times. Its overall value does not diminish during the product lifecycle (if perf is a critical factor for release). It also doesn’t apply to true regression tests. Regression tests historically have a very low rate of identifying new defects; however are of significant value in providing information and confidence that specific features continue to function as expected. (Build verification tests are critical, provide high value, but the base set changes very little during the lifecycle.)

You state “A test case can be good in a number of different ways.” I was simply trying to point out those specific ways.

I don’t agree with your second point. In fact, some tests (specifically those for establishing baseline measures) will and do remain “good” during the product lifecycle, and there are some tests (primarily regression tests) that will remain “good” for several years while the product is in maintence.

WRT to variations, they should be built into the test case, not additional test cases. For example, I really don’t like hard coded test data. I realize there are times when it is valuable and necessary, but if I designing a test to validate an input control that takes a string of characters you have to believe my approach is going to throw strings of valid randomly generated characters of random length at that control. So, each time the test is executed it uses a different string Unicode characters of varying length. Now I have variability built into a single test.

BTW…I meant seven categories to match the bulleted list, and corrected that in the post. Thanks for catching that!

- Bj –
Monday, September 18, 2006 3:37 PM by I.M.Testy

We all know about “pesticide Paradox” – a test case can be good only initial few times if looked from a “bug finding ability”. As a test case is executed a number of times, it wears out and is less likely to find a bug or expose a new information about application behavior.

Here are my observations about “Good ness of a test case”
1. A test case can be good in number of different ways.
2. A good test case will not (can not)) remain “good” at all the time or during its life time.
3. A good test case offers easy ways to create variations of its own – hence does not become a victim of pesticide paradox.
4. Attributes of a Test case that important from overall ‘goodness” standpoint
a. Information Value
b. Ease of maintenance
c. Ease of creating variations of it
d. Credibility and Power
e. Provides insight into the feature being tested – serves as knowledge repository
f. Neither too simple (fit to be disposed after run once or twice) nor too complex ( requires good effort to understand the flow – hence more time for execution)

You mentioned that “The rationale for test case usually falls into one of six categories.”

Which six categories? Where are they mentioned?

Shrini