As I said in a previous post, I was responsible for the BVT test suite on international versions of Windows 95. The BVT suite was transparent to the developers, and we still found plenty of build breaks and other defects.

I think it is a common misconception to assume the pesticide paradox is a bad thing (perhaps because some testers assume the role of testing is to simply find bugs).

I think the pesticide paradox can be used to our advantage in testing. If developers use my automated tests, or if I can teach developers to write better unit tests to prevent defects upstream, how is this a bad thing?

Also, I think there is a big difference between a simple scripted test and well designed test automation that includes variability while adhereing to its intended purpose. (I will have to blog about this at some point.)

In our internal courses we discuss state transition testing and use exercises designed to teach people how to build abstract models of features in order to get new perspectives, check assumptions, and increase domain knowledge. I have been professing for years that testers need to develop their design and analysis skills in order to succeed in this profession. The ability to abstract out a feature set to prove or disprove a particular hypothesis is a necessary design skill of testers in my opinion.

Tuesday, August 14, 2007 2:58 PM by I.M.Testy

To strech this a bit — allow me deviate bit around this topic.

You say – some things in [along the long and never ending journey of ] testing [tend] to become “ordinary” (or routine, as_usual). Such things need to be automated to save time for more “non routine” and “extraordinary” tasks.

Fair Enough — two things to take care of as let go these things as “ordinary”.

Pesticide paradox – Developers know what BVTs are run and so superficially make sure that BVT’s still pass – You can say that you periodically review the BVT suite and keep it is variable and *interesting* as you can.

But this is a problem that all scripted tests suffer from. We need to find ways to counter it.

Second – what is ordinary or extra ordinary in micro tests (or testing in general) is according to some model of the system as per the observer (tester). Models present ways for the test designer to think and understand what happens under the skin.

Our model of micro tests – say checking that MS word can open a file on the file system – is based on some model of MS word that focuses on *few* observable attributes while ignoring others. So it is important to learn to model test sytstem in different ways and keep developing new models or adding different dimentsions, views, attributes, Zoom level to existing models …

I just noticed that you have not blogged about models and testing. I belive You should write one ..

Shrini

Monday, August 13, 2007 12:16 PM by Shrini

Mundane was perhaps not the best choice of words, because I don’t consider any test I design to be banal or unimaginary, or even ordinary. So, let me clarify my implied connotation of ‘mundane’ in this context.

Tests (both manual and automated) can be micro (discretely focused on specific functionality) or macro (user scenarios or exploration) and professional testers will design a library of tests that span this spectrum.

For example, a build verification test (BVT) is a micro-type test. Most tests in a BVT suite check for something very specific. The BVT suite is ran after each build (which may occur daily in iterative development models commonly used by many teams at Microsoft).

So, the frequency of the BVT suite becomes ‘ordinary’ in the sense the tests are ran repetatively, and ‘ordinary’ in the sense that I expect my BVT suite to pass (In other words, my BVT suite is a baseline functionality of each new build and I expect that new check-ins do not destablize the build). If there is an error in the BVT suite it is extra-ordinary or unexpected.

In this example, I would consider it valuable to automate these tests to free up some of my time to design (either manual or automated) a greater number of more complex tests.

Which brings us full circle to the intent of this post. I intended to provoke readers to think about the complexity of test design, and imagine different ways to use tools and design tests; including designing tests to emulate some human emotional traits for practical reasoning.

My approach in life when confronted with a problem is to never give up. Problems are temporarily unsolvable due to limitations of current technologies, information, knowledge, or skill sets.

Saturday, August 11, 2007 12:51 PM by I.M.Testy

I always wondered what types of tasks in Testing would be mundane or no brainer? Even a simplistic test that we can imagine can have multiple possible inputs and equally complex outcomes. We ASSUME certain tests/Tasks as mundane according to our model of the system.

Would not it be wrong to term any task of testing as mundane?

Yes, I can think of tasks like test data generation according to a *well documented* business rule – as *some what* mundane ….

What percentage of Testing can be assumed to be mundane?

Any thoughts?

Shrini

Saturday, August 11, 2007 1:06 AM by Shrini