Did you ever notice that when you ask someone to test something the first thing they do is to start ‘testing?’
I often see this in my classes and I ask the person, “what is the purpose of your test?” Typically the response is, “I’m testing this,” or “I’m trying to find a bug.”
Unfortunately this seems to indicate there is no or very little pre-thought that goes into the act of software testing. To some people, testing appears to be little more than simply pounding away at the keyboard and trying whatever flies into our subconscious mind as we interact with the software and declare a bug when we stumble upon unexpected behavior or see something we might disagree with.
This is why I found it especially interesting in my own research, and the case studies by Juha Itkonen that testers who were trained in formal software testing techniques or patterns there was no significant difference in terms of defect rates or coverage between pre-defined test cases and an exploratory testing approach. This is not to say that one approach to testing is preferred over the other. It is not an either or proposition as I explained in my post on the pesticide paradox, and there are certainly more than 2 approaches to software testing. Testing requires multiple approaches to most effectively aid us in collecting and presenting the appropriate information to the decision makers.
But, I am often puzzled that it seems we can easily think of negative or destructive tests once we have the product in hand, yet when we are designing a set of tests from the requirements the tests simply test the requirements and little else. I wonder why it is that we can think of ‘tests’ while executing other tests, but we can’t think of those same tests before hand. Is there some limitation in our psyche that prevents us from analyzing a problem until we are actually faced with the problem (software in hand)?
I don’t think so, but I suspect there is a mental hurdle in that we sometimes feel more productive when we are interacting with software as opposed to sitting back and analyzing the problem more prior to executing well-designed test cases. (More tests doesn’t equal better testing!)
The bottom line is that if we are given a set of requirements and can only design tests that only test the requirements, then we are probably not thinking critically about how to design test cases.
7 Comments
Hello Bj,
During your class I have asked you few times how would YOU approach some applications for testing. Since I haven’t worked as a tester only, my opinion on the problem whether there is psyche limitation ‘that prevents us from analyzing a problem until we are actually faced with the problem’ is that it all depends on our experience. In math school in Bulgaria we were taught first to solve simple problems. Then more complicated ones. But very popular phrase was “to bring/reduce a problem to a familiar one”. Once you master the basics(principles, algorithms) the rest is a lot of practice. What I try to say here is that if the AUT(application under test) is very complex you need a lot of experience to see before hand where the bugs might be, or you will need to break it down to a familiar tasks.
Well said!
Sadly, some folks learn at an early stage that they need to “look busy” for their managers. In their shops, they must hurry up and get testing – no time for thinking here! I fear that this is the root cause for some of the premature testulation that occurs – they just haven’t learned any better.
As Kaner, Bach and Pettichord wrote: “The difference between excellent testing and mediocre testing is how you think.”
http://www.sqablogs.com/jstrazzere/2285/Think+First%2C+Then+Test.html
Bj’s Response Thanks Joe. And you post on this topic is also a great read. Perhaps similar to the TDD movement in dev which is really a rebirth of D.Gelperin and B.Hetzel’s famous quote “Test, then code.” there should be a TTT (Think, Then Test) movement.
It is a lot easier to generate test ideas when you can use the application.
For example, you will not only see the actual design of the details, it is also possible to understand interactions between parts, interactions that can’t be modelled in a requirements document.
So if the application or a prototype is available, I would always use it, it is the fastest way to knowledge about the application.
But on the other hand, in real life, you often want to design tests before the implementation is finished, and in that case I agree that it seems like the test ideas tend to stop shortly after the requirements have been covered.
I believe that one reason for this is the shallowness of the classic test design techniques that are being taught. The examples used are too simple, too requirement-focused, and don’t help when trying to generate test ideas beyond the requirements.
There is the “error guessing” technique, but it isn’t as well described and developed as it should be, if we want to have a test design technique that can be used, in advance, to generate a multitude of test ideas, that aim to look at everything important, not only what is stated in the requirements (that can’t and shouldn’t be complete.)
Personally, I primarily use the product’s history, quality attributes, knowledge, my experience and subjectivity when designing tests.
Hi Nely,
The way I typicallly approach a problem is to break it down into its components to get a better understanding of how each component works and how various components interact. As you say, this is very much how we are taught how to solve complex math problems.
Also I agree that not only is experience important, but I also think a rich understanding of the underlying foundation is important as well.
But testing is not just about finding bugs. Testing includes verifying the various interdependent in a complex system work together as expected. In a nutshell testing serves 2 purposes; 1.) to identify and expose risks, and 2.) to instill a sense of confidence.
Hi Rikard,
It is interesting that when we use the triangle program as part of an interview (as G.Weinberg did at IBM) given just the requirements candidates are able to come up with several dozen tests that are both positive and negative.
It is also interesting to note that when I give students both the requirements and the application the primary focus of tests shifts from tests that primarily exercise computational logic, to behavioral tests that primarily focus on UI interaction and miss a lot of the underlying functional tests (permutations, boundaries, etc).
Perhaps you hit the nail on the head when you say part of the problem is the shallowness of how test techniques are taught. I designed our internal training at Microsoft and we go into great depth into various techniques and how to decompose a problem and apply the right technique in the right situation to identify specific types of faults. However, I will also say that most ‘classic techniques’ require a pretty thorough understanding of the underlying system (code, computer logic, etc) and a solid mathematics foundation.
I think many testers are very good at USING software and identifying issues while interacting with it. But, that seems to indicate that testers are limited to ‘testing’ after the fact (software in hand) rather than thinking critically about tests beforehand which in some case could very well prevent bugs.
“Did you ever notice that when you ask someone to test something the first thing they do is to start ‘testing?’”
What if you looked at things another way:
- did you ever notice that when you ask someone to test something the first thing they do is to start exploring?
- did you ever notice that when you ask someone to test something the first thing they do is to start learning?
I often see this in my classes and I ask the person, “what is the purpose of your test?” Typically the response is, “I’m testing this,” or “I’m trying to find a bug.”
What if you asked them a different question: “What are you doing now?” or “What are you trying to learn?”
In your version of the Triange problem, what would happen if you set up an iterative approach? After a short burst of testing, perform a debrief in which you ask people to describe their coverage so far, and what they would do to improve upon it?
—Michael B.
Hi Michael,
Thanks for the comments, and perspective.
Yes, I sometimes do ask people who dive right in, “what are you doing?” or “what exactly are you testing now?” Often their response is similar, esp. the “I’m trying to find a bug” response.
What I am pondering is if we thought more critically and anaylzed the problem for a few minutes before diving in head first could we design targeted tests in our subconscious that
- might expose some issues with less physical effort
- might also help improve our efficiency
- provide a perspective that we might overlook due to visual distractions in the UI (we found the triangle test cases are significantly different if the tester is given just the requirement as compared to the requirement and a coded simulation to play with. Testers given the requirement alone tend to produce more functional tests. My assertion here is this might occur becuase of less visual distractions)
(Next time we have a beer remind me to tell you about a student in my automation class who, after given a problem, just started coding without having a clear test objective in mind.)
I like your suggestion on the iterative approach. We do that in subsequent simulations and track code coverage and path profiles and test count to assess change, but not in the first ‘warm-up’ triangle simulation. I will have to give that a go this year and will let you know the outcome after a couple iterations. It will be nice to compare notes.