Originally Published Sunday, September 03, 2006
Exploratory testing is a topic often embroiled in unreasonable controversy. I didn’t always understand what all of the hullabaloo is about because the simple fact is that all testers engage in exploratory testing, and have used it in software testing for decades (although the early pioneers of software testing simply called it ‘testing’). Testers used exploratory testing approaches quite successfully long before Cem Kaner coined the phrase “exploratory testing.” And testers certainly engaged in exploratory testing long before a few consultants started exploiting the phrase by promoting exploratory testing as some new-fangled testing strategy and disregarding the need for greater technical knowledge and skills in professional testers.
Some readers may comment that I have expressed disdain for exploratory testing, which is simply not correct. I use exploratory testing approaches all the time (even before it was called exploratory testing) and I realize its value in the correct context. My contention with the subject of exploratory testing has nothing to do with the approach per se. My derision is with the ‘experts’ who divide our community into an ‘us’ versus ‘them’ political debate, separate testers into different ‘schools’ of thought, and proselytize exploratory testing as the holy grail of software testing with the zeal of religious extremists.
But, after responding to some comments from Michael Bolton this week, I sat down and thought about the topic of exploratory testing for awhile. (I have never met Michael, but I think we agree on the basic tenets of software testing, and I hope our paths cross sometime in the future so we can share our thoughts.) Michael is an articulate guy, and he said something that clicked and made all the dots connect for me. So, read on and see if the dots between exploratory testing and formal techniques connect for you.
The first dot: Checking assumptions
Michael wrote, “…testing is not merely a rote, mechanical task; that it’s not merely the application of engineering principles; that it’s not simply comparing the product with the spec; or other such simplifications.” Upon reading that I realized that exploratory testing zealots consistently either choose to ignore the value of testing techniques such as boundary value analysis or equivalence class partitioning, or they have never worked on a software testing project that required the testers to actually use their intellectual knowledge or technical skills.
I guess I have been privileged and have never had to work with or manage software testers who are simple-minded droids incapable of thinking for themselves, blindly use tools or techniques without understanding their purpose or capabilities, and have a very narrow interpretation of the role of the software tester. On the other hand, many of the ‘successful’ software testers who I have worked with and managed possessed the characteristics and traits we commonly look for during the interview process such as problem-solving and analytical thinking, precision questioning, and the capacity to learn new concepts and apply them. Early in the interview process we often dig around until we found an engineering principle or concept the interviewee is unfamiliar with, and take a few minutes to explain it to them. Later in the interview process a different person on the interview loop will ask the interviewee about that principle or concept and see if they were not only to remember it, but to process the knowledge in a manner in which they could think of innovative ways to apply it.
I assume that many good testers already have the knowledge and capacity to learn, to think critically, to analyze, to ask relevant questions, and to “understand, diagnose, and solve problems”. Therefore, as a mentor, a teacher, and a software testing professional my goal is to help novice testers unlock their intellectual horsepower and their ability to successfully exploit new ideas such as testing techniques in order to be more effective in their role as a professional software tester, and more efficient with their time.
I don’t assume that a majority of testers lack the capacity to think for themselves or incapable of logical reasoning. And, contrary to Michael’s assertion I don’t agree that “…one doesn’t have to be an expert in anything to be successful (i.e. employed) as a professional tester; one doesn’t even need to be expert in testing itself.” First, I don’t equate success with employment. If Michael’s use of the term ‘professional’ implies “a person who earns a living in an occupation frequently engaged in by amateurs” then I concur that our industry has seen its share of self-proclaimed testers who are simply keyboard monkeys or checklist drones. Fortunately, the maturation of the testing discipline is weeding out the amateurs and faux testers. So, when I write or speak of professional testers I am referring to knowledgeable, highly skilled individuals who are “expert in their work”, who “show great skill”, and who “engage in a pursuit or activity professionally.”
The second dot: Techniques mature from exploration
Exploratory testing activists state heuristics are a key component of exploratory testing. In fact, Kaner and and Tinkham state “…exploratory testing is a highly heuristic process that uses heuristics.” So, let’s talk about heuristics. To put it simply heuristics:
- serve to indicate or point out; stimulate interest as a means of furthering investigation
- encourage a person to learn, discover, understand, or solve problems on his or her own, as by experimenting, evaluating possible answers or solutions, or by trial and error
- of, pertaining to, or based on experimentation, evaluation, or trial-and-error methods
- Computers, Mathematics. pertaining to a trial-and-error method of problem solving used when an algorithmic approach is impractical
- Of or relating to a usually speculative formulation serving as a guide in the investigation or solution of a problem
- A rule of thumb, simplification, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood
Experimentation and investigation often leads to patterns or repeatable steps to replicate a desired outcome in a given situation, or systematic procedures by which a complex or scientific task is accomplished (which happens to be the definition of technique). So, can’t we say the progenitors of software testing hypothesized systematic procedures such as boundary value analysis through experimentation and evaluation? And, don’t we agree that when boundary value testing is applied correctly it accomplishes the complex task of verifying the extreme ranges of data more effectively and more efficiently than continuing to use trial and error or educated guesses? So, when a pattern of systematic procedures emerges from experimentation that proves or disproves a hypothesis within specific context our experimentation matures into a technique that can save the tester time and has been proven effective through previous investigation and trail and error.
Functional and structural software testing techniques impart consistent guidelines for highly skilled knowledgeable testers to be more effective, and improve the tester’s efficiency. Techniques are not step by step or rote procedures. In fact, the application of techniques requires a great deal of knowledge and skill.
Educated guessing, trial and error, experimentation, and speculation are tribal knowledge. Attrition of the testers with the knowledge will cause a skill or performance gap in the organization. So, as long as companies continue to hire unqualified, sub-standard keyboard monkeys then the consultants who advocate exploratory testing as the Tao can continue to sell their exploratory testing medicine show. But, I agree with Michael that software testing is a “is a deeply intellectual and challenging activity.” And unfortunately for the exploratory snake oil salesmen many companies also understand that and the caliber of testers hired today is much higher than in years past. The era of hiring the butcher, the baker, and the candle-stick maker to find bugs is over. Companies such as Microsoft, Google, Freddie Mac, and others are targeting knowledgeable, technically skilled individuals as professional testers.
Drawing the line between the dots: Stop fighting and play nicely!
Michael described the testing role quite eloquently stating, “… testing is the process of investigating the product to reveal quality-related information about it, for people to whom that information might matter.” At the end of the day, there are multiple approaches and techniques required to examine or inspect any complex software project to accurately and adequately assess quality attributes and exposure to risk. Simply put, no single approach to software testing is sufficient. Exploratory testing and functional and structural software testing techniques are only some of the tools in the repertoire of the professional tester. They are all valuable when employed in the appropriate context. (BTW…everything has context because nothing occurs in a vacuum.) There is no us versus them, there are not 4 schools of testing, and since everyone does exploratory testing to some degree doesn’t that make everyone an "exploratory tester?" (Of course, if exploratory testing is the only approach to assessing the capabilities and attributes of a software project, then the testing strategy is probably not very mature, and numerous studies prove that approach leaves a lot of holes (untested areas of the product = 100% exposure to risk)).
I think myself, Michael, and other notable professionals in the field are exposing the need for highly qualified, smart, and innovative testers in the industry because testing is extremely challenging. We are also commonly aligned in our desire to teach bright and creative individuals additional testing skills to be more effective as professional software testers. In essence aren’t we just building different walls of a sand castle? Can’t we just get along and play nicely in the same sandbox? Or, do we have to choose sides?
5 Comments
Hi, BJ…
Thanks for writing this post. Alas, it confuses me somewhat.
“…testers certainly engaged in exploratory testing long before a few consultants started exploiting the phrase by promoting exploratory testing as some new-fangled testing strategy and disregarding the need for greater technical knowledge and skills in professional testers.”
If you’re referring to prominnent voices in the community that talk about exploratory testing (people Cem Kaner, James Bach, Elisabeth Hendrickson, James Lyndsay, Mike Kelly, Jonathan Kohl, Jonathan Bach, and me come to mind), I’m confused as to where any of us makes the claim that exploratory testing is some new-fangled strategy. None of us has ever done that. As for the claim that any of the above people disregard the need for greater technical knowledge and skills in professional testers, it would be libellous if it weren’t laughable.
“And, don’t we agree that when boundary value testing is applied correctly it accomplishes the complex task of verifying the extreme ranges of data more effectively and more efficiently than continuing to use trial and error or educated guesses?”
Boundary value testing is swell, but it is itself a heuristic. (Many of us define “heuristic” succinctly as “a fallible method for solving a problem”.) It’s not that we ignore the value of boundary value testing. On the contrary, we use it all the time; it can be used in a scripted approach, or in an exploratory approach. So, sure, we agree that the technique has value. But because boundary value testing is a fallible method for solving a problem, it’s still a sophisticated form of trial and error or educated guess, just like every other test technique. We promote understanding the technique (and lots of others), but we also vigourously advocate understanding how it can fail, and how it can be limited, misunderstood, or misused.
One more thing confuses me. On the one hand, you say “Can’t we just get along and play nicely in the same sandbox? Or, do we have to choose sides?” But earlier in the message, you use this kind of language: “…the consultants who advocate exploratory testing as the Tao can continue to sell their exploratory testing medicine show.” I think this mischaracterizes me and my colleagues, but it certainly seems inconsistent with your plea to play nicely.
Thursday, September 07, 2006 10:47 AM by Michael Bolton
Hi Michael,
After re-reading my post my words seem contradictory to my question as to whether or not we are really promoting the same thing from two perspectives. I guess sometimes in my zealousness I get a little carried away.
Just like you I am also confused at times because James Bach stated “Testing is not computer science.” I guess in general testing is not computer science, but testing computer software is certainly one aspect of computer science.
It also confuses me when the denotation of words are mutated to support an argument. How you define “heuristic” is not necessarily how the world defines the word heuristic. I have looked and nowhere can I find a definition that matches your stated connotation of “a fallible method for solving a problem.”
I think we will agree to disagree on boundary testing. I will refer to it as a technique which is a mature, directed, efficient approach as compared to a heuristical (trial and error, speculation, best guess) approach.
I suspect we are not that different you and I. I absolutely concur with your statement “We promote understanding the technique (and lots of others), but we also vigourously advocate understanding how it can fail, and how it can be limited, misunderstood, or misused.” I don’t know of any respected teacher of software testing who wouldn’t take such an approach.
However, while you state you promote the “…understanding of technique…”, James wrote, “I don’t teach boundary testing in my classes because it’s too trivial to worry about.” So, which is it?
Also, as a side note: there are not “…a buh-zillion techniques” as James would have us believe. In fact there are really quite few techniques. For example, functional testing techniques are fairly limited to boundary value analysis, eqivalence class partitioning, combinatorial analysis (not just pair-wise), state transition testing (aka model-based), and cause and effect (although I agree with Beizer that cause effect graphing is a “complex solution to a complex problem”). This is a much smaller set compared to possible ‘try-this’ heuritics. Also note that I would typically include exploratory testing as a technique (systematic procedure for solving a complex problem); however, James vehemently disagrees with calling exploratory tesitng a technique.
So, here is my world view. I think most testers are extremely bright, analytical thinkers, with great problem solving skills. I teach people how to be most effective within the limited time periods to evaluate a software product and provide qualified information about the assessment for managers to make strong business decsions regarding exposure to risk.
My approach to teaching testers involves teaching formal techniques (targeted testing) to examine the product in a way to increase coverage and confidence while minimizing the guessing aspect. I take this approach because I assume that testers already test (exploratory, ad hoc, error guessing, or whatever we want to call it). It doesn’t preclude exploratory testing. Exploratory testing is adjunct (added or joined as an accompanying object or circumstance – Merriam-Webster) to the application of other formalized techniques.
- Bj –
Thursday, September 07, 2006 1:21 PM by I.M.Testy
Well, I guess I’ll give this one more kick at the can, and then stop.
You said, “I guess sometimes in my zealousness I get a little carried away.” That can happen to anyone, but I must say it’s a risk to my ability to take your postings seriously.
When James says that “testing is not computer science”, I agree with him, for reasons outlined in my previous reply. Computer scientists (at least those who contribute to the ACM/IEEE curriculum guidelines) don’t seem to think of it as computer science, much; as such, I feel they trivialize testing. But the shoe fits on the other foot, too: to call testing computer science trivializes testing. When we test, I hold that we’re not merely testing computer software; we’re testing systems. We’re testing the cognitive systems whereby software developers, requirements developers, documentors, business people, and a host of others develop and test software. We’re testing business systems in which the software is intended to work. We might be testing social, political, or economic systems, in which the software is intended to work.
[Bj - so let me get this straight. You write "We're testing business systems in which the software is intended to work." So, instead of testing the software to evaluate whether or not it provides a technological solution to the customer based on their business needs, we're testing the customers business system to evaluate whether they can use the software we create? You really don't think that do you? Maybe I am all confused. I thought a software tester tests software and a business analyst evaluates business systems. I don't think software testers really test the social system. I think software testers do attempt to create scenarios that personify the social system to evaluate the software's attributes and its ability to provide a technological solution that a social system deems necessary or beneficial.]
When James says that he doesn’t teach boundary testing in his class because it’s too trivial, I (as the co-author of the Rapid Software Testing class) take it to mean that it’s a topic that’s so easy to explain that we can cover in a few moments. Here: we divide things into some kind of conceptual groupings. Some of those things are at the edges of a group, and some aren’t. Often people screw up when they’re dealing with things at the edges of a group, largely because of the way programming languages work–the difference between less-than and less-than-or-equal-to, and similar constructs–but also because the cognitive boundaries between any given observer’s grouping of things. There; we’ve done boundary testing and we’re moving on to something else more significant.
Which is this: teaching boundary testing is trivial because most testers–the extremely bright analytical thinkers to which you refer–are already ready to go way past that. We believe–and our experience confirms–that such people are ready and hungry for a much deeper understanding: the idea that two things are equivalent (or not), or have a boundary between them (or not), based on some person’s model. If you model your equivalence classes trivially, there are four numbers that you need to use for the canonical two-digit integer adding program. If you model your classes richly, there is literally an infintite number of tests that you could run, even for the two digit adding program. The trick is to be conscious of the way that you’re modelling the problem space, and deciding consciously which boundaries and which equivalence classes you’re willing to ignore.
[Bj] This is a great point. The real key to effective boundary testing is through ‘rich’ equivalence classing of the data which not only identifies boundaries at extreme edges, but also boundaries within nominal ranges. But, if boundary testing is so trivial, and if so many testers are ready to move past it, then why in the world do we still see so many problems around boundary values? (Plugging in the data to execute the tests is simple, designing effective tests is where the real brain power comes into play.) Of course, personally I think boundary testing should be done at the unit tseting level. If boundary bugs are found at the system level of testing there are some serious issues with the development team’s ability to do adequate unit tesitng (in most cases).
For example, I think you understimate the number of techniques based on your model of what techniques are.
[Bj] I don’t have a model of what techniques are. A technique to me is exactly how the dictionary defines technique (The systematic procedure by which a complex or scientific task is accomplished – American Heritage Dictionary)
Above, you list techniques where we would view these as classes of techniques. We identify nine classes of test techniques–functional testing, domain testing, stress testing, flow testing, scenario testing, claims testing, user testing, risk-based testing, automatic testing.
[Bj - I would classify these as methods based on the definition (A means or manner of procedure, especially a regular and systematic way of accomplishing something; The procedures and techniques characteristic of a particular discipline or field of knowledge). So, perhaps we are saying the same thing, just using different words.]
For our purposes, techniques (and there are a buh-zillion, the way we look at it) fall into these broad classes, and any given technique could pop up under several of them at once. Here’s a technique: paste a million characters after a question mark on a URL that points to a CGI script.
[Bj - this is not a technique...this is a wild speculation or trail and error.]
Here’s another one: take a pile of data from a log file, put it into Excel, and use conditional formatting, based on some condition, to see what patterns are revealed.
[Bj - investigation (if you are in fact expecting patterns to emerge) or speculation (if your guessing patterns will emerge). Let me flip this and instead of taking "a pile of data" how about if we form the data in such a manner so that it has the greatest probability exposing an error based on failure indicators.]
Here’s another: when you see five fields that accept four digits each, put 9999 into each field, and press OK.
[Bj - assuming the fields are dependent (some calculation going on between all 5 fields) this is a classic BO attack to make sure the dev isn't using a 16 bit signed int (-32768 - + 32767), or if they are for some wierd reason (such as artificially constrained output) then to ensure they handle the exception properly.]
Here’s another one: when you see a field that’s supposed to accept only numbers, put in some letters instead.
[Bj - classic equivalence class subset]
Another: when you see a field that’s supposed to accept four digits, put in nothing at all, and press the OK button.
[Bj - Another classis equivalence class subset]
Another: instead of typing the data in, paste it in from the clipboard.
[Bj - this is actually testing OLE]
Now: you may disagree that these (very trivial) techniques are techniques at all. If so, I’d contend that it’s because you would lump them into the same equivalence class.
[Bj - and there lies the faulty assumption on your part.]
See how this kind of disagreement (which is reasonable, but based on different models of equivalence) could add risk to a test project?
I would argue that exploratory testing is not a technique; it’s an approach or a mindset. We differentiate exploratory testing and scripted testing on a continuum, based on where the test idea comes from and when it arrived. In a scripted approach, the next test idea comes from someone who is often not the tester, it comes from some time in the past, and it ignores the result of the previous test.
[Bj - just to check assumptions and be clear, I have said nothing about scripted testing so I don't know why you continually try to apply this argument or comparison. Certainly you don't assume the judicious application of established formal techniques in which the data, machine states, and system environment (down to the program language) is critically analyzed is 'scripted' testing do you?]
In an exploratory approach, the test idea comes from the person who is executing the test, it comes from right now, and it is directed by the result of the last test. It should be easy to see that a given test can appear at any point on this continuum; that a given test can be a little bit exploratory or a whole lot exploratory, a little bit scripted or a whole lot scripted. The more exploratory the test, the more the tester has freedom to choose the next test and the technique that she uses.
[Bj - How do we know the next test is not redundant? How does the "exploratory tester" know what they have missed? I have nothing against exploratory testing, but I will say the more tester knows about the data, the domain, the system, and the context of the problem space the more effective and more efficient that tester will be. As I've said, techniques are effective systematic (not scripted) procedures to evaluate specific areas of the software in the most efficient manner possible. Exploratory testing is a great adjucnt to the application of techniques used to establish a solid, systematically achieved foundation upon which further methods and experiementation can build.]
One can use functional testing techniques in an exploratory or a scripted approach.
[Bj - Perhaps this is the crux of the problem. It seems to me exploratory pundits have two views of the world...that of exploration, and that which is scripted. Maybe this is an oversimplification, but scripted vs. exploratory seems to be a recurring theme. I don't believe testing is approached from simply either a scripted or an exploratory approach, and I don't think the practice of testing should be polarized by comparing two completely opposite approaches.]
If that’s the case, then there’s some grouping, other than technique, by which we differentiate the two–and understanding that principle of differentiation is a general systems thing, by the way.
But here’s the thing: all testing, on some fundamental level, is the forming of hypotheses and designing experiments to see if those hypotheses play out.
[Bj - yes, but there is a difference between a working hypothesis or provisional conjecture, and a hypothesis which is highly probably based on established facts. For example, some people hypothsized they could fly if they tied enough feathers onto their bodies. That of course is provisional conjecture because nobody has yet to succeed. But, if I hypothsize that a value of 0 in the Width control of the Attributes feature of MS Paint will return an error message because I know the minimum value for a valid width is 1, then that hypothsis is highly probable based on established facts of well-formed code. I will test that hypothsis to verfiy the code is well-formed and the instantiation of the error message when a value of 0 is entered then become fact. In this case, my hypothsis was neither a guess or conjecture.]
Richard Feynman, in his 1961 lectures on freshman physics, described a hypothesis as a guess. You say that boundary testing is “a mature, directed, efficient approach as compared to a heuristical (trial and error, speculation, best guess) approach.” I would argue that boundary testing is a mature, directed, efficient, trial and error, speculation, best guess approach. Boundary testing is the epitome of a heuristic approach; it’s a fallible method for solving the problem of finding bugs. Isn’t it?
Finally, there’s this weird thing which seems to have happened again in this post: you assert “I have looked and nowhere can I find a definition that matches your stated connotation of ‘a fallible method for solving a problem.’” –yet in your previous message, you offer “A rule of thumb, simplification, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood”. What is a rule of thumb if not a fallible method for solving a problem? What is a simplification if not a fallible method for solving a problem?
[Bj - You are mixing arguments again. I am not arguing that a rule of thumb is not fallible. But, you wrote "Many of us define "heuristic" succinctly as "a fallible method for solving a problem." There is a distinct difference between denotation and connotation. So, in actuality the statement should be 'Many of us imply or suggest that a heuristic is a fallible method for solving a problem.']
I urge you: read “Introduction to General Systems Thinging” by Gerald M. Weinberg. It’s a pivotal book, I think you’ll find much in it with which you’ll agree. I’ll bet, if you read it and absorb it, that it will both expand and refine your view of some of the approaches and techniques you promote.
[Bj – in fact I have read all of Weinbergs books. I expecially like his Quality Software Management series. Excellent thoughts. I would also highly recommend books by Beizer, Myers, Jorgensen, Copeland, Whittaker, Hetzel, and Binder.
Cheers,
—Michael B.
Thursday, September 07, 2006 3:57 PM by Michael Bolton
BJ: So, instead of testing the software to evaluate whether or not it provides a technological solution to the customer based on their business needs, we’re testing the customers business system to evaluate whether they can use the software we create?
Yep. We’re questioning things in order to evaluate them. In these kinds of cases, we’re testing the relationships between the (customer, business, cognitive) system and the software; we’re testing to see how the software system fits within the larger system. If we don’t understand that, I can practically guarantee inadequate testing on some level, unless the testing mission has been set very simplistically.
[Bj] “Testing how the software fits within the larger system” is very different than your previous statement that you’re “…testing the business systems in which the software is intended to work.”
BJ: “But, if boundary testing is so trivial, and if so many testers are ready to move past it, then why in the world do we still see so many problems around boundary values?”
My answer is that the topic is being treated simplistically by people who are proponents of shallow models of boundary testing. Or so I believe.
Michael: Here’s a technique: paste a million characters after a question mark on a URL that points to a CGI script.
[Bj - this is not a technique...this is a wild speculation or trail and error.]
Actually, many people would call it an input constraint attack, a technique that is designed to exploit a buffer (a.k.a. boundary) overflow. That’s what Jim Whittaker calls it in “How to Break Software”, for instance. It’s a serious security problem.
[Bj - I should have been more clear in my response. The specific attack is quite valuable, the wild speculation comment referred to the "a million characters." A (one) million is a nominal value. If your trying to overflow a buffer set to a max size of a 16 bit int type then you score. However, do we really need a million characters to exploit a 16 bit buffer? Or can I overflow that buffer with 65,537 characters? If the buffer used the buffer size of a signed 32 bit int, then a million characters wouldn't overflow the buffer because you would need at least 2,147,483,648.
----
Michael: "We differentiate exploratory testing and scripted testing on a continuum..."
BJ: "I don't believe testing is approached from simply either a scripted or an exploratory approach, and I don't think the practice of testing should be polarized by comparing two completely opposite approaches."
Check the dictionary entry on "continuum".
[Bj - yep, I got that part. But, comparisons always seem to be exploratory vs. scripted testing.
BJ: I would also highly recommend books by Beizer, Myers, Jorgensen, Copeland, Whittaker, Hetzel, and Binder.
Thanks for the referrals. I read them years ago (except Lee's book, which I read on the way home from STAR East). But if you've done Weinberg, do the Weinberg thing here.
[from above]
Here’s another: when you see five fields that accept four digits each, put 9999 into each field, and press OK.[Bj - assuming the fields are dependent (some calculation going on between all 5 fields) this is a classic BO attack to make sure the dev isn't using a 16 bit signed int (-32768 - + 32767), or if they are for some wierd reason (such as artificially constrained output) then to ensure they handle the exception properly.]
[/from above]
Rule of Three: can you think of at least three more possible bugs here?
[Bj - I can actually think of more than 3 potential defects that could occur depending on the context. But, without context and more information I don't know which of those defects are probable or even possible, and testing for issues that are improbable or not likely is probably not the best use of my limited time. For example, by digit do you mean whole number or real number? What is the intrinsic type? Are there any artificial constraints on the inputs or output(s) (In fact nothing is known about outputs at this point). The language the program is written in also offers critical insight as to what can go wrong and hence what test provide value, and what tests may be redundant. Instead of wasting my time with wild speculation of all the possibilities, I would rather gather the pertinent information necessary to focus my testing in areas that will provide the highest ROI for my customers.]
—Michael B.
Friday, September 08, 2006 2:25 AM by Michael Bolton
BJ – here are my responses to this post …
simple fact is that all testers engage in exploratory testing, and have used it in software testing for decades (although the early pioneers of software testing simply called it ‘testing’).
By saying this you can not simply take away the new shape this ‘exploratory testing” is taking. It is true that all testers do ET – but recognizing that there are better ways to do ET and if so called ‘ET’ consultants are helping to shape ET – what is wrong with that?
disregarding the need for greater technical knowledge and skills in professional testers.
From where did you get fact that ET consultants disregard technical knowledge – that too greater? Give me an example of a technical knowledge (Application of formal techniques like Boundary value or EQ partition?) ET is itself an important skill of tester (professional or otherwise) so ET consultant can not disregard at least *skill* part. In my opinion the mission of ET is promote useful exploration of the software being tested – ET encourages using whatever helps that mission be it technical knowledge or any other thing that a tester might find useful in a given context.
[Bj] OK…you want an example, Bach has said on numerous occasions that testers don’t need to know a programming language, and should not know a programming language because it could bias their testing. I, on the other hand believe that if a tester is unable to engage in code reviews, or analyze code coverage results, or design white box tests then their skill set is limited, and they should strive to improve in these areas to make themselves a greater asset to their organization. But, please dont misunderstand or mis-quote me. I don’t equate technical knowledge only with coding skills. In-depth knowledge of character encoding methods, network protocols, or other technologies is ‘technical knowledge.’ And, just so we understand each other, I have never claimed a tester only needs technical knowledge; it is just one trait of professional testers.
My derision is with the ‘experts’ who divide our community into an ‘us’ versus ‘them’ political debate, separate testers into different ‘schools’ of thought,
“Us vs them” debate is relevant here because there are two different sets of Testing philosophies here. It is good to have that debate (there is nothing political here) so that testing world understands there are differences. Dividing testers into different schools of thought is a very intelligent and neat approach. Just you are classifying testers as “professional”, ‘keyboard monkeys” – People like James and Cem are justified in putting testers in “factory”, “quality’, “analytical’ and “context driven” school.
[Bj] Once again you are taking my words and use them out of context. I have never classified or segragated testers as professionals and keyboard monkeys. I used the phrase keyboard monkey to describe a group of people who call themselves testers; however their contribution is often limited to simple GUI input/output type testing (I hesitate to call it black box tesitng because API testing often includes a black box approach to test design, but requires testers who can develop stubs or mock objects).
They acknowledge the fact that they represent one school of testing while there are other schools that may be relevant in some context. They identified themselves. Where as people who are so called “professional” testers declare that “testing is a branch of computer science and that “Testing techniques (discovered and used 20-30 years ago) are holy grail of software testing”. It is a great disregard to skilled testing world and human thinking capabilities.
[Bj] I have never claimed formalized testing techniques are the holy grail of testing. I have said countless times they are systematic procedures that help focus the testing effort and significantly aid in the testers ability to identify specific classes of defects.
Aren’t you trying to proselytize “testing techniques” and “professional Testing” (a branch of computer science) as Holy Grail of software testing? You are also speaking with the zeal of religious extremists – only difference is that your philosophy is different.
[Bj] No, I am not proselytizing techniques as the holy grail. Never have, never will.
see if the dots between exploratory testing and formal techniques connect for you.
In my opinions there are no dots between the two. All formal techniques (any technique that will help you in better exploration of problem space) are embedded in ET or its essence.
ET zealots don’t ignore formal testing techniques; they just don’t “oversell” it. A good exploratory tester uses these techniques as appropriate in a given context while recognizing the fact that real testing is much more than and beyond these formal techniques. There is no “blind” faith on any “best practice” – everything is questioned and every probed for its usefulness in a context – ALL THE TIMES.
[Bj] I reread all my posts and can’t find any reference where I suggest “blind” faith in any technique or approach to testing.However, I guess when James Bach states that he doesn’t “teach boundary testing in my classes because it’s too trivial to worry about.” he really means that it is important or useful, but he just doesn’t want to “oversell” it.
or they have never worked on a software testing project that required the testers to actually use their intellectual knowledge or technical skills.
You appear to believe that *only* that tester who is using the formal test techniques is the one qualified to claim his/her technical knowledge or technical skill and others who practice ET can not. This is incorrect.
[Bj] I have never stated formal techniques == technical knowledge or skill and ET != formal technical knowledge and skill. In fact, one’s ability to perform ET effectively relies heavily upon their technical, domain, and system knowlege and their skills to effectively use various tools, techniques and methods.
I guess I have been privileged and have never had to work with or manage software testers who are simple-minded droids incapable of thinking for themselves, blindly use tools or techniques without understanding their purpose or capabilities, and have a very narrow interpretation of the role of the software tester. On the other hand, many of the ‘successful’ software testers who I have worked with and managed possessed the characteristics and traits we commonly look for during the interview process such as problem-solving and analytical thinking, precision questioning, and the capacity to learn new concepts and apply them. Early in the interview process we often dig around until we found an engineering principle or concept the interviewee is unfamiliar with, and take a few minutes to explain it to them. Later in the interview process a different person on the interview loop will ask the interviewee about that principle or concept and see if they were not only to remember it, but to process the knowledge in a manner in which they could think of innovative ways to apply it.
Well said… All that you have mentioned in this paragraph resonates WELL with principle and practice of a GOOD exploratory testing.
when I write or speak of professional testers I am referring to knowledgeable, highly skilled individuals who are “expert in their work”, who “show great skill”, and who “engage in a pursuit or activity professionally.”
You seem to have obsession for word “professional” tester. This is one word that appears in your almost every blog post that talks about tester. Why do you use this word that frequently?
[Bj] Because software testing is the profession I (and many others) have chosen as a career, and so I strive to become a professional in the practice of software testing much like doctors choose to become professionals in the practice of medicine.
According to Merriam Webster dictionary – A professional is one who engages in a *profession* to make is his livelihood. If you add word “Tester” to this, then a professional tester is some one who is engaged in a ‘testing” profession and makes is livelihood. I am sure that is not Michael Bolton’s interpretation of a ‘professional tester”. This *dictionary* meaning does not say anything about thinking abilities and intelligence of tester – it just mentions about *engaging* in a profession and making *making livelihood. You might want to switch over to word “Skilled Tester”.
[Bj] I suggest you look at the dictionary again. In fact, in the paragraph above where you quote me are additional denotations of the word professional that more aptly apply in this context. So, when the denotation of the word professional states “expert in their work” I am assuming one understands that a professional tester is able to think and has a high degree of intellegence. (But, I guess for some folks, everything must be spelled out.)
Exploratory testing activists bloviate the use of heuristics as a key component of exploratory testing.
Your main concern regarding ET comes from the fact that it is a heuristic process
*especially* when heuristic is defined in terms of words that contain –
• Speculative
• Trial and Error
• Simplification or educated guess
What would you say when the word heuristic is defined (and used) in terms of the words like
• Investigation
• Experimentation (not random or sloppy)
• A method of evaluation
• A rule of thumb
• A method of solution that is non algorithmic
• Or simply as “fallible method to solve a problem”.
When ET is defined to follow a heuristic process (a fallible method to solve a problem) – ET can use any of the formal techniques as appropriate.
It is important to note that all approaches to Testing – ET or a bunch of formal test techniques- are based on *FALLIBLE* methods. Also don’t forget that Formal test techniques like boundary value and EQ partitioning also DO NOT guaranty 100% correct results in a sense they are also FALLIBLE – then why have a blind faith in formal techniques and simply attack ET?
[Bj] I am not sure in why you think, or in what alternate universe live in, to suggest that I ever proposed or advocated blind faith in formal techniques because as I said I would never make such a statement. It is simply a cheap lie to suggest that I would write such nonsense. This, my friend, is your assertion. You continuously suggest there is an either / or situation, ET vs. formal techniques (e.g. us vs. them). Again, I would argue software testing is not either ET or formal techniques,and testers must use a multidude of approaches, techniques and methods to be successful in our roles.
Functional and structural software testing techniques impart consistent guidelines for highly skilled knowledgeable testers to be more effective, and improve the tester’s efficiency.
Too many “heavily loaded” words and sentences “consistent guidelines”, “highly skilled knowledgeable tester”, “testers’ efficiency”. Can you explain each of these? I can see a *blind faith* and *obsession* with formal techniques here.
[Bj] It’s funny that you seem to be smart enough to understand big fancy words like epistemology, and cognitive psychology but don’t understand simple compound phrases such as “consistent guidelines,” highly skilled,” and the word “efficiency.” Sorry but I fail to see where my words imply blind faith or obsession with techniques.
Techniques are not step by step or rote procedures.
What are they – then?
[Bj] Techniques are systematic procedures to help solve complex problems, and their effectiveness depends heavily on one’s ability and knowledge to use it correctly. They are not infallible, but they are more effective than best guess. For example, I bet most people know what sutures are, and they probably think they stitch up a wound; however, there are specific techniques to apply sutures. Incorrectly applied sutures can lead to more scarring than necessary, and the sutures may possibly become undone. The same is true of the application of formal techniques. Just because a person can repeat the phrase “boundary value analysis” or “boundary testing” doesn’t necessarily mean they know how to do it correctly or most effectively.
In fact, the application of techniques requires a great deal of knowledge and skill.
This is what ET also says – Knowledge and skill are required to better ET – so where is the confusion?
So, as long as companies continue to hire unqualified, sub-standard keyboard monkeys then the consultants who advocate exploratory testing as the Tao can continue to sell their exploratory testing medicine show. And unfortunately for the exploratory snake oil salesmen many companies also understand that and the caliber of testers hired today is much higher than 3 years ago.
There is no commonality or relation of any degree between ‘ET consultants” and “unqualified, sub-standard keyboard monkeys”. I strongly object this. I would rather construct this as ‘disregard’ or “insult” to “ET consultant and practitioner” community. If you agree that caliber of testers hired today is much higher than 3 years ago – that is largely due to efforts of people MANY of whom are in ET consultant community and those who assert that Testing is a deep intellectual, human thinking based activity.
[Bj] Your assumption is incorrect. My assertion that testers today are better qualified as compared to 3 years ago is based on the fact that many companies have realized that hiring a bunch of people to bang away on the keyboard to find bugs is not as productive as some people would have us believe, and hence many companies are going back to the practice of hiring CS grads, folks with technically strong computer, math and other types of engineering backgrounds into the testing roles.
They are all valuable when employed in the appropriate context. (BTW…everything has context because nothing occurs in a vacuum.)
100% Correct!! Everything has a context – the moment you start forcing “formal techniques as Holy Grail of testing without understanding the context – you ARE ignoring the context. In this connection ET helps you to acknowledge the existence of the context and that all methods in testing are fallible methods of identifying the problems and risks in the application.
[Bj] Please, please, please, show me where I have ever stated formal techniques are the holy grail. Please stop this foolishness.
There is no us versus them, there are not 4 schools of testing, and since everyone does exploratory testing to some degree doesn’t that make everyone an “exploratory tester?”
4 schools of testing DO EXIST as there are as many as philosophies and beliefs practiced by testers. You can not deny that fact. You would agree that you belong to “Analytic school of testing that says “Software testing is a branch of computer science” and that “there is one correct answer”
[Bj] As i stated in the other response to your post, I do not agree there are 4 schools, so I don’t agree I belong to any particular “school.” I think the concept of segregation of values into schools is a rediculous idea.
I am not sure whose blog you are reading, but I have never said there is one correct answer. I also never stated testing is a branch of computer science. (I have stated the testing discipline is a field of computer science.) So, if you are going to quote or paraphrase me, please be accurate.
Of course, if exploratory testing is the only approach to assessing the capabilities and attributes of a software project, then the testing strategy is probably not very mature,
True – ET is a continuum – A good test strategy involves a good mix of scripted and exploratory Testing.
[Bj] Here is a hint, scripted != application of formal techniques,
And numerous studies prove that approach leaves a lot of holes
Can you point me or give references to those studies (ones that explored ET in detail and have concluded that ET leaves lots of holes in testing)
[Bj] Try reading Beizer, my notes from StarEast2005, SQS case study, etc.
untested areas of the product = 100% exposure to risk.
What makes you to believe or assert that ET leaves or causes “Lots of untested” areas of the product. And what other *reliable* methods and techniques do a 100% perfect job of testing all areas of testing – Formal test techniques?
[Bj] Did I ever state techniques do a perfect job? No I didn’t so can you please stop perpetuating that lie about what I wrote.
Again, with regards to untested areas see the references stated above.
Shrini Kulkarni
Tuesday, October 31, 2006 9:21 AM by Shrini