But, one statement that has always troubled me that I saw repeated in the Agile Testing book is in regards to exploratory testing as the most effective way to expose important bugs. Every time I hear/read this or one of its many derivatives it makes me ponder. If this is really true then why do we invest so much time and resources in other approaches to software testing. Why do we spend so much time in static and dynamic analysis, reviews, and coverage analysis? Why do we try to prevent bugs when they can (and presumably are) found easier and ‘quicker’ downstream via exploratory testing? Why don’t we simply continue to hire hoards of diverse people to rapidly explore the product just before release and beat out all those important/critical bugs?

I wonder why it seems that we can ‘think critically’ when we have the product to play with, but we can’t seem to apply critical thinking while reviewing documented requirements, or a model? Personally, I am not sure that ‘trying’ something and then deciding what to do based on the outcome or state of the computer, or deciding whether something is a bug after the fact constitutes critical thinking. Perhaps sometimes it is, but I suspect that sometimes it is just guessing. To me, critical thinking involves the ability to foresee potential issues; not simply stumble upon them, or apply an attack from my bag of exploratory tricks to reveal them downstream in the development lifecycle when I have the product to play with. 

I sometimes wonder if exploratory testing really is the holy grail of software testing, or if we have just given up on other approaches because they are too hard. Could it be that our testers are not capable of ‘testing’ without also engaging their fingers? Are our testers untrained or under-trained in our profession? Is the reason why our pre-defined test cases are seemingly ineffective due to the realization that we suck at test design, and learning how to design more robust and ineffective tests is just too hard? Is the reason we can’t think of tests until we get the product in hand perhaps due to our inability to model or is it because we really don’t know that much about the systems we are testing? (You can’t model what you don’t understand and you don’t understand what you can’t model.) Besides, why should we burden testers with the overhead of learning about programming concepts, computer science, and math. You don’t need all that stuff to find bugs.

Now many of you might think that I despise exploratory testing. I am not anti-exploratory testing (whichever form that takes); because we all do it! Show me a tester who doesn’t include some form of exploratory testing in his or her repertoire and I’ll show you a thoughtless simpleton who needs explicit instructions for determining which side of the heal of a loaf of bread to butter. But, I have always thought that we could actually improve the craft of testing. I thought we might play a much bigger role in an organization. I thought we could help reduce overall production costs by being involved earlier and throughout the lifecycle. I thought we could help reduce risk through defect prevention and in-depth analysis. And, I thought we could learn to use tools and techniques to help us design better tests up front, and rely on a team of testers who are capable of thinking while executing a test case.

Yes, it is true that exploratory testing is fun, and we don’t really need to understand all that computer science stuff to find bugs using an exploratory approach. Regardless of what empirical studies on the topic reveal, exploratory testing ‘feels’ so much more productive. Why should we base our profession on evidence when we can get by so easily with emotion? Why should we have to think critically during the design or implementation of software when we can simply muddle our way through some graphical user interface to find bugs? If our approaches to unit testing and component level testing are so shoddy as to simply invoke happy path” tests then of course finding bugs by exploration seems rather logical. But, as developers become more adept at unit can component level testing in flushing out functional bugs earlier we may only need ‘testers’ to explore the product before release for behavioral issues, compatibility problems, and the few remaining obscure functional bugs. Or possibly exploratory testing providing a ‘good enough’ service to our team by finding the bugs that matter to the majority of customers, and perhaps that is ‘good enough.’

Personally, I think we can improve our testing processes. I think we will need to improve the practice of software testing. I think that our job is not simply to find bugs, but a more important objective is to prevent defects (which can help reduce overall product costs). I also don’t view testing as destructive, and I think those who do are doomed to become obsolete. Successful businesses don’t hire folks who just want to ‘destroy’ the products they build, I suspect they want to hire people who can help the company grow and improve. Finding bugs helps me ship a product, but it doesn’t necessarily help me grow the business, reduce production costs, or improve quality long-term.

But maybe we don’t need to change. Maybe our businesses are satisfied with the status quo. Maybe we don’t need to think about cost savings or helping our teams mature their processes, reduce overall risk, and potentially improve quality earlier. Maybe I am too  idealistic thinking that testers can provide greater value to an organization beyond finding bugs at the end of the development cycle.  Maybe our craft has reached the the apex of of our profession and this is as good as it gets.

Of course, I know it is un-unpopular to question the claims of exploratory testing ‘experts,’ but I just happen to be one of those people who question things that are founded on anecdotal observations without any hard data to substantiate those claims. I certainly don’t have all the information, but I personally like to be able to back up my position with facts (known at the time) and several verifiable/repeatable data points so I can answer questions from a defendable position rather than trying to convince or cajole someone with my subjective opinion. (I know a lot of studies show that many Americans base their decisions on their emotional state at the time. But I learned a long time ago that you should never buy the boat you fall in love with because you will spend more time maintaining her than sailing her.) Also, it’s easier to persuade me that I might be wrong with solid, verifiable information and repeatable data versus emotional rhetoric or personal insults.

I think most people who promote exploratory testing are well intentioned and realize in conjunction with other testing approaches that exploratory testing adds value to any testing effort. I also think that many practitioners realize that while we must not only hone our intellectual capabilities of critical thinking and logical reasoning, we must also constantly build our knowledge and skills of the other approaches, methods, and techniques used in our professional trade.

At Microsoft, I can’t think of any testing group that does not use exploratory testing as part of its overall strategy. We have learned not to rely on exploratory testing as our primary approach because it simply doesn’t scale as project size and complexity increase, and it is easy for testers to focus too much on out of context issues in hopes of finding another bug. As one Principal Test Manager summarized, exploratory testing helps

• flush out “low hanging fruit” (identify obvious issues very quickly)
• provide welcomed context switching by getting folks to look at other areas of the product
• to seed new testing ideas or helps identify holes (which is great as long as we have a way to preserve those ideas and they are learnable by other testers)

But, of course, it was also noted that greater ‘system knowledge’ and an understanding of other various testing techniques and approaches enriched the overall effectiveness of the testers on the teams. My job as a teacher and mentor of software testing is to take really smart people who already know how to think critically about problems and provide them with the foundational knowledge of alternative techniques, methods, approaches, and the skills that are specific to the profession of software testing that will enable them to decide what approach to use depending on the context.

Similar to other testing approaches exploratory testing has benefits and limitations and is more effective in exposing certain categories of issues, and is less effective at exposing other types of problems. (See post on Pesticide Paradox.) And now we have researched case studies that begin to help us understand how to utilize exploratory testing as part of our overall testing strategy. Of course, further research could be done in this area, but it is very interesting that the independent studies used in the article reached similar findings and conclusions.

Anyway, I look forward to comments or feedback on the article.

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.

Let’s use a simple example to get started. Suppose we had to drive from Lynnwood, Washington to Puyallup, Washington without a map or (GPS auto navigation system). Just as we have ‘clues’ to point us in the various directions while performing exploratory testing at the user interface we have the numerous highway signs to help us navigate various routes to complete our journey. And, it is up to us to decide which route to take. The shortest route is I-405 south to SR-167. But, I-405 is always at a stand-still, so another popular route is I-5 to SR-18 east then SR-167 south. Of course, after traversing those routes a couple of times the scenery (and crawling in traffic) gets a bit boring, so we might find additional less travelled routes. But, regardless of how many times we make the journey or how many different drivers we choose to complete this journey it is highly unlikely that we will traverse every possible route in any reasonable amount of time. Some routes may not be obvious such as I-5 south to Seattle, then taking the ferry from Seattle to Bremerton and continuing to SR-310 south to SR-16, then I-5 north to SR-167. And, of course some routes are impossible (or at least so convoluted they would be improbable).

Fortunately, control flow through even complex algorithms is not as labyrinthine as the state roadways in western Washington. And, just as the department of transportation uses various tools to measure traffic volumes testers can use path profiling tools to measure frequently traversed paths through the code. We can also use code coverage tools to see what paths have or have not been traversed, and which decisions are made at branching statements. Using code coverage and profiling tools to map control flow through the algorithm we are able to more thoroughly explore the code. Using our ‘map’ we can learn what paths have not been traversed and even whether or not certain paths through the code are even possible. After we explore the ‘map’ we can more effectively design additional tests to traverse un-tested paths through an algorithm. Common structural test design techniques include  to evaluate code statements, code blocks, simple decisions or branches, or multiple Boolean conditional clauses in a single predicate statement. Then, using those test designs we can execute those tests either using stubs or mock objects at the unit or component level, or through the user interface to traverse those paths to reduce overall susceptibility to risk.

I discuss the various techniques commonly used in structural testing in Chapter 6 of our book How We Test Software At Microsoft, and also address the subject here, and here. Of course, the application of structural techniques is usually referred to as code coverage analysis. But, using this simple analogy hopefully other testers can begin to understand how exploratory testing approaches are used not only from the user interface, but also below the GUI at the code level. As Boris Beizer initially stated, "all testing is essentially exploratory in nature," and code coverage analysis (analyzing code coverage results to learn about, design additional tests, then execute those tests) also makes great use of exploratory approaches inside the box.

I just left Stockholm after spending a week there. That was my second visit to Stockholm and it is truly a remarkable city. I spoke at EuroStar which is the largest software testing conference in Europe, and had the opportunity to meet some old friends and colleagues as well as chat with many other speakers and delegates throughout the week. My friend Steve Allott invited me to meet with the senior managers of Know IT, which is the largest IT consulting company in Sweden where I discussed controlling test environments and techniques to generate real-world and random test data. This next week I will be in Denmark where I will be teaching at the Microsoft campus in Vedbaek.

While at an after-event gathering (a really great party thrown my the wonderful folks at Know IT AB) one evening last week I met a gentleman whom I had previously challenged regarding his comparison of the agile development model to the waterfall model. His response at the time was simply that most people in the industry were only familiar with the waterfall model, so that’s what he used as a comparison. At the gathering he approached and chided me by asking if I remembered giving him such a hard time in Dusseldorf. After some light-hearted banter I told him I only gave him such a ‘hard time’ because often we only see concepts compared to a completely opposite concept or approach. What I had asked at the conference was for him to compare agile approaches to other iterative development approaches. This evening he was better prepared for that conversation and he was also able to discuss the advantages and disadvantages of the agile approach in relation to various other approaches, as well as state when other approaches may be more relevant as compared to an agile approach. (Yes, believe it or not, some people still understand that agile is not, and should not be, the only way!)

Often when I hear people talk about exploratory testing (just to clarify again, I think everyone does and should do explorative type testing) it is often compared with scripted testing. But, I often ask myself "Is the testing problem really so simple that it can be solved from either an exploratory or scripted testing approach? Or, are there other approaches, methods, and techniques that I should consider when attempting to prove or disprove a hypothesis of a complex system?"

When I think of scripted testing I think of a test case (often poorly written) that is so prescriptive in its set of instructions that it leaves no ability on the part of the tester to deviate while still proving or disproving the hypothesis or purpose of the test. For example, a scripted test to me is exemplified by the following set of prescriptive steps:

1. Click Start button and select Run… menu item
2. Enter "iexplore" in the Open textbox and press the OK button
3. Enter http://www.google.com in the address bar and press Enter
4. Press ALT + q and enter "pickaxe" in the search field
5. Click the Google Search button
6. Verify the search results contain "Programming Ruby"

An automated version of this test in Ruby might look something like the following:

require ‘watir’
include Watir
test_site = ‘http://www.google.com’
ie = IE.new
ie.goto(test_site)
ie.text_field(:name, "q").set("pickaxe")
ie.button(:name, "btnG").click
if ie.contains_text("Programming Ruby")
   puts "Test Passed.
else
   puts "Test Failed!
end

Now, there may be times when I need a prescriptive set of steps in a scripted test to validate specific requirements, or I need absolute control over what is being tested and how it is being tested. There are other times I want to use an explorative approach to test and test design, such as when I am not familiar with a new undocumented or under-documented feature, or when I want to evaluate the system in ways which I have not previously considered.

But, I also know that between the spectrum of explorative type testing and purely scripted testing there are a wide variety of approaches to testing and test design that are extremely beneficial to the professional tester. When I talk about techniques or methods or approaches commonly used in software testing I always discuss the specific context in which they are useful and situations in which they are not applicable. I discuss strengths and weaknesses, and also the different types of information that can be revealed beyond simply the discovery of potential defects.

From my point of view, professional testing is perhaps the most challenging discipline in the software industry. The complexity of the software, and the virtual infinite number of tests that are possible surely require us to approach the situation from multiple approaches and not simply from an explorative approach or a scripted approach. It is certainly easy to bucket things for simplicity (e.g. if it isn’t exploratory it must be scripted). However, when we view a problem from only two, or one of two directions we are certainly more likely to miss important information (other than simply defects) that may be critical for our managers to better analyze risk and make better business decisions. A professional tester not only realizes that there are multiple approaches to a specific situation, but also understands that each approach has advantages and disadvantages and knows how to leverage those advantages in order to gather the appropriate information.

So, let’s break out of only viewing testing within the context of testing approaches (exploratory vs, scripted), and start understanding the contexts of the software under test (the specific set of facts or circumstances surrounding a particular event or situation) and let’s really learn about all the various approaches, techniques, and methods that are available to the professional tester and better understand when and how to best apply those ‘tools of our trade’ in the pursuit of our profession.

A few weeks ago a read an interesting post on SQA Forums about exploratory testing. It was interesting not because there was anything ‘new’ to learn about ‘exploratory’ testing; but because it offered a compelling counter-argument to ad hoc testing. It is also a good read because it differentiates exploratory testing from ad hoc testing in a surprising revelation.

I, perhaps like many of you, have always assumed ad hoc testing to imply an unplanned, random approach to testing. In our internal defect tracking system at Microsoft two entries for the How Found category included "Ad hoc (directed)" and "Ad hoc (undirected)" and I once thought…’How can random testing be directed?’ Even Lee Copeland writes in his excellent book A Practitioner’s Guide to Software Test Design,  "…ad hoc testing which (by my definition, although others may disagree) often denotes sloppy, careless, unfocused, random and unskilled testing."

But, a Moderator on the SQA Forums site pointed out the actual definition of ad hoc (as an adjective – because the context of its use is important and in this context ad hoc describes the type of testing being performed) is "concerned or dealing with a specific subject, purpose, or end" and "improvised and often impromptu."

In sharp contrast to it’s commonly assumed connotation, the denotative implication of ‘ad hoc testing’ might be better interpreted as "improvised testing dealing with a specific subject or a clear purpose without previous preparation."

Now let’s contrast the denotative implication of ad hoc (or a bug bash) testing to exploratory testing. James Bach defines exploratory testing as "simultaneous learning, test design, and test execution." So, I interpret this to mean that a tester thinks about what to do next based on his/her cognitive abilities to apply their knowledge of testing techniques or methods as well as the domain and system space, and then performs or executes an action (or test) while learning about the capabilities or attributes of the application under test (without having a specific goal or purpose in mind other than perhaps the hope of finding a defect.)

When comparing the bastardized connotative assumption of ad hoc testing with exploratory testing I agree with the SQAForums moderator mentioned above who also stated that "…trying to equate exploratory testing to ad hoc testing is incorrect" and that "They [ad hoc testing and exploratory testing] are really on the opposite ends of the spectrum." But I also don’t think that exploratory testing is equitable to the denotative inference of ad hoc testing. Unless of course exploratory testing is largely improvised, doesn’t require previous preparation and does have a purpose or deal with a specific subject; in which case exploratory testing would literally be the same as ad hoc testing (based on the denotation of that adjectival phrase in the context of software testing)?

(BTW…antonyms of the word improvised include planned and predetermined, and we all know that exploratory testing is not planned or predetermined.)

Now, I don’t think for a moment that this revelation about the denotation of the word ad hoc is going to change the connotative implication of the phrase ad hoc testing. I also don’t think some people will ever change their opinion about exploratory testing as the magic snake oil or holy grail of testing. But this post in SQAForums made me think about the ridiculousness of the whole ad hoc/exploratory testing debate a bit. So, to expunge the emotion and religious rebuttals between ad hoc and exploratory testing, and to perhaps expose some common ground among testers I suggest we simply start using the phrase improvised testing to refer to any testing performed without a pre-defined test case with the intent of learning and/or evaluating the attributes and capabilities of a software project and exposing defects. Improvised testing…yeah…that’s it! No, wait. On second thought the word improvised may also carry negative connotations, so maybe we can call it extemporaneous testing, or maybe autoschediastical testing. Now, that sounds pretty damn impressive and sexy! Way better sounding than just simply saying testing. Who’s with me?

(Oh..we do (and have for many years) perform ‘directed ad hoc testing’ all the time at Microsoft…but we lovingly refer to it as a bug bash! A bug bash is when we ‘direct’ the testing effort to focus on a specific feature area or type of testing such as security or globalization for a given period of time. A bug bash involves executing improvised tests with a goal or purpose of exposing defects in a feature area or defects of a particular type or classification often with minimal preparation.)

Exploratory testing is a topic often embroiled in unreasonable controversy. I didn’t always understand what all of the hullabaloo is about because the simple fact is that all testers engage in exploratory testing, and have used it in software testing for decades (although the early pioneers of software testing simply called it ‘testing’). Testers used exploratory testing approaches quite successfully long before Cem Kaner coined the phrase “exploratory testing.” And testers certainly engaged in exploratory testing long before a few consultants started exploiting the phrase by promoting exploratory testing as some new-fangled testing strategy and disregarding the need for greater technical knowledge and skills in professional testers.

Some readers may comment that I have expressed disdain for exploratory testing, which is simply not correct. I use exploratory testing approaches all the time (even before it was called exploratory testing) and I realize its value in the correct context. My contention with the subject of exploratory testing has nothing to do with the approach per se. My derision is with the ‘experts’ who divide our community into an ‘us’ versus ‘them’ political debate, separate testers into different ‘schools’ of thought, and proselytize exploratory testing as the holy grail of software testing with the zeal of religious extremists.

But, after responding to some comments from Michael Bolton this week, I sat down and thought about the topic of exploratory testing for awhile. (I have never met Michael, but I think we agree on the basic tenets of software testing,  and I hope our paths cross sometime in the future so we can share our thoughts.) Michael is an articulate guy, and he said something that clicked and made all the dots connect for me. So, read on and see if the dots between exploratory testing and formal techniques connect for you.

The first dot: Checking assumptions

Michael wrote, “…testing is not merely a rote, mechanical task; that it’s not merely the application of engineering principles; that it’s not simply comparing the product with the spec; or other such simplifications.” Upon reading that I realized that exploratory testing zealots consistently either choose to ignore the value of testing techniques such as boundary value analysis or equivalence class partitioning, or they have never worked on a software testing project that required the testers to actually use their intellectual knowledge or technical skills.

I guess I have been privileged and have never had to work with or manage software testers who are simple-minded droids incapable of thinking for themselves, blindly use tools or techniques without understanding their purpose or capabilities, and have a very narrow interpretation of the role of the software tester. On the other hand, many of the ‘successful’ software testers who I have worked with and managed possessed the characteristics and traits we commonly look for during the interview process such as problem-solving and analytical thinking, precision questioning, and the capacity to learn new concepts and apply them. Early in the interview process we often dig around until we found an engineering principle or concept the interviewee is unfamiliar with, and take a few minutes to explain it to them. Later in the interview process a different person on the interview loop will ask the interviewee about that principle or concept and see if they were not only to remember it, but to process the knowledge in a manner in which they could think of innovative ways to apply it.

I assume that many good testers already have the knowledge and capacity to learn, to think critically, to analyze, to ask relevant questions, and to “understand, diagnose, and solve problems”. Therefore, as a mentor, a teacher, and a software testing professional my goal is to help novice testers unlock their intellectual horsepower and their ability to successfully exploit new ideas such as testing techniques in order to be more effective in their role as a professional software tester, and more efficient with their time.

I don’t assume that a majority of testers lack the capacity to think for themselves or incapable of logical reasoning. And, contrary to Michael’s assertion I don’t agree that “…one doesn’t have to be an expert in anything to be successful (i.e. employed) as a professional tester; one doesn’t even need to be expert in testing itself.” First, I don’t equate success with employment. If Michael’s use of the term ‘professional’ implies “a person who earns a living in an occupation frequently engaged in by amateurs” then I concur that our industry has seen its share of self-proclaimed testers who are simply keyboard monkeys or checklist drones. Fortunately, the maturation of the testing discipline is weeding out the amateurs and faux testers. So, when I write or speak of professional testers I am referring to knowledgeable, highly skilled individuals who are “expert in their work”, who “show great skill”, and who “engage in a pursuit or activity professionally.”

The second dot: Techniques mature from exploration

Exploratory testing activists state heuristics are a key component of exploratory testing. In fact, Kaner and and Tinkham state “…exploratory testing is a highly heuristic process that uses heuristics.” So, let’s talk about heuristics. To put it simply heuristics:

• serve to indicate or point out; stimulate interest as a means of furthering investigation
• encourage a person to learn, discover, understand, or solve problems on his or her own, as by experimenting, evaluating possible answers or solutions, or by trial and error
• of, pertaining to, or based on experimentation, evaluation, or trial-and-error methods
• Computers, Mathematics. pertaining to a trial-and-error method of problem solving used when an algorithmic approach is impractical
• Of or relating to a usually speculative formulation serving as a guide in the investigation or solution of a problem
• A rule of thumb, simplification, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood

Experimentation and investigation often leads to patterns or repeatable steps to replicate a desired outcome in a given situation, or systematic procedures by which a complex or scientific task is accomplished (which happens to be the definition of technique). So, can’t we say the progenitors of software testing hypothesized systematic procedures such as boundary value analysis through experimentation and evaluation? And, don’t we agree that when boundary value testing is applied correctly it accomplishes the complex task of verifying the extreme ranges of data more effectively and more efficiently than continuing to use trial and error or educated guesses? So, when a pattern of systematic procedures emerges from experimentation that proves or disproves a hypothesis within specific context our experimentation matures into a technique that can save the tester time and has been proven effective through previous investigation and trail and error.

Functional and structural software testing techniques impart consistent guidelines for highly skilled knowledgeable testers to be more effective, and improve the tester’s efficiency. Techniques are not step by step or rote procedures. In fact, the application of techniques requires a great deal of knowledge and skill.

Educated guessing, trial and error, experimentation, and speculation are tribal knowledge. Attrition of the testers with the knowledge will cause a skill or performance gap in the organization. So, as long as companies continue to hire unqualified, sub-standard keyboard monkeys then the consultants who advocate exploratory testing as the Tao can continue to sell their exploratory testing medicine show. But, I agree with Michael that software testing is a “is a deeply intellectual and challenging activity.” And unfortunately for the exploratory snake oil salesmen many companies also understand that and the caliber of testers hired today is much higher than in years past. The era of hiring the butcher, the baker, and the candle-stick maker to find bugs is over. Companies such as Microsoft, Google, Freddie Mac, and others are targeting knowledgeable, technically skilled individuals as professional testers.

Drawing the line between the dots: Stop fighting and play nicely!

Michael described the testing role quite eloquently stating, “… testing is the process of investigating the product to reveal quality-related information about it, for people to whom that information might matter.” At the end of the day, there are multiple approaches and techniques required to examine or inspect any complex software project to accurately and adequately assess quality attributes and exposure to risk. Simply put, no single approach to software testing is sufficient. Exploratory testing and functional and structural software testing techniques are only some of the tools in the repertoire of the professional tester. They are all valuable when employed in the appropriate context. (BTW…everything has context because nothing occurs in a vacuum.) There is no us versus them, there are not 4 schools of testing, and since everyone does exploratory testing to some degree doesn’t that make everyone an "exploratory tester?" (Of course, if exploratory testing is the only approach to assessing the capabilities and attributes of a software project, then the testing strategy is probably not very mature, and numerous studies prove that approach leaves a lot of holes (untested areas of the product = 100% exposure to risk)).

I think myself, Michael, and other notable professionals in the field are exposing the need for highly qualified, smart, and innovative testers in the industry because testing is extremely challenging. We are also commonly aligned in our desire to teach bright and creative individuals additional testing skills to be more effective as professional software testers.  In essence aren’t we just building different walls of a sand castle? Can’t we just get along and play nicely in the same sandbox? Or, do we have to choose sides?

Last year James Bach wrote that he “…”invented” testing…mainly by discovering that the problems of testing have already been solved in the fields of cognitive psychology, epistemology, and general systems thinking.” Well, there are simply 2 things wrong with the 2 points James makes in his statement. First, I am pretty sure James didn’t invent testing. Although, he has been very successful at taking the most commonly used testing method, wrapping it with the fancy moniker of ‘exploratory testing,’ and having us believe that we really don’t understand it (until of course we drink the ‘kool-aid’ and concede exploratory testing is the best thing since sliced bread). Secondly, the problems of software testing are complex and they certainly aren’t already solved. I won’t pretend to know what the solutions are to the problems we encounter in software testing, but I am pretty confident the solutions are not going to be found by studying cognitive psychology, epistemology, or systems thinking.

Now, don’t assume that I simply dislike James Bach. I agree with James on several points regarding the need for better tester education, the value (or lack thereof) of tester certifications, and some of his earlier writings. However, I do admit that James and I disagree on several issues. Professional disagreement is not bad; in fact it sometimes sparks healthy debate, and motivates thought so we can reach our own logical conclusion and reconcile alternative points of view.

One point of contention is the correlation of software testing with branches of philosophical, psychological, and sociological fields of study. The idea that solutions to the problems in software testing practices and processes can be found by studying philosophy, psychology, and systems theory is twaddle. These are interesting topics and can certainly heighten a person’s ability to think abstractly and question assumptions. But, are these subjects really unique to software testing or do they also apply as equally to development, or to any other job requiring a high degree of innovation and ingenuity?

Epistemology

Epistemology is the philosophical study of the origin, the nature, and the scope of knowledge. When I find a defect in software testing, I must admit I really don’t spend a lot of time philosophizing how I know that I know this is a defect? Current theories in epistemology tend to focus on the subjects of truth and belief. Imagine a Venn diagram that illustrates an intersection between belief and truth as knowledge. So, I guess that if I believe an unexpected behavior to be a defect, and the requirements confirm it to be a defect, then I now know it is a defect. But, of course, if the requirements are wrong, then I can only theorize it is a defect, but I may be wrong because then it is only my belief (which may or may not be correct). Frankly, I am not too sure that understanding the difference between priori knowledge versus posteriori knowledge will help me decide whether or not the unexpected behavior is a defect. I am also not too sure how understanding the differences between the foundationalism and coherentism approaches to knowledge will help me design better software tests. Personally if I were to correlate software testing to a non-engineering field of study, I would say that software testing is more akin to archaeology than it is to philosophy. To quote Indiana Jones, “Archaeology is the search for fact… not truth. If it’s truth you’re looking for, Dr. Tyree’s philosophy class is right down the hall.”

Cognitive psychology

Cognitive psychology is the branch of psychology that studies mental processes. Basically, it studies how we learn and process that knowledge into behavior. As an educator and consultant I am very interested in how we learn and how we use the knowledge we acquire to reach logical conclusions, make reasonable decisions, and solve complex problems. Understanding cognition helps me design and develop better curriculum to train new software testers into professional testers and work with teams to resolve problems or implement new ideas. But, do new testers really need to study how people learn and how they process information, or do they simply need the capacity to learn engineering principles and apply that knowledge to design effective tests.

Systems thinking

Systems thinking is related to general systems theory which studies the organization and interdependent relationships of systems. This seems logical in computer science, but general systems theory is usually applied to natural sciences and systems thinking is typically used to model human organizational behavior. Systems thinking views the ‘big picture’ of organizational relationships and the environment on a system. Systems thinking is mostly applied to change management processes because small events in an organization can sometimes cause catastrophic consequences. Effectively communicating decisions and continuously managing the processes effecting the change in an organization to minimize isolation of any part or element of the organization. Consultants must understanding change management and system thinking in order to be successful because their primary role is to introduce change to an organization. But, from a software testing viewpoint I am thinking that systems thinking may not be the best field of study to pursue. Perhaps a better focus would be systems philosophy, which is a form of systems thinking that focuses on design and root cause analysis. Systems philosophy studies the development of systems expressed as models. We often create models to help understand complex systems. In software testing we create models to help us design state transition tests.

I am not an expert in systems thinking, cognitive psychology, or epistemology. And, the simple fact is that I don’t need to be an expert in social sciences to be successful as a professional tester.(Of course taking courses in social sciences in university helps one think abstractly.) And I certainly wouldn’t try to convince anyone who is serious about a career in software testing that they should spend a lot of time studying various branches of philosophy, psychology, or other human sciences to be an effective tester. As I said above, while these topics are certainly fascinating, they are of no greater importance to the practice of software testing then they are to software development or any other discipline that requires a high degree of intellectual horsepower and creative innovation.