Originally posted June 01, 2006
Last year James Bach wrote that he “…”invented” testing…mainly by discovering that the problems of testing have already been solved in the fields of cognitive psychology, epistemology, and general systems thinking.” Well, there are simply 2 things wrong with the 2 points James makes in his statement. First, I am pretty sure James didn’t invent testing. Although, he has been very successful at taking the most commonly used testing method, wrapping it with the fancy moniker of ‘exploratory testing,’ and having us believe that we really don’t understand it (until of course we drink the ‘kool-aid’ and concede exploratory testing is the best thing since sliced bread). Secondly, the problems of software testing are complex and they certainly aren’t already solved. I won’t pretend to know what the solutions are to the problems we encounter in software testing, but I am pretty confident the solutions are not going to be found by studying cognitive psychology, epistemology, or systems thinking.
Now, don’t assume that I simply dislike James Bach. I agree with James on several points regarding the need for better tester education, the value (or lack thereof) of tester certifications, and some of his earlier writings. However, I do admit that James and I disagree on several issues. Professional disagreement is not bad; in fact it sometimes sparks healthy debate, and motivates thought so we can reach our own logical conclusion and reconcile alternative points of view.
One point of contention is the correlation of software testing with branches of philosophical, psychological, and sociological fields of study. The idea that solutions to the problems in software testing practices and processes can be found by studying philosophy, psychology, and systems theory is twaddle. These are interesting topics and can certainly heighten a person’s ability to think abstractly and question assumptions. But, are these subjects really unique to software testing or do they also apply as equally to development, or to any other job requiring a high degree of innovation and ingenuity?
Epistemology
Epistemology is the philosophical study of the origin, the nature, and the scope of knowledge. When I find a defect in software testing, I must admit I really don’t spend a lot of time philosophizing how I know that I know this is a defect? Current theories in epistemology tend to focus on the subjects of truth and belief. Imagine a Venn diagram that illustrates an intersection between belief and truth as knowledge. So, I guess that if I believe an unexpected behavior to be a defect, and the requirements confirm it to be a defect, then I now know it is a defect. But, of course, if the requirements are wrong, then I can only theorize it is a defect, but I may be wrong because then it is only my belief (which may or may not be correct). Frankly, I am not too sure that understanding the difference between priori knowledge versus posteriori knowledge will help me decide whether or not the unexpected behavior is a defect. I am also not too sure how understanding the differences between the foundationalism and coherentism approaches to knowledge will help me design better software tests. Personally if I were to correlate software testing to a non-engineering field of study, I would say that software testing is more akin to archaeology than it is to philosophy. To quote Indiana Jones, “Archaeology is the search for fact… not truth. If it’s truth you’re looking for, Dr. Tyree’s philosophy class is right down the hall.”
Cognitive psychology
Cognitive psychology is the branch of psychology that studies mental processes. Basically, it studies how we learn and process that knowledge into behavior. As an educator and consultant I am very interested in how we learn and how we use the knowledge we acquire to reach logical conclusions, make reasonable decisions, and solve complex problems. Understanding cognition helps me design and develop better curriculum to train new software testers into professional testers and work with teams to resolve problems or implement new ideas. But, do new testers really need to study how people learn and how they process information, or do they simply need the capacity to learn engineering principles and apply that knowledge to design effective tests.
Systems thinking
Systems thinking is related to general systems theory which studies the organization and interdependent relationships of systems. This seems logical in computer science, but general systems theory is usually applied to natural sciences and systems thinking is typically used to model human organizational behavior. Systems thinking views the ‘big picture’ of organizational relationships and the environment on a system. Systems thinking is mostly applied to change management processes because small events in an organization can sometimes cause catastrophic consequences. Effectively communicating decisions and continuously managing the processes effecting the change in an organization to minimize isolation of any part or element of the organization. Consultants must understanding change management and system thinking in order to be successful because their primary role is to introduce change to an organization. But, from a software testing viewpoint I am thinking that systems thinking may not be the best field of study to pursue. Perhaps a better focus would be systems philosophy, which is a form of systems thinking that focuses on design and root cause analysis. Systems philosophy studies the development of systems expressed as models. We often create models to help understand complex systems. In software testing we create models to help us design state transition tests.
I am not an expert in systems thinking, cognitive psychology, or epistemology. And, the simple fact is that I don’t need to be an expert in social sciences to be successful as a professional tester.(Of course taking courses in social sciences in university helps one think abstractly.) And I certainly wouldn’t try to convince anyone who is serious about a career in software testing that they should spend a lot of time studying various branches of philosophy, psychology, or other human sciences to be an effective tester. As I said above, while these topics are certainly fascinating, they are of no greater importance to the practice of software testing then they are to software development or any other discipline that requires a high degree of intellectual horsepower and creative innovation.
3 Comments
I think the quotes in James’ use of “invented” are intended to convey irony; they refer to the previous paragraph, in which James had to discover many things about testing for himself, and that many of the things that he read in the literature ranged from the impractical to the preposterous. I’ve followed the same kind of path; I noticed over many years as a tester that I could identify serious problems in software without drawing cause-and-effect diagrams, and that the people who were doing the cause-and-effect diagrams were missing lots of important problems. This was largely because they were okay at the diagramming technique, but they didn’t have a sufficient understand cause and effect and other critical thinking skills.
I’m confused by your assertion that
“The idea that solutions to the problems in software testing practices and processes can be found by studying philosophy, psychology, and systems theory is twaddle.”
You seem to contradict yourself when you say, immediately after that,
“These are interesting topics and can certainly heighten a person’s ability to think abstractly and question assumptions.”
Are thinking and questioning assumptions not core to the testing mission? You seem to agree that they are when you ask
“But, are these subjects really unique to software testing or do they also apply as equally to development, or to any other job requiring a high degree of innovation and ingenuity?”
So I’m confused; if these skills are important to development (as I agree) and any other job, etc. (as I agree) are they not important to testing. Can you help me understand? Perhaps it would help for you to clarify your ideas about what the big problems of testing might be.
You point out that “Secondly, the problems of software testing are complex and they certainly aren’t already solved.” I don’t think that James is suggesting that problems that we’re having in testing have been solved; I think he’s suggesting that we might find solutions if we study these other disciplines.
I’m not sure how you (or a fictitious archaeologist spouting philosophy while disclaiming it) distinguish fact and truth. But thinking and reading about epistemology reminds me that issues related to testing and quality are subjective and fundamentally uncertain, just as you point out in your description of trying to decide whether something is a bug. Asking “wait–how do I know this?” is a question that expert testers ask themselves all the time, isn’t it? I’d argue that if there’s not some fundamental uncertainty lurking about, I’m not asking enough questions to be testing effectively.
You suggest (and I agree) that “Systems thinking is related to general systems theory which studies the organization and interdependent relationships of systems.” Isn’t software testing all about questioning the organization and interdependent relationships of systems?
You suggest (and I disagree) that “general systems theory is usually applied to natural sciences and systems thinking is typically used to model human organizational behavior.” I would have thought that general systems theory is applied to systems generally (else it ceases to become general), and that systems thinking is applied to whatever systems you care to think about. That is, I think your interpretation of general systems is specialized somehow.
I’d also suggest that in software testing we create models to help us design all of our tests, not just state transition tests. A model is a subset of a space; it’s a representation of a slice of some reality; it’s a map of some territory. We model users; we model the input space for functions; we model workflows; we model platforms; we model everything because we can’t cover everything.
Finally, you assert “the simple fact is that I don’t need to be an expert in social sciences to be successful as a professional tester.” Indeed. These days, one doesn’t have to be an expert in anything to be successful (i.e. employed) as a professional tester; one doesn’t even need to be expert in testing itself. The quality of the software that we ship and that we use is testament to that. James (and I, and many others) would like to change that by recognizing that testing is a deeply intellectual and challenging activity; that we provide value to management and to our customers when we think critically about software in ways that others don’t.
Perhaps your focus is on what you’d teach new testers. I would (and do) teach them that testing is not merely a rote, mechanical task; that it’s not merely the application of engineering principles; that it’s not simply comparing the product with the spec; or other such simplifications. Instead, testing is the process of investigating the product to reveal quality-related information about it, for people to whom that information might matter. I contend that this work starts with thinking skills. Where does it start for you?
—Michael B.
Thursday, August 31, 2006 6:43 AM by Michael Bolton
Hi again Michael,
Your assertion that techniques, or engineering principles or concepts are “merely rote, mechanical task[s]” is simply fallacious, and trivializes the excellent work of world renowned industry professionals such as Copeland, Myers, Beizer, Binder, Whittaker, etc. I would expect such a juvenile statement from someone who does not understand formal testing theory or the judicious application of techniques to improve effectiveness and efficiency; not from another professional.
Techniques are not gimmicks or rote mechanical tasks. In fact they require a great deal of knowledge and skill to apply effectively. Techniques are not the only tools in the professional tester’s toolbox, and similar to tools techniques can be fallible if applied incorrectly or by untrained, unknowledgeable individuals. It is also no secret that all professional testers have used exploratory testing as a testing tool long before Kaner coined the term, and long before a handful of consultants exploited the phrase for monetary gain promoting it as the holy grail of software testing.
You and I agree that great testing starts with great thinking skills. The difference is I don’t try to teach people how to think, I teach them to unlock their intellectual horsepower and use their knowledge to solve complex problems in software testing effectively and efficiently. The testing courses I teach at Microsoft and the University of Washington, and the presentations at various conferences focus on fundamental discipline knowledge and technical skills necessary to succeed as professional software testers.
I can teach an intelligent person with great problem solving skills and who knows how to think analytically the fundamental techniques, methods, and engineering concepts and principles used in software testing. More importantly I also need to be able to assess whether or not the people I teach can effectively apply what I taught them via Human Performance Technology (HPT) models and other learning effectiveness measures.
Software testing is not a religion, it is not a political debate between “us” and “them,” everything has context because real life doesn’t occur in a vacuum, and there are not 4 distinct schools separating testers or their approaches to solving problems in software testing. Software testing is a professional discipline in the field of computer science. Professional testers require great breadth of knowledge and the skills because there is not a single approach that works in all scenarios. At the end of the day, I think we both want professional testers making smart decisions about what and how to test each unique problem space they encounter using the most appropriate technique, method, or approach, in order to present qualified information for accurate risk assessment.
- Bj –
Saturday, September 02, 2006 4:34 AM by I.M.Testy
Hi, BJ…
“Your assertion that techniques, or engineering principles or concepts are “merely rote, mechanical task[s]” is simply fallacious”.
Yes, it would be, if that’s what I had said. I’d invite you to have a look at the sentence to which you refer and read what it says. Here it is again.
“I would (and do) teach them that testing is not merely a rote, mechanical task; that it’s not merely the application of engineering principles;…”
That’s not an assertion that engineering principles are rote mechanical tasks. HERE is an assertion that engineering principles are rote mechancal tasks:
“Engineering procedures are rote, mechanical tasks.”
See the difference between the two sentences? I do. Perhaps I can help. Note that I differentiate the two in the first sentence, and equate the two in the second. Note that I disagree with the second sentence, and stand by the first one. Let’s go on.
“…that it’s not simply comparing the product with the spec; or other such simplifications. Instead, testing is the process of investigating the product to reveal quality-related information about it, for people to whom that information might matter.”
You agree with that, don’t you?
You suggest…
“Software testing is a professional discipline in the field of computer science.”
I think that’s true to some degree. Alas it’s not very well supported by the certain professional organizations in the field itself. The content of the ACM/IEEE’s curriculum guide to this day is less than one full course in a four-year program. I wish those guidelines addressed testing more; as it is, the I think professional organizations trivialize it.
So to be clear, testing IS a professional discipline in the field of computer science, but it’s also much more than that. Testing involves asking questions about the way humans interact with the software. It involves asking questions about the way that humans can make mistakes as they’re developing software, and as they’re testing it. So in additional to technical skills, we need other kinds of skills too.
I agree when you say that “Techniques are not rote mechanical tasks”, but I fear that they can become that way if we teach testers that other skills aren’t important too. In your original post, you assert “I am not an expert in systems thinking, cognitive psychology, or epistemology.” I’m not either, but I have studied them a good deal in the effort to gain more expertise. The value that these studies provide to me is that they add to “great breadth of knowledge and the skills”, and help me to understand ways in which “there is not a single approach that works in all scenarios”, as you put it. Technical skills are important things, but they’re not the only important things.
My guess is that we agree on too. Don’t we? It’s weird; I see you objecting strenuously to my message, and then proceeding to agree violently with it on many points.
Thursday, September 07, 2006 10:17 AM by Michael Bolton