I.M. Testy

Some of you know that I am an organic gardener. I do this not only because I like the fresh taste of vegetables and fruits that are not tainted with chemicals, but my daughter loves to eat things right off the vine or stem as we are harvesting our bounty and I am perhaps overly protective of my daughter. Here in the Pacific Northwest slugs are a particular problem, and I must use several different techniques and approaches to try to ward off or destroy these nasty, slimy creatures throughout the year because no single approach is 100% effective. Yes it is unfortunately true, even with my diligent efforts and myriad of tactics a few slugs still get by into the raised gardens. This sure seems a lot like software testing and how some tests find some bugs and miss others, and how iterative builds seem to allow new bugs to continually creep in. But, this really isn’t a revelation.

In 1983, in his seminal book Software Testing Techniques, Boris Beizer compared the diminishing effects of insecticides on boll weevils destroying cotton fields to the decreasing effectiveness of testing methods in exposing defects in software. This became well known as the Pesticide Paradox. The Pesticide Paradox states “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”

Now, unlike insects (or pests), software doesn’t magically build up an immunity to bugs. What happens is that we design and execute tests using one approach that exposes some set of issues. But, then the number of issues being exposed by that approach starts to diminish, yet there are still some ‘residue of subtler bugs’ that haven’t been detected by that set of tests or testing approach. Also similar to how the ladybugs in my garden that eat aphids and mites have no effect on slugs, not all approaches or techniques we might use in our testing is effective in detecting or preventing all types of bugs.

For the past 10 years, I have been teaching various software testing practices and approaches for solving complex problems at Microsoft. One cool aspect of my job is that I get to experiment a lot in a reasonably controlled environment with diverse groups of people. We often design these studies to better understand the benefits and limitations of various testing approaches and methods in exposing different categories of defects to better understand how each approach can be used more effectively within the appropriate context. Of course, it should be of no surprise to anyone that the pesticide paradox holds true not just in the classroom, but also in practice.

I often explain Beizer’s paradox by stating, “there is no single approach or method used in software testing that is completely effective in exposing all bugs, and some approaches or methods are more effective in exposing different types or categories of bugs.”

At Microsoft there is no “one size fits all” solution, and the Engineering Excellence group doesn’t dictate how to test or what testing methods can or can’t be used. But, through a series of problem solving exercises in our new SDET training program each tester experiences the benefits and limitations of various approaches and techniques. Based on their experiences in the classroom and in ‘real life’ they also learn the most effective strategy for testing is not to rely too heavily on a single approach, and to use a variety of test design principles and patterns throughout the product lifecycle. But, that’s just how we roll.

Software Testing Techniques 2nd Eidition

Originally Published Monday, November 24, 2008

I just returned from a business trip to Israel. I was a long time on the road (a week at EuroStar followed by a trip to Israel to teach at our 2 R&D centers there). So, I really lucked out because I got the opportunity to go sailing this past Saturday and unwind a bit. I checked the day before and all the boats were all reserved, but since the sky was a bit gray many people cancelled their reservations. Having lived in the PNW for the past 15 years or so I have become quite accustomed to gray skies, so it didn’t bother me in the least.

Wow…sailing in the Med, and sailing a Performance Club 420. It has been a long time since I was in a 14’ racing boat, and even longer since I was in one that we launched from the beach in 1 – 2’ waves. It was wet, it was wild, and it was fast. Saturday brought back a lot of memories from my childhood, and it also reminded me of all the various things I learned about sailing as a teenager. These days I mostly sail aboard my Cooper 416. She is a heavy cruiser at 42’ long and weighing in at 30 thousand pounds; she is very stable, turns like a dead elephant, she is generally is very forgiving, and most of all…she is comfortable. Small racing boats are light (@ 250 lbs), nimble, wet, and not so forgiving when you make a mistake. In brisk winds and breaking seas you must be very vigilant, act quickly, time your tacks, and utilize every bit of skill and knowledge to keep her upright and prevent pitch-poling or swamping when running downwind and returning to the beach.

In all honesty, sailing itself isn’t really all that hard. Any chowder-head can eventually learn how to work lines and get a sailboat moving in some odd direction. But, knowing how to sail well and how to sail in a variety of circumstances takes a lot of skill and knowledge. Sailors are at the mercy of mother nature, so they must understand such things as weather patterns, and geographical influences on the wind and tidal flows. Sailors must also have in-depth knowledge of navigation and navigational techniques, and boat handling in different contexts such as light air or heavy seas. Sailors (not sunny weather day sailors) must also understand math and physics, and theoretical concepts such as Bernoulli’s principle. While some sailors may not fully understand Bernoulli’s principle, they understand the concept every time they trim the sails for maximum performance. Those who do understand it can easily explain to novices the physics of why a Marconi rigged sailboat can sail faster into the wind  compared to running the wind from dead astern or on a broad reach, or why and how adjusting the rake of the mast affects sailing angle.

As a child my father taught me that if I really wanted to be good at something it wasn’t enough to just do it. Practice is important, but if I wanted to really understand something I have to also understand how and why things work. He taught me the more I understood how something works from both a theoretical and practical perspective the better equipped I would be to apply critical thinking skills to various situations, to face new challenges, and also potentially come up with alternate ideas and approaches because I could think through the situation both logically and abstractly.

So, you can imagine that I was somewhat surprised when I came across a posting on a software testing distribution list in which someone suggested that teaching testers various techniques and methods commonly used in our profession is controversial. Considering studies indicate a significant number of testers have never received formal training in software testing, and anecdotal evidence suggests less than 10% have read more than 2 books on the specific subject it boggles my mind to try to understand how anyone who considers themselves to be a professional in this discipline could even contemplate the notion that training testers in the foundational knowledge of our profession and its ‘systems’ is controversial?

Of course, if your entire perspective of testing is simply bug finding, and you are easily amused by parlor tricks that expose inane issues, blindly accept wild hyperbole without empirical evidence or a logical explanation then perhaps actually studying about software testing practices or computer engineering, and learning about professional practices used in the discipline might be controversial.

Clarke’s third law – “Any sufficiently advanced technology is indistinguishable from magic.”

Originally Published Friday, February 08, 2008

I have been tardy in my writing…too many irons in the fire these days. This week I have been teaching new SDETs in the morning, then rushing downtown to teach evening classes on software testing at University of Washington. These days I spend a great deal of my time teaching (mostly about systematic testing approaches and test automation), consulting on testing practices, designing and developing hands-on technical training, and talking about the future of the discipline. This role is personally satisfying, and it is certainly one of the most challenging positions I have ever had. So, as a member of the Engineering Excellence group at Microsoft I am very proud to announce that we are in the top 10 of the world’s most successful learning and development organizations. When Microsoft embarked on its journey in 2004 to refocus our product teams on engineering practices including the drive to increase the technical skills and acumen of our software testers our training was not even listed in Training Magazines Top 125 awards. In 2005 we were listed in 38th place. In 2006 we moved up to 23rd place, and 19th place in 2007. Last week, Microsoft was ranked 9th among the top 125 rated training organizations around the world. Companies are ranked based on various factors including number of training hours per employee, detailed formal programs, use of various learning technologies, and quantitative and qualitative case studies demonstrating strategic business impact. I must admit, I consider myself very lucky to be part of such a great team of professionals who are incredibly passionate about their discipline and strive to have a positive impact on how we design, develop and test our products.

Originally Published Saturday, April 28, 2007

I had a professor who once told me her greatest reward in life would be to influence just one of her students to think differently. She sadly passed away a few years later, and University of Maryland now has a scholarship program in her honor. She accomplished many things during her life, and if influencing just one person was her goal, her life was more successful than she could ever have imagined. I never really understood why she set her bar for accomplishment so low (mostly because I knew she achieved so much more), but she was a humble woman and did not like to boast.

Influencing one person doesn’t seem like a stretch goal, but unless you have ever been an educator or a trainer you may not really understand how hard it is to effectively influence someone to impact their way of thinking. I am not talking about teaching new facts, or showing someone how to use a tool. I am talking about getting people to change behavior based on new perspectives and awakening the curiosity of someone’s mind to want to learn even more.

At Microsoft I train about 400 – 450 new testers around the world each year in our intensive, hands-on new tester course and test automation courses. The new tester course teaches testers (and some developers) the theory and application of several functional and structural testing techniques, test case design, and debugging skills including multiple interactive exercises designed to give the attendees practical experience. About 2 years ago I took some of the core lessons of our internal training and wrapped them up into a workshop that I present regularly at the Software Testing and Performance conference. At the internal training offerings and at the conferences I tell people my only goal is to get them to think about testing and how they approach testing from different perspectives, to think about different ways of problem solving, and how these techniques (at least the functional techniques) are tacitly employed (to varying degrees) even during exploratory testing approaches. (The effectiveness of the application of these techniques varies depending on the testers’ abilities and knowledge; but that is a tangent.)

While I hope I could reach everyone in each class or workshop I am a pragmatist. So, I set my personal objective to influence just one person to change or think about just one aspect of their testing. Well, at the STP conference in San Mateo I guess I was successful. Andrew Binstock, who writes for SD Times and InfoWorld, attended the workshop in San Mateo and posted his comments about one of the functional techniques discussed in the workshop. It seems Andrew realized how pairwise analysis could be used effectively in unit testing.

So, it seems I did reach at least one person at the conference, and hopefully his post will inspire others in his sphere of influence, and those that realize the value and change their behavior (with regards to pairwise testing) will influence others…and so it goes.