I.M. Testy

Originally Published Tuesday, August 11, 2009

I remember as a young lad working construction for my uncle one summer. The hours were long, it was hot, and I would much rather have been somewhere else. But, I was saving up for my first motorcycle, so I did whatever jobs I could find. Perhaps it was because my mind wasn’t really engaged in what I was doing, but as I was hammering nails into drywall the hammer slipped off the head of the nail and mashed my thumbnail. Man that hurt. Of course after yelling out a few expletives I screamed,  “stupid [expletive deleted] hammer!” Years later, I was working an engine and the wrench slipped and the back of my hand slammed into the engine block. Covered with grease (and now a bit of blood) I examined my hand to assess the damage in order to decide whether to keep working or tend to my wounds, and the first thing I thought of as I looked at my somewhat mangled hand was “stupid wrench.” Do you see the pattern here? Isn’t it a bit funny that often when we use a tool incorrectly, misuse a tool to do something the tool was not designed to do, or do not really know how to use the tool to begin with and a problem ensues our initial reaction is to blame the tool. Of course, it couldn’t be our fault; it has to be the tool!

I sometimes see this deflection of responsibility repeated by testers who attempt to apply various techniques or methodologies to a software testing problem and later discover they missed or overlooked a bug. Their immediate reaction is “such and such” technique sucks because it didn’t find “this” bug. Of course it could never be our own fault! It could never be that we don’t sufficiently understand the principles of a technique or approach in order to apply it correctly. It could never be that we don’t fully understand the ‘system’ in depth that we are testing. And it could never be that we are using a particular technique or approach in the wrong context. The problem could never be the fault of the person wielding the tool…it must be the tool.

A few years ago I was rebuilding a sailing dinghy. I was ready to mount the rub strake and started drilling holes through the fiberglass hull and the teak strip. For some bizarre reason, I decided to simply hold the rub strake against the gunwale as I moved forward drilling the holes rather than use C-clamps. My grip was sufficiently tight to hold the rub strake tight against the hull, and I had used an electric drill for years and knew well how to use the tool. But, after drilling several holes I suddenly experienced an excruciating pain in my left index finger. Yep…the drill bit went right into the proximal interphalangeal joint on my left index finger. After screaming a few profanities (which by the way helps us deal with pain) I flung the drill across the garage only to put a hole in the drywall and hear it crash to the floor in pieces. With that I hurled a few more obscenities, wrapped my finger with ice, and headed off to the doctor and thought to myself…”man, that was stupid of me not to use clamps.” To add insult to injury, the doctor asked, “Why didn’t you use a clamp?” I replied, “Well, I was in a bit of a hurry.” He shook his head, smiled, and asked the redundant question, “In retrospect that wasn’t too smart was it?”

When we misuse tools, apply them in the wrong context, or if we really don’t understand how to use tools appropriately bad things can happen. (And sometimes the scars are permanent!)

While I don’t think that misapplication of a testing technique or approach would require a trip to the hospital, it might cost us a missed bug or added redundant tests. This also doesn’t mean that all techniques or approaches to testing are 100% effective in finding all categories of anomalies. But, we have to remember for the most successful application of any tool, technique, or approach used in testing depends heavily on the person wielding the tool in the appropriate situation and we must learn to use them smartly!

Originally Published Wednesday, July 22, 2009

This week, I will keep this post quite short and redirect you to my answers to an interview by the great folks at What Is Testing. The questions covered various topics from the book How We Test Software At Microsoft, to my current role at Microsoft, to my perspectives on things such as open source, certifications, and future directions for our profession.

Speaking about our book How We Test Software At Microsoft, I am generally not one to pat myself on the back, but I am really glad to announce our book was translated into Chinese and Korean. In my past life as a Test Manager I worked closely with our teams in Korea, and I was actively engaged in establishing some of Microsoft’s initial partnerships with Chinese vendors. So, I would personally like to thank the many translators and editors, and the publishers who are making our book available to the testers in those countries. I hope our book provides testers in Korea and China some new ideas or alternative perspectives on software testing. Both translated versions should be available soon!

Originally Published Friday, April 17, 2009

This week I came across a discussion [regarding test design] in which a tester wrote, "…the main goal is having something that is ‘good enough’." Every time I hear a tester utter the phrase "good enough" my head wants to explode!

Wrapping duct tape around a splint on the broken handle on my hoe is good enough to finish the job until I can go buy a new handle. While I may sometimes temporarily improvise a "good enough" solution; I am never truly satisfied with good enough, and I personally aspire to be better than good enough. My father always told me if something was worth doing, I should do it right! He also raised me to always put forth my best effort, and constantly strive to improve myself.

I seriously can’t think of any professional (in any discipline) who seriously considers good enough to ever really be good enough? The "good enough" argument is the ultimate cop out! In my opinion "good enough" epitomizes an unprofessional, apathetic attitude sanctioning mediocrity.

From a job performance perspective I suspect that if we told our employers that we were going to simply design and execute tests that are "good enough" we probably wouldn’t be in a job very long. I certainly would not want people on my team who are satisfied with "good enough;" I want people who want to do their best, and to strive for better!

I spent some time in the US Air Force and we often used the phrase "it’s good enough for government work" to describe slop-shoddy work. So, it amazes me that some people seem to be satisfied by consciously condoning ignominious practices. But, I guess some people are taught to expend just enough effort to be good enough!

In my opinion, good enough may be "good enough for government work" or for individuals who don’t have a vested interested in helping organizations improve, or who don’t really care about improving themselves; but, there is no room for the slovenly "good enough" mentality among professional testers.

Originally Published Friday, March 27, 2009

I often find discussions about quality to be hypothetical, and in fact unless you define your specific context the word itself is nebulous, vague, or simply meaningless philosophical psycho-babble. For a while now, I previously posted my opposition to the simplistic notion that quality is value to some person. Sure, most thesaurus’ equate the two words as synonyms, and the context-driven posse constantly regurgitate Weinberg’s  quote about "quality is value to some person." Yes, that is one perspective of quality, but only one.

In the past I have taught that quality is not value in the context as one of the goals of software testing. My definition of value is the purpose or usefulness of a product in satisfying a customer’s needs or wants. My definition of quality was based on tangible aspects of the attributes or capabilities of a product from an engineering perspective. Our definition of software testing as any task designed to evaluate or assess the attributes and capabilities of a software project in relation to implicit or explicit guidelines in order to provide information to the management team (the people who make the business decisions). Those evaluations result in measurements we call quality measurements or criteria (the "essential or distinctive characteristics, properties, or attributes" that are critical for the success of our project) and that is part of the information we present to the decision makers to help them make more informed business decisions. Remember, Weinberg also stated "Thinking about measurability from the beginning is an essential part of creating a well-formed effort."

The other day I met with members of one of our research teams and they were talking about quality in terms of tenet and non-tenet quality; and I clarified that basically they were essentially talking about the customer perceptions of quality versus the engineering aspects of quality. Surely, from a holistic point the observation and reliable measurements of both perspectives of quality are important for the success of any organization. Customers usually buy/download software because it helps them satisfy some need or want. The decision of which software product to buy made be based on personal bias, or perhaps the herding instinct, or it may be more rational based on the comparison of features (measureable attributes and capabilities). Once the customer begins to use the software they form their own opinions based on expectations and/or previous experiences that provides the company with information regarding customer satisfaction.

So, the next time someone starts talking about quality stop them and ask them if they are talking about the engineering aspects of quality, or the customer perceptions of quality. They are closely related, but different perspectives of the same topic.

Originally Published Friday, February 06, 2009

I am an avid fly-fisherman, and I am spending a few of these last winter evenings tying flies in preparation for the new year. The lakes are still too cold so the trout are deep and lethargic, and many of the rivers are closed and too damn cold and swollen anyway. So, now is a good time restock my fly boxes, reflect on past years, and dream of the up-coming season. While fly fishing is an enjoyable escape from the day to day torrent of technology, when I can’t stand in a river and wave a stick I can still relax with a good book that conjures memories of years past or engulfs my mind in an adventurous narrative as if I were there. I don’t really enjoy reading most fiction books, but I do enjoy reading the memoirs of people such as John Gierach. (Perhaps it is the sailor in me that loves a good yarn, or the fact that I can relate personally to the stories.) Anyway, this weekend I acquired a book entitled Fishless Days, Angling Nights by Sparse Gray Hackle that introduced a legend in American fly-fishing by the name of Theodore Gordon. I had never heard of Mr. Gordon before this (perhaps that is because I tend to fish a lot more soft hackle flies rather than dry flies), but I found a quote by him quite interesting. He said, “The great charm of fly fishing is that we are always learning.”

This morning I thought how apropos this statement is to software testing. But, there of course, as it pertains to the practice of software testing I would rephrase Mr. Gordon’s statement to state, “The great demand of software testing is that we must always learn.”

There are different types of people who fly fish. There are those who buy a fly rod and a pocket full of store bought flies and head to the water just to catch fish. This group of people don’t seem to care much about the techniques of casting, learning how to read water, or understand patterns of fish or aquatic entomology; they are simply there to try to catch fish using whatever slop-shod mooching approach seems to work. Yes, they still catch fish…usually the small, dumb farm raised trout stocked into regional lakes by the state’s department of wildlife who will bite at anything. On the other end of the spectrum are true purists who fish with cane poles, tie their own flies to match the hatch (sometimes right beside the river), and study trout, regional entomological lifecycles, and the geological formations of a river bed to better pin-point where the big, smart trout are hiding. Then there are the group in between these extremes with varying degrees of skill and knowledge. Depending on how much time a person devotes to both practice and learn (and their capacity to learn) about the sport will often make a huge difference in both their enjoyment and their effectiveness in catching large, persnickety trout.

From my observations of the testing community over the past few decades, I can see a similar pattern regarding the spectrum of skills and knowledge of people who participate in the practice. In the past, it was not uncommon for some companies to hire ‘clever’ people who were simply good at finding bugs into testing roles. Some companies hired developers who would (often times begrudgingly) take the job as a stepping stone into a developer position. Unfortunately, some people at both extremes of this spectrum often stagnated because they did not learn more about software testing or the technological advances that were happening around them. At one extreme, I suspect that some people thought as long as they were finding behavioral type issues they were providing a benefit to the company because they were ‘good at representing the customer.’ At the other end of the spectrum the developer’s in testing roles who failed to realize the challenges in software testing. In both extremes complacency and stagnation usually occurs. Of course, there are many other testers between these extremes; some who will go on to become professional testers and have significant impact, and others who will simply belly-ache and whine about how unfair it is or claim how wrong any change is and why change won’t work.

As professional testers we must constantly strive to improve our knowledge of testing, technology, and the systems we are working on. We must also increase our skills and abilities as the demands of the role expand beyond the traditional comfort zone of behavioral testing and ‘playing customer advocate’ by executing ‘tests’ to find bugs at the end of a cycle. The challenges of testing complex systems built around advancing technologies significantly raises the aptitude bar for testers. Emerging practices such as TDD and agile lifecycles designed to drive engineering quality upstream and form closer customer connections is also impacting the role of testing and how testing adds value in the lifecycle (and I don’t think the role of testing in an agile lifecycle is trying to wedge testing between the end of a sprint cycle and the release to customers in order to provide a pseudo-proxy customer buffer…that’s a bottleneck.) Reinstituting best practices such as design and code reviews and inspections (when warranted), or developing new approaches or tools to help increase testing effectiveness and reduce costs also require greater skills and knowledge among testers.

The formidable challenges of testing software that lie ahead will require highly intelligent critical thinkers who also have an in-depth understanding of the systems they are working on, and who possess the technical aptitude to provide valued input in throughout the product cycle. Indeed we work in a very dynamic industry filled with diverse challenges that demand continued learning and greater proficiency of the skills used in our profession.

Originally Published Friday, January 30, 2009

Last week I discussed the fallacy of the minefield analogy misrepresented by some people to suggest regression testing as uninteresting or unlikely to reveal new or important information.  Their premise is that executing the same test is similar to walking in someone’s footsteps through a minefield. While this argument seems logical on the surface, this interpretation of the minefield analogy trivializes the importance of regression testing. However, there may be specific instances where regression testing (especially the execution of redundant, poorly designed, basic scripted test cases that simply reuse hard-coded data or mindlessly follow a prescriptive steps to retest unchanged or unaffected areas of code) may not provide great value to the organization. Some of those situations include:

• retesting scripted, or simple procedural programs
• retesting programs that are relatively static (unchanging),
• retesting programs with no internal or external dependencies
• rapid testing approaches where the test strategy is to simply sample the program behavior in order to provide a quick assessment and find some bugs

So, everything being equal, I would tend to agree that walking through someone else’s footsteps in a minefield may be somewhat redundant. But, there is a huge difference between redundant testing and effective regression testing. Redundant testing has zero probability of revealing new or valuable information. But, well-designed tests are not all equal, and in iterative and agile software development lifecycles some portion of effective regression test suites have some reasonable probability in exposing new information. For example, regression testing is typically useful where:

• the underlying code base is changing (new features, refactoring, or bug fixes)
• poor or incorrectly used revision control processes
• a change in one module affects other modules in software developed using object oriented and procedural programming paradigms
• the complex system has other internal and external dependencies
• the software is regulated by some governing authority or law (Sarbanes-Oxley, FDA, FAA, etc)
• the system is highly critical
• an established baseline is required or important
• legacy code bases where new functionality can unintentionally destabilize older code
• code bases contain lots of ‘spaghetti’ code that is simply patched together
• the results of a well-designed regression test suite helps instill a sense of confidence in the decision makers

In these contexts, an effective regression testing strategy can not only help identify important and potentially destabilizing changes in expected functionality quicker, but can also increase overall confidence in the product; especially in key areas.

Overcoming some common misconceptions of regression testing

One common misconception of regression testing strategies is that all tests are equal and all documented (manual or automated) tests go into the regression test suite. But, the simple fact is that not all tests are equal, and not all tests need to be re-executed repeatedly (every build) throughout the development lifecycle. For example, a test to check for proper tab order on a dialog or property sheet doesn’t need to be re-executed on each new build unless the UI elements have changed on that dialog or property sheet. So, in most situations this type of test is probably a poor candidate for inclusion in a regression test suite. (As a side note, if the UI elements are changing every build, I wouldn’t even test tab order until there was at least some semblance of UI stabilization.) When redundant test cases or tests that don’t provide great value to the overall testing effort are slovenly added to the regression test suite to simply increase code coverage or to artificially inflate the number of tests for ‘feel-good’ generally only results in an overloaded test suite that rapidly grows out of control and becomes a management nightmare.

Another common misconception of regression testing strategies is the suggestion that unit tests are sufficient for retesting code churn. Of course, this is simply a fool-hearty assumption and idealistic. Unit tests are a type of smoke tests performed by developers. The primary purpose of unit testing is to verify a method or function does what it is supposed to do, and may include ancillary negative tests as well. However, unit tests generally do not include comprehensive data coverage, data permutations, combinatorial analysis of parameters, etc. (which is why API testing is important in a well-rounded test strategy). Also, unit tests are usually executed in a “clean-room” environment using stubs or mock-objects, although in some cases unit tests are reran on private builds before the code is checked into the main build tree as a form of low level regression tests. While there have been significant advances in both static and dynamic analysis tools to increase the robustness of unit tests, if unit testing were the answer for effectively evaluating code churn I suspect a lot of testers would be out of a job and testing would simply be a rudimentary process of behavioral-type acceptance testing performed by non-technical ‘end-user-like’ individuals poking about the UI looking for errant anomalies and subjectively evaluating their perception of ‘usefulness’ of a product.

Another common misconception of regression testing strategies is that regression tests are less valuable because of their ineffectiveness in identifying or exposing ‘new’ bugs. This is a myopic view of testing that focuses on the current version of the product, and on ‘bug-finding’ as the main purpose of testing. Of course, value is very subjective, and industry reports suggest that less than 15% of the bugs reported during a product development lifecycle are detected via regression testing. But, if even half that number are critical issues that would cause the company to pay upwards of $100K to release a patch (or even worse, the issue leads to a major calamity) then regression testing (especially highly automated regression testing) is most probably worth the time and effort. Of course, the cost of designing effective regression test cases is quite high and executing those tests requires valuable resources. So, we should not just consider the legal or financial liability issues to the company. We should also consider the product shelf-life as a factor. An effective suite of well-designed regression test cases will not only help provide a baseline assessment of the current version, but a significant portion of the test suite will/should be reused during maintenance or sustained engineering of that product for upwards of 10 years, and some subset of those tests in that suite will also be reused on the next release or version of the product. There are additional benefits to well-designed regression test cases (or any test case for that matter) in that it preserves knowledge and helps eliminate or at least reduce tribal knowledge and hero worship mentalities.

Some effective regression testing strategies

The regression test suite is essentially a set of test cases that provide a baseline measure of expected, or important functionality, and to verify previously fixed bugs do not reoccur. Based on those assertions, test cases included in the regression test suite must be carefully selected to prevent overloading or rerunning unimportant tests during a regression test pass. I generally recommend 4 types of test cases for inclusion in a regression test suite:

1. Test cases designed to verify critical functional attributes and capabilities of the software program
2. Test cases designed to validate baseline functionality/behavior specified in project requirements or user acceptance criteria
3. Test cases designed to verify functional anomalies that are found and fixed during the software development lifecycle
4. Test cases designed to evaluate collateral or dependent areas associated with a code fix

Even with these targeted tests the regression test suite can grow quite large. Within the regression test suite categorize the test cases by the feature or component that test directly targets, and sub-categorized to identify dependent modules or features. Associating test cases in the regression test suite with the primary features or components they are designed to evaluate allow the tester to prioritize the regression test pass to focus initially on the modules in which code churn occurred,  followed by dependent or interrelated modules, and finally the other remaining tests in the suite. Also, prioritize test cases within each category based on criticality. Prioritization of tests is another mechanism we can use to help us better identify an appropriate set of tests based on the constraints of available time and resources.

Finally, automate, automate, automate! In order to gain the full benefit of an effective regression test suite (similar to the build verification/acceptance (BVT/BVA) test suite ) automate as many regression test cases as is reasonable possible. But, I am referring to well-designed automated test cases; not just a bunch of simple, elementary UI script monkeys. While some regression test cases will be very targeted prescriptive tests designed to verify specific functionality, most well-designed test cases include variability in the execution while still evaluating the hypothesis, or purpose of the test case. Also, some test cases in the regression test suite will be executed through the UI layer, but in applications where the functional code is separated from the UI layer some regression test cases are more effectively executed below the UI or through an abstraction layer.

Similar to other testing approaches, regression testing is an organizational investment, and its value to the team must be considered as an investment and not done simply because it seems like the right thing to do. In some cases it may not make much sense to invest heavily in regression testing, but in cases where a missed regression in functionality results in fines, legal costs, or potential loss of hundreds of thousands of dollars in post-production costs I suspect regression testing is a critical component of the overall testing strategy to validate a baseline and instill confidence.

Originally Published Monday, January 19, 2009

In my studies at university I studied anthropology. Several courses I took surveyed folklore and its relevance in modern society. Many people mistakenly believe that most folklore (folktales, legends, myths, ballads, etc.) are purely fictional and simply fanciful tales. However, folklore is usually based on some grain of truth, or is used to instill societal or religious mores and values. For example, social scientists have found that many ancient civilizations have folklore regarding a massive “flood” in the distant past which wiped out huge populations of people. Did this actually occur? Well, we don’t know for certain, but geological evidence does suggests is that at one time coastal waters did rise significantly. Was this caused by cyclical change in the earth’s temperatures or by a series of earthquakes causing tsunami’s to ravage coastal villages? We don’t know; but the folklore may indicate that at some point many societies suffered a devastating travesty caused by rising waters. Was the story embellished over time…certainly. Another example is the “Cinderella” story. There are over 450 versions of the “Cinderella” story around the world.  The story is about over-coming adversity and oppression, and avoiding self-pity and selfishness. Basically, it is much more than a Disney animation, in traditional folklore it has been passed down through the generations to reinforce societal values.

The first time I read about a minefield analogy was in the context of sampling. Later, Brian Marick used a similar analogy to suggest repeating tests (regression testing) is not likely to reveal new bugs. Marick’s analogy is perpetuated by Bach, Kaner, and others who tend to diminish the value of regression testing (especially automated testing) because we are simply traversing a minefield by following a previously cleared path.

The Marick minefield analogy is simply an alternate perspective of Beizer’s pesticide paradox which states “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. Basically, no single approach to software testing is effective in identifying all categories of defects, and we must use many approaches in software testing and vary our tests. In that context I absolutely agree with the analogy.

However, a basic problem of Marick’s minefield analogy as it is often misrepresented is that it seems to treat the software under test as a static, unchanging field of easily exposed mines.

If you were hired as a consultant to come in a perform a rapid evaluation of a software product using a sampling approach such as exploratory testing, then Marick’s minefield analogy is a wonderful strategy. In that context re-running a test provides no new value and has little probability of exposing new information.  However, for the rest of us who work in iterative software development lifecycles (including agile lifecycles) building complex systems with interdependent components the minefield analogy may not be as useful.

For example, in complex systems with interdependent modules we know that a change in one module can adversely affect other modules that have some dependence on that module. So, a change in one module can impact the functional behavior of other modules. In layman’s terms, activating a mine while traversing one path through the minefield may reactivate an already cleared mine in another part of the minefield, or even plant a new mine in a previously traversed path.

In iterative development lifecycles, the minefield is in constant flux (at least until the code complete stage, but even then the code is changing as issues are being addressed.) In iterative lifecycles features are being added, changed, and possibly removed during the process. Depending on the length of your product lifecycle the changes can be massive. The PDC release of Windows 95 ‘looked’ very different as compared to the final release.  The build verification/acceptance test suite for Windows 95 was a relatively static baseline regression test suite that continued to find ‘regression’ problems up to the final weeks of the project due to code churn.

Also, not all mines are equal! Some mines are quite easy to detect while others are very hard to find which is why systematic probing is still used by professional’s to clear latent minefields. Similarly, an exploratory approach to testing software will easily reveal some bugs very quickly, but without ‘systematic probing’ we could just as easily overlook other types of issues.

There are also different types of mines which may be activated differently, so traversing a minefield with a size 10 boot may not activate the mine, but someone with a size 12 boot, or who weighs more than the previous person may in fact activate the mine. Likewise, traversing the same path through software using different data or applying a more systematic analysis of a path may reveal interesting information or expose anomalies that were not previously discovered. For example, throwing simple ASCII characters at a text input control is not likely to expose any bugs (or restated it is likely to show us a clear path through the minefield). However, when we take that same exact path using Unicode characters, or Unicode surrogate pair characters we are very likely to expose problems not revealed previously.

In part 2 I will discuss regression testing and specific situations where regression testing is very valuable.

Originally Published Wednesday, December 24, 2008

I have a library of books and white papers on software testing, engineering processes and management, and software development that I have read and reference quite often. For new testers I generally recommend A Practitioner’s Guide to Software Test Design by Lee Copeland, and How to Break Software: A Practical Guide to Testing by James Whittaker. There are 5 books I highly recommend (not including How We Test Software at Microsoft which I co-authored and also highly recommend).

In my current role as a teacher, trainer, and mentor of new testers the 2 books that are constantly on my desktop are Testing Object-Oriented Systems: Models, Patterns, and Tools, by Robert V. Binder, and Software Testing Techniques, 2nd edition by Boris Beizer. Not that I don’t frequently reference other books, but to me these are the quintessential books on the foundational knowledge of software testing techniques and methodologies for intermediate to advanced testers with a strong technical background.

But, the booklet that I would keep in my shirt pocket if I tested products on a day-to-day basis would be Josh Poley’s Black Book. Josh’s Black Book is the ultimate desktop reference for software testers (and developers). While this book is primarily intended to aid those who work on projects developed in C/C++, it has loads of information that is valuable to any tester working on just about any technology. From decimal and named entities of ISO characters to error codes for DOS, VB, JScript, HTTP, and of course Windows Errors this book is jammed packed with great information and quick reminders for both developers and testers.

Originally Published Monday, November 24, 2008

I just returned from a business trip to Israel. I was a long time on the road (a week at EuroStar followed by a trip to Israel to teach at our 2 R&D centers there). So, I really lucked out because I got the opportunity to go sailing this past Saturday and unwind a bit. I checked the day before and all the boats were all reserved, but since the sky was a bit gray many people cancelled their reservations. Having lived in the PNW for the past 15 years or so I have become quite accustomed to gray skies, so it didn’t bother me in the least.

Wow…sailing in the Med, and sailing a Performance Club 420. It has been a long time since I was in a 14’ racing boat, and even longer since I was in one that we launched from the beach in 1 – 2’ waves. It was wet, it was wild, and it was fast. Saturday brought back a lot of memories from my childhood, and it also reminded me of all the various things I learned about sailing as a teenager. These days I mostly sail aboard my Cooper 416. She is a heavy cruiser at 42’ long and weighing in at 30 thousand pounds; she is very stable, turns like a dead elephant, she is generally is very forgiving, and most of all…she is comfortable. Small racing boats are light (@ 250 lbs), nimble, wet, and not so forgiving when you make a mistake. In brisk winds and breaking seas you must be very vigilant, act quickly, time your tacks, and utilize every bit of skill and knowledge to keep her upright and prevent pitch-poling or swamping when running downwind and returning to the beach.

In all honesty, sailing itself isn’t really all that hard. Any chowder-head can eventually learn how to work lines and get a sailboat moving in some odd direction. But, knowing how to sail well and how to sail in a variety of circumstances takes a lot of skill and knowledge. Sailors are at the mercy of mother nature, so they must understand such things as weather patterns, and geographical influences on the wind and tidal flows. Sailors must also have in-depth knowledge of navigation and navigational techniques, and boat handling in different contexts such as light air or heavy seas. Sailors (not sunny weather day sailors) must also understand math and physics, and theoretical concepts such as Bernoulli’s principle. While some sailors may not fully understand Bernoulli’s principle, they understand the concept every time they trim the sails for maximum performance. Those who do understand it can easily explain to novices the physics of why a Marconi rigged sailboat can sail faster into the wind  compared to running the wind from dead astern or on a broad reach, or why and how adjusting the rake of the mast affects sailing angle.

As a child my father taught me that if I really wanted to be good at something it wasn’t enough to just do it. Practice is important, but if I wanted to really understand something I have to also understand how and why things work. He taught me the more I understood how something works from both a theoretical and practical perspective the better equipped I would be to apply critical thinking skills to various situations, to face new challenges, and also potentially come up with alternate ideas and approaches because I could think through the situation both logically and abstractly.

So, you can imagine that I was somewhat surprised when I came across a posting on a software testing distribution list in which someone suggested that teaching testers various techniques and methods commonly used in our profession is controversial. Considering studies indicate a significant number of testers have never received formal training in software testing, and anecdotal evidence suggests less than 10% have read more than 2 books on the specific subject it boggles my mind to try to understand how anyone who considers themselves to be a professional in this discipline could even contemplate the notion that training testers in the foundational knowledge of our profession and its ‘systems’ is controversial?

Of course, if your entire perspective of testing is simply bug finding, and you are easily amused by parlor tricks that expose inane issues, blindly accept wild hyperbole without empirical evidence or a logical explanation then perhaps actually studying about software testing practices or computer engineering, and learning about professional practices used in the discipline might be controversial.

Clarke’s third law – “Any sufficiently advanced technology is indistinguishable from magic.”

Originally Published Wednesday, October 08, 2008

As a young lad growing up on the shores of the Chesapeake Bay I would often spend part of my summer vacation from grade school helping my grandfather work the crab pots on the north shore. Now, don’t think “Dangerous Catch,” crabbing in the Chesapeake is much different than crabbing in Alaskan waters, although we did have our share of squalls and nor’easters that will humble any man. Crabbing is hard work, especially for a young boy. On one particular day working the pots with my grandfather I recall something he said which has stuck with me my entire life. The water was rough that day and we were being tossed around a bit. It was raining…not like the wimpy Seattle rain, I mean hard East coast rain that comes down in sheets with drops so large they hurt when they hit you. Even in the driving rain the smell of dead fish, rotten chicken pieces and turkey necks that we used for bait permeated everything, the slime of fish guts coated the decks making footing somewhat precarious, and the brine of the Chesapeake seeped into every little cut and nick on my hands. I wasn’t having a lot of fun that day, and although my grandfather knew it he didn’t say anything to me until I complained aloud. Now my grandfather was a tall, barrel-chested man with a stern face, and he never put up with any senseless, petty whining. But, instead of scolding me he simply said, “Son, we are not rich and there is no inheritance coming your way, so you better find a job you like doing.” At first, I was a bit puzzled. What did that have to do with how I was feeling at the moment? But, I soon realized what he meant, and my grandfather’s sage advice has stuck with me ever since and guided me in my career pursuits.

My father, who was also every bit as practical and pragmatic as my grandfather taught me the value of working hard, and the virtues of responsibility. He taught me to take pride in the things I did well, and take responsibility for things that didn’t work out so well. My father also taught me never to quit, to always look at alternative options, and that the path of least resistance or the easy path through life is not always the best path to take. He also said if something was worth doing, do the very best you can do, and never sit back and rest on my laurels. My father was not the type of man to do something half-way, or ‘good-enough.’ One summer we built a new barn on our property. This wasn’t the typical barn, instead my father got some telephone poles and even got his friend at the telephone company to bring out a truck with an auger to drill the holes 10 feet deep! Next we used oak planks between the stalls. Now hammering nails through oak is not easy business and the entire project took about 2 weeks. But, my father was adamant about not having to rebuild the walls of the stall if a horse tried to kick it down.

Practicality in practice

Now perhaps it was my upbringing, or perhaps it is my Myers-Briggs ISTP personality traits, but it is hard for me to understand how some people embark on a career, or pursue a job opportunity without knowing (or perhaps worse..completely ignoring the fact) that they will be challenged to continually improve their knowledge and skills everyday. This is especially true of many of our chosen profession of software testing. We work in a highly dynamic industry and we must continue to learn and develop new skills to meet the growing diversity and increasingly complex challenges that will potential advance the discipline as well as provide benefit to our employers.

Sure, we can whine and complain about the changes. We can point fingers, and otherwise express negative emotions and utter baseless, counter-productive propaganda slogans such as, “automation can’t do such and such,” or “I know non-coding testers who find more bugs than (fill in the blank here)” or the virtually inverse argument “tester’s  who write code are biased and don’t know how to think like an end-user.” I learned a long time ago that although misery loves company, it is usually not those who whine the most or put forth pointless and petty gripes, or who only see the world through bi-polar, black and white lenses who drive meaningful change. Sure, it is easy to empathize with some people in some situations; but empathy without practical solutions to resolve the situation effectively is simply a pathetic play of the irresponsible victim card.

For example, we unfortunately often get mired down in a this vs. that, exploratory vs. scripted, or STE vs SDET debate that is generally too myopic (on my project… blah, blah, blah) and often counter-productive (meaningless comparisons, ridiculous measures such as raw bug count, and baseless assertions that lack empirical evidence to substantiate the claim, etc). There seems to be an opinion that if someone writes automation, or comes with a coding background then they are not good ‘exploratory’ or behavioral, or (and I hate using the phrase) “black-box” testers.  (All testers are “black-box testers!). Yet, some of the most talented testers I know in the industry began with very little in-depth knowledge of the ‘system’ or an understanding of programming concepts and applied themselves to  grow their knowledge and skills about the overall system and technologies to advance their careers, their teams, and the industry by taking on greater challenges. I also know highly talented testers with great industry experience and very strong coding backgrounds who are masters at explorative, behavioral,  or “black-box” testing.

So, for those of you who have been living in a cave, or have your head permanently implanted in the posterior end of your alimentary canal it is obvious the industry and the profession of software testing is changing and there is an increasing demand for greater system knowledge, experience, and technical skills for many testing positions in the industry. There are many reasons for these changes, and the simple fact is that the industry’s needs and strategic direction primarily dictate what skills and knowledge are required to remain competitive in the industry and advance the business.

Are ya’ gonna lay there and bleed, or ya’ gonna’ cowboy up!?

Now, many people mistakenly assume that I only promote technical knowledge or advancement though increased technical skills. (Of course, many of these people are so narrow minded and biased they only equate ‘technical knowledge and skill with ‘coding’ skills and programming knowledge.) I do talk about these subjects a lot because…well, they are directly related to our professional growth, and because I studied and interviewed thousands of testers to perform a skills gap analysis (delta between the disciplinary needs of the business in relation to the actual skills and knowledge of the existing workforce at the time). The skills gap analysis indicated that many people in the testing industry at the time were well-educated, intelligent people who were very capable of thinking critically about problems within the appropriate context. But, it also illustrated that a significant population in the testing profession at that time lacked in-depth knowledge of the system they were working and the technical skills to expand their own productivity or become more influential in driving product quality. So, as a manager and mentor it was (and is) my responsibility to help people understand changes based on the maturation of the industry and the profession, and to help them grow professionally, and that is the role I took on with great passion. I decided to focus on growing people’s technical skills and knowledge, and their understanding of established, and contextually effective software testing processes.

One of my first public talks revolved around the idea that good-enough was simply not good enough any longer based on changing customer views and increasing maintenance costs. That was in 2003! In 2005, I presented empirical evidence suggesting that exploratory testing by ‘presumable power-users’ and ‘domain experts’ is not as effective as some claims, and I advised testers to begin thinking about the future and the increasing demand for testers with more in-depth system knowledge and technical skills to not only remain effective and productive in the industry, but to help drive it forward. Some people jeered me, and one respondent from the StarWest 2005 conference wrote “Bj is a fool, and his talk shows why Microsoft products suck and why Microsoft will fail” on the feedback forms. A few others said they agreed with the message, but they really didn’t like hearing it out loud. One person told me “Yes, we know we need to improve, but your talk was like hitting us in the face with a frying pan full of hot grease.”

I mention this not to “toot my own horn;” that is simply not my style, for I am yet a simple man who is still learning to improve myself. I only mention this because here we are 5 years later still arguing over petty things that we sometimes do not have the knowledge, skill, experience, business insight, scope of influence, or credibility to change.  (And in case you don’t know, whining. griping, and emotional conjecture is simply not viewed as being very credible in trying to influence change in a room full of decision makers or business leaders.) Right now, the industry, and the testers in the industry need leaders to help them adjust to changes, provide strategic vision, and guide career growth that enables each person to be successful in their professional development and compete with their peers. Responsible leaders are those who face challenges head on, who put forth logical and rational arguments based on analysis and empirical evidence, but who continue to search for ways to solve not only the immediate problems, but also have the ability to envision potential side-effects and deal with those as well before they become problems.

Our profession is multi-dimensional, and as testers we must continually strive to increase our knowledge and our skills in order to take on new and exciting challenges in this dynamic and agile world of software testing. The path is not always easy, but it does provide many rewards to those who work in a job they truly enjoy.