I.M. Testy

Originally Published Thursday, September 18, 2008

There are rare occasions when an exceptionally bright and innovative person comes along and actually builds a better ‘mousetrap.’  My friend Ken Smith is one such person. Ken is the inventor of SmartPlug. Anyone with a boat that connects to shore power has experienced the aggravation of lining up the 3 keyed prongs in the dark, over-heated and burnt out receptacles, and cross threading the sealing ring on those circular Marinco power inlets. Finally, Ken came up with a better design that offers a safer solution for shore power. But, Ken has owned boats for much of his life, and he has been in the marine industry for 38 years. His invention was based on years of personal and industry experience coupled with the knowledge and experience of several other boat owners and electrical experts.

So, I am often amused (not in a funny way) by people who are relatively new to the industry who decide to build yet another automation testing framework or test tool. Unfortunately, many of these ‘new’ and ‘innovative solutions are simply rewrites of existing frameworks or tools. Sometimes they may introduce a new and useful feature, but mostly it is just another ‘mousetrap.’ Often these "new" mousetraps are designed and developed to solve problems that are unique to a specific team or product. But, rather than building on existing frameworks or tools some people seem compelled to completely re-invent the wheel and then want to promote ‘their’ solution as the holy grail (which, of course, are not compatible with any existing solution).

Upon recently seeing another such proposal for yet another test automation framework I decided that I would also join the re-invention foray and provide a "new, and innovative" solution myself. Now, I need to ballyhoo my revolutionary project in order to gain greater adoption and win political points with the not-so-clued-in managers who actually think completely redesigning a mousetrap over and over again by inexperienced people working in isolation is a good idea as a way to motivate people who are bored.

So, without further ado, here is my announcement for the new, exciting, and revolutionary Wheeleze!

“I am wasting precious time and mindless resources working on yet another needless re-invention for which I don’t yet have the breadth of experience or span of influence to succeed, but it is really cool and fun and I don’t have to work on other stuff that I think is boring. I call my senseless re-invention “Wheeleze.” Let me tell you about my new innovative solution that solves the problem of transporting various cargo over land.  Wheels are already in common use and provide effective solutions to the transportation problem, but there are several different types of wheels in existance and everyone knows there should only be one wheel because having several different types of wheels is expensive and it is mind-numbing to keep track of which wheel should be used in which specific context. In contrast, Wheeleze will replace all of these existing wheels with one easily adaptable and extensible solution. And, although there are some other wheel type solutions being developed in isolated pockets around the world that are also claiming to optimize rolling, Wheeleze is really new and different. (Really, it is, and if I get enough people to say it enough everyone else will believe it). As the name suggests, Wheeleze is intended to make rolling easier by providing a vehicle agnostic, intuitive, simple to use, modular solution to roll public and private conveyances down paved, cement, gravel, dirt, and even sand roadways. Best of all, Wheeleze can be recycled as fenders on tug boats, can be buried in playgrounds for children to climb on, or have one side turned inside out to form red-neck flower planters. This innovate and unique solution optimizes reuse and minimizes landfill waste. Wheeleze is destined to revolutionize the wheel industry.  Wheeleze is fully globalized and I even have a localized version called “Wheelese” for non-US English markets (such as Canada, Ireland, England, and Australia). Wheeleze is an attempt to eliminate duplication of effort because nobody would ever dream of recreating another wheel after using Wheeleze. My new, unique, and innovate Wheeleze is designed for unicycles, bicycles, oxen-carts, wagons, cars, sports-utility vehicles, light trucks, tractor trailers, combines, and even wheelbarrows. I am also planning to extend the Wheeleze to support motorboats, canoes, and surfboards. (NOTE: Wheeleze on a surfboard appears to be similar to a skateboard, but they really are different...you can’t surf on a skateboard!) Unfortunately there is no plan to support Wheeleze on sailboats (the keels present too hard of a problem that I don’t know how to solve, and of course I don’t want to ask anyone else for help because this solution is mine) because sailboats are tool slow and "old tech" and most people will probably not want wheels on a sailboat anyway. Please visit http://www.anothercompletelystupidwasteoftime.com for more information.”

Published Thursday, September 18, 2008

Originally Published Tuesday, September 02, 2008

It has been some time since I posted. Quite frankly I was burnt out, and I finally realized that I needed a vacation. So, I decided to unplug and take the month of August off to catch up on chores around the house, tend to my gardens, and finally wrapped it up with a week of single-handed sailing down to Olympia and back. I love gardening because it gets my mind off computers and I love to work with my hands. But, the last week of my vacation I decided to sail alone and completely unplug from all electronics and get back to the basics of sailing and navigation.

This was my first voyage to Olympia (I usually head north from Seattle) so I was heading into unfamiliar waters, and decided to do so without the conveniences of GPS, the depth finder, autopilot, wind instruments, and other modern conveniences to refresh my basic sailing and navigation skills. I even kept my cell phone and VHF turned off, and used oil lamps inside the cabin at night as I re-read the first edition of Robinson Crusoe. The only luxury I would afford myself was using propane for cooking and taking a hot shower once securely anchored (I even used an oil lamp as an anchor light, but only because my masthead anchor light was burnt out and I didn’t feel like climbing the stick to change the bulb).

So, I studied the charts and the tide and current tables, read through the Coast Pilot and the Waggoner Cruising Guide, and plotted my course. There are several channels where timing the currents are critical so, if my calculations were wrong I might not make it through a pass for at least 6 hours. The land masses also cause the wind to shift quite a bit, so tending to the sheets and trimming the sails meant there was little free time once the sails were raised. Despite the gray skies, pulling a shoulder muscle, ripping off part of a fingernail, and sustaining a rather deep gash near my right eye while reefing my mainsail the trip restored confidence in my abilities, provided me with peace and solitude to contemplate life, and prepare me for la rentrée (the return) back to the real world.

So, you are probably wondering what the relevance of my vacation is to software testing?

It’s all about the basics. Without a solid foundation of basic skills and knowledge of the discipline then a person’s ability to fully engage in all the tasks associated with professional software testing are limited. There are some who mistakenly assume that testing from an end-user’s perspective is sufficient; and that may be the case in some limited situations. But, in highly complex or mission critical systems, or software products that must be maintained over several years require professional testers who possess a rich understanding the system and who are also proficient using basic skills and tools of the trade. Understanding the basics enables a person to better understand the how and why, and provides a base for approaching a problem from multiple rational perspectives given a specific context.

Originally Published Sunday, June 01, 2008

I started diving in the late 70′s, and in1985 I became a PADI certified open water scuba instructor. In those days, scuba instruction involved a lot of classroom time discussing various concepts that impacted human physiology such as Boyle’s Law, Dalton’s Law, Haldane’s Principle, and also immediate responder first aid for scuba diving related maladies (most doctors are not trained in hyperbaric medicine and divers were generally more aware of potential symptoms of DCS as compared to most doctors). The training at the time emphasized in-depth knowledge of the theory of diving as well as repetitive practicing of fundamental and critical skills in controlled environments (usually a swimming pool) and ‘real-world’ environments (the open water). Certification entailed successfully completing multiple written examinations as well as demonstrated competence in the water.

I remember those early days when PADI and NAUI instructors would compete for students, and many instructors would ridicule other certification agencies while claiming the agency they supported to be superior as compared to the others. Some instructors would completely denounce other certifications as inadequate or claim that those certifications were not accepted worldwide, or make some other malicious attacks based on ignorance and their own personal motives.

Of course, we know that some people are incapable of thinking on their own, or they are easily persuaded with scare tactics, and some simply blindly follow the bloviated jabberwocky of  charismatic people. But, most intelligent people who are capable of processing cognitive rational thoughts are able to see through the prevarication and realize the hypocrisy of people who lambaste some certifications while selflessly pander their own certification.

My personal views on certifications in software testing haven’t changed. I can see both the perceived benefits of certifications by employers as well as the limitations of certifications. For example, certifications in software testing or other professional disciplines are generally based on knowledge of the discipline rather than on the ability of a person to perform a particular task.

But this is really no different than other professional organizations. For example, it is possible to get a certification (Juris Doctor) to practice law in California without ever having gone to law school or presenting a case before a judge under the tutelage of a mentor. And a person only has to drive Interstate 5 through Seattle to know that the Washington state certified engineers who completed their requirements for certification succeeded in concocting a major transportation boondoggle. Perhaps the hundreds of successful major corporations in Europe and elsewhere around the world see the value of a well-established testing certifications such as the ISTQB and ASQ for their ability to help professional testers effectively communicate using a common discipline jargon rather then constantly coming up with confusing neologisms. (Of course, all these successful organizations could be wrong…but I tend to think they are successful because the  influential decision makers in those companies make the right decisions most of the time regardless of what some external person with a limited perspective or an idealistic neophyte thinks. Perhaps that’s why they are successful!)

Of course, it would be ideal for certifications in our industry to also include practical skill-building exercises that taught testers how to test a product using various techniques and approaches, and certification required both practical demonstration and in-depth knowledge of the discipline. This doesn’t simply mean that we teach people to find bugs by banging on the GUI, or asking ‘probing’ questions such as should a button control enlarge when a user mouse’s over it. (Please…finding bugs is really not that hard, and if I wanted to know if a button control should enlarge or can enlarge I would look at the button properties for that control rather than sit there and ponder the question for 5 minutes.)

While I would like to see certifications include skill based learning along with teaching in-depth knowledge, we should also realize there are potential limitations with practical skill-building exercises. For example, in our own training we can assess if an individual learns to correctly apply a systematic procedure to design effective tests after in-depth analysis and logically decomposing a feature area or data set for a simulation used in our training. The concepts and application of some approaches and techniques such as combinatorial analysis are applicable across multiple software projects within appropriate contexts. So, we  instruct our SDETs to identify the well-defined contexts in which certain approaches or techniques are appropriate and when they are not, and we also explain how they are sometimes misused so they don’t also fall into the same traps that untrained people fall into. However, current certification schemas can not yet accurately assess how well a person will perform on a real project until that person is put into that situation. This is true of software testing just as much as it is true of scuba diving.

Fortunately in scuba diving the certification wars are mostly in the past. But, it seems the testing certification wars are heating up. Today, certifications are valued in some business sectors for various reasons. I suspect some new certifications will come along and claim some great benefit beyond the others. And perhaps they will, or perhaps they will be isolated communities of zealots who want to simply be different. I just find it rather Pecksniffish for any person to claim all certifications are bogus; of course except for the one in which he or she has some personal vested interest.

Originally Published Wednesday, March 19, 2008

Last October after presenting a keynote at the Conquest software testing conference I was invited to speak at an internal quality conference at SAP. At first it may seem a bit odd because Microsoft and SAP do compete within one market segment; however we also do collaborative work on other projects. Regardless of the company we work for, I do believe that most software engineers are intent on doing the best possible job they can to produce the best possible product they can that will ultimately provide a high value solution to the end user customer.

The common theme throughout the conference was the need to improve quality. In my keynote I suggested that customers demand higher quality because the end user customers of today are very different than the customers of yesterday. Today, software permeates virtually all aspects of our life. Today, software is found in children’s toys, in our automobiles, and even in toothbrushes. A decade ago computer users were accustomed to periodic anomalies and  assumed it was the price of technology; however the end user customers today have much higher expectations of software and  presume it will simply work and provide an easy solution that offers some perceived value in their lives!

In my keynote address to the engineers at SAP I described how I would sometimes say “my Mom wouldn’t understand how to do such and such” when describing ambiguous functionality, but in order to remain successful and competitive in today’s market we need to consider designing and developing software for our children and future generations. Of course we want our existing customers (and my mom) to feel comfortable using our software, but it is readily apparent that our children want a very different experience and have higher expectations from software.

Originally Published Wednesday, February 13, 2008

Many testers are familiar with the concept of inattentional blindness (or at least should be in my opinion). Basically inattentional blindness occurs when we are so visually focused on a task  or object that we completely fail to see something out of the ordinary.

But, I am going to introduce my own neologism that I will refer to as contextual blindness. Contextual blindness occurs when someone is so restrictive in their thinking or so biased by their own opinion that they take references to a document or study completely out of context to support their own biased argument. In essence, they are ignoring the original context in which the statements are made, and perverting the sentence or using a statement out of its original context to support an oppositional point of view.

I too have been guilty of this when I made statements without completely researching available data or carefully reviewing empirical, or factually substantiated evidence. (These days, if I don’t have sufficient data or information to support a strong argument for or against something then I try to preface my statement with “I suspect…” or “in my opinion...”). However, some people seem to make a habit out of making wild, and often fallacious statements often in seemingly juvenile attempts of one-upsmanship. I suspect that sometimes people do this because they think they are beyond reproach; that they assume to know more than others, or that they consider themselves to be such an expert that nobody should question anything they say.

As I have gotten older and a bit more wiser, I have learned to question things and reassess my position or my ideas from time to time. I often speak with recognized industry experts, read several books and studies (often presenting  contradictory approaches or perspectives), review empirical data (and just to be clear, IMHO bug count as the only data point offered as empirical data is about as useful as nipples on men), and when I can I try to experiment or experience new things in order to draw my own conclusions. By now your probably asking where I am going with all this…there is a point…read on!

This evening a person sent me mail asking me if my keynote at EuroStar last year was “an attack on certification programs.” She knew I gave one of the keynotes at EuroStar, but was a little shocked that I would attack certification programs. I told her that I gave one of 5 keynote addresses at EuroStar 2007 and my talk was entitled The Path to Professionalism: Skills of star performers, and gave her my perspective of certifications. In her response she forwarded me some mail from a distribution list which included excerpts from a rather lively debate between two other members of the list in which one  of the participants in the debate stated,

“The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT. The keynote speech at Eurostar, in December was an attack on certification programs such as the CSQE.”

So, I reminded her that the second sentence should more appropriately read, “One of the keynote speeches at Eurostar…”. I told her that Michael Bolton gave that talk. (I must say that it was the first time I had seen him speak and I his stage presence is excellent, but I wasn’t overly impressed with his arguments against certification because he simply denigrated the existing programs without offering any other solutions. Several delegates at the conference later asked me about his comments and I deferred by stating, “when in Rome.” (Certifications in Europe are highly valued by employers for a variety of reasons.)  But, the above mentioned statement was merely misleading; it is actually the first statement that is most fallacious, and taken completely out of context.

The statement “The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT” used the following study as a reference. So, let’s take a critical look at that statement and question its truth or validity. (Because, in our jobs as professional testers understanding the correct context, critical thinking and logical questioning are important skills!)

“All we want are the facts, ma’am”

Fact #1. The DoD did not identify anything. The study was conducted by the National Defense Industrial Association (NDIA), which is not in anyway shape or form a part of the Department of Defense (DoD). NDIA members include “individuals from academia, government, the military services, small businesses, prime contractors, and the international community, the opportunity to network effectively with the government.”  There were 26 participants in this study and one participant represented the DoD.

Fact #2. The study did not identify “the failure of traditional testing processes.”  The purpose of the study conducted by 26 participants was to “Identify the top 5 software engineering problems or issues  prevalent within the defense industry.” The participants actually came up with 7 issues, and they determined that one of the top issues in software engineering included “Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems.”

Fact #3. The problem of over-documentation was in the context of a discussion related to tests with a “disproportionate effort on detailed procedures.” This was not one of the 5 (actually 7) problems identified, the participants concluded “Tests are over-documented with disproportionate effort on detailed procedures.” as a possible reason to explain the 5th top issue of “Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems.” I suspect this statement speaks to the fact that many documented tests that I have seen by under-trained testers are simply prescriptive, regimented scripts based on some interpretation of an ambiguous requirements document rather than well-formed tests designed from an in-depth analysis of the system under test.

What the study really said…

But, not only did this person take parts of several statements in the report, munge them together in an attempt to use the findings completely out of context; this person also completely ignored (or purposefully omitted)  other points discussed in relation to this specific problem such as:

• Over-reliance on testing alone rather than robust SW verification techniques.
• Manual testing techniques are labor-intensive, scale poorly, and are unproductive relative to the large investment of resources.”
• Compliance based tools do not adequately cover risks or failure conditions
• Tests are over-documented with disproportionate effort on detailed procedures
• Education, training, certifications are inadequate to develop effective test skills.

The person also ignored (or purposefully omitted) the recommendations by the participants which included:

• Sponsor a study of state-of-the-practice verification and testing approaches.
• Review/update testing policies and guidance to emphasize robust, productive approaches that maximize ROI.
• Review adequacy of verification plans/approaches early in the acq. life cycle.
• Emphasize skilled investigation throughout the life cycle, based on coverage, risk mitigation, high volume automation.
• Strengthen curricula, training, certifications, career incentives for testing roles.

Now, I really don’t understand how someone can read this report and misconstrue the information to imply that traditional testing processes (and it is not clear what they are referring to here), or that over-documentation is one of the top 5 problems identified by the DoD; especially when the report also recommends strengthening policies or guidelines for full requirements traceability.

I have my suspicions as to why the person who made the fallacious remark above might omit all the facts and additional counterpoints in their argument. I suspect those details were not revealed because those points do not support (in fact, they completely dispute) the context-driven ideology that emphasizes manual GUI testing and the vehement opposition to documentation, test automation, coverage analysis, robust verification techniques, strengthening of certifications, or essentially anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

Originally Published Friday, February 08, 2008

I have been tardy in my writing…too many irons in the fire these days. This week I have been teaching new SDETs in the morning, then rushing downtown to teach evening classes on software testing at University of Washington. These days I spend a great deal of my time teaching (mostly about systematic testing approaches and test automation), consulting on testing practices, designing and developing hands-on technical training, and talking about the future of the discipline. This role is personally satisfying, and it is certainly one of the most challenging positions I have ever had. So, as a member of the Engineering Excellence group at Microsoft I am very proud to announce that we are in the top 10 of the world’s most successful learning and development organizations. When Microsoft embarked on its journey in 2004 to refocus our product teams on engineering practices including the drive to increase the technical skills and acumen of our software testers our training was not even listed in Training Magazines Top 125 awards. In 2005 we were listed in 38th place. In 2006 we moved up to 23rd place, and 19th place in 2007. Last week, Microsoft was ranked 9th among the top 125 rated training organizations around the world. Companies are ranked based on various factors including number of training hours per employee, detailed formal programs, use of various learning technologies, and quantitative and qualitative case studies demonstrating strategic business impact. I must admit, I consider myself very lucky to be part of such a great team of professionals who are incredibly passionate about their discipline and strive to have a positive impact on how we design, develop and test our products.

Originally Published Monday, January 28, 2008

The debate over whether testers need to at least understand programming concepts is still raging within the discipline. To me this debate is puzzling because it seems to suggest that as a professional, I don’t have to really understand or be completely proficient in critical aspects of my trade. Even Cem Kaner noted, “I think that the next generation of testers will have to have programming skills.” Actually, there was a time not so long ago when testers had to have programming skills, so it is nice that Cem now acknowledges that skill as useful in testing.

Unfortunately, occasionally even within Microsoft a few people still want to differentiate between STE and SDET by blindly assuming that STE meant non-programming testers. The fact is, that the old STE ladder level guidelines clearly stated skills such as debugging production code, and design and develop effective automation as required skills for Microsoft testers. Unfortunately, some managers chose to selectively ignore these skill requirements and some groups chose to differentiate between GUI testers and  any tester who could write code by labeling them as STE and SDET respectively.  (This was a horrible abomination of job titles in my opinion.) The new SDET competencies at Microsoft are designed, and supposed to be implemented in a manner, to reinforce the essential skills we expect from our testers so a tester at a certain level in their career stage in one business unit essentially has equitable skills of any other tester at the same level in their career stage in any group in the company.

But, people are often resistant to change, and as I wrote in my last post some people choose to wallow in self-pity, pretend they are a victim of some evil plot, hypercriticize change with dogmatic arrogance, and incessantly bemoan dubiously negative aspects of change from an often overly emotional narrow-minded perspective. A person who moved from a testing role to program management stated, “I was a tester because I understand how users think and how they use products and I wanted to use that knowledge to make our software better.” Really? We make software better by beating quality into it? Does this demonstrate a good understanding of software processes and good business logic? I only ask these questions because it is pretty well-known that it is much cheaper to prevent defects, and that many defects can be found in the design process. So, I am asking myself why in the world didn’t this person start as a Program Manager (responsible for interpreting marketing analysis and customer feedback into requirements and product design) or become one before now? What is even more amazing about this statement is that it doesn’t even acknowledge the fact that as a program manager this person is now in a role that should have a direct connection to the customer and a greater impact on making our software better. A development strategy or process that emphasizes customer advocacy primarily in the testing phases is ridiculously immature and a gross waste of resources since it is widely known through empirical studies that it is cheaper to prevent defects by better designs and clear requirements as opposed to finding them during a testing cycle.

The same person stated, “I wanted to keep breaking software in the incredibly fun, very effective way I had been doing.” (Personally, I find API testing (which can also use a black-box approach), and white box test design extremely fun and intellectually challenging, and is also very effective when used appropriately.) Unfortunately, this comment seems to perpetuate a myth that testers make software better by finding bugs, and it also demonstrates an extremely limited view of the role and overall potential value of software testing to an organization. This is indeed a very narrow, antiquated (in technology time), and immature view of software testing that emphasizes testing as primarily a bug finding endeavor. However, Beizer wrote that black box testing exercises approximately 35 – 65% of the product, and Marne Hutcheson and I have empirical data that demonstrates that GUI testing (which is one type of black box testing most people are familiar with, and the type of testing most non-technical testers are limited to perform) is not as effective as most people want to believe, and it is often more costly as compared to using a variety of approaches to software testing. Again, even Kaner notes, “Programmer productivity has grown dramatically over the years, a result of paradigmatic shifts in software development practice. Testing practice has evolved less dramatically and our productivity has grown less spectacularly. This divergence in productivity has profound implications—every year, testers impact less of the product. If we continue on this trajectory, our work will become irrelevant because its impact will be insignificant.” (I highly suspect the ‘testers’ Kaner is referring to in this context are primarily non-technical, GUI testers since that is the type of testing emphasized in his BBST course.)

There is no doubt that a person who does not at least understand programming concepts or have an in-depth technical understanding of the system they are testing is unable to perform various activities that may be required in the role of a professional software tester. That person cannot perform code reviews (which have been proven to find certain classes of defects more effectively than any other type of testing); they cannot analyze code to determine which areas of the code have not been tested and design tests from a white-box approach to increase testing effectiveness and reduce risk, they cannot debug errors and identify root causes of defects, they cannot automate tests to free up their time or reduce costs during the maintenance phase of the product lifecycle, they may not be able to adequately analyze and decompose test data, etc. While some companies don’t rely on their testers to do this type of work, these are certainly tasks that any professional tester should be able to perform.

I guess there are some software companies that are not interested in actually maturing their processes, reducing long term costs, have no interest in the intellectual property value of testing artifacts, or simply want to continue to rely primarily on GUI testing to get a ‘gut-feel’ of their product before releasing it. However, many large companies that produce software (Microsoft, Cisco, Google, Siemens, etc.) understand the value add proposition that professional testers provide to the organization health and specifically hire people into testing roles who have both broad technical skills as well as the common traits we tend to associate with good testers.

This post is not to question the need for non-technical people who have in-depth and current domain or business knowledge of the application space, or who understand the market expectations and customer demands/needs in the software engineering process. The question I ask is whether the value these individuals bring to the software development process is misplaced, or would their contribution be more cost effective and provide greater overall value to the customer if they were in a role (other than testing) that better utilized their knowledge by contributing to defining requirements and designing high quality software rather than trying beat in quality through bug finding?

Originally Published Monday, January 21, 2008

"If a man is called to be a streetsweeper, he should sweep streets even as Michelangelo painted, or Beethoven composed music or Shakespeare wrote poetry. He should sweep streets so well that the hosts of heaven and earth will pause to say, here lived a great streetsweeper who did his job well."- Rev. Martin Luther King Jr.

As I was growing up my parents taught me the value of working for things I wanted, and to how to bear the burdens associated with responsibility. When I was a teen we moved to the country because my mother had become very active in horseback riding and my father wanted to stop boarding the horses. So, we moved out of the suburbs and away from the upper Chesapeake. I was not really happy about this because I love the water. As a young boy I spent a great deal of time with my friends hanging out in the marinas where I learned a lot about boats, hard work (from the bright work to the bilge), and life in general. As I young lad I remember listening intently to the trials and tribulations of old blue crabbers, fishermen, and sailors. I loved swimming in the tributaries of the upper Chesapeake, fishing for yellow perch, sunfish, and going crabbing and walking the riverbeds at low tide for soft-shell crabs. I also was not really looking forward to the change because it meant getting up very early in the morning to feed horses, and cleaning stalls after coming home from school. But, since I also opted to get a horse, that is burden of responsibility I had to bear.

Throughout life our environment changes. That’s just the way it is. New ‘doors’ open, and some ‘doors’ close. I often recall my first manager at Microsoft telling me that it is up to me to control my own destiny at the company. But, I had been around long enough to realize that life also sometime throws us curves or unexpectedly changes direction, and Microsoft is certainly a dynamic company. It is often during times of change where new and exciting opportunities present themselves. Even during times of change I believe we still control our own destiny (at least somewhat). When our environment changes (as it always does) we generally have many choices. For example, often we can embrace the change and dynamically adapt and/or rise up to new challenges that life presents. Or, we can choose to wallow in self-pity, pretend we are a victim of some evil plot, hypercriticize the change with dogmatic arrogance, and incessantly bemoan the dubiously negative aspects of the change from an often overly emotional narrow-minded perspective. This latter choice is usually not an especially productive one (personally or professionally), and generally only demonstrates one’s extremely biased, and limited view and their incapacity to grasp the "big picture."

Let’s face it; we have chosen to work in one of the most dynamic industries in the world. Change is all around us, although some things remain relatively the same. For example, the techniques medical doctors use in the initial screening of certain maladies have remained relatively constant for decades, but at the same time advancements medical imagery has made tremendous technological leaps forward. Likewise, the practice of exploratory testing has been used extensively in software testing for decades, but the effective application of combinatorial analysis (pair-wise, triplet, n-wise) of interdependent or semi-coupled parameters has only recently become a mainstream best practice.

The testing discipline is undergoing tremendous change these days. There are many people around the world who are serious about maturing and advancing our profession. Some ideas are great, some still need to be refined, but at least they are seriously investigating at ways to advance the profession. As a tester working in a rapidly changing industry we must constantly re-evaluate our skills, and increase our professional knowledge of software testing and computer systems in general in order to provide the best possible service to our employer.

I think if someone chooses testing as a profession, then they should strive to become a professional in the discipline, and develop and refine the skills and knowledge that entails.

Originally Published Friday, December 28, 2007

The easiest thing to criticize is that which one does not fully comprehend.

There has been a lot of discussion lately about Jerome Groopman’s book How Doctor’s Think and a correlation between doctors in the medical profession and software testers. The book is an excellent read, and provides readers with valuable insights not only with medicine, but we can certainly abstract many of the fundamental lessons to testing software. Indeed there are many similarities between doctors practicing medicine, and software testers testing software. I won’t belabor this point since many others have discussed the similarities, and one only has to read the book to discover the obvious.

However, there are several aspects that some people seemingly blindly overlook when comparing medical doctors or others in the medical profession to software testers. For example, medical doctors are highly educated in biology, chemistry, human anatomy, physiology, etc. In other words, doctors spend a great deal of time being formally educated in subjects directly related to their profession. Compare this with many testers working in the field of software testing. This is not to imply that in order to be a professional tester that one must study computer science at a university; however, I know a great number of testers who have worked in the industry for several years and still do not understand basic programming concepts, lack the ability to perform an in-depth analysis of test data, and are incapable of adequately decomposing or modeling features below the user interface. In other words, testers who lack the basic knowledge of computer systems and software are utterly blind to the internal workings of the system on which they are working. Doctors also spend a great deal of time post graduation reading medical journals and attending medical conferences and workshops to continue their education and build their knowledge and skills. Dorothy Graham, Marne Hutcheson, Steve McConnell, and I have collected a lot of empirical evidence to suggest that very few testers have ever received any formal training in software testing methods, and less than 1% of testers polled have read more than one book on software testing. Lee Copeland presented an excellent talk at a conference entitled "The Nine Forgettings" in which he asks, "How can we call ourselves professional testers if we are ignorant of the fundamental techniques of our craft?"

Doctors also realize there are practical techniques (systematic procedures) commonly used in their profession that are extremely useful in the correct situation and valuable in either diagnosing an ailment or identifying contraindications, and competent doctors know when and how to apply those techniques. Conversely, I often hear testers whom I suspect have a superficial understanding of the overall system or fail to adequately perform an in-depth investigation of the system on which they are working refer to testing techniques as folklore. Who needs techniques…if we pound on the keyboard or touch screen long enough we are surely bound to find a bug! (The only folklore I’ve ran across in the industry is a belief that only a handful of people really understand, or are even capable of understanding exploratory testing.)

Medical science also has an common body of knowledge and professional jargon that is universally accepted throughout the trade. When doctors refer to exploratory surgery there is a common understanding of what is being discussed. Doctor’s don’t pointlessly argue how one person’s practice of exploratory surgery is different than another person’s approach and therefore it is not really the ‘exploratory’ surgery that "I" preach. Reputable doctors also don’t needlessly make up words to make themselves sound like they offer something new or revolutionary. When they discuss medical conditions they discuss things within rational contexts. For example, when someone complains about numbness in the fingers they may perform tests to check for adequate blood flow to the hand, or they may ask the patient about neck injuries or pain in the neck or upper shoulder region to see if the problem might be caused by a pinched nerve. I don’t suspect that too many doctors would ask the patient what they had for breakfast, or if they recently stubbed their toe to diagnose numbness in the fingers (but, I am not a medical doctor so that is just guessing on my part). Basically doctor’s understand the head bone is connected to the neck bone and don’t waste a lot of time hypothesizing "what if the head bone was connected to the knee bone?"

Finally, a really big difference about doctors and many testers is in how they fundamentally approach their job. Doctors are constantly striving to help people stay healthy. In other words, they are trying to prevent the spread of illnesses, or searching for ways to help people from becoming sick to begin with. Of course, this means that doctors must have an in-depth understanding of the ‘system,’ the types of ‘bugs’ a system might be exposed to, and how those ‘bugs’ can act on the ‘system’ or how the ‘system’ reacts to those foreign agents.

As Groopman discussed in his book, "There are primary care physicians in every hospital who speak with great sensitivity and concern, and their longtime patients love them, but clinically they are incompetent…" Likewise, I have met many people, some at MS and some in the industry, who still think software testing is simply about finding bugs, and that additional skills or knowledge of the profession is unnecessary if one is a good bug finder. I often listen with amusement at talks or when I read articles by people who profess that testing is about providing information, but the only ‘information’ they seem to provide is how to expose yet another obscure bug, or denigrate practices by incorrectly applying techniques out of context (such as attempting to apply pairwise analysis on a function requiring sequential inputs then claiming pairwise analysis is not a best practice because it is not good at finding bugs while failing to mention where and when the technique is best applied and other types of information that the technique provides such as increased code coverage). As I said in the opening, it is easy to criticize those things we least understand (unless of course we are purposefully obfuscating facts for personal motives). And the uselessness of any critique is only exacerbated by irrational arguments, illogical alternatives, or personal attacks.

So, how does all this relate to Microsoft’s hiring practices and why we primarily hire people with computer science backgrounds?

Easy, graduate coming from university with a computer science background have a strong understanding of the ‘system’ and the ‘system internals’ much like the doctor understands physiology and human anatomy. This doesn’t mean that we only hire testers with a computer science degree. Myself, and many other testers at Microsoft do not have a computer science degree; however, we have also not stagnated or simply rested on our ability to find bugs. We have constantly strived to improve our technical skills and overall understanding of computer systems, software testing practices, and testing knowledge. Interestingly enough, Microsoft career job guidelines have always required testers to be able to automate tests, debug production code, and design tests from both a white box and black box perspective, etc. The difference now is that those skill requirements are universally applied across the company.

Also, much of our internal training can now be based on a common baseline of expectations. For example, when we teach testers how to design tests from a white box perspective, or when I talk about the best practices of code reviews I expect that the testers already know how to read code. Our automation courses require that testers already know how to write code using procedural programming and object oriented programming approaches. Our training simply exposes testers to namespaces, classes, and methods commonly used in test automation (not in application development), and mentor them on how to design tests that will provide value to the organization from an automated testing perspective. (Also, I can attest that our testers are very good critical thinkers and not simply mindless droids who bang on keyboards or pump out simple rote, prescriptive scripts (similar to what is described here) and label them automated tests.

Finally, one of the things we reinforce in our training and throughout a person’s career growth at Microsoft is driving quality upstream and defect prevention. We have senior testers who are mentoring developers in how to design and write better unit tests. Our testers are engaged in design and code reviews. Many teams now rely on testers to debug problems to the line of code in order to identify patterns of problems. We have testers who are doing root cause analysis of specific classes or types of defects and building tools or enabling processes to identify those types of defects earlier in the project or development cycle. And yes, our testers still design and execute a lot of manual tests. They use the product in development everyday by dog-fooding. And most testers even participate in newsgroups and sit with or listen in on product support calls to indirectly hear from end users.

So, much like you probably seek out the best possible medical doctor as a primary care provider or when struck with an illness, at Microsoft we want highly competent individuals in our software testing roles who understand that testing is a huge challenge and requires a great breadth of skills and knowledge in order to test software from various perspectives and approaches, and who possess a great deal of in-depth knowledge of the system they are working on. We seek professionals who realize that constantly bemoaning the inadequacies or drawbacks of an approach or perspective solve nothing, and instead actively engage in finding great, scalable solutions to problems in order to reduce testing costs and help drive overall quality upstream.

In a dynamic industry that changes rapidly as professional testers we must constantly reevaluate our skills and knowledge to stay abreast of changes in technology, and learn more about our chosen profession. We must be vigilant and strive unceasingly to improve our skills and knowledge of computers, computer software, and our profession in order to remain competitive with our peers and remain a valuable team member in our organization or to our employer.

Originally Published Sunday, December 23, 2007

It never ceases to amaze me that every time we see a calamity involving software the immediate reaction of the sensationalist media types and other people who are generally misinformed is to blame inadequate testing. Recently, it seems that Joel Spolsky not only fell victim to rumors and misinformation, but also to the lunacy of blaming the testing organization for the rather lack luster adoption by individual consumers to move to Windows Vista.

Sales of Windows Vista are slow for a myriad of reasons. The UI is a radical departure (which most design experts applaud), security features are annoying (which most security experts applaud, yet many individual consumers dislike), machine requirements (means many would have to purchase new hardware), performance issues, etc. But inadequate testing or too much automation? Pah…leaze! I usually like reading Spolsky’s blog, but must admit this is the first time I can recall in which he has used unsubstantiated rumors and innuendo to reach such a faulty conclusion as suggested in his post. Too bad…

Joel’s attack was not about testing per se, but more about his assumption that Windows Vista suffered because of too much test automation. He based his conclusion on the information from a blog noted for its petty bitching about things inside of Microsoft. (Let me say this…there are a lot of things about MS and corporate policy that I don’t like. Bitching about those things is easy and usually non-productive. But, working with the diversity of people who make up one of the most influential software companies of our time to effect change is difficult and challenging. I mostly choose the later route.) And the non-productive bitch-fest blog based its disinformation on the observations of Chris Pirillo’s blog post written in May of 2006 exposing several critical issues such as:

• Shouldn’t the phrases used during the setup state (Copying Windows files, Expanding files, etc) be intercapped? i.e., Copying Windows Files, Expanding Files, Installing Features, etc.?
  [Uh....no. Check with the terminologist and documentation experts]
• The "You’re Ready to Start" graphic flickered and jumped when I pressed the Windows button.
  [Does it happen every time, or was it an anomaly, or did was it perhaps an electrical or wind storm...because here in Washington when we get wind storms my lights flicker all the time.]
• Control Panel>Additional Options displays…get this…"There are no items to display." Well, if there are no items to display, then why did you show me the link in the first place?
  [Perhaps because when user's install Additional options such as the Java Control Panel, they have a common place to access them]
• The Undo and Redo button for the Photo Gallery Viewer are at the bottom of the window, whereas I would’ve expected them to be at the top. Thought they weren’t there until I looked hard.
  [ Is this inconsistent with other Task Panes? Or is this based on your personal preference? BTW...I happen to like them at the bottom no cluttering up other stuff.]
• What if I don’t like the red "X" close button? Why can’t I change that to…purple?
  [Great suggestion that I am sure millions of user's will appreciate. I think they have that scheduled as a hotfix somewhere around the turn of the next century]
• Microsoft Windows Mail splash screen needs an overhaul
  [Great...I am sure the customers who really want less functional defects or better performance will be more than happy to know we instead decided to spend time overhauling the splash screen based on your valuable input.]

This is not to say that Chris’ feedback was ignored or not valued (I also enjoy reading Chris’ blog from time to time). Personally, I would like to think we investigate a lot of the feedback we get from our customers (we simply don’t act on all of it because…we’ll we have bigger problems to solve other than allowing users to change the color of the close button.)

Spolsky stated, " The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one. And, None of those things could be checked by automated scripts."

Unfortunately, I guess Joel hasn’t remained abreast of advances in test automation and perhaps assumes that automation is still primarily record/playback or simple rote scripts in some proprietary or limited scripting language. As software becomes more complex, and as the capabilities of test automation improve the fact is that quite a lot can be done with automation.

For example, checking the consistency of fonts in each dialog throughout a product is more effectively and efficiently accomplished via automation than by human eyes. Automation can also detect differences in the alignment of controls on dialog boxes and also detect clipping and truncation of controls or text in controls more efficiently than humans (and we actually have empirical data to prove it). Spelling and politically sensitive words are more efficiently detected via automation. Most testers and developers that I know are not experts in English grammar and I would never hire an English language major as a tester just to test grammar. We made our documentation/content experts responsible for messages and other ‘string’ content thus pushing quality upstream!

(Also, as a side note…the majority of the testers that worked on Windows Vista were the "old testers at Microsoft" doing they same work they did shipping Windows Xp.)

Isn’t if funny that when certain things don’t meet our personal expectations (Vista sucks), or we are fearful of change (test automation is evil and we should continue to hire hoards of non-technical people to bang on keyboards) it is so easy to jump to faulty conclusions, base assumptions on unfounded hearsay and rumor, constantly utter petty unproductive complaints, and generally play the victim. Of course, being a victim provides you with attention from others, and useless bitching and wild speculation is way more fun than facts and research…besides…it makes people laugh, and when people laugh they are happy, and the role of the tester is to make people happy…right?