I.M. Testy

Originally Published Friday, March 27, 2009

I often find discussions about quality to be hypothetical, and in fact unless you define your specific context the word itself is nebulous, vague, or simply meaningless philosophical psycho-babble. For a while now, I previously posted my opposition to the simplistic notion that quality is value to some person. Sure, most thesaurus’ equate the two words as synonyms, and the context-driven posse constantly regurgitate Weinberg’s  quote about "quality is value to some person." Yes, that is one perspective of quality, but only one.

In the past I have taught that quality is not value in the context as one of the goals of software testing. My definition of value is the purpose or usefulness of a product in satisfying a customer’s needs or wants. My definition of quality was based on tangible aspects of the attributes or capabilities of a product from an engineering perspective. Our definition of software testing as any task designed to evaluate or assess the attributes and capabilities of a software project in relation to implicit or explicit guidelines in order to provide information to the management team (the people who make the business decisions). Those evaluations result in measurements we call quality measurements or criteria (the "essential or distinctive characteristics, properties, or attributes" that are critical for the success of our project) and that is part of the information we present to the decision makers to help them make more informed business decisions. Remember, Weinberg also stated "Thinking about measurability from the beginning is an essential part of creating a well-formed effort."

The other day I met with members of one of our research teams and they were talking about quality in terms of tenet and non-tenet quality; and I clarified that basically they were essentially talking about the customer perceptions of quality versus the engineering aspects of quality. Surely, from a holistic point the observation and reliable measurements of both perspectives of quality are important for the success of any organization. Customers usually buy/download software because it helps them satisfy some need or want. The decision of which software product to buy made be based on personal bias, or perhaps the herding instinct, or it may be more rational based on the comparison of features (measureable attributes and capabilities). Once the customer begins to use the software they form their own opinions based on expectations and/or previous experiences that provides the company with information regarding customer satisfaction.

So, the next time someone starts talking about quality stop them and ask them if they are talking about the engineering aspects of quality, or the customer perceptions of quality. They are closely related, but different perspectives of the same topic.

Originally Published Wednesday, March 19, 2008

Last October after presenting a keynote at the Conquest software testing conference I was invited to speak at an internal quality conference at SAP. At first it may seem a bit odd because Microsoft and SAP do compete within one market segment; however we also do collaborative work on other projects. Regardless of the company we work for, I do believe that most software engineers are intent on doing the best possible job they can to produce the best possible product they can that will ultimately provide a high value solution to the end user customer.

The common theme throughout the conference was the need to improve quality. In my keynote I suggested that customers demand higher quality because the end user customers of today are very different than the customers of yesterday. Today, software permeates virtually all aspects of our life. Today, software is found in children’s toys, in our automobiles, and even in toothbrushes. A decade ago computer users were accustomed to periodic anomalies and  assumed it was the price of technology; however the end user customers today have much higher expectations of software and  presume it will simply work and provide an easy solution that offers some perceived value in their lives!

In my keynote address to the engineers at SAP I described how I would sometimes say “my Mom wouldn’t understand how to do such and such” when describing ambiguous functionality, but in order to remain successful and competitive in today’s market we need to consider designing and developing software for our children and future generations. Of course we want our existing customers (and my mom) to feel comfortable using our software, but it is readily apparent that our children want a very different experience and have higher expectations from software.

Originally Published Monday, July 16, 2007

I previously blogged about quality and value, but after giving it more thought I determined that quality and value are indirectly related, but quality is not value!

I define testing as any activity designed to evaluate an attribute or capability of a program in order to determine its’ ability to meet or exceed implicitly and tacitly applicable standards, guidelines, and/or customer expectations. Even Jerry Weinberg clearly states "Whatever else it means, quality at least means meeting specifications for each job." This does not mean that testing merely verifies functionality against some specification or requirements document. (Only a novice tester would assume testing involves writing tests that simply validate the spec.) But, quality is the specific, measurable, and quantifiable results of the evaluation of the attributes and/or capabilities (especially those that are critical for the success) of a product. Quality attributes and traits can be quantified via various measures, and those metrics should ideally track towards established goals, or compared against baselines or competitors products for example. For example, we can assess quality in terms of some degree of better-ness of comparable attributes for features, or exceeding stated or implied goals or expectations.

Conversely, value is something that serves a useful purpose to some person. When most consumers purchase software, or a vehicle, or a piano they do so because they will, or perceive they will, derive some personal benefit from that purchase. In other words, when something fills a personal need or provides some benefit (either real or perceived), that thing is providing a value to that person. Value cannot be directly qualified; although people can agree or disagree whether or not something provides value to them and they can debate relative degrees of value.

Let’s take the example of a vehicle. When I decided to purchase I vehicle I bought a truck because a truck provides a greater value based on my personal needs as compared to a car, a motorcycle, or a SUV. When I started looking for a truck I compared the various attributes and capabilities of the available models in my price range and narrowed it down to a Chevrolet, Ford, and Dodge. Some key attributes I compared were towing capacity, fuel mileage, fuel tank capacity, and warranty. The final decision breaker for me was the crew cab feature with full sized rear doors. For me this provided additional value because when I go skiing I want to be able to take along adult friends (and I don’t know about you, but I would be miserable if I were squeezed into an extended cab for a 1 1/2 hour ride up to the ski lodge). So, I went to each dealer and sat in the back seat of each truck and finally settled on the Chevy 1500 HD. For me, a truck provided value and the Chevy 1500 HD met or exceeded the quality expectations I expected in a truck. Now, when people think of a truck they typically don’t think of a quality vehicle. Perhaps that is because a truck doesn’t provide much value to them, or perhaps they are mentally comparing a truck to a more comfortable sedan which is not a realistic comparison. But, I also think it is quite common for people to confuse quality and value.

Unfortunately, it seems that many managers and testers in the software industry also confuse quality and value. That is why many companies attempt to assess the ‘quality’ of their products on subjective measures such as smiley sheet surveys that go out to customers, or anti-quality metrics (such as defects found during the SDLC, or defects found post release). Also, some testers are convinced that ‘customers define quality.’ Poppycock! Customers don’t define quality; the project team defines the set of attributes, capabilities or features they think are critical to the success of the project (and will provide some perception of value to the customer), and sets goals that suggest some desired measurable level of acceptance (‘quality bar’) that meet or exceed expectations in a positive manner for those features and attributes.

Customers who decide to purchase or download software or use services do so because it provides a value for them at reasonable cost (money, time, ease of use, etc.) If the customer likes that software, then to them it is quality software. In this case, the software not only does what they need it to do (value); from their perspective it does it sufficiently well or better than a competitor’s product (quality).

Now, I am not providing this perspective simply to be contentious, or insight unreasonable debate and random objections. But, I am beginning to think that one of the primary reasons we can’t get our heads wrapped round software metrics and start proposing meaningful solutions to the difficult problem of software quality measurement is because testing is focused on trying to satisfy the end user customer and make the user customer ‘happy’ rather than providing real, qualitative information to the primary customer of testing whom I consider to be the project’s management team. We know the most successful businesses establish quantifiable measurement programs in order to improve in certain areas and reduce costs. But, it sometimes seems that the commercial software industry is content with the status quo, and we trick ourselves into believing we (our company or our product or processes) are getting better based on customer feedback, sales or adoption, and feel-good assessments. Establishing an effective measurement program that establish baselines, and assess improvements in processes, productivity, or cost savings is not easy; it is not quick, it exposes weaknesses or failures in the organization, and it takes time to implement changes and even the measurements must adapt over time. Perhaps we don’t need to be more efficient, or increase effectiveness, or reduce costs.  Perhaps our business practices are ‘good enough.’ In which case, I guess we shall continue to go with the smiley sheets and anti-quality metrics because they are cheap, easy, provide instant-gratification, are non-judgmental, and it doesn’t matter if we continue to make the same mistakes over and over again. But, we should acknowledge the fact that smiley faces and anti-metrics do not tell us whether we are improving our processes, or increasing our productivity, or reducing overall (and long term) costs.

Originally Published Wednesday, April 04, 2007

The value of testing is important and critical to the success of many projects. However, testing is NOT solely responsible for the quality of a project!

When we discuss the role of testing and quality in our internal training we emphasize the primary objectives of the tester as providing information and assessing (preferably via quantifiable measurements) quality (and specifically focusing on the quality traits or attributes that are most critical to the success of the project).

But, there are still some managers who would like to believe that the testing group within an organization is responsible for quality. And there are even some testers who will tell you they are responsible for quality. I won’t begin to speculate how this folktale came to be, or why it is perpetuated. But, it is a folktale (any belief or story passed on traditionally, esp. one considered to be false or based on superstition). Folktales usually persist in societies for a variety of reasons. There is no doubt that quality is important. But, quality is hard to define; quality is hard to measure, and it is hard to translate quality into value for the customer. There is no doubt that as a tester I can influence the quality; however, responsibility implies the ability to also control. How much control does a typical testing organization in a commercial software company actually have upon the quality of a project?

There are 3 simple tests to refute the conjecture that testing is responsible for quality.

1. Can we test in quality?
2. Can testing find all the defects?
3. Will all the defects be fixed? (Here a defect implies deviation from explicit or implicit requirements or expectations, or unexpected behavior.)

The answer to these three questions is no! Unless you are an absolute novice to software engineering you will probably agree we cannot test in quality. Testing cannot find all the defects in any complex software project (and there is ample evidence of this). And anyone who has any experience on a project of significant size will admit that not all defects are fixed (including functional defects). So, as a tester I know I can’t test in quality, I will never find all the defects, and I know that some of the defects I find will not get fixed (because I can’t use electric cattle prods on developers and I am not going to go into the code and fix the bug myself), so how in the world can I be responsible for the quality of a product?

As a tester I am responsible for providing information regarding the perceived quality and/or risk of a project as assessed by a series of tests, observations, and experiments, and that information is provided to management (who hopefully use it) to determine whether a commercial software product is released to the public.

So, who is ultimately responsible for the quality of a product? Often times folks will rally around the “everyone is responsible for quality” mantra. But, W. Mark Manduke wrote an nice article in STQE magazine a few years ago stating “When quality is declared to be everyone’s responsibility, no one is truly designated to be responsible for it, and quality issues fade into the chaos of the crisis du jour.” He concluded “…when management truly commits to a quality culture, everyone will, indeed, be responsible for quality.”

Thus, ultimately the management team (the decision makers) is responsible and accountable for quality.

Originally Published Sunday, February 18, 2007

Quality is a mysterious thing. It seems difficult to articulate definitively, but it is relatively apparent when it’s lacking. The term quality has different meanings to different people, and even the dictionary defines the word broadly. But, ‘quality’ is integral to the discipline of software testing, so at some point we simply have to wrap our heads around it and understand how it relates to our profession. We can say that at one end of the spectrum is Jerry Weinberg’s relativist point of view that “Quality is value to some person.” At the other end of the spectrum is a more prescriptive perspective by Philip Crosby who stated “Quality is conformance to requirements.”

Several years ago when I first became a test manager at Microsoft I read Weinberg’s seminal works entitled Quality Software Management, and I still highly recommend leads and managers responsible for leading engineering teams and shipping software products read all 4 volumes. In Volume 2; First Order Measurement, Weinberg stated “Quality is value” and ‘value is always perceived value’.  The quality as value proposition is important for managers to understand because it largely determines business success. Basically, if our customers realize a value then for them there is also some perception of intrinsic quality. In this context quality is largely regarded as a value proposition for the business goal of increased customer satisfaction which Weinberg refers to as first-order measurement.

But, the quality as value is a business proposition. Quality as conformance to requirements implies that there are essential or distinctive characteristics, properties, or attributes that must be evaluated and measured. In fact, even Jerry Weinberg clearly states “Whatever else it means, quality at least means meeting specifications for each job.”  As software testers our most important objective is to provide accurate, qualitative information. So, in this context quality is composed of measurable goals that are indirectly related to customer value, and Weinberg refers to this as second-order measurement.

So, when some people talk about quality in terms of ‘value’ they are perhaps approaching the topic from a high level perspective which is mostly external customer focused. However, since I often interact with new individual contributor testers and try to teach them important concepts and skills of the profession I tend to have a more measurement-centric approach to defining quality. It is the second-order measures that testers are generally responsible for obtaining and relating back to management as quality metrics and/or risk assessment information. But, qualitative software measures are also important for managers to understand as well. If senior managers (the decision makers) don’t understand second-order measures they may not use them effectively as part of their decision process, or they will likely ignore or do away with second-order measures when the project spins out of control, and the result will likely be what Weinberg refers to as “quality crisis.”

Certainly, understanding the customer needs may be first-order measurement of quality. But, defining the indirect measures of quality is a much harder problem. Testers must interpret customer values into goals and goals into measurable tasks as second-order measurements that correlate to ‘quality’ assessment. Things that make this an especially challenging task include collecting the wrong measures, abuse or ignore the metrics, and we must constantly make adjustments based on the information and on changing customer values. Also, the measures for one project are often different than the metrics needed on another project. But, one thing is for certain; without the ability to measure the “essential or distinctive characteristics, properties, or attributes” that are critical for the success of our project then we really have no way of knowing whether or not we achieve/exceed our goals other than feel-good or best guess.

“Thinking about measurability from the beginning is an essential part of creating a well-formed effort.” – Jerry Weinberg.

Quality is the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. Not to be mistaken for “degree of excellence” or “fitness for use” which meet only part of the definition.
[ISO8402].