I love autumn! Yes, I am definitely a boy of summer and very much prefer warmer weather; however, there is something special about autumn. This past weekend my daughter, and my 2 friends Dongyi and her husband Yuning and I participated in the Rum Run sailboat fun race with an overnight raft up at Bainbridge Island’s Port Madison. Saturday morning was quite rainy, but the wind was blowing 15 knots with gusts to 25 knots and NOAA weather radio announcing gale force warnings in Puget Sound. Wow…what a ride! But, it was actually the rather relaxing sail back to my marina on Sunday morning that rekindled the beauty of autumn in my mind. The bright reds, golden yellows, and pastel browns of the foliage seemed to blend into a collage framed by the darkness of the waters of Puget Sound and the snow covered peaks of the Olympic mountains. The beauty of autumn reminds me about change. A sloughing of the old, the cleansing brought about by the pure white snows, eventually followed by the new and fresh growth that blossoms in spring.

Just as the earth goes through variable cycles of rejuvenation, we must also continually update our tests, and (more importantly) the test data we use in our test cases to prevent them from becoming stale. Trees shed their leaves in the autumn and new leaves emerge in the spring, but the tree is fundamentally still the same tree. Similarly, a well-designed test case has a unique fundamental purpose and by changing the variables we can grow the value of that test case. Of course, the cycle of change in test data should be dramatically shorter in duration as compared to the seasonal changes of mother earth.

Here is a simple example of how a well-designed test case using variable test data can increase the value of the information each  test iteration provides through increased confidence and also potentially reduce overall risk. In my role at Microsoft I am in a unique position to not only conduct controlled studies, but I can also implement ideas into practice on enterprise level software projects. One experiment I started about 2 years ago involved multiple groups of testers (sessions) located around the world divided into 3 separate control groups. Each control group tested the identical web page that would display the stock price if the user input a valid stock ticker symbol into a single textbox on the page and pressed the OK button. The only difference in the control groups was the instructions to perform single positive test case with the specific purpose of “ensure any valid stock ticker symbol displays the current stock price for the publicly traded stock specified by its symbol.” The purpose of the study was to determine if different cultural and experiential backgrounds impacted the test data used in a test based on the instructions for a test case. The study collected demographic information on the participants as well as specific inputs applied to the web page. Information on the oracle used by the students was collected anecdotally. Step one in each test was identical because we were not interested in how the tester launched the browser. (Of course this assumes there are other tests that test the multitude of ways to launch a browser and navigate to a URL. Also, if the browser failed to launch the test case is blocked.)

Group 1 was given the most vague instructions for the test case. The instruction was simply:

1. Launch browser and navigate to [url address]
2. Enter a valid stock ticker symbol and press the OK button and verify the accuracy of the returned stock price.

The instructions in the test case given to Group 2 were also somewhat vague, but provided a little guidance both on input and oracle.

1. Launch browser and navigate to [url address]
2. Enter a valid stock ticker symbol (e.g. “MSFT”)
3. Press the OK button
4. Verify the returned stock price is identical to the current stock price listed on the appropriate exchange

Group 3 had similar instructions to Group 2, but the group was given additional guidance as indicated below.

1. Launch browser and navigate to [url address]
2. Enter a valid stock ticker symbol from a publicly traded stock listed on any public stock exchange
   Listings of valid stock ticker symbols are on stock exchange web sites such as:
   http://www.nyse.com
   http://www.eoddata.com/Symbols.aspx
   http://www.nasdaq.com
   http://www.londonstockexchange.com
3. Press the OK Button
4. Verify the returned stock price is identical to the current stock price listed on the appropriate exchange

Results

The results were mostly not surprising, but rather reinforcing. For example, we expected Group 1 to be rather random, but mostly aligned with ticker symbols they were familiar with. Of course, the majority (90%) of stock ticker symbols entered was MSFT and there was no significant difference in cultural background, locale, experience or educational background. (As this study was conducted at Microsoft I am sure there was some bias as to the symbol entered.) What was most interesting was that testers with no formal training (no previous courses in testing, no CS degree, and read less than one discipline specific book) and with more than 2 years of test experience were approximately more likely (25%) to violate the purpose of the test and enter random or completely invalid data as their first action. In other words, instead of executing the required test their initial reaction was to immediately go on a bug hunt.

In group 2 99% of the participants simply entered the stock ticker symbol “MSFT.” But, what was even more surprising was the fact that one the next day, the same people in that group were given the same exact test, and 95% of them simply reentered MSFT. Perhaps this is laziness, perhaps this is related to the superficial nature of the study, or perhaps this is due to individuals taking the path of least resistance. The percentage of people who entered identical stock ticker symbols on consecutive days was not significantly different between group 1 and group 2.

It should be no surprise that group 3 had the greatest distribution of variable test data applied to the web page. Demographics had no impact on any of the people who were in group 3. The majority of people in group 3 (78%) would select the first stock exchange listed (regardless of what link it was) but there was no significant overlap in the selected stock ticker symbols. When asked to repeat the test on the next day 83% of the participants selected a different link and and a different symbol. Of those who selected the same link 97% selected a different stock ticker symbol. On the down side, approximately 4% of the people simply took the path of least resistance and input MSFT as the test data on both days of the experiment.

Conclusion

One of the most common problems I hear about ‘scripted,’ or pre-defined test cases is that they are too prescriptive and not flexible enough to allow the tester to try things. Of course, a well-designed test case is not simply a prescriptive set of steps inputting the same hard coded test data they run over and over. So, in this study we made the assumption that a scripted test case that specified “Enter MSFT in the textbox” would simply result in the tester entering “MSFT” without any thinking on the part of the tester. Hard-coding variable test data is often times the worse possible way to design a test case.

Vaguely written test cases added some level of variability, but also seemed to increase the probability of the tester executing context free tests outside the scope of the purpose of the test. In fact, what we found was some testers (approx 2%) simply went on a bug hunt and never actually input a valid stock ticker symbol at all during the session.

A test case that provided only one example that is representative of the type of test data required for the test case produced the least desirable results in this study. I am not sure this would be the case in practice. However, based on this study if I were to outsource execution of a test case similar to that used by group 2 the only thing I could guarantee is that MSFT would definitely be tested numerous times, and the variability of other test data would be extremely limited regardless of the number of testers executing that test or the number of iterations.

When faced with a virtually infinite number of possibilities for input variables as test data used in either positive or negative tests we need to test as many possibilities as possible given the available resources in order to increase test coverage and reduce overall risk. So, one way increase the coverage of test data while still achieving the specific purpose of the test case is to provide useful resources that help guide the tester while relying on the tester’s creative thinking skills and curiosity to expand the test coverage.

Of course, we can also increase variability of test data and capture the essence of the tester’s creativity using a similar approach in a well-designed automated test case as well. In fact, a similarly designed automated test case enables us to significantly increase the amount of variable test data that is exercised in order to expand test coverage and increase overall confidence.

Something that raises red flags in my brain is hard-coded strings or test data in either a manual test or an automated test. Yes, I know that sometimes there are times when a test must be very prescriptive and use specific data and follow specific procedures, but I am absolutely amazed how often I see examples of test cases that are so prescriptive in the detail of execution that it completely takes any thought out of executing that test. While it can well be argued that the execution of that test might very well be a brain-dead activity, I would also argue that the person who wrote such a test also lacks creativity and generally has no clue of how to actually design a test.

We did a simple experiment on test design and execution. The purpose was to see if we could design a ‘scripted test’ that provided the tester with greater freedom, cognitive engagement, and deductive reasoning.

The simulation in this experiment was a simple web page in which the user entered a stock ticker symbol (test data), pressed the "Get Quote" button, and compared the displayed result against the expected price at that time (using a real-time 3rd party stock quote monitoring system). The test was a positive test, but was written 3 different ways as illustrated below.

The first test was very prescriptive and was written as follows:

Purpose: Verify the web page displays the most recent quote for a valid stock ticker symbol registered on a major stock exchange

Steps:

1. Enter "MSFT" in the Stock symbol text box
2. Press the "Get Quote" button on the web page

Verify: The displayed quote matches the real-time quote.

Given this test over 95% of the subjects simply entered MSFT and looked for a result. Some did not even appear to compare the result against the real-time quote.

We modified the steps in the test as follows and used a second study group.

Steps:

1. Enter a valid stock ticker symbol in the Stock symbol text box (e.g. "MSFT")
2. Press the "Get Quote" button on the web page

In this second session more than 75% of the the test subjects still only entered MSFT and looked for a result. On a later day in the week, we asked the same group to run the test again. Once again over 75% entered MSFT as the test data.

So, we modified the steps in the test once more as follows and used a third group in the experiment.

Steps

1. Enter a valid stock ticker symbol in the Stock symbol text box from a list of available stock ticker symbols at
   <Link to NYSE>
   <Link to NASDAQ>
   <Link to S&P>
   <Link to London stock exchange>
   <additional links>
2. Press the "Get Quote" button or press the Enter key

In the third part of the experiment over 95% of the third group clicked the links and selected a stock ticker symbol at random. Some testers copied and pasted the ticker symbols from the linked web pages into the Stock symbol text box, but the majority simply entered the symbol via the keyboard. Some (less than 5%) of the participants simply entered MSFT. (Which is not really surprising since they work there!) What was more interesting was that when the same groups were given this test at a later time over 95% of the testers selected a different link and 99% selected a different stock ticker symbol.

This third part of the experiment essentially is the same test (proves the same hypothesis) but uses a descriptive ‘scripted’ test approach. A more descriptive test can still achieve the stated purpose and provides 2 more important benefits.

• The purpose of the positive test (verify the web page displays the most recent quote for a valid stock ticker symbol registered on a major stock exchange) is achieved without hard-coding specific test data or specific results to check against. This means the tester has to use basic deductive reasoning in order to validate the results of the test.
• The breadth of test data used in the test significantly increased (even if the test was executed by the same person), thus increasing the variability of each successive test and provides the tester with great freedom in selecting the data to use in each test and how to interact with the system under test.

Whether or not the "Get Quote" button is pressed or the Enter key is pressed is tangential to the purpose of the test, so in this case it is not important what action the tester takes to send the request to the web service to get the stock quote; he or she has to trigger that event.

I suspect that one reason why many ‘scripted’ tests are very prescriptive in nature is because they are written from the "watch me" perspective. In other words, the test is crafted from "this is what I did, so that is how I will write my test…word for word." I also suspect that in many of these cases the tester really doesn’t have a clear purpose of what he or she is trying to prove or disprove, that tester is simply writing a script or developing an automated test to satisfy some thoughtless process or increase some magic number.

Watching a tester perform a set of steps and then recording that same set of steps in either a manual or an automated test does not constitute test design; it is a brain-dead activity. Test design is not simply watching a person perform a set of actions and ‘scripting’ that into a ‘test.’ And test design doesn’t mean reacting to the results of one test and thinking of another test ‘on-the-fly.’ Designing robust tests is a separate activity from test execution. Designing a robust, descriptive scripted test that enhances the effectiveness of the testing effort requires incredible creativity in order to achieve its desired objective.

So, the next time someone tells you that scripted tests are too restrictive, impede the freedom of the tester, or limits your creativity I would suggest to you that that perspective is rather narrow-minded and limited to a vision in which ‘scripted’ tests are all highly prescriptive in nature and result in a set of brain-dead steps. Conversely, any professional tester realizes that test design is a very creative process involving the application of your cognitive and analytical skills to help you design a test that aids you in proving or disproving your deduced hypothesis from pluralistic perspectives within the context of the situation, and to ultimately enhance the effectiveness of your testing effort.

Many experts have written articles and devoted chapters of books on the attributes of what constitutes a ‘good’ test case. Unfortunately, most books repeat a common (yet limited) perspective that a good test case has a high probability of finding bugs, and Kaner goes to the extreme by stating “A test that did not reveal a problem was a waste of time.” I pondered this for a while and thought, if I run a test to prove that a feature functions as expected is that really a waste of time? Isn’t it a good thing that testers actually spend some time executing tests that demonstrates the product does what the customer expects it to do? Or do we simply want to restrict the value of testing and reduce our testers to keyboard monkeys who bang away at the keyboard in search of bugs? (I believe limiting the perspective of the purpose of a test in testing literature has only perpetuated the faulty assumption that the purpose of testing is to simply find bugs; a point I previously dismissed). So, what is the purpose of a test?

Jorgensen states in his book, Software Testing: A Craftsman’s Approach that “a test case has two purposes: either to expose an error, or to demonstrate correct execution.” This explanation broadens the purpose of a test case to include both verifying the product meets the requirements, and revealing potential errors or defects. I tend to agree more with Jorgensen in this matter, but also think the purpose of a test case involves even more.

There are several objectives of any effective test whether manually executed or automated. The rationale for test case usually falls into one of seven categories. Each category provides value to the organization in different ways, but they all essentially function to reduce risk and qualify the testing effort. So, a good test provides some measurable value to the organization with the explicit purpose of:

• Verifying conformance to applicable standards and guidelines and customer requirements
• Validating expectations and customer needs
• Increasing control flow coverage
• Increasing logic flow coverage
• Increasing data flow coverage
• Simulating ‘real’ end user scenarios
• Exposing errors or defects