I.M. Testy

Well, it has been quite some time since I last posted to this blog, and I apologize to my regular readers for the inconsistency. Since the December timeframe I have been a bit busy behind the scenes searching for a new direction at Microsoft, and last week I was finally able to officially announce that I will soon be a Principal Test Lead in the Windows Phone group on the Communications team. My feature area will be Social Networking. I am so excited about the change, and getting back into the “real-world.” I have already started making the transition (more mentally than physically) and I am recalling the feelings I had when I first started at Microsoft. It’s incredible!

For the past 5 weeks I have also been teaching a course on Software Testing at the UW on Monday and Wednesday evenings. Tonight was my last class with a final that culminated in the attendees writing “scripted tests” from vague requirements, running those tests followed by some exploratory testing, and finally reviewing structural coverage effectiveness and defect detection effectiveness of the testing approaches.All in all, it is a fun class.

One lesson in the class focused state-transition testing.  The focus of the lesson is to identify the important “states” the machine is in at a given time, and identify various ways a customer might navigate from one state to another (traversals). In my opinion, one of the easiest ways to present the concept of modeling states is with a setup or installer of a program with a minimum number of configuration variables. I have found that minimizing the configuration variables helps people focus on “state” rather than be distracted by the different configuration settings (combinatorial testing).

After a brief explanation and a demo, I cut them loose on a problem to solve. In this case I used a shareware application called 3rd Plan It. The objective was to identify all possible paths a customer could navigate while setup is running and the important states the machine might be in at any given moment in time during the installation process. The value of a state diagram or map is that it can help us identify paths that we might miss via exploratory testing. Also, by identifying important characteristics of each state we can better assess whether the machine is in the expected state (oracle) as we navigate the various paths towards the objective of our test (in this situation it is the installation of a program).

The state map below illustrates a model of what we might consider the “important states,” and the various ways to traverse or navigate between states. Coming up with a state transition diagram often requires us to explore the feature. But, it also seems to force us to really understand the feature we are exploring at a much deeper level, and consider paths that we might otherwise have missed. A state transition diagram also helps us think about what “state” the machine should be in at any given moment during the setup process (it is also important to understand the “state” within the appropriate context).

This is not to suggest that we need to create a state transition diagram for every feature. That would be silly. But, even if we don’t create a state diagram on paper we certainly create one in our mind. I suspect that most of the time we are purposeful in our testing approach and use heuristic patterns as we design tests (that is of course unless someone is high on PCP or some other hallucinogen and are completely reckless in their test approach and simply bang away indiscriminately in hopes of finding bugs).

Of course, some people found 1 or 2 issues using an exploratory approach without first creating a state diagram. But, after viewing the state diagram those folks were also able to realize paths they had not considered or traversed and were able to find more issues. The single biggest epiphany that most people have after participating in this exercise is how much more they understand the feature they are modeling compared to what they thought they knew.

Then of course, when the setup is littered with problems, the uninstall functionality which is part of the setup engine has its own set of problems as well.

Often, after teaching concepts and demoing examples of situations where various testing techniques might apply I am asked how diligently we should apply these patterns of testing. Of course, functional techniques are only one way to think though a problem. Test techniques are useful to help us systematically analyze a problem, and come up with some model of what we are testing. Models are sometimes useful to help us think of the problem differently, or identify things we might otherwise miss. State transition testing is a useful functional technique that is useful for helping us visualize various ways a customer might use a feature, and in turn come up with various tests and potentially find more issues.