I.M. Testy

In response to my last post, Shrini suggested, “You should probably do a post on types of bugs that unit testing (or developer testing at any level) would’nt catch. That would be a fitting reply to all those who swear by automated unit testing/API Testing.”

I am a big proponent of well-designed automated tests, and I have written a lot about the value of automation as an effective tool in the development process. But, I also know that automated tests find a relatively low number of bugs throughout a product lifecycle. If you think the primary goal of an automated testing effort is to find lots of bugs, or the same types of bugs as a person then you probably don’t know very much about test automation. Personally, I don’t think of automated testing as a proxy for the tester or as a bug finding solution. Sure, some automated tests can help find bugs, but more importantly it is one approach I might use for

• defect prevention (esp. computational logic problems),
• earlier identification of key integration issues,
• potential degradation of critical areas (battery, performance, memory),
• efficient execution of redundant ‘checks’ (if necessary or for confidence)
• more effective/precise ‘oracles’ as compared to humans
• cost reduction in long term sustained engineering

In my new role at Microsoft I am leading a team of great SDETs that tests primarily at the foundation (API) level. Everyday I see first hand the value that automated regression testing at the unit and API levels of testing provides to the overall production lifecycle (because we build everyday). While the tests we run ultimately affects the customer’s experience, it also helps reduce our overall production costs and drives certain aspects of ‘quality’ upstream. This is especially important in large scale, enterprise systems where a build breaks or integration failures can be costly and lead to unnecessary delays.

But, this is just one level of testing, and I realize that most testers are testing at the ‘system’ level of testing and rely heavily on GUI automated tests. I have never been a big fan of GUI automated tests; especially GUI regression testing. This is not to suggest that all GUI automation is bad, and there are several situations where GUI automated tests can be especially valuable to a test team. However, there are a few situations where I think automating tests that manipulate the GUI is mostly a complete waste of time such as attempting to emulate the behavior of a customer “scenario,” or trying to verify the ‘correctness’ of what the customer “sees” (e.g. visual verification).

I once told a colleague that the computer is really bad at emulating “me.” I said, “for example, sometimes when I type I hit the wrong key, or I lay my finger on a key for too long and that stupid sticky keys message box appears, and sometimes my hand position on my laptop causes my insertion point to jump to some random point in the text body and I have to reset it to the correct location. You can’t automate the unpredictability of me typing!” He said, “Sure I can!” I said, “Ok..I know that we can automate randomness or errant behaviors to some extent, but why would we?”

I sometimes think that in our zeal to “automate everything” we forget that our products often get a lot of “face time” through self-hosting, product partners, beta releases, and other strategies that are intended to get feedback on unanticipated or escaped functional issues, behavioral issues in how people might use the features in different ways (scenarios), and of course the “look” or visual anomalies that might occur while using the product.

As an example, the other day I was searching for a new program for my students to practice their GUI automation skills (yes, while I generally dislike GUI automation it is still a good skill to have). I came across text editor application called XINT. Within minutes of downloading the application and exploring the features I found a bug with the feature that inserts a URL into the text body.

As a simple example I was going to show how we can automatically go through the menu structures to make sure there are no changes, and that the menu items trigger the appropriate events (e.g. displaying a dialog). I was going to develop an automated test demo that systematically marched through the menu structure and validated the expected event triggered by that menu item. An automated GUI test such as this could provide a high level ‘check’ much more quickly than could be performed by a tester. Also, it automates a redundant ‘check’ of the application under test that we might want to perform on each new build to potentially ‘look’ for changes. I wouldn’t expect this automated test to find a lot of bugs, but it would clue us in very quickly to any changes in the menu structure and any anomalies with basic functional expectations triggered by those menu items. Quick and simple.

So, we get to this menu item and programmatically simulating the menu item “click” via a SendMessage() call to the appropriate menu item the ‘correct’ dialog appeared. OK so far. We are not testing the “insert a URL” functionality; my high level ‘check’ is simply checking that the correct event occurred (in this case a dialog appeared). So, now I am going to send a message to “click” the Cancel button and my expected result is for this dialog to go away and focus will return back to the application under test (XINT). But, in this case the URL insertion dialog is repainted with the sample text removed, and a new label. (It gets even better because clicking Cancel again forces the sample text into the text edit control. You have no choice at this point…you selected to insert a URL…so you are going to get whether you like it or not!) But, there is a bug, and my automation could have found it.

But, a little more exploration and I discover another anomaly that almost defies logic, and that automation would certainly not have found. When I pressed the ALT key, I noticed something odd. The fricking Cancel button disappears! However, as far as the ‘system’ is concerned the handle to this control is still there and I can programmatically send a message to that button control. But, to me the user…it’s gone!

This is just one example of the types of issues that automation is not especially suited for, and even trying to automate a test for these types of issues is not just an effort in futility, I would say it is damn near insanity. Of course there are other types of issues that automation is not especially efficient or effective in detecting such as ease of use, consistency in layout or behavior, general “look and feel,” and most importantly customer scenarios or user stories.

Bottom line, use automation for things that computers are really good at such as computational logic and redundant ‘checks’ that we might want to do after each new build. And, use humans to test for the things that humans are really good at which often happen to be the things that will delight your customer if you get it right, or the things that will piss them off if you screw it up!

…developers basically suck at testing and relying on developers to test your product will cost you money! Hey you pointy haired managers who think you can save money by cutting testing costs…are you listening?

Don’t get me wrong, I don’t think developers suck in their jobs; in fact, I know many really great developers (but, I also know and have seen the work of some hacks who call themselves developers). Great developers are usually pretty good at writing unit tests that test discrete functions. Some developers are even good at writing higher component or API level tests. Unit testing and API testing are valuable approaches to help identify certain types of problems in a new build after refactoring or bug fixing.

I don’t distrust developers; I think similar to testers they are concerned about the quality of their code and of the product they are working on. But, testers have a different mindset as compared to developers, and we have different perspectives. Some testers may write, or partner with developers for component level or API tests, but most testers generally focus on integration and system levels of testing. Using various approaches in our craft we often expose functionality issues as well as behavioral or usability issues that might adversely your customers.

For example, the other day I came across a website to make a travel reservation. When I get to the payment page I realize that I can’t complete it because it is missing 3 months.

“Hmm…this is odd,” I thought to myself. I refreshed the page thinking there might have been a glitch in getting the list of months. Ultimately, I had to call the travel company to make reservations.

At first they didn’t believe me, then they confessed they just commissioned this new website. So, I told them that after they fire the developer and hire some testers for the next go round they should also figure out why “test” is listed as a State.

Or, among other things, why this dropdown list contains seeming unrelated options, why some options are all upper case, and why one city name (scottsdale) is not capitalized. And of course our all time novice faux pas…”abc_test.”

These are just of few of the examples of the lack of perceived quality of this site. And the company really expects customers to enter personal data into this site; especially credit card information?

Now, of course testing doesn’t guarantee the absence of bugs, but I am pretty confident that most testers would have easily found these before staining this company’s reputation.

So, the next time your company thinks it can cut testing costs by having developers do the testing…point them here and say…I sure hope the developers you hire do a better job than this.

I am almost through my first week in my new job here at Microsoft and trying to get my head wrapped around everything. It is really exciting to work on new technologies and in a feature area that helps users stay connected with friends and family in amazing ways. For now, let’s just say that I feel like I am treading water which is not unusual when changing jobs within Microsoft. But, of course, that is a good thing. If I felt like I could’ve easily stepped into this job I would not feel challenged, I would not open my mind to learn new things, and I would likely be bored after a few months. Yes…I am a bit overwhelmed in a very exciting way!

Of course, I am still teaching classes at the University of Washington Extension, and am currently teaching a class in test automation design. A few sessions ago we hit an issue with some canned examples failing to run properly on 64-bit operating systems. This problem appeared for 2 reasons:

• I developed the lab examples 2 years ago before I upgraded my home system to a 64-bit machine and have only ran them in the classroom environment since
• I hard-coded the Program Files folder environment path to C:\Program Files.

Of course, it is a bit embarrassing as an teacher when your samples do not run (it’s even worse when they don’t even build, but that is a different story). In a training situation it is often easier to control the environment to minimize problems and allow learners to (initially) focus on the main topic or skill being presented. But, the downside is that many of us don’t work in ‘controlled’ environments in the real world. And while I am adamantly opposed to hard-coded strings in code, I find that sometimes I ignore my own advice in samples or demo code (and it often comes back to haunt me).

The specific problem was that on a 64-bit operating system the application is installed to the Program Files (x86) folder instead of simply the Program Files folder and the application failed to launch when we attempted to start the process. The solution is actually quite simple in this case because we can actually call the ProgramFiles member of the Environment.SpecialFolder enumeration. For example, a hard-coded path on a default installation

string autPath = @"c:\program files\[autFolder]\[autName.exe]";

failed on a 64-bit system, so a better approach to detect the ‘correct’ program files directory might be

string autDefaultFolderName = "[aut_folder_name]";
string autExecutableName = "[filename.exe]";
string autPath = System.IO.Path.Combine(
  Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
   autDefaultFolderName,
   autExecutableName);

The ProgramFiles member of the Environment.SpecialFolder will return Program File on a 32-bit machine, and Program Files (x86) on a 64-bit machine.Of course, this works fine for the default installation folder, but if for some reason you need to run your automation on installations of an application under test in a non-default directory then you would need to qualify the path in other ways.

Also, this is just one difference between a 32 and 64-bit operating system environment, but there are other situations where we want to detect bit-ness so we can write a single test rather than one test for 32 bit operating system environments and another for 64-bit operating system environments. Fortunately, the .NET 4.0 framework makes it really easy to detect whether a machine is 64-bit or not with the Environment.Is64BitOperatingSystem property. For example, you can control flow for 64 vs 32 bit with a simple decision such as:

if (Environment.Is64BitOperatingSystem)
{
  // TODO 64-bit stuff
}
else
{
  // TODO 32-bit stuff
}

But, if you are running the .NET 3.5 framework or earlier it becomes a bit more complicated to detect 64-bit operating systems. The following method can help determine whether or not your operating system environment is 64-bit

public static bool Is64BitOperatingSystem()
{
  if (!string.Equals(Environment.GetEnvironmentVariable(
    "PROCESSOR_ARCHITECTURE"), "x86", StringComparison.OrdinalIgnoreCase) ||
    !String.IsNullOrEmpty(Environment.GetEnvironmentVariable(
    "PROCESSOR_ARCHITEW6432")))
    {
      return true;
    }

     return false;
}

There are also Win32 API functions we can P-Invoke such as GetSystemInfo,

So, rather than having separate automated tests for 32 and 64 bit operating systems a better test design approach might be to simply detect the bit-ness and auto-magically set variables or redirect the control flow path through your test for the appropriate operating system environment.