In the previous post I attempted to explain the subtle differences between unit testing and API testing. It should also be noted that testing at the API layer is different than testing through the GUI. API testing is primarily focused on the functionality of the business logic of the software, and not necessarily the behavior or the “look and feel” from the end user customer perspective. In fact, the 1st tier customer of the API tester is the developer who designed and develops the API. The second tier customers are the developers who utilize the APIs in building the user interface or applications that sit on top of the underlying infrastructure of a program. And finally, API testers must also consider the various end-to-end scenarios of the end user customers at the integration level of testing (without a GUI).

This post will discuss why API testing is an important activity in the complete software development lifecycle (SDLC). Teams that have multiple developers and a continuous integration (CI) build process can greatly benefit from API testing. Key benefits of API testing include:

• Reduced testing costs
• Improved productivity
• Higher functional (business logic) quality   

Reduced Testing Costs

It shouldn’t be a surprise to anyone that finding most types of functional bugs early in the SDLC is more cost effective. The primary goal of unit testing, and component/integration levels of testing (API testing) is to flush out as many functional issues in the business logic layer of a software program as early as possible in the SDLC. Driving functional quality upstream not only reduces production costs, but can also reduce testing costs.

API testing can reduce the overall cost of test automation.  Automated API tests are based on the API’s interface. So, once the API interface is defined testers can begin to design and develop automated tests. Having a battery of automated tests ready as the functional APIs come on-line pushes testing upstream in parallel with development rather than later in the SDLC. This also enables earlier tester engagement and closer collaboration between testers and developers.

Also, since API interfaces are generally very stable, so automated API tests are less impacted by changes as compared to GUI based automated tests. Many testers are familiar with the constant upkeep and maintenance typically associated with GUI based automated tests. The constant massaging of GUI automation is often a huge cost in a test automation effort and a contributing factor to why so many automation projects fail. Automated API tests in general require a lot less maintenance unless there is a fundamental change in the underlying infrastructure or design of the program.

Another significant way that API testing can reduce testing costs is by refocusing testing. Many test strategies rely  heavily on finding functional bugs typically using exploratory type testing through the GUI. But, most software produced today is developed in “layers” (see Testing in Layers). A more robust test strategy should focus the bulk of functional testing at the API layer that contains the “business logic” of the program. Of course some functional issues will still be found while testing through the GUI, but the focus of testing at the GUI layer should be on  behavioral testing. A test strategy that provides a multi-tiered approach is more effective than the typical approach of throwing a bunch of bodies to bang on the GUI in an attempt to beat out the bugs. A multi-tiered test strategy may even reduce the total testing time by reducing the need to spend long cycles trying to uncover a lot of functional bugs through the GUI.

Improved Productivity

There are different ways to evaluate productivity, but certainly one way is to ensure production keeps moving forward. Continuous integration is a keystone of Agile development projects, and at Microsoft this means daily builds. If the build breaks, production grinds to a virtual halt and forward momentum is blocked until the issue is fixed. A build break negatively impacts the productivity of the entire team. A suite of low level integration tests can help identify potential build breaks especially involving dependent modules before new fixes or features are merged into a higher level branch.

API testing can also improve productivity of testing. For example, structural testing is a white box test approach intended to test the structure or flow of a program. If increased levels of code coverage is an important goal, then the most efficient way to improve structural coverage is to identify untested code paths and design and develop API level tests that will tactically target untested code.

Perhaps the most significant improvement to productivity is gained through teamwork. Building and releasing great software products require a team effort. A team of people working closely together. Gone are the days of the adversarial relationship between developers and testers. Changes in technologies, changes in customer demands, and changes in how we build software require close collaboration between developers and testers, and testers being actively engaged throughout the SDLC and not just at the beginning (picking apart a spec), or at the end (banging out  bugs via the GUI pretending to mimic a ‘customer’). A team focused on delivering high quality can greatly add to a team’s overall productivity.

Higher Functional Quality

One of the advantages and also disadvantages of testing at the API layer is that you can test the API in ways that are different then how the GUI interacts with the API. For example, the Morse Code Trainer has an interface for the methods that parse the dots and dashes and plays a system beep of 1 unit duration for each dot in the stream, and a system beep for 3 units of duration for each dash in the stream. The duration of a unit is based on the WordsPerMinute property value.

   1:    interface ISoundGenerator

   2:    {

   3:      void PlayMorseCode(string morseCodeString);

   4:   

   5:      int WordsPerMinute { get; set; }

   6:    }

Testing this property at the API level we could “set” a negative integer value to make sure nothing really bad happens. But, a well-designed GUI would never accept an integer value less than 1 (which is painfully slow) nor above 150 (which is ridiculously high). A better design might be to use a drop-down list of values ranging from 5 (the minimum requirement for a basic license) to 20 words per minute (required for the highest level amateur radio operator license). Of course, it may be possible to find functional anomalies while API testing that could not be found via testing through the GUI. But, the important thing an API tester must consider is how a bug found at the component or integration levels of testing adversely affects a scenario, or the customer.

API testers work alongside of the developers. An API tester may also provide input into the initial API design, engage in code reviews before check-ins, and of course write automated tests to test the API (component level) and APIs in end-to-end scenarios (integration level). Having testers engage with developers early and throughout the SDLC helps ensure team work and instills the idea that quality is a collaborative effort.

Application Programming Interface (API)

The Microsoft Press Computer Dictionary defines API as “A set of routines used by an application program to direct the performance of procedures by the computer’s operating system.” So, referring to the abstract levels of testing, an API can be a unit, but is more likely a component because it is usually “an integrated aggregate of one or more units.”

An API provides value to both the developer and to the customer. For example, an API:

• provides developers with common reusable functions so they don’t have to rewrite common routines from scratch
• provides a level of abstraction between the application and lower level ‘privileged’ functions
• ensures any program that uses a given API will have identical behavior/functionality (for example, many Windows programs use a common file dialog such as IFileSaveDialog API to allow customers to save files in a consistent manner)

Essentially, an API contains the core functionality of a program, or the business logic as some people refer to it. Customers don’t interact with API’s directly. Customers interact with software via the Graphical User Interface (GUI) which in turn interacts with an abstraction layer (e.g. controller in the MVC design pattern) that interacts with APIs exposed via Interfaces.

Testing an API as a Black Box

Some people assume that API testing is a ‘white-box’ testing activity in which the tester has access to the product source code. But in reality, API testing is truly black-box testing in the truest sense of the testing approach. API testers make no assumptions about how the functionality is implemented, and are not limited by constraints or distracting behaviors of a graphical user interface.

As an example, let’s use a program I developed called Morse Code Trainer. As a boy I was really into electronics (HeathKit projects were routinely on my Christmas list), and in order to pursue my amateur radio license I had to learn Morse code, or CW for short. Although Morse code is not required any longer to get a HAM operators license I think it is not hard to learn (memorize) about 55 sequences of dits and dahs, and in my opinion learning additional languages is good for the brain.

A core bit of functionality in this program is to convert a string of characters (a sentence) to the dits (represented as a period character “.”) and dahs (represented as a dash character “-“). The API to do this bit of magic is:

        string AlphaNumericCharacterStringToMorseCodeString(string input)

and is exposed to the developer who will code the UI and controller via the IMorseCodeEncoder interface.

        interface IMorseCodeEncoder
        {
            string AlphaNumericCharacterStringToMorseCodeString(string input);

           string MorseCodeStringToAlphaNumericCharacterString(string input);
        }

Notice that we don’t see any of the underlying code of how this method actually does its magic.

Let’s assume the developer didn’t do any unit testing and simply threw the code over the proverbial wall for testers to beat on. Since the tester (me) knows the developer (me) didn’t do any unit testing of any of the private methods the API under test relies on, the API tester (me) writes a simple test just to see if this code “works” like the developer (me) assured the tester (me) that it would. The most basic API test looks very similar to the unit test illustrated below, and in fact it this is a unit test the developer should write and execute before chucking a program at testers to bang on. A proper API test would call the method under test from the dynamic link library (DLL), and include initialization, clean-up, utilize the proper test design, have a robust oracle,  and of course have no hard-coded strings.

   1:      [TestMethod()]

   2:      [DeploymentItem("Morse Code Trainer.exe")]

   3:      public void GetMorseCodeStreamTest()

   4:      {

   5:        try

   6:        {

   7:          MorseCodeEncoder_Accessor target = new MorseCodeEncoder_Accessor();

   8:          string input = "A QUICK TEST";

   9:          string expected = ".-  --.- ..- .. -.-. -.-  - . ... -";

  10:          string actual;

  11:          actual = target.GetMorseCodeStream(input);

  12:          Assert.AreEqual(expected, actual);

  13:        }

  14:        catch (Exception e)

  15:        {

  16:          Assert.Fail(e.ToString());

  17:        }

  18:      }

Interestingly enough, had the developer (me) ran this unit test the developer would have discovered an unhandled exception. The unit test failed because this API called a method to get a Dictionary in another class and the Dictionary was created from 2 string arrays (an array of alpha-numeric characters, and an array of Morse code sequences). The specific error was a duplicate key/value in the Dictionary; in other words a duplicate entry in the alphaCharacterArray string array was throwing an System.ArgumentException for duplicate keys. But, because the methods in the MorseCodeLibrary class weren’t unit tested the API to encode a string of alpha-numeric characters to Morse code characters failed its basic unit test.

   1:      public Dictionary<string, string> GetAlphaCharacterToMorseCodeDictionary()

   2:      {

   3:        Dictionary<string, string> AlphaToMorseCodeDictionary = new Dictionary<string, string>();

   4:        for (int i = 0; i < this.alphaCharacterArray.Length; i++)

   5:        {

   6:          AlphaToMorseCodeDictionary.Add(this.alphaCharacterArray[i], this.morseCodeArray[i]);

   7:        }

   8:   

   9:        return AlphaToMorseCodeDictionary;

  10:      }

But, it actually gets worse. Another API in another class to decode a string of Morse code to alpha-numeric characters would have failed as well because it used the same faulty string array of data to create a Dictionary calling the public method

        public Dictionary<string, string> GetMorseCodeToAlphaCharacterDictionary().

This is actually a good example of 2 very different bugs with the same root cause. This is also good example of how it is more efficient to find functional bugs at the unit,  or component or integration levels of testing (API testing) as compared to finding this problem via functional testing through the user interface.

Unit vs. API Testing

So, you’re probably asking yourself “if the above example is really an example of a unit test the developers should do before throwing their code at testers, then how does unit testing differ from API testing?” When testing a single API call the most significant difference is in the thoroughness of test coverage. Most unit tests are rather simple things. Unit tests are not very complex; unit tests are not comprehensive in test coverage (although a good suite of unit tests should achieve good structural coverage); and unit tests often rely on simplistic oracles.

API tests by contrast are usually more comprehensive as compared to unit tests. API tests usually include both positive tests (does it do what its supposed to do) as well as negative tests (how well does it handle error conditions). While API tests should strive for a high level of code coverage (structural testing) a more important goal is test coverage. For example API tests of this same method might include a series of data-driven tests that:

• test every known alpha-numeric character defined in Morse code (the population of the variable is small enough to test every element, if the population of a given variable is large then testers should define equivalent partitions and test an adequate number of samples from the population for confidence)
• test character casing
• test pangrams, and special signals (e.g. end of message, attention, received, etc)
• test boundary conditons (e.g string max len although 2 billion+ characters seems excessive for a Morse code transmission)
• test strings with invalid or characters that are not defined in Morse code
• test strings with non-ASCII letters that have Morse code encodings (Ä, Á, Å, Ch, É, Ñ, Ö, Ü)
• test performance to provide baseline measures of individual methods

More complex APIs such as this MessageBox.Show method that have several parameters with variable argument values might benefit from additional testing techniques such as combinatorial testing.

Testing API End–To–End Scenarios

Testing a single API is usually considered unit or component level testing in the abstract levels of testing. Some people consider unit and component level testing to be “owned” by the developer. I certainly agree that unit tests must be owned by the developer, and that developers can do a much better job of component level testing. But, I also think this is a key area where API testers can collaborate more closely with developers to increase the effectiveness of the tests, the data used in the test, and even the test design (e.g. data-driven unit testing).

But, I will suggest that the integration level of testing or “testing done to show that even though the components were individually satisfactory, as demonstrated by successful passage of component tests, the combination of components are incorrect or inconsistent“ is the domain of the API tester. Software applications are complex beasts that often rely on sequences of API calls interacting with databases, cloud services, or other background workers. So, although API testing rarely involves testing through the GUI, API testers must also understand how the various APIs will be used to effect various customer scenarios. The only difference is that API testers emulate these scenarios without navigating a graphical user interface.

For example, one scenario is to convert a string of text into dits and dahs and use the system’s beep to “play” the Morse code sequence over the computer’s speaker. So, this program contains a class to convert the sequences of dits and dahs into sound; the SoundGenerator class. The interface for the sound functions includes a getter and setter, and the PlayCharacterCode API.

   1:    interface ISoundGenerator

   2:    {

   3:      void PlayMorseCode(string morseCodeString);

   4:   

   5:      int WordsPerMinute { get; set; }

   6:    }

So, although we don’t know exactly yet how the developer and GUI designer will implement the GUI for this program, we can still create a test that inputs a string of alphanumeric characters, encodes the alphanumeric string into a Morse code encoded string, and then passes the string of Morse code dits and dahs as an argument to the PlayMorseCode method. This is a rather simple example of an end-to-end scenario. In more complex application the API functions/methods would likely be compiled in one or more dynamic link libraries (DLLS), rely on mocks, fake servers, and possibly other emulators. Of course, the oracles for this type of API testing is also more complex and generally involves checking multiple outcomes or states.

API testing focuses on an application’s functional capabilities, whereas testing through a GUI should focus primarily on behavior, usefulness and general ‘likeability.’

Now, I don’t read Arabic, but I am pretty good at noticing globalization bugs when they are staring me right in the face. The “box” character (actually a glyph) in a Unicode string either represents an Unicode code point that is unassigned (it doesn’t have a character associated with that code point value), or the system doesn’t have a font that maps a glyph (the character we see) to that particular Unicode code point. So, curiosity got the better of me, and I decided to investigate a bit. The first thing I did was to right click on the email subject line and paste it into Notepad and notice that the “box” glyph did not appear.

A few years ago I developed a utility for decoding Unicode Strings aptly called “String Decoder” and also wrote a post that discusses the tool. So, I launched String Decoder and copied the Arabic string from Notepad and pasted it into the String Decoder tool.

The first thing I notice when reading through the list of Unicode code point values is the value U+FEFF. Now, I happen to know that this particular value is a byte order mark (BOM). This seems pretty unusual and ask myself how a BOM character could get inserted in a string. So, I look up the character in the Unicode Charts and discover that in the Arabic Presentation Forms-B character set this was a special character for a zero width no-break space that as been deprecated. Ah, so the Unicode BOM code point value appearing in the string is not so magical after all!

Interestingly enough, the U+FEFF character only displays as a “box” glyph in the subject line in the Junk E-mail folder. When I copied the email message from the Junk folder to my Inbox (or other folder) the code point U+FEFF is treated as a zero width non-breaking space character so no box glyph appears. This is due to the fact that when an email gets shunted into the Junk E-mail folder “links and other functionality have been disabled.” In other words, it is plain-text.

I previously also wrote about using “real world” test data for globalization testing, and this is another example of “real-world” data can be useful in testing text inputs and outputs to evaluate how unexpected character code points in a string are parsed or handled. I think this also bolsters the argument to include some amount of test data randomization using tools such as the Babel tool in globalization testing to potentially test for other unexpected characters or sequences of mixed Unicode characters.

If you ask any lead they will likely agree that their success as a lead hinges largely on their team. But, if you ask leads what their first priority is they will likely say shipping a product, or managing testing of some feature area they have been assigned. Yes, ultimately we need to ship a product and do our best to make sure our feature areas are adequately tested in an attempt to improve our customer’s overall experience. There are many ‘managers’ throughout the industry who are good at manipulating ‘resources’ to achieve some desired result or filling in magic numbers on a balanced scorecard. Balanced scorecards provide some value to a business, but sometimes managers lose sight of what is most important and focus on doing mundane things that will twiddle the numbers to make it fit into the scorecard to hype success. But, managing resources to ship a product is different than leading a team of people to achieve, and sometimes exceed goals and visions.

Leadership is much more than management. A successful leader manages projects by articulating a clear vision, guiding people towards achieving goals, and motivates people by helping them grow. When folks ask me what my first priority is as a Test Lead I say it is the people on my team. But, what does that mean?

Open doorways to dreams!

One of my primary responsibilities as a lead is to help the people on my team grow and expand their scope of influence and impact not only on my team, but ultimately within other teams across Microsoft. Of course it is always hard to see someone on our team leave for new opportunities, but good leaders understand the career aspirations of the people on the team and work with them to help them achieve those dreams. Leaders find opportunities that will help people develop skills that will benefit both the project and the person. Leaders should be truly invested and take an active role in helping people on their team grow even if that means the person will eventually leave the team to find new challenges. Managers fear losing the people on their team; leaders nurture people on their teams and open doorways to dreams and new opportunities. Think of it this way, would you rather join a team in which the manager holds on to people until they burn out, or a team in which the leader has a track-record of helping people grow into their next job.

Delegate responsibility not just work!

Like many other leaders, I have many balls to juggle, and I can’t juggle them all alone. So, as leads we must delegate some of the things on our plates. But, delegation is more than assigning tasks to people. Delegation is endorsing people on your team who will represent you have be responsible for driving a project that has a broad scope of impact. Of course, delegation also doesn’t mean just throwing ideas out there and seeing what happens. A leader who delegates work will set clear expectations and realistic goals, coach for success, provide guidance on how to build upon success, and perhaps most importantly empower the person to make decisions on their own. When we delegate we should set people up for success; not throw them into the fire of failure.

Encourage risk and accept failure!

Sometimes when people know that I am an avid sailor they will ask me to teach them to sail. I love sharing knowledge and experiences about things that I am passionate about with people who are interested in learning. Sometimes people are hesitant to do something because they don’t want to break something, or do something wrong. I make it very clear from the start that every inanimate object on the boat is replaceable, and while back-winding a sail means we steered too far into the wind it can always be corrected. I know many ‘captains’ who yell and shout when a line gets twisted and jams in a sheave, or someone accidentally releases the main halyard while under sail. It’s our reactions to such situations that provide a positive learning experience or turn our experience into a day of hell on the water. Leaders encourage people to try new things, innovate, and experiment. Leaders also know that sometimes things might not work out perfectly and should be willing to protect people from harm (either physical harm on the boat, or professional/political harm at work), and rebuild a persons confidence when things don’t work out so well.

The burden of blame!

At the end of the day I can’t point fingers and say “so and so didn’t do such and such,” or “if things weren’t so screwed up to begin with we wouldn’t be in this mess.” As a lead I am accountable. If things go wrong I first look at my own leadership to see if I failed to set clear expectations, or neglected to provide adequate guidance (without hand-holding), or did I “delegate and disappear.” Ultimate the responsibility of achieving my team’s goals and objectives is mine. We succeed as a team, or I fail as an individual.

Attitude adjustment!

I sometimes see managers who are grumpy or apathetic. I sometimes hear managers say, “I don’t like this either, but we have to do it to satisfy some other manager or scorecard criteria.” A good leader understands asks and explains why and how they might provide value to the requestor. I know that my attitude affects the people on my team, and if I appear empathetic towards their ideas then they will likely not share their innovative ideas with me. If I am constantly complaining about something, then my team learns to complain about similar things and we start to look like a bunch of whiners. (Nobody really likes whiners. People might try to appease whiners from time to time, but ultimately they just want the whiners to go away.) Leaders know they are being watched and should always project a positive attitude.

So, after 6 months of being back in the trenches, shipping a product, and facing some tough challenges I will say that I am still loving it!

Another place I have been known to explore from time to time is the Stack Exchange Software Quality Assurance and Testing forum. There are many interesting questions and a great variety of responses that offer a wealth of information or provide different perspectives. Recently a question was posed about how to read in static test data for a specific locale or language. Many regular readers know that I am a strong proponent of pseudo-random test data generation in conjunction with automated testing to increase the variability of test data used in each test iteration and generally improve test coverage. But I also understand the value of static test data in providing a solid baseline, and in some cases enabling access to specific test data in different locales or languages.

For example, suppose I am testing a text editor application and I want to read in a text file in the appropriate language based on the operating system current users locale settings. In this situation, I could save a text file containing strings or sentences for each target language or locale dialect. Each file would get a unique name based on the 3 letter ISO-639-2 language name (the complete list is at http://www.loc.gov/standards/iso639-2/php/code_list.php), prepended it to a common filename that describes the contents and the appropriate extension. For example,

• ENG[TestData].txt would be English
• ZHO[TestData].txt would be Chinese
• DEU[TestData].txt would be German

To get the appropriate text file auto-magically read in to the test at runtime the only thing we would need to do is to get the current user locale using the CultureInfo class Three Letter ISO Language Name property in C#.

   1:              string testDataFileName = "testdata.txt";

   2:   

   3:              CultureInfo ci = CultureInfo.CurrentCulture;

   4:   

   5:              // Path to server location where static files exist 

   6:              string path = Path.GetFullPath(

   7:                  Environment.GetFolderPath(Environment.SpecialFolder.Desktop));

   8:   

   9:              // Read file contents

  10:              using (StreamReader readFile = 

  11:                  new StreamReader(Path.Combine(

  12:                      path, string.Concat(

  13:                      ci.ThreeLetterISOLanguageName, testDataFileName))))

  14:              {

  15:                  //parse test data and do test stuff

  16:              }

Notice we concatenate the filename (and extension) and the 3-letter ISO language name in line 13 and then combine that with the path to the file location and read the file contents using StreamReader.

But, we might need more specialization depending on what we are testing. For example, if we were testing a spell checker for US versus Great Britain (and Canada), or testing simplified Chinese and also traditional Chinese. In this case the ISO 639-2 specification does not delineate between simplified Chinese and traditional Chinese or US English and British English.  In this case we could “make up” a 3-letter designation such as GBR for Great Britain, or CHT for Chinese (traditional).

Or, perhaps a better solution would be to use the Locale Identifiers (LCID) used by Windows to identify specific locales (rather than languages). The solution is identical to the above except instead of calling the ThreeLetterISOLanguageName property we call the LCID property as illustrated below.

   1:              string testDataFileName = "testdata.txt";

   2:   

   3:              CultureInfo ci = CultureInfo.CurrentCulture;

   4:   

   5:              // Path to server location where static files exist 

   6:              string path = Path.GetFullPath(

   7:                  Environment.GetFolderPath(Environment.SpecialFolder.Desktop));

   8:   

   9:              // Read file contents

  10:              using (StreamReader readFile = 

  11:                  new StreamReader(Path.Combine(

  12:                      path, string.Concat(

  13:                      ci.LCID, testDataFileName))))

  14:              {

  15:                  //parse test data and do test stuff

  16:              }

Of course, now we would need to name our static file names with the appropriate LCID decimal number such as

• 1028testdata.txt would be traditional Chinese used in Taiwan, and
• 2052testdata.txt would be for simplified Chinese used in PRC

Personally, I prefer getting the LCID as it provides greater control and more specificity. But the down side of using LCIDs is that if you may end up having multiple files that contain the same contents. For example, although Singapore, Malaysia, and PRC all use simplified Chinese there are 3 different LCIDs.

There are other properties that allow you to get the culture info for the current user in Windows, and the right property to use ultimately depends on your specific needs. But, CultureInfo class members can easily be used to manage localized static data files or even manage control flow through an automated test that has specific dependencies on a language or a locale setting.

In this previous post I explained the business reasons why Microsoft changed the title from STE to SDET. But, for some reason people commonly mistake the title with the role or job function. In the good ol’ days our internal job description for STE at level 59 included ‘must be able to debug other’s code,’ and ‘design automated tests.’ Almost all STEs hired prior to 1995 had coding questions as part of their interview and were expected to grow their ‘technical skills’ throughout their career.  That was the traditional role of the STE.

As I explained in this previous post we established the title of SDET to ensure that testers at a given level in one organization in the company had comparable skills to another tester in a different organization. As part of the title change, the company decided that we needed to reestablish the base skill set of our testers to include ‘technical competence.’ Unfortunately when the career profiles were introduced some managers misinterpreted ‘technical competence’ with raw coding skills and the naive ideology of 100% automation. These same managers now complain their SDETs don’t excel at ‘bug finding’ and customer advocacy.

On my current team, the program managers are big customer advocates. They run their own set of ‘scenarios’ against new builds at least weekly. My feature area is testing private APIs on our platform. Our primary customers are the developers who consume those APIs, but we also must understand how bugs we find via our automated tests might manifest themselves and impact our customers. So, our team spends quite a bit of time also self-hosting, doing exploratory testing, and we even started a new approach that takes customer scenarios to the n-th degree that we call "day in the life" testing to help us better understand how customers might use our product throughout their busy days. Our product has 93% customer satisfaction.

So, if its true that the SDETs on some teams aren’t finding bugs and lack customer focus (and I suspect it is for some teams) then they hired the wrong people onto their test team. If SDETs don’t balance their technical competence with customer empathy then we have a problem; and I will say it is likely a management problem.

The testing profession is diverse and requires people to perform different roles or job functions during the development process and over the course of their career. Microsoft didn’t eradicate the STE “role” we simply changed the title of the people we hire in our testing “roles” and reestablished the traditional expectations of people in that role.

Differentiating between STE and SDET in our industry seems nonsensical to me, and I also think this false differentiation ultimately limits our potential to positively impact our customer’s experience and advance the profession. Testers today face many challenges, and hiring great testers (regardless of the job title) is about finding people who not only have a passion and drive to help improve our customer’s experience and satisfaction, but can also solve tough technical challenges to advance the craft and help improve the company’s business.

The weekend hockey game was a good break for me. You see, we have been in ship mode for our Mango release on the Windows Phone. The hockey game was a good outlet for some pent up frustration. Every D-man had at least 2 shots on goal, and I blocked a couple of shots; one off the mask and one off my inner thigh of course where there are no pads, and yes it left a pretty good bruise. But, as they say, “pain is temporary; a win is forever.”

Seemingly against the odds, we ended up winning the hockey game 5 to 1.

Ship mode often times gets a little crazy. Second guessing takes on a whole new meaning. “Did you test this"?” “What about that?” “I have a situation when I do such and such, and the sun came out (remember we’re in Seattle) something bad happened. Have you seen this before?” Some people run around looking for fires, others are trying to start them.

As I get older I have learned not to react to fires as I did in my younger days. I have learned that sometimes fires aren’t really fires at all; it’s just a spark that someone is recklessly trying to fan into a flame. Sometimes there are fires that burn themselves out, but you just have to manage them in a control burn. And then there are the fires that have to be dealt with. Dealing with fires late in the product cycle is not fun for on the team. But, a team of people are responsible for doing just that and it is seldom easy; and it happens in the ship room.

Our ship room looks at a lot of data every day throughout the product cycle to help us manage our release schedule and stay focused. In ship mode, data is scrutinized even closer and every bug goes under the microscope. Managers must now work together to make some hard decisions about whether to take a fix. There is often intense discussion, but you will never hear anyone play the consultant card saying “it depends.” These guys in ship room have been in the game a long time, they know the risks and they know the business. Of course they know “it depends.” They don’t want a bunch of hand waving and bloviating, they need facts to make hard decisions. If you say an issue going to adversely affect customers you better be able to explain how customers are impacted, how many customers are impacted, how customers get into that predicament, and know if there is a potential work-around.

In the end, a team of senior managers must make hard decisions about what issues to punt and which to fix based on the information that is presented. This never easy at anytime during the product cycle, but in ship mode each issue is carefully investigated down to root cause, the fix is understood, and the impact of fix and testing considerations are well defined before the final decision is made. Of course, many products are schedule driven, but at the forefront of every decision in our ship room is customer impact. Perhaps that is why customer satisfaction for Windows Phone 7 is at 93%, and why I am glad to be on a team that works hard to do the right thing for our customers.

“A unit is the smallest testable piece of software, by which I mean it can be compiled or assembled, linked, loaded, and put under the control of a test harness or driver.”

“A component is an integrated aggregate of one or more units. A unit is a component a component with subroutines it calls is a component, etc. By this (recursive) definition, a component can be anything from a unit to an entire system.”

“Unit testing is the testing we do to show that the unit does not satisfy its functional specification and/or that its implemented structure does not match the intended design structure.”

It’s interesting to note Beizer’s explanation of the purpose of unit testing. Perhaps this is the inspiration behind TDD concepts rediscovered 10 years later. It also is counter to the typical ideology that unit tests are intended to “prove it works.”

“Integration is a process by which components are aggregated to create larger components.”

“Integration testing is testing done to show that even though the components were individually satisfactory, as demonstrated by successful passage of component tests, the combination of components are incorrect or inconsistent. “Integration testing is specifically aimed at exposing the problems that arise from the combination of components.”

“A system is a big component.”

“System testing is aimed at revealing bugs that cannot be attributed to components as such, to the inconsistencies between components, or to the planned interactions of components and other objects. “System testing concerns issues and behaviors that can only be exposed by testing the entire integrated system or a major part of it.”

I often us the following visual model to illustrate the concept of ‘level of testing.’ This model is intended to show how each level builds upon the previous level, and also shows the increasing scope of each level of testing.

The problem with this model is that if we don’t understand the internal structure of software such as methods (or functions), classes, APIs or interfaces,  forms or other UI elements, etc., it is difficult to differentiate between the different levels other than system (the entire integrated system, or product), and the other levels.

One way to explain this model is that unit testing level tests individual methods or functions in a class. Component level testing tests a public methods that call one or more methods in a class. Integration tests are targeted at testing individual APIs or combinations of APIs treating the APIs as black boxes but usually without traversing through a user interface. And system testing tests both functional aspects and behavioral aspects of the entire product’s components together in a single build.  Obviously, the scope of testing at the system level of testing is very large and includes functional testing of computational logic, non-functional testing such as performance, security, etc, and behavioral testing such as usability, look and feel, etc.

Another problem with this model is that it may appear that by focusing testing efforts at the system level (e.g. testing each build through a user interface) we would have greater coverage and implicitly cover the ‘levels’ below the system. Unfortunately this is not always the case because

• the user interface can mask or hide some functional bugs in public APIs that are called by other developers
• the system is large and complex and we may miss underlying functional bugs

As Beizer indicated, each ‘level’ of testing has different objectives and can help identify certain types of bugs more efficiently as compared to the other ‘levels of testing.’ In our experience at Microsoft, we have learned that hiring massive numbers of people to test at the system level with little or no unit/component or integration testing does not necessarily result in higher “quality,” and may cost more in maintenance and support due to undetected functional issues.

Another model that I also like to show comes from Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and Janet Gregory. In their chapter on automation they illustrate the “test automation pyramid” with 3 layers of automated tests.

This model also shows how automated tests benefit from the underlying automation, and each layer is targeted towards helping the tester (and developer) identify different types of issues during the SDLC. I also like this example because it shows where automated tests are most effective or provide the greatest value to developers and testers.

Ideally there should be heavy emphasis on unit/component level testing, additional investment in API level testing, and as Crispin and Gregory state, “The top tier represents what should be the smallest automation effort, because the tests generally provide the lowest ROI.” While ROI means different things to different people, I would generally agree that GUI automated tests tend to be less reliable and require much more maintenance as compared to other levels of automation.

There are also some illustrations of testing levels that show a sequential progression from unit testing to component testing to integration testing and finally to system testing. In my opinion these don’t provide much value to anyone other than process wonks who are only capable of linear thought. There are also other models of ‘levels of testing’ that folks have devised they use usually to separate unique classes of system testing. Models are abstract concepts that can be used to help explain complex systems, and sometimes a little more detail in the model helps people understand the system.

But, I guess that is both the benefit and the drawback of models. Models can help explain complex concepts, but they can also be misused when single-minded individuals attempt to create rigid linear processes from an abstract model, or assume how someone actually implements a model is representative of the abstract concepts that someone attempted to present in a model.

Last week on my way to speak at TestNet in the Netherlands I posted about “bad automation.” More to the point, it was about how we sometimes cobble UI automated tests that attempt to mimic some rudimentary set of sequential actions that we think a customer might execute and pass it off as an automated functional test. Some of these types of “automated tests” might make sense in limited situations, perhaps similar to how unit tests give us a very basic level of confidence after code churn. But, I think one of the reasons that test automation generally gets a bad rap is because of overly simplistic scripts that mindlessly perform the same thing over and over again and provides little real value to the team beyond emotional ‘feel-good’ or ‘frustration’ that often accompanies automation projects. So, building upon the example from last week, let’s look at how we might consider a few simple design principles and craft a more robust functional automated test that increases test coverage, helps reduce overall risk, and may potentially expose unexpected errors.

The purpose of the test
Think about abstract big picture rather than a myopic singular purpose

In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.

The purpose of the this test is not to just to verify “Boob” displays in the message box. Even if this were a regression test we could manually test the word “Boob” if necessary, but in the bigger picture the purpose is to provide confidence that any Unicode string (with a max length of 25 characters) is rendered correctly in the text of the message box. Of course, we can’t test every possible permutation of Unicode characters in strings with varying lengths between 0 and 25 characters. But, we can design a functional test that passes in multiple Unicode string variant to increase overall test coverage and still achieve our specific purpose without hard-coded values.

As Gaurav Pandey suggested in the comments of the previous post one way to increase test data through put is to use a data-driven automation approach. Data-driven automation is certainly one way to increase the breadth of test data in an automated test especially if the test data contains historical failure indicators (test data that revealed problems in the past in this given context), and ‘customer-like’ data. The approach I took in this example is to use a random string generator (Babel). Each time we iterate through the for loop line 66 calls the GetRandomTestDataVariant() method to generate a new random Unicode string using the Babel library. This random string variable is also used as the oracle for this test.

Automation perspective
Think about programming the computer rather than trying to program human interactions

In the previous example we used Sendkeys method to navigate the user interface using key mnemonics, enter text into the text box and manipulate the button controls. That approach to automating the user interface is often unreliable because Sendkeys send the message to whichever window has focus. Another problem with this approach is that key mnemonics could change leading to additional maintenance costs, and test failures on localized versions of the software where key mnemonics are different from the English language version.

In this example, instead of using Sendkeys methods we used UI Automation to get the automation elements to access to user interface elements on the desktop programmatically. This can also be accomplished by p-invoking various native functions in the user32.dll library. For example, the NativeMethod class in line 21 p-invokes the GetForegroundWindow() function and we use this in line 227 of our oracle to get the window handle of the message box. Line 51 gets the automation element objects on the main form that we will use to enter text into the name textbox and emulate clicking the button controls.

But sometimes we can accomplish tasks in a UI automated test without manipulating UI elements programmatically. For example, suppose we want to change our user locale settings during a test. From the customer perspective we would launch the Regional and Language control panel applet in the control panel and change a desired settings on the appropriate tab. However, in a previous post last year I explained how we could programmatically manipulate user locale settings using the GlobalTester library without trying to manipulate UI elements.

Just about any interaction a user can do via the user interface we can programmatically emulate or manipulate the system below the UI via Windows APIs. The point here is that when we design our automated tests we should harness the power of the computer and think about what the computer is capable of doing, and not limit our automated test design to a set of simplistic sequential steps that an end user might perform in a scripted end-to-end scenario.

Predict programmatic unpredictability
Think about what can go wrong and try to deal with it gracefully

Poorly designed UI automation fails exactly 2.5 seconds after someone stops watching it run. When we launch an application we can visually ‘see’ when it is ‘ready’ to interact with. If an application launches in .25 seconds on one machine, but then takes 1 second on a different machine the test might fail due to a race condition in the automated test. Synchronization issues between the automated test and the application under test are a frequent problem with UI automation.

Polling loops are one way to help synchronize the automated test with the application under test to ensure that automation elements or windows are in the proper state before executing the next statement in the program. For example, the polling loop in the LaunchApplication() method starting at line 115 ‘looks’ for the appropriately ‘named’ form on the windows desktop for up to 5 seconds. Another polling loop in the oracle on line 225 looks for the Message box window to ensure it is instantiated before trying to get the the message box text.

Of course some people might argue that the example code in the previous post is more simple, and is only about 50 lines of code or so. But, of course, short and simple often only provides simplistic value. Many test consultant now agree that testers require some degree of proficiency in a programming language in order to craft well-designed automated tests. Record/playback script-lets and rudimentary scripted tests can provide some value in limited situations. But, more robust automation can help reduce overall costs, improve confidence by efficiently increasing test coverage, and can even expose unexpected anomalies that other approaches may or may not discover.

The image to the right illustrates 25 randomly generated strings in one test pass. Although this particular automated test does not use ‘real world’ data, it significantly increases the use of Unicode characters across the spectrum of possibilities, and provides greater confidence that any Unicode character or combination of characters would not cause an error. Also, it provides greater breadth of coverage of test data more efficiently (time & effort) as compared to manual testing.

Below exemplifies some concepts discussed above, and is one possible solution (that can be improved…such as the logging). The wonderful thing about coding automated tests is that it can provide another perspective that creative testers can use to help them craft tests that can add value to the project.

   1:  namespace HelloTest

   2:  {

   3:    using System;

   4:    using System.Diagnostics;

   5:    using System.IO;

   6:    using System.Runtime.InteropServices;

   7:    using System.Windows.Automation;

   8:    using TestingMentor.TestTool.Babel;

   9:  

  10:    internal class Constant

  11:    {

  12:      internal const string AutName = "hello.exe";

  13:      internal const string AutFormTitle = "HelloForm";

  14:      internal const string AutNameTextBox = "textboxFirstName";

  15:      internal const string AutOKButton = "buttonOK";

  16:      internal const string AutClearButton = "buttonClear";

  17:      internal const int FirstNameTextboxLength = 25;

  18:      internal const int MaxPollCount = 50;

  19:    }

  20:  

  21:    public class NativeMethod

  22:    {

  23:      [DllImport("user32.dll")]

  24:      public static extern IntPtr GetForegroundWindow();

  25:    }

  26:  

  27:    class SampleHelloTest

  28:    {

  29:      static void Main(string[] args)

  30:      {

  31:        try

  32:        {

  33:          Stopwatch timer = new Stopwatch();

  34:          timer.Start();

  35:  

  36:          int testIterationCount = 25;

  37:  

  38:          // Declare a new process and automation elements

  39:          Process autProcess = new Process();

  40:          AutomationElement desktopPath = AutomationElement.RootElement;

  41:          AutomationElement autForm = null;

  42:          AutomationElement nameTextbox;

  43:          AutomationElement clearButton;

  44:          AutomationElement okButton;

  45:  

  46:          // Launch the AUT

  47:          autProcess = LaunchApplication(

  48:            Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),

  49:            Constant.AutName),

  50:            desktopPath,

  51:            Constant.MaxPollCount,

  52:            ref autForm);

  53:  

  54:          // Get the UI automation element object on main form

  55:          GetUIAutomationElements(autForm, out nameTextbox, out okButton, out clearButton);

  56:  

  57:          // Verify the textbox control is 'visible'

  58:          if (nameTextbox != null &&

  59:            (bool)nameTextbox.GetCurrentPropertyValue(AutomationElement.IsEnabledProperty))

  60:          {

  61:            // iterate through a specified number of test data variants

  62:            for (int testCount = 0; testCount < testIterationCount; testCount++)

  63:            {

  64:  

  65:              int seedValue;

  66:              string testData = GetRandomTestDataVariant(out seedValue);

  67:  

  68:              // Put test string into textbox

  69:              SetTextboxText(nameTextbox, testData);

  70:  

  71:              // Emulate clicking the OK button to instantiate the message box

  72:              PushButton(okButton);

  73:  

  74:              // Oracle to compare actual vs expected result and log results/important information

  75:              string actualResult = string.Empty;

  76:              if (ValidateMessageBoxStringOracle(testData, Constant.MaxPollCount, out actualResult))

  77:              {

  78:                // Log Test Results as appropriate 

  79:                Console.WriteLine(testData);

  80:              }

  81:              else

  82:              {

  83:                Console.BackgroundColor = ConsoleColor.Red;

  84:                Console.WriteLine(

  85:                  "FAIL: Expected Result = {0}, Actual Result = {1}, Seed Value = {2}",

  86:                  testData, actualResult, seedValue);

  87:              }

  88:  

  89:              // Emulate clicking the clear button

  90:              PushButton(clearButton);

  91:            }

  92:          }

  93:          else

  94:          {

  95:            Console.WriteLine("TEST FAILURE: Textbox control not found");

  96:          }

  97:  

  98:          // Clean up the test environment 

  99:          CloseAutProcess(autProcess);

 100:  

 101:          // Log total elapsed time and any additional important info

 102:          timer.Stop();

 103:          Console.WriteLine("Total elapsed time: {0} seconds", timer.Elapsed.TotalSeconds);

 104:        }

 105:        catch (Exception e)

 106:        {

 107:          Console.WriteLine(e.ToString());

 108:        }

 109:      }

 110:  

 111:      #region HELPER METHODS

 112:      /// <summary>

 113:      /// Launches AUT process; or throws exception if process fails to launch

 114:      /// </summary>

 115:      internal static Process LaunchApplication(

 116:        string autPathAndFilename,

 117:        AutomationElement desktop,

 118:        int maxPollCount,

 119:        ref AutomationElement myAutForm)

 120:      {

 121:        Process myProc = new Process();

 122:        myProc.StartInfo.FileName = autPathAndFilename;

 123:        if (myProc.Start())

 124:        {

 125:          // Find the MyFontForm window

 126:          int pollCount = 0;

 127:          do

 128:          {

 129:            myAutForm = desktop.FindFirst(

 130:              TreeScope.Descendants, new PropertyCondition(

 131:                AutomationElement.AutomationIdProperty, Constant.AutFormTitle));

 132:            pollCount++;

 133:            System.Threading.Thread.Sleep(100);

 134:          }

 135:          while (myAutForm == null && pollCount < maxPollCount);

 136:  

 137:          if (myAutForm == null)

 138:          {

 139:            throw new ArgumentNullException("Failed to find AUT form");

 140:          }

 141:        }

 142:        return myProc;

 143:      }

 144:  

 145:      /// <summary>

 146:      /// Get UI automation element objects on main form by property name

 147:      /// </summary>

 148:      private static void GetUIAutomationElements(

 149:        AutomationElement autFormName,

 150:        out AutomationElement nameTextbox,

 151:        out AutomationElement okButton,

 152:        out AutomationElement clearButton)

 153:      {

 154:        nameTextbox = autFormName.FindFirst(

 155:          TreeScope.Descendants, new PropertyCondition(

 156:            AutomationElement.AutomationIdProperty, Constant.AutNameTextBox));

 157:        okButton = autFormName.FindFirst(

 158:          TreeScope.Descendants, new PropertyCondition(

 159:            AutomationElement.AutomationIdProperty, Constant.AutOKButton));

 160:        clearButton = autFormName.FindFirst(

 161:          TreeScope.Descendants, new PropertyCondition(

 162:            AutomationElement.AutomationIdProperty, Constant.AutClearButton));

 163:      }

 164:  

 165:      /// <summary>

 166:      /// Generate a pseudo-random Unicode string using the Babel.dll automation library

 167:      /// See http://www.testingmentor.com/automation/sdk/babel/babel.htm

 168:      /// </summary>

 169:      private static string GetRandomTestDataVariant(out int seedValue)

 170:      {

 171:        string randomString = string.Empty;

 172:        do

 173:        {

 174:          Random prng = new Random();

 175:          seedValue = prng.Next();

 176:          StringGenerator sg = new StringGenerator();

 177:          sg.Info.Seed = seedValue;

 178:          sg.Info.MaximumCharacterCount = Constant.FirstNameTextboxLength;

 179:          sg.Info.RandomizeCharacterCount = true;

 180:          sg.Info.AllowSurrogatePairCharacters = false;

 181:          randomString = sg.Polyglot();

 182:        }

 183:        while (randomString.Contains("\0"));

 184:  

 185:        return randomString;

 186:      }

 187:  

 188:      /// <summary>

 189:      /// Generic method that enters the test data into the a specified textbox

 190:      /// Throws exception if the length of the testData argument exceeds the size

 191:      /// property of the textbox control, or textbox control in not enabled

 192:      /// </summary>

 193:      private static void SetTextboxText(AutomationElement inputTextbox, string testData)

 194:      {

 195:        try

 196:        {

 197:          ValuePattern input =

 198:            (ValuePattern)inputTextbox.GetCurrentPattern(ValuePattern.Pattern);

 199:          input.SetValue(testData);

 200:        }

 201:        catch (ElementNotEnabledException)

 202:        {

 203:          throw new ElementNotEnabledException("Textbox is not enabled.");

 204:        }

 205:        catch (InvalidOperationException)

 206:        {

 207:          throw new InvalidOperationException("Test data is invalid or exceeds maximum length.");

 208:        }

 209:      }

 210:  

 211:      /// <summary>

 212:      /// Oracle that compares the randomly generated string against the variable

 213:      /// result in the message box text

 214:      /// </summary>

 215:      private static bool ValidateMessageBoxStringOracle(

 216:        string testData, int maxPollCount, out string actualResult)

 217:      {

 218:          bool result = false;

 219:          IntPtr msgBoxHandle = IntPtr.Zero;

 220:          AutomationElement messageBox;

 221:          int pollCount = 0;

 222:          actualResult = string.Empty;

 223:          try

 224:          {

 225:            do

 226:            {

 227:              msgBoxHandle = NativeMethod.GetForegroundWindow();

 228:              messageBox = AutomationElement.FromHandle(msgBoxHandle);

 229:              string className =

 230:                (string)messageBox.GetCurrentPropertyValue(AutomationElement.ClassNameProperty);

 231:              System.Threading.Thread.Sleep(100);

 232:              pollCount++;

 233:            }

 234:            while (msgBoxHandle == IntPtr.Zero && pollCount < maxPollCount);

 235:  

 236:            if (messageBox != null)

 237:            {

 238:              AutomationElement textbox = messageBox.FindFirst(

 239:                TreeScope.Descendants, new PropertyCondition(

 240:                  AutomationElement.AutomationIdProperty, "65535"));

 241:              string messageBoxText =

 242:                (string)textbox.GetCurrentPropertyValue(AutomationElement.NameProperty);

 243:              actualResult =

 244:                messageBoxText.Substring(messageBoxText.IndexOf('\u0020') + 1, testData.Length);

 245:  

 246:              if (actualResult.Equals(testData))

 247:              {

 248:                result = true;

 249:              }

 250:  

 251:              AutomationElement messageboxOKButton = messageBox.FindFirst(

 252:                TreeScope.Descendants, new PropertyCondition(

 253:                  AutomationElement.ClassNameProperty, "Button"));

 254:              PushButton(messageboxOKButton);

 255:            }

 256:            else

 257:            {

 258:              actualResult = "Messagebox not found.";

 259:            }

 260:  

 261:            return result;

 262:          }

 263:          catch (ArgumentOutOfRangeException)

 264:          {

 265:            throw new ArgumentOutOfRangeException("POSSIBLE TEST DATA FAILURE");

 266:          }

 267:          catch (NullReferenceException)

 268:          {

 269:            throw new NullReferenceException("Messagebox unexpectedly became null");

 270:          }

 271:      }

 272:  

 273:      /// <summary>

 274:      /// Generic method to push a button control

 275:      /// Throws exception if button control is not enabled, or buttonName is null

 276:      /// </summary>

 277:      private static void PushButton(AutomationElement buttonName)

 278:      {

 279:        try

 280:        {

 281:          InvokePattern pushButton =

 282:            (InvokePattern)buttonName.GetCurrentPattern(InvokePattern.Pattern);

 283:          pushButton.Invoke();

 284:        }

 285:        catch (ElementNotEnabledException)

 286:        {

 287:          throw new ElementNotEnabledException("Button is not enabled.");

 288:        }

 289:        catch (InvalidOperationException)

 290:        {

 291:          throw new InvalidOperationException("Button not found.");

 292:        }

 293:      }

 294:  

 295:      /// <summary>

 296:      /// Generic method to close an AUT process

 297:      /// </summary>

 298:      private static void CloseAutProcess(Process autProcess)

 299:      {

 300:        try

 301:        {

 302:          autProcess.CloseMainWindow();

 303:          autProcess.WaitForExit(500);

 304:          if (!autProcess.HasExited)

 305:          {

 306:            autProcess.Kill();

 307:          }

 308:        }

 309:        catch (InvalidOperationException)

 310:        {

 311:          throw new InvalidOperationException("Specified process not found.");

 312:        }

 313:      }

 314:      #endregion HELPER METHODS

 315:    }

 316:  }

I am still surprised how often I hear people disparage automated testing. It’s almost like a rampant phobia similar to fear of water, fear of heights, or fear of the unknown. What leads people to proclaim disingenuous tripe that I sometimes read on blogs, or discussion forums? For example, the other day I read “automation will not find any new bugs.” Well, I guess that is the experience of some folks. But of course we could say that any inexperienced, untrained, or non-thinking person who uses a tool is likely to misuse the tool and get hurt, do damage, or come to a false conclusion that the tool doesn’t work. Automated tests are not bad; stupid automated tests are bad. If your test automation is not providing value don’t blame the tool…maybe a little introspection is in order.

All Automation Is Not Equal

The first problem in almost any discussion about automation is the lack of context. Automated tests are different and when done well can be effective at helping us identify some types of functional issues. An automated unit test suite can be a powerful tool for developers during refactoring. Of course some bugs escape unit testing because unit tests are not intended to be comprehensive. I have long been a proponent of functional testing at the API layer and this is where my team plays today. I see first hand the value of our automated tests in the early identification of potentially critical functional issues during our continuous integration process. While this level of testing is effective in helping prevent build breaks, it is not comprehensive and does not test the behavior of the application through the user interface. And then, comes GUI automation.

Why (Most) GUI Automation Sucks!

Although I am not a big fan of GUI automation I have done it, I teach it, and I know that it can be of value in many ways. Last week I was discussing reasons for intermittent failures in test automation with a colleague. Of course, a large number of intermittent failures occur in GUI automation for a variety of reasons (that’s a different post somewhere in the future). But, as I was reviewing the test code something dawned on me…the tests sucked!

It seems that when many people sit down to automate a GUI test (using just about any test automation paradigm) the automation is about a brain dead as a zombie. I suspect that when folks sit down to automate a GUI test the first thing they do is step through the UI. Then the tester automates each step until he/she arrives at some final state and validates the results with an equally brain dead oracle. In other words a lot of GUI automation is comprised of simplistic scripts that attempts to emulate human behavior patterns rather than being a well-designed test that capitalizes on the power of the computer as a tool.

Let’s look at a simple example; the “Hello” application. The application takes a first name string then displays a message box that reads “Hello [name]!”

Here is an example of a brain dead automated test that we might find if we asked someone to automate this “feature.”

   1:  namespace BraindDeadAutomatedGuiTest

   2:  {

   3:    class Program

   4:    {

   5:      [STAThread]

   6:      static void Main(string[] args)

   7:      {

   8:        // Launch the AUT

   9:        Process testAut = new Process();

  10:        testAut.StartInfo.FileName = "Hello.exe";

  11:        testAut.Start();

  12:  

  13:        // Hard coded sleep to give AUT time to instantiate

  14:        System.Threading.Thread.Sleep(1000);

  15:  

  16:        // Use key mnemonic to set focus to textbox 

  17:        // Often this is skipped because people assume this textbox has focus

  18:        SendKeys.SendWait("%f");

  19:  

  20:        // Enter hard-coded test data

  21:        SendKeys.SendWait("Boob");

  22:  

  23:        // Tab to the OK key and press

  24:        SendKeys.SendWait("{Tab}");

  25:        SendKeys.SendWait("{ENTER}");

  26:  

  27:        // Another hard coded sleep to make sure messagebox appears

  28:        System.Threading.Thread.Sleep(1000);

  29:  

  30:        // Now comes the brain dead oracle

  31:        Clipboard.Clear();

  32:  

  33:        // Capture the messagebox text

  34:        SendKeys.SendWait("^(+c)");

  35:  

  36:        // Now get the text from the clipboard and see if it contains the

  37:        // hard-coded test data; often I see people compare strings

  38:        // verbatim which is a recipe for false positives & maintenance 

  39:        string actualResult = Clipboard.GetText(TextDataFormat.Text);

  40:        if (actualResult.Contains("Boob"))

  41:        {

  42:          Console.WriteLine("Pass");

  43:        }

  44:        else

  45:        {

  46:          Console.WriteLine("Fail");

  47:        }

  48:  

  49:        // Clean up - kill the messagebox then close the AUT

  50:        SendKeys.SendWait("{ESC}");

  51:        testAut.CloseMainWindow();

  52:      }

  53:    }

  54:  }

Now, you might be saying that this is an absurd example, Really? I don’t think so. I see this sort of automation all the time in examples on the web and at conferences. For example, how often have you ever seen an example along the lines of:

1. Launch IE (or some other browser)
2. Navigate to “http://google.com”
3. Enter “Ruby” in test box
4. Press Search button
5. Verify “Ruby” appears in results

Now, if you really expect this type of automation to add value then you should really consider a new line of work…perhaps assembly line work.

Well, we are about 1 1/2 hours out from Amsterdam and breakfast is about to be served, so I am going to wrap this up now. Yes, I learned a long time ago not to complain if I don’t at least offer a solution (whining is never appreciated). So, later this week, perhaps on the trip home on Wednesday, I will suggest one possible solution to automate even this simple app that could provide value by increasing test coverage and potentially exposing some errant behavior (bugs).

(Oh…and by the way, while writing this a little dirt merchant in pajamas and a pacifier sticking out of his pie-hole started doing laps through the cabin banging off the sides of the chairs…but at least he wasn’t crying!)