Skip to content

API Testing–Functional Testing Below the User Interface

After a long hiatus from writing I am finally carving out some time to put thoughts to words again. A lot has been going on both professionally and personally. On the personal side I will simply say that never take for granted the time someone you care for has on this earth, and make a habit of spending quality time with that person regularly. On the professional side, things have been crazy busy in a very good way. I have settled in at work (still have lots to learn as always), and squeezed out a day to drive to Portland, OR to speak at PNSQC on random test data generation, and also present an online presentation discussing API testing best practices for the STP Online Summit: Achieving Business Value With Test Automation. Based on questions from that session I thought I would follow up with a few posts discussing API testing. Let’s start with describing API testing and how it differs from other “levels of software testing.”

Application Programming Interface (API)

The Microsoft Press Computer Dictionary defines API as “A set of routines used by an application program to direct the performance of procedures by the computer’s operating system.” So, referring to the abstract levels of testing, an API can be a unit, but is more likely a component because it is usually “an integrated aggregate of one or more units.”

An API provides value to both the developer and to the customer. For example, an API:

  • provides developers with common reusable functions so they don’t have to rewrite common routines from scratch
  • provides a level of abstraction between the application and lower level ‘privileged’ functions
  • ensures any program that uses a given API will have identical behavior/functionality (for example, many Windows programs use a common file dialog such as IFileSaveDialog API to allow customers to save files in a consistent manner)

Essentially, an API contains the core functionality of a program, or the business logic as some people refer to it. Customers don’t interact with API’s directly. Customers interact with software via the Graphical User Interface (GUI) which in turn interacts with an abstraction layer (e.g. controller in the MVC design pattern) that interacts with APIs exposed via Interfaces.

Testing an API as a Black Box

Some people assume that API testing is a ‘white-box’ testing activity in which the tester has access to the product source code. But in reality, API testing is truly black-box testing in the truest sense of the testing approach. API testers make no assumptions about how the functionality is implemented, and are not limited by constraints or distracting behaviors of a graphical user interface.

As an example, let’s use a program I developed called Morse Code Trainer. As a boy I was really into electronics (HeathKit projects were routinely on my Christmas list), and in order to pursue my amateur radio license I had to learn Morse code, or CW for short. Although Morse code is not required any longer to get a HAM operators license I think it is not hard to learn (memorize) about 55 sequences of dits and dahs, and in my opinion learning additional languages is good for the brain.

A core bit of functionality in this program is to convert a string of characters (a sentence) to the dits (represented as a period character “.”) and dahs (represented as a dash character “-“). The API to do this bit of magic is:

        string AlphaNumericCharacterStringToMorseCodeString(string input)

and is exposed to the developer who will code the UI and controller via the IMorseCodeEncoder interface.

        interface IMorseCodeEncoder
            string AlphaNumericCharacterStringToMorseCodeString(string input);

           string MorseCodeStringToAlphaNumericCharacterString(string input);

Notice that we don’t see any of the underlying code of how this method actually does its magic.

Let’s assume the developer didn’t do any unit testing and simply threw the code over the proverbial wall for testers to beat on. Since the tester (me) knows the developer (me) didn’t do any unit testing of any of the private methods the API under test relies on, the API tester (me) writes a simple test just to see if this code “works” like the developer (me) assured the tester (me) that it would. The most basic API test looks very similar to the unit test illustrated below, and in fact it this is a unit test the developer should write and execute before chucking a program at testers to bang on. A proper API test would call the method under test from the dynamic link library (DLL), and include initialization, clean-up, utilize the proper test design, have a robust oracle,  and of course have no hard-coded strings.

   1:      [TestMethod()]
   2:      [DeploymentItem("Morse Code Trainer.exe")]
   3:      public void GetMorseCodeStreamTest()
   4:      {
   5:        try
   6:        {
   7:          MorseCodeEncoder_Accessor target = new MorseCodeEncoder_Accessor();
   8:          string input = "A QUICK TEST";
   9:          string expected = ".-  --.- ..- .. -.-. -.-  - . ... -";
  10:          string actual;
  11:          actual = target.GetMorseCodeStream(input);
  12:          Assert.AreEqual(expected, actual);
  13:        }
  14:        catch (Exception e)
  15:        {
  16:          Assert.Fail(e.ToString());
  17:        }
  18:      }


Interestingly enough, had the developer (me) ran this unit test the developer would have discovered an unhandled exception. The unit test failed because this API called a method to get a Dictionary in another class and the Dictionary was created from 2 string arrays (an array of alpha-numeric characters, and an array of Morse code sequences). The specific error was a duplicate key/value in the Dictionary; in other words a duplicate entry in the alphaCharacterArray string array was throwing an System.ArgumentException for duplicate keys. But, because the methods in the MorseCodeLibrary class weren’t unit tested the API to encode a string of alpha-numeric characters to Morse code characters failed its basic unit test.

   1:      public Dictionary<string, string> GetAlphaCharacterToMorseCodeDictionary()
   2:      {
   3:        Dictionary<string, string> AlphaToMorseCodeDictionary = new Dictionary<string, string>();
   4:        for (int i = 0; i < this.alphaCharacterArray.Length; i++)
   5:        {
   6:          AlphaToMorseCodeDictionary.Add(this.alphaCharacterArray[i], this.morseCodeArray[i]);
   7:        }
   9:        return AlphaToMorseCodeDictionary;
  10:      }


But, it actually gets worse. Another API in another class to decode a string of Morse code to alpha-numeric characters would have failed as well because it used the same faulty string array of data to create a Dictionary calling the public method

        public Dictionary<string, string> GetMorseCodeToAlphaCharacterDictionary().

This is actually a good example of 2 very different bugs with the same root cause. This is also good example of how it is more efficient to find functional bugs at the unit,  or component or integration levels of testing (API testing) as compared to finding this problem via functional testing through the user interface.

Unit vs. API Testing

So, you’re probably asking yourself “if the above example is really an example of a unit test the developers should do before throwing their code at testers, then how does unit testing differ from API testing?” When testing a single API call the most significant difference is in the thoroughness of test coverage. Most unit tests are rather simple things. Unit tests are not very complex; unit tests are not comprehensive in test coverage (although a good suite of unit tests should achieve good structural coverage); and unit tests often rely on simplistic oracles.

API tests by contrast are usually more comprehensive as compared to unit tests. API tests usually include both positive tests (does it do what its supposed to do) as well as negative tests (how well does it handle error conditions). While API tests should strive for a high level of code coverage (structural testing) a more important goal is test coverage. For example API tests of this same method might include a series of data-driven tests that:

  • test every known alpha-numeric character defined in Morse code (the population of the variable is small enough to test every element, if the population of a given variable is large then testers should define equivalent partitions and test an adequate number of samples from the population for confidence)
  • test character casing
  • test pangrams, and special signals (e.g. end of message, attention, received, etc)
  • test boundary conditons (e.g string max len although 2 billion+ characters seems excessive for a Morse code transmission)
  • test strings with invalid or characters that are not defined in Morse code
  • test strings with non-ASCII letters that have Morse code encodings (Ä, Á, Å, Ch, É, Ñ, Ö, Ü)
  • test performance to provide baseline measures of individual methods

More complex APIs such as this MessageBox.Show method that have several parameters with variable argument values might benefit from additional testing techniques such as combinatorial testing.

Testing API End–To–End Scenarios

Testing a single API is usually considered unit or component level testing in the abstract levels of testing. Some people consider unit and component level testing to be “owned” by the developer. I certainly agree that unit tests must be owned by the developer, and that developers can do a much better job of component level testing. But, I also think this is a key area where API testers can collaborate more closely with developers to increase the effectiveness of the tests, the data used in the test, and even the test design (e.g. data-driven unit testing).

But, I will suggest that the integration level of testing or “testing done to show that even though the components were individually satisfactory, as demonstrated by successful passage of component tests, the combination of components are incorrect or inconsistent“ is the domain of the API tester. Software applications are complex beasts that often rely on sequences of API calls interacting with databases, cloud services, or other background workers. So, although API testing rarely involves testing through the GUI, API testers must also understand how the various APIs will be used to effect various customer scenarios. The only difference is that API testers emulate these scenarios without navigating a graphical user interface.

For example, one scenario is to convert a string of text into dits and dahs and use the system’s beep to “play” the Morse code sequence over the computer’s speaker. So, this program contains a class to convert the sequences of dits and dahs into sound; the SoundGenerator class. The interface for the sound functions includes a getter and setter, and the PlayCharacterCode API.

   1:    interface ISoundGenerator
   2:    {
   3:      void PlayMorseCode(string morseCodeString);
   5:      int WordsPerMinute { get; set; }
   6:    }


So, although we don’t know exactly yet how the developer and GUI designer will implement the GUI for this program, we can still create a test that inputs a string of alphanumeric characters, encodes the alphanumeric string into a Morse code encoded string, and then passes the string of Morse code dits and dahs as an argument to the PlayMorseCode method. This is a rather simple example of an end-to-end scenario. In more complex application the API functions/methods would likely be compiled in one or more dynamic link libraries (DLLS), rely on mocks, fake servers, and possibly other emulators. Of course, the oracles for this type of API testing is also more complex and generally involves checking multiple outcomes or states.

API testing focuses on an application’s functional capabilities, whereas testing through a GUI should focus primarily on behavior, usefulness and general ‘likeability.’


  1. Johan Hoberg wrote:

    Excellent post!

    Very valuable to reference for me when I create company internal presentations!

    [Bj's Reply] Thank you for your kind words Johan. I am glad the post is of use to you. I will follow up with a few more based on my presentation I gave at the STP online summit. Please don’t hesitate to let me know if I can be of assistance in providing additional references for your company’s internal presentations.

    Tuesday, November 22, 2011 at 2:52 AM | Permalink
  2. Tarun wrote:

    Indeed valuable post.
    Only thing I am not very comfortable with is keeping test methods in try-catch block (GetMorseCodeStreamTest).
    I have seen instance when it is forgotten that there should exception thrown from catch () and test ends up in suppressing errors.

    [Bj's Reply] Hi Tarun. Thank you. I am glad you found the post of interest. I agree with you that swallowing an exception is generally always a bad thing. In my test automation design class I teach people if the expectation is an exception, then they should use a nested try/catch block. The inner try/catch should be placed to catch the “expected” exception (which is common for example in some negative testing). The outer try/catch block is simply used to catch any unexpected anomolies during the test execution.

    Tuesday, November 22, 2011 at 9:02 AM | Permalink
  3. santi wrote:

    I am a big fan of your blog. Can you please write a post on how a STE can move to SDET role and what points he needs to work on. I liked your last post on STE/SDET.

    [Bj's Reply] Hi Santi, thank you for your kind words. I started giving advice about the mental and skill shift from “STE” to “SDET” or really about how testers had to upgrade their skills in 2003. Dispite the riducule from several so-called “experts” in the field I have continued to suggest for ways for people to improve. But, perhaps a post similar to your and posts from my perspective may be in order. Thanks for the suggestion.

    Monday, November 28, 2011 at 11:31 PM | Permalink

One Trackback/Pingback

  1. [...] • I.M.Testy размышляет о тестировании API в сравнении с тестированием через пользовательский интерфейс и с помощью модульных тестов. [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *
* storniolo-cyril