Also this week the newest edition of the Automated Software Testing magazine published an article I wrote entitled UI Automation Beneath the Presentation Layer. It discusses some tips on how to use reflection for automating applications developed using managed code. If you have questions or comments about my article please let me know. Your feedback is appreciated.

I have always respected the work of Dion Johnson, and I was quite honored when he asked me to write an article for the magazine. The magazine and the Automated Testing Institute website provide a wonderful resource to people interested in software test automation. I really look forward to working with Dion and the Automated Testing Institute more closely in the future to provide whatever help I can to this monumental task he has undertaken. I also hope there are others out there who will share some of their ideas as well. I think the days of Not Invented Here (NIH) and I’ve Got A Secret (IGAS) syndrome are becoming issues of the past. We still have too much reinvention and not enough reuse of test code. I think that we have well established there is no single approach to testing that is effective in all contexts. We might approach problems from different perspectives, but I also think that all testers are passionate and have very similar goals of personal and professional improvement. To mature our discipline, and increase our effectiveness in our roles we should collaborate more and pool our resources, and the Automated Testing Institute is a growing community of professionals.

Technorati Tags: Automation,Reflection
Windows Live Tags: Automation,Reflection
WordPress Tags: Automation,Reflection

I am generally not a big fan of static data in test automation, but being a pragmatic person, I know there are clearly times when using data-driven testing is just plain common-sense. For example, data-driven testing is an effective automation approach when designing ‘black-box’ tests for testing an API.

Data-driven testing is a common approach to test automation where static test data is passed to application parameters and the expected result (which is usually also read from static data) is compared against the actual result. This automation approach is effective when the actual result compared to some expected result can be resolved as a Boolean outcome. In other words, if the actual and expected results match the outcome is true and the test passes; otherwise the outcome is false and the test fails. (Of course, if something occurs during the test where there is no actual result then that particular test is usually logged as indeterminate.)

Of course, the key to effective data-driven testing is the data! If we don’t identify the most appropriate data to use in the test then the test case may have holes and we might overlook important information or miss critical anomalies. If we have too much redundant data then we may be simply running unnecessary tests (yes, even with test automation redundant testing is not an efficient use of resources).

Let’s say we had to test an API method such as:

public bool IsValidNetBiosName(string name)

where the return value is true if the string argument passed to the name parameter is a valid NetBIOS name on the Windows operating environment; otherwise it returns false.

With a data-driven testing approach we could use a simple CSV file that contained the string arguments and the expected result for each string passed to the name parameter. A partial sample of the CSV data file would be:

a,true
validname,true
validnamexxxxxx,true
invalidnamexxxxx,false
invalidnamexxxxxxxxxxxxxxxxxxxxxxx,false
,false
null,false,
xx\x,false
xx/xx,false
x:x,false
xxx*xx,false
x?xx,false
xxxx",false
;,false
xxx|xxx,false

(NOTE: null is a special case in which we need to convert the string “null” to a null in the test code, and the test above null is an empty string. An empty string and null are two different things and both must be tested in this case.)

Next, we need to read in the CSV file into our automated test, and perhaps the easiest way I found to read in a text or CSV file in C# is with the File.ReadAllLines method. The ReadAllLines method opens a text file, reads each line of text as an element in a string array, and then closes the file. Once we have a array of all lines in our data file, we simply need to parse each element in the string array into test data and/or expected result, and then compare the actual result against the expected result as illustrated in this example.

   1: // Read each line in the entire CSV file into a string array

   2: string[] testDataArray = System.IO.File.ReadAllLines("myTestData.csv");

   3:  

   4: // Iterate through each element in the test data file

   5: foreach(string test in testDataArray)

   6: {

   7:     // Split each element in each line into an array where the elements are the

   8:     // test data and the expected result

   9:     string[] testElement = test.Split(',');

  10:     string testData = testElement[0];

  11:     string expectedResult = testElement[1];

  12:     

  13:     // Special case for passing a null to the API parameter

  14:     if (string.Equals(testData, "null", 

  15:         StringComparison.OrdinalIgnoreCase))

  16:     {

  17:         if (string.Equals(api.IsValidNetBiosName(null).ToString(), expectedResult, 

  18:             StringComparison.OrdinalIgnoreCase))

  19:         {

  20:             result = "Pass";

  21:         }        

  22:         else

  23:        {

  24:             result = "Fail";

  25:        }

  26:     }

  27:  

  28:      // Compare the return value against the expected result

  29:     else if(string.Equals(api.IsValidNetBiosName(testData).ToString(), expectedResult,

  30:         StringComparison.OrdinalIgnoreCase))

  31:     {

  32:         result = "Pass";

  33:     }

  34:     else

  35:     {

  36:         result = "Fail";

  37:     }

  38: }

This is rather simple example, but data-driven testing is effective for unit testing, API testing, and can even be used in automated GUI testing (although data-driven automation may only have limited applicability in GUI automation). I am a firm believer in the KISS principle when it comes to developing automated tests, and the ReadAllLines method is perhaps the easiest and most efficient way to read in data file for data-driven development. Of course, data-driven testing doesn’t solve all problems. Chan Chaiyochlarb has a good post on some pitfalls to watch out for. But, in the right context, data-driven testing can be one approach used in automated testing.