Archive for the ‘Test Automation’ tag
Random Test Data – Credit Card Numbers
Things are winding down for the year. The Christmas lights are up on the house, my gardens are tilled and mulched for next spring, people are disappearing from the office like there is a plague, it hasn’t snowed in a while which means the mountains are mostly ice (I dislike skiing on ice), the next GSHL ice hockey league doesn’t start for awhile and pick up games are few and far between (I suck at hockey but it is fun). So, what to do? Oh…I forgot Christmas shopping. I hate Christmas shopping! So, I have spent the past few idle nights refactoring the automation libraries for some of my test data generation tools after my daughter goes to bed.
One of the most popular random test data generators that I have developed so far has been a tool called CCMaker to generate random valid and invalid credit card numbers. (Sometimes I wonder why that is, but I don’t dwell on it for too long and I haven’t been interrogated by the FBI lately.) Testing forms that require a credit card has always been risky business because you certainly don’t want to use your own card. Often times developers will include a check on web forms or client apps to do a high level verification of a credit card number before sending all the data across the wire to be validated. This early or high level verification prevents flooding the pipe with bad data. So, one test we can do prior to testing the end-to-end scenario is to test to see if and how the developer is validating credit cards numbers prior to submission.
As far as data goes, generating credit card numbers are fairly simple. There is a bank ID number (BIN), there is a number of digits between 12 and 19 depending on the card type, and there is a checksum. So, if we know the valid BINs for each issuing bank, the valid number of digits for each card type, and how to calculate the checksum we can generate valid credit card numbers. (Of course this is a bit oversimplified because many credit and debit card companies are issued multiple BINs and use varying number lengths.)
Testing for invalid credit card numbers should include using numbers that look close to being correct in some way but are slightly altered. For example for the 3 defined equivalent partitions (BIN, length, checksum) there are seven possible invalid combinations (23 – 1) we could test.
- Valid BIN, invalid length and valid checksum
- Valid BIN, valid length, and invalid checksum
- Valid BIN, invalid length, and invalid checksum
- Invalid BIN, valid length and valid checksum
- Invalid BIN, invalid length, and valid checksum
- Invalid BIN, valid length, and invalid checksum
- Invalid BIN, checksum and length
This doesn’t mean I run 7 tests and call it good because there are numerous invalid lengths and invalid BINs for the different card types. A common mistake when using an equivalent partition testing approach is to simply plug in values for each combination listed above and call it good. The problem is that there are several hundred BINs and 8 different valid lengths. For example, for just the Discover card there are 829 valid BIN numbers, and for the Maestro cards there are 56 combinations of BINs and card lengths ranging from 12 to 19 numbers in length. This doesn’t include the permutations of the other numbers that compose the entire card number.
The question every tester must ask him or herself every day when designing tests is how many tests do I need to have any reasonable sense of confidence that risk is minimal and the perception of quality is high. Of course, there is no single right answer here and not magic formula, but since we can’t possibly execute every possible positive or negative test we should at least understand that ultimately testing is sampling.
For example, one strategy for positive testing might be to test every valid BIN for every valid card length for any given credit card. For example for American Express I would want to test at least one number with a BIN of 34 and a card length of 15 that satisfies the checksum requirement, and at least one number with a BIN of 37 and a card length of 15 that also satisfies the checksum requirement. For a card type of Visa I would need a minimum of 2 tests in which the BIN is 4, the checksum requirement is satisfied, and one has a card length of 13 numbers and the other has a card length of 16 numbers.
That probably sounds like quite a bit of testing, and tests which most likely would not produce an error (unless of course the BIN is miss identified (e.g. instead of checking for a BIN of 5020 the BIN is incorrectly assigned as 5002), or if a valid BIN is not recognized as valid because it is omitted from a list or enumeration of valid BINs for that credit card). Certainly testing of this magnitude would be expensive if done manually. But when automated using a random test data generator and a data-driven automation approach to set the random generator properties comprehensive testing becomes a much more reasonable proposition and can significantly increase overall confidence.
This is where my CCMaker 3.1 test data generator can help by randomly generating both valid and invalid credit card numbers. The updated CCMaker test automation library has just been posted to my web site with documentation and examples. If you have any questions, or find any issues with the new library please let me know.
Randomizing Static Test Data in Automated Tests
Originally Published Sunday, October 11, 2009
A significant percentage of static test data is stored in tabular comma delimited or tab-delimited formats and saved in Excel spreadsheets. Reading in comma or tab-delimited static test data into an automated test is pretty straight forward and there are numerous examples in many programming languages illustrating how to read in these types of test data repositories. Reading in rows of data is the foundation of data-driven automation and definitely has its place in any automation project.
I am a big proponent of stochastic (random) test data generation that is customized to the context, but I also know that sometimes static test data is useful for establishing baselines and more exact emulation of ‘real-world’ customer-like inputs. But, if the automated test is simply passing the same variable arguments to the same input parameters in the same order over and over again the value of subsequent iterations of that automated test using that static data set diminishes rather quickly. So how can we more effectively utilize static test data in our automated tests?
One possible solution is to randomly select an argument from a collection of static variables that is passed to the specific input parameter. The advantage of this approach is that it effectively increases the test data permutations in each iteration of the test case. For example, let’s consider 2 input parameters; one for a given name and one for a surname. In a traditional data-driven approach in which the static test data is read in by rows our test data file might be similar to:
Bob,Smith
John,Johnson
Roger,Williams
Steve,Abbot
This static data file would give us 4 sets of test data, but each time the test data is read into the test case the given and surnames are always the same.
However, if we read in the given names and surnames into 2 collections, and then randomly select a given name and surname from the appropriate collection to pass to the respective parameter we effectively have 16 possible combinations of static test data to work with. An advantage of this approach is that our ‘collections’ of given names and surnames can contain differing numbers of elements (in which case the number of possible combinations of test data is the Cartesian product of the number of elements in each collection).
Of course there are many ways to accomplish this. For example, one approach is to continue to use a comma or tab-delimited file format and list given names in one row and surnames in a second row. Another approach is to list the given names and surnames in columns in a spreadsheet and read in each column into a collection of some sort. The latter is the approach I used in developing my PseudoName test data generator tool. I chose this approach for 2 reasons; first an Excel spreadsheet is a simple yet powerful file format for storing static test data, and secondly because lists of test data are sometimes better represented in columns rather than rows.
The following code shows one way to read in test data by columns from an Excel spreadsheet.
1: // <copyright file="datareader.cs" company="TestingMentor">
2: // Copyright © 2009 by Bj Rollison. All rights reserved.
3: // </copyright>
4:
5: namespace TestingMentor.TestTool
6: {
7: using System;
8: using System.Collections;
9: using System.Globalization;
10: using System.Runtime.InteropServices;
11: using System.Threading;
12: using Excel = Microsoft.Office.Interop.Excel;
13:
14: /// <summary>
15: /// This class contains methods for reading test data from Excel spreadsheets
16: /// </summary>
17: public class TestDataReader
18: {
19: /// <summary>
20: /// This method reads all the data elements in the specified number of
21: /// columns in the specified Excel spreadsheet containing the test data
22: /// and copies the data into a multi-dimensional array
23: /// </summary>
24: /// <param name="dataFileName">The filename containing the test data</param>
25: /// <param name="columnCount">The number of columns in the Excel
26: /// spreadsheet to read</param>
27: /// <returns>A multi-dimensional array containing the data eleements for
28: /// each column </returns>
29: public static string[][] ExcelColumnReader(string dataFileName, uint columnCount)
30: {
31: CultureInfo originalCulture = null;
32: Excel.Application excelApp = null;
33: Excel.Workbook excelWorkbook = null;
34: Excel.Worksheet excelActiveWorksheet = null;
35: string[][] testData = new string[columnCount][];
36:
37: originalCulture = Thread.CurrentThread.CurrentCulture;
38: Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");
39:
40: excelApp = new Excel.Application();
41: excelWorkbook = excelApp.Workbooks.Open(
42: dataFileName,
43: 0,
44: false,
45: 5,
46: String.Empty,
47: String.Empty,
48: false,
49: Type.Missing,
50: String.Empty,
51: true,
52: false,
53: 0,
54: true,
55: false,
56: false);
57: excelActiveWorksheet = (Excel.Worksheet)excelWorkbook.ActiveSheet;
58:
59: for (int i = 0; i < columnCount; i++)
60: {
61: // Start at column 1
62: object columnIndex = i + 1;
63:
64: // Row 1 is the column title; test data starts on Row 2
65: object rowIndex = 2;
66: ArrayList tempCollection = new ArrayList();
67: while (
68: ((Excel.Range)
69: excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2 != null)
70: {
71: tempCollection.Add(
72: ((Excel.Range)
73: excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2);
74: rowIndex = (int)rowIndex + 1;
75: }
76:
77: testData[i] = new string[tempCollection.Count];
78: testData[i] = (string[])tempCollection.ToArray(typeof(string));
79: }
80:
81: // Clean up
82: excelWorkbook.Close(false, Type.Missing, Type.Missing);
83: excelWorkbook = null;
84: excelApp.Quit();
85: excelApp = null;
86:
87: // Garbage collection is not pretty, but necessary to release Excel proc
88: System.GC.Collect();
89: System.GC.WaitForPendingFinalizers();
90:
91: if (originalCulture != null)
92: {
93: Thread.CurrentThread.CurrentCulture = originalCulture;
94: }
95:
96: return testData;
97: }
98: }
99: }
I must tell you that performance can be an issue especially if the columns contain a lot of data. For example, to read in approximately 700 elements of test data in 3 separate columns took slightly less than 1 second, and reading in 1800 elements in 3 columns required just over 4 seconds. Unfortunately, I didn’t compare total byte counts, but it is pretty obvious the greater the number of test data elements being read the longer the read operation will take and you certainly will have to take the read time into consideration in your automated test case.
Reading static test data line by line from a data file while looping through a data-driven automated test case is a useful test design approach in some situations, this is another useful approach that will allow the test designer to randomize the combinations of static test data values applied to multiple input parameters in multiple iterations of an automated test case.
Test Automation ROI (Part II)
Originally Published Wednesday, September 02, 2009
Last week I talked about the silliness of wasting time calculating the return on investment (ROI) of an automation effort on any non-trivial software project; especially if it has an extended shelf-life. As my friend Joe Strazzre commented, “If you need an ROI analysis to convince business management that test automation is a good thing when used intelligently, than you have already lost.”
But, management might need to be educated on the limitations of record/playback, rudimentary hard-coded scripts and keyword driven automation efforts because these it is often more appealing for bean counters to invest in low cost tools and continue to rely on non-coding bug finders or domain experts to script out ‘tests’ which do nothing more than repeat some rote set of steps over and over again. But, as E.Dustin, T. Garrett, and B. Gauf wrote in Implementing Automated Software Testing any serious software automation effort “is software development.” Well designed automated tests requires highly skilled, technically competent, extremely creative, analytical testers capable of designing and developing automated tests using procedural programming paradigms.
We should still apply ROI concepts in test automation, but at a much lower level. Essentially, each tester must evaluate the return on investment of any test before automating it. The most fundamental purpose of an automated test effort is to provide some perceived value to the tester, the testing effort, and the organization. As a tester, the primary reason I automate a test is to:
- Free up my time,
- Re-verify baseline assessments (BVT/BAT, regression, acceptance test suites)
- Increase test coverage (via increased breadth of data or state variability),
- Accomplish tasks that can’t easily be achieved via manual testing approaches.
For example, the build verification and build acceptance test suites are baseline tests that must be ran on each new build; these tests should be 99.9% automated because they free up my time to design other tests. Tests that evaluate a non-trivial number of combinations or permutations are generally good candidates for automation because they increase test coverage. Performance, stress, load, and concurrency tests should be heavily automated because they are difficult to conduct manually.
It is important to note that I am not simply referring to UI type automation. A significant amount of “functional tests” designed to evaluate the computational logic of methods or functions can be automated below the UI layer in software architectures using OOP and procedural paradigms where the business and computational logic is separate from the UI layer.
There are many papers that discuss specific factors to take into consideration when deciding what tests to automate. Unfortunately, there is no single cookie-cutter approach in deciding what tests to automate. Different projects have different requirements and expectations, and, of course, not all tests are equal. One of the best papers I’ve read on deciding what tests to automate is When Should a Test Be Automated by Brian Marick. I like the simplicity in his 3 key questions:
- How much more will this test cost to automate versus running it manually?
Some people think that automating a test reduces costs because it eliminates the tester from manually executing that test. Unfortunately, this is not always the case. As i talked about in a previous post, visual comparative oracles are notoriously error prone requiring the testers to constantly massage the test code and manually verify results anyway. Sometimes paying a vendor to run a test periodically is cheaper than paying an SDET to tweak the test every build. But, if the population of potential test data is large, or combinatorial testing of non-trivial features then automating that test case is probably a good investment. - What is the potential lifetime of this automated test?
How many times will this test be re-ran during the development cycle and in maintenance or sustained engineering efforts? Can this test be reused in the next iteration of the product? - Does the automated test have some probability of exposing new issues?
Although I don’t necessarily agree with this question because many automated tests may not expose new issues, but they still provide value to the overall testing effort. For example, I wouldn’t expect tests in my regression test suite to expose new defects because if they do there was a regression in the product. So, I would rephrase this question to ask, “Does this automated test have some probability of exposing new issues, providing additional information that increases confidence, or increases test coverage?”
A few other questions I ask myself when I am deciding whether to automate a particular test include:
- What exactly is being evaluated?
This is perhaps the first question I ask myself. If the test is evaluating functional or non-functional (stress, perf, security, etc.) capabilities then automation may be worthwhile. But, behavioral tests such as usability tests and content testing are generally not good candidates for automated testing. - What is the reliability of automating this test?
I don’t want to have to constantly massage a test in order to get it to run. So, what is the probability this test will throw a lot of false positives or false negatives? How much tweaking will this test require due to code or UI instability? - What are the oracles for this test and can they be automated?
I don’t want to sit in front of a computer and watch software run software. Also, there is a difference between an automated test and a macro (A single, user-defined command that is part of an application and executes a series of commands). There are different types of oracles, and the professional test designer needs to also design the most effective oracle for the test. By the way…if the most effective oracle is a human reviewing the results then that test should probably not be automated using current technologies.
For each test I consider these questions in deciding whether to automate that test. For some tests, I may ask additional questions depending on the context and the business needs of my organization. I don’t use a cookie-cutter template, or try to fill out some spreadsheet to do a cost comparison based on dollar amounts. It’s hard to put a price on value. Instead, I ask myself a few key questions to help me decide if automating a test is worth it to me, my team, and the organization. Is automating a particular test the right thing to do or am I automating something because it’s challenging, or to increase some magical percentage of automated tests compared to all currently defined tests. The key message here is not to blindly automate everything; use your brain and make smart decisions about whether each test should be automated and being able to explain how automating that test benefits the testing effort.
A Different Perspective on Random Name Generation
Originally Published Saturday, August 15, 2009
My daughter made me laugh today when she offered a bit of her philosophy. She told me that her favorite candy is gummy bears “because gummy bears get stuck between your teeth, and then you can dig out a second helping with your tongue.” I never really thought of it that way, but how many of us have not picked at a piece of licorice stuck between our teeth with our tongue (or a toothpick) and savor that last little bit? Ummm….
Perhaps it is my own twisted logic, but as I started writing this post I thought about my daughter’s predilection for gummy bears and somehow made a connection to static test data used in tests. Static test data that is simply reused over and over in a test is similar to that last little bit of licorice we dig out of our teeth. The last bit tastes just like the first bite, and all the other bites between. This may be good for those who like the flavor of licorice, but it is not so good for hard-coded test data in rudimentary test scripts, especially in automated tests.
If you have followed my posts or my personal website then you know that I am a big proponent of probabilistic stochastic test data (statistically unbiased, parameterized randomly generated test data that is representative of the population of possible inputs for a specific variable). The latest addition to my random test data generator toolbox is PseudoName, a random name (pseudonym) generator library for automated testing.
Before designing and developing PseudoName I researched the plethora of available random name generators currently available because I am not a big fan of reinventing the wheel either. In fact, there are many very good online html based random name generators. For example, Fake Name Generator that not only generates a pseudonym, but also generates an address, phone number, etc. essentially creating a fictitious persona. However, while this tool is useful for manual testing it is not so useful for automated tests. The Automated Testing Institute website provides code samples in VBScript and Ruby for generating random names from a built in collection of names stored in an array. These examples are also useful and the collections can certainly be expanded to include a greater variety of names, but they are still limited in scope.
A common problem that I noticed among all available random name generators is the Romanization (representing a written language with the Latin alphabet) of the pseudonym. Basically this means the random names are always represented with ASCII characters. Romanization may be satisfactory for those who only know the letters “A” through “z” or for those whose eyes glaze when the displayed character glyphs are in a foreign language. But, for those of us dealing with modern software or services that supports Unicode and may be adapted (or localized) or used in different locales where it is important to support the native language we soon realize that Romanization using simple ASCII characters is simply not enough for effective globalization testing.
Unlike most random name generators PseudoName generates a random name (pseudonym) from columns of name data in an Excel spreadsheet. The name data in the Excel spreadsheet is stored as Unicode so the characters can be the same as those used in the desired region or locale. For example to generate a random female Chinese name most name generators would produce a string such as “Dongyi Li.” However, PseudoName can randomly generate a name using Chinese characters such as “冬怡 李.” (Actually, Dongyi Li is not a pseudonym. Dongyi is my friend and she was kind enough to produce the Chinese name list of female, male, and surnames, and also helped me with refactoring the code used in this tool.)
The PseudoName library is simple to use in an automated test. The PseudoName members page also includes simple examples, and the NameInfo properties allow customization of the pseudonym output. If additional properties are necessary to generate reasonably realistic names in different locales please let me know. Also, if there is enough demand I might consider slapping on a GUI.)
The format for the Excel sheet is simple. The first column is female names, the second is male names, and the third is surnames. The names listed in the currently available US and Japanese names data files are the most common names in those countries according to census data. The names in the Chinese data file are the characters used for feminine and masculine names, as well as the most common surnames used in China. (I could really use some help collecting name lists using Unicode character scripts from other countries around the world. If you want to contribute please send me a name list in Excel and I will post it on the tool website for other testers to use.)
UI Automation Out of Control
Originally Published Saturday, August 01, 2009
When many people think of test automation they envision rudimentary scripts with hard-coded events and data that manipulate user interface objects much the same way a customer might interact with the software to accomplish a pre-defined, robot-like task. Perhaps this is the reason there is a plethora of tools available to business analysts or super-users hired as ‘black-box’ testers to help them record and playback (or list keywords to sequentially step through) some contrived set of steps they think a customer might perform. Sure…it’s cool to watch windows open and close, and the mouse cursor move across the desktop as if by magic. But, for anyone with half-a-brain the visual amazement lasts for for oh….about 1.7 seconds….after that it is mind numbingly boring! Unfortunately, this automation is usually short lived, requires tremendous overhead in terms of maintenance costs, and contributes to the exceedingly high percentage of failed or less than successful automation projects.
I will say that in general I am not a big fan of GUI automation for a litany of reasons, but mostly because it is misused, or applied to tests that are simply more effectively and efficiently performed by a person. However, I also understand the GUI automation does provide value when used in the right context. I am not against GUI automation, but I certainly don’t advocate automating a test just because we think we can, or because we have some bizarre idealistic belief that everything should be automated.
For example, in one situation I spoke with a tester whose manager wanted him to maintain a legacy test designed to detect the correct color of an arrow symbol after an action was performed. If the action completed correctly the arrow was green; and if it was unsuccessful the arrow appeared red. Now, besides the fact that we could have just as easily automated a test to check the HRESULT value, this test could have been executed by a user within a reasonable time, there was little probability of change in this area of the code, and there were no dependencies. However, the manager insisted this GUI test run despite this test which used image comparison as an oracle was notoriously problematic. (This shouldn’t be surprising since many image comparison oracles are notoriously problematic and throw an inordinate number of false positives.)
The tester said the manager claimed by automating this test it would negate a tester from having to execute the test manually thus saving time. What??? This tester was spending hours per week chasing down false positives and tweaking the automation to “make it work” on the daily builds just to make his manager happy. So, although this feature was used repeatedly by hundreds of people dog-fooding the daily build, another few thousand people around the company self-hosting internal releases, and thousands of customers using beta releases this particular manager determined continued tweaking of this test would save some tester’s time!
In another example a tester inquired how to automate a a test to determine IF the order of the slides in a power point presentation had changed between different copies of a .ppt file. Of course, the question was followed by a flurry of responses suggesting creating a base set of images of each slide in the deck, and then using an image comparison tool to identify changes. I responded a bit differently. First, there are several ways to programmatically detect file changes, and if we detect changes in the binary properties we can easily open the Power Point presentation in slide sorter view and take a few seconds (depending on the number of slides) and visually compare it against an original. Sure it is a bit slower than an automated test, but I really suspect it would be more effective and probably even more efficient in the long run. I also wondered how many times this “test” would actually need to be ran during whatever project this person was working on (it wasn’t PowerPoint) in comparison to the hours/days it would take to develop such a test, and the ensuing maintenance nightmare.
These are just 2 examples of the misuse of automated UI testing that I think illustrate a few important points:
- Not all automated UI tests save time!
Tests that require constant massaging and tweaking because they constantly throw false positives take up a huge amount of a tester’s time in wasted maintenance. - Sometimes a human is a more efficient oracle than a computer algorithm!
Sure, just about anything a computer does can be automated to some degree in some fashion, but there really are clearly some tests where it is more prudent and simpler to rely on a tester. - Don’t rely on automation to emulate your customers!
Test automation does not effectively emulate a human user. Sure, we have test methods in some of our internal automation frameworks to slow down simulated keystrokes (the actual keys are not being pressed on the keyboard), or simulate multiple or repeated clicks on a control or the mouse, and other tricks that try to emulate various user behaviors; however, test automation is generally poor at detecting behavioral issues such as usability, ease of use, or other customer value type assessments. Rely on the feedback from internal and external customers who are dog-fooding, self-hosting, and beta-testing your product (and act on it). - Go under the covers!
I think many testers rely too heavily on UI automation because they think it emulates user behavior (although most things such as populating a text box are simulated via Windows APIs), or perhaps because they don’t know how to dig into the product below the surface of the UI. Whatever the case, think about the specific purpose of the test. If it is easier to check a return value, or call an API to change a setting then go deep…and stop messing around on the surface. (It only complicates the test, wastes valuable machine cycles, reduces reuse across multiple versions, and often leads to long term maintenance costs. (For an example of this see my previous post.) - Constantly massaging code contributes to false negatives!
I have seen many cases where a tester designs a a UI automated test, and then tweaks a bit here and there to get it to run. Often times this tweaking contributes to a tests ineffectiveness in exposing problems, and may even hide other problems. Also, some tweaks are geared around synchronization issues (sync’ing the automated test with the system under test) and involve artificially slowing down the automation (usually by stopping or ‘sleeping’ the automated test process for a specific period of time). Other tweaks might hard-code parameters that then make the test fail on a different resolution or non-portable across different environments. - STOP trying to automate every damn test!
As I stated before…just because we can automate something doesn’t mean that we should try to automate everything! We need to make rational decisions about what tests to automate, and what is the best approach to automating that test.
It is easy to be lured in by the siren call of UI automation. I write automated tests to free up my time to design and develop more and different tests, and so I don’t have to sit in front of the computer executing redundant tests, or constantly massage code to make it run. Automation is a great tool in the arsenal of competent professionals who understand its capabilities and know how to exploit its potential. But, it is one of many tools in our toolbox; and the best tool is the one sitting on our shoulders. Use it!
Programming Paradigms in Test Automation
Originally Published Thursday, May 14, 2009
Regardless of the personal opinions of a few people, the simple fact is that the demand for software testers who can design and develop effective test automation is increasing. Perhaps one reason for the distain by some folks in the industry is due to limitations of the test automation approach they are most familiar with, and they sometimes assume those limitations apply to all types of test automation. However, not all test automation approaches are equal, and there are advantages and disadvantages for any approach.
At its core an automated test case is software code, and similar to the various approaches used in developing product software there are different programming paradigms used to develop test automation such as:
- Record and playback automation
- Keyword or action-word driven automation
- Scripted automation
- Procedural automation
- Model based automation
Record and playback automation
The record and playback paradigm simply records sequences of keyboard and mouse events, auto-magically codifies them usually into some proprietary scripting language which can then be replayed (executed) over and over again. There are usually severe limitations to this type of automation and it tends to be extremely fragile requiring constant massaging (re-recording). Although many record/playback paradigm allows ‘test developers’ to modify the scripted actions to some extent, and possibly even incorporate simple yes/no oracles I think many people view the record/playback paradigm as being slightly more useful than trained monkeys in limited situations.
Keyword or action-word driven automation
Keyword or action-words are simple scripts usually in some tabular format that ‘describe’ a sequence of ‘actions’ for the computer to perform. Of course, the key to keywords is the underlying architecture of the tool that interprets the keywords and executes the sequence of events. The beauty of keyword driven testing is that it hides the actual code, and similar to record and playback can be more easily used by business analysts or ‘user domain experts’ hired to into testing roles to automate something. I do see the benefit of keyword driven testing in some limited contexts (especially for companies who rely on business analysts/user domain experts for testing software), but let’s be real…these people aren’t automating anything…they are simply filling out a form that is then feed into a tool that performs the actions as prescribed by the listed instructions. The keyword form does nothing by itself, and the only thing a ‘tester’ has to think about is using the correct key words to sequentially get from point A to point Z for a ‘test.’
Scripted automation (imperative programming)
The primary difference between keyword and scripted automation is the tester actually develops the test in a programming language rather than filling in a form with abstracted key words that drive some automation engine. However, similar to keywords, scripted automation tends to use rudimentary statements of basic instructions that manipulate the software to perform a pre-determined sequence of events as illustrated below.
1: def test_b_googlenews
2: #-------------------------------------------------------------------------
3: # Test to demonstrate WATIR select from drop-down box functionality
4: #
5: #variables
6: test_site = 'http://news.google.com'
7: puts '## Beginning of test: google news use drop-down box'
8: puts ' '
9: puts 'Step 1: go to the google news site: news.google.com'
10: $browser.goto(test_site)
11: puts ' Action: entered ' + test_site + ' in the address bar.'
12: puts 'Step 2: Select Canada from the Top Stories drop-down list'
13: $browser.select_list( :index , 1).select("Canada English")
14: puts ' Action: selected Canada from the drop-down list.'
15: puts 'Step 3: click the "Go" button'
16: $browser.button(:caption, "Go").click
17: puts ' Action: clicked the Go button.'
18: puts 'Expected Result: '
19: puts ' - The Google News Canada site should be displayed'
20: puts 'Actual Result: Check that "Canada" appears on the page by using an assertion'
21: assert($browser.text.include?("Canada") )
22: puts ' '
23: puts '## End of test: google news selection'
24: end # end of test_googlenews
25: def test_c_googleradio
Most examples of scripted automation appear as codified versions of a set of steps listed in a less-than-adequately designed manual test case using hard-coded arguments for variables, mindless progression between steps, and simple deterministic oracles. Scripted automation is probably most beneficial for automating specific sub-tasks in "computer assisted testing." However, scripted automation is usually too prescriptive, and rely heavily on nothing going wrong during the execution of the test case.
Procedural automation (procedural programming)
In procedural automation the testers also develops a test by writing a series of computational steps to achieve a desired purpose. However unlike scripted automation the procedural automation paradigm generally provides better control flow options during the execution of the automated test case, allows for greater complexity in the design, improves reuse and reduces maintenance through modularity, and can employ both deterministic and heuristic oracles.
1: // Procedural programming example
2: static void Main(string[] args)
3: {
4: string logResult = string.Empty;
5: // Path to the data file passed as a string argument to the test case
6: string pictTestData = args[0];
7: //Stopwatch to measure test case duration
8: Stopwatch sw = new Stopwatch();
9: sw.Start();
10: // Launch the AUT
11: AutomationElement desktop = AutomationElement.RootElement;
12: AutomationElement myAutForm = null;
13: Process myProc = new Process();
14: myProc.StartInfo.FileName = myConstantAutFileName;
15: if (myProc.Start())
16: {
17: // Polling loop to find AUT window by window property
18: int pollCount = 0;
19: do
20: {
21: myAutForm = desktop.FindFirst(TreeScope.Children,
22: new PropertyCondition(AutomationElement.AutomationIdProperty,
23: myConstantAUTPropertyID));
24: pollCount++;
25: System.Threading.Thread.Sleep(100);
26: }
27: while (myAutForm == null && pollCount < 50);
28:
29: if (myAutForm == null)
30: {
31: throw new Exception("Failed to find dialog");
32: }
33:
34: // Get UI element collection here...
35: // Call method to read in test data
36: string[] testData = ReadTabDelimitedFile(pictTestData);
37: // iterate through each set of test data (data-driven test example)
38: foreach (string test in testData)
39: {
40: // Call method to execute each set of test data and assign the return
41: // value to the logResult variable; Oracle is separate method called
42: // from the test method
43: LogResultMethod(ExecuteCombinatorialTestMethod(test));
44: }
45:
46: // close AUT and clean-up
47: TimeSpan ts1 = sw.Elapsed;
48: // log test case duration...
49: // Deal with situation if AUT failed to launch
50: }
51: }
Procedural automation can be used for anything from API to GUI automated test cases designed to evaluate functionality (computational logic), non-functional areas such as stress, performance, and security, and also behavioral . Using a language similar to the programming language removes abstraction layers, and also enables other members of the team (developers) to easily review test cases.
Model based automation
Model based automation is a relatively new automation paradigm, and its complexity is beyond the scope of this single post. Basically, model based automation involves codifying abstracted state machines and state traversals and couples these parts with an automation engine that uses some form of graph traversal logic to drive the system under test between the various state machines identified in the model. In some sense model based automation is similar to exploratory testing because tests are generally not pre-determined or pre-scripted, what constitutes a single test is really hard to describe, and the oracles generally detect errant behavior (or being in an unexpected state). Personally, I think there is tremendous potential in model based automation, but the industry has just begun to scratch the surface of this automation paradigm and it is still largely misunderstood. This automation paradigm requires more complex skill sets of the person designing the test automation such as the ability to abstract important machine states as a model, and encode system behaviors. For more information about model based automation I recommend taking a look at http://research.microsoft.com/en-us/projects/specexplorer.
So, approach which is best?
In my opinion, there may be some limited value in record/playback, keyword, and scripted automation in specific contexts; however, a robust automated test case that will run on multiple environments, multiple languages, and distributed across multiple platforms without rewriting the test for each variation requires well designed tests developed using procedural automation or model based automation approach.
Test Automation: Look Below the UI for More Effective and Robust UI Automated Test Case Designs
Originally Published Tuesday, April 14, 2009
Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.
Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.
For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!
To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.
In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below
1: namespace TestingMentor.PInvokeSample
2: {
3: using System;
4: using System.Runtime.InteropServices;
5:
6: /// <summary>
7: /// This class contains Native Win32 API functions that are marshalled
8: /// over for use in C#
9: /// </summary>
10: class NativeMethod
11: {
12: /// <summary>
13: /// Sets an item of information in the user override portion of the
14: /// current locale. This function does not set the system defaults.
15: /// </summary>
16: /// <param name="locale">the locale identifier of the locale with the
17: /// code page used </param>
18: /// <param name="localeType">Type of locale information to set.</param>
19: /// <param name="localeData">A null-terminated string containing the
20: /// locale information to set</param>
21: /// <returns>Returns true if successful; otherwise false</returns>
22: [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
23: public static extern bool SetLocaleInfo(
24: uint locale,
25: uint localeType,
26: string localeData);
27:
28: /// <summary>
29: /// Sets an item of information in the user override portion of the
30: /// current locale. This function does not set the system defaults.
31: /// </summary>
32: /// <param name="locale">the locale identifier of the locale with the
33: /// code page used </param>
34: /// <param name="localeType">Type of locale information to set.</param>
35: /// <param name="localeData">An integer value representing the locale
36: /// information to set</param>
37: /// <returns>Returns true if successful; otherwise false</returns>
38: [DllImport("kernel32.dll", SetLastError = true)]
39: public static extern bool SetLocaleInfo(
40: uint locale,
41: uint localeType,
42: int localeData);
43: }
44: }
The argument values that we can pass to these functions are enumerated in a separate class similar to the one below
1: namespace TestingMentor.NlsInfo
2: {
3: /// <summary>
4: /// Constant values for SetLocaleInfo API
5: /// </summary>
6: class NlsConstant
7: {
8: public enum Locale : uint
9: {
10: Invariant = 0x007F,
11: SystemDefault = 0x0800, // use system default for setlocaleinfo
12: UserDefault = 0x0400,
13: Neutral = 0x0000,
14: CustomDefault = 0X0C00, // Vista and later
15: CustomUiDefault = 0x1400, // Vista and later
16: CustomUnspecified = 0X1000 // Vista and later
17: };
18:
19: public enum LocaleType : uint
20: {
21: // VALUE LCDATA TYPES
22: CalendarType = 0x00001009, // type of calendar specifier
23: CurrencyDigits = 0x00000019, // local monetary fractional digits
24: CurrencySymbol = 0x0000001B, // position of positive currency symbol
25: FractionalDigits = 0x00000011, // number of fractional digits
26: NativeDigitSubstitution = 0x00001014, // native digit substitution
27: FirstDayOfWeek = 0x0000100C, // first day of week specifier
28: FirstWeekOfYear = 0x0000100D, // first week of year specifier
29: LeadingZeros = 0x00000012, // leading zeros for decimal
30: Measure = 0x0000000D, // 0 = metric, 1 = US
31: NegativeCurrency = 0x0000001C, // negative currency mode
32: NegativeNumber = 0x00001010, // negative number mode
33: PaperSize = 0x0000100A, // paper size
34: TimeFormat = 0x00000023, // time format specifier
35:
36: // STRING LCDATA TYPES
37: // Valid Unicode characters
38: AM = 0X00000028, // AM designator
39: PM = 0x00000029, // PM designator
40: CurrencySymbol = 0x00000014, // local monetary symbol
41: DecimalSeparator = 0x0000000E, // decimal separator
42: DigitGrouping = 0x00000010, // digit grouping
43: ListSeparator = 0x0000000C, // list item separator
44: LongDate = 0x00000020, // long date format string
45: MonetaryDecimalSeparator = 0x00000016, // monetary decimal separator
46: MonetaryGrouping = 0x00000018, // monetary grouping
47: MonetaryThousandSeparator = 0x00000017, // monetary thousand separator
48: NativeDigits = 0x00000013, // native ascii 0-9
49: NegativeSign = 0x00000051, // negative sign
50: PositiveSign = 0x00000050, // positive sign
51: ShortDate = 0x0000001D, // short date format string
52: ThousandSeparator = 0x0000000F, // thousand separator
53: TimeSeparator = 0x0000001E, // time separator
54: TimeFormat = 0x00001003, // time format string
55: YearMonthFormat = 0x00001006 // year month format string
56: };
57:
58: public enum LocaleData : int
59: {
60: // LOCALE_ICALENDARTYPE VALUES
61: Gregorian = 1, // Gregorian (localized)
62: GregorianUS = 2, // Gregorian(Always English)
63: GregorianMEFrench = 9, // Middle East French
64: GregorianArabic = 10,
65: GregorianXlitEnglish = 11, // transliterated English
66: GregorianXlitFrench = 12, // transliterated French
67: Japan = 3,
68: Taiwan = 4,
69: Korea = 5,
70: Hijri = 6,
71: Thai = 7,
72: Hebrew = 8,
73: Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later
74:
75: // LOCALE_ICURRENCY
76: PositiveCurrencyPrefixNoSeparation = 0,
77: PositiveCurrencySuffixNoSeparation = 1,
78: PositiveCurrencyPrefixSeparation = 2, // one character separation
79: PositiveCurrencySuffixSeparation = 3, // one character separation
80:
81: // LOCALE_IDIGITSUBSTITUTION
82: DigitSubstitutionContextBased = 0,
83: DigitSubstitutionNone = 1, // use this setting for full unicode support
84: DigitSubstitutionNative = 2, // uses digits based on national conventions
85: // according to LOCALE_SNATIVEDIGITS
86:
87: //LOCALE_IFIRSTDAYOFWEEK
88: Monday = 0, // LOCALE_SDAYNAME1
89: Tuesday = 1, // LOCALE_SDAYNAME2
90: Wednesday = 2, // LOCALE_SDAYNAME3
91: Thursday = 3, // LOCALE_SDAYNAME4
92: Friday = 4, // LOCALE_SDAYNAME5
93: Saturday = 5, // LOCALE_SDAYNAME6
94: Sunday = 6, // LOCALE_SDAYNAME7
95:
96: // LOCALE_IFRISTWEEKOFYEAR
97: FirstDay = 0, // Week containing 1/1 even if single day
98: FirstFullWeek = 1, // first full week following 1/1
99: FirstWeek = 2, // first week with at least 4 days after 1/1
100:
101: // LOCALE_ILZERO
102: NoLeadingZero = 0, // .975 119:
103: LeadingZero = 1, // 0.975
104:
105: // LOCALE_IMEASURE
106: Metric = 0,
107: US = 1,
108:
109: // LOCALE_INEGCURR
110: ParenthesisSymbolNumber = 0, // ($1.1)
111: NegativeSignSymbolNumber = 1, // -$1.1
112: SymbolNegativeSignNumber = 2, // $-1.1
113: SymbolNumberNegativeSign = 3, // $1.1-
114: ParenthesisNumberSymbol = 4, // (1.1$)
115: NegativeSignNumberSymbol = 5, // -1.1$
116: NumberNegativeSignSymbol = 6, // 1.1-$
117: NumberSymbolNegativeSign = 7, // 1.1$-
118: NegativeSignNumberSpaceSymbol = 8, // -1.1 $
119: NegativeSignSymbolSpaceNumber = 9, // -$ 1.1
120: NumberSpaceSymbolNegativeSign = 10, // 1.1 $-
121: SymbolSpaceNumberNegativeSign = 11, // $ 1.1-
122: SymbolSpaceNegativeSignNumber = 12, // $ -1.1
123: NumberNegativeSignSpaceSymbol = 13, // 1.1- $
124: ParenthesisSymbolSpaceNumber = 14, // ($ 1.1)
125: ParenthesisNumberSpaceSymbol = 15, // (1.1 $)
126:
127: // LOCALE_INEGNUMBER
128: Parenthesis = 0, // (1)
129: NegativeSignNumber = 1, // -1
130: NegativeSignSpaceNumber = 2, // - 1
131: NumberNegativeSign = 3, // 1-
132: NumberSpaceNegativeSign = 4, // 1 -
133:
134: // LOCALE_PAPERSIZE
135: USLetter = 1,
136: USLegal = 5,
137: A3 = 8,
138: A4 = 9,
139:
140: // LOCALE_ITIME
141: FormatAM_PM = 0,
142: Format24Hour = 1
143: };
144: }
145: }
You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-dd. The automated oracle verified the year, month and day values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.
I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals. Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.
GUI Test Automation Is Not Child’s Play
Originally Published Thursday, March 12, 2009
There are many approaches to test automation from unit testing to system level testing through the GUI. Of course, the most often discussed approach is the automation approach that drives the GUI to perform some action; or GUI automation. This also happens to be the most controversial approach to test automation, and is perhaps the hardest type of automation to design and develop. One reason why an automated GUI test fails or doesn’t achieve its potential is due to a lack of understanding of the "system" by the tester, which in turn leads to a poorly designed test from the outset.
This problem is especially obvious when people who may have specialized business knowledge but lack a in-depth understanding of the systems they are working on (non-technical testers) are asked to ‘automate’ something. The automation in this case is usually in the form of using record/playback tools or perhaps creating a rote script to drive a keyword-driven framework, and the ‘test’ is usually consists of nothing more than merely mimicking some contrived behavior by the ‘tester.’ In fact, this over-simplified view of test automation is sometimes perpetuated by tool vendors. A manager at one tool vendor said, "By automatically capturing the tester’s process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating."
There are 2 fundamental problems with the above quote, and this approach to ‘automation’ (and I use that term loosely in this context). First, while I don’t totally discount the value of record/playback, and in the right context it could very well be the best approach in a specific situation, the general consensus in the industry is that due to the limited capabilities of record/playback type tools this type of automation is simply one level above using a hoard of monkeys trained to repeat a set of sequences using the keyboard and mouse. In fact, some might suggest trained monkeys may be a better alternative because bananas are much cheaper than the costs of licensing a tool and then realizing that you have to hire someone to try to patch together some proprietary script in an often vain attempt to build an automation test suite that is beyond the reasonable capabilities of a record/playback automation approach. Unfortunately, in either case the organization is usually left with a horrible mess that nobody wants to clean up. Secondly, it makes a ridiculous assumption that testers are too stupid to automate complex scenarios or test automation is a brain-dead, non-interesting, zero-value-add activity.
If simply recording or ‘documenting’ rudimentary scripts that essentially repeat a sequence of ‘hard-coded’ steps over and over again is someone’s idea of well-designed test automation then I would agree that automation is a brain-dead activity. When you automate poorly designed tests, you simply get poorly designed automated tests! And, since I don’t mind calling the kettle black, I will say it…recording a set of actions, or documenting a sequence of steps with hard-coded values to feed into a keyword driven architecture is not test automation! Surely it automates tasks, but well designed automated tests are much more powerful than the production of some crude script that automates the actions performed by someone sitting in front of a computer.
Of course, well designed automated tests require highly skilled professional testers with in-depth knowledge of the systems they are working on, as well as proficiency in programming concepts and languages. Similar to how doctors study anatomy, physiology, pharmacology, immunology, biochemistry, etc., professional testers need to constantly study the various systems they are tasked to test. Developing a well designed automated test is very different than simply using a tool to automatically repeat a sequence of actions. Designing robust tests (automated or not) requires not only incredible creativity and problem-solving skills, but an in-depth knowledge of the system and an understanding of how to manipulate the system programmatically.
But, if you buy into the idea that automation is simple and merely rote recording or documenting some sequence of steps performed by some person then you get exactly that; simplistic repetitive automated actions. Simple automation is simply automated simplicity.
Data-Driven Testing
Originally Published Sunday, January 04, 2009
I am generally not a big fan of static data in test automation, but being a pragmatic person, I know there are clearly times when using data-driven testing is just plain common-sense. For example, data-driven testing is an effective automation approach when designing ‘black-box’ tests for testing an API.
Data-driven testing is a common approach to test automation where static test data is passed to application parameters and the expected result (which is usually also read from static data) is compared against the actual result. This automation approach is effective when the actual result compared to some expected result can be resolved as a Boolean outcome. In other words, if the actual and expected results match the outcome is true and the test passes; otherwise the outcome is false and the test fails. (Of course, if something occurs during the test where there is no actual result then that particular test is usually logged as indeterminate.)
Of course, the key to effective data-driven testing is the data! If we don’t identify the most appropriate data to use in the test then the test case may have holes and we might overlook important information or miss critical anomalies. If we have too much redundant data then we may be simply running unnecessary tests (yes, even with test automation redundant testing is not an efficient use of resources).
Let’s say we had to test an API method such as:
public bool IsValidNetBiosName(string name)
where the return value is true if the string argument passed to the name parameter is a valid NetBIOS name on the Windows operating environment; otherwise it returns false.
With a data-driven testing approach we could use a simple CSV file that contained the string arguments and the expected result for each string passed to the name parameter. A partial sample of the CSV data file would be:
a,true validname,true validnamexxxxxx,true invalidnamexxxxx,false invalidnamexxxxxxxxxxxxxxxxxxxxxxx,false ,false null,false, xx\x,false xx/xx,false x:x,false xxx*xx,false x?xx,false xxxx",false ;,false xxx|xxx,false
(NOTE: null is a special case in which we need to convert the string “null” to a null in the test code, and the test above null is an empty string. An empty string and null are two different things and both must be tested in this case.)
Next, we need to read in the CSV file into our automated test, and perhaps the easiest way I found to read in a text or CSV file in C# is with the File.ReadAllLines method. The ReadAllLines method opens a text file, reads each line of text as an element in a string array, and then closes the file. Once we have a array of all lines in our data file, we simply need to parse each element in the string array into test data and/or expected result, and then compare the actual result against the expected result as illustrated in this example.
1: // Read each line in the entire CSV file into a string array
2: string[] testDataArray = System.IO.File.ReadAllLines("myTestData.csv");
3:
4: // Iterate through each element in the test data file
5: foreach(string test in testDataArray)
6: {
7: // Split each element in each line into an array where the elements are the
8: // test data and the expected result
9: string[] testElement = test.Split(',');
10: string testData = testElement[0];
11: string expectedResult = testElement[1];
12:
13: // Special case for passing a null to the API parameter
14: if (string.Equals(testData, "null",
15: StringComparison.OrdinalIgnoreCase))
16: {
17: if (string.Equals(api.IsValidNetBiosName(null).ToString(), expectedResult,
18: StringComparison.OrdinalIgnoreCase))
19: {
20: result = "Pass";
21: }
22: else
23: {
24: result = "Fail";
25: }
26: }
27:
28: // Compare the return value against the expected result
29: else if(string.Equals(api.IsValidNetBiosName(testData).ToString(), expectedResult,
30: StringComparison.OrdinalIgnoreCase))
31: {
32: result = "Pass";
33: }
34: else
35: {
36: result = "Fail";
37: }
38: }
This is rather simple example, but data-driven testing is effective for unit testing, API testing, and can even be used in automated GUI testing (although data-driven automation may only have limited applicability in GUI automation). I am a firm believer in the KISS principle when it comes to developing automated tests, and the ReadAllLines method is perhaps the easiest and most efficient way to read in data file for data-driven development. Of course, data-driven testing doesn’t solve all problems. Chan Chaiyochlarb has a good post on some pitfalls to watch out for. But, in the right context, data-driven testing can be one approach used in automated testing.
Test Automation: Temporary Test Files
Originally Published Tuesday, December 02, 2008
There are occasionally times during an automated test needs to create a temporary file during the execution of that test. The problem is that often this file is left behind on the system, or even worse stored in some obscure directory on a server. I say worse because those files will be discovered by someone approximately 9.25 months from the time they were created, and that person will spend about 6.5 hours trying to figure out who they belong to before realizing they are no longer needed and can blow them away.
The test designer must decide whether or not to leave test artifacts once created. My general rule of thumb is if the test run is a system level (or dirty room) test then it is probably ok to leave any artifacts from previous tests lying about on the local machine. But, if the test run is a integration or component level (or primarily clean room) test, then in order to preserve the clean room environment (in other words, I want to eliminate or control the number of unknown environmental variables) then I probably want to remove any test artifacts that are created during the execution of a test.
There are many ways to create a file and delete a file in an automated test. But, one of the easiest ways is to use File class members in C# to create a file that will automatically delete itself when the automated test ends. The FileOptions enumeration in File.Create method provides additional options for FileStream objects including a member that will automatically delete the file when the thread that created it closes.
FileStream fs = File.Create(path, bufferSize, FileOptions.DeleteOnClose)
Of course if you wanted to design in the option to delete the file at the end of a test based on some Boolean decision you could simply create the file using WriteAllText(), WriteAllLines() or WriteAllBytes() methods.
Another problem with temporary test files is that the test designer frequently hard-codes a clever name such at “test.txt” to name the temporary file in the test case itself. Then of course, the next test fires off and that test also has a hard-coded file name which also happens to be “text.txt” that will either overwrite the first file, or in the worse case append to it. Again, the tester can easily create a random string generator to generate random strings for use as file names, or use another method that is already built into C#.
Another useful method for creating temporary file names is found in the Path class. The Path class member Path.GetRandomFileName() will generate a random file name. This random file name also includes a random extension which may not be desired, so you can use the Path.ChangeExtension method to automatically generate a randomly named file and change the extension to the desired extension.
string filename = Path.ChangeExtension(GetRandomFileNamePath.GetRandomFileName(), "txt");
Now, we can create a temporary test file at a desired location (say the My Documents folder) on the local machine with a random name and that file will be automatically deleted at the end of the test.
FileStream fs = File.Create(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments),
Path.ChangeExtension(Path.GetRandomFileName(), "txt")),
bufferSize, FileOptions.DeleteOnClose);