One of the most popular random test data generators that I have developed so far has been a tool called CCMaker to generate random valid and invalid credit card numbers. (Sometimes I wonder why that is, but I don’t dwell on it for too long and I haven’t been interrogated by the FBI lately.) Testing forms that require a credit card has always been risky business because you certainly don’t want to use your own card. Often times developers will include a check on web forms or client apps to do a high level verification of a credit card number before sending all the data across the wire to be validated. This early or high level verification prevents flooding the pipe with bad data. So, one test we can do prior to testing the end-to-end scenario is to test to see if and how the developer is validating credit cards numbers prior to submission.

As far as data goes, generating credit card numbers are fairly simple. There is a bank ID number (BIN), there is a number of digits between 12 and 19 depending on the card type, and there is a checksum. So, if we know the valid BINs for each issuing bank, the valid number of digits for each card type, and how to calculate the checksum we can generate valid credit card numbers. (Of course this is a bit oversimplified because many credit and debit card companies are issued multiple BINs and use varying number lengths.)

Testing for invalid credit card numbers should include using numbers that look close to being correct in some way but are slightly altered. For example for the 3 defined equivalent partitions (BIN, length, checksum) there are seven possible invalid combinations (2³ – 1) we could test.

1. Valid BIN, invalid length and valid checksum
2. Valid BIN, valid length, and invalid checksum
3. Valid BIN, invalid length, and invalid checksum
4. Invalid BIN, valid length and valid checksum
5. Invalid BIN, invalid length, and valid checksum
6. Invalid BIN, valid length, and invalid checksum
7. Invalid BIN, checksum and length

This doesn’t mean I run 7 tests and call it good because there are numerous invalid lengths and invalid BINs for the different card types. A common mistake when using an equivalent partition testing approach is to simply plug in values for each combination listed above and call it good. The problem is that there are several hundred BINs and 8 different valid lengths. For example, for just the Discover card there are 829 valid BIN numbers, and for the Maestro cards there are 56 combinations of BINs and card lengths ranging from 12 to 19 numbers in length. This doesn’t include the permutations of the other numbers that compose the entire card number.

The question every tester must ask him or herself every day when designing tests is how many tests do I need to have any reasonable sense of confidence that risk is minimal and the perception of quality is high. Of course, there is no single right answer here and not magic formula, but since we can’t possibly execute every possible positive or negative test we should at least understand that ultimately testing is sampling.

For example, one strategy for positive testing might be to test every valid BIN for every valid card length for any given credit card. For example for American Express I would want to test at least one number with a BIN of 34 and a card length of 15 that satisfies the checksum requirement, and at least one number with a BIN of 37 and a card length of 15 that also satisfies the checksum requirement. For a card type of Visa I would need a minimum of 2 tests in which the BIN is 4, the checksum requirement is satisfied, and one has a card length of  13 numbers and the other has a card length of 16 numbers.

That probably sounds like quite a bit of testing, and tests which most likely would not produce an error (unless of course the BIN is miss identified (e.g. instead of checking for a BIN of 5020 the BIN is incorrectly assigned as 5002), or if a valid BIN is not recognized as valid because it is omitted from a list or enumeration of valid BINs for that credit card). Certainly testing of this magnitude would be expensive if done manually. But when automated using a random test data generator and a data-driven automation approach to set the random generator properties comprehensive testing becomes a much more reasonable proposition and can significantly increase overall confidence.

This is where my CCMaker 3.1 test data generator can help by randomly generating both valid and invalid credit card numbers. The updated CCMaker test automation library has just been posted to my web site with documentation and examples. If you have any questions, or find any issues with the new library please let me know.

For some time I have wanted to add surrogate pair character support to a tool I developed called GString, and this week I managed to find some time to do that work and more! As I developed the methods for surrogate pair support I rewrote (refactored in developer parlance) some of the previous methods to reduce complexity. And wouldn’t you know it…the simple act of refactoring exposed some otherwise hard to find defects (and one pretty obvious one). I discovered these defects because I had to approach the problem space from a different perspective, and that perspective (working primarily with int types instead of char types) exposed the problems.

So, I decided to retire the GString code base, and I ported what I could into a new tool named Babel (and this is my shameless plug for that tool.) I know it is not ‘customer friendly’ when someone goes and renames a tool, especially when it comes with a library for test automation because now the ‘customer’ has to change their references in order to use the functionality in the new DLL. However, the name Babel seems more fitting in the purpose of this tool to generate random characters across the Unicode spectrum of language scripts; and besides Java also has a class called GString and I didn’t want to cause any confusion.

The obvious bug fixed in Babel is a problem that occurred when generating character in the ASCII only range. For some bizarre reason I neglected to exclude Japanese half-width katakana characters (and for an even more bizarre reason I failed to find it; which is a really good reason why unit testing only goes so far and we really need a second set of eyes for sufficient testing). One not so obvious defects included exclusion of a range of code points between U+1A20 and U+1AFF instead of U+1B80 and U+1CFF. This was a classic boundary bug! But unless we did a formal code review it is unlikely this one would have never been found.) The other not so obvious defect that has been fixed involved the the programs inability to exclude some valid Unicode code points that have not been assigned a character if the user selected to exclude unassigned code points (again a similar problem to that described above.)

The good news is these are now fixed, and the new Babel tool also includes support for Unicode surrogate pair characters in the range of U+10000 through U+10FFFF as an option. Also, I included a feature to save the output to a text file rather than having to copy and paste. The installation package include a desktop tool, a DLL for test automation, and the user’s guide and can be found at Testing Mentor.

If you encounter any problem using the tool, or if you have any feedback please let me know. Enjoy!

I am not a big fan of static test data, so this month’s issue of Software Testing and Performance magazine published an article I wrote outlining one approach for generating random string data (although the basic concepts can be used for generating other types of random data).

Unfortunately, it appears that some of the numbers got a little screwed up and the printer did not superscript the exponents correctly so the numbers in the third paragraph are probably looking pretty strange. So, to clarify, the paragraph should read:

Using only the characters ‘A’ – ‘Z’ the total number of possible character combinations using for a filename with an 8-letter filename and a 3-letter extension is 26⁸ + 26³, or 208,827,099,728. If we were assigned to test long filenames on a Windows platform using only ASCII characters (see Table 1), the number of possibilities increases because there are 86 possible characters we can use in a valid filename or extension and a maximum filename length is 251 characters with a 3 character extension is 86²⁵¹ + 86³. Trust me, that is one big number.

(NOTE: There have been several assertions regarding the above formula for determining the number of tests, here is the explanation. Essentially, the Windows platform file system treats the base filename and the file extension as 2 separate components and there is no interaction or dependencies between these two components. (For example, we cannot save a filename as CON.txt, but we can save a filename as myFile.CON.) Since there is no dependencies between the base filename component and the extension component they are treated as 2 independent parameters which would mathematically result in 26⁸ + 26³, or 208,828,082,152 tests if we elected to test all possible combinations of the base filename component with a nominal valid extension, then test all possible extension component combinations with a nominal valid base filename. One could argue we could combine the 17576 unique 3-character extension combinations with various combinations of the 8-character base filename component to reduce the overall number of tests by 17576; however I choose not to use that approach and instead test each parameter independently. If we mistakenly assumed dependency or inter-relationship between the base filename and extension components of a filename on the Windows platform testing all combinations (or 26⁸ * 26³ (or simply 26¹¹) on a Windows OS would result in approximately 3,670,135,659,905,624 redundant tests (if we could do exhaustive testing). This is where in-depth knowledge of the ‘system’ really pays off.)

Of course, the filename length and extension length is variable. Also, 251 characters assumes a base filename component length from the root directory (it does not take into account the MAXPATH constant). So, the total number of combinations using only ASCII characters is much greater because the base filename component length with a ‘default’ 3-letter extension from the root directory is actually 86²⁵¹ + 86²⁵⁰ + 86²⁴⁹ + 86²⁴⁸ + 86²⁴⁷ … + 86¹. Then, of course vary the length of extensions, and the total number of combinations increases even further. But, all this is only to provide some scope the magnitude of the testing problem.

Also, the equivalence class table (Table 2) is simplified and does not include reserved device names. For example, Windows will/should prevent a user from saving a filename of LPT1, or COM6, or CON, etc. (The behavior for saving filenames with strings composed of reserved device names is different on Windows Xp and Windows Vista…Vista finally got it right!).

Unfortunately, I did not get a chance to read the edited copy before print, but I think the basic idea comes through and I hope you find value of using intelligent random test data in your testing and would be interested in hearing your feedback.

Well, for those of you living outside the Pacific Northwest you are probably unaware of the recent wind storm with winds gusting to 60+ miles per hour that left more than 1 million people on the eastern side of the state without power. The damage was pretty extensive, and since I live in a fairly remote area I was without power for more than 7 days and without the Internet for almost 9 days. I do have a generator, but it hadn’t been used in almost 4 years. Sure, I started it every 6 months for about 15 minutes each time, but after the first full day of operation the generator started doing wierd things. So, during the past week I have become pretty good at fixing generators (mine and my neighbors), tracing electrical systems, troubleshooting furnace problems, splitting a lot of firewood, cutting up fallen trees, and repairing fences.

After the sun set (which is quite early) I had little else to do (other than making sure nobody stole my generator), so between stoking the fire I started developing a DLL for Unicode string generation in automated tests based on the GString utility. While reviewing the data tables I created for the GString utility with the Unicode Handbook I noticed some holes (OK…defects). Some of the boundaries for code ranges that are not assigned to any Unicode script group were incorrect. (That will teach me to use a web page with the listing put together by a web developer rather than using the Unicode handbook.) But, I also found a problem that prevented unassigned code points from being generated even if the Only use assigned code points check box was unchecked.So, the (hopefully final) update to GString is complete, including the GString.DLL! So, along with the massive overhaul of the Unicode data tables, the new GString package available from my personal website also includes a new DLL for anyone needing to generate strings of random Unicode characters in test automation. The GString zip file also includes detailed documentation on the utility and the dll usage. Let me know if you have any questions about the tool or using random string generation in your testing.

Well, now back to (mostly) normal life.