Archive for the ‘Testing’ tag
Perceptions of Testing
This week I was in Ho Chi Minh City, Vietnam. After 3 days of attending the VistaCon 2010 conference and 1 day visiting the MS office here I got to explore the city, visit some museums, and end my trip by zipping around the city on a rented scooter. What great fun! Michael Hackett and I rented scooters on Saturday, drove around for a bit, then around noon got separated. Three hours later we met back at the place we rented the bikes and Hung Nguyen was there as well. We once again set out for a coffee shop, then on to a Czech style beer hall well hidden in the middle of the city. After a beer (or two) and a great squid dinner Hung and I headed off to Vive for a few games of pool with his son Denny. Next we headed to a karaoke place I embarrassed myself and also got a treat listening to Hung, Denny, and our companions sing songs in Vietnamese. Today was a continuation of the day before and I spent the day riding out to Hiep Phuoc to explore the countryside a bit. Getting caught in 2 torrential down pours was not part of the plan and forced me to stop once in a small coffee shop for an hour, then a pho stand for another hour to wait out the rain. Away from the city communication is mostly smiles, head movements and hand gestures, but ice coffee is well understood and I knew how to say pho bo to order noodles with beef. All in all, I had way too much fun this week immersing myself in the culture and getting to meet so many wonderfully friendly Vietnamese!
Back to the conference, I was very honored when I was invited to present the opening keynote for VistaCon 2010, the first software testing conference in Vietnam. My dear friends Hung Nguyen and Michael Hackett and the rest of the staff at LogiGear organized a fantastic conference that hosted about 180 people from 6 different countries. I also presented a 2 hour tutorial on combinatorial testing practices, and another talk on random test data generation in automated tests. The talks went well although culturally the Vietnamese people are a bit shy about speaking out, and many opted to talk with me one on one or in smaller groups during breaks or the lunch hour. One attendee later told me, “I didn’t understand everything you said, but your talks really made me think.”
The software industry in Vietnam is growing rapidly, and Ho Chi Minh City has tremendous potential as both an outsourcing destination and for distributed development. So, it was not a surprise to repeatedly hear the recurring question, “What skills will we need as software testers in the future?”
Those of you who read this blog regularly or have listened to me speak at conferences know that I think that software testers should have a rich understanding of the “system.” In my opinion, the less you know about the “system” in which you are testing the greater the potential to miss important issues, or decrease your ability to troubleshoot issues and identify patterns of software testing that can then be applied in the appropriate context.
For example, many testers have heard of the problems with the antiquated double byte encoding systems (DBCS) and will continually cut and paste hard-coded strings containing “problematic” CJK characters into various input textboxes. Unfortunately, much of this ‘testing’ is simply wasted effort. Due to a lack of “system” knowledge they don’t know is that these characters were problematic in file I/O operations on ANSI based systems, or where an application is thunking between Unicode and ANSI. Today the native character encoding of the Windows operating system is Unicode. Different encoding; different problems.
I also think testers should be competent in at least one programming language for several reasons that I have discussed in previous posts. Over the past few years, my message has been pretty clear; to grow as a professional in the software testing discipline many of us must improve our overall technical skills and knowledge. I have leaned in this direction mostly because I saw this skill gap in many testers and understood the needs of companies that produce software will require a greater breadth and depth of skills from their testers. The key operative phrase in that last sentence is “companies that produce software” because that is the lens through which I view the maturing role of software testing. My perspective of testing is biased towards testing practices at companies that produce software.
Cem Kaner also gave a keynote at VistaCon. At first I wasn’t sure if his keynote was on investment banking or testing, but he eventually drew a parallel between the job of “quants” in the financial sector and the role of a business domain expert tester working for an outsourcing company. But he did say that if you work for a company that produces software then you probably want to increase your technical skills, and if you work for a outsourcing company that tests software produced by software producer then you should become a specialist in the business domain of the software you are testing. You know…I agree. If you want to grow your career as a vendor for companies that outsources the testing of their products to specialists that test software from an end-user perspective then you should definitely become an expert in that business domain. On the other hand if you work for a company that produces software, or creates software solutions then I think you need to constantly improve your overall knowledge of the complete system; including some understanding of the domain in which you are testing.
Surely in the future there is demand for testers who are business domain specialist working for outsourcing vendors, and there will be demand for testers who want to work at a company that produces software and will need an increasingly richer set of technical skills, and I suspect that professional testers in the future will have a mix of skills and knowledge that will enable them to adapt to the changing job market and demands of the industry.
Shipping software is a team effort
Since I was young I was involved in team sports. I played little league baseball during my elementary years, in junior high I switched to football, and in high school I played lacrosse. Growing up on the right (east) coast of America a lot of us boys would also play pick-up pond hockey when the rivers and ponds froze over in the winter. Hockey has always been a passion of mine, and I am a big Bruins fan (perhaps because Maryland didn’t have a hockey team), and Bobby Orr was amazing!
There is an ice rink very close to my home, and my daughter started skating at 4 years of age. She has no ambition to be the next Nancy Kerrigan, but she loves to ice skate. She also loves to watch hockey (she is a die hard Penguins fan and her favorite player is Sid The Kid).
The past few weeks I have been involved in a hockey tournament in conjunction with my regular Monarchs schedule, work schedule, and getting the boat ready for the up-coming Leukemia Cup Regatta and summer sailing season. (I’ll use that as an excuse for my long absence from the blogosphere.) In the tournament we won some games, and we lost some games. Eventually our team won the silver medal yesterday afternoon in the final match of the series. One thing all this practice and play has reminded me of is the importance of the team. Whether it is playing hockey or shipping software a team that plays well together is required for success.
Playing well together is more than simply showing up on the ice. Each player has a position, each player is expected to communicate effectively to move the puck around and “read the play,” occasionally players will “block a shot” on goal, and when you are on the ice you give 150% (which is why the shifts (time on ice) are short). Essentially no player can do everything alone, and the weakest link means that others on the team must take up the slack (which eventually leads to mistakes and players getting “out of position”).
I play defenseman, and usually left defense.There are general skills for the game that all players must have, but there are skills and strategies for defense positions that are different than for forwards. And although the 3 forwards and the 2 defensive players must work together as a team to score (or prevent a goal) the responsibilities of each position are different, and each player must play their position.
I have read a lot of analogies between individual sports such as martial arts and the discipline of software testing. These analogies are sometimes good in that they help testers understand why they must constantly learn new things and practice in order to grow in our (or any) professional discipline. Similarly, I could make an analogy between the defensemen on a hockey team and software testers. But, while we might all aspire to be a “testing ninja” (or a Bobby Orr) in our career, we also need a team of other professionals each doing their part to ship a great software product.
When players in the software game are out of position or not playing their best the team is looking for a loss, or at best a very hard earned win (that usually burns players out). For example, program managers who don’t to provide useful customer scenarios or models of potential feature designs, or developers who fail to unit test their code or run unit tests that are only slightly better than “does it compile” put the burden on the test team (the defensive players) to hopefully “score the goal.”
While a defensive player may occasionally score a goal, their primary role (on offense) is to put the puck in front of the opposing net. As defensive players we must be able to read the play (understand our customer scenarios and analyze abstract models of a design and provide critical feedback). Communicate effectively to other defensemen and forwards to get the puck into position, or protect our goal (put the important information in front of those who make the critical business decisions).
At times defensemen need to help the forwards by driving the puck out of the zone, but without the forwards doing their part the defensemen would surely get exhausted very quickly and the outcome would not likely be favorable. Likewise, if the forwards don’t help defend their goal it is likely the opposing team will wear down the defense and score.
So, while we (as testers) sometimes like to think that the success or failure of a software project depends solely on our defensive skills to find bugs and drive in quality, the simple fact is that we need a team of professional to be successful and “quality” depends on the value each player brings to the table. If we don’t start a game with a good strategy and people playing their best in their assigned positions the likelihood of success is marginal at best. A strong defense is critical, but no one ever wins by playing defense.
"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest." – Thomas Reid’s Essays on the Intellectual Powers of Man, 1786
API Testing – Thinking Differently About the Problem
Last year the University of Washington Extension Program started running a new Software Test Automation using C# program that I designed and developed for experienced testers with little or no programming background. The program is very popular and has more than 60 people waiting for the next offering. Unfortunately, the pay is not that great so I have no intention of quitting my day job. It helps with the moorage costs for my sailboat, but the stipend I receive is not my motivation for teaching this course.
A few years ago I realized the industry would once again require software testers to have a richer understanding of the complete ‘systems’ they are testing, and also require testers to have a wider range of ‘testing’ skills beyond emulating user behavior in an attempt to expose as many bugs as possible before the software is released. I also realized there are many testers in the Seattle area who are good testers but simply lacked the coding skills necessary to design and develop automated test cases (that more and more companies are expecting from their testing staff).
So, this program is one way I can help testers in the community gain additional skills and share some ideas with my colleagues in the local community. Don’t tell the program coordinator from UW, but my real reward comes when a student tells me about how he/she was able to solve a test problem using something they learned in class. Frankly, I don’t think I am a really great teacher, but it is nice to think that in some small way I can sometimes help testers unleash their own potential to overcome challenges and succeed.
Anyway, the final project after the first 10 weeks of the course is to design automated tests of 3 simple API methods from a ‘black box’ perspective (e.g. they had to design a test that called the API method in a DLL). Each method required one or more argument variables to be passed to the method’s parameters when it was called in the automated test case, and each method returned a type (bool, int, and string) that had to be checked against the expected result based on the variables used in the test. The final project also introduces data-driven automation concepts. The focus of the project was to reinforce the programming concepts and skills they learned over the previous 9 weeks and put that knowledge and skill to use in a reasonably realistic testing project.
I am a big fan of API testing, and at Microsoft we do a lot of API testing and I would venture to say that a significant portion of our test automation runs below the UI layer banging away at various APIs. If API is broken…well it’s that whole “lipstick on a pig” thing; you might mask it for awhile, but it is still a pig and eventually the lipstick wears off.
Prior to the project I try to set the stage by telling everyone that the key to data-driven testing is dependent on the test data crafted by the tester. If the test data is insufficient you potentially miss a critical error. If the data is wrong then you are likely to throw a false positive; an error or exception thrown by the test and not by the system under test (or API method in this case). If a C# method parameter takes an intrinsic data type of int (Integer32) then trying to pass a string variable into the test case from a test data file to that parameter will throw an exception in the test code well before it makes the call to the API method being tested.
For example, the simplified sample test case below is testing a simple API static method ConvertValueToUnicodeChar(int value) that takes a integer value and converts it to a UTF-16 Unicode character. If the integer value is outside the UTF-16 range (0 through 65535) the method ConvertValueToUnicodeChar(int value) will throw an ArgumentOutOfRangeException.
1: // <copyright file="simpletestcase.cs" company="TestingMentor">
2: // Copyright © 2009 by Bj Rollison. All rights reserved.
3: // </copyright>
4:
5: namespace TestingMentor.Sample
6: {
7: using System;
8: using System.IO;
9: using TestingMentor.Simulation;
10:
11: class TestCase
12: {
13: static void Main(string[] args)
14: {
15: int testCounter = 0;
16: // Read in an array of strings representing the test data.
17: // Of course this would likely come from a static test data file
18: // on a server or copied to a folder on the local machine
19: string[] testData = new string[]
20: { "90,Z",
21: "24798,惞",
22: "0,null",
23: "65536,Error",
24: "-1,Error",
25: "1.5,",
26: "xyz,xyz"
27: };
28:
29: // Loop through each test data string
30: foreach (string test in testData)
31: {
32: testCounter++;
33: // This nested try/catch block catches invalid test data
34: // but allow additonal tests in the testData array
35: try
36: {
37: // Parse each string into the test data and expected result
38: string[] testElement = test.Split(',');
39: string expectedResult = testElement[1];
40: string actualResult = String.Empty;
41:
42: // Convert the string to a type int value
43: int value = int.Parse(testElement[0]);
44:
45: // We need a way to handle int values 0 through 32 which are
46: // control characters, this is an example of how to deal with
47: // a int value of 0 which is a null character
48: if (expectedResult.Equals("null", StringComparison.OrdinalIgnoreCase))
49: {
50: expectedResult = '\0'.ToString();
51: }
52:
53: // This nested try/catch block tests catches exceptions thrown by
54: // the method under test. If the method under test throws an
55: // exception we certainly want to test for that case!
56: try
57: {
58: // Call the API method under test
59: char result = Converter.ConvertValueToUnicodeChar(value);
60: actualResult = result.ToString();
61: }
62:
63: catch (ArgumentOutOfRangeException)
64: {
65: actualResult = "Error";
66: }
67:
68: catch (Exception)
69: {
70: // if this happens this is a failure because the documentation
71: // states that this method will only throw an
72: // ArgumentOutOfRangeException.
73: actualResult = "Non-specific or unexpected error thrown";
74: }
75:
76: // Call a simple oracle and log results
77: if (String.Equals(actualResult, expectedResult))
78: {
79: // log pass
80: Console.WriteLine("{0} Pass", testCounter);
81: }
82: else
83: {
84: // log fail...of course log as much detail as possible
85: Console.WriteLine("{0} Fail", testCounter);
86: }
87: }
88:
89: catch (FormatException)
90: {
91: // log the test data for this test as incorrect, test is skipped
92: Console.WriteLine("{0} Bad test data. Test skipped.", testCounter);
93: }
94: }
95: }
96: }
97: }
Instead of reading in test data from a file I simply created a string array called csvTestData to simulate a partial list of test data that might be contained in our csv formatted test data file. Notice that the test data on lines #25 and #26 are invalid integer types. So, when these test data variables are converted from strings to type int values in line #43 the int.Parse method will throw a FormatException which is caught by the outer catch block on line #89, marked as bad data and the oracle is skipped. Of course, we want to test the integer values that represent the physical boundaries for a UTF-16 char in C# (which are 0 and 65535) and the values immediately above and below those values (e.g. –1, 0, 1, 65534, 65535, and 65536). Then of course, we need to determine how many samples from the population of possible input variables (integer values between 0 and 65535) we need to test to attain a reasonable degree of confidence that the API method would return the correct UTF-6 Unicode character for a given integer value. (or in this case the population of test data is relatively small and we could simply run through all 65536 values because it would only take a minute or two).
Unfortunately, some of the test data files submitted in the final project contained invalid test data for the API method being called. In some test cases the parameter type required was a type int, but the test data read in from the file for that parameter was a real number such as 1.5, or a string such as “xyz” similar to the example above. I asked myself why would someone include these variables in a test that are being passed to a parameter of type int? The only thing I can think of is that when these testers designed their test data files, they were thinking about the problem as if they were testing the API method through a user interface. (And, in fact my suspicion was confirmed later when I asked them.)
The bottom line here is that we often times throw a lot of ‘tests’ or a lot of data at something in an attempt to trigger an unexpected error. Sometimes we are successful, and hopefully we document that information and share it with others so we can all learn. But, a lot of times it seems we can’t see the trees because of the forest and execute tests or include test data in our tests just for the sake of physical activity. I sometimes wonder whether or not it matters to think critically about the problem, analyze the situation, and design well-thought out tests, or is simply throwing stuff against the wall and seeing what sticks good enough testing?
Random Test Data Generation
Originally Published Wednesday, May 30, 20
I am not a big fan of static test data, so this month’s issue of Software Testing and Performance magazine published an article I wrote outlining one approach for generating random string data (although the basic concepts can be used for generating other types of random data).
Unfortunately, it appears that some of the numbers got a little screwed up and the printer did not superscript the exponents correctly so the numbers in the third paragraph are probably looking pretty strange. So, to clarify, the paragraph should read:
Using only the characters ‘A’ – ‘Z’ the total number of possible character combinations using for a filename with an 8-letter filename and a 3-letter extension is 268 + 263, or 208,827,099,728. If we were assigned to test long filenames on a Windows platform using only ASCII characters (see Table 1), the number of possibilities increases because there are 86 possible characters we can use in a valid filename or extension and a maximum filename length is 251 characters with a 3 character extension is 86251 + 863. Trust me, that is one big number.
(NOTE: There have been several assertions regarding the above formula for determining the number of tests, here is the explanation. Essentially, the Windows platform file system treats the base filename and the file extension as 2 separate components and there is no interaction or dependencies between these two components. (For example, we cannot save a filename as CON.txt, but we can save a filename as myFile.CON.) Since there is no dependencies between the base filename component and the extension component they are treated as 2 independent parameters which would mathematically result in 268 + 263, or 208,828,082,152 tests if we elected to test all possible combinations of the base filename component with a nominal valid extension, then test all possible extension component combinations with a nominal valid base filename. One could argue we could combine the 17576 unique 3-character extension combinations with various combinations of the 8-character base filename component to reduce the overall number of tests by 17576; however I choose not to use that approach and instead test each parameter independently. If we mistakenly assumed dependency or inter-relationship between the base filename and extension components of a filename on the Windows platform testing all combinations (or 268 * 263 (or simply 2611) on a Windows OS would result in approximately 3,670,135,659,905,624 redundant tests (if we could do exhaustive testing). This is where in-depth knowledge of the ‘system’ really pays off.)
Of course, the filename length and extension length is variable. Also, 251 characters assumes a base filename component length from the root directory (it does not take into account the MAXPATH constant). So, the total number of combinations using only ASCII characters is much greater because the base filename component length with a ‘default’ 3-letter extension from the root directory is actually 86251 + 86250 + 86249 + 86248 + 86247 … + 861. Then, of course vary the length of extensions, and the total number of combinations increases even further. But, all this is only to provide some scope the magnitude of the testing problem.
Also, the equivalence class table (Table 2) is simplified and does not include reserved device names. For example, Windows will/should prevent a user from saving a filename of LPT1, or COM6, or CON, etc. (The behavior for saving filenames with strings composed of reserved device names is different on Windows Xp and Windows Vista…Vista finally got it right!).
Unfortunately, I did not get a chance to read the edited copy before print, but I think the basic idea comes through and I hope you find value of using intelligent random test data in your testing and would be interested in hearing your feedback.
Allpairs, Pairwise, Combinatorial Analysis
Originally Published Wednesday, October 25, 2006
Last week I went to StarWest as a presenter and as a track chair to introduce speakers. Being a track chair is wonderful because you get to interface more closely with other speakers. Anyway…one of the speakers I introduced was Jon Bach. Jon is a good public speaker, and I was pleasantly surprised that he was doing a talk on the allpairs testing technique (also known as pairwise or combinatorial analysis). I wish Jon dedicated a little more time to the specifics of the technique during his talk and was generally more aware of available tools and information for folks to investigate further, but I think he successfully raised the general awareness and interest in pariwise testing as an effective testing technique among the audience.
Pairwise testing is one approach to solving the potential explosion in the number of tests when dealing with multiple parameters whose variables are semi-coupled or have some dependency on variable states of other parameters. For example, in the font dialog of MS Word there are 11 checkboxes for various effects such as superscript, strikethrough, emboss, etc. Obviously these effects have impact on how the characters in a particular font are displayed and can be used in multiple combinations such as Strikethrough + Subscript + Emboss. The total number of combinations of effects is the Cartesian product of the variables for each parameter, or 211 or 2048 in this example. This doesn’t include different font types, styles, etc. which also interdependent. So, you can see how the number of combinations increases rapidly especially as additional dependent parameters are included in the matrix.
The good news is the industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters. So, from a risk based perspective where it may not be feasible to test all possible combinations how do we choose the combinations out of all the possibilities? Two common approaches include orthogonal arrays and combinatorial analysis.
But, true orthogonal arrays require that the number of variables is the same for all parameters. (Rarely true in software.) It is possible to create "mixed orthogonal arrays" where some combinations of variables will be tested more than once. For example, if we have 5 parameters and one parameter has 5 variables and the remains 4 parameters only have 3 variables each, we can see from the orthogonal array selector (available on FreeQuality website) the size of the orthogonal array is L25 (which basically means the test case will require 25 tests which is still significantly less than the total number of combinations of 405).
The other approach is combinatorial analysis (often referred to as pairwise or allpairs testing) because the approach most commonly used is to use a mathematical formula to reduce the total number of combinations in such a way that each variable for each parameter is tested with each variable from the other parameters at least once. In the above example, the number of tests would be reduced to 16. (Note: some tools will give slightly different results.) However, some tools (such as Microsoft’s PICT) also allow for more complex analysis of variable combinations such as triplets and n-wise coverage.
One problem that is hopefully not overlooked by testers using these tools is that some combinations of variables are simply not possible. For example, in the Effects group of the Font dialog it is impossible to check the Superscript checkbox and the Subscript checkbox simultaneously. Therefore, the tester either has to manually modify the output, or use a tool that allows constraints. Again, this is another situation where Microsoft’s free tool PICT excels. PICT uses a simply basic-like language for conditional and unconditional constraining of combinations of variables. PICT also allows weighting variables, seeding, output randomization, and negative testing.
I didn’t want this to be a PICT sales job, but alas my bias has influenced this post. So, I will conclude by pointing the readers to the Pairwise Testing website. My colleague Jacek Czerwonka has pulled together great resources on the technique of combinatorial analysis including a list of free and commercially available tools, and white papers supporting the value and practicality of this testing technique.
The Purpose of a Good Test Case
Originally Published Friday, September 15, 2006
Many experts have written articles and devoted chapters of books on the attributes of what constitutes a ‘good’ test case. Unfortunately, most books repeat a common (yet limited) perspective that a good test case has a high probability of finding bugs, and Kaner goes to the extreme by stating “A test that did not reveal a problem was a waste of time.” I pondered this for a while and thought, if I run a test to prove that a feature functions as expected is that really a waste of time? Isn’t it a good thing that testers actually spend some time executing tests that demonstrates the product does what the customer expects it to do? Or do we simply want to restrict the value of testing and reduce our testers to keyboard monkeys who bang away at the keyboard in search of bugs? (I believe limiting the perspective of the purpose of a test in testing literature has only perpetuated the faulty assumption that the purpose of testing is to simply find bugs; a point I previously dismissed). So, what is the purpose of a test?
Jorgensen states in his book, Software Testing: A Craftsman’s Approach that “a test case has two purposes: either to expose an error, or to demonstrate correct execution.” This explanation broadens the purpose of a test case to include both verifying the product meets the requirements, and revealing potential errors or defects. I tend to agree more with Jorgensen in this matter, but also think the purpose of a test case involves even more.
There are several objectives of any effective test whether manually executed or automated. The rationale for test case usually falls into one of seven categories. Each category provides value to the organization in different ways, but they all essentially function to reduce risk and qualify the testing effort. So, a good test provides some measurable value to the organization with the explicit purpose of:
- Verifying conformance to applicable standards and guidelines and customer requirements
- Validating expectations and customer needs
- Increasing control flow coverage
- Increasing logic flow coverage
- Increasing data flow coverage
- Simulating ‘real’ end user scenarios
- Exposing errors or defects