I.M. Testy

Automating Screen Captures

Friday, September 3, 2010

When I first started at Microsoft I worked on the Windows 95 international test team. Not only did we focus on globalization testing, but we also did a lot of the localization testing for the East Asian language versions (Japanese, Korean, Simplified Chinese, and Traditional Chinese). Localization is the adaptation of software to a particular target market. Translation of resource strings is one of the most visible parts of the localization process, and in those days part of our testing effort was spent on translation validation (e.g. checking the strings for appropriate translation). (In retrospect, I now think it is a huge waste of time and resources to use professional testers to validate the translation “quality.”) During our localization testing cycles it was common practice to take screen shots of issues we found during localization testing. These screen shots often help put things in context for the localization engineers, and helped them troubleshoot the issue. Of course, there were other times when we took screen shots to put other anomalies in context.

I am generally not a big fan of just attaching screen images to bug reports carte blanche. Joe Strazzere has an excellent post describing why a screen shot doesn’t always add value in bug reports. But, I also know that there are times when screen shots of the desktop can be of value. When we are testing using pre-defined tests or exploratory approaches we are physically there to see anomalies as they occur. But, (hopefully) nobody is babysitting the machines our automated GUI test scripts are running on. So, when an unexpected anomaly occurs with automated GUI test scripts is might sometimes be beneficial to capture the desktop state as an image. This screen capture might provide some clues as to the state of the desktop at the time of unexpected automated test case failures resulting in an indeterminate automated test script result.

There are several 3rd party tools that some testers use to capture the desktop image and save it to a file. Automated test frameworks should also have methods that test developers can call to capture screen shots at important points during the execution of an automated test script, or when an anomaly occurs. But, if not, here is a simply method that will take a snapshot of the desktop and save a jpeg file of the desktop state.

   1: using System.Drawing;

   2: using System.Drawing.Imaging;

   3: using System.Windows.Forms;

4:

   5: ...

6:

   7: public static void GetDesktopImage(string imageFilePath)

   8: {

   9:   // Declare a structure for the width and height of the desktop

  10:   Rectangle rect = Screen.GetBounds(Point.Empty);

11:

  12:   // Declare a new instance of the bitmap class

  13:   Bitmap image = new Bitmap(rect.Width, rect.Height);

14:

  15:   // Capture the desktop

  16:   using (Graphics desktopImage = Graphics.FromImage(image))

  17:   {

  18:     desktopImage.CopyFromScreen(Point.Empty, Point.Empty, rect.Size);

  19:   }

20:

  21:   // Save the image to the specified path & filename

  22:   image.Save(imageFilePath, ImageFormat.Jpeg);

  23: }

Overall it is a pretty simple process. Basically we get the size of the desktop, use those values to declare a new instance of the bitmap class, and finally capture the desktop as a jpeg image and save it to the specified location. Also notice that you need to also add the System.Drawing, System.Drawing.Imaging, and System.Windows.Forms references to the class where this method lives.

I would not capture tons of images during a test run; they just aren’t that valuable. And, I do not advocate capturing images to use as oracles…they are just too unreliable in my opinion. But, there are times when a screen capture of the desktop might add value and provide some context for the tester or the developer in troubleshooting issues.

Code Coverage: Unreachable Code and Hard to Reach Code

Friday, August 27, 2010

Well, I am back from a sailing excursion to the San Juan Islands. I wanted to go to the Gulf Islands, but considering an unexpected ordeal with a kidney stone just before taking off on the trip I decided it might be better to be a bit closer…just in case. The weather was great, and we spent a lot of time exploring Stuart and James Islands, and dropped into Roche Harbor the first night and Friday Harbor the last night in the islands. We limited out on Dungeness crabs on all but 2 days where we only managed to get 3 legal size crabs on those 2 days. Basically this translates to a lot of crab cakes in the freezer…yum! This was the first time my daughter went crabbing with me. My daughter would ride out in the dinghy with me to check the pots, and would point out the male crabs for me, but she wouldn’t reach in an help me throw back the females or undersized males. Come to think about it, she didn’t help me cook and clean the crabs either…she just ate the crab cakes we made on the boat. I think the rules might have to change next year! All in all it was a great month decompressing and recharging, and contemplating my personal and professional future.

But enough about me. My last 2 posts have been discussing code coverage analysis. The primary purpose of using code coverage tools as either a developer or as a tester is not to try to obtain some magical ROMA number. The biggest value of measuring code coverage is to help us analyze untested areas of code and make informed decisions of whether or not we need to design additional tests to increase test coverage and help reduce exposure to risk.

The last post illustrated how we might use code coverage results to help us design additional tests we might have missed during the execution of any pre-defined tests (automated or manual) and additional exploratory testing efforts. But remember, the goal is simply not to design tests in order to get the tool to report 100% code coverage. In fact, in just about any complex system executing 100% of the statements in code may not be feasible or provide any practical value. This is generally referred to as unreachable code.

For example, let’s look at this (albeit antiquated) code snippet.

The code coverage tool is indicating that this conditional statement has been exercised to its true outcome, but not it’s false outcome. This was a common approach used in 16-bit applications to prevent multiple instances of the same application on a single machine. However, in the 32-bit world hPrevInstance always returns null, which means there is no practical way to make this conditional statement return false.

This is a bit of an obscure example, but is used to illustrate why a greater understanding of the programming language used by the development team would help testers from banging their heads against the wall ‘trying’ different things until someone realizes we could never make this conditional statement return false. By analyzing this section of the code coverage results we might suggest refactoring for a Win32/64 environment, or at least be able to explain why this conditional will not return false. (Remember…it’s all about information.)

Another example of unreachable code is sometimes caused by coding style or possibly unnecessary code. For example, the following lines are also in the WinMain() function that is called when the ‘user’ launches the application.

In this situation when the application initially starts and calls the WinMain() function these 2 conditional statements in WinMain() determine whether the Frog and Car bitmaps are true. Since we just launched the application and the bitmaps have not yet been loaded by any other calls to LoadBitmap() then the conditional statements in lines 104 and 109 will never go true, and lines 105 and 110 will never be executed. Again, following an analysis of the section of the code we can provide information regarding why we can’t design a test to cause these conditional statements to return true without fault injection or code mutation. Additional information that we might provide based on our analysis of the code coverage results may be a suggestion to refactor this code to improve testability.

A similar example of unreachable code is a common coding style involving switch statements where developers included a case statement for each possible value, and also included a default statement. For example in the last post we saw how we saw this code chunk which is essentially the menu structure.

When the menu item is selected the submenu displays the submenu items Start and Exit. When the submenu is displayed the only actions possible is to select the Start submenu item (line 270), or the Exit submenu item (line 274). Without fault injection there is no practical way to execute the default statement in line 277. Again, this may be another example where refactoring could improve testability because if the default statement is removed control flow would simply pass out of the switch block.

However, this is not always the case with switch statements. Here is an example in which a modal message box is displayed and draws 2 buttons; a yes button to restart the game, and a no button to quit the game. But notice that the default case statement (line 295) has not been executed.

In this situation if we launch the game, move the frog to get hit by a car this modal message box will display showing the ‘end user’ 2 possible buttons to press (in this case the ESC key or the Close control button in the upper right corner of the modal dialog are not possible options). However, if we put the game into a state where this modal dialog is displayed and then kill the application process using Windows Task Manager control flow will pass to line 295 as the process terminates.

Of course, it may not be practical or reasonable to terminate the application process from every possible machine state. Also, this simply increases the costs of testing without adding any real practical value. Providing this information to the decision makers along with suggestions to refactor and improve testability to reduce overall testing costs is another way code coverage analysis can be a valuable tool in a tester’s toolbox.

Another example of hard to reach code. In this case the conditional statement is if the RegisterClass() function fails then we want to return false.

The RegisterClass() function is also called within the WinMain() function when the application initially launches. So, while analyzing the code coverage results the question we ask ourselves is, “Without fault injection can we make the conditional statement in line 88 return true, and if so how?”

Well, we can. All we have to do is launch about 450 instances of this application to cause line 88 return true. Now, we have to ask ourselves, “What value does this test provide?” Especially since the code design should only allow 1 instance of the application (although it fails to do that because it is a 16-bit app running on a 32-bit environment and that is the nature of hPrevInstance as explained earlier.

From a testing perspective the primary goal of code coverage is not to achieve some magic number; the objective of code coverage is to analyze code coverage results in order to

Improve test coverage
Reduce overall risk
Potentially increase testability of the project

The code coverage number is not really useful information to anyone. It is the analysis of the code coverage results that can help us decide whether we need to design additional tests, identify areas of the code that can’t be executed without even more expensive testing such as fault injection and/or code mutation, or refactor the code to improve testability (which often increases the code coverage measure).

But, this is not to suggest that we should employ code coverage and analyze the results for all software projects. Analyzing code coverage results and designing additional tests from a white box perspective, or refactoring code are all additional expenses for any project. For each project we (or our managers) must decide whether the cost is worth the improved coverage and potentially a reduction in overall risk.

Another way to look at it…it is our responsibility as testers to provide valued information to the decision makers. If the only information we are providing is that we achieved 80% code coverage then we really aren’t doing an effective job. Yes, many managers are number focused; however, the valuable information is in the rest of the story about the 20% that has not been executed.

Filed in General Testing Topics, Testing Practices | Tagged Code Coverage, White Box Testing | Comments (0)

Code Coverage: Finding missing tests

Sunday, August 22, 2010

[NOTE: This was written last week but due to a glitch did not get automatically posted before I left on a boat trip where I disconnected from the world. How refreshing…but more about that later.]

Well, we got a buy in the quarter final play-offs, and we won our semi-final game against the Sockeyes (2 – 0) last Tuesday. Unfortunately, while celebrating with my teammates I was literally knocked to the floor by a kidney stone in my bladder. After a trip to the hospital I am now taking a battery of pain-killers until I pass the stone. Unfortunately, this put me on the injured roster and knocked me out of the final game and until I get an OK from the doctor. Fortunately, I am not prone to kidney stones and this shall pass (literally). I have only had one kidney stone in the past and that was about 20 years ago. For those of you who have not experienced a kidney stone, trust me on this…be very, very glad and I hope you never experience this malady.

Last week I attempted to to illustrate how we might achieve high levels of code coverage (structural control flow) but potentially overlook critical tests, especially from a ‘black-box’ testing approach. The bottom line message…high code coverage does not necessarily equal good test coverage. In reality it is really unlikely to get 100% measured code coverage of an reasonably complex application under test. Unfortunately this often begs the question, “What is the right amount of code coverage for [my] application?” To which I have heard several leads/managers reply, “Our goal is 80% code coverage?” Really? C’mon…that’s just plain ROMA data. Setting arbitrary goals for code coverage is about as pointless as tits on a boar hog. The real answer is that we simply don’t know what measure of code coverage is the ideal level for any product.

Also, for those who have read this article will know that regardless of the testing approach at some point the effectiveness of our tests to hit untested areas of code diminishes. While more time and effort may increase data flow coverage and expose issues, it is unlikely to increase control flow (code) coverage. Remember, just because you exercise a line of code doesn’t mean you found all the bugs, but you have 0% probability of finding any bugs in any untested code.

Fortunately, the majority of customers will traverse the same code paths covered by many of our tests. Also, if your team/organization does robust unit testing then there is a good probability that unit tests at least provided some minimal level of code coverage. (NOTE: While I highly recommend that unit tests and coverage results should be transparent to the test team, I do not recommend using unit tests as part of the battery of tests designed by the test team and executed against the whole build to measure code coverage.) So, there are a couple of questions we have to ask ourselves.

“Does the untested code present significant risk to our customers?”

“Do we need to reduce exposure to risk in the untested areas of code?”

“What is the most efficient way to effectively evaluate the untested code?”

As I wrote last week, code coverage is not about the number. Code coverage is about analyzing the results and potentially designing additional functional tests, or at least being able to explain why areas of the code are untested. If we determine that it is important for our business to better understand the untested code, or improve overall confidence and reduce potential risk then we should use a tool to measure code coverage. But, again it is not about simply measuring code coverage and reporting some magical metric.

Code coverage analysis is the most efficient method to help testers evaluate untested code. Code coverage analysis basically involves the tester reviewing untested code reported by the code coverage tool and determining why some code was not exercised, and possibly design additional tests to exercise the previously untested code. (Remember I also wrote last week the future of professional testing is about analyzing information and designing tests…so, here we go!)

Missing tests

For several years I’ve used the triangle simulation to help set a ‘test effectiveness’ baseline for new testers who had never been formally trained in different test techniques, patterns, or approaches. After a few years of analyzing the results we found that there was about a 70 to 75% probability of tests exercising true branch of the first conditional expression in a compound predicate statement in a key method in the program. There was about a 20 to 25% chance of tests exercising the true branch of the second conditional expression, and there was less than a 10% probability of tests exercising the third conditional expression in the predicate statement. When I found these results I could hardly believe it, so I changed the third conditional expression to inject a bug and sure enough the results held true; in any class of 20 people on average only 1 or 2 people found the bug in the software.

From a black box approach let’s say our tests used the following values for sides A, B, and C respectively:

1, 2, 3 – an error message indicating the values would not produce a triangle
2, 1, 3 – an error message indicating the values would not produce a triangle
4, 5, 6 – scalene triangle
2, 1, 2 – isosceles triangle
5, 5, 5 – equilateral triangle

In this case our code coverage tool would report our coverage is less than 100%. As we drill down we see that the IsValidTriangle() method illustrated below is not completely covered. So, (assuming the arguments values passed to the parameters in this method are all validated to be greater than 0) we analyze the code below in our coverage tool and realize that we need a test to evaluate the third conditional expression to true (e.g. 1, 3, 2 for sides A, B, and C respectively).

   1: internal bool IsValidTriangle(int sideA, int sideB, int sideC)

   2: {

   3:   bool result = true;

   4:   if ((sideA + sideB <= sideC) || (sideB + sideC <= sideA) || (sideA + sideC <= sideB))

   5:   {

   6:     result = false;

   7:   }

8:

   9:   return result;

  10: }

This is a simple example, and things aren’t always so easy in the ‘real-world.’ So, let’s look at something a bit more complicated. In this example, we are testing a game (Frogger) and discover that we called the MoveFrogLeft function, but we didn’t evaluate the conditional expression in line 499 to its false outcome as illustrated below by the red arrow and blue ‘T’.

For this we must find where the frogExists variable is set to true (or 1). So, we search the code for the frogExists variable and soon find the FrogStart() function where the frogExists variable is set to 1.

Now we must find when the FrogStart() function is called and we find the following case statement that basically deals with sending windows messages from the window menu (WM_COMMAND msg). In this example, when we start the game and click the Start menu item (IDM_START) the StartWorkers() and FrogStart() functions are called.

So, now we need to put the puzzle pieces together and design a test that will make the conditional expression in line 499 to evaluate false. In this case the test will be to launch the game, then press the left arrow key before clicking the game’s Start menu item.

These are rather simple examples, but should serve to illustrate how we can use code coverage tools and analyze the results to help us design additional tests from a white box test design approach to identify additional tests to increase control flow (code) coverage and improve functional testing effectiveness.

Next week I will discuss unreachable code.

Filed in General Testing Topics, Testing Practices | Tagged Code Coverage, White Box Testing | Comments (2)

Code Coverage: Did we run the right tests?

Monday, August 9, 2010

It has been awhile since my last post, and I apologize for the few folks who follow my rants. Over the last few months I have been busy working on my boat (she is almost 30 years old now and needs some major refitting). Since starting to play hockey again I have been spending a lot of time at the gym and on the ice and trying to keep my body from getting too battered in the games (for I too am older now…let’s just say more than 30). I have also done a bit of soul searching on the past year or so reflecting on the high points and the low points as well.

So, the refitting on the boat is almost complete and she is ready for a nice long trip to the Gulf Islands starting next week. The Monarch’s Div 7A team finished the regular summer hockey season in first place and playoffs are this week. Winter season doesn’t start until October so I have a month and a half to recuperate. And, I have contemplated a few things to help nurture my professional and personal growth that I will proactively begin working on…beginning with a vacation to the Gulf Islands next week.

But, for now I want to talk more about code coverage. Not the number! Forget the measure! The code coverage percentage is simply a magic metric for pointy-haired managers and other thoughtless chowderheads who like to wave it around as if it actually means something. Look…as a manager I really don’t give a rat’s butt about what percentage of coverage was achieved by testing. If you tell me that your testing achieved 80% code coverage I will probably say, “That’s great! Now tell me about the 20% of the code that you didn’t test!” As a manager what I want to know is:

Did we run the right tests? Did the test suites we ran give us the information that we need to make good business decisions?
What is my potential exposure to risk in the areas of untested code and how do/can we reduce that risk?

I have said for years now that the future of professional testing is not simply about beating on a product and finding bugs. The future of testing lies in our ability to design effective tests and critically analyze results in order to provide better information to our primary customers (the decision makers). Code coverage is about analyzing the results and potentially designing additional functional tests, or at least being able to explain why areas of the code are untested.

Did we run the right tests?

The code coverage metric simply measures the number of statements in the code have been exercised by monitoring control flow through the code. Control flow through code is sequential until it hits some type of branch such as an if statement, a for loop, or an exception. For example, when we call the CalculateMonthlyMortgage method below control flows sequentially from the statement in line 6 to the statement in lines 7 and 8 and finally to the return statement in line 9.

   1: public static double CalculateMonthlyMortgage(

   2:   double principle,

   3:   double annualInterest,

   4:   int months)

   5: {

   6:   double monthlyInterest = (0.01 * annualInterest) / 12;

   7:   double monthlyPayment = principle * monthlyInterest /

   8:     (1 - Math.Pow((1 + monthlyInterest), (-1) * months));

   9:   return monthlyPayment;

  10:  }

So, by passing in a value of 250000 for the principle, 4.5 for the annual interest rate, and 360 for the months parameter the method would return an expected value of 1266.71327456471 and we would measure 100% coverage of this method. That’s a happy path test. Yes, it does what we think it is supposed to do. But, what happens when we pass in a value of 0 to the annualInterest parameter? Control flows through the method as described above giving us 100% code coverage, but the return value is NaN, or Not a Number. Similarly, if the value for the months parameter is 0 we get 100% code coverage and the return value is Infinity. Of course, negative values for any of the parameters would also produce undesirable output results.

This example illustrates sequential control flow through a method that contains simple statements. When branching conditions occur in software the control flow through the method becomes a bit more complicated as does the testing required. The following snippet counts the number of characters in a string. But, to prevent a null reference exception we use a predicate or conditional statement in line 4 that branches control flow from line 4 to line 12 if the input argument is null, and from line 4 to the loop structure starting in line 6 if the string is not null. If the input argument is an empty string control will flow from line 6 to line 10 bypassing the inner block that increments the count variable. But, if the input argument is a string of at least 1 character control flows from line 6 to line 8 and loops back to line 6 until all characters in the string are counted. (This previous post shows how you can create a model or control flow graph to map control flow through an algorithm.)

   1: public static int CharacterCounter(string input)

   2: {

   3:   int count = 0;

   4:   if (input != null)

   5:   {

   6:     foreach (char c in input)

   7:     {

   8:       count++;

   9:     }

  10:   }

11:

  12:    return count;

  13: }

With 3 tests (null, empty string, and a string of at least 1 character) we can achieve 100% code coverage of this method. However, there are still 2 basic problems. The first problem is that a null and an empty string both produce an output of 0. This may be desired output, but not in all cases. Secondly, if the input argument is the string “ꇔㄣᦅrえꞄ௏Жᾁ” the return value will be 11 instead of the expected 10. This is because the last character in the string is actually a Unicode surrogate-pair character composed of 2 UTF-16 code points. (Assuming that 1 character glyph is 1 byte or one UTF-16 encoded code point value is a common mistake in string parsing.)

These are just 2 simple examples designed to illustrate how we can get high levels of code coverage yet still have bugs lurking in our software. Code coverage tools tell us whether we exercised code; it doesn’t tell us if we ran the right set of tests to expose potential issues.

Next week let’s continue this with some thoughts on analyzing code coverage results to help reduce risk.

Filed in General Testing Topics, Testing Practices | Tagged Code Coverage, White Box Testing | Comments (4)

Should we use boundary values in our combinatorial tests?

Thursday, July 1, 2010

I have been a little busy at work lately designing 2 new advanced software testing courses. One of the courses is on combinatorial testing. The course focuses primarily on feature decomposition to identify input parameter interactions, modeling input variables, using the more advanced features of our PICT tool to customize the model file, how to generate a variety of subsets of combinatorial tests from a single model to increase test coverage using PICT, and how to design oracles for data-driven automated combinatorial tests.

In this particular course I used the Page Setup dialog in Paint as a feature to model in one of the exercises. And as it turns out, this was a good choice because as it turns out, it has made me rethink how to model input variables for use in combinatorial testing.

Paint’s Page Setup Dialog

I generally don’t advocate hard-coding specific values for input parameters that have a linear range of values. The reason should be reasonably obvious; if we have a range of values from 1 to 100, and I hard-code the values of 1, 10, 50, and 75, 100 (for a positive test) then I have absolutely 0 probability of ever including the value of 42 in combination with other input parameters. To avoid hard-coding values I usually recommend creating equivalent partitions of appropriate input parameters (e.g. xsmall (1-10), small (11 – 25), medium (26-50), etc). Modeling a range of input values using equivalent partitions allows me to randomly select a value in each set, increases my probability of testing with values that I might not otherwise include in a hard-coded set, and adds some degree of variability of inputs for improved test coverage of all possible input values.

However, sometimes we might want to include specific values in the model file we use to generate combinatorial tests. These specific values might include boundary conditions or other values based on historical failure indicators for that feature. In the past I suggested that we don’t necessarily have to specify boundary values in our combinatorial tests. The reason for this suggestion is based on the idea that:

many boundary issues are single mode faults (meaning the error occurs when 1 parameter is set at or immediately above or below its boundary condition
testing for single mode errors is often easier and less costly as compared to combinatorial testing
combinatorial testing might obfuscate the cause of a boundary bug

However, I am now convinced that

some developers are so inept at unit testing and completely overlook boundary conditions (If you are a developer and only write “happy path” unit tests, please read Pragmatic Unit Testing by Hunt and Thomas, and Clean Code by Robert Martin)
we find boundary bugs so late in the test cycle that someone determines they are too obscure to fix
we have “trained” customers to avoid boundaries (due to the number of issues and resultant failures that often occur around boundaries) so we don’t care about them anymore either
we don’t understand the fault model and therefore don’t now how to adequately identify boundary conditions and test for them

But boundary issues are still fun to find, and they always make for good examples in training or conference demos.

Anyway, on to the bug. While ‘checking’ the ranges of the margins on Paint’s Page Setup dialog for the exercise in this course I came across an interesting anomaly. When the margins were set to values that were grossly outside the allowable margin and I pressed the OK button I got an appropriate error message. But, when I changed the Scaling variable state from Fit to: to Adjust to: the Fit to: value changed to 0 although the textbox control was grayed out. I now realized the margin values are being used to auto calculate the Fit to: output values.

Margin values used to calculate Fit to: value

Since the boundary value for letter size paper with a portrait orientation is 8.5 inches, I decided to see what happens when I set the left margin to 8.501 and the right margin to 0 and then change from Fit to: to Adjust to: to check and see what happens. Interestingly enough, the Fit to: value now changed to 4,294,965,329. OK…now, I just overflowed a variable (the developer only allows the user to input a maximum of 2 characters (99) in the Fit to textboxes).

Overflow in Fit to: value

Surely, I am thinking that a page size boundary is a standard value, and surely someone tested this. But, I decided to check the specific boundary value just to see what happens anyway. So, I set the left margin to 8.5 and the right margin to 0, change the Scaling from Fit to: to Adjust to: and…

Uh-oh!

Game over!

There are many ways to expose this failure. Another fun way is to set the Scaling parameter to Adjust to: first. Next set the left margin to 8.5 and the right margin to 0 (assuming letter size paper with a portrait orientation), and click the OK button. Then, open the Page Setup dialog again and…game over!

Now, I really didn’t find this bug doing combinatorial testing. In fact, although combinatorial testing might ultimately reveal this problem (depending on the model of inputs provided to the tool that generates variable combinations), this bug was discovered during the data modeling process and discovering where calculations were occurring on certain variables. Once I saw an output boundary anomaly caused by other input variables I forced those input values to target the output boundary conditions of the output variable I wanted to further investigate. So, while we should use failure indicators and experience to specify important values in our combinatorial tests in conjunction with random values within the total population of possible values) I am still not thoroughly convinced that we should always include specific boundary values in our combinatorial test models because I suspect that even the process of modeling this feature for combinatorial testing would likely have exposed this issue.

But, in the end this is really just another example of a simple boundary bug that could have easily been found during unit testing.

Filed in Testing Practices | Tagged Boundary Testing, Combinatorial Testing | Comments (3)

Agile is not a process; Agile is a mindset.

Sunday, June 20, 2010

When someone compares agile with “traditional’ development I have no clue what they are talking about. I have never worked in a software company that produced software as if it were a widget on a factory assembly line. I have never worked in an organization where people didn’t talk to each other constantly. I have never worked for a company that didn’t periodically reflect on its own processes and/or procedures to identify areas of improvement. I have never worked for a company that wasn’t capable of adapting to changing to ‘requirements’ when necessary. And, I would say that the majority of people that I have worked with are highly motivated individuals who strive to do the best possible work, and who are capable of adapting, improvising, and over-coming obstacles in order to ship complex world class software.

This is the way it has been for me during my almost 2 decades in this industry. So, when I read books and papers that compare agile to “traditional” approaches I ask myself, “What in the hell do they mean by “traditional” approach?

For example, when I joined the Windows 95 team we had weekly builds, and at least once per week the build had to satisfy the requirements for “self-host” status. Self-host meant that anyone in the company could use that build for their day to day work. Also, many of the self-host builds are released to key strategic industry partners and 3rd party vendors so they could take advantages of new features in their software.

After the PDC release of Windows 95 we made several key changes in the operating system based on feedback from our customers and partners. Also, late in the cycle after beta 2 we had to make a major change in how Windows started because of a quirk in a relatively popular MS DOS based game. Throughout a typical lifecycle features are added, removed, and tweaked based on several factors including customer feedback.

Instead of sprints we generally have milestones. At the end of each milestone many teams have a post-mortem or retrospective to review key objectives, discuss things that went well, things that could have been improved, and “tune and adjust its behavior” to make sure everyone is aligned and find ways to increase effectiveness.

Most developers were also doing unit testing. (TDD is simply another way of restating Dave Gelperin’s and Bill Hetzel’s famous quote from around 1987 “Test then code.”). Testers and developers talked frequently, and on many occasions the developer I worked closely with would come to my office to debug a hard to reproduce issue. And the majority of testers who were hired in the early 90’s were typically required to have an in-depth technical and often coding background in order to be able to engage in all aspects of the software development lifecycle and perform all the various job roles that might be required from a testing professional.

By 1998 we were doing daily builds. We have well established unit test libraries that not only assist developers in refactoring, but also help prevent build breaks and down-stream regressions. We still do a lot of API (component) level testing, as well as integration and system level testing.

Of course some of our products have longer release lifecycles than others. However, as indicated earlier we deliver “working software” at least once a week internally, and even to some premier customers (especially after the third milestone leading up to the final release of that version. But, some of our teams release every month.

Stepping back and looking at the debates, the books, and conference presentations I realized that I can relate well to the Agile principles. We often use models, principles and other abstractions to describe things. And, when all is said and done Agile principles are simply another way to present concepts that are intended to enable a team of highly motivated people to work together to ship a high quality product that satisfies their customers.

We have different models to generalize our processes to help us describe our typical workflow to others. But in truth there is no single way to build software, and models are simply our abstract expression of how we view the process. In fact, each team chooses processes and procedures that work for them and their particular product in their unique context.

(BTW, I am also really tired of Agile pundits always comparing Agile to waterfall models. Why don’t they compare their understanding of Agile to the V-model, W-model, or especially to spiral or other iterative models of software lifecycles? These are models; how your team chooses to implement a development lifecycle may resemble one of these models or you may adopt things from different models to implement your processes and approaches to software development.

Remember, George Box stated, “All models are wrong, some models are useful.” We should implement the concepts embodied in the model; we should not implement the model.)

So, to me, Agile is the ability of a team to interact with each other, to adapt to changes, to periodically reassess its productivity in order to strive to become more effective and efficient in delivering software to its customers. Agile (and every other lifecycle models) is not a process; Agile is a mindset.

Technorati Tags: Agile,Agile Testing

Filed in General Testing Topics | Tagged Agile Testing | Comments (5)

UI Automation Beneath the Presentation Layer using .NET’s Reflection

Wednesday, June 9, 2010

This week I am in Bad Homburg, Germany where I gave a keynote at the Testing & Finance conference. It was really great to see some very dear friends and meet new colleagues from around the world. Bad Homburg is about 30 minutes by train from Frankfurt, and it is a quaint little town with beautiful parks and Schloss Homburg which was built around 1680 and was the summer residence of the German Kaiser.

Also this week the newest edition of the Automated Software Testing magazine published an article I wrote entitled UI Automation Beneath the Presentation Layer. It discusses some tips on how to use reflection for automating applications developed using managed code. If you have questions or comments about my article please let me know. Your feedback is appreciated.

I have always respected the work of Dion Johnson, and I was quite honored when he asked me to write an article for the magazine. The magazine and the Automated Testing Institute website provide a wonderful resource to people interested in software test automation. I really look forward to working with Dion and the Automated Testing Institute more closely in the future to provide whatever help I can to this monumental task he has undertaken. I also hope there are others out there who will share some of their ideas as well. I think the days of Not Invented Here (NIH) and I’ve Got A Secret (IGAS) syndrome are becoming issues of the past. We still have too much reinvention and not enough reuse of test code. I think that we have well established there is no single approach to testing that is effective in all contexts. We might approach problems from different perspectives, but I also think that all testers are passionate and have very similar goals of personal and professional improvement. To mature our discipline, and increase our effectiveness in our roles we should collaborate more and pool our resources, and the Automated Testing Institute is a growing community of professionals.

Technorati Tags: Automation,Reflection
Windows Live Tags: Automation,Reflection
WordPress Tags: Automation,Reflection

Filed in Test Automation | Tagged Data-Driven Testing, Test Automation | Comments (0)

Globalization Testing: Basic International Sufficiency

Thursday, June 3, 2010

I started my career at Microsoft in 1994 working on the Windows 95 International Test team. Globalization testing is a unique specialty in software testing just like performance, security, and other specific areas of testing. Globalization testing doesn’t necessarily require a tester to be bi-lingual, or be from a country other than the United States. A good globalization tester has an in-depth understanding of such things as character encoding types and issues associated with the different types, character mapping and conversion issues, data manipulation by the application, operating system, and network protocols.

Many people might also say that globalization testers also need to know that different locales (places) around the world use different formats for date and time (national conventions). For example, in the United States the default long date format is Thursday, June 03, 2010 but in Germany it is Donnerstag, 3. Juni 2010. A tester doesn’t have to ‘read’ German to see the abstract date format has changed from dddd, MMMM dd, yyyy to dddd, d. MMMM yyyy.

Testing for support of these different national conventions used around the world is referred to as basic international sufficiency testing. I suspect the reason why some people might assume basic international sufficiency testing these different national conventions is the domain of the globalization tester is because the national conventions are set by default on the different localized versions of a software product so that’s when they are tested. But, this reasoning is absurd!

First, not all products are “localized” into all languages or ‘locales.” So, who tests the Canadian long date format of MMMM-dd-yy, or the Georgian (Georgia) long date format of yyyy ‘წლის’ dd MM, dddd? Also, Vista and later versions of Windows allow the user to ‘customize’ the date and time “format pictures” to use different separator symbols and orderings.

Secondly, way too many bugs such as hard-coded date formats are found way too late in the testing cycle (because localized versions tend to lag US English language version). And of course, we all know the cost of finding bugs later in the lifecycle are more costly to correct.

So, we must ask if there is a way for basic international sufficiency testing to be ‘pushed upstream?’ And of course the answer is yes. The easiest way is to host a “globalization bug bash” early in the cycle. (A “bug bash” is a day where testers are given some basic training on attack patterns, fault models, etc., in a general focus area and then spend a day exploring different areas of the product trying to flush out bugs in a competition style format.) Another way is to assign each tester a different locale (preferably one that is not associated with a localized language version) and have them set their test and self-host environments to that locale during their testing.

This is easily accomplished on Windows test environments by having testers launch the Regional and Language control panel applet (the short cut is Start –> Run, then type “intl.cpl” without the quotes, and press the OK button).

This just tests for a basic level of international sufficiency, and any good tester would want to explore their project’s capability to support the more than 150 different locale national conventions at a deeper level. This is especially true if your product is going to be used by customers around the world (including Canada). But, of course, we don’t want to run the same tests on all 150+ locales supported by the operating system.

The national convention settings for a particular locale are stored in a data type called the LCID, and when we change our locale (Format on the latest Regional and Language control panel applet) through the user interface we are actually calling various National Language Support (NLS) APIs. A “world-wide” application should use the universal NLS APIs and data available via the operating system.

One way to test our application’s ability to correctly use the national convention data supplied by the operating system is to set customized conventions. For example, did you know the Windows 7 operating system allows a digit grouping symbol to be a string of up to 3 characters? Or the Negative sign symbol can be a string of up to 4 characters.

Although having testers change their default locale (Format) on their test environment and self-host machines is a good first step in basic international sufficiency testing, we also want to see if our application can process a negative value of “!NEG7” instead of just “–7,” and any textboxes correctly display the customized negative sign symbol (especially at the upper extreme boundary of the textbox size property.

To customize the national convention settings we simply click the Advanced settings… button on the Formats property sheet of the Region and Language control panel applet which instantiates a new dialog with 4 property sheets for Numbers, Currency, Time, and Date.

Solution for Test Automation

That’s all well and fine for basic testing, or testing a “few” customized values, but if we wanted to test the permutations for each convention, or the combination of different conventions on numbers, currency, time, or date formats the number of tests is astronomical. Typically, testers writing an automated test would try to navigate the user interface of the Regional and Language control panel applet and the Customize Format property sheets in order to set custom conventions.

In the past I provided some code snippets for changing the convention settings on the Customize Format property sheets on versions of Windows pre-Vista. Earlier this year I also provided code snippets for customizing the date format picture and the time format picture.

That’s all well and good, but I recently released a new test automation library called GlobalTester for test developers to use in their automated test scripts. The GlobalTester library provides testers methods to set custom national conventions for the current user without having to navigate the user interface of the Region and Language options control panel applet. These national conventions include number formats, currency formats, date formats, time formats, and also current location.

The following example illustrates how we might design a test script to customize the date format for a test and reset the date format to its original setting (restoring the test environment to pre-test conditions). (Usage documentation for the GlobalTester library is on the Testing Mentor website.)

   1: namespace CustomizeDateSettingsExampleScript

   2: {

   3:   using System;

   4:   using System.Globalization;

   5:   using TestingMentor.TestTool.GlobalTester;

6:

   7:   class MyTest

   8:   {

   9:     static void Main()

  10:     {

  11:       try

  12:       {

  13:         CustomDateFormat time = new CustomDateFormat();

14:

  15:         string defaultLongDateFormat =

  16:           CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern;

  17:         string newLongDateFormat = "MMM - d (yyyy) gg";

18:

  19:         if (time.ChangeLongDateFormat(newLongDateFormat))

  20:         {

  21:           // Launch AUT

  22:           // Execute test - (e.g. AUT implements long date string)

  23:           // Oracle - (e.g. compare long date format against customized pattern)

24:

  25:           // Reset test platform to original configuration

  26:           time.ChangeLongDateFormat(defaultLongDateFormat);

  27:         }

  28:         else

  29:         {

  30:           // Date format not changed; test not executed (e.g. invalid

  31:           // day, month, year, and era format pictures)

  32:         }

  33:       }

34:

  35:       catch(ArgumentOutOfRange e)

  36:       {

  37:         // Test script failure - (e.g. long date format string argument out of range)

  38:       }

39:

  40:       finally

  41:       {

  42:         // Log test results

  43:       }

  44:     }

  45:   }

  46: }

Basic international sufficiency testing is just as important as testing for boundary conditions, and it doesn’t require a globalization testing specialist. With a little fundamental understanding of national conventions and where they are used in your application you can easily start incorporating basic international sufficiency testing in your tests, and with GlobalTester library aids testers exercise a greater number of variables and combinations in a more efficient way.

Filed in Internationalization Testing, Test Automation | Tagged Globalization Testing, Test Automation, Test Tools | Comments (3)

Shipping software is a team effort

Monday, May 24, 2010

Since I was young I was involved in team sports. I played little league baseball during my elementary years, in junior high I switched to football, and in high school I played lacrosse. Growing up on the right (east) coast of America a lot of us boys would also play pick-up pond hockey when the rivers and ponds froze over in the winter. Hockey has always been a passion of mine, and I am a big Bruins fan (perhaps because Maryland didn’t have a hockey team), and Bobby Orr was amazing!

There is an ice rink very close to my home, and my daughter started skating at 4 years of age. She has no ambition to be the next Nancy Kerrigan, but she loves to ice skate. She also loves to watch hockey (she is a die hard Penguins fan and her favorite player is Sid The Kid).

The past few weeks I have been involved in a hockey tournament in conjunction with my regular Monarchs schedule, work schedule, and getting the boat ready for the up-coming Leukemia Cup Regatta and summer sailing season. (I’ll use that as an excuse for my long absence from the blogosphere.) In the tournament we won some games, and we lost some games. Eventually our team won the silver medal yesterday afternoon in the final match of the series. One thing all this practice and play has reminded me of is the importance of the team. Whether it is playing hockey or shipping software a team that plays well together is required for success.

Playing well together is more than simply showing up on the ice. Each player has a position, each player is expected to communicate effectively to move the puck around and “read the play,” occasionally players will “block a shot” on goal, and when you are on the ice you give 150% (which is why the shifts (time on ice) are short). Essentially no player can do everything alone, and the weakest link means that others on the team must take up the slack (which eventually leads to mistakes and players getting “out of position”).

I play defenseman, and usually left defense.There are general skills for the game that all players must have, but there are skills and strategies for defense positions that are different than for forwards. And although the 3 forwards and the 2 defensive players must work together as a team to score (or prevent a goal) the responsibilities of each position are different, and each player must play their position.

I have read a lot of analogies between individual sports such as martial arts and the discipline of software testing. These analogies are sometimes good in that they help testers understand why they must constantly learn new things and practice in order to grow in our (or any) professional discipline. Similarly, I could make an analogy between the defensemen on a hockey team and software testers. But, while we might all aspire to be a “testing ninja” (or a Bobby Orr) in our career, we also need a team of other professionals each doing their part to ship a great software product.

When players in the software game are out of position or not playing their best the team is looking for a loss, or at best a very hard earned win (that usually burns players out). For example, program managers who don’t to provide useful customer scenarios or models of potential feature designs, or developers who fail to unit test their code or run unit tests that are only slightly better than “does it compile” put the burden on the test team (the defensive players) to hopefully “score the goal.”

While a defensive player may occasionally score a goal, their primary role (on offense) is to put the puck in front of the opposing net. As defensive players we must be able to read the play (understand our customer scenarios and analyze abstract models of a design and provide critical feedback). Communicate effectively to other defensemen and forwards to get the puck into position, or protect our goal (put the important information in front of those who make the critical business decisions).

At times defensemen need to help the forwards by driving the puck out of the zone, but without the forwards doing their part the defensemen would surely get exhausted very quickly and the outcome would not likely be favorable. Likewise, if the forwards don’t help defend their goal it is likely the opposing team will wear down the defense and score.

So, while we (as testers) sometimes like to think that the success or failure of a software project depends solely on our defensive skills to find bugs and drive in quality, the simple fact is that we need a team of professional to be successful and “quality” depends on the value each player brings to the table. If we don’t start a game with a good strategy and people playing their best in their assigned positions the likelihood of success is marginal at best. A strong defense is critical, but no one ever wins by playing defense.

"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest." – Thomas Reid’s Essays on the Intellectual Powers of Man, 1786

Filed in General Testing Topics | Tagged Testing | Comments (1)

Globalization Testing: Customizing Time Formats

Wednesday, April 28, 2010

Time is a commodity in short supply. I have been juggling a lot lately and there never seems to be enough time to do everything I need to do, and even less time to do the things I want to do. (Blogging falls under the want to do category.) I wish sometimes I could slow down the hands of time, but that is beyond my control. What is within my control is changing the time format displayed on the computer. And if I need to do that in an automated test to increase the robustness of my test to include globalization, then I can programmatically change the time format without having to manipulate the Region and Language settings control panel applet.

Time and date information is commonly pulled from the operating system by many developers for use in headers or footers on documents, default file names, printing, and other places time/date stamps are useful or important. To ensure our products are “world-ready” we should modify the formats to validate whether our product supports various national conventions used in different regions (locales) around the world. In the previous post I illustrated how to programmatically customize the date formats on a Windows environment for including some basic globalization tests in your test automation. This week let’s look at how we can programmatically change both the short time and long time formats.

We will again need the 2 Win32 API functions SetLocaleInfo() and PostMessage() that we marshaled over into the NativeMethods class. Since that code doesn’t change I won’t repeat it here you can simply refer to the code snippet in the previous post. In this situation we need to set the lcType in SetLocaleInfo() to the LOCALE_STIMEFORMAT constant. Then we can pass a null-terminated string to the lcData variable in the SetLocaleInfo() function. MSDN explains “The maximum number of characters allowed for this string is 80, including a terminating null character. The string can consist of a combination of hour, minute, and second format pictures.”

Once again, to simplify that a bit I wrote some more wrapper methods to change the time format. Also, since we will be calling SetLocaleInfo() and PostMessage() a lot for customizing date, time, and other national conventions I created a wrapper method called UpdateLocaleInformation() to remove redundancy.

   1: namespace TestingMentor.TestTool.GlobalTester

   2: {

   3:   using System;

4:

   5:   public enum TimeFormatType

   6:   {

   7:     LongTimeFormat = 0x00001003,

   8:     ShortTimeFormat = 0x00000079

   9:   }

10:

  11:   public class CustomTimeFormat

  12:   {

  13:     private int timeFormatType = (int)TimeFormatType.ShortTimeFormat;

  14:     private string timeFormatPicture = string.Empty;

15:

  16:     public int SetTimeFormatType

  17:     {

  18:       set

  19:       {

  20:         if (value == (int)TimeFormatType.ShortTimeFormat ||

  21:           value == (int)TimeFormatType.LongTimeFormat)

  22:         {

  23:           this.timeFormatType = value;

  24:         }

  25:         else

  26:         {

  27:           throw new ArgumentOutOfRangeException("TimeFormatType invalid");

  28:         }

  29:       }

  30:     }

31:

  32:     public string SetTimeFormatPicture

  33:     {

  34:       set { this.timeFormatPicture = value; }

  35:     }

36:

  37:     public bool ChangeTimeFormat()

  38:     {

  39:       return UpdateLocaleInformation(

  40:         this.timeFormatType,

  41:         this.timeFormatPicture);

  42:     }

43:

  44:     private bool UpdateLocaleInformation(int localeType, string localeData)

  45:     {

  46:       bool success = false;

  47:       if (NativeMethods.SetLocaleInfo(

  48:         NativeMethods.SystemDefaultLocale,

  49:         localeType,

  50:         localeData))

  51:       {

  52:         NativeMethods.PostMessage(

  53:           NativeMethods.BroadcastMessage,

  54:           NativeMethods.SettingChangeMessage,

  55:           IntPtr.Zero,

  56:           IntPtr.Zero);

  57:         success = true;

  58:       }

59:

  60:       return success;

  61:     }

  62:   }

  63: }

Once again, we simply have to set the SetTimeFormatType property to either the Short time or Long time format, provide the format picture by setting the SetTimeFormatPicture property, and then call ChangeTimeFormat(). The sample below illustrates how to change the short time format with different time separators and a reverse order.

   1: static void Main(string[] args)

   2: {

   3:   CustomTimeFormat time = new CustomTimeFormat();

   4:   time.SetTimeFormatType = (int)TimeFormatType.ShortTimeFormat;

   5:   time.SetTimeFormatPicture = "ss'mm,hh - tt";

   6:   if (time.ChangeTimeFormat())

   7:   {

   8:     Console.WriteLine("Success");

   9:   }

  10: }

Now, we can also customize the AM/PM designator as well. To change the AM/PM designator we need to add a few more properties and another wrapper method. In this case, I’ve added the SetAmPmDesignator property, the SetAmPmString property, and the ChangeAmPmDesignator() method.

   1: public enum AmPmDesignator

   2: {

   3:   AM = 0x00000028,

   4:   PM = 0x00000029

   5: }

6:

   7: public class CustomTimeFormat

   8: {

   9:   public int SetAmPmDesignator

  10:   {

  11:     set

  12:     {

  13:       if (value == (int)AmPmDesignator.AM || value == (int)AmPmDesignator.PM)

  14:       {

  15:         this.designatorForAmPm = value;

  16:       }

  17:       else

  18:       {

  19:         throw new ArgumentOutOfRangeException("AmPmDesignator invalid.");

  20:       }

  21:     }

  22:   }

  23: }

  24: public string SetAmPmString

  25: {

  26:   set { this.timeDesignator = value; }

  27: }

28:

  29: public bool ChangeAmPmDesignator()

  30: {

  31:   return UpdateLocaleInformation(

  32:     this.designatorForAmPm,

  33:     this.timeDesignator);

  34: }

The code snippet below illustrates how to change the AM designator from “AM” to “In the morning.”

   1: static void Main(string[] args)

   2: {

   3:   CustomTimeFormat time = new CustomTimeFormat();

   4:   time.SetAmPmDesignator = AmPmDesignator.AM;

   5:   time.SetAmPmString = "In the morning.";

   6:   if (time.ChangeAmPmDesignator())

   7:   {

   8:     Console.WriteLine("Success");

   9:   }

  10: }

Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle. Time and date are perhaps the most visible national conventions used in many different ways in our applications. We should test the common (equivalent) conventions used in various regions around the world, and customizing these settings helps ensure the developer is properly calling NLS APIs and not using custom functions.

Also, check out the beta release of the GlobalTester automation library that has this functionality and more, and let me know what you think.

Filed in Internationalization Testing, Test Automation | Tagged Globalization Testing, Test Automation | Comments (2)