Skip to content

Globalization Testing: Customizing the Date Format

Saturday, April 17, 2010

The ability of our software products to function correctly in a global environment is becoming more and more important. Our software should support national conventions used by the various locales around the globe. For example, in some regions of the world the period character is used as the number group separator and the comma is used as the decimal symbol (radix). European calendars generally start on Monday rather than Sunday which is customary in the United States. Era based calendars are still in common use in Japan and Korea, date formats and order, and time formats also vary by region or locale. As testers we need to test our software to ensure our customers around the world can use the national conventions they are accustomed to, and not force them down a US-centric, one-size-fits-all format or standard.

There are several settings that we can modify and customize for more robust globalization testing such as number, currency, time and date formats. Modifying these settings can help us test that our application is globalized to use National Language System (NLS) APIs provided by the system.Although a user would change these settings using the Regional Options user interface property sheets, if the purpose of our test is not to emulate user interaction, then modifying the custom regional settings for globalization testing programmatically is more efficient.

clip_image001

Last year I talked about how to programmatically make changes to the settings in the Region and Language control panel applet when doing globalization testing. Unfortunately, the code sample provided in the previous post was appropriate for versions of Windows XP and earlier. For versions of Windows Vista and later things have changed a bit. Also, the previous sample tried to be a one-size fits all and relied on the test developer to set the appropriate lcType constants and lcData argument variables required by the Win32 function SetLocaleInfo().

This time, I decided to simplify things a bit and wrapped some methods to call the appropriate Win32 API functions and properties to set lcType and lcData values to make it easier to incorporate into automated tests. I also separated the various advanced custom formats for Region and Language options into separate classes. Of course, I have a beta version of an automation library (DLL) called GlobalTest.DLL on my website that testers can use in their automated test cases, but this week let’s look at the class for setting custom date formats.

Making these changes programmatically still requires the Win32 SetLocaleInfo() function. MSDN also states this function modifies the specified values for all applications, so to prevent potential issues in other applications running on the system we should also broadcast the WM_SETTINGCHANGE message. To broadcast the WM_SETTINGCHANGE message we will also need the Win32 PostMessage() function. Since we are Process Invocation (PInvoke) to call these unmanaged functions we should put them in a separate class that I’ve called NativeMethods. I also included all necessary constant values required by these methods in the NativeMethods class also as illustrated below. 

   1: namespace TestingMentor.TestTool.GlobalTester

   2: {

   3:   using System;

   4:   using System.Runtime.InteropServices;

   5:  

   6:   internal sealed class NativeMethods

   7:   {

   8:     internal const int SystemDefaultLocale = (int)0x00000800;

   9:     internal const int BroadcastMessage = (int)0x0000FFFF;

  10:     internal const int SettingChangeMessage = (int)0x0000001A;

  11:  

  12:     private NativeMethods() { }

  13:  

  14:     [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]

  15:     [return: MarshalAs(UnmanagedType.Bool)]

  16:     internal static extern bool SetLocaleInfo(

  17:       int locale,

  18:       int localeType,

  19:       string localeData);

  20:  

  21:     [DllImport("user32.dll", SetLastError = true)]

  22:     [return: MarshalAs(UnmanagedType.Bool)]

  23:     internal static extern bool PostMessage(

  24:       int handle,

  25:       int message,

  26:       IntPtr wParam,

  27:       IntPtr lParam);

  28:   }

  29: }

The class for the custom wrapper method is TestingMentor.TestTool.GlobalTester.SetDateFormat. There is a public enumeration for the short date and long date constants. One of these values must be assigned to the SetDateType property. The other property that must be set is the SetDateFormatPicture. The big change in the SetLocaleInfo() function is that the lcData type is a null-terminated string that MSDN refers to as a format picture. Current versions of Windows allow users to customize the order of the month, day and year, the format for each, and even allow different separators between the date elements. The format picture enables the user to select various format types in different orders for either the short date or the long date. See MSDN’s Month, Day, Year and Era Format Pictures for the various supported format types. 

   1: namespace TestingMentor.TestTool.GlobalTester

   2: {

   3:   using System;

   4:  

   5:   public enum DateFormatType

   6:   {

   7:     ShortDate = 0x0000001F,

   8:     LongDate = 0x00000020

   9:   }

  10:  

  11:   public class CustomDateFormat

  12:   {

  13:     private string dateFormatPicture = string.Empty;

  14:     private int dateType = (int)DateFormatType.ShortDate;

  15:  

  16:     public string SetDateFormatPicture

  17:     {

  18:       set { this.dateFormatPicture = value; }

  19:     }

  20:  

  21:     public int SetDateType

  22:     {

  23:       set

  24:       {

  25:         if (value == (int)DateFormatType.ShortDate ||

  26:           value == (int)DateFormatType.LongDate)

  27:         {

  28:           this.dateType = value;

  29:         }

  30:         else

  31:         {

  32:           throw new ArgumentOutOfRangeException("Invalid DateType");

  33:         }

  34:       }

  35:     }

  36:  

  37:     public bool ChangeDateFormat()

  38:     {

  39:       bool success = false;

  40:       if (NativeMethods.SetLocaleInof(

  41:         NativeMethods.SystemDefaultLocale,

  42:         this.dateType,

  43:         this.dateFormatPicture))

  44:       {

  45:         NativeMethods.PostMessage(

  46:           NativeMethods.BroadcastMessage,

  47:           NativeMethods.SettingChangeMessage,

  48:           IntPtr.Zero,

  49:           IntPtr.Zero);

  50:       }

  51:  

  52:       return success;

  53:     }

  54:   }

  55: }

Once the SetDateType and SetDateFormatPicture properties are assigned we simply have to call ChangeDateFormat() method to change the settings and broadcast the message to the system. The code snippet below illustrates how a tester would change the default long date format in an automated test to determine globalization support in the application under test. Customizing the date format is useful if the application under test uses a date string in any way. For example, if the application includes a function to insert a date string in an edit control, or if the date is printed as a header or footer in a document, or if a date string is appended to a record. 

   1: using TestingMentor.TestTool.GlobalTester;

   2: ...

   3:     static void Main(string[] args)

   4:     {

   5:       CustomDateFormat date = new CustomDateFormat();

   6:       date.SetDateType = (int)DateType.LongDate;

   7:       date.SetDateFormatPicture = "[ dd % MM | yyyy ]";

   8:       if (date.ChangeDateFormat())

   9:       {

  10:         Console.WriteLine("Long date was changed");

  11:       }

  12:     }

Programmatically changing the date format is an easy way testers can customize date formats in their automated tests without having to manipulate the controls on Region and Language property sheet. Also note, that since the format picture is a string the order of the supported date format types is now controlled by the arrangement in the string, and the separator characters can be different between the day and month and the month and year as illustrated in the example above.

Modifying national conventions is one way to test for globalization support upstream and should be done early in the testing cycle rather than relying on a separate globalization testing cycle.

Next week I will discuss customizing the time format. Also, check out the beta release of the GlobalTester automation library that has this functionality and more and let me know what you think.

Filed in Internationalization Testing, Test Automation, Testing Tools | Tagged | Comments (1)

Is this as good as it gets?

Thursday, April 8, 2010

Last month I was travelling throughout Europe. A few days in Switzerland to speak at Swiss Testing Days and visit our communication server team in Zurich. About a week and a half in Denmark teaching at our offices there, a few days in Dublin teaching, and finally a full day meeting with one of our teams in Reading, UK. The only good thing about being on the road that much is getting in more reading time. Two of the books I read while travelling were Agile Testing by Lisa Crispin and Janet Gregory. The other was Clean Code by Robert Martin. I found both books thought provoking and informative. Of course I didn’t agree with everything, but even the statements I didn’t agree with made me think about that topic a bit more. Martin’s book inspired me to improve my testing and coding craft. If nothing else, testers should read the first chapter of Clean Code, those who design and develop automated tests should read the whole book.

But, one statement that has always troubled me that I saw repeated in the Agile Testing book is in regards to exploratory testing as the most effective way to expose important bugs. Every time I hear/read this or one of its many derivatives it makes me ponder. If this is really true then why do we invest so much time and resources in other approaches to software testing. Why do we spend so much time in static and dynamic analysis, reviews, and coverage analysis? Why do we try to prevent bugs when they can (and presumably are) found easier and ‘quicker’ downstream via exploratory testing? Why don’t we simply continue to hire hoards of diverse people to rapidly explore the product just before release and beat out all those important/critical bugs?

I wonder why it seems that we can ‘think critically’ when we have the product to play with, but we can’t seem to apply critical thinking while reviewing documented requirements, or a model? Personally, I am not sure that ‘trying’ something and then deciding what to do based on the outcome or state of the computer, or deciding whether something is a bug after the fact constitutes critical thinking. Perhaps sometimes it is, but I suspect that sometimes it is just guessing. To me, critical thinking involves the ability to foresee potential issues; not simply stumble upon them, or apply an attack from my bag of exploratory tricks to reveal them downstream in the development lifecycle when I have the product to play with. 

I sometimes wonder if exploratory testing really is the holy grail of software testing, or if we have just given up on other approaches because they are too hard. Could it be that our testers are not capable of ‘testing’ without also engaging their fingers? Are our testers untrained or under-trained in our profession? Is the reason why our pre-defined test cases are seemingly ineffective due to the realization that we suck at test design, and learning how to design more robust and ineffective tests is just too hard? Is the reason we can’t think of tests until we get the product in hand perhaps due to our inability to model or is it because we really don’t know that much about the systems we are testing? (You can’t model what you don’t understand and you don’t understand what you can’t model.) Besides, why should we burden testers with the overhead of learning about programming concepts, computer science, and math. You don’t need all that stuff to find bugs.

Now many of you might think that I despise exploratory testing. I am not anti-exploratory testing (whichever form that takes); because we all do it! Show me a tester who doesn’t include some form of exploratory testing in his or her repertoire and I’ll show you a thoughtless simpleton who needs explicit instructions for determining which side of the heal of a loaf of bread to butter. But, I have always thought that we could actually improve the craft of testing. I thought we might play a much bigger role in an organization. I thought we could help reduce overall production costs by being involved earlier and throughout the lifecycle. I thought we could help reduce risk through defect prevention and in-depth analysis. And, I thought we could learn to use tools and techniques to help us design better tests up front, and rely on a team of testers who are capable of thinking while executing a test case.

Yes, it is true that exploratory testing is fun, and we don’t really need to understand all that computer science stuff to find bugs using an exploratory approach. Regardless of what empirical studies on the topic reveal, exploratory testing ‘feels’ so much more productive. Why should we base our profession on evidence when we can get by so easily with emotion? Why should we have to think critically during the design or implementation of software when we can simply muddle our way through some graphical user interface to find bugs? If our approaches to unit testing and component level testing are so shoddy as to simply invoke happy path” tests then of course finding bugs by exploration seems rather logical. But, as developers become more adept at unit can component level testing in flushing out functional bugs earlier we may only need ‘testers’ to explore the product before release for behavioral issues, compatibility problems, and the few remaining obscure functional bugs. Or possibly exploratory testing providing a ‘good enough’ service to our team by finding the bugs that matter to the majority of customers, and perhaps that is ‘good enough.’

Personally, I think we can improve our testing processes. I think we will need to improve the practice of software testing. I think that our job is not simply to find bugs, but a more important objective is to prevent defects (which can help reduce overall product costs). I also don’t view testing as destructive, and I think those who do are doomed to become obsolete. Successful businesses don’t hire folks who just want to ‘destroy’ the products they build, I suspect they want to hire people who can help the company grow and improve. Finding bugs helps me ship a product, but it doesn’t necessarily help me grow the business, reduce production costs, or improve quality long-term.

But maybe we don’t need to change. Maybe our businesses are satisfied with the status quo. Maybe we don’t need to think about cost savings or helping our teams mature their processes, reduce overall risk, and potentially improve quality earlier. Maybe I am too  idealistic thinking that testers can provide greater value to an organization beyond finding bugs at the end of the development cycle.  Maybe our craft has reached the the apex of of our profession and this is as good as it gets.

Filed in General Testing Topics | Tagged | Comments (3)

Complex != Better

Wednesday, March 31, 2010

I know all about over-engineering. I previously wrote about a barn my father designed and built using telephone poles and oak planks and pallets for the stall walls. From a structural perspective this barn would have stood virtually anything mother nature could have thrown at it. And if someday people wanted to tear down that barn they would surely curse the builders because the job would be much harder then they expect.

I sometimes find code that is way over-engineered. In some cases it may be necessary for making the code more robust. But, over-engineered code may also result from ignorance of more efficient algorithms or design patterns. And, sometimes overly complex algorithms are a result of developers trying to craft something that obfuscates the simplicity of the solution and fool others into believing the complexity of the solution is somehow better than a more simple solution.

I sometimes see this in test code. I am a firm believer in robust, ‘bullet-proof’ test code because each time an automated test case throws a false positive or ‘breaks’ the team loses confidence in the automation project, and it takes our time to ‘massage’ the test back to health. But, there seems to be 2 extremes in test automation. Simplistic prescriptive scripted tests with a bunch of hard-coded ‘test-data,’ or overly complex test code that is virtually undecipherable by anyone other than the original developer that renders any downstream maintenance virtually impossible. I have seen many instances where complete libraries of automation was scrapped and re-developed simply because it was easier to re-write the code rather than trying to wade through the quagmire of complexity.

In a recent example, some of my students submitted their test automation projects with a library that contained a method to generate a random string. When I first looked at their method for generating a random string I was a bit perplexed. Besides the fact that the method only produced a grand total of 26 upper case characters, the code was much more complicated than necessary. Despite having showed them how to use the Babel random string generator they choose to make their own, so I asked them how they came up with this solution and they said “they searched on the Internet. ” Others in the class indicated they also used the same method in their projects. So, I when I got home I did a quick search and found the code sample.

   1: private static string RandomString(int size)

   2: {

   3:   StringBuilder builder = new StringBuilder();

   4:   Random random = new Random();

   5:   

   6:   char ch;

   7:   for (int i = 0; i < size; i++)

   8:   {

   9:     ch = Convert.ToChar(

  10:       Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));

  11:     builder.Append(ch);

  12:   }

  13:  

  14:   return builder.ToString();

  15: }

OK…for a moment let’s overlook the fact that if we make 2 consecutive calls to this method we will get the same identical random string of characters (which the author of this code also discovered), and let’s assume that the range of characters is limited to upper case A through Z, and let’s also ignore the fact that we are not seeding our Random generator for reproducibility of the randomly generated output from this method.

Let’s focus on the statement in lines 9 and 10. Why in the world would we generate a number between 0.0 and 1.0, multiply it by 26 and add 65, and then use Math.Floor to return the largest integer less then or equal to the resultant double, then convert that double to a type Int32, and then convert the Int32 to a type char? Now I ask you, can we make this any more complicated?

Now, I am not critiquing the style of the code or its inherent limitations, but this solution is an example of over-engineering. As I have said before, complexity cultivates chaos. At first I thought maybe this approach might give me a better distribution of characters, but when I tested this hypothesis it simply didn’t pan out. So, the way the random character is generated is simply too complex. An easier, more readable, and effective alternative might be to simply generate a random integer within the allowable range and cast that int to a type char as illustrated below in line 10. (Yes, I realize the limitations with this approach if trying to generate surrogate pair characters above U+FFFF.)

   1: private const int minCharacter = 65;

   2: private const int maxCharacter = 91;

   3: private static string SimpleRandomString(int size)

   4: {

   5:   StringBuilder sb = new StringBuilder();

   6:   System.Threading.Thread.Sleep(1);

   7:   Random r = new Random();

   8:   for (int i = 0; i < size; i++)

   9:   {

  10:     sb.Append((char)r.Next(minCharacter, maxCharacter + 1));

  11:   }

  12:  

  13:   return sb.ToString();

  14: }

BTW…the Sleep() in line 6 is a easy solution to preventing identical return values for consecutive calls. But, a more effective solution would be to pass in a seed value as a parameter to the method and use that to seed the new Random generator as illustrated below. Different seeds not only guarantee randomness in the resultant string, but also allow for repeatability if necessary as long as the seed value is preserved.

   1: private static string SimpleRandomString(int size, int seed)

   2: {

   3:   StringBuilder sb = new StringBuilder();

   4:   Random r = new Random(seed);

   5:   ...

   6: }

But, I digress. This post isn’t about random generation or style…it’s about complexity. Overly complex code tends to harbor errors that might go undetected (at least initially). Also, maintenance of complex code not only becomes more problematic, it often leads to costly re-writes down the road.

If we consider that greater complexity may increase the likelihood of error, then we don’t want our development partners to unnecessarily over-engineered algorithms. Also, overly complex solutions often require overly complex testing. This not only leads to increased testing costs, but it also increases the probability of an important test being missed or overlooked. Think testability!

Finally, with regard to test automation we should consider that our automated tests might be reused by other teams, and they certainly will be used during maintenance or sustained engineering efforts well after the product has released. And, in some cases, the people maintaining the product might not be the same people who shipped the product. Well-written automated tests are not just reviewable by someone other than the author of the code, but they should also be easily maintainable. Otherwise, we might end up paying double for that automated test case. Ouch!

Filed in General Testing Topics, Test Automation | Tagged , | Comments (1)

Boundary Bugs…like shooting fish in a barrel

Wednesday, March 24, 2010

If there is a bug at a boundary that doesn’t lead to an unhandled exception or security exploit should we care?

Perhaps an even more important question is why do we find so many boundary type bugs via exploratory testing when they can and should be caught earlier? Why don’t we find these types of bugs in our unit testing? Why don’t we find these types of bugs by more systematically testing the software? Maybe we do find them, and those who make the decisions to fix these types of bugs just don’t care if they are fixed because there is no severe negative impact to the user. Maybe someone just wants to give me fodder for my blog!

This week I wanted to compare the range of allowable font sizes for a simulation program I developed as an example for a magazine article that I am working on. I knew that Office applications allow a font size within the range of 1 – 1638. I thought that range might be too large for my purposes, and since I knew that Windows Notepad included a font dialog I decided to check the allowable range of font sizes in Notepad.

The first thing I discovered was that the combobox control allows up to 5 characters! Really? Someone decided it is a good idea to allow users to enter 5 characters? notepad 1

OK, I’ll play along. Maybe if I put in a size of 99999 and press the OK button on the dialog I will get an error message, or at least Notepad defaults to the last ‘valid’ selected font size. That might seem reasonable. But is that what happens? NO! Instead of doing something reasonable (e.g. error message, default font size) the font changes to a size of 1 (yes that is a font size 1 in the upper left corner in the image below).

Notepad 2

I am sure that defaulting to a font size of 1 makes sense when the allowable size value overflows! Really…someone thought that was a good idea? Now I wanted to see what magical boundary value the developer decided was an acceptable font size. Since the combobox size property allowed 5 characters I immediately tried 65535. No, that also resulted in the overflow and displayed the text in a font size of 1. Next I tried 32767. Wait…32767 didn’t display the string in Notepad’s edit control at a font size of 1. Now, I am thinking the developer is using a data type of signed short for the font size variable. So, I enter 32768 expecting the value to overflow and display my string as a size 1 font again. But, no…that doesn’t happen.

Now, when I am design boundary tests I generally rely on 2 heuristics for identifying boundary values for input or output parameters.

  1. Values at the extreme edges of a physical range of values
  2. Values at the edges of equivalence partitions of physical values

So, in these situations I ask myself “What sort of demented developer debauchery have I now found myself?” I can’t think of any other obvious edge values that might apply, so out of curiosity I quickly narrow down the magical value to 39321. I then ask myself, “OK…even if there were a display capable of rendering or a printer capable of printing a font of this size, what is so unique about 39321?” In hexadecimal it is 0×9999, and in binary it is 1001100110011001. OK…nothing obviously special here, but I am certain the implementation details are much more complex then a simple range of values and at this point I really don’t care because this bug just doesn’t make sense.

Maybe it’s not supposed to make sense! Maybe nobody really cares about these types of bugs!

(BTW…somebody please take the Thesaurus away from the developer…’Oblique?’ Are you serious…why not just be consistent and use the word ‘Italic?’)

Filed in General Testing Topics, Testing Practices | Tagged , | Comments (6)

Meaningful Measures

Sunday, March 21, 2010

I arrived in Switzerland on Monday morning and met with our team here in Zurich who work on the communication server. Tuesday I presented a tutorial on advanced combinatorial testing and delivered a keynote address at Swiss Testing Days on Wednesday. Unfortunately, I really didn’t get to spend a lot of time exploring the city, but it was great to catch up with my long time friend James Whittaker. James and I also gave brief presentations at an executive dinner the night prior to the conference. It was also really nice to meet new friends from SwissQ who put together Swiss Testing Days. This was my first time to present at this conference and I was greatly impressed. More than 750 people attended the conference! It was quite an event and I hope to return next year.

At the executive dinner and during my keynote I discussed various challenges in software engineering that directly impact testers. One of those challenges we need to get our heads wrapped around is software measures. By software measures I am referring to objects in software engineering mapped to various scales in the mathematical world. Although we sometimes also use biased qualitative measures, such as “too slow,” if we are to be taken with any degree of credibility we have to define what to slow is and set a reasonable goal for ‘acceptable’ based on customer values.

As testers we expend a lot of cycles collecting buckets full of metrics. We spend time producing fancy charts, and spend countless hours ‘looking’ at the data as if it were some type of oracle that would speak to us and tell us what we wanted to know. In the best case we convince ourselves that the numbers are telling us what we want the numbers to tell us. In the worse case the decision makers do not even consider the measures, or we don’t analyze the data in an attempt to identify ways to improve some of our engineering processes and practices. In the end, all the fancy charts are taken off the walls only to be shredded and we start over.

We often get caught up in tracking mostly useless data such as bug count and code coverage. What in the world does bug count or code coverage tell us (or the decision makers) about quality? Nothing; absolutely nothing! Some people want to believe that finding a lot of bugs or have high levels of code coverage means better quality, but that is sort of like believing that you’ll find a pot of gold and a leprechaun at the end of every rainbow. So, why do we measure bug counts and code coverage? Simple…because they are easy to measure!

Good metrics are hard to define mostly because we don’t always have clear goals, or we use a scatter-gun approach to setting a bunch of disparate wishful goals (goals that we hope we can achieve, but nobody is accountable if we don’t). I personally advocate the Goal/Question/Metric paradigm by Victor Basili. But, the biggest problem I have in using this approach is in establishing meaningful goals! People are generally good with coming up with superfluous objectives such as 100% automation or 80% code coverage. But, when you ask those people why they want 100% automation or 80% code coverage they retort only with a bunch of hand-waving and philosophical arguments. It seems we sometimes have difficulty expressing the ‘why’ of setting certain goals. Of course the answer in most cases is to ‘get better’ or ‘improve’ something! But, why? What is the business value?

Once we establish clear goals the next step is to understand the variables that we can manipulate to help us achieve those goals. Then we must decide on which ones we want to change that we think will have the biggest bang for the buck. Finally, we figure out which measures will let us know whether we are progressing towards our goal. (This usually isn’t a single point of measurement.)

At one time I naively believed that there was a core set of metrics that all teams should be collecting all the time that we could put into a ‘dashboard’ and compare across teams. In retrospect that was really a bone-headed notion. Identifying these measures is not easy, and there is no cookie-cutter approach. Each project team needs to decide on their specific goals that may increase customer value or impact business costs. Testers should ask themselves, “why are we measuring this?” “What actions will be taken as a result of these measures?” And, “if there is no actionable objective associated with this measure, then why am I spending time measuring this?”

At times is seems we are locked in a vicious cycle of relearning things via tribal knowledge, and we make decisions based mostly on ‘gut-feel’ and emotion. We collect a bunch of measures and display them similar to how the ancient Chinese used the mystical ‘dragon bones’ as oracles. But, if we are interested in being able to articulate business impact (either positive or negative) in a professional manner then we must be able to find ways to measure the things that are really important and actionable, and spend less time collecting numbers for wall decorations. At the end of the day someone is going to ask, “How do we know?” And trust me on this…really great managers will eat you alive if you answer with “well, we think…” or “we feel…” or try to evaluate success on some other subjective measure.

Filed in General Testing Topics, Test Management | Tagged | Comments (2)

Do I Really Need To Automate This Test?

Monday, March 8, 2010

For the past 2 weeks my students in my automation course at University of Washington have been tasked with designing automated test cases through the GUI for a shareware program. In my opinion, GUI automation is the least effective approach for testing the functional or business logic of a program (assuming a well designed architecture where the business logic code is separate from the form (GUI) and the event hander (GUI object behavior) code). However, if a tester doesn’t have access to the underlying APIs used in the application under test (AUT) and is given just a compiled application (‘GUI application’) to test then functional testing through the GUI may be the only alternative.

GUI automation can be effective for some types of behavioral and non-functional tests such as performance and stress testing. It can also be useful in checking for layout issues such as control alignment, and clipping or truncation of controls on a dialog much more effectively than compared to the human eye.

However, there are some behavioral tests that are more efficient to perform manually by ‘me’ the tester. For example, end-2-end user scenarios are designed to simulate a customer completing some task involving multiple features and system interactions. Sure, we could automate these types of tests and I can even design my automated test to simulate emotions such as frustration by timing out if an event takes ‘too long’ or anger because of ‘too many’ pop-ups. (Of course, I’d have to specify ‘too long’ and ‘too many.’)  But, in my opinion we shouldn’t automate things like end-2-end scenarios because automation is poor at emulating a real person. I write automated tests to provide value to ‘me’ the tester; to free up my time to test the things that are better tested by ‘me.’

There are other types of GUI test cases that I need to execute, but shouldn’t be automated. One student wanted to automate a test that clicked the buttons on the toolbar but was having difficulty accessing the toolbar buttons on a native code application using C#. Now, in my opinion, spending time to automate a test case to ‘validate’ the toolbar buttons makes about as much sense as automating a test to validate the tab order of a dialog or checking duplicate access key mnemonics. The question is not how can we can automate ‘test cases’ for tab-order, key mnemonics, or the toolbar buttons; the question is should we?

First, I explained to the student that the difficulty was due to the fact that toolbar buttons are not the same as common button controls (e.g. OK or Cancel buttons). Toolbar “buttons” are actually bitmap images that sit on a toolbar control and just look and act similar to small buttons. Next, I asked the student, “Since I know you are not testing the toolbar control itself, what is the purpose of this test; what exactly are you testing?” He replied, “To make sure it works.” Again I asked, “What exactly are you testing, what specifically are you making sure works?” Finally he replied, “To make sure the toolbar button triggers the appropriate event handler.” I thought to myself, “Great! They are starting to think about how this stuff works below the covers.” The questioning continued, “Are there other ways to trigger the same events? The student replied, “Yes, there are menu items.” In fact, most toolbar buttons are essentially shortcuts so users don’t have to navigate dropdown menus. The example program below illustrates how toolbar buttons provide a visual cue to the user, but end up calling the same event handler as the menu item.

menu items toolbar buttons

So I asked, “Since there is a menu item that calls the same apparent event as the toolbar button, do you think there are two separate event handlers for the same behavior; one for the menu item click event and another for the toolbar button click event, or do you think the menu item click event and the toolbar button click event call the same event handler?”

The answer here could depend on whether or not we are dealing with competent developers. For example, as we build out the event handlers for the UI element I guess we could create 4 separate events (2 that do the same thing) as illustrated below.

4 event handlers

Competent developers would of course realize we only need 1 event handler for the ‘click’ events for the align right menu item and toolbar button, and 1 event handler for the align left menu item and toolbar button since there is no behavioral difference between clicking the menu item or clicking the toolbar button in this situation. So, our developer refactors the code to have 1 event handler for each specific behavior similar to:

2 event handlers

and then updates the appropriate UI element Click event statements in the form designer code to call the appropriate event handler for the menu items and as illustrated below for the toolbar buttons.

update click events

But, I still wasn’t completely convinced of the purpose of his test case. So, I asked, “Are you testing the event handler, or are you testing to make sure the toolbar button “click” event calls the appropriate event handler?” To which he responded, “To make sure the toolbar button ‘click’ event calls the ‘correct’ event handler.”

“OK,” I said in a pondering sort of way, “Let me get this right. You are going to spend some amount of time to automate a test that will validate whether or not each toolbar button click event calls the appropriate event handler.” Then I proceeded to click each toolbar button on the application under test to trigger the expected behavior. The few buttons only took a matter of a few seconds. Then I looked at him and asked, “Are you sure you want to spend time automating a test to do what I just did in a few seconds? “Are you sure you want to automate a test that has an extremely low probability of changing during the product development lifecycle? “Are you sure you want to automate a test that will probably get a lot of “face time” by testers, developers, beta testers, and others on the team? “Are you sure you want to automate a test case that you will likely spend even more time massaging and maintaining over the product shelf-life? “Or, do you think it might be a more efficient use of your time to take a few seconds and test this once per sprint cycle or milestone and let dog-fooding, beta testing, self-hosting, etc. help in ‘testing’ the behavior of those toolbar buttons?’”

I suspect this is a case of “well, this is a test that I need to test at least once, so we should automate it if we can.” Certainly we need to test toolbar buttons to make sure they trigger the appropriate event handler; once, maybe once per milestone or sprint cycle. But, do I really need to automate this test? In a similar case, one tester at Microsoft said to me, “we have to constantly retest this in sustained engineering and if we don’t automate this test then we will have to hire testers to test it manually.

Besides the faulty logic of retesting unchanged code or code that is not impacted by other changes repeatedly (and we have lots of tools to show us code churn and dependencies between modules that might be affected by churn) and beside the foolish notion that automation will replace testers, I will say that I would rather have a tester spend a few seconds each cycle testing whether a toolbar button event calls the appropriate event handler rather than have a tester spend hours/days/weeks baby-sitting and massaging temperamental GUI test code.

This is not to say that all GUI automation is finicky. And this is not to say that we shouldn’t consider automating our test cases. But, we shouldn’t automate for the sake of trying to automate all our test cases, and we certainly shouldn’t automate mindlessly simple tests; especially automated tests that might require more of my time in the long run or that have little value (virtually zero probability of  new information) to the overall testing effort when executed. (Just because a test is automated doesn’t mean it’s free!)

Before we develop an automated test we should really think about the test design from a “what am I REALLY testing here” perspective and then ask, “Does this really make sense to have a separate automated test case, or is this behavior or functionality being covered by other tests (manual and/or automated) sufficiently?”

Filed in General Testing Topics, Test Automation | Tagged , | Comments (4)

Programmatically Detecting The Operating System Version (Part II)

Wednesday, March 3, 2010

Time is a commodity in short supply! It has been more than 2 weeks since my last post. I have not been sitting idle, but really haven’t had a lot of free time to write. In preparation for my up-coming trip to Zurich, Switzerland to give a workshop and keynote at Swiss Testing Day I was interviewed by Marco van der Spek for TESTNIEUWS.NL. The interview provides a bit more background about me and some of my perspectives of Microsoft and software testing.

Also sucking up some of my time over the past couple of weeks were ‘reorgs’ at work. Change at Microsoft is a constant. Most people get used to it after awhile; others still freak out even with the slightest change. For me it mostly means shifting some priorities based on the new General Manager’s strategic vision, acclimating to a new manager and letting him know what I am working on, and generally making sure the day to day business on our team keeps moving forward during the transition.

Just like life at Microsoft and technology in general the Windows operating system continuously goes through changes. New versions, new service packs, Ultimate, Home, Server versions, etc. Sometimes it is hard to keep up with all the changes. And from a test automation perspective it is sometimes important to know which operating system version the test is running on. In some cases control flow in the automated test case may need to branch in order for the test to execute on different operating system versions. Branching in an automated test based on the operating system version eliminates the need to write separate test cases for variances in operating system versions.

And, certainly if our test matrix includes multiple product versions (Home, Ultimate, Professional, etc) and our automated test exposes a bug then we certainly want to collect information about the operating system version the test was running on. This is especially important if the automated test fails on one machine, but passes on another. Sometimes the cause of the failure may be a slight difference in the machine configuration or the operating system version.

Almost 2 years ago I described a way to get the operating system version information using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members in this blog post. But, I also mention some limitations such as detecting the specific edition or product type of a Windows Version. Another limitation is the difficulty in detecting whether the operating system version is Windows 7 or Windows 2008 Server R2.

I also mentioned that in order to identify a particular version  and/or edition of Windows we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function. But, to use these Win32 APIs we need to use platform invocation services (P/Invoke) to marshal native code into our managed test code.

The code snippet below illustrates the required Win32 function marshaled using the DLLImport attribute in C#, constant values, wrapper methods to get common operating system information, and public properties to get the operating system version, any installed service pack information, and the operating system edition for Windows Vista, Windows Server 2008, and Windows 7.

   1: // <copyright file = VersionInfo.cs" company = "Testing Mentor">

   2: // Copyright © 2010 All Rights Reserved. Test developers can simply copy and

   3: // paste the code into their code, but may not reproduce or publish the code

   4: // snippets on any web site, online service, or distribute as source on any

   5: // media without express written permission. </copyright>

   6:  

   7: namespace TestingMentor.Snippet.OperatingSystemVersionInfo

   8: {

   9:   using System;

  10:   using System.Runtime.InteropServices;

  11:  

  12:   public class WindowsVersionInfo

  13:   {

  14:     public string GetOSVersion

  15:     {

  16:       get { return this.GetOSVersionInfo(); }

  17:     }

  18:  

  19:     public string GetServicePack

  20:     {

  21:       get { return this.GetServicePackInfo(); }

  22:     }

  23:  

  24:     public string GetProductType

  25:     {

  26:       get { return this.GetProductTypeInfo(); }

  27:     }

  28:  

  29:     private string GetOSVersionInfo()

  30:     {

  31:       string version = "Unsupported Version";

  32:  

  33:       NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();

  34:       osvi.VersionInfoSize = 

  35:         Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

  36:       NativeMethods.GetVersionEx(ref osvi);

  37:  

  38:       if (OsviConstant.SupportedPlatform == osvi.PlatformId &&

  39:         osvi.MajorVersion > 4)

  40:       {

  41:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  42:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows2000)

  43:         {

  44:           version = "Windows 2000";

  45:         }

  46:  

  47:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  48:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsXP)

  49:         {

  50:           version = "Windows XP";

  51:         }

  52:  

  53:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT5 &&

  54:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsServer2003)

  55:         {

  56:           if (osvi.ProductType == (byte)OsviConstant.WorkStation)

  57:           {

  58:             version = "Windows XP Professional x64";

  59:           }

  60:           else

  61:           {

  62:             version = "Windows Server 2003";

  63:             if (NativeMethods.GetSystemMetrics(OsviConstant.ServerR2) != 0)

  64:             {

  65:               version += " R2";

  66:             }

  67:           }

  68:         }

  69:  

  70:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&

  71:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.WindowsVista)

  72:         {

  73:           if (osvi.ProductType ==

  74:             (byte)OsviConstant.WorkStation)

  75:           {

  76:             version = "Windows Vista";

  77:           }

  78:           else

  79:           {

  80:             version = "Windows Server 2008";

  81:           }

  82:         }

  83:  

  84:         if (osvi.MajorVersion == (int)OsviConstant.MajorVersion.NT6 &&

  85:           osvi.MinorVersion == (int)OsviConstant.MinorVersion.Windows7)

  86:         {

  87:           if (osvi.ProductType == (byte)OsviConstant.WorkStation)

  88:           {

  89:             version = "Windows 7";

  90:           }

  91:           else

  92:           {

  93:             version = "Windows Server 2008 R2";

  94:           }

  95:         }

  96:       }

  97:  

  98:       return version;

  99:     }

 100:  

 101:     private string GetServicePackInfo()

 102:     {

 103:       NativeMethods.OSVersionInfoEx versionInfo = new NativeMethods.OSVersionInfoEx();

 104:       versionInfo.VersionInfoSize = Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

 105:       NativeMethods.GetVersionEx(ref versionInfo);

 106:       return versionInfo.CSDVersion; 

 107:     }

 108:  

 109:     private string GetProductTypeInfo()

 110:     {

 111:       string product = String.Empty;

 112:  

 113:       NativeMethods.OSVersionInfoEx osvi = new NativeMethods.OSVersionInfoEx();

 114:       osvi.VersionInfoSize =

 115:         Marshal.SizeOf(typeof(NativeMethods.OSVersionInfoEx));

 116:       NativeMethods.GetVersionEx(ref osvi);

 117:  

 118:       if (osvi.MajorVersion > 5)

 119:       {

 120:         uint productType = 0;

 121:  

 122:         NativeMethods.GetProductInfo(

 123:           osvi.MajorVersion,

 124:           osvi.MinorVersion,

 125:           osvi.ServicePackMajor,

 126:           osvi.ServicePackMinor,

 127:           ref productType);

 128:  

 129:         switch (productType)

 130:         {

 131:           case (uint)OsviConstant.ProductInfo.Business:

 132:             product = "Business Edition";

 133:             break;

 134:           case (uint)OsviConstant.ProductInfo.BusinessN:

 135:             product = "Business N Edition";

 136:             break;

 137:           case (uint)OsviConstant.ProductInfo.ClusterServer:

 138:             product = "HPC Edition";

 139:             break;

 140:           case (uint)OsviConstant.ProductInfo.DatacenterServer:

 141:             product = "Server Datacenter (Full)";

 142:             break;

 143:           case (uint)OsviConstant.ProductInfo.DatacenterServerCore:

 144:             product = "Server Datacenter (Core)";

 145:             break;

 146:           case (uint)OsviConstant.ProductInfo.DataCenterServerCoreV:

 147:             product = "Server Datacenter without Hyper-V (Core)";

 148:             break;

 149:           case (uint)OsviConstant.ProductInfo.DataCenterServerV:

 150:             product = "Server Datacenter without Hyper-V (Full)";

 151:             break;

 152:           case (uint)OsviConstant.ProductInfo.Enterprise:

 153:             product = "Enterprise Edition";

 154:             break;

 155:           case (uint)OsviConstant.ProductInfo.EnterpriseE:

 156:             product = "Enterprise E Edition";

 157:             break;

 158:           case (uint)OsviConstant.ProductInfo.EnterpriseN:

 159:             product = "Enterprise N Edition";

 160:             break;

 161:           case (uint)OsviConstant.ProductInfo.EnterpriseServer:

 162:             product = "Server Enterprise (Full)";

 163:             break;

 164:           case (uint)OsviConstant.ProductInfo.EnterpriseServerCore:

 165:             product = "Server Enterprise (Core)";

 166:             break;

 167:           case (uint)OsviConstant.ProductInfo.EnterpriseServerCoreV:

 168:             product = "Server Enterprise without Hyper-V (Core)";

 169:             break;

 170:           case (uint)OsviConstant.ProductInfo.EnterpriseServerIA64:

 171:             product = "Server Enterprise for Itanium-based Systems";

 172:             break;

 173:           case (uint)OsviConstant.ProductInfo.EnterpriseServerV:

 174:             product = "Server Enterprise without Hyper-V (Full)";

 175:             break;

 176:           case (uint)OsviConstant.ProductInfo.HomeBasic:

 177:             product = "Home Basic Edition";

 178:             break;

 179:           case (uint)OsviConstant.ProductInfo.HomeBasicE:

 180:             product = "Home Basic E Edition";

 181:             break;

 182:           case (uint)OsviConstant.ProductInfo.HomeBasicN:

 183:             product = "Home Basic N Edition";

 184:             break;

 185:           case (uint)OsviConstant.ProductInfo.HomePremium:

 186:             product = "Home Premium Edition";

 187:             break;

 188:           case (uint)OsviConstant.ProductInfo.HomePremiumE:

 189:             product = "Home Premium E Edition";

 190:             break;

 191:           case (uint)OsviConstant.ProductInfo.HomePremiumN:

 192:             product = "Home Premium N Edition";

 193:             break;

 194:           case (uint)OsviConstant.ProductInfo.HomeServer:

 195:             product = "Home Server Edition";

 196:             break;

 197:           case (uint)OsviConstant.ProductInfo.HyperV:

 198:             product = "Microsoft Hyper-V Server";

 199:             break;

 200:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerManagement:

 201:             product = "Windows Essential Business Server Management Server";

 202:             break;

 203:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerMessaging:

 204:             product = "Windows Essential Business Server Messaging Server";

 205:             break;

 206:           case (uint)OsviConstant.ProductInfo.MediumBusinessServerSecurity:

 207:             product = "Windows Essential Business Server Security Server";

 208:             break;

 209:           case (uint)OsviConstant.ProductInfo.Professional:

 210:             product = "Professional Edition";

 211:             break;

 212:           case (uint)OsviConstant.ProductInfo.ProfessionalE:

 213:             product = "Professional E Edition";

 214:             break;

 215:           case (uint)OsviConstant.ProductInfo.ProfessionalN:

 216:             product = "Professional N Edition";

 217:             break;

 218:           case (uint)OsviConstant.ProductInfo.ServerForSmallBusiness:

 219:             product = "Windows Server 2008 for Windows Essential Server Solutions";

 220:             break;

 221:           case (uint)OsviConstant.ProductInfo.ServerForSmallBusinessV:

 222:             product = "Windows Server 2008 without Hyper-V for Windows Essential Server Solutions";

 223:             break;

 224:           case (uint)OsviConstant.ProductInfo.ServerFoundation:

 225:             product = "Server Foundation";

 226:             break;

 227:           case (uint)OsviConstant.ProductInfo.SmallBusinessServer:

 228:             product = "Windows Small Business Server";

 229:             break;

 230:           case (uint)OsviConstant.ProductInfo.SmallBusinessServerPremium:

 231:             product = "Windows Small Busines Server Premium";

 232:             break;

 233:           case (uint)OsviConstant.ProductInfo.StandardServer:

 234:             product = "Server Standard (Full)";

 235:             break;

 236:           case (uint)OsviConstant.ProductInfo.StandardServerCore:

 237:             product = "Server Standard (Core)";

 238:             break;

 239:           case (uint)OsviConstant.ProductInfo.StandardServerCoreV:

 240:             product = "Server Standard without Hyper-V (Core)";

 241:             break;

 242:           case (uint)OsviConstant.ProductInfo.StandardServerV:

 243:             product = "Server Standard without Hyper-V (Full)";

 244:             break;

 245:           case (uint)OsviConstant.ProductInfo.Starter:

 246:             product = "Starter Edition";

 247:             break;

 248:           case (uint)OsviConstant.ProductInfo.StarterE:

 249:             product = "Starter E Edition";

 250:             break;

 251:           case (uint)OsviConstant.ProductInfo.StarterN:

 252:             product = "Starter N Edition";

 253:             break;

 254:           case (uint)OsviConstant.ProductInfo.StorageEnterpriseServer:

 255:             product = "Storage Server Enterprise";

 256:             break;

 257:           case (uint)OsviConstant.ProductInfo.StorageExpressServer:

 258:             product = "Storage Server Express";

 259:             break;

 260:           case (uint)OsviConstant.ProductInfo.StorageStandardServer:

 261:             product = "Storage Server Standard";

 262:             break;

 263:           case (uint)OsviConstant.ProductInfo.StorageWorkgroupServer:

 264:             product = "Storage Server Workgroup";

 265:             break;

 266:           case (uint)OsviConstant.ProductInfo.Ultimate:

 267:             product = "Ultimate Edition";

 268:             break;

 269:           case (uint)OsviConstant.ProductInfo.UltimateE:

 270:             product = "Ultimate E Edition";

 271:             break;

 272:           case (uint)OsviConstant.ProductInfo.UltimateN:

 273:             product = "Ulitmate N Edition";

 274:             break;

 275:           case (uint)OsviConstant.ProductInfo.Undefined:

 276:             product = "Unknown Product";

 277:             break;

 278:           case (uint)OsviConstant.ProductInfo.Unlicensed:

 279:             product = "Unlicensed or Expired";

 280:             break;

 281:           case (uint)OsviConstant.ProductInfo.WebServer:

 282:             product = "Web Server (Full)";

 283:             break;

 284:           case (uint)OsviConstant.ProductInfo.WebServerCore:

 285:             product = "Web Server (Core)";

 286:             break;

 287:         }

 288:       }

 289:  

 290:       return product;

 291:     }

 292:   }

 293:  

 294: // ****************************************************************************

 295: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE

 296: // ****************************************************************************

 297:   

 298:   internal class OsviConstant

 299:   {

 300:     internal const int SupportedPlatform = 2;

 301:     internal const int ServerR2 = 89;

 302:     internal const int WorkStation = 0x00000001;

 303:  

 304:     private OsviConstant()

 305:     {

 306:     }

 307:  

 308:     internal enum MajorVersion

 309:     {

 310:       NT5 = 5,

 311:       NT6 = 6

 312:     }

 313:  

 314:     internal enum MinorVersion

 315:     {

 316:       Windows2000 = 0,

 317:       WindowsXP = 1,

 318:       WindowsServer2003 = 2,

 319:       WindowsVista = 0,

 320:       Windows7 = 1

 321:     }

 322:  

 323:     internal enum ProductInfo : uint

 324:     {

 325:       Business = 0x00000006,

 326:       BusinessN = 0x00000010,

 327:       ClusterServer = 0x00000012,

 328:       DatacenterServer = 0x00000008,

 329:       DatacenterServerCore = 0x0000000C,

 330:       DataCenterServerCoreV = 0x00000027,

 331:       DataCenterServerV = 0x00000025,

 332:       Enterprise = 0x00000004,

 333:       EnterpriseE = 0x00000046,

 334:       EnterpriseN = 0x0000001B,

 335:       EnterpriseServer = 0x0000000A,

 336:       EnterpriseServerCore = 0x0000000E,

 337:       EnterpriseServerCoreV = 0x00000029,

 338:       EnterpriseServerIA64 = 0x0000000F,

 339:       EnterpriseServerV = 0x00000026,

 340:       HomeBasic = 0x00000002,

 341:       HomeBasicE = 0x00000043,

 342:       HomeBasicN = 0x00000005,

 343:       HomePremium = 0x00000003,

 344:       HomePremiumE = 0x00000044,

 345:       HomePremiumN = 0x0000001A,

 346:       HyperV = 0x0000002A,

 347:       MediumBusinessServerManagement = 0x0000001E,

 348:       MediumBusinessServerSecurity = 0x0000001F,

 349:       MediumBusinessServerMessaging = 0x00000020,

 350:       Professional = 0x00000030,

 351:       ProfessionalE = 0x00000045,

 352:       ProfessionalN = 0x00000031,

 353:       ServerForSmallBusiness = 0x00000018,

 354:       ServerForSmallBusinessV = 0x00000023,

 355:       ServerFoundation = 0x00000021,

 356:       SmallBusinessServer = 0x00000009,

 357:       StandardServer = 0x00000007,

 358:       StandardServerCore = 0x0000000D,

 359:       StandardServerCoreV = 0x00000028,

 360:       StandardServerV = 0x00000024,

 361:       Starter = 0x0000000B,

 362:       StarterE = 0x00000042,

 363:       StarterN = 0x0000002F,

 364:       StorageEnterpriseServer = 0x00000017,

 365:       StorageExpressServer = 0x00000014,

 366:       StorageStandardServer = 0x00000015,

 367:       StorageWorkgroupServer = 0x00000016,

 368:       Undefined = 0x00000000,

 369:       Ultimate = 0x00000001,

 370:       UltimateE = 0x00000047,

 371:       UltimateN = 0x0000001C,

 372:       WebServer = 0x00000011,

 373:       WebServerCore = 0x0000001D,

 374:       Unlicensed = 0xABCDABCD,

 375:       HomeServer = 0x00000013,

 376:       SmallBusinessServerPremium = 0x00000019,

 377:     }

 378:   }

 379:  

 380: // ****************************************************************************

 381: // NEW CLASS - SHOULD BE PLACED IN SEPARATE FILE

 382: // ****************************************************************************

 383:  

 384:   internal class NativeMethods

 385:   {

 386:     private NativeMethods()

 387:     {

 388:     }

 389:  

 390:     [DllImport("kernel32")]

 391:     [return: MarshalAs(UnmanagedType.Bool)]

 392:     internal static extern bool GetVersionEx(ref OSVersionInfoEx osvi);

 393:  

 394:     [DllImport("kernel32.dll")]

 395:     [return: MarshalAs(UnmanagedType.Bool)]

 396:     internal static extern bool GetProductInfo(

 397:       int osMajorVersion,

 398:       int osMinorVersion,

 399:       int spMajorVersion,

 400:       int spMinorVersion,

 401:       ref uint type);

 402:  

 403:     [DllImport("kernel32.dll")]

 404:     internal static extern int GetSystemMetrics(

 405:       int index);

 406:  

 407:     [StructLayout(LayoutKind.Sequential)]

 408:     internal struct OSVersionInfoEx

 409:     {

 410:       public int VersionInfoSize;

 411:       public int MajorVersion;

 412:       public int MinorVersion;

 413:       public int BuildNumber;

 414:       public int PlatformId;

 415:       [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]

 416:       public string CSDVersion;

 417:       public Int16 ServicePackMajor;

 418:       public Int16 ServicePackMinor;

 419:       public Int16 SuiteMask;

 420:       public byte ProductType;

 421:       public byte Reserved;

 422:     }

 423:   }

 424: }

Now, some of my readers have indicated that these code snippets are not very useful because they can’t copy them directly and put them to use. So, to help resolve that issue I have created a new section on my web site called the Code Snippet Library. This snippet is posted there along with fully annotated (mostly FxCopy and StyleCopy compliant) file available for download or to copy for inclusion in your automated test cases, or compiled into a dynamic link library (DLL).

This example doesn’t differentiate between 32-bit and 64-bit Windows operating systems, but that is not really difficult to add, and if I get enough requests I will certainly add that into the pot. If the operating system version is no longer supported by Microsoft the GetOsVersion property will return "Unsupported Version." If no Service Packs are installed the GetServicePack property will return an empty string. If you need to detect an unsupported operating system version use the example here.

Filed in Test Automation | Tagged , , | Comments (0)

Scary Stories and GUI Automation

Wednesday, February 17, 2010

I remember going camping with my cousins as I was growing up. It was great fun despite sleeping inside a musty smelling canvas tent that retained heat so well it was more like a sauna. But, my father was adamant that the old canvas tent he bought at an Army Surplus store and took 2 men and 4 boys to carry and assemble was much better then those new fangled nylon tents. Nylon ripped too easily he reasoned, but canvas will withstand anything short of a raging bear. I’m not too sure there were too many raging bears roaming the camp-grounds of Maryland, Pennsylvania, and Virginia but I was confident that even if there were I would have stood a better chance inside a canvas tent as compared to those paper thin nylon tents. To this day I remember those camping trips when I smell old canvas, or perhaps it’s dried mold spores embedded in the canvas. Whichever, it takes me back to a time of fun and fond memories.

One of the best parts of the trips were sitting around the camp fire at night listening to my uncle concocting some story intended to scare the wits out of us young boys. You know, the kind of stories about headless Confederate soldiers, or werewolves, vampires, or other such wicked creatures of the night. I think these campfire chats are remnants of man’s tribal roots where the elders tried to scare the hell out of the juvenile hominids to prevent them from wandering off at night. As we got older we realized that these stories were simply fictitious folktales; sort of like successful, value-add GUI automation projects.

Last week I had lunch with a colleague who wanted to talk to me about an automation project on his team that went horribly awry. As he started to tell me his story I thought, “Wait…I heard this tale before. I can tell this story because I’ve heard it so many times over and over again…just like those scary stories I heard around the campfire growing up.” The story goes something like this.

Our team bought a new tool, or built another framework, and taught everyone how to script “black box” test cases. They developed quite a number of automated test scripts, and of course everything was working very well. The scripts were running and managers were happy because the team had a lot of automated test scripts. But, the tests weren’t finding any bugs, so just out of curiosity the managers suggested a bug bash. And sure enough as the testers started exploring the project it didn’t take long for them to fill the database with bugs. The developers were shell shocked, and the managers couldn’t believe it! They couldn’t understand why the automated GUI tests weren’t finding any bugs? And so, in a typical knee-jerk reaction fashion, the managers immediately halted the GUI automation project and required every tester to embark on an exploratory testing adventure in search of bugs. Of course, the managers decided this approach was better than investing in more GUI automation bringing an end to another GUI automation project.

Unfortunately, unlike the scary stories my father and uncles told around the campfires, stories of failed GUI automation are often true, and usually much scarier. Why are they so scary? Because I hear these sorts of stories repeated so often. It seems that we as a discipline rely on tribal knowledge where each generation simply learns through trial and error and the folktales of our elders and thrive more on hero worship of people who are often remarkably good at finding bugs by poking and prodding hour upon hour.

Now, if you haven’t caught on already you will know that I am no big fan of GUI automation. Not because I don’t think it can be useful. In fact, I think GUI automation can provide tremendous value in some situations. But unfortunately much of the automated GUI test cases I see (especially in examples) are poorly designed, simple rudimentary script-lets. Many of these automated tests are nothing more than mindless automated sequences of events contrived because the testers have been told to automate, but are not given strategic vision (why) and little to no tactical direction (what and how).

With little or no direction or goals, or without an in-depth understanding of the system they are testing the biggest problems with GUI automation is that many testers attempt to automate

  • functional tests intended to expose computational errors in the business logic layer or in the underlying APIs
  • usability tests intended to imitate ‘me’ trying to emulate the scenarios or tasks I think the customer might do

GUI automation is probably the least effective approach for functional testing. This is not to say that GUI automation will not find functional issues in the lower logic layers. I suspect we will always find ‘functional’ bugs (e.g. boundary issues, unhandled exceptions, string parsing errors, calculation problems, etc.) while testing through the UI. But, as indicated in my previous post, well-designed software is usually built in layers and a good many of the ‘functional’ issues we find today can likely be more efficiently found through more robust unit and component (API) levels of testing.

Perhaps even more silly than trying to use GUI automation for low level functional testing is the notion of using GUI automation to emulate a ‘user’ by scripting out prescriptive sequences of actions (often with hard-coded data) that are then played over and over again. Test automation cannot and should not attempt to replicate ‘me.’ I’ve said before the purpose of automation is to provide value to me, to free up my time, to increase my efficiency, and to help me be more effective in my job; automation does not replace me. Let’s face it…we (humans) are much better at evaluating the ease of use of software and whether scenarios that represent target customer segments are intuitive for those customers.

For example, I once had a conversation explaining that GUI automation runs much faster than I can interact with software, and sometimes I make mistakes when typing in something that throws an unexpected message or takes me down a path. My colleague replied, “Well, we can slow down the automation.” Why? Why in the hell would I want to slow down my automated tests? C’mon…we all should know by now that the 100% automation (or automate all tests) mantra is a ridiculous dream and I’ve heard more plausible fantasies from people on acid trips.

So, where does GUI automation add value. In my opinion, GUI automation is probably most effective in testing UI control properties and the event handlers between the UI layer and the API layers. It is also effective in behavioral testing areas such as performance and stress.  And GUI automation is also much more effective in evaluating UI layout issues such as misaligned controls, or clipped or truncated controls on a window as compared to the human eye.

Similar to how we use different techniques to expose different categories of defects, and how we use different approaches to testing depending on the context or test objective test automation is a useful tool in our toolbox. It is certainly not the only tool. We can do some remarkable things with automation, but we must learn where GUI automation adds value and where other approaches testing (automated or manual) might be more effective.

Filed in Test Automation | Tagged | Comments (7)

API Testing: Testing in Layers

Tuesday, February 2, 2010

For the past few weeks my test automation class at the University of Washington has been focused on API (application programming interface) testing, or component level testing. Boris Beizer defines component level testing as “an integrated aggregate of one or more units” and that a “component can be anything from a unit to an entire system.”

This seems a bit confusing at first but then we realize that a single method (or function) may be a unit, or a component, or may (although unlikely) be the ‘system.’ A collection of methods wrapped in a library or DLL that interact to meet a functional requirement is a component. Rather than a developer having to call each method individually to achieve some usually repetitive functional outcome from the library that functionality is usually exposed via a call to a single API.

In this situation the students are testing a public API in a single library (DLL) that calls several methods to produce a randomly generated string (outcome) based on parameterized property values. The interface is the single API call (and the property variables) in an automated test, so we might consider the DLL that contains this API as the ‘system’ under test based on Beizer’s definition of component. Also, since students don’t actually see the underlying code, API testing in this context is ‘black-box’ testing. clip_image001

The debate of who is responsible for API or component testing is tangential to the practice. I promote API testing because even in today’s extreme programming and TDD development lifecycle models we testers are still finding way too many ‘functional’ bugs (as opposed to behavioral bugs) during the integration and system levels of testing.

Also, generally (not all) software is designed and developed in layers similar to this simplified illustration. In well-designed, more easily testable projects the business logic or logical functionality is (should be) contained in classes or libraries (DLLs), and the public methods or APIs in those libraries know nothing about the user interface. and visa versa. End user inputs at the GUI are marshaled to the business logic layer via event handlers and properties (get/set accessors in C#) in different classes.

So, it shouldn’t be a surprise to anyone that certain categories of functional issues are more easily exposed at the API level of testing as opposed to testing through the UI. In fact, sometimes the UI properties and event handler layers in one project may actually mask some bugs in the APIs which aren’t exposed until much later when someone else uses that API in a different application or feature. The value of API testing is that a lot of functional testing can be performed very early in the project cycle, and functional testing can progress while the UI layer is unstable or in flux.

I sometimes think we are stuck testing from the end-user’s perspective. It seems that we often approach testing by trying to expose both behavioral type bugs and functional type bugs by testing completely through the UI. But, if we think of testing in layers the same way many products are developed then I wonder if we could better focus our test designs to target specific categories of functional bugs earlier and concentrate on behavioral issues and end-2-end customer scenarios when we have a more stable UI?

Filed in General Testing Topics | Tagged | Comments (5)

Code Coverage: More Than Just a Number

Thursday, January 21, 2010

When I was growing up I would sometimes go down into my grandfather’s basement. He had amassed a variety of tools during his lifetime and he was an excellent wood craftsman. I wasn’t allowed to touch any of the power tools, because his rule was, “if you don’t know how to use a tool properly then you shouldn’t play with it.”

Of course, I am a bit of a hard head (even back then) and one day I started playing with the wood lathe while my grandfather was upstairs. Everything seemed to be going pretty well until I pushed the chisel in too far too fast and the wood split and went flying. One piece shattered the overhead light and the other piece ricocheted off the back of my hand leaving an nice gash. I shut off the machine and ran upstairs. After my grandmother cleaned and wrapped my hand, my grandfather made me go back downstairs and clean up the mess and stood over me with a stern look of disapproval making sure I wiped up my blood trail. After that incident, I heeded my grandfather’s advice, at least in his basement shop.

Anyway, with the recent discussions of code coverage around the testing blogosphere I started thinking about what was really being discussed. The discussions (as is the case with most discussions about code coverage) were not actually about the application code coverage as a tool, but more about the code coverage metric. And more specifically the discussions were about how not to assume a high measure of code coverage implies something is well tested. Interestingly enough, 2 years ago I wrote a post illustrating how the metric can be gamed and how the code coverage measure tells us nothing about quality or test effectiveness, but also alluded to how it might be used more effectively.

I thought that how the metric is sometimes misused is mostly self-evident, but then I realized that almost every time testers start talking about code coverage the discussion tends to focus on the metric. This may seem a bit harsh, but if a person’s only contribution to a conversation about code coverage is about how the metric doesn’t relate to quality or testing effectiveness then that person should not be allowed to play with hammers, and employing more complex tools such a wheel-barrows are well beyond that person’s comprehension.

Only thinking of code coverage as a means to get some magic number is akin to thinking “how many nails can I pound with this hammer. The metric itself is mostly irrelevant; and it is completely irrelevant if you don’t know how to interpret it in a way that helps you as a tester. Think about it this way; if we told our managers “our tests achieved 80% code coverage” some of our managers would be elated. (Of course IMHO, these types of managers are metric morons.) But, what do you think these same pointy headed number zombies would say if we told them “we ran our tests and we only missed testing 20% of the code.” I suspect they would start pacing back and forth in the room mumbling “We must run more tests, we must run more tests.”

When we stop thinking of code coverage as a simply measure where our only use of the tool is to try and achieve some magical number then perhaps we can start thinking about how to actually use code coverage as an effective tool to help us design tests (in under-tested or untested areas of the code), reduce potential risk, and possibly even drive quality upstream.

For example, one of my mentees is currently working on a project that uses just in time code coverage as a tool to evaluate how tests exercise changed code and downstream dependencies prior to checking code changes (e.g. bug fixes) back into the main tree. The initial pushback by some members of the team (including some pointy headed managers) was “code coverage doesn’t tell us about product quality” or “its too hard to achieve 80% code coverage” (although no such goal had been mentioned), and my personal favorite, “it’s too difficult to get everyone to measure coverage.” I reminded my mentee that the project is not about achieving some magic number, and in fact, it’s really not even about measuring at all. It’s about using the tool to discover information and to help us design additional functional tests at the API or component level that we might otherwise overlook to help prevent downstream regressions. In a nutshell, its about using code coverage as a defect prevention tool in this case.

Bottom line, code coverage is a tool! If you don’t know how to use it to improve your testing, well…

Filed in General Testing Topics, Testing Practices | Tagged , | Comments (1)