I.M. Testy

Well, it has been 60 days on my new team. It has been a challenge going from an IC back to a management role. Time management is a huge challenge. I also am still ramping up on our feature area (I now know more about social networks then I ever thought I would). By day I find my time split between people management (team meetings, 1:1’s, etc.) and project management (triage meetings, status meetings, external partner sync-ups, etc.) At night I read through the API specs and brush up on my C++. While I come up to speed I am fortunate to lead a great team of highly competent SDETs. Someone once asked me if I were to ever leave the classroom and go back to manage a team in a product group if I would hire people as smart as I was. I replied, “Well, I am not really that smart, so I would want a team of people who are much smarter than me.” And, that’s exactly the team I have!

So, what does my team do? Essentially we are responsible for integrating the social networking functionality onto the Windows Phone devices. The Windows Phone is already the “real Facebook Phone” according to Wired magazine, and also helps you manage feeds from your LinkedIn contacts. The next release will also include Twitter integrated into the People’s hub (peeps) as well so you won’t need to install and switch between multiple apps to read or send updates from your contacts on different social networks. (The photo illustrates key aspects of the Peeps hub for those of you that don’t have a Windows Phone yet).

How we test in our corner of Microsoft

Last year I wrote about testing at the importance of functional testing at the API layer. So it is interesting that 1 year later I am leading a team that does just that. Our feature area includes the functional capabilities of the APIs that integrate social network features into the Windows Phone. 100% of our automated tests are designed to test functional scenarios at the API layer. Like many other teams at Microsoft we have daily builds of our feature branch. In addition to code reviews, our dev team runs their unit tests and a series of critical tests authored by our test team before checking in fixes and updates. Then an automated test suite is fired off in a lab on each nightly build to further validate the code changes did not destabilize key functional areas. Next lower priority automated tests are ran in the lab, as well as stress, performance and other test suites. This is continuous integration continuously.

While the team members are highly competent coders writing tests in C++ and some C#, they are more than developers in testing roles. Although a sister team is responsible for testing the customer experience, we also spend a time exploring our functional area via the user interface using an emulator or a device. This helps us develop customer empathy, but more importantly when our automated tests expose an issue we are better able to explain how that anomaly might impact our customer. We also spend a lot of time eating our own dog food and self hosting on test devices and some of us even flash our personal phones.

So, while I feel I am starting to come up to speed and beginning to contribute to the team I know I still have a lot more to learn. But, most importantly…I am having a blast!

On March 1st I switched roles at Microsoft from an individual contributor (IC) in the Engineering Excellence team to a test lead in the Windows Phone team. This not only meant going from an IC role to a role in which I would lead a team of people, but also going from an academia/consultant role right into the fire pit of the newest Windows Phone 7 release. People often relate the experience of starting a new job at Microsoft, or changing jobs within Microsoft as ‘drinking from a fire hose,’ or in my case right from the hydrant itself.

It has been more than 8 years since I directly managed a team. In that time some ‘operational’ things have changed, but most importantly I have changed. Personally, I think my responsibilities as a lead include helping the people on my team grow their career, providing a vision and guidance towards clear goals, protecting the team from political fallout and pressure, managing the features(s) I am responsible for, preventing project and personal ‘fires,’ and also actively participating in the testing effort. There are skills, experiences, and values that I have learned in the past that will help me in my transition, there are some adaptations to make, and there is a whole lot to learn.

So, here are a few brief thoughts after my first 30 days back in the saddle.

Meet the “people” on your team – this seems rather obvious, but I don’t mean meet the team, I mean really get to know the people on your team at a personal level. You will be spending a lot of time working together, and occasionally socializing outside of the office at morale events, team lunches, etc. People have lives outside of ‘the project.’ Get to know the people and actively support their work-life balance.

Many times the first 1:1 meeting with a new manager mostly spent discussing my role on the team the project status, or other items related to the business. Leading people is different than leading projects.  Leaders should be able to relate to the people on his/her team at a personal level, so get to know them a bit. Getting to know your team not only helps build trust, but helps a lead think about parental leave, vacations, children getting sick, dropping off/picking up children at school, wedding planning, and a host of other things that are not accounted for in the project schedule.

Meet your peers – Within the first 2 weeks I set up meetings with my developer and program manager counterparts. At Microsoft the Dev/Test/PM triad meets regularly (often daily), and must work together to manage the project and make critical business decisions. I also wanted to get their perceptions about the test team. Your peers can brief you on group policies, help you get ramped up on team processes, and also bring you up to speed on the project. They can point you to the right aliases to join, and make sure you are scheduled for the appropriate meetings.

Ask (a lot of) questions – you don’t know jack! Of course, leads/managers generally have pretty good track records and are hired not only on past accomplishments but also on future potential. But, when you move to a new team you are going to spend a fair amount of time learning new things. Teams are different, but many people say that it takes about 6 months on a new team before they feel comfortable and able to fully contribute. That doesn’t mean you have 6 months to slack off, it means that the first few months you might be cut some slack. But, the pressure is on and you need to step up and push the envelope of your personal development. Ask your peers and the people on your team for help; they often have a vested interest in your success.

Time management – There are triage meetings, the 1:1 meetings with directs, leads meetings, meetings with partners, etc. Some days I might have an hour or 2 of free time between 9 am and 5 pm that I don’t have meetings, but those days are rare. As a new lead/manager your time is going to be pulled in a lot of directions at once, and time management is a lot more challenging. Expect to come in earlier and sometimes stay later or log on in the evenings to prepare for meetings or catch up on things. Oh yeah…you still have to make personal time for yourself.

Provide stability – change often invokes uncertainty. The people on your team are usually highly capable and experienced. They likely understand their role on the team and know what needs to be done in their areas of responsibility. One of the first things a new lead/manager should do is provide some degree of stability, and provide some clear goals to help focus the team on achieving the project goals.

So far, my transition back into the product groups and a management role is at times a bit overwhelming, but it is a total rush and each day brings new and exciting challenges. On to the next 30 days!

In response to my last post, Shrini suggested, “You should probably do a post on types of bugs that unit testing (or developer testing at any level) would’nt catch. That would be a fitting reply to all those who swear by automated unit testing/API Testing.”

I am a big proponent of well-designed automated tests, and I have written a lot about the value of automation as an effective tool in the development process. But, I also know that automated tests find a relatively low number of bugs throughout a product lifecycle. If you think the primary goal of an automated testing effort is to find lots of bugs, or the same types of bugs as a person then you probably don’t know very much about test automation. Personally, I don’t think of automated testing as a proxy for the tester or as a bug finding solution. Sure, some automated tests can help find bugs, but more importantly it is one approach I might use for

• defect prevention (esp. computational logic problems),
• earlier identification of key integration issues,
• potential degradation of critical areas (battery, performance, memory),
• efficient execution of redundant ‘checks’ (if necessary or for confidence)
• more effective/precise ‘oracles’ as compared to humans
• cost reduction in long term sustained engineering

In my new role at Microsoft I am leading a team of great SDETs that tests primarily at the foundation (API) level. Everyday I see first hand the value that automated regression testing at the unit and API levels of testing provides to the overall production lifecycle (because we build everyday). While the tests we run ultimately affects the customer’s experience, it also helps reduce our overall production costs and drives certain aspects of ‘quality’ upstream. This is especially important in large scale, enterprise systems where a build breaks or integration failures can be costly and lead to unnecessary delays.

But, this is just one level of testing, and I realize that most testers are testing at the ‘system’ level of testing and rely heavily on GUI automated tests. I have never been a big fan of GUI automated tests; especially GUI regression testing. This is not to suggest that all GUI automation is bad, and there are several situations where GUI automated tests can be especially valuable to a test team. However, there are a few situations where I think automating tests that manipulate the GUI is mostly a complete waste of time such as attempting to emulate the behavior of a customer “scenario,” or trying to verify the ‘correctness’ of what the customer “sees” (e.g. visual verification).

I once told a colleague that the computer is really bad at emulating “me.” I said, “for example, sometimes when I type I hit the wrong key, or I lay my finger on a key for too long and that stupid sticky keys message box appears, and sometimes my hand position on my laptop causes my insertion point to jump to some random point in the text body and I have to reset it to the correct location. You can’t automate the unpredictability of me typing!” He said, “Sure I can!” I said, “Ok..I know that we can automate randomness or errant behaviors to some extent, but why would we?”

I sometimes think that in our zeal to “automate everything” we forget that our products often get a lot of “face time” through self-hosting, product partners, beta releases, and other strategies that are intended to get feedback on unanticipated or escaped functional issues, behavioral issues in how people might use the features in different ways (scenarios), and of course the “look” or visual anomalies that might occur while using the product.

As an example, the other day I was searching for a new program for my students to practice their GUI automation skills (yes, while I generally dislike GUI automation it is still a good skill to have). I came across text editor application called XINT. Within minutes of downloading the application and exploring the features I found a bug with the feature that inserts a URL into the text body.

As a simple example I was going to show how we can automatically go through the menu structures to make sure there are no changes, and that the menu items trigger the appropriate events (e.g. displaying a dialog). I was going to develop an automated test demo that systematically marched through the menu structure and validated the expected event triggered by that menu item. An automated GUI test such as this could provide a high level ‘check’ much more quickly than could be performed by a tester. Also, it automates a redundant ‘check’ of the application under test that we might want to perform on each new build to potentially ‘look’ for changes. I wouldn’t expect this automated test to find a lot of bugs, but it would clue us in very quickly to any changes in the menu structure and any anomalies with basic functional expectations triggered by those menu items. Quick and simple.

So, we get to this menu item and programmatically simulating the menu item “click” via a SendMessage() call to the appropriate menu item the ‘correct’ dialog appeared. OK so far. We are not testing the “insert a URL” functionality; my high level ‘check’ is simply checking that the correct event occurred (in this case a dialog appeared). So, now I am going to send a message to “click” the Cancel button and my expected result is for this dialog to go away and focus will return back to the application under test (XINT). But, in this case the URL insertion dialog is repainted with the sample text removed, and a new label. (It gets even better because clicking Cancel again forces the sample text into the text edit control. You have no choice at this point…you selected to insert a URL…so you are going to get whether you like it or not!) But, there is a bug, and my automation could have found it.

But, a little more exploration and I discover another anomaly that almost defies logic, and that automation would certainly not have found. When I pressed the ALT key, I noticed something odd. The fricking Cancel button disappears! However, as far as the ‘system’ is concerned the handle to this control is still there and I can programmatically send a message to that button control. But, to me the user…it’s gone!

This is just one example of the types of issues that automation is not especially suited for, and even trying to automate a test for these types of issues is not just an effort in futility, I would say it is damn near insanity. Of course there are other types of issues that automation is not especially efficient or effective in detecting such as ease of use, consistency in layout or behavior, general “look and feel,” and most importantly customer scenarios or user stories.

Bottom line, use automation for things that computers are really good at such as computational logic and redundant ‘checks’ that we might want to do after each new build. And, use humans to test for the things that humans are really good at which often happen to be the things that will delight your customer if you get it right, or the things that will piss them off if you screw it up!

…developers basically suck at testing and relying on developers to test your product will cost you money! Hey you pointy haired managers who think you can save money by cutting testing costs…are you listening?

Don’t get me wrong, I don’t think developers suck in their jobs; in fact, I know many really great developers (but, I also know and have seen the work of some hacks who call themselves developers). Great developers are usually pretty good at writing unit tests that test discrete functions. Some developers are even good at writing higher component or API level tests. Unit testing and API testing are valuable approaches to help identify certain types of problems in a new build after refactoring or bug fixing.

I don’t distrust developers; I think similar to testers they are concerned about the quality of their code and of the product they are working on. But, testers have a different mindset as compared to developers, and we have different perspectives. Some testers may write, or partner with developers for component level or API tests, but most testers generally focus on integration and system levels of testing. Using various approaches in our craft we often expose functionality issues as well as behavioral or usability issues that might adversely your customers.

For example, the other day I came across a website to make a travel reservation. When I get to the payment page I realize that I can’t complete it because it is missing 3 months.

“Hmm…this is odd,” I thought to myself. I refreshed the page thinking there might have been a glitch in getting the list of months. Ultimately, I had to call the travel company to make reservations.

At first they didn’t believe me, then they confessed they just commissioned this new website. So, I told them that after they fire the developer and hire some testers for the next go round they should also figure out why “test” is listed as a State.

Or, among other things, why this dropdown list contains seeming unrelated options, why some options are all upper case, and why one city name (scottsdale) is not capitalized. And of course our all time novice faux pas…”abc_test.”

These are just of few of the examples of the lack of perceived quality of this site. And the company really expects customers to enter personal data into this site; especially credit card information?

Now, of course testing doesn’t guarantee the absence of bugs, but I am pretty confident that most testers would have easily found these before staining this company’s reputation.

So, the next time your company thinks it can cut testing costs by having developers do the testing…point them here and say…I sure hope the developers you hire do a better job than this.

I am almost through my first week in my new job here at Microsoft and trying to get my head wrapped around everything. It is really exciting to work on new technologies and in a feature area that helps users stay connected with friends and family in amazing ways. For now, let’s just say that I feel like I am treading water which is not unusual when changing jobs within Microsoft. But, of course, that is a good thing. If I felt like I could’ve easily stepped into this job I would not feel challenged, I would not open my mind to learn new things, and I would likely be bored after a few months. Yes…I am a bit overwhelmed in a very exciting way!

Of course, I am still teaching classes at the University of Washington Extension, and am currently teaching a class in test automation design. A few sessions ago we hit an issue with some canned examples failing to run properly on 64-bit operating systems. This problem appeared for 2 reasons:

• I developed the lab examples 2 years ago before I upgraded my home system to a 64-bit machine and have only ran them in the classroom environment since
• I hard-coded the Program Files folder environment path to C:\Program Files.

Of course, it is a bit embarrassing as an teacher when your samples do not run (it’s even worse when they don’t even build, but that is a different story). In a training situation it is often easier to control the environment to minimize problems and allow learners to (initially) focus on the main topic or skill being presented. But, the downside is that many of us don’t work in ‘controlled’ environments in the real world. And while I am adamantly opposed to hard-coded strings in code, I find that sometimes I ignore my own advice in samples or demo code (and it often comes back to haunt me).

The specific problem was that on a 64-bit operating system the application is installed to the Program Files (x86) folder instead of simply the Program Files folder and the application failed to launch when we attempted to start the process. The solution is actually quite simple in this case because we can actually call the ProgramFiles member of the Environment.SpecialFolder enumeration. For example, a hard-coded path on a default installation

string autPath = @"c:\program files\[autFolder]\[autName.exe]";

failed on a 64-bit system, so a better approach to detect the ‘correct’ program files directory might be

string autDefaultFolderName = "[aut_folder_name]";
string autExecutableName = "[filename.exe]";
string autPath = System.IO.Path.Combine(
  Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
   autDefaultFolderName,
   autExecutableName);

The ProgramFiles member of the Environment.SpecialFolder will return Program File on a 32-bit machine, and Program Files (x86) on a 64-bit machine.Of course, this works fine for the default installation folder, but if for some reason you need to run your automation on installations of an application under test in a non-default directory then you would need to qualify the path in other ways.

Also, this is just one difference between a 32 and 64-bit operating system environment, but there are other situations where we want to detect bit-ness so we can write a single test rather than one test for 32 bit operating system environments and another for 64-bit operating system environments. Fortunately, the .NET 4.0 framework makes it really easy to detect whether a machine is 64-bit or not with the Environment.Is64BitOperatingSystem property. For example, you can control flow for 64 vs 32 bit with a simple decision such as:

if (Environment.Is64BitOperatingSystem)
{
  // TODO 64-bit stuff
}
else
{
  // TODO 32-bit stuff
}

But, if you are running the .NET 3.5 framework or earlier it becomes a bit more complicated to detect 64-bit operating systems. The following method can help determine whether or not your operating system environment is 64-bit

public static bool Is64BitOperatingSystem()
{
  if (!string.Equals(Environment.GetEnvironmentVariable(
    "PROCESSOR_ARCHITECTURE"), "x86", StringComparison.OrdinalIgnoreCase) ||
    !String.IsNullOrEmpty(Environment.GetEnvironmentVariable(
    "PROCESSOR_ARCHITEW6432")))
    {
      return true;
    }

     return false;
}

There are also Win32 API functions we can P-Invoke such as GetSystemInfo,

So, rather than having separate automated tests for 32 and 64 bit operating systems a better test design approach might be to simply detect the bit-ness and auto-magically set variables or redirect the control flow path through your test for the appropriate operating system environment.

Well, it has been quite some time since I last posted to this blog, and I apologize to my regular readers for the inconsistency. Since the December timeframe I have been a bit busy behind the scenes searching for a new direction at Microsoft, and last week I was finally able to officially announce that I will soon be a Principal Test Lead in the Windows Phone group on the Communications team. My feature area will be Social Networking. I am so excited about the change, and getting back into the “real-world.” I have already started making the transition (more mentally than physically) and I am recalling the feelings I had when I first started at Microsoft. It’s incredible!

For the past 5 weeks I have also been teaching a course on Software Testing at the UW on Monday and Wednesday evenings. Tonight was my last class with a final that culminated in the attendees writing “scripted tests” from vague requirements, running those tests followed by some exploratory testing, and finally reviewing structural coverage effectiveness and defect detection effectiveness of the testing approaches.All in all, it is a fun class.

One lesson in the class focused state-transition testing.  The focus of the lesson is to identify the important “states” the machine is in at a given time, and identify various ways a customer might navigate from one state to another (traversals). In my opinion, one of the easiest ways to present the concept of modeling states is with a setup or installer of a program with a minimum number of configuration variables. I have found that minimizing the configuration variables helps people focus on “state” rather than be distracted by the different configuration settings (combinatorial testing).

After a brief explanation and a demo, I cut them loose on a problem to solve. In this case I used a shareware application called 3rd Plan It. The objective was to identify all possible paths a customer could navigate while setup is running and the important states the machine might be in at any given moment in time during the installation process. The value of a state diagram or map is that it can help us identify paths that we might miss via exploratory testing. Also, by identifying important characteristics of each state we can better assess whether the machine is in the expected state (oracle) as we navigate the various paths towards the objective of our test (in this situation it is the installation of a program).

The state map below illustrates a model of what we might consider the “important states,” and the various ways to traverse or navigate between states. Coming up with a state transition diagram often requires us to explore the feature. But, it also seems to force us to really understand the feature we are exploring at a much deeper level, and consider paths that we might otherwise have missed. A state transition diagram also helps us think about what “state” the machine should be in at any given moment during the setup process (it is also important to understand the “state” within the appropriate context).

This is not to suggest that we need to create a state transition diagram for every feature. That would be silly. But, even if we don’t create a state diagram on paper we certainly create one in our mind. I suspect that most of the time we are purposeful in our testing approach and use heuristic patterns as we design tests (that is of course unless someone is high on PCP or some other hallucinogen and are completely reckless in their test approach and simply bang away indiscriminately in hopes of finding bugs).

Of course, some people found 1 or 2 issues using an exploratory approach without first creating a state diagram. But, after viewing the state diagram those folks were also able to realize paths they had not considered or traversed and were able to find more issues. The single biggest epiphany that most people have after participating in this exercise is how much more they understand the feature they are modeling compared to what they thought they knew.

Then of course, when the setup is littered with problems, the uninstall functionality which is part of the setup engine has its own set of problems as well.

Often, after teaching concepts and demoing examples of situations where various testing techniques might apply I am asked how diligently we should apply these patterns of testing. Of course, functional techniques are only one way to think though a problem. Test techniques are useful to help us systematically analyze a problem, and come up with some model of what we are testing. Models are sometimes useful to help us think of the problem differently, or identify things we might otherwise miss. State transition testing is a useful functional technique that is useful for helping us visualize various ways a customer might use a feature, and in turn come up with various tests and potentially find more issues.

I am generally not a big fan of static test data. I do know that in the proper context static test data can provide some value. Of course we should be aware of the common problems with files of static test data or (even worse) hard coded test data in a test case. Some problems with static test data include:

• Stagnation – static test data may add some initial value, but over time simply reusing the same test data over and over in a test diminishes the value of that test. For example, retesting the same name strings in a first name input textbox is not providing any new information if those ‘static’ names worked in the previous build and the underlying functionality has not changed.
• Contextual blindness – sometimes we have files of static test data that was identified as “problematic” in one situation (context), so we reuse the “problematic” test data regardless of the context. In 1995 I wrote a white paper on “problematic double-byte encoded characters (DBCS) explaining why each code point was problematic in a given context. For example, a Japanese character that began with a 0x5C trail-byte might be problematic in a filename on an ANSI based system that parsed characters by bytes instead of wide bytes. This is not true on Windows systems where the default encoding is the Unicode transformation format of UTF-16. However, some people continue to use obsolete DBCS problem characters perhaps because they don’t fully understand the underlying contextual differences between ANSI based encodings and Unicode.

Perhaps on the opposite end of the test data spectrum is random test data. Many of you that read this blog or have heard me speak know that I am a big proponent of parameterized random test data generation. Parameterization allows us to better model our test data. I know that even parameterized random data can be crafted to be representative of real data, but it is not “real” data.

But, there may be a happy medium between static test data and random test data. And, best of all it is abundantly available. One of the best sources for (especially non-English) test data comes from sources that most of us already use on a daily basis. The test data source I speak of are social networks.

I have met many wonderful people from around the world both in person and virtually, and stayed in contact with many of them. Last year while keynoting at the first software testing conference in Vietnam (VistaCon 2010) I was privileged to meet my dear friend Thuyen, who helped organize the conference. Since the conference we have stayed in contact via email and Facebook. When she posts on Facebook it is usually in Vietnamese. Since I don’t (yet) read Vietnamese I use Bing Translator to help me figure out the comment.

Last week she had an entry on her Facebook wall that began “Tối nay vô tình nghe trên TV 1 bài hát mà giai điệu…” So, I copied the entry and opened Bing Translator to translate the entry.

Many of you will quickly notice the strange anomaly in the translation. I initially thought that this service might be incrementing this numeric value for some reason, but when I changed the number value to 2 the number 2 displayed in the translated string. I tried various other numbers and quickly discovered that 6 incremented to the number 7, 8 decremented to 7, and 9 decremented to the number 3. I didn’t see a clear pattern here so I thought this might be an issues resulting from parsing a particular sequence of characters.

So, I modified different parts of the string (removed words) to narrow down the problem. I found the string “tình nghe trên TV 1 bài hát mà giai điệu” contained the problematic sequence. Removing any ‘word’ from this string displayed the translated string with a number of 1, with the exception of 1 word. Removing the word “nghe” from the above string resulted in the translation illustrated below.

By the way…the Google translation engine doesn’t fair much better. And, the results are different between www.google.com/ig and http://translate.google.com.

But, the purpose of this post is not to illustrate this particular bug, but to give you ideas of how we can use social network feeds in our testing. People around the world use social networks and you can find “real world” strings in various languages that you can use as test data in various contexts. Most of the time this ‘test data’ will not likely result in a bug; but sometimes it can reveal interesting issues. Best of all, strings taken from social networks are not some manufactured static or random test data. Using strings copied from social networks is about as “real world” as we can get…this is the “data” from our customers.

Well, it’s another new year. Like a lot of folks I have spent the last few days reflecting on the past year and contemplating this coming year. I won’t bore you by rambling on about my thoughts, reflections, or ambitions; they are mostly personal. Professionally, I will continue to strive to improve myself in my chosen discipline, seek out new challenges and opportunities, and share my experiences with those who follow my posts. The beginning of the year is a busy time for me arranging my conference schedule (which I am cutting back on) and preparing to teach a software testing course and a software test automation course at the University of Washington, and getting engaged with 2 key internal projects intended to help improve our internal engineering processes at Microsoft.

Many of you know that I started at Microsoft on the Windows International Test Team, and internationalization (I18N), globalization (G11N) and even localization (L10N) are topics that I have always been interested in. In 2009 I posted a series on localization testing (Part 1, Part 2, Part 3, and Part 4). Last year, I designed and developed an internal class on globalization testing basics for non-globalization experts who want to incorporate globalization test strategies into their test designs in an attempt to find bugs sooner and drive quality upstream. As the world becomes even more connected and there is rapid growth of customers around the globe it just makes sense that we need to design our software that our customers will value in their (local) context. So, over the next few weeks I will do a series of posts on globalization testing.

But first, I’d like to share a globalization type bug that I found…or should I say that found me. This is a great example of bugs that you might come across using a strategy I used in my previous teams to help drive international sufficiency testing upstream. In this post on international sufficiency testing I explained one strategy was to get folks on the team to set their default user locale to something other than US-English. This particular bug was initially detected the day after writing the post discussing a bug that was found by an SDET taking my globalization testing basics course. To replicate the bug for screen shots, I customized my number format via the Region and Language control panel applet by changing the decimal symbol from the default period character (‘.’) to the small Latin letter d (‘d’). After writing the post, I did not restore the machine to it’s default state (using the period character as the decimal symbol).

The next day I came into the office, logged onto my computer and noticed the following cryptic scripting error message on the desktop. I hadn’t launched any other app yet, so I deduce this is likely caused by some process that starts automatically when logging in. So, I give the faithful Windows 3 finger salute to bring up the task manager and scan through the list of running processes.

It didn’t take long to associate this error dialog with the communicator.exe process. But, at first I wasn’t quite sure what was causing this error. Later that morning, I reset the user locale settings back to the default settings. After lunch I logged back onto my machine and no error message!

So, being a tester, curiosity got the better of me and I felt compelled to try to replicate the initial anomaly. To make a long story short, it didn’t take too long for me to put the pieces of the puzzle together and figure out the customized decimal symbol was causing an error. But, after troubleshooting a bit it was much worse than thought because this error just didn’t occur with a letter character, it also occurred when I changed my decimal symbol to a comma character (‘,’) which some international locales use as the decimal symbol. So, I contacted my friend and colleague Alan Page who just so happens to work on the Communications team.  He quickly looked into it for me and announced this problem does not occur in the latest release. (Note to self…update your machine to the latest self host bits.)

What is important here is that this bug did not require a different language version. I didn’t need to know a different written language. This bug was exposed simply by customizing the settings in the Regional and Language control panel applet and using the system as a ‘normal user.’ So, what I did need to know was a bit of technical knowledge of the system (specifically around the National Language Support or NLS) and how to configure the system to help me expose potential globalization issues. It’s not magic and it’s not rocket science. So, over the next few weeks I plan to share some testing approaches to help other testers find problems similar to this and incorporate globalization testing into their test strategies.

This has been a rather eventful year; both positive and negative. I have done quite a bit this past year professionally and personally, but certainly not as much as I had hoped to accomplish. As the year is winding down, my attention is focused on fulfilling my daughter’s Christmas dreams and wishes. Christmas is still a magical time for her and I both.

I don’t know if she still truly believes in Santa Claus, but at least she does a good job of pretending. Tonight we will perform our yearly ritual of putting milk and cookies on the fireplace hearth and toss some carrots out onto the back lawn for the reindeer. We won’t build a fire in that fireplace on Christmas eve because she doesn’t want Santa Claus to get burnt as he comes down the chimney. Of course, after she is tucked soundly in bed, I will eat the cookies, drink the milk, and throw the carrots into the greenbelt behind my house for the critters to munch on.

She is delighted by finding the present she asks for from Santa Claus under the tree. She only asks for one thing from Santa so “he” feels obliged to satisfy her wish. There are some smaller gifts as well; puzzles, books, American Girl doll accessories, etc.). But, it is not the presents that make this a special time for her and I. It is all the things we do leading up to Christmas day that make this a wonderful time of year for our family.

Well, I am won’t blather on about Christmas, or what a special time it has become for me since my daughter was born. Instead, I would actually like to thank Michael Larsen for taking the time to write several detailed reviews of How We Test Software at Microsoft. As I read his reviews of chapter 5 and chapter 6 (the ones I wrote) he actually brought out several key points that I think were a little obscure such as, “Bj champions the use of Exploratory Testing (ET). ET is a great way to get a handle on an application, especially when a tester is new to it.” Michael also caused me to reflect on the following point in the book, “Bj argues that Exploratory Testing can be sufficient for small software projects, or software with limited distribution, or software with a limited shelf life. But that it doesn’t scale well for large-scale, complex or mission-critical applications.” In retrospect I wish I would have expanded on that statement by saying that in our experiences at Microsoft ET does not scale well as a primary approach to testing, and the teams that tried to use ET as a primary approach have changed their strategy. However, ET is used throughout MS; it always has been and always will be a valued tool in any professional tester’s toolbox. Fortunately, Michael was able to read between the lines and wrote, “I get what he’s trying to say, in that Exploratory Testing techniques will not be the be all and the end all with testing (nor should it be; all tools have their right time and their right place).” Right on!

So, whether you have read the book or not, I highly recommend that you visit Michael’s blog to read the reviews for insightful thoughts on the book. I also recommend his other thoughtful posts on the topic of software testing.

A person who speaks 3 languages is trilingual; a person who speaks 2 languages is bilingual; and a person who speaks one language is an American (or Brit…depending on which side of the pond you are on).

I work at a company with tremendous cultural diversity embodied in very smart people from around the globe. Yet, it seems that when people come to Redmond their cultural uniqueness seems to disappear. Maybe it is an engineering thing, maybe it is an assimilation thing, but whatever it is, it is not good thing especially considering that a growing number of our customers come from non-US markets and the way they interact with software is often quite different compared to the US centric scenarios and personas I so often seen used to design and develop our software and services.

Monday afternoon I taught a class on globalization testing basics geared towards SDETs who are not experts in globalization. These testers usually come to the training mostly because they’ve been tagged by their manager to help with globalization testing efforts on their team. But, what they learn is that with a little understanding of some basic concepts and with the aid of a few tools they can incorporate international sufficiency testing concepts into their test designs (both exploratory and automated) and drive quality upstream (e.g. find some bugs sooner).

One of the topics I discuss in the class is customizing the current user locale settings. I also discussed this in an earlier post this year. Customizing the national conventions settings for the current user locale include things like custom date and time pictures, as well as number and currency formats. Of course to really understand custom locale settings we should have a good technical understanding of the national language support (NLS) API functions and specifically the LCType parameter Locale Information Constants.

<tangent alert>  When I and other people talk about encouraging testers to develop their technical skills or knowledge there are some people who simply assume that we mean “technical” equates to coding. This is really a shallow view of how they interpret ‘technical.’ When I refer to technical knowledge and skills I am primarily referring to an understanding of how the system (or parts of the system) work and how to use that knowledge and their skills to design more effective tests.</tangent alert>

In this case, our technical understanding of the variable arguments that can be passed to the lpLCData parameter of an NLS API function such as SetLocaleInfo() for the various LCType constants can help is more fully explore the functionality of features that use the operating system’s NLS settings. For example, the constant value for the decimal symbol for number formats is LOCALE_SDECIMAL. When this LCType is specified the value we can pass to the lpLCData parameter is a string of up to 3 characters in length (“maximum number of characters allowed for this string is four, including a terminating null character.”)

In class I often use a custom decimal symbol as an example of customizing national convention settings by manually changing the number format decimal symbol from a period character (.) to random Unicode characters. From previous examples I knew of a minor anomaly in the calculator (calc.exe) in which the decimal button changed to show “abc” (or some other random string of 1 to 3 characters), but the decimal symbol in the result window only displayed the first character (“a”).

But, in this week’s class I gave an example of changing the decimal symbol from a period character to the literal string “dot.” Then, Andreas Schiffler, one of the SDETs in the class used the example “dot” string in the calculator and quickly discovered an anomaly.

It seems that the calculator does not like the lower case letter ‘d’ as a decimal point. If you customize the decimal symbol for the number format to the Unicode lower case letter d, then launch the calculator (calc.exe) and press the decimal symbol key (or perform any calculation in which the result includes a decimal value) the calculator result window will show “Overflow.”

Andreas initially thought this might be caused due to the fact that the letter ‘d’ is a formatting character. However, we quickly discovered the upper case letter ‘D’ is also a formatting character, but the upper case ‘D’ and other formatting characters do not cause a similar incorrect condition. (I went home and automated this test looking for clues using the GlobalTester library and pumped in over 10000 random Unicode characters, and none of them have caused an overflow.)

Now this particular problem might seem out of context, or have no ‘real-world’ scenario because the commonly used characters for the decimal symbol are the period (.), the comma (,) and the space ( ) characters. And for the life of me I can’t think of any national convention in this world that uses the letter ‘d’ as a decimal symbol in their number formats. But, this application is using the NLS settings (at least to some level of implementation), and since we allow the user to customize these settings, then I shouldn’t expect that functionality to break. The bottom line is that something out of the ordinary is going on that probably needs investigating.

Sometimes simply changing the current user default locale settings might reveal basic internationalization issues. And sometimes customizing the national conventions and using randomized data for those conventions might reveal problems with how the developer implements national language support in the product. You don’t have to be an expert in globalization to discover these issues! You just have to know a little bit about the technicalities of NLS, and have a desire to potentially find some pretty cool, or at least weird anomalies earlier in your testing.