But, I think Shrini has some good points to make and encourage him to refute my writings with strong arguments based on factual evidence and specific examples or suggestions.

Saturday, February 10, 2007 4:21 PM by I.M.Testy

The inference that Shrini does not do this strikes me as fairly odious.

—Michael B.

Thursday, February 08, 2007 11:34 AM by MichaelBolton
# re: Regression Testing Strategies
Pardon me; I meant implication, rather than inference.

—Michael B.

Thursday, February 08, 2007 11:37 AM by MichaelBolton

Please clarify what you mean by impact analysis. In my experience impact analysis is used to assess or predict behavioral changes as a result of a modification to a “system.” (For example, if I rename a variable or declare a different data type for a variable I must also assess how that change will impact the rest of the code.)

I did not say or imply that ALL high priority tests should be included in the regression test suite. The size of the regression test suite must be managed properly for the suite to be effective. So, it is important to carefully evaluate each test that is incorporated into the regression test suite.

I did say that “any” functional bug that is found and fixed should be included in the regression test suite based on the assumption that if the team made a business decision to fix it once, they probably don’t want that defect to reappear. However, are role as testers involves making smart decisions all the time, so perhaps we should also analyze these tests from an ROI perspective as well.

For example, some functional defects are not catastrophic (no crash, no data loss, etc) and the probability of a user encountering said defect may be extremely low, but we fix that defect. Should that defect be put into the regression test suite? Probably not, because the cost of automating it would probably exceed any reasonable return. I might want to execute that test at some later milestone (just in case) but, I might not want to run it every regression pass. So, again, it is up to the tester to make smart decisions about what tests to include in the regression test suite.

Tuesday, January 30, 2007 1:45 PM by I.M.Testy

I am taking it that the two categories of tests to include in the regression test suite are selected after impact analysis. Or are you saying that ALL “high priority tests for commonly expected functionality” and ALL “functional defects that are found and fixed” would be part of regression test suite always?

Tuesday, January 30, 2007 3:28 AM by kvinayaks

This is a great question! The short answer is I don’t consider the BVT as part of the regression test suite. I do consider the BVT as a separate (and most likely the first) baseline measurement established after each new build.

But, in my experience the regression test suite (assuming it to be 99.999% automated) is often ran (almost) immediately following the BVT, and generally using the same lab machines as used for the BVT, (or more if necessary to distribute the load of the regression test suite.) I should also add here, tests in the BVT suite were not included in the regression suite (that would simply be redundant).

When I designed the BVT test suite for the Windows 95 international versions produced in Redmond I had a constraint of 30 minutes to validate the integrity of 4 language versions of each new weekly build of the operating system (This is before the single world-wide binary devleopment models often used today when each language version was recompiled with #ifdef’s – meaning there was functional differences in each language version.) At the pinicle the intl. BVTs were distributed across 12 machines in my office. (I still remember the warmth and constant whir of the fans when curling up to catch some sleep under the bench.)

As I developed the BVT suite from mostly manual tests to 99% automated my manager would occasionally ask me the status of the BVT and I would reply with vague, non-commital answers such as “well, it’s pretty good, but…blah blah blah.” To which he replied I must make a firm decision…no sitting on the fence post…it was my responsiblity to make a decision. My manager made it very clear that the purpose of the BVT was to establish 1 of 3 possible outcomes.

1) If the build failed pre-determined critical tests it was rejected and kicked back to development. Now this was a decision not to be taken lightly, because this meant that not only were the dev’s going to be working around the clock to get the weekly build out, it probably also meant most of the team would have to work the weekend to make up for lost time.

2) The second outcome could be what we referred to as Release for Test Only. This meant the build had some problems detected by the BVT, but some of the problems could be “worked around” and the build was stable enough to regress fixed defects, and continue testing large areas of the product. Some minor areas may not have been testable, but they were not usually critical areas. For example, in one build the menu links in the Start menu failed to launch the applets (Wordpad, Notepad, etc.) but a work-around to the problem was to launch the applet via the Run dialog or command prompt, or by double clicking the executable, so the build was released to the test team only.

3) The third option was to determine whether or not the build was Released for self-host (this was later called dog-fooding). This typically meant the new build passed the BVT (at least above 90%) and was deemed stable enough for everyone on the team (including managers) to format their main machines and install the latest build to conduct their day to day work. Trust me, when I got this wrong I was in the dog house for the entire week.

I should take this and make a new post about BVTs because I have more to add here. But, I hope you get the gist of the BVT and the distinction between the BVT and the regression suite.

By the way Vinayak, I like your blog…the last few posts are indeed interesting.

Thursday, January 25, 2007 12:33 AM by I.M.Testy

Can we consider BVTs a part of regression testing startegy?

v-vinaku@microsoft.com

Wednesday, January 24, 2007 6:47 AM by kvinayaks

Great Reply — Thanks. Let me get back to you with Strategies for building a good REgression Suite…

Shrini

Wednesday, January 24, 2007 5:29 AM by Shrini

I thnk we may both be saying similar things with different words. I stated “the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition.” What I did not explicitly state is that if the regression test suite passes then our defined baseline measurement has not changed and the “application passes the set”. This doesn’t mean there haven’t been changes in the code, this doesn’t mean there are still not a lot of defects in the system. It simply means that the criteria we used to establish a baseline measurement for identifying regression of specific capabilities or attributes of the system has not changed. (Perhaps we are getting hung up around what baseline measurements are.)

You stated “Execute a set of Tests (Regression test) to validate that Application passes the set.” I am going to assume here that you have well crafted stochastic tests that prove or disprove the same hypothesis each time they are executed (which is how we establish baseline measurements) and the ideal is the application under test passes the defined test suite. However, if a test in the regression suite fails, then the tests have in fact identified a change from the established baseline that requires investigation.

If you disagree and think we are still at odds then please provide specific arguments with factual data as to why “regression testing in the way that you [I] mention is not feasible.” You might also consider adding constuctive strategies how you would develop a regression testing suite. That approach would add more value to this discussion.

Also, my argument of responsibility is in no way similar to “Police are responsible to maintain Law and order situation – They are responsible and accountable for any crime that happens in their area.” Unfortunately, your lateral thinking took a left turn if you think police are responsible and accountable for crime. I agree police are responsible to maintain law and order within their abilities. It is the individual who is responsible and held accountable for commiting a crime. If police see a crime and do not act, then they are held accountable for failure to uphold the law. (Perhaps you can have Michael coach you on using analogies and rebuttals.)

My suggestions provide strategic thought on how to develop an effective regression test suite. The ideas may not be perfect, and they may not apply in all situations, but they do offer solid, specific advice based on applied practice. So, yes, I will continue to adovcate strategic thinking about how to constructively develop more effective regression testing strategies with some specific examples and suggestions. (Thank you for your permission to do so.) . In return, you should feel free to continue your “lateral thinking” and “feasibility angle” approach.

Tuesday, January 23, 2007 5:16 PM by I.M.Testy

My position is:

Regression testing objective when stated as “To validate that the application has not regressed to an error condition or non functioning State” becomes impractical and impossible to achieve.

I would like to restate the objective of Regression testing to

“Execute a set of Tests (Regression test) to validate that Application passes the set”

Main Key words here are “Regression suite” and “pass” – When I say this build has passed regression test – I mean it is passed the test cases in the suite.

>>>> Conversely, vague rhetorical statements such as “regression testing is nearly *IMPOSSIBLE* to achieve to its totality” or analyzing “to find out the feasibility of fulfilling such objective” are without meaningful substance.

Please note that I have never said “Regression Testing is nearly *IMPOSSIBLE* to achieve to its totality” – what all I said is – the way you have stated regression testing makes it impossible to achieve. I would say that Regression testing when defined as a “process of executing a set of test cases and checking if they pass” is POSSIBLE. Then the focus of the tester is to design and continusly to improve that Regression suite.

>>>Responsibility is the capacity of rational thought or action, the ability to discharge obligations, characterized by good judgment and sound thinking, involves accountability, and yes, is given to individuals who are reliable and dependable. Thus, responsibility is usually given to individuals who are accountable for something within their ability.

Your argument is similar to “Police are responsible to maintain Law and order situation – They are responsible and accountable for any crime that happens in their area”. There are certain things that are beyond good and able human intentions.

By setting up testers that “you shall be responsible for ensuring this ….” and giving them a target of confirming that current build has not regressed to a non-functional/error condition — a sure way of setting them for failure. It is something similar to “count the number of stars you see in the sky”. For the application of sound judgment, sound thinking with accountability – one needs a rational goal.

>>>Also, your statement that “if we knew complete product features and ALL error conditions – there would not be any bugs slipped out of testing” is simply foolish and erroneous.

Read carefully – my statement emphasizes the fact that it is NOT possible to know complete product feature and error conditions considering that software is complex and one can never claim that they know enough. If you believe in 100% Testing and Zero defect software product (your assertion that my comments are “foolish” – indicate that you are do believe in them)

- feel free and continue to preaching that “it is possible to perform regression testing in your way – validate that application has not regressed to a non-functioning/error condition.

I am not sure if I wrongly quoted some one my replies to this post. As far as Michael is concerned – I just showed him this whole post and checked if wrongly quoted him. He replied that my reply is correct.

In my attempt to “play with words” – I am not trying to hide any information or message or trying to obfuscate the problem with extraneous information. I am trying to expose a dimension of your argument. In other words – I am viewing your argument from “feasibility angle” (lateral thinking) and say that regression testing in way that you mention is not feasible”

Tuesday, January 23, 2007 6:58 AM by Shrini

The difference between you and I is that I take a position and attempt to put forth clear, unambiguous, and concise arguments to explain or defend my position. (My position may be incorrect in which case I will admit I am wrong or not completely aware of the facts and modify accordingly.) Conversely, vague rhetorical statements such as “regression testing is nearly *IMPOSSIBLE* to achieve to its totality” or analyzing “to find out the feasibility of fulfilling such objective” are without meaningful substance. It’s sort of like telling a 5 year old child to do something and they reply “No!”, and when you ask why they simply reply, “Because I said so!”

Responsibility is the capacity of rational thought or action, the ability to discharge obligations, characterized by good judgment and sound thinking, involves accountability, and yes, is given to individuals who are reliable and dependable. Thus, responsibility is usually given to individuals who are accountable for something within their ability.

That is why I stated, a “tester is responsible to understand the product’s attributes and capabilities, and thus should know what a non-functioning state or error condition is.” If they don’t have that ability, then I will not rely on them for rational thoughts regarding the product’s attributes or capabilities, or give them the responsibility for making sound judgments regarding the product.

As a test manager I was responsible and accountable to my managers to provide them with information that I could defend based on hard data and facts. To accomplish that, I relied on the talented people on my teams whom I trusted and could depend upon. If someone lacked the ability we attempted to train them. If a person was incapable of performing necessary obligations they were often reassigned roles that matched their abilities.

Also, your statement that “if we knew complete product features and ALL error conditions – there would not be any bugs slipped out of testing” is simply foolish and erroneous. (Also, if you are going to quote someone, quote them correctly.)

I have taught a variety of subjects to a variety of people for many years. It is truly a gift to be able to rephrase a statement, or use synonymous terminology, or even apply a metaphor to clearly communicate a complex idea or fact under a variety of circumstances. (For example, it is quite difficult for some people to fully understand JBS Haldane’s concept without understanding Boyle’s and Dalton’s laws, while some people simply accept the theory at face value.)

This is quite different than “playing with words” in which a person twists or manipulates the meaning of words to imply some alternate philosophy or definition because they are attempting to disguise their message. Yes people who play with words are masterful with the language also, but the intent of playing with words is not to inform or educate, it is simply to mislead or to obfuscate the the problem with extraneous information.

Saturday, January 20, 2007 1:34 PM by I.M.Testy