Friday, March 07, 2008 1:03 PM by I.M.Testy

Wednesday, March 05, 2008 4:44 AM by puretesting

Thank you for bringing the conversation back to my original point about taking things completely out of context.

I do understand the difference between design and execution, and I only spoke about designing tests and I did not even refer to or infer execution of tests in my response. So, your arguments about design vs. execution are tangential.

Also, my statement was “Unless of course, you do not consider…” not simply “you do not consider…”; which changes the context of my statement.

Wednesday, February 27, 2008 12:57 AM by I.M.Testy

Doing demos (or showing) is not testing (exploratory or otherwise). You could be demo’ing a software to your family, or business group or to a customer.

Before one could demo a peice of software, it has to be configured and operated (dry run) or rehearsal. That part of demo cycle where you prepare the software, setup the data and scenarios, do a dry run (few scenarios) is what I am referring as “Non explopratory”.

Depending upon the length of the demo, it could involve a great amount of scripted testing to make sure that specified scripts “pass” and demo will be successful.

>>> but see the point below because I think tester’s ‘explore’ the requirement to understand what to audit or how to audit something?

That portion of actual execution of the audit or regulatory confirmation that involves execution of specific scripts (created via some sort of exploration) is “Non exploratory”.

You know *exactly* what to do (execute) and what exactly to expect (results). I believe most of the audits and compliance tests happen as per pre-defined scripts – no room for exploration.

>>>I guess we could say that even scripted tests are exploratory because they are derived from a ‘sapient’ person ‘exploring’ the requirements and designing tests!

Right. But I would make a distiction between “design” and “execution”. Design of scripted tests is often “exploratory” (some engage in mechanial ways of “deriving” test cases from specification)

Execution of scripted is Non exploratory. You will do exactly as mentioned in script and *observe* only what test tells you to observe. Nothing more or nothing less (deviating from script is considered as indescipline according to QA/Process department)

>>> you do not consider that a person who applies critical thinking and use of their cognitive skills to analyze (since you state analysis is highly exploratory) the requirements and use sound judgement (sapience) to design a set of both positive and negative tests (scripted tests) from the requirements to be exploratory testing.

Did I say that …? No. Design scripted tests is exploration. Execution of scripted tests with sole intention of checking if those tests pass is not.

If you seperate the creation and execution activities, you might see the difference. IT is important to note that often, since the person who designed tests is not the one who will be executing them – line of seperation between exploratory and scripted testing is clear.

Shrini

Wednesday, February 27, 2008 12:14 AM by Shrini

Thanks for reiterating the demo point Michael made above. I am not sure why staged demos would be considered testing in anyone’s mind; perhaps that is why they call them demos instead of testing?

The audit/compliance makes perfect sense, and I agree, but see the point below because I think tester’s ‘explore’ the requirement to understand what to audit or how to audit something?

I find it quite interesting that you only list problems, issues and bugs as the “kind of new information that would come out of a sapient human testing.” A professional tester who is truly wise and uses sound judgement would of course realize that defects, problems and issues are only a fraction of the valuable information testing should provide to the organization to reduce exposure to risk.

So based on the above, I would suspect that you would consider structural testing mythical and ellusive, especially if the only information one can provide is bug count, time, and ‘feel-good’ observations.

In essence (according to your definition), everything which provides new information is exploratory. I guess we could say that even scripted tests are exploratory because they are derived from a ‘sapient’ person ‘exploring’ the requirements and designing tests!

Unless of course, you do not consider that a person who applies critical thinking and use of their cognitive skills to analyze (since you state analysis is highly exploratory) the requirements and use sound judgement (sapience) to design a set of both positive and negative tests (scripted tests) from the requirements to be exploratory testing.

Tuesday, February 26, 2008 12:18 PM by I.M.Testy

Oh… and since Shrini has apparently stopped playing, I’ll give you three instances in which an exploratory approach might not be appropriate.

I am back … (was on vacation) … BJ, Here are three contexts that *I* think ET would be inappropriate.

1. Demos – In demos, where one wants demonstrate a software application, tried and tested paths are preffered – scripted tests are good. You would not want to explore when you are demonstrating …

2. Audits/Compliance Testing – Where the mission of testing is to show confirmance to some regulatory standard (pretty clearly known in advance). Exploration is not appropriate when you are auditing or checking if a confirmance test passes. Exploration in that case would mean “doing something” else other than the confirmance.

3. Any other forms or types of testing when you do not look for any information other than the one provided by scripted tests.

So, you have heard Michael Bolton, me … Please please make a note … we, in context driven school do not *sell* exploratory testing as “holy grail”.

Also I would not agree with the cases you mentioned where ET is inappropriate ….

>>> – efficient structural testing

I am not sure if “efficient structural testing” exists in this world. It is mythical and ellusive. Irrespective of state of structural testing – ET would always be appropriate – if you are looking for problems, issues and bugs (any kind of new information that could come out of a sapient human testing)

>>>>- perfoming an analysis of untested code and either designing tests to exercise those areas or understanding why those tests may not be cost effective

Analysys, understanding – are highly exploratory human process. You can not have a static script for these things. These are exploratory by definition ….

>>>- performance, stress, and load testing

These too, involve great deal of exploratory elements – especially performance tuning work

>>> – concurrency testing

Setting up concurrency conditions, observing results, learning from the results and re-doing things – see exploration at every point. A scripted concurrency testing would be pretty narrow and limited utility as one can not (even a professional tester) can not predict some interesting outcomes and develop a good scripted concurrency test …

Shrini

Tuesday, February 26, 2008 7:51 AM by Shrini

In my experience, the companies I have worked for review my past accomplishments and then hire me for my demonstrable knowledge and skills, my passion for personal growth, and my potential to benefit the organization.

I think we (myself and readers of this thread) understand that you do not hold certifications in high regard. I am not sure why or what you are trying to convince me of here. I am neutral on the subject. I understand the value some businesses perceive, and I also understand the limitations.

Regarding the phrase professional tester, I suppose if someone were ignorant of the term professional, then I suppose they could misinterpret it anyway they saw fit. But, as I have said previously, I consider a professional anyone “who is expert at his or her work” or “having or showing great skill.”

Notice there is no mention of some sacred doctrine to adhere to, nor do I state anything regarding a philosopy the way Shrini and others talk about “the philosophy of context driven thinking.” There is no special club, and there is no special forum where moderators remove people who disagree with some pre-established principles. No secret hand-shakes, and no us versus them mentality.

I refer to professionalism as merely the state of mind of any person who strives to constantly improve his or her in-depth knowledge and broaden the diverse set of skills required to be considered an expert in a myriad of tasks used in our discipline to provide the best possible service to our organization.

Thanks for your reasons where exploratory would not be appropriate.

Mandate seems rather narrow minded of management. Personally, I have never seen this happen, but I guess it could in a shop where the only thing the organization wanted was to perform strict verification tests.

I would think that even in closed systems a professional tester should think critically about the system and question the model from different perspectives, perhaps analyze parameter interaction and not only perform combinatorial analysis of semi-coupled parameters from t=2 through t=6, but also randomize the combinatorial output for breadth of coverage.

I agree with benchmarking, except I would say the the desired results are consistent with previous iterations of the same test when ran against a new build, because of course each new build could introduce new defects even in areas of the product that have been previously tested; that’s why we refer to them regression tests.

I am not sure I follow your example for avoidance. I call that setting up a demo and most presenters do this because they don’t want the demo to fail unexpectedly; it wastes people’s time.

But, if a tester actively avoids testing or learning in the commission of their job then I wouldn’t consider that person a professional tester, would you?

Personally, I can think of other areas where a tester might consider exploratory testing inappropriate such as:

- efficient structural testing

- perfoming an analysis of untested code and either designing tests to exercise those areas or understanding why those tests may not be cost effective

- performance, stress, and load testing

- concurrency testing

On the other hand, I would suggest that debugging is highly exploratory in nature with logical deduction sprinkled in.

I am glad that you and James teach reasons to repeat tests in your course. BVT/BAT testing, regression testing, compliance testing, and other repeatable test suites are important for many reasons. See, common ground!

Thursday, February 21, 2008 3:34 AM by I.M.Testy

Quite true–but it does in others, and that’s why it’s so useful to me, both as a positive and negative test of my potential relationship with a potential employer. I like to belong to clubs that would have me as a member, and I’m not interested in clubs that don’t want me. If someone is impressed by my actions, my writing, my speech, my experiences, and so on, then we’re more likely to get along. If I find that someone isn’t interested in my own credentials, but is instead interested in the fact that I’ve taken a less-than-trivial certification exam, that tells us both very useful information: neither of us is likely to be interested in the other. That’s cool.

Respect from others is also not a form of certification because some people may not necessarily respect the same people as you. So, with regards to an industry recognized certification, self selection or respect from peers simply doesn’t cut it.

I think you might mean “in comparison to” rather than “with regards to”. Certification is simply the granting of respect to an individual by a group–largely anonymous, from the perspective of most hiring managers–that has appointed itself worthy of granting respect. Some people buy into that; others don’t. Some organizations that I’m aware of actively reject “certified” testers. I think that’s extreme, but I can understand why they might make that choice: trivial certification is inconsistent with their values.

Again, if someone doesn’t respect the same people that I do, that gives me an important hint. If we engage, maybe one or both of us will learn something, or maybe we’ll simply not engage and avoid wasting each other’s time.

I personally disdain all types of needless segregationist type labels.

Hmmm. You frequently refer to the “professional tester”. Is that a label that someone could interpret as segregationist?

Oh… and since Shrini has apparently stopped playing, I’ll give you three instances in which an exploratory approach might not be appropriate.

1) Mandate. Client mandates against it and issues an edict. “You shall not perform exploratory tests; you shall perform these tests and no others.”

2) Sufficiency. A very deterministic, already well-tested, and essentially closed system where inputs and outputs are highly constrained and the test space has been rigidly modelled; scripted tests have been deemed sufficient to reveal the information we need. Components in embedded systems might provide a lot of contextual examples here.

3) Benchmarking. When it’s important for the results of the test to be entirely consistent with previous iterations of the same test, exploration is the wrong thing to do.

Here’s another one for free:

4) Avoidance: if we want to actively avoid testing or learning new information, avoiding exploration is a great way to do it. For example, when a tester receives a request from the marketing department, “Please give us a suite of tests that have been run before lots of times and that don’t crash, so that we can give them to Bill for tomorrow’s trade show.”

James Bach and I teach at least six more reasons to repeat tests, rather than to explore, as an exercise in our Rapid Software Testing course. Text associated with this exercise is published on James’ site: Reasons to Repeat Tests (http://www.satisfice.com/repeatable.shtml)

—Michael B.

Wednesday, February 20, 2008 11:44 PM by Michael A Bolton

In my opinion, self certification simply does not hold credible value in many businesses and some social circles. Respect from others is also not a form of certification because some people may not necessarily respect the same people as you. So, with regards to an industry recognized certification, self selection or respect from peers simply doesn’t cut it.

The Oxford dictionary is an interesting example, but I think all professions rely upon commonly understood jargon. Mutate the jargon used to convey consistent meaning within an established profession and effective communication breaks down.

I agree with some of your other examples, and apologize for not highlighting some of them. I agree tt would be nice if exams not only included ticking checkboxes on a piece of paper, but also required individuals to sit down and analyze the application under test to discover what was tested and what wasn’t tested, and understand why it wasn’t tested.(Finding bugs is only one component of testing, but it really doesn’t tell me anything about how well something is tested.)

With respect to my comments about the context-driven school, I have read much about the ideology, and I actually agree with what is written; who couldn’t agree with it, it is really mostly common sense. Personally, I am sure you know that I don’t think it really makes much sense to segregate testing into ‘schools’ and I think my assessment is correct in that there are those who self-identify with one particular school and then there is everyone else. But, if some people want to self segregate that is fine. Unfortunately, it seems to me that the perception of some people who like to also identify with the context-driven ideology is different than what is stated.

At the end of the day, I really don’t suspect that you and I disagree as much as we agree. For example, I suspect we both agree that code coverage by itself is a poor (and misleading) measure (similarly bug count by itself is a poor measure of anything other than we found another bug), and test coverage is also important and we are discovering ways to evaluate and measure effective test coverage.

Our focuses may be different, but we both strive to enrich the skills and knowledge of testers, and I suspect to our respective audiences we are both considered successful; otherwise I doubt we would continually be invited back to conferences.

I don’t consider myself an empiricist (I personally disdain all types of needless segregationist type labels), but yes, being a man, and seeing other men without shirts on, I did happen to notice that the vast majority of human men have nipples (and I have not seen or heard of an instance in which a human man (homo-sapiens) was born without nipples). There is actually only one reason why nipples exist on men. The reason why men have nipples is because it is encoded in our genetic autosomes.

Now, there might be various reasons why men derive pleasure (or not) from the fact they have nipples (which I am not at all interested in discussing), but there is really only one scientifically established reason for the existence of nipples in men (and women) of the homo-sapiens species…and the answer is genetic encoding.

Wednesday, February 20, 2008 9:22 PM by I.M.Testy

I’d like to thank you for the kind words, but I’d also like to suggest that there may have been more to the talk than “simply denigrating the existing programs without offering any other solutions”. You can review the notes for the talk at http://www.developsense.com/presentations/notyetcertified.pdf.

On pages 3 and 4, I give examples of ways in which I certify myself; self-certification is form of certification. On pages 5 and 6, I note personal endorsement as another way of thinking about of certification. On pages 14, 15, and 16, I offer several suggestions on elements of a certification program that I could believe in, with an example question or answer for each.

On page 17, I suggest that a body of knowledge for certification schemes should be built upon a descriptive, Oxford English Dictionary model (and in the talk, I spoke of the process by which this was done; refer to The Meaning of Everything by Simon Winchester for a very accessible history of the project). I suggested that foundation-level certifications based on multiple choice tests should be free, to remove the perception of the conflict of interest between those who provide training and those who provide certification. I further suggest on page 19 that we could easily assess a bevy of additional skills at the foundation level. On page 19, I have specific proposals for skills to be examined at the advanced level, and I allude to an existing model (Cisco’s) that takes that kind of approach.

With respect to your comments on context-driven testing, I think that if you looked, you would find much support in the community for the things that you claim that we denigrate. We’re not opposed to documentation, though I think you’ll find that most of us are opposed to wasteful documentation. We’re not opposed to automation (especially in the broader sense of automation as “any use of tools to support testing”); we more likely to be opposed to the inefficient, wasteful, and gratuitous use of automation. We’re not single-mindedly focused on testing the GUI using hands as the input mechanism, but we often try to emphasize its value and the risk associated with not using human testing. We’re not opposed to coverage analysis; far from it–we’re in favour of far more expansive notions of test coverage than that afforded by mere code coverage. I’m not opposed to strengthening of certifications if those certifications are relevant and meaningful, but in the models being delivered by QAI and the ISTQB, I don’t think they are. Still, see the references to my talk above.

One more thing: as a good empiricist, you’ll notice that men have nipples, apparently useless or not. Can you think of at least three reasons why nipples on men might exist?

Cheers,

—Michael B.

Wednesday, February 20, 2008 5:41 PM by Michael A Bolton