This is a good example. I have just one little nit. It is common for people to say that ET requires recording, because of the possibility of having trouble remembering what you did. Certainly I am enthusiastic about recording. I recommend recording if it does not interfere with the progress of testing.
But, here's the thing: so what if we have trouble reproducing what we did before? Isn't it better to experience a problem, even if only one time, than never to experience that problem at all? In most cases, I would say that answer is obviously yes. ET is about maximizing the power and coverage and value of our tests. This is a good thing, even if we have partial amnesia about what we did.
To maximize the integrity and reproducibility of a test, you will want to script it (whether for human or automated execution) or perfectly record it. But beware of the attitude that it is better to learn nothing than to learn something imperfectly. What that 8-year old kid provided was an indication of a risk. That indication helps us make decisions about where to put our future testing efforts, whether or not his specific bug is repro'd.
Tests do not simply result in bug finding. They also help create better, more knowledgeable testers, who are better adapted to the potential of finding important problems in the product. Don't lose sight of the whole equation.
1. One does not need to be a domain expert to find a bug or expose a risk.
2. Some times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.
3. Having a child like curiosity or some times creativity to be good at testing. As kids all of us grew with asking questions to our parents and making them say "you do not understand certain things. you need to grow up to understand them". Keeping child in you alive and active can make a big difference to testing
4. We might be ignoring horribly big set of problems with the application if we concentrate narrowly on scripted tests that supposed to 100% mapping to requirements.
From your observations:
>>>This story vividly demonstrates two points about exploratory testing. First is that you have to be very careful to record everything you do during a testing session
Like James, I am also of the opinion is that "recording" is an important aspect of ET. Though recording can be a powerful aid is not an essential ingredient of good ET. The more emphasis you pay on recording, making the process repeatable - more you are tending towards "scripted" approach testing. With a focus on recording, a tester performing ET might get distracted from the core goal of exploring.
Having a non-intrusive tool - like Test explorer would be a good thing in ET but mandating that recording is must - would amount to overstating the value of recording.
>>> Approach your testing with an awareness of your assumptions, and try to find out what happens when you break them.
I think this is a great point. Thanks for mentioning this. Listing out assumptions while testing and constantly keep checking them for "what if this assumption is false" - can be a great Test idea generator....
Let us say someone is testing Google search feature - it will really help if he/she puts down the assumptions that are being made and design tests to expose the possible issues when each of those assumptions become false ...
Shrini Kulkarni Principal Consultant iGATE Global solutions Bangalore, India
Some times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.
Novices can often be great testers.
I once worked for an organization that purposely hired college students in the summer that had no domain knowledge of what was being tested. Why?
When performing scripted testing: The novices would follow the scripts instead of doing it the way they thought it should be done. (As James has often pointed out: this is only to the degree of the detail of the scripts.) Also, the novices were more likely to ask questions about how to execute a test than people that thought they knew how the system should operate.
When performing testing that was not completely scripted: The novices were more likely to act like the real-world novice users of the systems under test.
The questions asked by the novices also gave the engineers that oversaw the tests useful information about how real users think.
Testers that mimic real-world users can be very valuable.
Many years ago a friend of mine was working in the video games industry. He kept getting recurring bugs that would crash the game, but would sometimes only occur for certain testers and then only after some hours of game play.
When asked what they had done just prior to the crash, the testers would often be unable to remember clearly.
His solution was to take a trip to the local junk shop where he bought a number of old beta-max video recorders. When the testers were able to simply record on tape everything they did, they were better placed to determine the steps to reproduce the failure.
I have though of this many times in the years since, when I get a bug report from a tester that I cannot for the life of me reproduce. Every now and again I suggest using video recorders in project meetings. It usually raises a smile, but never as yet raised any action. Ho Hum.
Hi Allen,
ReplyDeleteThis is a good example. I have just one little nit. It is common for people to say that ET requires recording, because of the possibility of having trouble remembering what you did. Certainly I am enthusiastic about recording. I recommend recording if it does not interfere with the progress of testing.
But, here's the thing: so what if we have trouble reproducing what we did before? Isn't it better to experience a problem, even if only one time, than never to experience that problem at all? In most cases, I would say that answer is obviously yes. ET is about maximizing the power and coverage and value of our tests. This is a good thing, even if we have partial amnesia about what we did.
To maximize the integrity and reproducibility of a test, you will want to script it (whether for human or automated execution) or perfectly record it. But beware of the attitude that it is better to learn nothing than to learn something imperfectly. What that 8-year old kid provided was an indication of a risk. That indication helps us make decisions about where to put our future testing efforts, whether or not his specific bug is repro'd.
Tests do not simply result in bug finding. They also help create better, more knowledgeable testers, who are better adapted to the potential of finding important problems in the product. Don't lose sight of the whole equation.
-- James
Allen,
ReplyDeleteFrom the story -
1. One does not need to be a domain expert to find a bug or expose a risk.
2. Some times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.
3. Having a child like curiosity or some times creativity to be good at testing. As kids all of us grew with asking questions to our parents and making them say "you do not understand certain things. you need to grow up to understand them". Keeping child in you alive and active can make a big difference to testing
4. We might be ignoring horribly big set of problems with the application if we concentrate narrowly on scripted tests that supposed to 100% mapping to requirements.
From your observations:
>>>This story vividly demonstrates two points about exploratory testing. First is that you have to be very careful to record everything you do during a testing session
Like James, I am also of the opinion is that "recording" is an important aspect of ET. Though recording can be a powerful aid is not an essential ingredient of good ET. The more emphasis you pay on recording, making the process repeatable - more you are tending towards "scripted" approach testing. With a focus on recording, a tester performing ET might get distracted from the core goal of exploring.
Having a non-intrusive tool - like Test explorer would be a good thing in ET but mandating that recording is must - would amount to overstating the value of recording.
>>> Approach your testing with an awareness of your assumptions, and try to find out what happens when you break them.
I think this is a great point. Thanks for mentioning this. Listing out assumptions while testing and constantly keep checking them for "what if this assumption is false" - can be a great Test idea generator....
Let us say someone is testing Google search feature - it will really help if he/she puts down the assumptions that are being made and design tests to expose the possible issues when each of those assumptions become false ...
Shrini Kulkarni
Principal Consultant
iGATE Global solutions
Bangalore, India
http://shrinik.blogspot.com
Shrini said...
ReplyDeleteSome times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.
Novices can often be great testers.
I once worked for an organization that purposely hired college students in the summer that had no domain knowledge of what was being tested. Why?
When performing scripted testing: The novices would follow the scripts instead of doing it the way they thought it should be done. (As James has often pointed out: this is only to the degree of the detail of the scripts.) Also, the novices were more likely to ask questions about how to execute a test than people that thought they knew how the system should operate.
When performing testing that was not completely scripted: The novices were more likely to act like the real-world novice users of the systems under test.
The questions asked by the novices also gave the engineers that oversaw the tests useful information about how real users think.
Testers that mimic real-world users can be very valuable.
Ben Simo
QualityFrog.com
Many years ago a friend of mine was working in the video games industry. He kept getting recurring bugs that would crash the game, but would sometimes only occur for certain testers and then only after some hours of game play.
ReplyDeleteWhen asked what they had done just prior to the crash, the testers would often be unable to remember clearly.
His solution was to take a trip to the local junk shop where he bought a number of old beta-max video recorders. When the testers were able to simply record on tape everything they did, they were better placed to determine the steps to reproduce the failure.
I have though of this many times in the years since, when I get a bug report from a tester that I cannot for the life of me reproduce. Every now and again I suggest using video recorders in project meetings. It usually raises a smile, but never as yet raised any action. Ho Hum.
Great, great, great things to learn. Awsome blog.
ReplyDeleteQTP