Testing Blog

8-year-old exploratory testing

Wednesday, May 02, 2007
Share on Twitter Share on Facebook
Google
Labels: Allen Hutchison

5 comments :

  1. James Marcus BachMay 2, 2007 at 1:10:00 PM PDT

    Hi Allen,

    This is a good example. I have just one little nit. It is common for people to say that ET requires recording, because of the possibility of having trouble remembering what you did. Certainly I am enthusiastic about recording. I recommend recording if it does not interfere with the progress of testing.

    But, here's the thing: so what if we have trouble reproducing what we did before? Isn't it better to experience a problem, even if only one time, than never to experience that problem at all? In most cases, I would say that answer is obviously yes. ET is about maximizing the power and coverage and value of our tests. This is a good thing, even if we have partial amnesia about what we did.

    To maximize the integrity and reproducibility of a test, you will want to script it (whether for human or automated execution) or perfectly record it. But beware of the attitude that it is better to learn nothing than to learn something imperfectly. What that 8-year old kid provided was an indication of a risk. That indication helps us make decisions about where to put our future testing efforts, whether or not his specific bug is repro'd.

    Tests do not simply result in bug finding. They also help create better, more knowledgeable testers, who are better adapted to the potential of finding important problems in the product. Don't lose sight of the whole equation.

    -- James

    ReplyDelete
    Replies
      Reply
  2. Shrini KulkarniMay 3, 2007 at 11:12:00 AM PDT

    Allen,

    From the story -

    1. One does not need to be a domain expert to find a bug or expose a risk.

    2. Some times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.

    3. Having a child like curiosity or some times creativity to be good at testing. As kids all of us grew with asking questions to our parents and making them say "you do not understand certain things. you need to grow up to understand them". Keeping child in you alive and active can make a big difference to testing

    4. We might be ignoring horribly big set of problems with the application if we concentrate narrowly on scripted tests that supposed to 100% mapping to requirements.

    From your observations:

    >>>This story vividly demonstrates two points about exploratory testing. First is that you have to be very careful to record everything you do during a testing session

    Like James, I am also of the opinion is that "recording" is an important aspect of ET. Though recording can be a powerful aid is not an essential ingredient of good ET. The more emphasis you pay on recording, making the process repeatable - more you are tending towards "scripted" approach testing. With a focus on recording, a tester performing ET might get distracted from the core goal of exploring.

    Having a non-intrusive tool - like Test explorer would be a good thing in ET but mandating that recording is must - would amount to overstating the value of recording.

    >>> Approach your testing with an awareness of your assumptions, and try to find out what happens when you break them.

    I think this is a great point. Thanks for mentioning this. Listing out assumptions while testing and constantly keep checking them for "what if this assumption is false" - can be a great Test idea generator....

    Let us say someone is testing Google search feature - it will really help if he/she puts down the assumptions that are being made and design tests to expose the possible issues when each of those assumptions become false ...

    Shrini Kulkarni
    Principal Consultant
    iGATE Global solutions
    Bangalore, India

    http://shrinik.blogspot.com

    ReplyDelete
    Replies
      Reply
  3. Ben SimoMay 3, 2007 at 5:32:00 PM PDT

    Shrini said...

    Some times being an expert or some who knows the stuff very well - might work against you as you tend to take things for granted and do not question certain things as a novice would.

    Novices can often be great testers.

    I once worked for an organization that purposely hired college students in the summer that had no domain knowledge of what was being tested. Why?

    When performing scripted testing: The novices would follow the scripts instead of doing it the way they thought it should be done. (As James has often pointed out: this is only to the degree of the detail of the scripts.) Also, the novices were more likely to ask questions about how to execute a test than people that thought they knew how the system should operate.

    When performing testing that was not completely scripted: The novices were more likely to act like the real-world novice users of the systems under test.

    The questions asked by the novices also gave the engineers that oversaw the tests useful information about how real users think.

    Testers that mimic real-world users can be very valuable.

    Ben Simo
    QualityFrog.com

    ReplyDelete
    Replies
      Reply
  4. AnonymousMay 8, 2007 at 6:19:00 AM PDT

    Many years ago a friend of mine was working in the video games industry. He kept getting recurring bugs that would crash the game, but would sometimes only occur for certain testers and then only after some hours of game play.

    When asked what they had done just prior to the crash, the testers would often be unable to remember clearly.

    His solution was to take a trip to the local junk shop where he bought a number of old beta-max video recorders. When the testers were able to simply record on tape everything they did, they were better placed to determine the steps to reproduce the failure.

    I have though of this many times in the years since, when I get a bug report from a tester that I cannot for the life of me reproduce. Every now and again I suggest using video recorders in project meetings. It usually raises a smile, but never as yet raised any action. Ho Hum.

    ReplyDelete
    Replies
      Reply
  5. Sachin DhallJuly 26, 2007 at 3:44:00 AM PDT

    Great, great, great things to learn. Awsome blog.

    QTP

    ReplyDelete
    Replies
      Reply
Add comment
Load more...

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

  

Labels


  • TotT 104
  • GTAC 61
  • James Whittaker 42
  • Misko Hevery 32
  • Code Health 31
  • Anthony Vallone 27
  • Patrick Copeland 23
  • Jobs 18
  • Andrew Trenk 13
  • C++ 11
  • Patrik Höglund 8
  • JavaScript 7
  • Allen Hutchison 6
  • George Pirocanac 6
  • Zhanyong Wan 6
  • Harry Robinson 5
  • Java 5
  • Julian Harty 5
  • Adam Bender 4
  • Alberto Savoia 4
  • Ben Yu 4
  • Erik Kuefler 4
  • Philip Zembrod 4
  • Shyam Seshadri 4
  • Chrome 3
  • Dillon Bly 3
  • John Thomas 3
  • Lesley Katzen 3
  • Marc Kaplan 3
  • Markus Clermont 3
  • Max Kanat-Alexander 3
  • Sonal Shah 3
  • APIs 2
  • Abhishek Arya 2
  • Alan Myrvold 2
  • Alek Icev 2
  • Android 2
  • April Fools 2
  • Chaitali Narla 2
  • Chris Lewis 2
  • Chrome OS 2
  • Diego Salas 2
  • Dori Reuveni 2
  • Jason Arbon 2
  • Jochen Wuttke 2
  • Kostya Serebryany 2
  • Marc Eaddy 2
  • Marko Ivanković 2
  • Mobile 2
  • Oliver Chang 2
  • Simon Stewart 2
  • Stefan Kennedy 2
  • Test Flakiness 2
  • Titus Winters 2
  • Tony Voellm 2
  • WebRTC 2
  • Yiming Sun 2
  • Yvette Nameth 2
  • Zuri Kemp 2
  • Aaron Jacobs 1
  • Adam Porter 1
  • Adam Raider 1
  • Adel Saoud 1
  • Alan Faulkner 1
  • Alex Eagle 1
  • Amy Fu 1
  • Anantha Keesara 1
  • Antoine Picard 1
  • App Engine 1
  • Ari Shamash 1
  • Arif Sukoco 1
  • Benjamin Pick 1
  • Bob Nystrom 1
  • Bruce Leban 1
  • Carlos Arguelles 1
  • Carlos Israel Ortiz García 1
  • Cathal Weakliam 1
  • Christopher Semturs 1
  • Clay Murphy 1
  • Dagang Wei 1
  • Dan Maksimovich 1
  • Dan Shi 1
  • Dan Willemsen 1
  • Dave Chen 1
  • Dave Gladfelter 1
  • David Bendory 1
  • David Mandelberg 1
  • Derek Snyder 1
  • Diego Cavalcanti 1
  • Dmitry Vyukov 1
  • Eduardo Bravo Ortiz 1
  • Ekaterina Kamenskaya 1
  • Elliott Karpilovsky 1
  • Elliotte Rusty Harold 1
  • Espresso 1
  • Felipe Sodré 1
  • Francois Aube 1
  • Gene Volovich 1
  • Google+ 1
  • Goran Petrovic 1
  • Goranka Bjedov 1
  • Hank Duan 1
  • Havard Rast Blok 1
  • Hongfei Ding 1
  • Jason Elbaum 1
  • Jason Huggins 1
  • Jay Han 1
  • Jeff Hoy 1
  • Jeff Listfield 1
  • Jessica Tomechak 1
  • Jim Reardon 1
  • Joe Allan Muharsky 1
  • Joel Hynoski 1
  • John Micco 1
  • John Penix 1
  • Jonathan Rockway 1
  • Jonathan Velasquez 1
  • Josh Armour 1
  • Julie Ralph 1
  • Kai Kent 1
  • Kanu Tewary 1
  • Karin Lundberg 1
  • Kaue Silveira 1
  • Kevin Bourrillion 1
  • Kevin Graney 1
  • Kirkland 1
  • Kurt Alfred Kluever 1
  • Manjusha Parvathaneni 1
  • Marek Kiszkis 1
  • Marius Latinis 1
  • Mark Ivey 1
  • Mark Manley 1
  • Mark Striebeck 1
  • Matt Lowrie 1
  • Meredith Whittaker 1
  • Michael Bachman 1
  • Michael Klepikov 1
  • Mike Aizatsky 1
  • Mike Wacker 1
  • Mona El Mahdy 1
  • Noel Yap 1
  • Palak Bansal 1
  • Patricia Legaspi 1
  • Per Jacobsson 1
  • Peter Arrenbrecht 1
  • Peter Spragins 1
  • Phil Norman 1
  • Phil Rollet 1
  • Pooja Gupta 1
  • Project Showcase 1
  • Radoslav Vasilev 1
  • Rajat Dewan 1
  • Rajat Jain 1
  • Rich Martin 1
  • Richard Bustamante 1
  • Roshan Sembacuttiaratchy 1
  • Ruslan Khamitov 1
  • Sam Lee 1
  • Sean Jordan 1
  • Sebastian Dörner 1
  • Sharon Zhou 1
  • Shiva Garg 1
  • Siddartha Janga 1
  • Simran Basi 1
  • Stan Chan 1
  • Stephen Ng 1
  • Tejas Shah 1
  • Test Analytics 1
  • Test Engineer 1
  • Tim Lyakhovetskiy 1
  • Tom O'Neill 1
  • Vojta Jína 1
  • automation 1
  • dead code 1
  • iOS 1
  • mutation testing 1


Archive


  • ►  2025 (1)
    • ►  Jan (1)
  • ►  2024 (13)
    • ►  Dec (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (1)
    • ►  May (3)
    • ►  Apr (3)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2023 (14)
    • ►  Dec (2)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (3)
    • ►  Aug (1)
    • ►  Apr (1)
  • ►  2022 (2)
    • ►  Feb (2)
  • ►  2021 (3)
    • ►  Jun (1)
    • ►  Apr (1)
    • ►  Mar (1)
  • ►  2020 (8)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  May (1)
  • ►  2019 (4)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Jul (1)
    • ►  Jan (1)
  • ►  2018 (7)
    • ►  Nov (1)
    • ►  Sep (1)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (1)
    • ►  Feb (1)
  • ►  2017 (17)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2016 (15)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (1)
    • ►  Sep (2)
    • ►  Aug (1)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (1)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2015 (14)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Aug (1)
    • ►  Jun (1)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (1)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2014 (24)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Sep (2)
    • ►  Aug (2)
    • ►  Jul (3)
    • ►  Jun (3)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2013 (16)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Jan (2)
  • ►  2012 (11)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (1)
    • ►  Aug (4)
  • ►  2011 (39)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (2)
    • ►  Aug (4)
    • ►  Jul (2)
    • ►  Jun (5)
    • ►  May (4)
    • ►  Apr (3)
    • ►  Mar (4)
    • ►  Feb (5)
    • ►  Jan (3)
  • ►  2010 (37)
    • ►  Dec (3)
    • ►  Nov (3)
    • ►  Oct (4)
    • ►  Sep (8)
    • ►  Aug (3)
    • ►  Jul (3)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (3)
    • ►  Mar (3)
    • ►  Feb (2)
    • ►  Jan (1)
  • ►  2009 (54)
    • ►  Dec (3)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (5)
    • ►  Aug (4)
    • ►  Jul (15)
    • ►  Jun (8)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (5)
    • ►  Jan (4)
  • ►  2008 (75)
    • ►  Dec (6)
    • ►  Nov (8)
    • ►  Oct (9)
    • ►  Sep (8)
    • ►  Aug (9)
    • ►  Jul (9)
    • ►  Jun (6)
    • ►  May (6)
    • ►  Apr (4)
    • ►  Mar (4)
    • ►  Feb (4)
    • ►  Jan (2)
  • ▼  2007 (41)
    • ►  Oct (6)
    • ►  Sep (5)
    • ►  Aug (3)
    • ►  Jul (2)
    • ►  Jun (2)
    • ▼  May (2)
      • Sign up to attend our Test Automation Conference
      • 8-year-old exploratory testing
    • ►  Apr (7)
    • ►  Mar (5)
    • ►  Feb (5)
    • ►  Jan (4)

Feed

  • Google
  • Privacy
  • Terms