Testing Blog

Burning Test Questions at Google

Tuesday, June 09, 2009
Share on Twitter Share on Facebook
Google
Labels: James Whittaker

15 comments :

  1. noneotherJune 9, 2009 at 8:13:00 PM PDT

    The key to this is ROI and metrics. Currently it is fashion to show efficiencies in execution and regression, based on volume of execution. The start of the debate should focus on risk mitigation, versus volume of execution, or regression.

    This is most common, in my opinion, with outsourced testing efforts, as the "shock and awe" effect of large numbers of Test Execution leads the customer (often an IT manager with little to no test acumen) to retaining and extending the service provider. Because it is believed that the volume of tests executed are directly proportional to mitigation of risk.

    Looking forward to following to following the blog. Best of luck.

    ReplyDelete
    Replies
      Reply
  2. James MartinJune 10, 2009 at 1:00:00 AM PDT

    This is not just about ROI or metrics (*shudder*), although those things will be in the minds of those paying the test team's wages. For me, discussions of this nature always overlook the fact that a good manual test cannot be automated.

    In order to fully appreciate that last statement I suggest reading James Bach's blog post on the subject, where he points out that "[t]est automation cannot reproduce the thinking that testers do when they conceive of tests, control tests, modify tests, and observe and evaluate the product."

    Automated testing removes (to a greater or lesser degree) the value that a trained human being adds to a test.

    The trick to balancing manual with automated testing is to allow humans to do what they do best: feel their way through software, make decisions based on hunches and patterns that a machine cannot be programmed to detect.

    Leave the repetitious checking to the machines by all means, and let good testers do what they do best: ask questions of the software that reveals information pertaining to its value.

    ReplyDelete
    Replies
      Reply
  3. Justin HunterJune 10, 2009 at 3:06:00 AM PDT

    James,

    Great post.

    You said "I think the issue is test design versus doing something because we can." I absolutely agree. Effective test design tools and methods are not given nearly enough attention by most testers, regardless of which side of the "manual v. automate" debate they support. Test design methods shoud be a part of every testers training and test design tools should be a part of every serious tester's tool box. Despite this, few testers, even in Fortune 100 firms that have otherwise sophisticated IT departments, are currently using test design tools and well-structured methods. Worse still, many testers have not even heard of them; as a result, testers generate and execute highly ineffective test cases week after week.

    I recently worked on a study that spanned 10 projects. In each project, we objectively compared the defects found by "2 testers in a room with a test design tool" to "4 testers in a room with no test design tool". Both teams tested the same "real world" application at the same time. I am looking forward to publishing the empirical findings (which is scheduled to happen in October).

    The results from those pilot projects were positive enough (consistently doubled tester productivity, consistently higher quality, etc.) that I decided to create my own test design software tool and help testing teams achieve more coverage in fewer tests. (Please see: http://www.hexawise.com).

    Thank you for emphasizing that identifying the right things to test is a critically important factor that too often gets overlooked in the manual testing v. automated testing debate.

    - Justin Hunter
    Founder and CEO of Hexawise

    ReplyDelete
    Replies
      Reply
  4. CalkelpdiverJune 10, 2009 at 8:00:00 AM PDT

    Jim,

    I'd say you nailed it pretty much on the head. It is like building a house. You need to look at what you are trying to build (test), design it, build it, 'test' it, and then use it.

    Manual testing does that, and when done you can (despite what some people say) use that as a blueprint for your automated test. I do agree that not all manual tests should be automated or can be (due to various reasons). Automation is a tool and not an end-all solution.

    And to agree with James Bach, there is not better tool than your own brain.

    Have fun at Google and look forward to more posts.

    Jim Hazen

    ReplyDelete
    Replies
      Reply
  5. James WhittakerJune 10, 2009 at 1:32:00 PM PDT

    So Ian thought it was funny and Wendy thought it was offensive. I intended the former and not that latter. But I suppose it will stick in the mind either way. Wendy, you have my apology!

    Now to the meat: thinking is the key issue but thinking is note the sole domain of the manual tester. Lots of thought should go into test design. If then those tests get executed by automation, this is very much OK. Like I said, all good automation starts it's life as a manual test. Maybe I should have said all good testing begins it's life as a good test design.

    ReplyDelete
    Replies
      Reply
  6. UnknownJune 11, 2009 at 7:13:00 AM PDT

    I've been tasked with setting up a QA department at my company.

    What books would you recommend to learning what QA is and how to do it effectively?

    Thanks,

    Simon Johnson

    ReplyDelete
    Replies
      Reply
  7. James WhittakerJune 11, 2009 at 3:40:00 PM PDT

    Simon (Ckwop): you're asking an author what books he recommends? Take a guess? Actually, my books will teach you testing, not necessarily how to set up a QA shop. And although he works at my former employer, Alan Page's book is top drawer stuff.

    ReplyDelete
    Replies
      Reply
  8. UnknownJune 11, 2009 at 10:01:00 PM PDT

    I was talking to my friend other day about this. It is very important to identify what exactly needs to be tested, the how is separate problem that should not be mixed. This helps us focus on designing better test cases.

    ReplyDelete
    Replies
      Reply
  9. sachinJune 12, 2009 at 3:09:00 AM PDT

    The moot point is clearly and simply well explained.

    Sachin

    ReplyDelete
    Replies
      Reply
  10. ElisabethJune 14, 2009 at 8:48:00 AM PDT

    Hi James,

    Good to see you online.

    I confess that I'm struggling to frame my comment because your post seems to assume a context that I just don't live in anymore, a test-last rather than test-driven context in which manual testing & automation compete for time, attention, & resources after the software is implemented.

    But the assertion "All good automation begins it's life as manual test cases" strikes me as just wrong for any context.

    Even in test-last contexts, I've automated tests that I never executed manually and that found bugs on their first run. Model-based testing FTW.

    Perhaps you meant to say, "All good automation involves test cases that would still be worth executing manually if that were necessary?"

    Elisabeth

    ReplyDelete
    Replies
      Reply
  11. Gitte OttosenJune 15, 2009 at 10:47:00 AM PDT

    This comment has been removed by the author.

    ReplyDelete
    Replies
      Reply
  12. Gitte OttosenJune 15, 2009 at 11:28:00 AM PDT

    Great post James. I totally agree when it comes to the value of test design. How the test design is documented is up to the individual, the importent thing is the process of designing the test - think about what you are doing befor you do it :-)

    Good to find you again, was wondering how to find you when you left MS. Maybe we'll see you at EUROSTAR sometime.

    Take care,
    Gitte
    Program Test Manager
    Systematic

    ReplyDelete
    Replies
      Reply
  13. Anup PatnaikJune 16, 2009 at 11:39:00 PM PDT

    James ... I'm glad you wrote this. The obsession with the product/solution is becoming a big problem with budding testers.

    I'd like to add here that I'm seeing it's not just the testers but also the clients/project managers at the client side who push for automation without having any understanding of if at all automation would help.

    Automation when used thoughtfully does do wonderful things. The industry needs to stop creating a buzz around automation and take it at face (real) value.

    -- Anup / Director QE - QA InfoTech

    ReplyDelete
    Replies
      Reply
  14. Quality GuyJuly 2, 2009 at 6:41:00 AM PDT

    James, good article, and it's about time someone is focussing on design. People who are not Testers dont understand when we talk about design, they think only Developers design (code).

    In my view this is part of the problem because if those people dont have a "tester brain" they just dont get it, they dont understand why the product is still buggy after being fully automated, or why all those person days of manual testing did not deliver a bug free system...and when the opposite occurs and the System is near perfect they still cant explain why this happened.

    Maybe we should be blogging more about the "tester brain" (what is it and how come those who have it can spot it in others), and then we can understand more about the underlying designs.

    Does this make sense? Anyone?

    ReplyDelete
    Replies
      Reply
  15. Ashwin PalaparthiJuly 26, 2009 at 8:53:00 PM PDT

    "Good test design" certainly is a topic not factored into what most testers call as "automation".

    Automation that refers to only automated test execution perhaps need to be more explicitly extended to include "automated test design" as well.

    Pairwise testing, markov-chains and so on... and of course the tester brain as well.

    Ashwin Palaparthi,
    Founder, TestersDesk.com

    ReplyDelete
    Replies
      Reply
Add comment
Load more...

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

  

Labels


  • TotT 104
  • GTAC 61
  • James Whittaker 42
  • Misko Hevery 32
  • Code Health 31
  • Anthony Vallone 27
  • Patrick Copeland 23
  • Jobs 18
  • Andrew Trenk 13
  • C++ 11
  • Patrik Höglund 8
  • JavaScript 7
  • Allen Hutchison 6
  • George Pirocanac 6
  • Zhanyong Wan 6
  • Harry Robinson 5
  • Java 5
  • Julian Harty 5
  • Adam Bender 4
  • Alberto Savoia 4
  • Ben Yu 4
  • Erik Kuefler 4
  • Philip Zembrod 4
  • Shyam Seshadri 4
  • Chrome 3
  • Dillon Bly 3
  • John Thomas 3
  • Lesley Katzen 3
  • Marc Kaplan 3
  • Markus Clermont 3
  • Max Kanat-Alexander 3
  • Sonal Shah 3
  • APIs 2
  • Abhishek Arya 2
  • Alan Myrvold 2
  • Alek Icev 2
  • Android 2
  • April Fools 2
  • Chaitali Narla 2
  • Chris Lewis 2
  • Chrome OS 2
  • Diego Salas 2
  • Dori Reuveni 2
  • Jason Arbon 2
  • Jochen Wuttke 2
  • Kostya Serebryany 2
  • Marc Eaddy 2
  • Marko Ivanković 2
  • Mobile 2
  • Oliver Chang 2
  • Simon Stewart 2
  • Stefan Kennedy 2
  • Test Flakiness 2
  • Titus Winters 2
  • Tony Voellm 2
  • WebRTC 2
  • Yiming Sun 2
  • Yvette Nameth 2
  • Zuri Kemp 2
  • Aaron Jacobs 1
  • Adam Porter 1
  • Adam Raider 1
  • Adel Saoud 1
  • Alan Faulkner 1
  • Alex Eagle 1
  • Amy Fu 1
  • Anantha Keesara 1
  • Antoine Picard 1
  • App Engine 1
  • Ari Shamash 1
  • Arif Sukoco 1
  • Benjamin Pick 1
  • Bob Nystrom 1
  • Bruce Leban 1
  • Carlos Arguelles 1
  • Carlos Israel Ortiz García 1
  • Cathal Weakliam 1
  • Christopher Semturs 1
  • Clay Murphy 1
  • Dagang Wei 1
  • Dan Maksimovich 1
  • Dan Shi 1
  • Dan Willemsen 1
  • Dave Chen 1
  • Dave Gladfelter 1
  • David Bendory 1
  • David Mandelberg 1
  • Derek Snyder 1
  • Diego Cavalcanti 1
  • Dmitry Vyukov 1
  • Eduardo Bravo Ortiz 1
  • Ekaterina Kamenskaya 1
  • Elliott Karpilovsky 1
  • Elliotte Rusty Harold 1
  • Espresso 1
  • Felipe Sodré 1
  • Francois Aube 1
  • Gene Volovich 1
  • Google+ 1
  • Goran Petrovic 1
  • Goranka Bjedov 1
  • Hank Duan 1
  • Havard Rast Blok 1
  • Hongfei Ding 1
  • Jason Elbaum 1
  • Jason Huggins 1
  • Jay Han 1
  • Jeff Hoy 1
  • Jeff Listfield 1
  • Jessica Tomechak 1
  • Jim Reardon 1
  • Joe Allan Muharsky 1
  • Joel Hynoski 1
  • John Micco 1
  • John Penix 1
  • Jonathan Rockway 1
  • Jonathan Velasquez 1
  • Josh Armour 1
  • Julie Ralph 1
  • Kai Kent 1
  • Kanu Tewary 1
  • Karin Lundberg 1
  • Kaue Silveira 1
  • Kevin Bourrillion 1
  • Kevin Graney 1
  • Kirkland 1
  • Kurt Alfred Kluever 1
  • Manjusha Parvathaneni 1
  • Marek Kiszkis 1
  • Marius Latinis 1
  • Mark Ivey 1
  • Mark Manley 1
  • Mark Striebeck 1
  • Matt Lowrie 1
  • Meredith Whittaker 1
  • Michael Bachman 1
  • Michael Klepikov 1
  • Mike Aizatsky 1
  • Mike Wacker 1
  • Mona El Mahdy 1
  • Noel Yap 1
  • Palak Bansal 1
  • Patricia Legaspi 1
  • Per Jacobsson 1
  • Peter Arrenbrecht 1
  • Peter Spragins 1
  • Phil Norman 1
  • Phil Rollet 1
  • Pooja Gupta 1
  • Project Showcase 1
  • Radoslav Vasilev 1
  • Rajat Dewan 1
  • Rajat Jain 1
  • Rich Martin 1
  • Richard Bustamante 1
  • Roshan Sembacuttiaratchy 1
  • Ruslan Khamitov 1
  • Sam Lee 1
  • Sean Jordan 1
  • Sebastian Dörner 1
  • Sharon Zhou 1
  • Shiva Garg 1
  • Siddartha Janga 1
  • Simran Basi 1
  • Stan Chan 1
  • Stephen Ng 1
  • Tejas Shah 1
  • Test Analytics 1
  • Test Engineer 1
  • Tim Lyakhovetskiy 1
  • Tom O'Neill 1
  • Vojta Jína 1
  • automation 1
  • dead code 1
  • iOS 1
  • mutation testing 1


Archive


  • ►  2025 (1)
    • ►  Jan (1)
  • ►  2024 (13)
    • ►  Dec (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (1)
    • ►  May (3)
    • ►  Apr (3)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2023 (14)
    • ►  Dec (2)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (3)
    • ►  Aug (1)
    • ►  Apr (1)
  • ►  2022 (2)
    • ►  Feb (2)
  • ►  2021 (3)
    • ►  Jun (1)
    • ►  Apr (1)
    • ►  Mar (1)
  • ►  2020 (8)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  May (1)
  • ►  2019 (4)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Jul (1)
    • ►  Jan (1)
  • ►  2018 (7)
    • ►  Nov (1)
    • ►  Sep (1)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (1)
    • ►  Feb (1)
  • ►  2017 (17)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2016 (15)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (1)
    • ►  Sep (2)
    • ►  Aug (1)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (1)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2015 (14)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Aug (1)
    • ►  Jun (1)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (1)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2014 (24)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Sep (2)
    • ►  Aug (2)
    • ►  Jul (3)
    • ►  Jun (3)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2013 (16)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Jan (2)
  • ►  2012 (11)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (1)
    • ►  Aug (4)
  • ►  2011 (39)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (2)
    • ►  Aug (4)
    • ►  Jul (2)
    • ►  Jun (5)
    • ►  May (4)
    • ►  Apr (3)
    • ►  Mar (4)
    • ►  Feb (5)
    • ►  Jan (3)
  • ►  2010 (37)
    • ►  Dec (3)
    • ►  Nov (3)
    • ►  Oct (4)
    • ►  Sep (8)
    • ►  Aug (3)
    • ►  Jul (3)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (3)
    • ►  Mar (3)
    • ►  Feb (2)
    • ►  Jan (1)
  • ▼  2009 (54)
    • ►  Dec (3)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (5)
    • ►  Aug (4)
    • ►  Jul (15)
    • ▼  Jun (8)
      • The plague of repetitiveness
      • The 7 Plagues of Software Testing
      • GTAC: Call for Proposals
      • Google Test Automation Conference 2009 - Zurich, S...
      • Burning Test Questions at Google
      • I'm a Googler now
      • James Whittaker joins Google
      • My Selenium Tests Aren't Stable!
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (5)
    • ►  Jan (4)
  • ►  2008 (75)
    • ►  Dec (6)
    • ►  Nov (8)
    • ►  Oct (9)
    • ►  Sep (8)
    • ►  Aug (9)
    • ►  Jul (9)
    • ►  Jun (6)
    • ►  May (6)
    • ►  Apr (4)
    • ►  Mar (4)
    • ►  Feb (4)
    • ►  Jan (2)
  • ►  2007 (41)
    • ►  Oct (6)
    • ►  Sep (5)
    • ►  Aug (3)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (7)
    • ►  Mar (5)
    • ►  Feb (5)
    • ►  Jan (4)

Feed

  • Google
  • Privacy
  • Terms