Google Testing Blog: Patricia Legaspi

Testing Blog

Partial Automation: Keeping humans in the loop

Tuesday, November 04, 2008

Posted by Patricia Legaspi, Test Engineering Manager

One of the challenges of automation is achieving complete automation. Ideally, complete or total automation would not require any human intervention or verification yet this is a difficult level to achieve. Investing Engineering time to completely automate tests is expensive and, many times, has diminishing returns. Rather than trying to achieve complete automation, investing in ways to make the most out of the automated test and the human time is time better spent.

Effective test report
Consider an automated UI test... Routinely, automated UI tests result in false negatives due to timing issues, the complexity of the steps taken, and other factors. These have to be investigated which can be time consuming. A thorough report can be created for automated UI tests to present log information, error reports, screen shots of the state of the application when the test failed, and an easy way to re-run the test in question. A human can make use of the information provided and effectively investigate the possible issue. If we can reduce the amount of work that someone has to do to investigate a test failure, the UI tests become more valuable and the human plays a much more important role as a "verifier." Had the report not provided any information for the failed test, the human would have spent, in some cases, hours investigating the issue rather than continuing to automate or run exploratory tests which would be of more value. Is there a way to maximize what people do well, while also maximizing the use of automation?

Applying human intelligence
What about tests that require a human eye? There are many arguments about tests that require a human make a judgment about the appearance of rendered UI objects. Machines are great at running tests that return in a firm pass or fail but for tests that require opinion or judgment, the human has an advantage. We've been experimenting with image comparison tests. Screen shots of a golden version of the application under test are compared to screen shots of the current release candidate to verify that the application continues to render properly and that the UI "looks good". Although image comparison techniques can determine if the screen shots are different, they cannot help determine if the difference is "important". The human comes back into the picture to complement the automated test. To make effective use of a person's time, the test results should be organized and well presented. For image comparison tests, a report which shows the golden and release candidate screen shots side-by-side with clearly highlighted differences or allows you to replace the golden screen shot is key. A tester can quickly navigate the reported differences and determine if they are actual failures or acceptable differences. Frequently, an application's UI will change which will result in expected image differences. If the differences are expected changes, the tester should be able to replace the golden with the new image with the click of a button for future comparison.

Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes. Automating the environment setup can be a major time saver and allow for added time to run in depth tests. Creating continuous builds to run automated tests ahead of time and preparing environments overnight results in added testing time.

Automated tests are rarely bullet proof, they are prone to errors and false failures. In some cases, creating an infrastructure to make test analysis easy for humans, is much more effective than trying to engineer tests to automatically "understand" the UI. We coined this, "Partial Automation." We found that by having a person in the loop dramatically reduces the time spent trying to over engineer the automated tests. Obviously, there are trade-offs to this approach and one size doesn't fit all. We believe that automation does not always need to mean complete automation; we automate as much as you can and consider how human judgment can be used efficiently.

6 comments

Google

Labels: Patricia Legaspi

Labels

TotT 113
GTAC 61
James Whittaker 42
Misko Hevery 32
Code Health 31
Anthony Vallone 27
Patrick Copeland 23
Jobs 18
Andrew Trenk 13
C++ 11
Patrik Höglund 8
JavaScript 7
Allen Hutchison 6
George Pirocanac 6
Zhanyong Wan 6
Harry Robinson 5
Java 5
Julian Harty 5
Adam Bender 4
Alberto Savoia 4
Ben Yu 4
Erik Kuefler 4
Philip Zembrod 4
Shyam Seshadri 4
Chrome 3
Dillon Bly 3
John Thomas 3
Lesley Katzen 3
Marc Kaplan 3
Markus Clermont 3
Max Kanat-Alexander 3
Sonal Shah 3
APIs 2
Abhishek Arya 2
Alan Myrvold 2
Alek Icev 2
Android 2
April Fools 2
Chaitali Narla 2
Chris Lewis 2
Chrome OS 2
Diego Salas 2
Dori Reuveni 2
Jason Arbon 2
Jochen Wuttke 2
Kostya Serebryany 2
Marc Eaddy 2
Marko Ivanković 2
Mobile 2
Oliver Chang 2
Simon Stewart 2
Stefan Kennedy 2
Test Flakiness 2
Titus Winters 2
Tony Voellm 2
WebRTC 2
Yiming Sun 2
Yvette Nameth 2
Zuri Kemp 2
Aaron Jacobs 1
Adam Porter 1
Adam Raider 1
Adel Saoud 1
Alan Faulkner 1
Alex Eagle 1
Amy Fu 1
Anantha Keesara 1
Antoine Picard 1
App Engine 1
Arham Jain 1
Ari Shamash 1
Arif Sukoco 1
Bartosz Papis 1
Benjamin Pick 1
Bob Nystrom 1
Bruce Leban 1
Carlos Arguelles 1
Carlos Israel Ortiz García 1
Cathal Weakliam 1
Christopher Semturs 1
Clay Murphy 1
Dagang Wei 1
Dan Maksimovich 1
Dan Shi 1
Dan Willemsen 1
Dave Chen 1
Dave Gladfelter 1
David Bendory 1
David Mandelberg 1
Derek Snyder 1
Diego Cavalcanti 1
Dmitry Vyukov 1
Eduardo Bravo Ortiz 1
Ekaterina Kamenskaya 1
Elliott Karpilovsky 1
Elliotte Rusty Harold 1
Espresso 1
Felipe Sodré 1
Francois Aube 1
Gene Volovich 1
Google+ 1
Goran Petrovic 1
Goranka Bjedov 1
Hank Duan 1
Havard Rast Blok 1
Hongfei Ding 1
Jason Elbaum 1
Jason Huggins 1
Jay Han 1
Jeff Hoy 1
Jeff Listfield 1
Jessica Tomechak 1
Jim Reardon 1
Joe Allan Muharsky 1
Joel Hynoski 1
John Micco 1
John Penix 1
Jonathan Rockway 1
Jonathan Velasquez 1
Josh Armour 1
Julie Ralph 1
Kai Kent 1
Kanu Tewary 1
Karin Lundberg 1
Kaue Silveira 1
Kevin Bourrillion 1
Kevin Graney 1
Kirkland 1
Kurt Alfred Kluever 1
Kyle Freeman 1
Manjusha Parvathaneni 1
Marek Kiszkis 1
Marius Latinis 1
Mark Ivey 1
Mark Manley 1
Mark Striebeck 1
Matt Lowrie 1
Meredith Whittaker 1
Michael Bachman 1
Michael Klepikov 1
Mike Aizatsky 1
Mike Wacker 1
Mona El Mahdy 1
Noel Yap 1
Palak Bansal 1
Patricia Legaspi 1
Per Jacobsson 1
Peter Arrenbrecht 1
Peter Spragins 1
Phil Norman 1
Phil Rollet 1
Pooja Gupta 1
Project Showcase 1
Radion Khait 1
Radoslav Vasilev 1
Rahul Singal 1
Rajat Dewan 1
Rajat Jain 1
Rich Martin 1
Richard Bustamante 1
Roman Govsheev 1
Roshan Sembacuttiaratchy 1
Ruslan Khamitov 1
Saicharan Nimmala 1
Sam Lee 1
Sean Jordan 1
Sebastian Dörner 1
Shahar Roth 1
Sharon Zhou 1
Shiva Garg 1
Siddartha Janga 1
Simran Basi 1
Stan Chan 1
Stephen Ng 1
Tejas Shah 1
Test Analytics 1
Test Engineer 1
Tim Lyakhovetskiy 1
Tom O'Neill 1
Vojta Jína 1
Zhe Lu 1
automation 1
dead code 1
iOS 1
mutation testing 1

Archive

▼ 2026 (7)
- ▼ Jul (1)
  - Prefactoring: Clear the Way for Your New Feature
- ► Jun (1)
- ► May (2)
- ► Apr (1)
- ► Mar (2)

► 2025 (3)
- ► Oct (1)
- ► Sep (1)
- ► Jan (1)

► 2024 (13)
- ► Dec (1)
- ► Oct (1)
- ► Sep (1)
- ► Aug (1)
- ► Jul (1)
- ► May (3)
- ► Apr (3)
- ► Mar (1)
- ► Feb (1)

► 2023 (14)
- ► Dec (2)
- ► Nov (2)
- ► Oct (5)
- ► Sep (3)
- ► Aug (1)
- ► Apr (1)

► 2022 (2)
- ► Feb (2)

► 2021 (3)
- ► Jun (1)
- ► Apr (1)
- ► Mar (1)

► 2020 (8)
- ► Dec (2)
- ► Nov (1)
- ► Oct (1)
- ► Aug (2)
- ► Jul (1)
- ► May (1)

► 2019 (4)
- ► Dec (1)
- ► Nov (1)
- ► Jul (1)
- ► Jan (1)

► 2018 (7)
- ► Nov (1)
- ► Sep (1)
- ► Jul (1)
- ► Jun (2)
- ► May (1)
- ► Feb (1)

► 2017 (17)
- ► Dec (1)
- ► Nov (1)
- ► Oct (1)
- ► Sep (1)
- ► Aug (1)
- ► Jul (2)
- ► Jun (2)
- ► May (3)
- ► Apr (2)
- ► Feb (1)
- ► Jan (2)

► 2016 (15)
- ► Dec (1)
- ► Nov (2)
- ► Oct (1)
- ► Sep (2)
- ► Aug (1)
- ► Jun (2)
- ► May (3)
- ► Apr (1)
- ► Mar (1)
- ► Feb (1)

► 2015 (14)
- ► Dec (1)
- ► Nov (1)
- ► Oct (2)
- ► Aug (1)
- ► Jun (1)
- ► May (2)
- ► Apr (2)
- ► Mar (1)
- ► Feb (1)
- ► Jan (2)

► 2014 (24)
- ► Dec (2)
- ► Nov (1)
- ► Oct (2)
- ► Sep (2)
- ► Aug (2)
- ► Jul (3)
- ► Jun (3)
- ► May (2)
- ► Apr (2)
- ► Mar (2)
- ► Feb (1)
- ► Jan (2)

► 2013 (16)
- ► Dec (1)
- ► Nov (1)
- ► Oct (1)
- ► Aug (2)
- ► Jul (1)
- ► Jun (2)
- ► May (2)
- ► Apr (2)
- ► Mar (2)
- ► Jan (2)

► 2012 (11)
- ► Dec (1)
- ► Nov (2)
- ► Oct (3)
- ► Sep (1)
- ► Aug (4)

► 2011 (39)
- ► Nov (2)
- ► Oct (5)
- ► Sep (2)
- ► Aug (4)
- ► Jul (2)
- ► Jun (5)
- ► May (4)
- ► Apr (3)
- ► Mar (4)
- ► Feb (5)
- ► Jan (3)

► 2010 (37)
- ► Dec (3)
- ► Nov (3)
- ► Oct (4)
- ► Sep (8)
- ► Aug (3)
- ► Jul (3)
- ► Jun (2)
- ► May (2)
- ► Apr (3)
- ► Mar (3)
- ► Feb (2)
- ► Jan (1)

► 2009 (54)
- ► Dec (3)
- ► Nov (2)
- ► Oct (3)
- ► Sep (5)
- ► Aug (4)
- ► Jul (15)
- ► Jun (8)
- ► May (3)
- ► Apr (2)
- ► Feb (5)
- ► Jan (4)

► 2008 (75)
- ► Dec (6)
- ► Nov (8)
- ► Oct (9)
- ► Sep (8)
- ► Aug (9)
- ► Jul (9)
- ► Jun (6)
- ► May (6)
- ► Apr (4)
- ► Mar (4)
- ► Feb (4)
- ► Jan (2)

► 2007 (41)
- ► Oct (6)
- ► Sep (5)
- ► Aug (3)
- ► Jul (2)
- ► Jun (2)
- ► May (2)
- ► Apr (7)
- ► Mar (5)
- ► Feb (5)
- ► Jan (4)

Feed

Google
Privacy
Terms