Testing Blog
From QA to Engineering Productivity
Tuesday, March 22, 2016
By Ari Shamash
In Google’s early days, a small handful of software engineers built, tested, and released software. But as the user-base grew and products proliferated, engineers started specializing in roles, creating more scale in the development process:
Test Engineers (TEs) -- tested new products and systems integration
Release Engineers (REs) -- pushed bits into production
Site Reliability Engineers (SREs) -- managed systems and data centers 24x7.
This story focuses on the evolution of quality assurance and the roles of the engineers behind it at Google. The REs and SREs also evolved, but we’ll leave that for another day.
Initially, teams relied heavily on manual operations. When we attempted to automate testing, we largely focused on the frontends, which worked, because Google was small and our products had fewer integrations. However, as Google grew, longer and longer manual test cycles bogged down iterations and delayed feature launches. Also, since we identified bugs later in the development cycle, it took us
longer and longer
to
fix them
. We determined that pushing testing upstream via
automation
would help address these issues and accelerate velocity.
As manual testing transitioned to automated processes, two separate testing roles began to emerge at Google:
Test Engineers (TEs) -- With their deep product knowledge and test/quality domain expertise, TEs focused on what should be tested.
Software Engineers in Test (SETs) -- Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.
The impact was significant:
Automated tests became more efficient and deterministic (e.g. by improving runtimes, eliminating sources of flakiness, etc.)
Metrics driven engineering proliferated (e.g. improving code and feature coverage led to higher quality products).
Manual operations were reduced to manual verification on new features, and typically only in end-to-end, cross product integration boundaries. TEs developed extreme depth of knowledge for the products they supported. They became go-to engineers for product teams that needed expertise in test automation and integration. Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code, and constantly designing better and more creative ways to identify weak spots and break software.
SETs (in collaboration with TEs and other engineers) built a wide array of test automation tools and developed best practices that were applicable across many products. Release velocity accelerated for products. All was good, and
there was much rejoicing
!
SETs initially focused on building tools for reducing the testing cycle time, since that was the most manually intensive and time consuming phase of getting product code into production. We made some of these tools available to the software development community:
webdriver
improvements
,
protractor
,
espresso
,
EarlGrey
,
martian proxy
,
karma
, and
GoogleTest
. SETs were interested in sharing and collaborating with others in the industry and established
conferences
. The industry has also embraced the Test Engineering discipline, as other companies hired
software engineers
into similar roles,
published articles
, and drove
Test-Driven Development
into mainstream practices.
Through these efforts, the testing cycle time decreased dramatically, but interestingly the overall velocity did not increase proportionately, since
other phases in the development cycle became the bottleneck
. SETs started building tools to accelerate all other aspects of product development, including:
Extending IDEs to make writing and reviewing code easier, shortening the “write code” cycle
Automating release verification, shortening the “release code” cycle.
Automating real time production system log verification and anomaly detection
, helping automate production monitoring.
Automating measurement of developer productivity, helping understand what’s working and what isn’t.
In summary, the work done by the SETs naturally progressed from supporting only product testing efforts to include supporting product development efforts as well. Their role now encompassed a much broader Engineering Productivity agenda.
Given the expanded SET charter, we wanted the title of the role to reflect the work. But what should the new title be?
We empowered the SETs to choose a new title
, and they overwhelmingly (91%) selected Software Engineer, Tools & Infrastructure (abbreviated to SETI).
Today, SETIs and TEs still collaborate very closely on optimizing the entire development life cycle with a goal of eliminating all friction from getting features into production. Interested in building next generation tools and infrastructure? Join us (
SETI
,
TE
)!
33 comments
EarlGrey - iOS Functional UI Testing Framework
Tuesday, February 16, 2016
By Siddartha Janga on behalf of Google iOS Developers
Brewing for quite some time, we are excited to announce EarlGrey, a functional UI testing framework for iOS. Several Google apps like
YouTube
,
Google Calendar
,
Google Photos
,
Google Translate
,
Google Play Music
and many more have successfully adopted the framework for their functional testing needs.
The key features offered by EarlGrey include:
Powerful built-in synchronization :
Tests will automatically wait for events such as animations, network requests, etc. before interacting with the UI. This will result in tests that are easier to write (no sleeps or waits) and simple to maintain (straight up procedural description of test steps).
Visibility checking :
All interactions occur on elements that users can see. For example, attempting to tap a button that is behind an image will lead to test failure immediately.
Flexible design :
The components that determine element selection, interaction, assertion and synchronization have been designed to be extensible.
Are you in need for a cup of refreshing EarlGrey? EarlGrey has been open sourced under the
Apache license
. Check out the
getting started guide
and add EarlGrey to your project using
CocoaPods
or manually add it to your Xcode project file.
9 comments
GTAC 2015 Wrap Up
Tuesday, December 08, 2015
by Michael Klepikov and Lesley Katzen on behalf of the GTAC Committee
The ninth
GTAC
(Google Test Automation Conference) was held on November 10-11 at the
Google Cambridge
office, the “Hub” of innovation. The conference was completely packed with presenters and attendees from all over the world, from industry and academia, discussing advances in test automation and the test engineering computer science field, bringing with them a huge diversity of experiences. Speakers from numerous companies and universities (Applitools, Automattic, Bitbar, Georgia Tech, Google, Indian Institute of Science, Intel, LinkedIn, Lockheed Martin, MIT, Nest, Netflix, OptoFidelity, Splunk, Supersonic, Twitter, Uber, University of Waterloo) spoke on a variety of interesting and cutting edge test automation topics.
All presentation videos and slides are posted on the
Video Recordings
and
Presentations
pages. All videos have professionally transcribed closed captions, and the YouTube descriptions have the slides links. Enjoy and share!
We had over 1,300 applicants and over 200 of those for speaking. Over 250 people filled our venue to capacity, and the live stream had a peak of about 400 concurrent viewers, with about 3,300 total viewing hours.
Our goal in hosting GTAC is to make the conference highly relevant and useful for both attendees and the larger test engineering community as a whole. Our post-conference survey shows that we are close to achieving that goal; thanks to everyone who completed the feedback survey!
Our 82 survey respondents were mostly (81%) test focused professionals with a wide range of 1 to 40 years of experience.
Another 76% of respondents rated the conference as a whole as above average, with marked satisfaction for the venue, the food (those Diwali treats!), and the breadth and coverage of the talks themselves.
The top five most popular talks were:
The Uber Challenge of Cross-Application/Cross-Device Testing (Apple Chow and Bian Jiang)
Your Tests Aren't Flaky (Alister Scott)
Statistical Data Sampling (Celal Ziftci and Ben Greenberg)
Coverage is Not Strongly Correlated with Test Suite Effectiveness (Laura Inozemtseva)
Chrome OS Test Automation Lab (Simran Basi and Chris Sosa).
Our social events also proved to be crowd pleasers. The social events were a direct response to feedback from GTAC 2014 for organized opportunities for socialization among the GTAC attendees.
This isn’t to say there isn’t room for improvement. We had 11% of respondents express frustration with event communications and provided some long, thoughtful suggestions for what we could do to improve next year. Also, many of the long form comments asked for a better mix of technologies, noting that mobile had a big presence in the talks this year.
If you have any suggestions on how we can improve, please comment on this post, or better yet – fill out the
survey
, which remains open. Based on feedback from last year urging more transparency in speaker selection, we included an individual outside of Google in the speaker evaluation. Feedback is precious, we take it very seriously, and we will use it to improve next time around.
Thank you to all the speakers, attendees, and online viewers who made this a special event once again. To receive announcements about the next GTAC, currently planned for early 2017, subscribe to the
Google Testing Blog
.
10 comments
GTAC 2015 is Next Week!
Friday, November 06, 2015
by
Anthony Vallone
on behalf of the GTAC Committee
The ninth GTAC (Google Test Automation Conference) commences on Tuesday, November 10th, at the Google Cambridge office. You can find the latest details on the
conference site
, including
schedule
,
speaker profiles
, and
travel tips
.
If you have not been invited to attend in person, you can
watch the event live
. And if you miss the livestream, we will post slides and videos later.
We have an outstanding speaker lineup this year, and we look forward to seeing you all there or online!
3 comments
Announcing the GTAC 2015 Agenda
Tuesday, October 27, 2015
by
Anthony Vallone
on behalf of the GTAC Committee
We have completed the selection and confirmation of all speakers and attendees for GTAC 2015. You can find the detailed agenda at:
developers.google.com/gtac/2015/schedule
.
Thank you to all who submitted proposals!
There is a lot of interest in GTAC once again this year with about 1400 applicants and about 200 of those for speaking. Unfortunately, our venue only seats 250. We will livestream the event as usual, so fret not if you were not selected to attend. Information about the livestream and other details will be posted on the GTAC site soon and announced here.
No comments
Labels
TotT
113
GTAC
61
James Whittaker
42
Misko Hevery
32
Code Health
31
Anthony Vallone
27
Patrick Copeland
23
Jobs
18
Andrew Trenk
13
C++
11
Patrik Höglund
8
JavaScript
7
Allen Hutchison
6
George Pirocanac
6
Zhanyong Wan
6
Harry Robinson
5
Java
5
Julian Harty
5
Adam Bender
4
Alberto Savoia
4
Ben Yu
4
Erik Kuefler
4
Philip Zembrod
4
Shyam Seshadri
4
Chrome
3
Dillon Bly
3
John Thomas
3
Lesley Katzen
3
Marc Kaplan
3
Markus Clermont
3
Max Kanat-Alexander
3
Sonal Shah
3
APIs
2
Abhishek Arya
2
Alan Myrvold
2
Alek Icev
2
Android
2
April Fools
2
Chaitali Narla
2
Chris Lewis
2
Chrome OS
2
Diego Salas
2
Dori Reuveni
2
Jason Arbon
2
Jochen Wuttke
2
Kostya Serebryany
2
Marc Eaddy
2
Marko Ivanković
2
Mobile
2
Oliver Chang
2
Simon Stewart
2
Stefan Kennedy
2
Test Flakiness
2
Titus Winters
2
Tony Voellm
2
WebRTC
2
Yiming Sun
2
Yvette Nameth
2
Zuri Kemp
2
Aaron Jacobs
1
Adam Porter
1
Adam Raider
1
Adel Saoud
1
Alan Faulkner
1
Alex Eagle
1
Amy Fu
1
Anantha Keesara
1
Antoine Picard
1
App Engine
1
Arham Jain
1
Ari Shamash
1
Arif Sukoco
1
Bartosz Papis
1
Benjamin Pick
1
Bob Nystrom
1
Bruce Leban
1
Carlos Arguelles
1
Carlos Israel Ortiz García
1
Cathal Weakliam
1
Christopher Semturs
1
Clay Murphy
1
Dagang Wei
1
Dan Maksimovich
1
Dan Shi
1
Dan Willemsen
1
Dave Chen
1
Dave Gladfelter
1
David Bendory
1
David Mandelberg
1
Derek Snyder
1
Diego Cavalcanti
1
Dmitry Vyukov
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Elliott Karpilovsky
1
Elliotte Rusty Harold
1
Espresso
1
Felipe Sodré
1
Francois Aube
1
Gene Volovich
1
Google+
1
Goran Petrovic
1
Goranka Bjedov
1
Hank Duan
1
Havard Rast Blok
1
Hongfei Ding
1
Jason Elbaum
1
Jason Huggins
1
Jay Han
1
Jeff Hoy
1
Jeff Listfield
1
Jessica Tomechak
1
Jim Reardon
1
Joe Allan Muharsky
1
Joel Hynoski
1
John Micco
1
John Penix
1
Jonathan Rockway
1
Jonathan Velasquez
1
Josh Armour
1
Julie Ralph
1
Kai Kent
1
Kanu Tewary
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Bourrillion
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Kyle Freeman
1
Manjusha Parvathaneni
1
Marek Kiszkis
1
Marius Latinis
1
Mark Ivey
1
Mark Manley
1
Mark Striebeck
1
Matt Lowrie
1
Meredith Whittaker
1
Michael Bachman
1
Michael Klepikov
1
Mike Aizatsky
1
Mike Wacker
1
Mona El Mahdy
1
Noel Yap
1
Palak Bansal
1
Patricia Legaspi
1
Per Jacobsson
1
Peter Arrenbrecht
1
Peter Spragins
1
Phil Norman
1
Phil Rollet
1
Pooja Gupta
1
Project Showcase
1
Radion Khait
1
Radoslav Vasilev
1
Rahul Singal
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roman Govsheev
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Saicharan Nimmala
1
Sam Lee
1
Sean Jordan
1
Sebastian Dörner
1
Shahar Roth
1
Sharon Zhou
1
Shiva Garg
1
Siddartha Janga
1
Simran Basi
1
Stan Chan
1
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Test Engineer
1
Tim Lyakhovetskiy
1
Tom O'Neill
1
Vojta Jína
1
Zhe Lu
1
automation
1
dead code
1
iOS
1
mutation testing
1
Archive
▼
2026
(7)
▼
Jul
(1)
Prefactoring: Clear the Way for Your New Feature
►
Jun
(1)
►
May
(2)
►
Apr
(1)
►
Mar
(2)
►
2025
(3)
►
Oct
(1)
►
Sep
(1)
►
Jan
(1)
►
2024
(13)
►
Dec
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(1)
►
May
(3)
►
Apr
(3)
►
Mar
(1)
►
Feb
(1)
►
2023
(14)
►
Dec
(2)
►
Nov
(2)
►
Oct
(5)
►
Sep
(3)
►
Aug
(1)
►
Apr
(1)
►
2022
(2)
►
Feb
(2)
►
2021
(3)
►
Jun
(1)
►
Apr
(1)
►
Mar
(1)
►
2020
(8)
►
Dec
(2)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
May
(1)
►
2019
(4)
►
Dec
(1)
►
Nov
(1)
►
Jul
(1)
►
Jan
(1)
►
2018
(7)
►
Nov
(1)
►
Sep
(1)
►
Jul
(1)
►
Jun
(2)
►
May
(1)
►
Feb
(1)
►
2017
(17)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(2)
►
Jun
(2)
►
May
(3)
►
Apr
(2)
►
Feb
(1)
►
Jan
(2)
►
2016
(15)
►
Dec
(1)
►
Nov
(2)
►
Oct
(1)
►
Sep
(2)
►
Aug
(1)
►
Jun
(2)
►
May
(3)
►
Apr
(1)
►
Mar
(1)
►
Feb
(1)
►
2015
(14)
►
Dec
(1)
►
Nov
(1)
►
Oct
(2)
►
Aug
(1)
►
Jun
(1)
►
May
(2)
►
Apr
(2)
►
Mar
(1)
►
Feb
(1)
►
Jan
(2)
►
2014
(24)
►
Dec
(2)
►
Nov
(1)
►
Oct
(2)
►
Sep
(2)
►
Aug
(2)
►
Jul
(3)
►
Jun
(3)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Feb
(1)
►
Jan
(2)
►
2013
(16)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
Jun
(2)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Jan
(2)
►
2012
(11)
►
Dec
(1)
►
Nov
(2)
►
Oct
(3)
►
Sep
(1)
►
Aug
(4)
►
2011
(39)
►
Nov
(2)
►
Oct
(5)
►
Sep
(2)
►
Aug
(4)
►
Jul
(2)
►
Jun
(5)
►
May
(4)
►
Apr
(3)
►
Mar
(4)
►
Feb
(5)
►
Jan
(3)
►
2010
(37)
►
Dec
(3)
►
Nov
(3)
►
Oct
(4)
►
Sep
(8)
►
Aug
(3)
►
Jul
(3)
►
Jun
(2)
►
May
(2)
►
Apr
(3)
►
Mar
(3)
►
Feb
(2)
►
Jan
(1)
►
2009
(54)
►
Dec
(3)
►
Nov
(2)
►
Oct
(3)
►
Sep
(5)
►
Aug
(4)
►
Jul
(15)
►
Jun
(8)
►
May
(3)
►
Apr
(2)
►
Feb
(5)
►
Jan
(4)
►
2008
(75)
►
Dec
(6)
►
Nov
(8)
►
Oct
(9)
►
Sep
(8)
►
Aug
(9)
►
Jul
(9)
►
Jun
(6)
►
May
(6)
►
Apr
(4)
►
Mar
(4)
►
Feb
(4)
►
Jan
(2)
►
2007
(41)
►
Oct
(6)
►
Sep
(5)
►
Aug
(3)
►
Jul
(2)
►
Jun
(2)
►
May
(2)
►
Apr
(7)
►
Mar
(5)
►
Feb
(5)
►
Jan
(4)
Feed