From QA to Engineering Productivity
Tuesday, March 22, 2016
By Ari Shamash
In Google’s early days, a small handful of software engineers built, tested, and released software. But as the user-base grew and products proliferated, engineers started specializing in roles, creating more scale in the development process:
This story focuses on the evolution of quality assurance and the roles of the engineers behind it at Google. The REs and SREs also evolved, but we’ll leave that for another day.
Initially, teams relied heavily on manual operations. When we attempted to automate testing, we largely focused on the frontends, which worked, because Google was small and our products had fewer integrations. However, as Google grew, longer and longer manual test cycles bogged down iterations and delayed feature launches. Also, since we identified bugs later in the development cycle, it took us longer and longer to fix them. We determined that pushing testing upstream via automation would help address these issues and accelerate velocity.
As manual testing transitioned to automated processes, two separate testing roles began to emerge at Google:
The impact was significant:
Manual operations were reduced to manual verification on new features, and typically only in end-to-end, cross product integration boundaries. TEs developed extreme depth of knowledge for the products they supported. They became go-to engineers for product teams that needed expertise in test automation and integration. Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code, and constantly designing better and more creative ways to identify weak spots and break software.
SETs (in collaboration with TEs and other engineers) built a wide array of test automation tools and developed best practices that were applicable across many products. Release velocity accelerated for products. All was good, and there was much rejoicing!
SETs initially focused on building tools for reducing the testing cycle time, since that was the most manually intensive and time consuming phase of getting product code into production. We made some of these tools available to the software development community: webdriver improvements, protractor, espresso, EarlGrey, martian proxy, karma, and GoogleTest. SETs were interested in sharing and collaborating with others in the industry and established conferences. The industry has also embraced the Test Engineering discipline, as other companies hired software engineers into similar roles, published articles, and drove Test-Driven Development into mainstream practices.
Through these efforts, the testing cycle time decreased dramatically, but interestingly the overall velocity did not increase proportionately, since other phases in the development cycle became the bottleneck. SETs started building tools to accelerate all other aspects of product development, including:
In summary, the work done by the SETs naturally progressed from supporting only product testing efforts to include supporting product development efforts as well. Their role now encompassed a much broader Engineering Productivity agenda.
Given the expanded SET charter, we wanted the title of the role to reflect the work. But what should the new title be? We empowered the SETs to choose a new title, and they overwhelmingly (91%) selected Software Engineer, Tools & Infrastructure (abbreviated to SETI).
Today, SETIs and TEs still collaborate very closely on optimizing the entire development life cycle with a goal of eliminating all friction from getting features into production. Interested in building next generation tools and infrastructure? Join us (SETI, TE)!
In Google’s early days, a small handful of software engineers built, tested, and released software. But as the user-base grew and products proliferated, engineers started specializing in roles, creating more scale in the development process:
- Test Engineers (TEs) -- tested new products and systems integration
- Release Engineers (REs) -- pushed bits into production
- Site Reliability Engineers (SREs) -- managed systems and data centers 24x7.
This story focuses on the evolution of quality assurance and the roles of the engineers behind it at Google. The REs and SREs also evolved, but we’ll leave that for another day.
Initially, teams relied heavily on manual operations. When we attempted to automate testing, we largely focused on the frontends, which worked, because Google was small and our products had fewer integrations. However, as Google grew, longer and longer manual test cycles bogged down iterations and delayed feature launches. Also, since we identified bugs later in the development cycle, it took us longer and longer to fix them. We determined that pushing testing upstream via automation would help address these issues and accelerate velocity.
As manual testing transitioned to automated processes, two separate testing roles began to emerge at Google:
- Test Engineers (TEs) -- With their deep product knowledge and test/quality domain expertise, TEs focused on what should be tested.
- Software Engineers in Test (SETs) -- Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.
The impact was significant:
- Automated tests became more efficient and deterministic (e.g. by improving runtimes, eliminating sources of flakiness, etc.)
- Metrics driven engineering proliferated (e.g. improving code and feature coverage led to higher quality products).
Manual operations were reduced to manual verification on new features, and typically only in end-to-end, cross product integration boundaries. TEs developed extreme depth of knowledge for the products they supported. They became go-to engineers for product teams that needed expertise in test automation and integration. Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code, and constantly designing better and more creative ways to identify weak spots and break software.
SETs (in collaboration with TEs and other engineers) built a wide array of test automation tools and developed best practices that were applicable across many products. Release velocity accelerated for products. All was good, and there was much rejoicing!
SETs initially focused on building tools for reducing the testing cycle time, since that was the most manually intensive and time consuming phase of getting product code into production. We made some of these tools available to the software development community: webdriver improvements, protractor, espresso, EarlGrey, martian proxy, karma, and GoogleTest. SETs were interested in sharing and collaborating with others in the industry and established conferences. The industry has also embraced the Test Engineering discipline, as other companies hired software engineers into similar roles, published articles, and drove Test-Driven Development into mainstream practices.
Through these efforts, the testing cycle time decreased dramatically, but interestingly the overall velocity did not increase proportionately, since other phases in the development cycle became the bottleneck. SETs started building tools to accelerate all other aspects of product development, including:
- Extending IDEs to make writing and reviewing code easier, shortening the “write code” cycle
- Automating release verification, shortening the “release code” cycle.
- Automating real time production system log verification and anomaly detection, helping automate production monitoring.
- Automating measurement of developer productivity, helping understand what’s working and what isn’t.
In summary, the work done by the SETs naturally progressed from supporting only product testing efforts to include supporting product development efforts as well. Their role now encompassed a much broader Engineering Productivity agenda.
Given the expanded SET charter, we wanted the title of the role to reflect the work. But what should the new title be? We empowered the SETs to choose a new title, and they overwhelmingly (91%) selected Software Engineer, Tools & Infrastructure (abbreviated to SETI).
Today, SETIs and TEs still collaborate very closely on optimizing the entire development life cycle with a goal of eliminating all friction from getting features into production. Interested in building next generation tools and infrastructure? Join us (SETI, TE)!
Would be able to provide more detail with regards to expectations of the TE when it comes to their technical abilities, is it similar to SETI. Would TEs at google say that they are automation experts, or are they more focused and specialized on manual testing?
ReplyDeleteI believe the article is saying the TEs are good at making test plans, risk analysis, and manual testing while SEIT/SETI excel at automating tests, builds, releases, deployment, gathering metrics, and finding ways to reduce cycle time.
DeleteI find this to be a useful distinction for job descriptions and it's worth noting that both roles are very valuable to product teams.
Should TE and SETI be physically separate people? I believe they should be, as the skills and mind-set of a TE and SETI should be/are different, as they have a different focus?
DeleteSteve that's more or less the accepted narrative, but I think it's a bit insulting to both groups. Are we really saying that they can't learn a skill?
DeleteThere is an older, but still accurate, post about the expectations and responsibilities of test engineers at Google.
Deletehttp://googletesting.blogspot.com/2013/01/test-engineers-google.html
As for should they be separate people, it depends on the team (and, thinking outside of Google, the company).
For a smaller team/company, it wouldn't make sense to have distinction between the roles because usually there'd only be one person responsible for everything. However, when products get larger and more complex, the team grows, and people tend to gravitate toward the responsibilities that interest them most.
It's not that SETIs can't do the work we primarily attribute to TEs, or vice versa. Many engineers find the primary responsibilities of one role or the other more interesting and more in line with what they want to do. Having separate roles makes things simpler versus having a combined role with a laundry list of responsibilities, many of which an individual engineer might not be doing.
Thank you for your feedback.
DeleteTo clarify, the purpose of this post was to describe the transition from Software Engineer in Test (SET) to Software Engineer, Tools & Infrastructure (SETI) and explain the new SETI role. Look for a future post on the evolution of the TE role at Google, but also refer to past post such as http://googletesting.blogspot.com/2013/01/test-engineers-google.html .
To answer some of the immediate questions:
- The TE role does have a coding bar candidates need to meet when they interview for a TE position, and automated testing is part of the TE role. The technical bar is quite high for all individuals at Google: All engineers, and many others (e.g. people managers, program managers, etc.) also contribute code.
- As people's passions change over time, they can (and do) learn new skills, leading to a mobility within teams and roles. Via rotations, engineers can also try out other roles before committing to a change.
@ Ari, how good is to involve a TE in JUnit Scripting
DeleteHi Nithiya --
DeleteAll engineers at Google are expected to contribute test cases for code that they write, not just TEs. We actively avoid situations where TEs end up writing all the test cases for code written by others.
From a team structure standpoint are SETIs embedded on the SWE teams that they support, or are they on their own smaller teams. What are the pros/cons with the current structure?
ReplyDeleteI think it's possible that you could have one of each. On an Agile Scrum team I think you would always want to have a TE. Having a SETI is the goal but it may not always be fiscally possible or needed for a mature team. More frugal organizations may have 'floating' SETI engineers which move from team to team as needed.
DeleteThen how about TE? Should they be embedded and report to the technical dev manager? Or should there by a separate group which the TE's belongs to, and are "loaned" to the feature/project teams?
DeleteBoth models exist within Google, depending on team size, structure, etc. There are trade offs for both models. Perhaps this is something that we can address in a future blog post.
DeleteThe key to long term success is to establish a community across Google, where SETI's and TE's can share ideas across organizational and product boundaries.
It is very interesting and excellent article showing the reality and evolution that has the area of testing. What I like most is that unlike the 2 roles (SETI and TE) and integrated work will allow a response time faster for the attention of the SDLC.
DeleteMany times you require 2 consulting roles to the same person
I agree with you, Alvaro, many times it takes disparate skills and team work to solve a problem. In an ideal world, the same person would have all the necessary skills, but that rarely happens, especially for complex issues.
DeleteI read this article recently: http://www.wired.com/2016/04/google-ensures-services-almost-never-go/
It talks about the SRE discipline at Google. What I find very interesting is the paragraph that talks about the two types of engineers that are hired, e.g. “set of technical skills that is useful to SRE but is rare for most software engineers"..
I find this very analogous to the SETI/TE split.
This post could not have come at a more appropriate time for my team. We are in the middle of the evolution from a more manual approach to an ever increasing set of automation and tooling. We have also recognized the necessity of our "Manual" testers, but it has been hard putting our finger on it in relation to the automation teams. This article helped to define that line a little better for us. I would love if you could provide any examples, allegorical or otherwise, about the two groups working together on a project? Maybe fodder for a future blog post?
ReplyDeleteA future blog post on this sounds like a good idea.
DeleteA suggestion: If you can attend GTAC (http://developers.google.com/gtac), feel free to reach out to any of the Google engineers there, I'm sure you'll get a lot of advice and ideas.
I am at the same stage with my team. Transitioning from 100% manual to automation dominant approach. Bumpy road but very rewarding!
DeleteAgreed on bumpy and very rewarding! Persevere through it, it'll be worth it at the end. Please do share your milestones, we'd love to hear all about it.
DeleteThank you for replying Ari. I just submitted my application for GTAC!
DeleteHi Ari, great article! I have a question on this excerpt:
ReplyDelete"Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code..."
So the TEs write scripts and create tools for the devs to use - not the SETs? Is this because the scripts and tools require domain / product knowledge which the TEs possess?
This distinction between the two roles is very interesting, as is the strong point made that engineers in general have coding skills. We are working towards this in my organisation but it is quite daunting, mainly due to the vastness of applications, products languages etc.
Simon Rigler
Hi Simon,
ReplyDeleteGreat question, thank you for asking for clarification!
Regarding the sentence, "Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code..."
We added that sentence to indicate that TEs, if they weren't already, started contributing tools and code in a big way as part of their evolution at Google. In fact, contributing technically is part of the job description for TEs. TEs at Google cannot be successful w/out technical contributions.
This task is by no means exclusive to TEs only. Every engineer within Google (TE or otherwise) ends up contributing tools and code during their career.
I hope this clarifies, feel free to follow up if you have any further questions.
Ari, Good article on your journey from SETs to SETI. Would like to know how the whole organization is now structured at Google specifically product development for various verticals and different/one productivity organization.
ReplyDeleteThere is an older, but still accurate, post about the expectations
ReplyDeleteDo you think for a technically oriented Test Engineer inside of Google its possible to transfer into the role of Software Engineer in Test and Infrastructure? Thanks :-)
ReplyDeleteYes, ladder transfers happen between SETI and TE in both directions (and between other positions, too).
DeleteThanks for sharing your eloquent story about Google evolution of quality assurance and the roles of the engineers , this is same experience we went through in our company. In order to close the test
ReplyDeletegap we improved our team cohesion and operational agility by redesign our organization away for a hierarchy organization to network organization. In addition be began merit badge system to encourage
our QA to shift left by allowing them examine other tools and methods to determine if want to pursue them as a vocation. Looking foward to sharing experience and discussing idea about
ways to increas engineering productivity.
I love this post, I need to read it a couple of time again before to get it totally.
ReplyDeleteVery happy to hear about that kind of evolution in IT and Testing Engineering!
BTW:
The g.con links at the end of the Post does not pass the Integration Test with careers.google.com.
Thanks for the heads up! We've updated the links.
ReplyDeleteCurios to know the organizational structure, do the REs and SREs in the EP organization same as TE and SETI? or in other structure style?
ReplyDeleteHi Ari, It's really great article!
ReplyDeleteJust one quick question here, as you mentioned SETI are building tools to accelerate all other aspects of product development, now container technique are becoming mainstream. So is there any overlap between SETIs and REs? Will these two roles combine together in future.
Hi Ari, An informative article.
ReplyDeleteDo the role SET's support cross platform organization? Meaning, SET's working for Software QA initiatives are expected to support Hardware QA initiatives as well? I am curious to understand how Google handles this after the new roles are coined, generally the Software QA engineers primarily focus on the productivity and efficiency measurements and they write and perform unit test using this as a basis whereas the Hardware engineers (in the testing organization) should have skills in the reliability level like Mean time between failure, Mean cycle between failure etc...
On GTAC blog (https://events.withgoogle.com/gtac-2017/), there was a mention about a plan to come up with an Engineering Productivity conference in 2018, but have not found any announcements or links yet on the same. Any idea when the new conference shall be held? Pointer to the details of such a conference would help.
ReplyDeleteI assume from the discussion that SETI's produce automation test harnesses. These would be used across multiple Agile/Scrum teams/squads. Are the SETI's embedded in the Agile entities? How do you resolve issues making use of this central investment? Are non-fuctional tests (chaos, stress, load, longevity, FMEA) performed by embedded individuals, and if so, how do you resolve cross-Agile-Entity impact?
ReplyDelete