Testing Blog

What Test Engineers do at Google: Building Test Infrastructure

Friday, November 18, 2016
Share on Twitter Share on Facebook
Google
Labels: Jobs , Jochen Wuttke

15 comments :

  1. sul4bhNovember 18, 2016 at 1:55:00 PM PST

    How do you mark a test failure as flaky? Do you have an automated/intelligent system that flags a test run failure as flaky or do you do it manually?

    ReplyDelete
    Replies
    1. sabujpNovember 18, 2016 at 2:39:00 PM PST

      https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

      Delete
      Replies
        Reply
    2. Reply
  2. UnknownNovember 18, 2016 at 4:00:00 PM PST

    Very interesting. The replay technique sounds like an interesting alternative/variation to contract tests. I gave a presentation last week that makes very similar recommendations to this article: https://skillsmatter.com/skillscasts/8567-testable-software-architecture

    ReplyDelete
    Replies
      Reply
  3. imbusyNovember 18, 2016 at 4:05:00 PM PST

    Sounds like the consumer driven contracts idea implemented here: https://docs.pact.io - cool in theory, but hard to write readable tests for in practice.

    ReplyDelete
    Replies
    1. UnknownNovember 19, 2016 at 12:54:00 AM PST

      Yes, pacts were a strong influence on what we did.
      However, we never went quite as far as they did, and cut out some of the stuff that makes pacts very powerful in theory, but hard to write in practice. Most importantly, instead of writing the contracts in code, we simply store the exchanged data as protocol buffers (https://developers.google.com/protocol-buffers/). That has the advantage of being far simpler, but also restricts what contracts can do, since you have a "passive" contract, instead of code that gets executed.

      Delete
      Replies
        Reply
    2. Reply
  4. AnonymousNovember 21, 2016 at 8:28:00 AM PST

    This is really interesting, thanks for sharing. Will you be open sourcing your tool?

    ReplyDelete
    Replies
      Reply
  5. testhackerNovember 24, 2016 at 7:59:00 PM PST

    Understood that integration testing is now carried out as part of unit testing. Just wondering, is functional testing also being covered as part of unit tests? Wouldn't functional testing require some of E2E tests be retained?

    ReplyDelete
    Replies
    1. UnknownDecember 6, 2016 at 12:34:00 AM PST

      Yes, functional and system testing do require some E2E tests to be retained. But these tests do not have to run during the developer cycle, which basic integration tests are quite important in SOA systems that change rapidly.

      Delete
      Replies
        Reply
    2. Reply
  6. Pratik ShahNovember 27, 2016 at 3:26:00 PM PST

    any chance this will be open sourced? We would love to contribute!

    ReplyDelete
    Replies
      Reply
  7. UnknownNovember 29, 2016 at 3:34:00 AM PST

    This is interesting approach. We are nearly in the same situation, but in the beginning. Could you share how did you solve connection to databases (and other services with different protocols). Do you have started DBs for tests or do you mock them as well?

    Another thing we need to cope with is order of returned data. Some of our methods are allowed to return items in array in random order (this random order originates form DB without order specification).
    Did you see such a problems?

    ReplyDelete
    Replies
    1. UnknownDecember 6, 2016 at 12:30:00 AM PST

      So, generally speaking, at Google DBs are also services that "speak" protocol buffer. But for most tests and languages, we also have very lightweight in-memory implementations that are more convenient to use where a DB is needed.

      For sorted/unsorted stuff in arrays, that's a common problem. In the final version we opted for always treating arrays as unsorted, so our matching algorithm just checks if each element (and duplicates) occur, but not in which position. In a previous version we tried to add a markup language to the stored data to modify the way things are matched, but in this particular case, it turned out that just un-ordered lists work well practically always.

      Delete
      Replies
        Reply
    2. Reply
  8. UnknownDecember 6, 2016 at 12:33:00 AM PST

    There have been several questions about open-sourcing the implementation of the library we built.

    There are currently no plans to do that. The two main reasons are:
    * A lot of what we did in the implementation is Google specific. Once we split of the parts that make sense in open source, there wouldn't be much left.
    * There are very good implementations of these principles out there that work well with common languages and OS stacks. For example https://docs.pact.io/.

    ReplyDelete
    Replies
      Reply
  9. AnuragFebruary 6, 2017 at 8:55:00 PM PST

    Hi, thank you for this post I agree with you that Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code.very useful information

    ReplyDelete
    Replies
      Reply
  10. divpMarch 22, 2017 at 11:56:00 PM PDT

    This is really interesting, thanks for sharing....Automation engineers design, program, simulate and test automated machinery and processes in order to complete exact tasks. They typically are employed in industries such as car manufacturing or food processing plants, where robots or machines are used to perform specific functions.i agree that a testing Engineer is required to fully test the product or system to ensure it functions properly and meets the business needs..worth reading

    ReplyDelete
    Replies
      Reply
  11. jianfengApril 20, 2017 at 9:37:00 PM PDT

    Thanks for sharing. if integration test need to prepare some data, such as: get account by id, integration test may need to create one first, so question is : how to handle the preparation when replay?

    ReplyDelete
    Replies
      Reply
Add comment
Load more...

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

  

Labels


  • TotT 77
  • GTAC 61
  • James Whittaker 42
  • Misko Hevery 32
  • Anthony Vallone 27
  • Patrick Copeland 23
  • Jobs 18
  • Code Health 13
  • C++ 11
  • Andrew Trenk 10
  • Patrik Höglund 8
  • JavaScript 7
  • Allen Hutchison 6
  • George Pirocanac 6
  • Zhanyong Wan 6
  • Harry Robinson 5
  • Java 5
  • Julian Harty 5
  • Alberto Savoia 4
  • Ben Yu 4
  • Erik Kuefler 4
  • Philip Zembrod 4
  • Shyam Seshadri 4
  • Chrome 3
  • John Thomas 3
  • Lesley Katzen 3
  • Marc Kaplan 3
  • Markus Clermont 3
  • Sonal Shah 3
  • APIs 2
  • Abhishek Arya 2
  • Adam Bender 2
  • Alan Myrvold 2
  • Alek Icev 2
  • Android 2
  • April Fools 2
  • Chaitali Narla 2
  • Chris Lewis 2
  • Chrome OS 2
  • Diego Salas 2
  • Dillon Bly 2
  • Dori Reuveni 2
  • Jason Arbon 2
  • Jochen Wuttke 2
  • Kostya Serebryany 2
  • Marc Eaddy 2
  • Marko Ivanković 2
  • Max Kanat-Alexander 2
  • Mobile 2
  • Oliver Chang 2
  • Simon Stewart 2
  • Stefan Kennedy 2
  • Test Flakiness 2
  • Tony Voellm 2
  • WebRTC 2
  • Yvette Nameth 2
  • Zuri Kemp 2
  • Aaron Jacobs 1
  • Adam Porter 1
  • Adel Saoud 1
  • Alan Faulkner 1
  • Alex Eagle 1
  • Anantha Keesara 1
  • Antoine Picard 1
  • App Engine 1
  • Ari Shamash 1
  • Arif Sukoco 1
  • Benjamin Pick 1
  • Bob Nystrom 1
  • Bruce Leban 1
  • Carlos Arguelles 1
  • Carlos Israel Ortiz García 1
  • Cathal Weakliam 1
  • Christopher Semturs 1
  • Clay Murphy 1
  • Dan Shi 1
  • Dan Willemsen 1
  • Dave Chen 1
  • Dave Gladfelter 1
  • Derek Snyder 1
  • Diego Cavalcanti 1
  • Dmitry Vyukov 1
  • Eduardo Bravo Ortiz 1
  • Ekaterina Kamenskaya 1
  • Elliott Karpilovsky 1
  • Espresso 1
  • Google+ 1
  • Goran Petrovic 1
  • Goranka Bjedov 1
  • Hank Duan 1
  • Havard Rast Blok 1
  • Hongfei Ding 1
  • Jason Elbaum 1
  • Jason Huggins 1
  • Jay Han 1
  • Jeff Listfield 1
  • Jessica Tomechak 1
  • Jim Reardon 1
  • Joe Allan Muharsky 1
  • Joel Hynoski 1
  • John Micco 1
  • John Penix 1
  • Jonathan Rockway 1
  • Jonathan Velasquez 1
  • Josh Armour 1
  • Julie Ralph 1
  • Karin Lundberg 1
  • Kaue Silveira 1
  • Kevin Bourrillion 1
  • Kevin Graney 1
  • Kirkland 1
  • Kurt Alfred Kluever 1
  • Manjusha Parvathaneni 1
  • Marek Kiszkis 1
  • Mark Ivey 1
  • Mark Striebeck 1
  • Matt Lowrie 1
  • Meredith Whittaker 1
  • Michael Bachman 1
  • Michael Klepikov 1
  • Mike Aizatsky 1
  • Mike Wacker 1
  • Mona El Mahdy 1
  • Noel Yap 1
  • Patricia Legaspi 1
  • Peter Arrenbrecht 1
  • Peter Spragins 1
  • Phil Rollet 1
  • Pooja Gupta 1
  • Project Showcase 1
  • Radoslav Vasilev 1
  • Rajat Dewan 1
  • Rajat Jain 1
  • Rich Martin 1
  • Richard Bustamante 1
  • Roshan Sembacuttiaratchy 1
  • Ruslan Khamitov 1
  • Sean Jordan 1
  • Sharon Zhou 1
  • Siddartha Janga 1
  • Simran Basi 1
  • Stephen Ng 1
  • Tejas Shah 1
  • Test Analytics 1
  • Test Engineer 1
  • Tom O'Neill 1
  • Vojta Jína 1
  • iOS 1
  • mutation testing 1


Archive


  • ►  2022 (2)
    • ►  Feb (2)
  • ►  2021 (3)
    • ►  Jun (1)
    • ►  Apr (1)
    • ►  Mar (1)
  • ►  2020 (8)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  May (1)
  • ►  2019 (4)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Jul (1)
    • ►  Jan (1)
  • ►  2018 (7)
    • ►  Nov (1)
    • ►  Sep (1)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (1)
    • ►  Feb (1)
  • ►  2017 (17)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ▼  2016 (15)
    • ►  Dec (1)
    • ▼  Nov (2)
      • What Test Engineers do at Google: Building Test In...
      • Hackable Projects - Pillar 3: Infrastructure
    • ►  Oct (1)
    • ►  Sep (2)
    • ►  Aug (1)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (1)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2015 (14)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Aug (1)
    • ►  Jun (1)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (1)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2014 (24)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Sep (2)
    • ►  Aug (2)
    • ►  Jul (3)
    • ►  Jun (3)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2013 (16)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Jan (2)
  • ►  2012 (11)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (1)
    • ►  Aug (4)
  • ►  2011 (39)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (2)
    • ►  Aug (4)
    • ►  Jul (2)
    • ►  Jun (5)
    • ►  May (4)
    • ►  Apr (3)
    • ►  Mar (4)
    • ►  Feb (5)
    • ►  Jan (3)
  • ►  2010 (37)
    • ►  Dec (3)
    • ►  Nov (3)
    • ►  Oct (4)
    • ►  Sep (8)
    • ►  Aug (3)
    • ►  Jul (3)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (3)
    • ►  Mar (3)
    • ►  Feb (2)
    • ►  Jan (1)
  • ►  2009 (54)
    • ►  Dec (3)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (5)
    • ►  Aug (4)
    • ►  Jul (15)
    • ►  Jun (8)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (5)
    • ►  Jan (4)
  • ►  2008 (75)
    • ►  Dec (6)
    • ►  Nov (8)
    • ►  Oct (9)
    • ►  Sep (8)
    • ►  Aug (9)
    • ►  Jul (9)
    • ►  Jun (6)
    • ►  May (6)
    • ►  Apr (4)
    • ►  Mar (4)
    • ►  Feb (4)
    • ►  Jan (2)
  • ►  2007 (41)
    • ►  Oct (6)
    • ►  Sep (5)
    • ►  Aug (3)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (7)
    • ►  Mar (5)
    • ►  Feb (5)
    • ►  Jan (4)

Feed

follow us in feedly
  • Google
  • Privacy
  • Terms