Google Testing Blog: September 2016

Testing on the Toilet: What Makes a Good End-to-End Test?

Wednesday, September 21, 2016

by Adam Bender
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.End-to-end tests can catch bugs that manifest across your entire systemslower, more flaky, and more expensive to maintain

focus on aspects of your system that cannot be reliably evaluated with smaller tests

For each important use case, there should be one corresponding end-to-end test. This should include one test for each important class of error. The goal is the keep your total end-to-end count low.

Be prepared to allocate at least one week a quarter per test to keep your end-to-end tests stable in the face of issues like slow and flaky dependencies or minor UI changes.

Focus your efforts on verifying overall system behavior instead of specific implementation details; for example, when testing login behavior, verify that the process succeeds independent of the exact messages or visual layouts, which may change frequently.

Make your end-to-end test easy to debug by providing an overview-level log file, documenting common test failure modes, and preserving all relevant system state information (e.g.: screenshots, database snapshots, etc.).

System components that are owned by other teams may change unexpectedly, and break your tests. This increases overall maintenance cost, but can highlight incompatible changes

It may be more difficult to make an end-to-end test fully hermetic; leftover test data may alter future tests and/or production systems. Where possible keep your test data ephemeral.

An end-to-end test often necessitates multiple test doubles (fakes or stubs) for underlying dependencies; they can, however, have a high maintenance burden as they drift from the real implementations over time.

20 comments

What Test Engineers do at Google

Monday, September 12, 2016

by Matt Lowrie, Manjusha Parvathaneni, Benjamin Pick, and Jochen Wuttke Testing on the ToiletTest Certified ProgramGoogle Test Automation Conference (GTAC)code coverageperformance testingprevious blog post

Automate a manual verification process for product release candidates so developers have more time to respond to potential release-blocking issues.

Design and implement an automated way to track and surface Android battery usage to developers, so that they know immediately when a new feature will cause users drained batteries.

Quantify if a regenerated data set used by a product, which contains a billion entities, is better quality than the data set currently live in production.

Write an automated test suite that validates if content presented to a user is of an acceptable quality level based on their interests.

Read an engineering design proposal for a new feature and provide suggestions about how and where to build in testability.

Investigate correlated stack traces submitted by users through our feedback tracking system, and search the code base to find the correct owner for escalation.

Collaborate on determining the root cause of a production outage, then pinpoint tests that need to be added to prevent similar outages in the future.

Organize a task force to advise teams across the company about best practices when testing for accessibility.

GTAC

7 comments

Testing Blog

Testing on the Toilet: What Makes a Good End-to-End Test?

What Test Engineers do at Google

Labels

Archive

Feed