Google Testing Blog: 2016

Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software

Thursday, December 01, 2016

By Mike Aizatsky, Kostya Serebryany (Software Engineers, Dynamic Tools); Oliver Chang, Abhishek Arya (Security Engineers, Google Chrome); and Meredith Whittaker (Open Research Lead). We are happy to announce OSS-Fuzz, a new Beta program developed over the past years with the Core Infrastructure Initiative community. This program will provide continuous fuzzing for select core open source software.
Recentsecuritystories buffer overflow use-after-free fuzz testingAFLlibFuzzerfuzz various components of the Chrome browserSanitizerssometimesAddressSanitizerClusterFuzz Early successesFreeTypebillion deviceswasFreeType fuzzernew heap buffer overflowERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa READ of size 2 at 0x615000000ffa thread T0 SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds) #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31 notified fixed confirmedfull list Contributions and feedback are welcome150 bugsprojects~4 trillion test casesheretrackerherebest practicesGitHub

5 comments

What Test Engineers do at Google: Building Test Infrastructure

Friday, November 18, 2016

Author: Jochen WuttkeWhat Test Engineers do at GoogleRefurbishing legacy systems makes new tools necessaryflaky tests

Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code.

The infrastructure used for the end-to-end tests had no good way to create and inject fakes or mocks for these services. As a result, the tests had to run the large number of servers for all these external dependencies. This led to very large and brittle tests that our existing test execution infrastructure was not able to handle reliably.

Exploring solutionsRPC callsgraphtreemethod calls

Adoption

The new tests are as effective in finding bugs as the end-to-end tests are.

The new tests run in about 3 minutes instead of 30 minutes for the end-to-end tests.

The client side tests are 0% flaky. The verification tests are usually less flaky than the end-to-end tests, and never more.

12 comments

Hackable Projects - Pillar 3: Infrastructure

Thursday, November 10, 2016

By: Patrik Höglund
This is the third article in our series on Hackability; also see the first and second article.Code HealthDebuggability Build Systems Speed
Question: What’s a change you’d really like to see in our development tools?

“I feel like this question gets asked almost every time, and I always give the same answer:
I would like them to be faster.”
-- Ian Lance TaylorninjagoldBazel

Figure 1: “Cheetah chasing its prey” by Marlene Thyssen.

Build and test times longer than a handful of seconds cause many developers’ minds to wander, taking them out of the zone.

Excessive build or release times* makes tinkering and refactoring much harder. All developers have a threshold when they start hedging (e.g. “I’d like to remove this branch, but I don’t know if I’ll break the iOS build”) which causes refactoring to not happen.

* The worst I ever heard of was an OS that took 24 hours to build!

How do you actually make fast build systems? There are some suggestions in the first paragraph above, but the best general suggestion I can make is to have a few engineers on the project who deeply understand the build systems and have the time to continuously improve them. The main axes of improvement are:

Reduce the amount of code being compiled.
Replace tools with faster counterparts.
Increase processing power, maybe through parallelization or distributed systems.

Note that there is a big difference between full builds and incremental builds. Both should be as fast as possible, but incremental builds are by far the most important to optimize. The way you tackle the two is different. For instance, reducing the total number of source files will make a full build faster, but it may not make an incremental build faster.

To get faster incremental builds, in general, you need to make each source file as decoupled as possible from the rest of the code base. The less a change ripples through the codebase, the less work to do, right? See “Loose Coupling and Testability” in Pillar 1 for more on this subject. The exact mechanics of dependencies and interfaces depends on programming language - one of the hardest to get right is unsurprisingly C++, where you need to be disciplined with includes and forward declarations to get any kind of incremental build performance.

Build scripts and makefiles should be held to standards as high as the code itself. Technical debt and unnecessary dependencies have a tendency to accumulate in build scripts, because no one has the time to understand and fix them. Avoid this by addressing the technical debt as you go.

Continuous Integration and Presubmit Queues

You should build and run tests on all platforms you release on. For instance, if you release on all the major desktop platforms, but all your developers are on Linux, this becomes extra important. It’s bad for hackability to update the repo, build on Windows, and find that lots of stuff is broken. It’s even worse if broken changes start to stack on top of each other. I think we all know that terrible feeling: when you’re not sure your change is the one that broke things.

At a minimum, you should build and test on all platforms, but it’s even better if you do it in presubmit. The Chromium submit queue does this. It has developed over the years so that a normal patch builds and tests on about 30 different build configurations before commit. This is necessary for the 400-patches-per-day velocity of the Chrome project. Most projects obviously don’t have to go that far. Chromium’s infrastructure is based off BuildBot, but there are many other job scheduling systems depending on your needs.

Figure 2: How a Chromium patch is tested.

As we discussed in Build Systems, speed and correctness are critical here. It takes a lot of ongoing work to keep build, tests, and presubmits fast and free of flakes. You should never accept flakes, since developers very quickly lose trust in flaky tests and systems. Tooling can help a bit with this; for instance, see the Chromium flakiness dashboard.

Test Speed

Speed is a feature, and this is particularly true for developer infrastructure. In general, the longer a test takes to execute, the less valuable it is. My rule of thumb is: if it takes more than a minute to execute, its value is greatly diminished. There are of course some exceptions, such as soak tests or certain performance tests.

Figure 3. Test value.

If you have tests that are slower than 60 seconds, they better be incredibly reliable and easily debuggable. A flaky test that takes several minutes to execute often has negative value because it slows down all work in the code it covers. You probably want to build better integration tests on lower levels instead, so you can make them faster and more reliable.

If you have many engineers on a project, reducing the time to run the tests can have a big impact. This is one reason why it’s great to have SETIs or the equivalent. There are many things you can do to improve test speed:

Sharding and parallelization. Add more machines to your continuous build as your test set or number of developers grows.
Continuously measure how long it takes to run one build+test cycle in your continuous build, and have someone take action when it gets slower.
Remove tests that don’t pull their weight. If a test is really slow, it’s often because of poorly written wait conditions or because the test bites off more than it can chew (maybe that unit test doesn’t have to process 15000 audio frames, maybe 50 is enough).
If you have tests that bring up a local server stack, for instance inter-server integration tests, making your servers boot faster is going to make the tests faster as well. Faster production code is faster to test! See Running on Localhost, in Pillar 2 for more on local server stacks.

Workflow Speed

We’ve talked about fast builds and tests, but the core developer workflows are also important, of course. Chromium undertook multi-year project to switch from Subversion to Git, partly because Subversion was becoming too slow. You need to keep track of your core workflows as your project grows. Your version control system may work fine for years, but become too slow once the project becomes big enough. Bug search and management must also be robust and fast, since this is generally systems developers touch every day.

Release Often

It aids hackability to deploy to real users as fast as possible. No matter how good your product's tests are, there's always a risk that there's something you haven't thought of. If you’re building a service or web site, you should aim to deploy multiple times per week. For client projects, Chrome’s six-weeks cycle is a good goal to aim for.

You should invest in infrastructure and tests that give you the confidence to do this - you don’t want to push something that’s broken. Your developers will thank you for it, since it makes their jobs so much easier. By releasing often, you mitigate risk, and developers will rush less to hack late changes in the release (since they know the next release isn’t far off).

Easy Reverts

If you look in the commit log for the Chromium project, you will see that a significant percentage of the commits are reverts of a previous commits. In Chromium, bad commits quickly become costly because they impede other engineers, and the high velocity can cause good changes to stack onto bad changes.

Figure 4: Chromium’s revert button.

This is why the policy is “revert first, ask questions later”. I believe a revert-first policy is good for small projects as well, since it creates a clear expectation that the product/tools/dev environment should be working at all times (and if it doesn’t, a recent change should probably be reverted).

It has a wonderful effect when a revert is simple to make. You can suddenly make speculative reverts if a test went flaky or a performance test regressed. It follows that if a patch is easy to revert, so is the inverse (i.e. reverting the revert or relanding the patch). So if you were wrong and that patch wasn’t guilty after all, it’s simple to re-land it again and try reverting another patch. Developers might initially balk at this (because it can’t possibly be their patch, right?), but they usually come around when they realize the benefits.

For many projects, a revert can simply be

git revert 9fbadbeef

git push origin master

If your project (wisely) involves code review, it will behoove you to add something like Chromium’s revert button that I mentioned above. The revert button will create a special patch that bypasses review and tests (since we can assume a clean revert takes us back to a more stable state rather than the opposite). See Pillar 1 for more on code review and its benefits.

For some projects, reverts are going to be harder, especially if you have a slow or laborious release process. Even if you release often, you could still have problems if a revert involves state migrations in your live services (for instance when rolling back a database schema change). You need to have a strategy to deal with such state changes.

Reverts must always put you back to safer ground, and everyone must be confident they can safely revert. If not, you run the risk of massive fire drills and lost user trust if a bad patch makes it through the tests and you can’t revert it.

Performance Tests: Measure Everything

Is it critical that your app starts up within a second? Should your app always render at 60 fps when it’s scrolled up or down? Should your web server always serve a response within 100 ms? Should your mobile app be smaller than 8 MB? If so, you should make a performance test for that. Performance tests aid hackability since developers can quickly see how their change affects performance and thus prevent performance regressions from making it into the wild.

You should run your automated performance tests on the same device; all devices are different and this will reflect in the numbers. This is fairly straightforward if you have a decent continuous integration system that runs tests sequentially on a known set of worker machines. It’s harder if you need to run on physical phones or tablets, but it can be done.

A test can be as simple as invoking a particular algorithm and measuring the time it takes to execute it (median and 90th percentile, say, over N runs).

Figure 5: A VP8-in-WebRTC regression (bug) and its fix displayed in a Catapult dashboard.

Write your test so it outputs performance numbers you care about. But what to do with those numbers? Fortunately, Chrome’s performance test framework has been open-sourced, which means you can set up a dashboard, with automatic regression monitoring, with minimal effort. The test framework also includes the powerful Telemetry framework which can run actions on web pages and Android apps and report performance results. Telemetry and Catapult are battle-tested by use in the Chromium project and are capable of running on a wide set of devices, while getting the maximum of usable performance data out of the devices.

Sources

Figure 1: By Malene Thyssen (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

3 comments

Hackable Projects - Pillar 2: Debuggability

Tuesday, October 11, 2016

By: Patrik Höglund
This is the second article in our series on Hackability; also see the first article.
“Deep into that darkness peering, long I stood there, wondering, fearing, doubting, dreaming dreams no mortal ever dared to dream before.” -- Edgar Allan Poe

Figure 1: computer log entry from the Mark II, with a moth taped to the page. Running on Localhost Read more on the Testing Blog: Hermetic Servers by Chaitali Narla and Diego Salas server stack on localhostb_serv --port=1234 --db=/tmp/fakedb c_serv --port=1235 a_serv --b_spec=localhost:1234 --c_spec=localhost:1235 hermetic tests, Debugging Mobile AppsRobolectric unit testsEspresso functional testsxcTestsEarl Grey When Debugging gets Trickyhard real-time systemsdebugging flags and instructionschrome --renderer-cmd-prefix='xterm -title renderer -e gdb --args' Proper Logging Read more on the Testing Blog: Optimal Logging by Anthony Vallone ChromiumThis article Monitoring and StatisticsStackdriver for Google Cloud System Under Test (SUT) SizeThis article

Figure 2. The smaller the SUT, the more valuable the test. Sources (Continue to Pillar 3: Infrastructure)

No comments

Testing on the Toilet: What Makes a Good End-to-End Test?

Wednesday, September 21, 2016

by Adam Bender
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.End-to-end tests can catch bugs that manifest across your entire systemslower, more flaky, and more expensive to maintain

focus on aspects of your system that cannot be reliably evaluated with smaller tests

For each important use case, there should be one corresponding end-to-end test. This should include one test for each important class of error. The goal is the keep your total end-to-end count low.

Be prepared to allocate at least one week a quarter per test to keep your end-to-end tests stable in the face of issues like slow and flaky dependencies or minor UI changes.

Focus your efforts on verifying overall system behavior instead of specific implementation details; for example, when testing login behavior, verify that the process succeeds independent of the exact messages or visual layouts, which may change frequently.

Make your end-to-end test easy to debug by providing an overview-level log file, documenting common test failure modes, and preserving all relevant system state information (e.g.: screenshots, database snapshots, etc.).

System components that are owned by other teams may change unexpectedly, and break your tests. This increases overall maintenance cost, but can highlight incompatible changes

It may be more difficult to make an end-to-end test fully hermetic; leftover test data may alter future tests and/or production systems. Where possible keep your test data ephemeral.

An end-to-end test often necessitates multiple test doubles (fakes or stubs) for underlying dependencies; they can, however, have a high maintenance burden as they drift from the real implementations over time.

20 comments

What Test Engineers do at Google

Monday, September 12, 2016

by Matt Lowrie, Manjusha Parvathaneni, Benjamin Pick, and Jochen Wuttke Testing on the ToiletTest Certified ProgramGoogle Test Automation Conference (GTAC)code coverageperformance testingprevious blog post

Automate a manual verification process for product release candidates so developers have more time to respond to potential release-blocking issues.

Design and implement an automated way to track and surface Android battery usage to developers, so that they know immediately when a new feature will cause users drained batteries.

Quantify if a regenerated data set used by a product, which contains a billion entities, is better quality than the data set currently live in production.

Write an automated test suite that validates if content presented to a user is of an acceptable quality level based on their interests.

Read an engineering design proposal for a new feature and provide suggestions about how and where to build in testability.

Investigate correlated stack traces submitted by users through our feedback tracking system, and search the code base to find the correct owner for escalation.

Collaborate on determining the root cause of a production outage, then pinpoint tests that need to be added to prevent similar outages in the future.

Organize a task force to advise teams across the company about best practices when testing for accessibility.

GTAC

7 comments

Hackable Projects - Pillar 1: Code Health

Thursday, August 18, 2016

By: Patrik Höglund IntroductionAccording to Wiktionary Adjective
hackable ‎(comparative more hackable, superlative most hackable)

(computing) That can be hacked or broken into; insecure, vulnerable.
That lends itself to hacking (technical tinkering and modification); moddable.

mainfocus for SETIs at Google In Practice

Developing is easy

Fast build

Good, fast tests

Clean code

Easy running + debugging

One-click rollbacks

Broken HEAD (tip-of-tree)

Slow presubmit (i.e. checks running before submit)

Builds take hours

Incremental build/link > 30s

Flakytests

Can’t attach debugger

Logs full of uninteresting information

The Three Pillars of Hackability

Pillar 1: Code Health “I found Rome a city of bricks, and left it a city of marble.”
-- Augustus Testsevenbe a hindrance

Figure 1: the Testing Pyramid. Readable Code and Code Reviewstyle guidecode reviewGerritSOLID principles Presubmit Testing and Lintgofmtclang-formatexample from Chrome$ git cl upload Error: the media/audio directory requires formatting. Please run git cl format media/audio.Enforce it with a checkAdd lintingVHDLFPGA Single Branch And Reducing Riskfeature flagsThisarticle Loose Coupling and Testabilitychat Aggressively Reduce Technical Debtdependency enforcement (Continue to Pillar 2: Debuggability)

8 comments

The Inquiry Method for Test Planning

Monday, June 06, 2016

by Anthony Vallone updated: July 2016 Creating a test plan is often a complex undertaking. An ideal test plan is accomplished by applying basic principles of cost-benefit analysis and risk analysis, optimally balancing these software development factors:

Implementation cost: The time and complexity of implementing testable features and automated tests for specific scenarios will vary, and this affects short-term development cost.
Maintenance cost: Some tests or test plans may vary from easy to difficult to maintain, and this affects long-term development cost. When manual testing is chosen, this also adds to long-term cost.
Monetary cost: Some test approaches may require billed resources.
Benefit: Tests are capable of preventing issues and aiding productivity by varying degrees. Also, the earlier they can catch problems in the development life-cycle, the greater the benefit.
Risk: The probability of failure scenarios may vary from rare to likely, and their consequences may vary from minor nuisance to catastrophic.

Effectively balancing these factors in a plan depends heavily on project criticality, implementation details, resources available, and team opinions. Many projects can achieve outstanding coverage with high-benefit, low-cost unit tests, but they may need to weigh options for larger tests and complex corner cases. Mission critical projects must minimize risk as much as possible, so they will accept higher costs and invest heavily in rigorous testing at all levels.
This guide puts the onus on the reader to find the right balance for their project. Also, it does not provide a test plan template, because templates are often too generic or too specific and quickly become outdated. Instead, it focuses on selecting the best content when writing a test plan.

Test plan vs. strategy

Before proceeding, two common methods for defining test plans need to be clarified:

Single test plan: Some projects have a single "test plan" that describes all implemented and planned testing for the project.
Single test strategy and many plans: Some projects have a "test strategy" document as well as many smaller "test plan" documents. Strategies typically cover the overall test approach and goals, while plans cover specific features or project updates.

Either of these may be embedded in and integrated with project design documents. Both of these methods work well, so choose whichever makes sense for your project. Generally speaking, stable projects benefit from a single plan, whereas rapidly changing projects are best served by infrequently changed strategies and frequently added plans.

For the purpose of this guide, I will refer to both test document types simply as "test plans”. If you have multiple documents, just apply the advice below to your document aggregation.

Content selection

A good approach to creating content for your test plan is to start by listing all questions that need answers. The lists below provide a comprehensive collection of important questions that may or may not apply to your project. Go through the lists and select all that apply. By answering these questions, you will form the contents for your test plan, and you should structure your plan around the chosen content in any format your team prefers. Be sure to balance the factors as mentioned above when making decisions.

Prerequisites

Do you need a test plan? If there is no project design document or a clear vision for the product, it may be too early to write a test plan.
Has testability been considered in the project design? Before a project gets too far into implementation, all scenarios must be designed as testable, preferably via automation. Both project design documents and test plans should comment on testability as needed.
Will you keep the plan up-to-date? If so, be careful about adding too much detail, otherwise it may be difficult to maintain the plan.
Does this quality effort overlap with other teams? If so, how have you deduplicated the work?

Risk

Are there any significant project risks, and how will you mitigate them? Consider:
- Injury to people or animals
- Security and integrity of user data
- User privacy
- Security of company systems
- Hardware or property damage
- Legal and compliance issues
- Exposure of confidential or sensitive data
- Data loss or corruption
- Revenue loss
- Unrecoverable scenarios
- SLAs
- Performance requirements
- Misinforming users
- Impact to other projects
- Impact from other projects
- Impact to company’s public image
- Loss of productivity
What are the project’s technical vulnerabilities? Consider:
- Features or components known to be hacky, fragile, or in great need of refactoring
- Dependencies or platforms that frequently cause issues
- Possibility for users to cause harm to the system
- Trends seen in past issues

Coverage

Tooling and Infrastructure

Do you need new test frameworks? If so, describe these or add design links in the plan.
Do you need a new test lab setup? If so, describe these or add design links in the plan.
If your project offers a service to other projects, are you providing test tools to those users? Consider providing mocks, fakes, and/or reliable staged servers for users trying to test their integration with your system.
For end-to-end testing, how will test infrastructure, systems under test, and other dependencies be managed? How will they be deployed? How will persistence be set-up/torn-down? How will you handle required migrations from one datacenter to another?
Do you need tools to help debug system or test failures? You may be able to use existing tools, or you may need to develop new ones.

Process

Are there test schedule requirements? What time commitments have been made, which tests will be in place (or test feedback provided) by what dates? Are some tests important to deliver before others?
How are builds and tests run continuously? Most small tests will be run by continuous integration tools, but large tests may need a different approach. Alternatively, you may opt for running large tests as-needed.
How will build and test results be reported and monitored?
- Do you have a team rotation to monitor continuous integration?
- Large tests might require monitoring by someone with expertise.
- Do you need a dashboard for test results and other project health indicators?
- Who will get email alerts and how?
- Will the person monitoring tests simply use verbal communication to the team?
How are tests used when releasing?
- Are they run explicitly against the release candidate, or does the release process depend only on continuous test results?
- If system components and dependencies are released independently, are tests run for each type of release?
- Will a "release blocker" bug stop the release manager(s) from actually releasing? Is there an agreement on what are the release blocking criteria?
- When performing canary releases (aka % rollouts), how will progress be monitored and tested?
How will external users report bugs? Consider feedback links or other similar tools to collect and cluster reports.
How does bug triage work? Consider labels or categories for bugs in order for them to land in a triage bucket. Also make sure the teams responsible for filing and or creating the bug report template are aware of this. Are you using one bug tracker or do you need to setup some automatic or manual import routine?
Do you have a policy for submitting new tests before closing bugs that could have been caught?
How are tests used for unsubmitted changes? If anyone can run all tests against any experimental build (a good thing), consider providing a howto.
How can team members create and/or debug tests? Consider providing a howto.

Utility

Who are the test plan readers? Some test plans are only read by a few people, while others are read by many. At a minimum, you should consider getting a review from all stakeholders (project managers, tech leads, feature owners). When writing the plan, be sure to understand the expected readers, provide them with enough background to understand the plan, and answer all questions you think they will have - even if your answer is that you don’t have an answer yet. Also consider adding contacts for the test plan, so any reader can get more information.
How can readers review the actual test cases? Manual cases might be in a test case management tool, in a separate document, or included in the test plan. Consider providing links to directories containing automated test cases.
Do you need traceability between requirements, features, and tests?
Do you have any general product health or quality goals and how will you measure success? Consider:
- Release cadence
- Number of bugs caught by users in production
- Number of bugs caught in release testing
- Number of open bugs over time
- Code coverage
- Cost of manual testing
- Difficulty of creating new tests

23 comments

GTAC 2016 Registration Deadline Extended

Thursday, June 02, 2016

by Sonal Shah on behalf of the GTAC CommitteeJune 1, 2016June 15, 2016June 15, 2016July 15, 2016August 15, 2016August 29, 2016thisthisGTAC websitegtac2016@google.com

No comments

Flaky Tests at Google and How We Mitigate Them

Friday, May 27, 2016

by John Miccoroot causesthis article

A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness. This prevents it from becoming a problem for developers, but could easily mask a real race condition or some other bug in the code being tested.

Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.

We have a new team dedicated to providing accurate and timely information about test flakiness to help developers and build monitors so that they know whether they are being harmed by test flakiness.

As we analyze the data from flaky test executions, we are seeing promising correlations with features that should enable us to identify a flaky result accurately without re-running the test.

36 comments

GTAC Diversity Scholarship

Wednesday, May 04, 2016

by Lesley Katzen on behalf of the GTAC Diversity Committee

Be 18 years of age or older.

Be from a traditionally underrepresented group in technology.

Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing.

Be able to attend core dates of GTAC, November 15th - 16th 2016 in Sunnyvale, CA.

To apply:formJune 1stJune 30thWhat the scholarship covers:https://www.google.com/events/policy/anti-harassmentpolicy.html

No comments

GTAC 2016 Registration is now open!

Wednesday, May 04, 2016

by Sonal Shah on behalf of the GTAC CommitteeGoogle Sunnyvale officeSpeakersApplicationcomplete this formDeadlineJune 1st, 2016CostMore informationhttps://developers.google.com/google-test-automation-conference/2016/faq

3 comments

GTAC 2016 - Save the Date

Friday, April 08, 2016

by Sonal Shah on behalf of the GTAC Committee

We are pleased to announce that the tenth GTAC (Google Test Automation Conference) will be held on Google’s campus in Sunnyvale (California, USA) on Tuesday and Wednesday, November 15th and 16th, 2016.

Based on feedback from the last GTAC (2015) and the increasing demand every year, we have decided to keep GTAC on a fall schedule. This schedule is a change from what we previously announced.

The schedule for the next few months is:

May 1, 2016 - Registration opens for speakers and attendees.

~~June 1, 2016~~ June 15, 2016 - Registration closes for speaker and attendee submissions.

~~June 30, 2016~~ July 15, 2016 - Selected attendees will be notified.

~~August 15, 2016~~ August 29, 2016 - Selected speakers will be notified.

November 14, 2016 - Rehearsal day for speakers.

November 15-16, 2016 - GTAC 2016!

As part of our efforts to increase diversity of speakers and attendees at GTAC, we will be offering travel scholarships for selected applicants from traditionally underrepresented groups in technology.

Stay tuned to this blog and the GTAC website for information about attending or presenting at GTAC. Please do not hesitate to contact gtac2016@google.com if you have any questions. We look forward to seeing you there!

7 comments

From QA to Engineering Productivity

Tuesday, March 22, 2016

By Ari Shamash

Test Engineers (TEs) -- tested new products and systems integration

Release Engineers (REs) -- pushed bits into production

Site Reliability Engineers (SREs) -- managed systems and data centers 24x7.

longer and longerfix themautomation

Test Engineers (TEs) -- With their deep product knowledge and test/quality domain expertise, TEs focused on what should be tested.

Software Engineers in Test (SETs) -- Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.

Automated tests became more efficient and deterministic (e.g. by improving runtimes, eliminating sources of flakiness, etc.)

Metrics driven engineering proliferated (e.g. improving code and feature coverage led to higher quality products).

there was much rejoicingwebdriverimprovementsprotractorespressoEarlGreymartian proxykarmaGoogleTestconferencessoftware engineerspublished articlesTest-Driven Developmentother phases in the development cycle became the bottleneck

Extending IDEs to make writing and reviewing code easier, shortening the “write code” cycle

Automating release verification, shortening the “release code” cycle.

Automating real time production system log verification and anomaly detection, helping automate production monitoring.

Automating measurement of developer productivity, helping understand what’s working and what isn’t.

We empowered the SETs to choose a new titleSETITE

33 comments

Testing Blog

Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software

What Test Engineers do at Google: Building Test Infrastructure

Hackable Projects - Pillar 3: Infrastructure

Hackable Projects - Pillar 2: Debuggability

Testing on the Toilet: What Makes a Good End-to-End Test?

What Test Engineers do at Google

Hackable Projects - Pillar 1: Code Health

The Inquiry Method for Test Planning

GTAC 2016 Registration Deadline Extended

Flaky Tests at Google and How We Mitigate Them

GTAC Diversity Scholarship

GTAC 2016 Registration is now open!

GTAC 2016 - Save the Date

From QA to Engineering Productivity

Labels

Archive

Feed