Google Testing Blog

How Google Tests Software - Part One

Tuesday, January 25, 2011

This is the first in a series of posts on this topic.and so on and so forthproduct teamservices teamEmbedded engineersmy GTAC talk

12 comments

Google Innovation and The Pretotyping Manifesto

Thursday, January 20, 2011

I'll be speaking very soon about innovation and The Pretotyping Manifesto (note: not prototyping). It's a concept that works well for any type of engineering, testing, or idea. Here's a brief preview...
The talk starts off by discussing the odds against innovators and how the deck is stacked against you from the start. Most engineers begin with trying to come up with THE killer idea. But you quickly realize that ideas are cheap. We all believe that our own ideas are good. As a matter of fact, most of us LOVE our own ideas to the point where it clouds our judgement. In parallel, there is a strong desire to jump in and build something after a great idea has been identified. Sometimes swarms of well intentioned people join in and "help." Quickly innovators can find themselves in the weeds or feeling disconnected from their original dream. So, what can be done? The idea is to use pretotyping and to focus on techniques that allow you to: Build the right it vs. Build it right.
Last time I did this talk, it was called: "the best keynote of any tech conference ever!"
I'm looking forward to seeing some of you the week of March7th when I'll be in London. In addition to dropping into the Google office, I'll be speaking at QCon London. If you want to attend (at a cheaper rate than normal) here are the details:

London event March 7-11, 2011.
My personal "promotion code" will save you £100 if they enter this code during registration.
Promotion code is: COPE100

Hope to see you in London,

Patrick Copeland

Senior Engineering Director, Google

1 comment

New Year's Resolutions

Tuesday, January 04, 2011

Year of the User1. I will listen to users more and developers less.
2. I will push all implementation testing issues to developers.
is3. I will endeavor to tie together all user-oriented testing.

7 comments

GTAC #5: videos, slides, abstracts

Thursday, December 30, 2010

The published content for this year's GTAC is finally available. You can find the links to all the material below and also at the GTAC site. Other interesting artifacts can be found via Google Search or Twitter or Facebook..
On behalf of the Committee and Google I want to thank all the speakers, attendees and volunteers who made this event a great professional engagement. Some moments of the conference are captured in photos.

Looking forward to next year’s GTAC in the city of Google’s headquarters.

Happy Holidays.
Sujay Sahni for the GTAC 2010 Committee

Day 1

Welcome and Opening Remarks
Sujay Sahni, Google Inc. & GTAC Committee Chair
video slides

Day 1 Opening Keynote
What Testability Tells us About the Software Performance Envelope
Robert Victor Binder, Founder and CEO, mVerify
video slides abstract

Twist, a next generation functional testing tool for building and evolving test suites
Vivek Prahlad, ThoughtWorks
video slides abstract

The Future of Front-End Testing
Greg Dennis & Simon Stewart, Google Inc.
video slides abstract

Lightning Talks/Interactive Session
GTAC Attendees
video slides

Testivus on Testability
Alberto Savoia, Google Inc.
video slides

Lessons Learned from Testability Failures
Esteban Manchado Velázquez, Opera Software ASA
video slides abstract

Git Bisect and Testing
Christian Couder
video slides abstract

Flexible Design? Testable Design? You Don’t Have To Choose!
Russ Rufer & Tracy Bialik, Google Inc.
video slides abstract

Day 2

Day 2 Opening Keynote
Automatically Generating Test Data for Web Applications
Jeff Offutt, Professor of Software Engineering, Volgenau School of Information and Technology, George Mason University
video slides abstract

Early Test Feedback by Test Prioritisation
Shin Yoo, University College London & Robert Nilsson, Google Inc.
video slides abstract

Measuring and Monitoring Experience in Interactive Streaming Applications
Shreeshankar Chatterjee, Adobe Systems India
video slides abstract

Crowd Source Testing, Mozilla Community Style
Matt Evans, Mozilla
video slides abstract

Lightning Talks/Interactive Session
GTAC Attendees
video slides

Closing Keynote - Turning Quality on its Head
James Whittaker, Engineering Director, Google Inc.
video slides abstract

Closing Panel Discussion
GTAC Attendees
video

Closing Remarks
Sujay Sahni, Google Inc. & GTAC Committee Chair
video slides

No comments

Test Sizes

Monday, December 13, 2010

by Simon Stewart
What do you call a test that tests your application through its UI? An end-to-end test? A functional test? A system test? A selenium test? I’ve heard all them, and more. I reckon you have too. Tests running against less of the stack? The same equally frustrating inconsistency. Just what, exactly, is an integration test? A unit test? How do we name these things?

Gah!

It can be hard to persuade your own team to settle on a shared understanding of what each name actually means. The challenge increases when you encounter people from another team or project who are using different terms than you. More (less?) amusingly, you and that other team may be using the same term for different test types. “Oh! That kind of integration test?” Two teams separated by a common jargon.

Double gah!

The problem with naming test types is that the names tend to rely on a shared understanding of what a particular phrase means. That leaves plenty of room for fuzzy definitions and confusion. There has to be a better way. Personally, I like what we do here at Google and I thought I’d share that with you.

Googlers like to make decisions based on data, rather than just relying on gut instinct or something that can’t be measured and assessed. Over time we’ve come to agree on a set of data-driven naming conventions for our tests. We call them “Small”, “Medium” and “Large” tests. They differ like so:

Feature	Small	Medium	Large
Network access	No	localhost only	Yes
Database	No	Yes	Yes
File system access	No	Yes	Yes
Use external systems	No	Discouraged	Yes
Multiple threads	No	Yes	Yes
Sleep statements	No	Yes	Yes
System properties	No	Yes	Yes
Time limit (seconds)	60	300	900+

Going into the pros and cons of each type of test is a whole other blog entry, but it should be obvious that each type of test fulfills a specific role. It should also be obvious that this doesn’t cover every possible type of test that might be run, but it certainly covers most of the major types that a project will run.

A Small test equates neatly to a unit test, a Large test to an end-to-end or system test and a Medium test to tests that ensure that two tiers in an application can communicate properly (often called an integration test).

The major advantage that these test definitions have is that it’s possible to get the tests to police these limits. For example, in Java it’s easy to install a security manager for use with a test suite (perhaps using @BeforeClass) that is configured for a particular test size and disallows certain activities. Because we use a simple Java annotation to indicate the size of the test (with no annotation meaning it’s a Small test as that’s the common case), it’s a breeze to collect all the tests of a particular size into a test suite.

We place other constraints, which are harder to define, around the tests. These include a requirement that tests can be run in any order (they frequently are!) which in turn means that tests need high isolation --- you can’t rely on some other test leaving data behind. That’s sometimes inconvenient, but it makes it significantly easier to run our tests in parallel. The end result: we can build test suites easily, and run them consistently and as as fast as possible.

Not “gah!” at all.

12 comments

Testing Blog

How Google Tests Software - Part One

Google Innovation and The Pretotyping Manifesto

New Year's Resolutions

GTAC #5: videos, slides, abstracts

Test Sizes

Labels

Archive

Feed