Testing Blog
ThreadSanitizer: Slaughtering Data Races
Monday, June 30, 2014
by Dmitry Vyukov, Synchronization Lookout, Google, Moscow
Hello,
I work in the Dynamic Testing Tools team at Google. Our team develops tools like
AddressSanitizer
,
MemorySanitizer
and
ThreadSanitizer
which find various kinds of bugs. In this blog post I want to tell you about ThreadSanitizer, a fast data race detector for C++ and Go programs.
First of all, what is a
data race
? A data race occurs when two threads access the same variable concurrently, and at least one of the accesses attempts is a write. Most programming languages provide very weak guarantees, or no guarantees at all, for programs with data races. For example, in C++ absolutely any data race renders the behavior of the whole program as completely undefined (yes, it can suddenly format the hard drive). Data races are common in concurrent programs, and they are notoriously hard to debug and localize. A typical manifestation of a data race is when a program occasionally crashes with obscure symptoms, the symptoms are different each time and do not point to any particular place in the source code. Such bugs can take several months of debugging without particular success, since typical debugging techniques do not work. Fortunately, ThreadSanitizer can catch most data races in the blink of an eye. See Chromium
issue 15577
for an example of such a data race and
issue 18488
for the resolution.
Due to the complex nature of bugs caught by ThreadSanitizer, we don't suggest waiting until product release validation to use the tool. For example, in Google, we've made our tools easily accessible to programmers during development, so that anyone can use the tool for testing if they suspect that new code might introduce a race. For both Chromium and Google internal server codebase, we run unit tests that use the tool continuously. This catches many regressions instantly. The Chromium project has recently started using ThreadSanitizer on
ClusterFuzz
, a large scale fuzzing system. Finally, some teams also set up periodic end-to-end testing with ThreadSanitizer under a realistic workload, which proves to be extremely valuable. When races are found by the tool, our team has zero tolerance for races and does not consider any race to be benign, as even the most benign races can lead to
memory corruption
.
Our tools are
dynamic
(as opposed to
static tools
). This means that they do not merely "look" at the code and try to surmise where bugs can be; instead they they
instrument
the binary at build time and then analyze dynamic behavior of the program to catch it red-handed. This approach has its pros and cons. On one hand, the tool does not have any false positives, thus it does not bother a developer with something that is not a bug. On the other hand, in order to catch a bug, the test must expose a bug -- the racing data access attempts must be executed in different threads. This requires writing good multi-threaded tests and makes end-to-end testing especially effective.
As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and
more
. It also natively understands atomic operations and thus can find bugs in
lock-free
algorithms (see e.g.
this bug
in the V8 concurrent garbage collector).
The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a
-fsanitize=thread
flag during compilation and linking. For Go programs, you simply need to add a
-race flag
to the go tool (supported on Linux, Mac and Windows).
Interestingly, after integrating the tool into compilers, we've found some bugs in the compilers themselves. For example, LLVM was
illegally widening stores
, which can introduce very harmful data races into otherwise correct programs. And GCC was injecting
unsafe code
for initialization of function static variables. Among our other trophies are more than
a thousand bugs
in Chromium, Firefox, the Go standard library, WebRTC, OpenSSL, and of course in our internal projects.
So what are you waiting for? You know what to do!
No comments
GTAC 2014: Call for Proposals & Attendance
Monday, June 16, 2014
Posted by
Anthony Vallone
on behalf of the GTAC Committee
The application process is now open for presentation proposals and attendance for GTAC (Google Test Automation Conference) (
see initial announcement
) to be held at the
Google Kirkland office (near Seattle, WA)
on October 28 - 29th, 2014.
GTAC will be streamed live on YouTube again this year, so even if you can’t attend, you’ll be able to watch the conference from your computer.
Speakers
Presentations are targeted at student, academic, and experienced engineers working on test automation. Full presentations and lightning talks are 45 minutes and 15 minutes respectively. Speakers should be prepared for a question and answer session following their presentation.
Application
For presentation proposals and/or attendance,
complete this form
. We will be selecting about 300 applicants for the event.
Deadline
The due date for both presentation and attendance applications is July 28, 2014.
Fees
There are no registration fees, and we will send out detailed registration instructions to each invited applicant. Meals will be provided, but speakers and attendees must arrange and pay for their own travel and accommodations.
Update
:
Our
contact
email was bouncing - this is now fixed.
7 comments
GTAC 2014 Coming to Seattle/Kirkland in October
Wednesday, June 04, 2014
Posted by
Anthony Vallone
on behalf of the GTAC Committee
If you're looking for a place to discuss the latest innovations in test automation, then charge your tablets and pack your
gumboots
- the eighth
GTAC
(Google Test Automation Conference) will be held on October 28-29, 2014 at Google Kirkland! The Kirkland office is part of the
Seattle/Kirkland campus
in beautiful Washington state. This campus forms our third largest engineering office in the USA.
GTAC is a periodic conference hosted by Google, bringing together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It’s a great opportunity to present, learn, and challenge modern testing technologies and strategies.
You can browse the presentation abstracts, slides, and videos from last year on the
GTAC 2013 page
.
Stay tuned to this blog and the GTAC website for application information and opportunities to present at GTAC. Subscribing to this blog is the best way to get notified. We're looking forward to seeing you there!
2 comments
Testing on the Toilet: Risk-Driven Testing
Friday, May 30, 2014
by Peter Arrenbrecht
This article was adapted from a
Google Testing on the Toilet
(TotT) episode. You can download a
printer-friendly version
of this TotT episode and post it in your office.
We are all conditioned to write tests
as we code: unit, functional, UI—the whole shebang. We are professionals, after all. Many of us like how small tests let us work quickly, and how larger tests inspire safety and closure. Or we may just anticipate flak during review. We are so used to these tests that often
we no longer question why we write them
. This can be wasteful and dangerous.
Tests are a means to an end:
To
reduce the key risks
of a project, and to
get the biggest bang for the buck
. This bang may not always come from the tests that standard practice has you write, or not even from tests at all.
Two examples:
“We built a new debugging aid. We wrote unit, integration, and UI tests. We were ready to launch.”
Outstanding practice.
Missing the mark.
Our key risks were that we'd corrupt our data or bring down our servers for the sake of a debugging aid. None of the tests addressed this, but they gave a false sense of safety and “being done”.
We stopped the launch.
“We wanted to turn down a feature, so we needed to alert affected users. Again we had unit and integration tests, and even one expensive end-to-end test.”
Standard practice.
Wasted effort.
The alert was so critical it actually needed end-to-end coverage for all scenarios. But it would be live for only three releases. The cheapest effective test? Manual testing before each release.
A Better Approach: Risks First
For every project or feature,
think about testing
. Brainstorm your key risks and your best options to reduce them.
Do this at the start
so you don't waste effort and can adapt your design.
Write them down
as a QA design so you can point to it in reviews and discussions.
To be sure,
standard practice remains a good idea in most cases
(hence it’s standard). Small tests are cheap and speed up coding and maintenance, and larger tests safeguard core use-cases and integration.
Just remember
: Your tests are a means.
The bang is what counts
. It’s your job to
maximize it
.
1 comment
Testing on the Toilet: Effective Testing
Wednesday, May 07, 2014
by Rich Martin, Zurich
This article was adapted from a
Google Testing on the Toilet
(TotT) episode. You can download a
printer-friendly version
of this TotT episode and post it in your office.
Whether we are writing an individual unit test or designing a product’s entire testing process, it is important to take a step back and think about
how effective are our tests at detecting and reporting bugs in our code
. To be effective, there are
three important qualities
that every test should try to maximize:
Fidelity
When the code under test is broken, the test fails.
A high-fidelity test is one which is very sensitive to defects in the code under test
, helping to prevent bugs from creeping into the code.
Maximize fidelity by ensuring that your tests cover all the paths through your code and include all relevant assertions on the expected state.
Resilience
A test shouldn’t fail if the code under test isn’t defective.
A resilient test is one that only fails when a breaking change is made to the code under test.
Refactorings and other non-breaking changes to the code under test can be made without needing to modify the test, reducing the cost of maintaining the tests.
Maximize resilience by only testing the exposed API of the code under test; avoid reaching into internals. Favor stubs and fakes over mocks; don't verify interactions with dependencies unless it is that interaction that you are explicitly validating. A flaky test obviously has very low resilience.
Precision
When a test fails,
a high-precision test tells you exactly where the defect lies
. A well-written unit test can tell you exactly which line of code is at fault. Poorly written tests (especially large end-to-end tests) often exhibit very low precision, telling you that something is broken but not where.
Maximize precision by keeping your tests small and tightly focused. Choose descriptive method names that convey exactly what the test is validating. For system integration tests, validate state at every boundary.
These three qualities are often in tension with each other. It's easy to write a highly resilient test (the empty test, for example), but writing a test that is both highly resilient and high-fidelity is hard.
As you design and write tests, use these qualities as a framework to guide your implementation
.
5 comments
Labels
TotT
113
GTAC
61
James Whittaker
42
Misko Hevery
32
Code Health
31
Anthony Vallone
27
Patrick Copeland
23
Jobs
18
Andrew Trenk
13
C++
11
Patrik Höglund
8
JavaScript
7
Allen Hutchison
6
George Pirocanac
6
Zhanyong Wan
6
Harry Robinson
5
Java
5
Julian Harty
5
Adam Bender
4
Alberto Savoia
4
Ben Yu
4
Erik Kuefler
4
Philip Zembrod
4
Shyam Seshadri
4
Chrome
3
Dillon Bly
3
John Thomas
3
Lesley Katzen
3
Marc Kaplan
3
Markus Clermont
3
Max Kanat-Alexander
3
Sonal Shah
3
APIs
2
Abhishek Arya
2
Alan Myrvold
2
Alek Icev
2
Android
2
April Fools
2
Chaitali Narla
2
Chris Lewis
2
Chrome OS
2
Diego Salas
2
Dori Reuveni
2
Jason Arbon
2
Jochen Wuttke
2
Kostya Serebryany
2
Marc Eaddy
2
Marko Ivanković
2
Mobile
2
Oliver Chang
2
Simon Stewart
2
Stefan Kennedy
2
Test Flakiness
2
Titus Winters
2
Tony Voellm
2
WebRTC
2
Yiming Sun
2
Yvette Nameth
2
Zuri Kemp
2
Aaron Jacobs
1
Adam Porter
1
Adam Raider
1
Adel Saoud
1
Alan Faulkner
1
Alex Eagle
1
Amy Fu
1
Anantha Keesara
1
Antoine Picard
1
App Engine
1
Arham Jain
1
Ari Shamash
1
Arif Sukoco
1
Bartosz Papis
1
Benjamin Pick
1
Bob Nystrom
1
Bruce Leban
1
Carlos Arguelles
1
Carlos Israel Ortiz García
1
Cathal Weakliam
1
Christopher Semturs
1
Clay Murphy
1
Dagang Wei
1
Dan Maksimovich
1
Dan Shi
1
Dan Willemsen
1
Dave Chen
1
Dave Gladfelter
1
David Bendory
1
David Mandelberg
1
Derek Snyder
1
Diego Cavalcanti
1
Dmitry Vyukov
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Elliott Karpilovsky
1
Elliotte Rusty Harold
1
Espresso
1
Felipe Sodré
1
Francois Aube
1
Gene Volovich
1
Google+
1
Goran Petrovic
1
Goranka Bjedov
1
Hank Duan
1
Havard Rast Blok
1
Hongfei Ding
1
Jason Elbaum
1
Jason Huggins
1
Jay Han
1
Jeff Hoy
1
Jeff Listfield
1
Jessica Tomechak
1
Jim Reardon
1
Joe Allan Muharsky
1
Joel Hynoski
1
John Micco
1
John Penix
1
Jonathan Rockway
1
Jonathan Velasquez
1
Josh Armour
1
Julie Ralph
1
Kai Kent
1
Kanu Tewary
1
Karin Lundberg
1
Kaue Silveira
1
Kevin Bourrillion
1
Kevin Graney
1
Kirkland
1
Kurt Alfred Kluever
1
Kyle Freeman
1
Manjusha Parvathaneni
1
Marek Kiszkis
1
Marius Latinis
1
Mark Ivey
1
Mark Manley
1
Mark Striebeck
1
Matt Lowrie
1
Meredith Whittaker
1
Michael Bachman
1
Michael Klepikov
1
Mike Aizatsky
1
Mike Wacker
1
Mona El Mahdy
1
Noel Yap
1
Palak Bansal
1
Patricia Legaspi
1
Per Jacobsson
1
Peter Arrenbrecht
1
Peter Spragins
1
Phil Norman
1
Phil Rollet
1
Pooja Gupta
1
Project Showcase
1
Radion Khait
1
Radoslav Vasilev
1
Rahul Singal
1
Rajat Dewan
1
Rajat Jain
1
Rich Martin
1
Richard Bustamante
1
Roman Govsheev
1
Roshan Sembacuttiaratchy
1
Ruslan Khamitov
1
Saicharan Nimmala
1
Sam Lee
1
Sean Jordan
1
Sebastian Dörner
1
Shahar Roth
1
Sharon Zhou
1
Shiva Garg
1
Siddartha Janga
1
Simran Basi
1
Stan Chan
1
Stephen Ng
1
Tejas Shah
1
Test Analytics
1
Test Engineer
1
Tim Lyakhovetskiy
1
Tom O'Neill
1
Vojta Jína
1
Zhe Lu
1
automation
1
dead code
1
iOS
1
mutation testing
1
Archive
▼
2026
(7)
▼
Jul
(1)
Prefactoring: Clear the Way for Your New Feature
►
Jun
(1)
►
May
(2)
►
Apr
(1)
►
Mar
(2)
►
2025
(3)
►
Oct
(1)
►
Sep
(1)
►
Jan
(1)
►
2024
(13)
►
Dec
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(1)
►
May
(3)
►
Apr
(3)
►
Mar
(1)
►
Feb
(1)
►
2023
(14)
►
Dec
(2)
►
Nov
(2)
►
Oct
(5)
►
Sep
(3)
►
Aug
(1)
►
Apr
(1)
►
2022
(2)
►
Feb
(2)
►
2021
(3)
►
Jun
(1)
►
Apr
(1)
►
Mar
(1)
►
2020
(8)
►
Dec
(2)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
May
(1)
►
2019
(4)
►
Dec
(1)
►
Nov
(1)
►
Jul
(1)
►
Jan
(1)
►
2018
(7)
►
Nov
(1)
►
Sep
(1)
►
Jul
(1)
►
Jun
(2)
►
May
(1)
►
Feb
(1)
►
2017
(17)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Sep
(1)
►
Aug
(1)
►
Jul
(2)
►
Jun
(2)
►
May
(3)
►
Apr
(2)
►
Feb
(1)
►
Jan
(2)
►
2016
(15)
►
Dec
(1)
►
Nov
(2)
►
Oct
(1)
►
Sep
(2)
►
Aug
(1)
►
Jun
(2)
►
May
(3)
►
Apr
(1)
►
Mar
(1)
►
Feb
(1)
►
2015
(14)
►
Dec
(1)
►
Nov
(1)
►
Oct
(2)
►
Aug
(1)
►
Jun
(1)
►
May
(2)
►
Apr
(2)
►
Mar
(1)
►
Feb
(1)
►
Jan
(2)
►
2014
(24)
►
Dec
(2)
►
Nov
(1)
►
Oct
(2)
►
Sep
(2)
►
Aug
(2)
►
Jul
(3)
►
Jun
(3)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Feb
(1)
►
Jan
(2)
►
2013
(16)
►
Dec
(1)
►
Nov
(1)
►
Oct
(1)
►
Aug
(2)
►
Jul
(1)
►
Jun
(2)
►
May
(2)
►
Apr
(2)
►
Mar
(2)
►
Jan
(2)
►
2012
(11)
►
Dec
(1)
►
Nov
(2)
►
Oct
(3)
►
Sep
(1)
►
Aug
(4)
►
2011
(39)
►
Nov
(2)
►
Oct
(5)
►
Sep
(2)
►
Aug
(4)
►
Jul
(2)
►
Jun
(5)
►
May
(4)
►
Apr
(3)
►
Mar
(4)
►
Feb
(5)
►
Jan
(3)
►
2010
(37)
►
Dec
(3)
►
Nov
(3)
►
Oct
(4)
►
Sep
(8)
►
Aug
(3)
►
Jul
(3)
►
Jun
(2)
►
May
(2)
►
Apr
(3)
►
Mar
(3)
►
Feb
(2)
►
Jan
(1)
►
2009
(54)
►
Dec
(3)
►
Nov
(2)
►
Oct
(3)
►
Sep
(5)
►
Aug
(4)
►
Jul
(15)
►
Jun
(8)
►
May
(3)
►
Apr
(2)
►
Feb
(5)
►
Jan
(4)
►
2008
(75)
►
Dec
(6)
►
Nov
(8)
►
Oct
(9)
►
Sep
(8)
►
Aug
(9)
►
Jul
(9)
►
Jun
(6)
►
May
(6)
►
Apr
(4)
►
Mar
(4)
►
Feb
(4)
►
Jan
(2)
►
2007
(41)
►
Oct
(6)
►
Sep
(5)
►
Aug
(3)
►
Jul
(2)
►
Jun
(2)
►
May
(2)
►
Apr
(7)
►
Mar
(5)
►
Feb
(5)
►
Jan
(4)
Feed
