Google Testing Blog: Patrik Höglund

Hackable Projects - Pillar 3: Infrastructure

Thursday, November 10, 2016

By: Patrik Höglund
This is the third article in our series on Hackability; also see the first and second article.Code HealthDebuggability Build Systems Speed
Question: What’s a change you’d really like to see in our development tools?

“I feel like this question gets asked almost every time, and I always give the same answer:
I would like them to be faster.”
-- Ian Lance TaylorninjagoldBazel

Figure 1: “Cheetah chasing its prey” by Marlene Thyssen.

Build and test times longer than a handful of seconds cause many developers’ minds to wander, taking them out of the zone.

Excessive build or release times* makes tinkering and refactoring much harder. All developers have a threshold when they start hedging (e.g. “I’d like to remove this branch, but I don’t know if I’ll break the iOS build”) which causes refactoring to not happen.

* The worst I ever heard of was an OS that took 24 hours to build!

How do you actually make fast build systems? There are some suggestions in the first paragraph above, but the best general suggestion I can make is to have a few engineers on the project who deeply understand the build systems and have the time to continuously improve them. The main axes of improvement are:

Reduce the amount of code being compiled.
Replace tools with faster counterparts.
Increase processing power, maybe through parallelization or distributed systems.

Note that there is a big difference between full builds and incremental builds. Both should be as fast as possible, but incremental builds are by far the most important to optimize. The way you tackle the two is different. For instance, reducing the total number of source files will make a full build faster, but it may not make an incremental build faster.

To get faster incremental builds, in general, you need to make each source file as decoupled as possible from the rest of the code base. The less a change ripples through the codebase, the less work to do, right? See “Loose Coupling and Testability” in Pillar 1 for more on this subject. The exact mechanics of dependencies and interfaces depends on programming language - one of the hardest to get right is unsurprisingly C++, where you need to be disciplined with includes and forward declarations to get any kind of incremental build performance.

Build scripts and makefiles should be held to standards as high as the code itself. Technical debt and unnecessary dependencies have a tendency to accumulate in build scripts, because no one has the time to understand and fix them. Avoid this by addressing the technical debt as you go.

Continuous Integration and Presubmit Queues

You should build and run tests on all platforms you release on. For instance, if you release on all the major desktop platforms, but all your developers are on Linux, this becomes extra important. It’s bad for hackability to update the repo, build on Windows, and find that lots of stuff is broken. It’s even worse if broken changes start to stack on top of each other. I think we all know that terrible feeling: when you’re not sure your change is the one that broke things.

At a minimum, you should build and test on all platforms, but it’s even better if you do it in presubmit. The Chromium submit queue does this. It has developed over the years so that a normal patch builds and tests on about 30 different build configurations before commit. This is necessary for the 400-patches-per-day velocity of the Chrome project. Most projects obviously don’t have to go that far. Chromium’s infrastructure is based off BuildBot, but there are many other job scheduling systems depending on your needs.

Figure 2: How a Chromium patch is tested.

As we discussed in Build Systems, speed and correctness are critical here. It takes a lot of ongoing work to keep build, tests, and presubmits fast and free of flakes. You should never accept flakes, since developers very quickly lose trust in flaky tests and systems. Tooling can help a bit with this; for instance, see the Chromium flakiness dashboard.

Test Speed

Speed is a feature, and this is particularly true for developer infrastructure. In general, the longer a test takes to execute, the less valuable it is. My rule of thumb is: if it takes more than a minute to execute, its value is greatly diminished. There are of course some exceptions, such as soak tests or certain performance tests.

Figure 3. Test value.

If you have tests that are slower than 60 seconds, they better be incredibly reliable and easily debuggable. A flaky test that takes several minutes to execute often has negative value because it slows down all work in the code it covers. You probably want to build better integration tests on lower levels instead, so you can make them faster and more reliable.

If you have many engineers on a project, reducing the time to run the tests can have a big impact. This is one reason why it’s great to have SETIs or the equivalent. There are many things you can do to improve test speed:

Sharding and parallelization. Add more machines to your continuous build as your test set or number of developers grows.
Continuously measure how long it takes to run one build+test cycle in your continuous build, and have someone take action when it gets slower.
Remove tests that don’t pull their weight. If a test is really slow, it’s often because of poorly written wait conditions or because the test bites off more than it can chew (maybe that unit test doesn’t have to process 15000 audio frames, maybe 50 is enough).
If you have tests that bring up a local server stack, for instance inter-server integration tests, making your servers boot faster is going to make the tests faster as well. Faster production code is faster to test! See Running on Localhost, in Pillar 2 for more on local server stacks.

Workflow Speed

We’ve talked about fast builds and tests, but the core developer workflows are also important, of course. Chromium undertook multi-year project to switch from Subversion to Git, partly because Subversion was becoming too slow. You need to keep track of your core workflows as your project grows. Your version control system may work fine for years, but become too slow once the project becomes big enough. Bug search and management must also be robust and fast, since this is generally systems developers touch every day.

Release Often

It aids hackability to deploy to real users as fast as possible. No matter how good your product's tests are, there's always a risk that there's something you haven't thought of. If you’re building a service or web site, you should aim to deploy multiple times per week. For client projects, Chrome’s six-weeks cycle is a good goal to aim for.

You should invest in infrastructure and tests that give you the confidence to do this - you don’t want to push something that’s broken. Your developers will thank you for it, since it makes their jobs so much easier. By releasing often, you mitigate risk, and developers will rush less to hack late changes in the release (since they know the next release isn’t far off).

Easy Reverts

If you look in the commit log for the Chromium project, you will see that a significant percentage of the commits are reverts of a previous commits. In Chromium, bad commits quickly become costly because they impede other engineers, and the high velocity can cause good changes to stack onto bad changes.

Figure 4: Chromium’s revert button.

This is why the policy is “revert first, ask questions later”. I believe a revert-first policy is good for small projects as well, since it creates a clear expectation that the product/tools/dev environment should be working at all times (and if it doesn’t, a recent change should probably be reverted).

It has a wonderful effect when a revert is simple to make. You can suddenly make speculative reverts if a test went flaky or a performance test regressed. It follows that if a patch is easy to revert, so is the inverse (i.e. reverting the revert or relanding the patch). So if you were wrong and that patch wasn’t guilty after all, it’s simple to re-land it again and try reverting another patch. Developers might initially balk at this (because it can’t possibly be their patch, right?), but they usually come around when they realize the benefits.

For many projects, a revert can simply be

git revert 9fbadbeef

git push origin master

If your project (wisely) involves code review, it will behoove you to add something like Chromium’s revert button that I mentioned above. The revert button will create a special patch that bypasses review and tests (since we can assume a clean revert takes us back to a more stable state rather than the opposite). See Pillar 1 for more on code review and its benefits.

For some projects, reverts are going to be harder, especially if you have a slow or laborious release process. Even if you release often, you could still have problems if a revert involves state migrations in your live services (for instance when rolling back a database schema change). You need to have a strategy to deal with such state changes.

Reverts must always put you back to safer ground, and everyone must be confident they can safely revert. If not, you run the risk of massive fire drills and lost user trust if a bad patch makes it through the tests and you can’t revert it.

Performance Tests: Measure Everything

Is it critical that your app starts up within a second? Should your app always render at 60 fps when it’s scrolled up or down? Should your web server always serve a response within 100 ms? Should your mobile app be smaller than 8 MB? If so, you should make a performance test for that. Performance tests aid hackability since developers can quickly see how their change affects performance and thus prevent performance regressions from making it into the wild.

You should run your automated performance tests on the same device; all devices are different and this will reflect in the numbers. This is fairly straightforward if you have a decent continuous integration system that runs tests sequentially on a known set of worker machines. It’s harder if you need to run on physical phones or tablets, but it can be done.

A test can be as simple as invoking a particular algorithm and measuring the time it takes to execute it (median and 90th percentile, say, over N runs).

Figure 5: A VP8-in-WebRTC regression (bug) and its fix displayed in a Catapult dashboard.

Write your test so it outputs performance numbers you care about. But what to do with those numbers? Fortunately, Chrome’s performance test framework has been open-sourced, which means you can set up a dashboard, with automatic regression monitoring, with minimal effort. The test framework also includes the powerful Telemetry framework which can run actions on web pages and Android apps and report performance results. Telemetry and Catapult are battle-tested by use in the Chromium project and are capable of running on a wide set of devices, while getting the maximum of usable performance data out of the devices.

Sources

Figure 1: By Malene Thyssen (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

3 comments

Hackable Projects - Pillar 2: Debuggability

Tuesday, October 11, 2016

By: Patrik Höglund
This is the second article in our series on Hackability; also see the first article.
“Deep into that darkness peering, long I stood there, wondering, fearing, doubting, dreaming dreams no mortal ever dared to dream before.” -- Edgar Allan Poe

Figure 1: computer log entry from the Mark II, with a moth taped to the page. Running on Localhost Read more on the Testing Blog: Hermetic Servers by Chaitali Narla and Diego Salas server stack on localhostb_serv --port=1234 --db=/tmp/fakedb c_serv --port=1235 a_serv --b_spec=localhost:1234 --c_spec=localhost:1235 hermetic tests, Debugging Mobile AppsRobolectric unit testsEspresso functional testsxcTestsEarl Grey When Debugging gets Trickyhard real-time systemsdebugging flags and instructionschrome --renderer-cmd-prefix='xterm -title renderer -e gdb --args' Proper Logging Read more on the Testing Blog: Optimal Logging by Anthony Vallone ChromiumThis article Monitoring and StatisticsStackdriver for Google Cloud System Under Test (SUT) SizeThis article

Figure 2. The smaller the SUT, the more valuable the test. Sources (Continue to Pillar 3: Infrastructure)

No comments

Hackable Projects - Pillar 1: Code Health

Thursday, August 18, 2016

By: Patrik Höglund IntroductionAccording to Wiktionary Adjective
hackable ‎(comparative more hackable, superlative most hackable)

(computing) That can be hacked or broken into; insecure, vulnerable.
That lends itself to hacking (technical tinkering and modification); moddable.

mainfocus for SETIs at Google In Practice

Developing is easy

Fast build

Good, fast tests

Clean code

Easy running + debugging

One-click rollbacks

Broken HEAD (tip-of-tree)

Slow presubmit (i.e. checks running before submit)

Builds take hours

Incremental build/link > 30s

Flakytests

Can’t attach debugger

Logs full of uninteresting information

The Three Pillars of Hackability

Pillar 1: Code Health “I found Rome a city of bricks, and left it a city of marble.”
-- Augustus Testsevenbe a hindrance

Figure 1: the Testing Pyramid. Readable Code and Code Reviewstyle guidecode reviewGerritSOLID principles Presubmit Testing and Lintgofmtclang-formatexample from Chrome$ git cl upload Error: the media/audio directory requires formatting. Please run git cl format media/audio.Enforce it with a checkAdd lintingVHDLFPGA Single Branch And Reducing Riskfeature flagsThisarticle Loose Coupling and Testabilitychat Aggressively Reduce Technical Debtdependency enforcement (Continue to Pillar 2: Debuggability)

8 comments

Audio Testing - Automatic Gain Control

Thursday, October 08, 2015

By: Patrik Höglund What is Automatic Gain Control? http://apprtc.appspot.com

Figure 1. How Auto Gain Control works [code here].
PID controller Testing the AGCwrite unit testsintegration testsdidn’t think about building that end-to-end test first, did youstable, low-maintenance audio quality testsPESQ

Add file support to Chrome’s fake audio input device, so we can play a known file. The original audio test avoided this by using WebAudio, but AGC doesn’t run in the WebAudio path, just the microphone capture path, so that won’t work.

Instead of running PESQ, run an analysis that compares the gain between input and output.

Adding Fake File Supportherethisint FileSource::OnMoreData(AudioBus* audio_bus, uint32 total_bytes_delay) { // Load the file if we haven't already. This load needs to happen on the // audio thread, otherwise we'll run on the UI thread on Mac for instance. // This will massively delay the first OnMoreData, but we'll catch up. if (!wav_audio_handler_) LoadWavFile(path_to_wav_file_); if (load_failed_) return 0; DCHECK(wav_audio_handler_.get()); // Stop playing if we've played out the whole file. if (wav_audio_handler_->AtEnd(wav_file_read_pos_)) return 0; // This pulls data from ProvideInput. file_audio_converter_->Convert(audio_bus); return audio_bus->frames(); } chrome --use-fake-device-for-media-stream \ --use-file-for-fake-audio-capture=/tmp/file.wav The Analysis StageAudioPowerMonitorA_diff = A_out - A_inA_diffimplement // ... size_t bytes_written; wav_audio_handler->CopyTo(audio_bus.get(), 0, &bytes_written); CHECK_EQ(bytes_written, wav_audio_handler->data().size()) << "Expected to write entire file into bus."; // Set the filter coefficient to the whole file's duration; this will make // the power monitor take the entire file into account. media::AudioPowerMonitor power_monitor(wav_audio_handler->sample_rate(), file_duration); power_monitor.Scan(*audio_bus, audio_bus->frames()); // ... return power_monitor.ReadCurrentPowerAndClip().first;A_inA_out What Went Wrong?A_diff

Figure 2. Plot of A_diff.A_outA_in

Figure 3. Plot of A_outand A_in. Clock Drift and Packet LossNetEqpacket lossclock drift Silence Splitting to the Rescue!least squares

Figure 4. Before silence splitting.

Figure 5. After silence splitting. Resultfinal test implementation base::FilePath reference_file = test::GetReferenceFilesDir().Append(reference_filename); base::FilePath recording = CreateTemporaryWaveFile(); ASSERT_NO_FATAL_FAILURE(SetupAndRecordAudioCall( reference_file, recording, constraints, base::TimeDelta::FromSeconds(30))); base::ScopedTempDir split_ref_files; ASSERT_TRUE(split_ref_files.CreateUniqueTempDir()); ASSERT_NO_FATAL_FAILURE( SplitFileOnSilenceIntoDir(reference_file, split_ref_files.path())); std::vector<base::FilePath> ref_segments = ListWavFilesInDir(split_ref_files.path()); base::ScopedTempDir split_actual_files; ASSERT_TRUE(split_actual_files.CreateUniqueTempDir()); ASSERT_NO_FATAL_FAILURE( SplitFileOnSilenceIntoDir(recording, split_actual_files.path())); // Keep the recording and split files if the analysis fails. base::FilePath actual_files_dir = split_actual_files.Take(); std::vector<base::FilePath> actual_segments = ListWavFilesInDir(actual_files_dir); AnalyzeSegmentsAndPrintResult( ref_segments, actual_segments, reference_file, perf_modifier); DeleteFileUnlessTestFailed(recording, false); DeleteFileUnlessTestFailed(actual_files_dir, true);void AnalyzeSegmentsAndPrintResult( const std::vector<base::FilePath>& ref_segments, const std::vector<base::FilePath>& actual_segments, const base::FilePath& reference_file, const std::string& perf_modifier) { ASSERT_GT(ref_segments.size(), 0u) << "Failed to split reference file on silence; sox is likely broken."; ASSERT_EQ(ref_segments.size(), actual_segments.size()) << "The recording did not result in the same number of audio segments " << "after on splitting on silence; WebRTC must have deformed the audio " << "too much."; for (size_t i = 0; i < ref_segments.size(); i++) { float difference_in_decibel = AnalyzeOneSegment(ref_segments[i], actual_segments[i], i); std::string trace_name = MakeTraceName(reference_file, i); perf_test::PrintResult("agc_energy_diff", perf_modifier, trace_name, difference_in_decibel, "dB", false); } } this

Figure 6. Average Adiff values for each segment on the y axis, Chromium revisions on the x axis.

4 comments

Multi-Repository Development

Friday, May 15, 2015

Author: Patrik Höglund roughly a hundred libraries

Figure 1. Holy dependencies, Batman!libphonenumber Updating Dependencies is Hard

Update the library to its latest release. If it’s been a long time since you did this, it can be quite risky and you may have to spend significant testing resources to ensure all the accumulated changes don’t break your application. You may have to catch up to interface changes in the library as well.

Cherry-pick the feature/bug fix you want into your copy of the library. This is even riskier because your cherry-picked patches may depend on other changes in the library in subtle ways. Also, you still are not up to date with the latest version.

Find some way to make do without the feature or bug fix.

How “Rolling” Works DEPS{ # ... 'src/third_party/webrtc': 'https://chromium.googlesource.com/' + 'external/webrtc/trunk/webrtc.git' + '@' + '5727038f572c517204e1642b8bc69b25381c4e9f', } 572703...DEPSrollinggetUserMedia has started crashing on Androidmany tests, on all platforms we ship on

Figure 2. Life of a Roll Patch. Bots That Can See the Future: “FYI Bots” DEPS

If all the FYI bots were green, we knew a roll most likely would go smoothly.

If the bots didn’t compile, we knew we would have to adapt Chrome to an interface change in the next roll patch.

If the bots were red, we knew we either had a bug in WebRTC or that Chrome would have to be adapted to some semantic change in WebRTC.

FYI “waterfall”main waterfall Making Gradual Interface Changes red blindness[1]API prime directive[2]class WebRtcAmplifier { ... int SetOutputVolume(float volume); }class WebRtcAmplifier { ... int SetOutputVolume(float volume, bool allow_eleven¹); }class WebRtcAmplifier { ... int SetOutputVolume(float volume); int SetOutputVolume2(float volume, bool allow_eleven); }

Roll into Chrome

Make Chrome use SetOutputVolume2

Update SetOutputVolume’s signature

Roll again and make Chrome use SetOutputVolume

Delete SetOutputVolume2

Resultsperformance tests Future Work: Optimistic Auto-rollingARB (AutoRollBot) References Footnotes

We actually did have a hilarious bug in WebRTC where it was possible to set the volume to 1.1, but only 0.0-1.0 was supposed to be allowed. No, really. Thus, our WebRTC implementation must be louder than the others since everybody knows 1.1 must be louder than 1.0.

6 comments

Chrome - Firefox WebRTC Interop Test - Pt 2

Tuesday, September 09, 2014

by Patrik Höglund
This is the second in a series of articles about Chrome’s WebRTC Interop Test. See the first. Verifying the Callapprtc.appspot.comremoteVideo.style.opacitybool WaitForCallToComeUp(content::WebContents* tab_contents) { // Apprtc will set remoteVideo.style.opacity to 1 when the call comes up. std::string javascript = "window.domAutomationController.send(remoteVideo.style.opacity)"; return test::PollingWaitUntil(javascript, "1", tab_contents); } Verifying Video is Playingexisting video detectorbool DetectRemoteVideoPlaying(content::WebContents* tab_contents) { if (!EvalInJavascriptFile(tab_contents, GetSourceDir().Append( FILE_PATH_LITERAL( "chrome/test/data/webrtc/test_functions.js")))) return false; if (!EvalInJavascriptFile(tab_contents, GetSourceDir().Append( FILE_PATH_LITERAL( "chrome/test/data/webrtc/video_detector.js")))) return false; // The remote video tag is called remoteVideo in the AppRTC code. StartDetectingVideo(tab_contents, "remoteVideo"); WaitForVideoToPlay(tab_contents); return true; }StartDetectingVideoWaitForVideoToPlayvideo_detector.js What to Send in the Call--use-fake-device-for-media-streamregular test infrastructureWebRTC waterfallsif (!HasWebcamOnSystem()) return; Downloading all Code and Components

The AppEngine SDK in order to bring up the local AppRTC instance,

The AppRTC code itself,

Chrome (already present in the checkout), and

Firefox nightly.

thousands and thousands of machinesDownloading the AppEngine SDKauthoritative sourceGoogle storagedepot_toolsdownload_from_google_storage.pyDownloading the AppRTC CodeChromium DEPS fileDownloading Firefoxin this scriptfails every now and then Putting it Togetherfinal code here.gclient solution

Also run on Windows/Mac.

Also test Opera.

Interop between Chrome/Firefox mobile and desktop browsers.

Also ensure audio is playing.

Measure bandwidth stats, video quality, etc.

3 comments

Chrome - Firefox WebRTC Interop Test - Pt 1

Tuesday, August 26, 2014

by Patrik Höglund WebRTCworking video chatsmall amount of Python and JavaScript Calling in WebRTC

Figure 1. Signaling and media connections.AppRTC reference application Test Designpeerconnection_serverapprtc.appspot.comvideo+canvas grab trickWebRTC getStats

Figure 2. Basic test design.hermetic Bringing up AppRTC on localhostGoogle App Engine applicationhello world examplegoogle_appengine/dev_appserver.py apprtc_code/bool LaunchApprtcInstanceOnLocalhost() // ... Figure out locations of SDK and apprtc code ... CommandLine command_line(CommandLine::NO_PROGRAM); EXPECT_TRUE(GetPythonCommand(&command_line)); command_line.AppendArgPath(appengine_dev_appserver); command_line.AppendArgPath(apprtc_dir); command_line.AppendArg("--port=9999"); command_line.AppendArg("--admin_port=9998"); command_line.AppendArg("--skip_sdk_update_check"); VLOG(1) << "Running " << command_line.GetCommandLineString(); return base::LaunchProcess(command_line, base::LaunchOptions(), &dev_appserver_); } ^[1] Figuring out Whether the Local Server is Up http://localhost:9999?r=some_roomdev_appserver.pybool LocalApprtcInstanceIsUp() { // Load the admin page and see if we manage to load it right. ui_test_utils::NavigateToURL(browser(), GURL("localhost:9998")); content::WebContents* tab_contents = browser()->tab_strip_model()->GetActiveWebContents(); std::string javascript = "window.domAutomationController.send(document.title)"; std::string result; if (!content::ExecuteScriptAndExtractString(tab_contents, javascript, &result)) return false; return result == kTitlePageOfAppEngineAdminPage; } 9998localhost:9999while (!LocalApprtcInstanceIsUp()) VLOG(1) << "Waiting for AppRTC to come up..."; Launching the Browsers launcher python scriptfrom mozprofile import profile from mozrunner import runner WEBRTC_PREFERENCES = { 'media.navigator.permission.disabled': True, } def main(): # Set up flags, handle SIGTERM, etc # ... firefox_profile = profile.FirefoxProfile(preferences=WEBRTC_PREFERENCES) firefox_runner = runner.FirefoxRunner( profile=firefox_profile, binary=options.binary, cmdargs=[options.webpage]) firefox_runner.start()GURL room_url = GURL(base::StringPrintf("http://localhost:9999?r=room_%d", base::RandInt(0, 65536))); content::WebContents* chrome_tab = OpenPageAndAcceptUserMedia(room_url); ASSERT_TRUE(LaunchFirefoxWithUrl(room_url));run_firefox_webrtc.py --binary /path/to/firefox --webpage http://localhost::9999?r=my_room1

2 comments

Testing Blog