By: Patrik Höglund
Introduction
Software development is difficult. Projects often evolve over several years,
under changing requirements and shifting market conditions, impacting developer
tools and infrastructure. Technical debt, slow build systems, poor
debuggability, and increasing numbers of dependencies can weigh down a project
The developers get weary, and cobwebs accumulate in dusty corners of the code
base.
According to Wiktionary ,
hackable is defined as:
Adjective
hackable (comparative
more hackable , superlative
most hackable )
(computing) That can be hacked or broken into; insecure, vulnerable.
That lends itself to hacking (technical tinkering and modification); moddable.
mainfocus for SETIs at Google as the role has evolved over the years.
In Practice
In a hackable project, it’s easy to try things and hard to break things.
Hackability means fast feedback cycles that offer useful information to the
developer.
Developing is easy
Fast build
Good, fast tests
Clean code
Easy running + debugging
One-click rollbacks
In contrast, what is not hackability?
Broken HEAD (tip-of-tree)
Slow presubmit (i.e. checks
running before submit )
Builds take hours
Incremental build/link > 30s
Flakytests
Can’t attach debugger
Logs full of uninteresting information
The Three Pillars of Hackability
There are a number of tools and practices that foster hackability. When
everything is in place, it feels great to work on the product. Basically no time
is spent on figuring out why things are broken, and all time is spent on what
matters, which is understanding and working with the code. I believe there are
three main pillars that support hackability. If one of them is absent,
hackability will suffer. They are:
Pillar 1: Code Health
“I found Rome a city of bricks, and left it a city of marble.”
Tests
Unit and small integration tests are probably the best things you can do for
hackability. They’re a support you can lean on while making your changes, and
they contain lots of good information on what the code does. It isn’t
hackability to boot a slow UI and click buttons on every iteration to verify
your change worked - it is hackability to run a sub-second set of unit tests! In
contrast, end-to-end (E2E) tests generally help hackability much less (and can
evenbe a hindrance if they, or the product, are in sufficiently bad shape).
Figure 1: the Testing Pyramid.
Readable Code and Code Review
At Google, “readability” is a special committer status that is granted per
language (C++, Go, Java and so on). It means that a person not only knows the
language and its culture and idioms well, but also can write clean, well tested
and well structured code. Readability literally means that you’re a guardian of
Google’s code base and should push back on hacky and ugly code. The use of a style guide enforces
consistency, and code
review (where at least one person with readability must approve) ensures the
code upholds high quality. Engineers must take care to not depend too much on
“review buddies” here but really make sure to pull in the person that can give
the best feedback.
Gerrit .
SOLID
principles , unit testing, or concurrency to encourage developers to learn.
Spread knowledge through code review, pair programming and mentoring (such as
with the Readability concept). You can’t just mandate higher code quality; it
takes a lot of work, effort and consistency.
Presubmit Testing and Lint
Consistently formatted source code aids hackability. You can scan code faster if
its formatting is consistent. Automated tooling also aids hackability. It really
doesn’t make sense to waste any time on formatting source code by hand. You
should be using tools like gofmt , clang-format , etc. If the
patch isn’t formatted properly, you should see something like this (example
from Chrome ):
$ git cl upload
Error: the media/audio directory requires formatting. Please run
git cl format media/audio.
Enforce
it with a check . Are there already inappropriate dependencies in your
project? Whitelist the existing ones for now, but at least block new bad
dependencies from forming. Should our app work on Android 16 phones and newer?
Add
linting , so we don’t use level 17+ APIs without gating at runtime. Should
your project’s VHDL code always
place-and-route cleanly on a particular brand of FPGA ?
Invoke the layout tool in your presubmit and and stop submit if the layout
process fails.
Single Branch And Reducing Risk
Having a single branch for everything, and putting risky new changes behind feature flags , aids
hackability since branches and forks often amass tremendous risk when it’s time
to merge them. Single branches smooth out the risk. Furthermore, running all
your tests on many branches is expensive. However, a single branch can have
negative effects on hackability if Team A depends on a library from Team B and
gets broken by Team B a lot. Having some kind of stabilization on Team B’s
software might be a good idea there. Thisarticle covers such situations, and how to integrate often with your
dependencies to reduce the risk that one of them will break you.
Loose Coupling and Testability
Tightly coupled code is terrible for hackability. To take the most ridiculous
example I know: I once heard of a computer game where a developer changed a
ballistics algorithm and broke the game’s chat . That’s hilarious, but
hardly intuitive for the poor developer that made the change. A hallmark of
loosely coupled code is that it’s upfront about its dependencies and behavior
and is easy to modify and move around.
Aggressively Reduce Technical Debt
It’s quite easy to add a lot of code and dependencies and call it a day when the
software works. New projects can do this without many problems, but as the
project becomes older it becomes a “legacy” project, weighed down by
dependencies and excess code. Don’t end up there. It’s bad for hackability to
have a slew of bug fixes stacked on top of unwise and obsolete decisions, and
understanding and untangling the software becomes more difficult.
dependency
enforcement can go a long way toward keeping people honest. It helps make
sure people are making the appropriate trade-offs instead of just slapping on a
new dependency, and it requires them to explain to a fellow engineer when they
want to override a dependency rule. This can prevent unhealthy dependencies like
circular dependencies, abstract modules depending on concrete modules, or
modules depending on the internals of other modules.
nice post, thank you
ReplyDeleteBravo.
ReplyDeleteGreat article, looking forward to the rest
ReplyDeleteReally great, it should be normal for a project to be designed this way e.g. without heavy external network interactions to run tests. Looking forward to the next post.
ReplyDeleteI like how you define what is hackability.
ReplyDeleteJust letting you know. There's a console error when the page loads. Error reads : GET https://2.bp.blogspot.com/_VvKHc_qcUVo/SarWOMfpOqI/AAAAAAAAASE/mM4LwFW8ysE/S45-s35/msn.jpg 404 ()
ReplyDeletecomments.js:3 Failed to execute 'write' on 'Document': It isn't possible to write into a document from an asynchronously-loaded external script unless it is explicitly opened.Xb @ comments.js:3comments @ comments.js:9(anonymous function) @ comments.js:11
Clearly we need more tests protecting Blogger deployments, haha.
DeleteEveryone else: thanks for the kind words.
Great post!
ReplyDelete