Builds Aren’t Code, But They Behave Like It
Backend builds that drive products like Quip and Slack Canvas used to take a full hour. That kind of latency makes the entire delivery pipeline sluggish: feedback arrives long after an engineer has moved on to the next task. The fix wasn’t abandoning classic engineering practices—it was applying them more rigorously, alongside modern build tooling like Bazel.
The mental model that makes this work treats a build as a graph of dependencies. Consider a simple app: a backend server handling API and storage, and a decoupled frontend. Arrows in a directed acyclic graph connect source files to the deployable artifacts that depend on them. A change to a Python file forces a backend rebuild; a TypeScript change triggers a frontend rebuild but leaves the backend’s outputs untouched.
Two Levers: Caching and Parallelism
Once a build is modeled as a graph of explicit units of work, the same performance levers used in application code apply directly:
- Do less work. Cache the results of expensive operations so they’re computed only once—trade memory for time.
- Share the load. Distribute work across more compute resources—trade compute for time.
A familiar Python example illustrates the caching lever. Calculating a factorial is potentially expensive; if the input doesn’t change, the answer doesn’t either. A cached version runs the calculation once per input, then looks up subsequent calls.
def factorial(n):
return n * factorial(n-1) if n else 1
@functools.cache
def factorial(n):
return n * factorial(n-1) if n else 1
For caching to be sound, the operation must be hermetic—it only uses explicitly given inputs—and idempotent—the same inputs always yield the same output. Violate those, and the cache produces surprising, incorrect behavior.
The cache key matters as much as the cache itself. A function that takes a list of images and a list of transforms will invalidate its cache whenever any element of either list changes, even if only one new image was added.
@functools.cache
def process_images(
images: list[Image],
transforms: list[Transform]
) -> list[Image]: ..The fix is granularity: cache at the level of a single image-transform pair, not the whole batch. The higher-level API stays, but internally it resolves each pair, only doing fresh work for combinations it hasn’t seen.
def process_images(
images: list[Image],
transforms: list[Transform]
) -> list[Image]:
new_images = []
for image in images:
new_image = image
for transform in transforms:
new_image = process_image(new_image, transform)
new_images.append(new_image)
return new_images
@functools.cache
def process_image(image: Image, transform: Transform) -> Image:
...
Parallelism asks for similar discipline. To fan image processing out across threads, each unit of work needs well-defined inputs and outputs, a path to move them across a boundary (thread, process, or network), and explicit handling of arbitrary completion order.
def process_images_threaded(
images: list[Image],
transforms: list[Transform]
) -> list[Image]:
with ThreadPoolExecutor() as executor:
futures = []
for image in images:
futures.append(executor.submit(process_images, [image], transforms))
# Returns images in any order!
return [future.result() for future in futures.as_completed(futures)]Units should be fine enough to spread across resources, but not so fine that orchestration overhead dwarfs the work. The right size varies by problem, but it’s a design decision, not an accident.
Applying the Principles to Bazel
Bazel builds are made of targets that form a directed acyclic graph. Each target declares three things:
- Dependencies (inputs)—the source files or other targets this step consumes.
- Outputs—the files this step produces.
- Commands—how inputs become outputs.
For the sample app, the graph and target definitions look like this:
python_build(
name = "backend",
srcs = ["core/http.py", "lib/options.py", "data/access.py"],
outs = ["backend.tgz"],
cmd = "python build.py",
)
ts_build(
name = "frontend",
srcs = ["cms/cms.ts", "collab/bridge.ts", "editing/find.ts"],
outs = ["frontend.tgz"],
cmd = "npm build"
)
A build target is conceptually a function definition, not an invocation. Inputs and outputs are explicit; commands run in a sandbox so they can only touch declared inputs. Outputs are just files, which solves the boundary problem—they’re copied wherever they need to go. In return, the build author promises Bazel that each command is hermetic and idempotent.
Those promises unlock capabilities a custom script can’t easily mimic:
- Automatic caching: unchanged inputs mean no build cost—the cached output is reused.
- Automatic distribution: actions spread across local cores or a remote build cluster.
- Automatic pruning: only the actions needed for the requested output ever run.
Just as with application code, a well-formed dependency graph of small, hermetic, idempotent units yields the best caching hit rates and the most parallelism. Theory, though, is only the starting point. The real challenge was applying it to a build that took an hour.
The Real Cost of a Coupled Build
Quip and Canvas are far more complex than the simple examples we have used so far. When we mapped the actual build graph, we found it was not just large — it was fundamentally unsuitable for the kind of caching and parallelization we wanted Bazel to provide.
Analyzing the graph revealed three critical flaws:
- The dependency graph was not a true directed acyclic graph — it contained cycles.
- Build execution units were enormous, not reliably idempotent, and harmed hermeticity because many steps mutated the working directory.
- Cache keys were so coarse that our cache hit rate was effectively zero. Every cached operation behaved like a function with 100 parameters, where 2–3 were always changing.
Throwing Bazel at this build would not have helped. With zero cache hits, its cache management was useless, and its parallelization would have added little over the ad-hoc parallelization already embedded in our scripts. Before Bazel could deliver any benefit, we had to do significant engineering work.
Untangling Application Code from Build Code
Our backend and build code were inseparable. Without a proper build framework, application code had taken over orchestration: Python logic managed Protobuf compilation and built Python and Cython artifacts, while more Python scripts coordinated tsc and webpack to turn TypeScript and Less into the frontend bundles shared by Slack Canvas and Quip’s desktop and web apps. Parallelization was handled by Python’s multiprocessing and the async routines in our core codebase. The result was that the entire built Python backend sat in the dependency tree above every frontend bundle.
That coupling meant a single Python source change altered the cache key for every frontend bundle, forcing an expensive full rebuild. That one dependency edge — between the Python application and the TypeScript build — cost us an average of 35 minutes per build, more than half the total build time.
This was not merely a performance problem. The coupling made it impossible for engineers to reason about the blast radius of a backend change, since it might break the frontend or the build system too. Because a full build took about an hour, we could not run builds at the Pull Request level to provide early warning. The result was frequent breakage on our main branch through no fault of any individual engineer.
We realized that no build-system change alone could fix this. We had to sever the dependencies between frontend and backend, between Python and TypeScript toolchains, and between application code and build code.
Over several months, we rewrote the Python build orchestration in Starlark, Bazel’s build definition language. Starlark’s deliberate constraints enforce the properties Bazel needs to function. Where Python scripts were still necessary, we stripped them down to depend only on the Python standard library, removing all links to backend code and additional build dependencies. We deleted all parallelization code — Bazel now handles that.
The original build code had essentially no tests, so defining “correct” behavior was difficult. To validate our work, we wrote a Rust tool that compared an artifact from the existing build process with one from our new code. The differences guided us to bugs in our new logic, and we iterated until the outputs matched.
Eventually, the build graph looked very different.
All three key couplings were gone. Build logic now lived in BUILD.bazel files alongside the units they built, with clean Starlark APIs and no entanglement with application code. Cache hit rates rose sharply because Python changes no longer affected TypeScript build keys. When the frontend was cached, we could build the whole application in as little as 25 minutes — a major improvement, though still not enough.
Layering Over Orchestration
With the backend-to-frontend coupling severed, we examined why the frontend build alone took 35 minutes. We found more separation-of-concerns problems.
The frontend builder took in all TypeScript, LESS, and CSS sources plus a range of switches controlled by environment variables and command-line options. It calculated the work needed, parallelized it across worker processes, and marshalled the output into deployable JavaScript and CSS bundles.
This was a reasonable trade-off when it was written — it did speed up the build versus a fully serial approach. But it had two key flaws. First, the cacheable units were much too large: all sources in, all bundles out. Changing one input file invalidated everything. There was no way to build a single bundle without depending on the entire set. Second, parallelizing across processes on one machine prevented parallelization across machines with far more cores. The script’s worker processes also competed with Bazel for the same CPU resources, and Bazel sometimes had to wait on work it already knew it did not need.
This was a layering violation.
The builder’s boundary cut across layers of capability, combining business logic with task orchestration and parallelization. We only wanted the top layer — the logic — so we could re-compose it within Bazel. Instead, we had one work orchestrator running inside another, both fighting for the same resources.
The fix was to do less. We deleted a lot of code. The new frontend builder had no parallelization, a much smaller API, and a simple contract: take one set of source files, produce one output bundle, with TypeScript and CSS processed independently.
This version is highly cacheable and parallelizable. Each output artifact is cached independently, keyed only on its direct inputs. A bundle’s TypeScript and CSS builds can run in parallel with each other and with other bundles’ builds. Bazel can now decide scope — one bundle, two, or all — instead of being forced to manage everything at once.
This mirrors the process_images() example from earlier: small, composable, well-keyed units of work. The results were clear:
- Independent caching of bundle, TypeScript, and CSS builds drove cache hit rates up.
- With sufficient resources, Bazel could run all bundle and CSS builds simultaneously, sharply reducing full-rebuild time.
- We removed our own parallelization code entirely, leaving the build script with a single responsibility: the logic for compiling a frontend bundle.
What We Learned
After applying these principles across our entire build graph, our build is now up to six times faster than when we started. That translates into shorter cycle times for engineers, quicker incident resolution, and more frequent releases.
The deeper lesson is that software engineering principles apply to the whole system — not just application code, but also build code, release pipelines, environment setup strategies, and the relationships between them. When you separate concerns and design for composability across all of those layers, every facet of your application gets stronger. A faster build is just the happy side effect.
