Shaving the overhead out of ML training builds

Machine learning engineers spend a meaningful share of their day not training models but waiting on builds, packaging, and distribution. At our scale, the overhead from slow compilation and heavyweight executable delivery was costing us double-digit percentages of engineering time. We tackled both sides of that problem: reducing how much code needs rebuilding, and making self-contained Python executables incremental to produce and fetch.

Why builds were slow

Two failure modes dominated. First, "cold" revisions — older commits that our build infrastructure hadn't cached recently — forced repeated compilation and linking of large component graphs. Second, build non-determinism meant that even identical inputs sometimes yielded different outputs, which invalidated cache entries and triggered unnecessary rebuilds.

We traced the non-determinism to two sources:

  • Tooling — compilers such as Clang, Rustc, and NVCC can emit different binaries for the same source input.
  • Build rules and source code — developers often introduced randomness intentionally or not, through temporary directories, random values, or timestamps baked into build logic.

Running nearly all build actions through Buck2's Remote Execution (RE) service let us mitigate these issues centrally. RE now returns consistent outputs for identical actions, which made a warm, stable revision practical: in many cases, builds disappear from the critical path entirely.

Dependency growth was the other lever. As graphs expanded, we improved our analysis tooling to find and strip unnecessary edges. That meant removing GPU code from binaries that didn't need it, identifying which Python modules were genuinely imported, and cutting native code with linker maps. The result was a leaner build graph and faster overall compilation.

Making packaging and distribution incremental

Self-contained Python executables have historically been packaged as XAR files. When unpacked, a typical one contains thousands of Python files plus native libraries and an interpreter — often hundreds of thousands of files totaling tens of gigabytes. Even a small edit to a few Python files forced a full XAR reassembly and redistribution before the executable could run on a remote host. For incremental work, that packaging and fetch overhead often exceeded the build time itself.

Our replacement, the Content Addressable Filesystem (CAF), treats packaging and fetching as incremental operations:

  • Packaging — a content-aware step skips uploads of files already present in Content Addressable Storage (CAS), whether from another executable or a different version of the same one.
  • Fetching — a destination-host cache ensures only content missing locally is downloaded.

A CAS daemon runs on most of our data center hosts. It maintains the local cache (materialization and garbage collection) and organizes a P2P network with peer daemons using Owl, our high-fanout content distribution system. That allows direct fetching from other hosts, which cuts both latency and storage bandwidth demand.

CAF defines an executable with a flat manifest listing symlinks, directories, hard links, and files along with their digests and attributes. That design deduplicates unique files across executables and enables a scheduling affinity/routing mechanism that maximizes local cache hits and minimizes downloads.

This approach resembles Docker's OverlayFS in spirit, but the details diverge. With so many executables sharing diverse dependencies, organizing proper layers is often impractical: layering loses efficiency and complexity climbs. Direct file access is also required for P2P shipping. We chose Btrfs as the underlying filesystem for its compression support and Copy-on-write (COW) behavior — compressed data can be written straight to extents without redundant decompression/recompression, and COW means updates only touch the affected file extents. Executables stay roughly the same size on disk as their XAR equivalents, while cache files are shared across executables.

Next steps: LazyCAF and uniform revisions

The combination of faster builds and incremental CAF packaging/distribution has reduced total overhead by double-digit percentages. Two further optimizations are on the horizon.

Workload analysis shows that only a fraction of an executable's content is used in many scenarios. Fetching parts on demand — a "lazy" variant of CAF — would cut materialization time and shrink disk footprint further. Separately, enforcing a uniform development revision across all ML engineers would raise cache hit ratios and increase the share of incremental builds, since most artifacts would already be cached.