Scaling CI in a Monorepo with Bazel

Dropbox’s server-side code lives in a large monorepo, and as the repository has grown, so has the cost of operating on it as a whole. Running the complete test suite on every commit was once feasible, but it became wasteful as the number of tests increased—particularly because most tests cannot be affected by any given change.

The solution leverages Bazel, which is used exclusively for building and testing the monorepo. Bazel models the repository as a graph of targets (binaries, libraries, tests, source files) and their dependencies. Dropbox’s continuous integration system extracts this dependency information using bazel query to compute the exact set of tests affected by a commit. This drastically cuts the number of tests run per commit while maintaining correctness.

Status Propagation and Test Batching

Because a test no longer runs on every commit, its status on skipped commits is inferred from history. If a test passes on commit N and is unaffected by commit N+1, its result is propagated forward. This yields a status for every test on every commit in which it exists.

To further conserve resources, affected tests are not run individually per commit. Instead, the CI system aggregates the affected test sets across all commits in a fixed time window, computing a union, and runs the combined set once on the last commit of that period. The window size is tunable, balancing resource usage against how quickly results are available.

This batching could mask the true culprit behind a failure. However, the automated breakage detection system, Athena, handles this by bisecting failing tests over the entire rollup window to isolate the precise commit that broke the build.

Deployment with Release Tests

Production software is shipped as SquashFS images, built by a custom Bazel rule. Historically, a commit was deployable only if every test in the repository passed on it—either directly or through propagation. That “monolithic green” model broke down as the repo grew, making it slow to prove a commit was fully green and blocking deployments on unrelated test failures.

The replacement is a per-package release criteria. Every deployable package can declare a set of release_tests, specified using a subset of Bazel’s target pattern syntax. Only these tests must pass before the package can be pushed.

For example, a simple C++ ping server would have a BUILD file like this:

cc_library(
     name = "ping_lib",
     srcs = ["ping.cc"],
)

cc_binary(
    name = "ping_server",
    srcs = ["ping_server.cc"],
    deps = [":ping_lib"],
)

cc_test(
    name = "ping_test",
    srcs = ["ping_test.cc"],
    deps = [":ping_lib"],
)

dbx_pkg_sqfs(
    name = "ping_server.sqfs",
    data = [
        ":server",
    ],
    release_tests = [
        "//ping_server/...",
    ],
)

This declares a C++ library, binary, and test using the standard Bazel C++ rules. The ping_server.sqfs target is a SquashFS image holding the ping_server binary, and it can be deployed once all tests in //ping_server/ and its subpackages have passed.

Immediate Deploys and Team Autonomy

Because test runs are aggregated in time windows, there can be extra latency between landing a commit and it becoming deployable. To avoid this, an engineer can request that the CI system run a package’s release tests immediately when a commit lands—regardless of where it sits in the rollup period. This allows hotfixes and urgent changes to move quickly.

The decision of what to include in release_tests is left to individual teams. Typically this includes the package’s own tests and those of critical dependency libraries. More conservative groups may also include tests of their reverse dependencies. During development, Dropbox experimented with auto-generating this set from the packaged code’s dependency graph, but found no intuitive heuristic that both matched engineer expectations and meaningfully reduced the test count.