Keeping 35,000 Daily Builds Green Without a Human in the Loop

Dropbox runs over 35,000 builds and millions of automated tests each day. At that volume, flaky tests and broken commits are inevitable. In the past, managing the fallout meant manual rotations: reverting commits that broke post-submit tests and quarantining flaky ones. As the operational burden grew, the responsibility split across multiple teams, all spending time on build health instead of feature work. The solution was a new service, Athena, which watches test results and automates the decision-making.

A flaky test that occasionally times out

Athena reduces human effort in two main ways: it identifies commits that deterministically break tests and notifies the author to revert, and it flags flaky tests for automatic quarantine. The system is designed to keep pre-submit testing fast and reliable while ensuring post-submit suites (like Selenium/UI tests, which are too slow and flaky to run before code lands) still gate releases.

Why Test Failures Are Hard to Classify

Distinguishing a deterministic breakage from a spurious failure is the core challenge. Tests are arbitrary user code and can fail for many reasons. Athena addresses three main classes of non-deterministic failures:

Non-hermetic tests. Some tests depend on external resources that are hard to fake, such as time. Dropbox sees failures that crop up when a new UTC day begins or a monthly discount expires. To spot these environmental failures, Athena tracks the latest "stable" commit where all tests passed. A test that fails on a stable commit likely points to non-hermetic behavior rather than a code change.

Flaky tests. These behave non-deterministically with no input change, often due to random numbers, thread scheduling, or concurrency. When a test fails, Athena reruns it up to ten times—a threshold chosen after experimentation. If results are inconsistent across those reruns, the test is deemed flaky.

Infrastructural flakiness. Even with containers, CPU/NUMA pinning, and resource quotas, a large cluster has performance variance. A test may simply time out on a slow host. Athena handles this by retrying the test on a different machine.

How Athena Triages a Failure

Athena monitors results for all new code submissions. When a test fails more than once within a few hours in post-submit testing, it marks that test as "noisy"—a temporary state while the system figures out whether the failure is a flake, infrastructure issue, or a real regression.

Noisy tests are temporarily ignored in pre-submit runs, so new code submissions aren't rejected due to failures that may not be the developer's fault. The tests continue running post-submit and still count toward overall build status, ensuring a broken release can't slip through.

Flake and Breakage Detection

To confirm a flake, Athena reruns the test on a commit where it previously passed. If it flaps between pass and fail across multiple hosts, it gets quarantined. Environmental failures are handled by rerunning on the latest stable commit.

Breakage detection is more involved. For each noisy test, Athena runs a bisect to find the transition point where the test went from passing to failing. It then reruns the test at that commit ten times to confirm the breakage is deterministic, not a flake.

def find_offending_commit(test, commits, latest_stable_commit):    
    # commits where the test might have transitioned from pass to fail
    potential_transition_points = [] 

    for commit in reversed(commits):
      if is_test_failing(test, commit) or is_test_unknown(test, commit):
          potential_transition_points.append(commit)
      else:
        # if we see a pass, we're guaranteed not to have broken the commit before
        # this point
        break

    if not potential_transition_points:
      return NoIssues() # the test isn't broken

    if len(potential_transition_points) == 1:
      # potential culprit
      commit = potential_transition_points[0]

      # deflake the test: run it 10 times to confirm it's broken
      is_failure = trigger_test(test, commit, deflake_num_runs=10)
      if not is_failure:
        return NoIssues() # just a flake

      # rerun on the stable commit to confirm it's not environmental issues
      is_failure = trigger_test(test, latest_stable_commit)
      if not is_failure:
        return Breakage(test=test, commit=commit)     

    # bisect logic
    midpoint = (len(potential_transition_points) / 2) - 1
    middle = potential_transition_points[midpoint]

    is_failure = is_test_failing(test, middle, run_if_unknown=True)

    # run bisect with a limited set
    if is_failure:
      new_candidates = potential_transition_points[:midpoint+1]
    else:
      new_candidates = potential_transition_points[midpoint+1:]

    return find_offending_commit(test, new_candidates, latest_stable_commit)

Results and Operational Lessons

After several months in production, Athena doubled the number of test quarantines while eliminating manual quarantine work. Quarantines are more aggressive than before, but that's intentional: it keeps pre-submit tests free of spurious flakes and enforces a consistent quality bar.

Athena doesn't auto-revert commits. Instead, it notifies the author and the owning team. Since broken test results are ignored in pre-submit, getting back to stable isn't time-sensitive, and an autoresponder could interfere with a forward fix for a critical issue.

One operational change was "rate limiting" expensive tests like Selenium UI suites, running them at most once every ten minutes instead of on most commits. While that cut down on demand spikes, teams lost the ability to visually scan for breakages. Developers also distrusted the system without visibility into what it was doing. A simple UI showing progress restored confidence and made Athena self-serve, reducing support load.

Overall, rate limiting reduced the testing cluster size by about 8%.

Roadmap and Takeaways

Athena hasn't yet been applied to desktop tests, which will introduce new failure modes across OS variants. For breakage detection, clusters with spare capacity may allow an "n-sect" approach, testing all possible transition points in parallel to catch breakages faster. Auto-revert is also on the table if a build stays broken for hours.

The experience reinforced three principles:

  • Keep notifications high-signal. Spamming developers with inaccurate alerts trains them to ignore the system. Precision builds trust and drives manual action.
  • Automation reduces bikeshedding. Like gofmt for code style, automated quarantine maintains a high test-quality bar without debate over individual cases.
  • Indicate progress on long-running actions. Asynchronous workflows need visible activity to reassure users the system isn't stuck.