Reconciling Speed and Safety

Product engineers want their code in front of real users yesterday, to validate ideas and iterate quickly. A new feature like the Dropbox Badge might require a CSS tweak one day and a change enabling it on a new file type the next—user studies are slow and statistically thin, so field feedback via staged rollouts is the fast loop. Platform engineers want the opposite: a stable development environment, reliable build machines, high sync quality, and minimal crashes. Their instinct is to say no to risky changes that could break the release.

For the Dropbox Desktop Client, this tension used to resolve in favor of caution and manual effort. Major releases took eight weeks each, required more than three full-time engineers to orchestrate, and suffered from frequent rollbacks and delays. Code took up to three days to reach internal users and anywhere from 10 to 18 weeks for full deployment. Today the cadence is two weeks. A single Technical Project Manager with two part-time engineers handles most of the release process, debugging is routed quickly to the owning team, code reaches internal users within one business day and everyone else within two to four weeks. This post breaks down the old process and what changed.

The Post-2015 Bottleneck

In 2015 the deliberate eight-week cycle existed to prevent shipping insecure or woefully broken software through a slow and imperfect auto-update channel. The costs of that caution were buried in operations work:

  • Integrating new code. The Release Engineering team devoted between one and three full-time people to the infrastructure that ingested commits.
  • Making builds. A "Primary On-Call" engineer started each day by picking a commit, manually tagging translations and versions, pushing to build machines, troubleshooting failures, smoke-testing the binaries, adjusting internal config to serve them, and writing announcement emails or forum posts. Days ended with triaging bug reports and occasionally rolling back the config and repeating the whole sequence.
  • Troubleshooting issues. A second Sync On-Call handled sync-specific bugs. Together with the Primary On-Call, they investigated or dispatched problems—often spending days tracing root causes across the three live versions: internal "office" builds, the external "forums" beta, and "stable." New engineers without institutional context struggled with this undocumented workflow.
  • Rolling out. A third Release Manager stewarded a specific version over its full eight-week life, tracking a matrix of deadlines: feature freeze, string freeze (for translations), code freeze, a manual QA pass, and finally the staged rollout. Any significant issue halted the train and forced partial re-runs of the cycle. Missing a deadline meant waiting a full eight weeks, which in turn encouraged risky last-minute changes with unpredictable side effects.

Only about 30 engineers worked on the Desktop Client, meaning a tenth of the team was perpetually on release duty. The environment was so full of undocumented context that part-time contributors from other teams were non-viable. The tooling and process did not scale, and something had to change.

Current Cadence

Major versions now ship on a two-week cadence, stewarded by one Technical Project Manager and two rotating part-time engineers—one for release troubleshooting, one for build system health. Internal builds are generated and deployed automatically every day. External builds are automatic, scheduled, or created as needed. Roughly 90 engineers contribute to each release, covering new features over dozens of teams.

High-Level Strategy

The Desktop Platform Team pursued two complementary themes to speed up iteration without sacrificing quality:

  • Reduce KTLO. "Keep The Lights On" overhead requires precision, not creativity. It is tedious, error-prone for humans, and grows linearly with release velocity—so it had to be automated or retired.
  • Automatically affirm quality. Auto-updates hit dozens of platforms and unpredictable user environments, so manual smoke tests only go so far. The goal became to catch edge cases programmatically before release and to monitor production behavior automatically afterward.

The specific steps in this effort were not tackled sequentially; they were developed in parallel. Each one multiplied the benefit of the others. Details on the exact tools and processes follow in the next part of this series.

Tuning Continuous Integration for the Desktop Client

Dropbox has used Continuous Integration (CI) organization-wide since early 2016. Every commit to a mainline branch triggers a suite of tests and builds, coordinated by Dropbox's open source tool, Changes. The tool's interface shows each commit as a bar—green for passing, red for at least one test failure, and black for system errors—with the bar's height representing job duration.

The expectation was that engineers would only commit code that passed the full test suite locally. When a build went red, the On-Call engineer for that area acted as "build cop," identifying the breaking change, asking the author to fix it, or backing out the commit. This troubleshooting was time-consuming, and while the suite stayed red, engineers committing new code received failure emails. Because these failures were often unrelated to their changes, trust in the system eroded. Engineers stopped investigating every failure, and multiple issues could pile up before being untangled.

To automate the job of keeping the mainline green, Dropbox built a "Commit Queue" (CQ). Engineers submit new commits to the CQ, which runs the test suite with the commit incorporated. Only passing commits are permanently added; failures are rejected and the author notified. The CQ also runs tests across a wider range of environments than a developer's laptop. A similar queue had existed for the Server codebase since 2015, but applying it to the Desktop Client required two things.

Setting the Stage: Git Migration and Organizational Change

The Server codebase had migrated from Mercurial to Git in 2014, and the tools built for it since then only supported Git. Rather than invest in supporting Mercurial workflows, Dropbox decided to migrate the Desktop Client repo to Git as well. This brought two benefits: it let engineers leverage existing tooling, and it removed the friction of working across two different version control systems daily.

This migration coincided with a broader reorganization. At the start of 2016, Dropbox's Engineering, Product, and Design (EPD) organization shifted from teams organized by platform ("Desktop", "Web", "Android") to explicit "Product" and "Product Platform" groups. This allowed for dedicated investment in platform goals like performance and ease of development. It also placed engineers facing similar problems across different codebases in the same organization, encouraging them to share and reuse solutions.

Reducing Flakiness Before Blocking Commits

A commit queue is only as useful as the tests it runs. If a build fails for reasons unrelated to the code change, developers waste time waiting or preemptively retrying, which adds load and can cause cascading failures. The two categories of such failures are infrastructure flakiness (issues in the systems running the tests) and test flakiness (non-deterministic test failures).

Test Flakiness

Consider a test that fails non-deterministically about 10% of the time due to a race condition. Run once in the CQ, it will likely pass, only to fail every ten builds later. This causes red builds post-commit and occasionally blocks unrelated changes—a poor developer experience as flaky tests pile up.

Flakiness can come from a poorly written test with too-short a timeout, or from state left over by a previous test interfering with the current one. Since tests run in random order, the latter can manifest as random failures across the suite. Sometimes the feature under test itself is flaky—a window that should open after a button click only does so sometimes. Categorizing these failures is challenging for both humans and automated tools.

To identify flaky tests, Dropbox configured the CQ to run any new or edited test many times before accepting the change, rejecting it if any run fails. This forces the authoring engineer—who has the most context—to address the issue. Once a test passes the CQ, it is run up to three times post-commit and on unrelated changes, with any single success counted as "green." A separate tool, Quarantine Keeper, monitors tests for failures that occur too often. It removes offending tests from the main pool, files a bug against the owning engineer to fix it, and re-adds the test once repaired. The goal is to keep signal-to-noise high: one-off random failures shouldn't alarm, but consistent intermittent failures must be eliminated.

Infrastructure Flakiness

Fixing infrastructure flakiness was a systematic effort of cataloging and addressing failure types, adjusting timeouts, and adding retries. The most impactful change was implementing a full-job retry for every job. Any failure for infrastructural reasons within the first 30 minutes was retried, up to three times total, which significantly improved build greenness.

Dropbox also had to treat its test infrastructure as a distributed system, measuring and anticipating computational needs. The Desktop Client is officially supported on over a dozen operating system versions across Windows, Mac, and Linux. A key goal was increasing CQ coverage across these OSes to reduce the pain of manual testing, but each additional configuration expanded the surface area for flakiness, potentially eroding trust even as other sources were being reduced. Enabling more job types than the infrastructure could support risked pushing machines past their limits.

One particularly instructive scaling incident involved rsync and git clone commands mysteriously hanging on Windows and Linux VMs—but not on Mac OS X—and more frequently as more code was committed. The root cause was that the Windows and Linux VMs shared the same network-attached storage, while Mac OS X used different hardware. As more test types were added, the disk I/O capacity of that shared storage was maxed out. rsync calls that copied files from one VM to another on the same machine would overload it, causing failures. The fix was to remove some test types from the Commit Queue until the storage hardware could be upgraded.

The article continues with a discussion of the technological and process changes needed to speed up making new builds and releasing with confidence.