Why Dropbox Bet on a Single Git Repository
Dropbox moved from Mercurial to Git in 2014, primarily for better local performance and because Git had become the industry standard that most new engineers already knew. At the time, backend code lived mainly in a monolith plus several dozen separately versioned repositories. That arrangement caused recurring pain: engineers wrote integration tests for smaller services inside the monolith's repository, and when a service's code changed, tests often broke at HEAD. With no dependency pinning, failures were hard to diagnose, release engineers spent excessive time debugging, and engineers lost trust in CI results.
The team briefly used a "super repo" that recorded a commit for every change to server-related repositories, providing a global ordering of changes. But the simplest fix was to merge all relevant repositories into one. At roughly 50,000 files, the combined size was manageable, and Git performance was projected to be acceptable for several years. The merge also simplified code sharing, large-scale refactors, integration with Bazel, and automatic bisects and reverts.
The Performance Wall
As predicted, Git performance degraded as the repository grew—apparently linearly with file count. By 2017, common operations like git status were slowing down. The problem was most acute on macOS, the primary development platform for Dropbox engineers. On Linux, git status was 5–10x faster.
The bottleneck was clear: many Git operations ran the lstat syscall on every file in the repository to check whether it was up to date, even though most developers modify only a small subset of files. The team also took a simple precaution early on, adding a pre-receive hook to reject large files that would otherwise degrade performance.
Controlling the Client Stack
To measure and fix the problem, Dropbox needed to control which Git version ran on macOS developer machines. They created a small fork of Git that recorded timing for operations like git status and git pull, auto-provisioned it on developer laptops, and adjusted $PATH so their Git took precedence over system or Homebrew installs.
At the time, git status averaged over two seconds. The team turned to improvements in open-source Git:
- fsmonitor: a Git hook that delegates file-change detection to Watchman, a daemon that watches and buffers filesystem changes. Git operations that need local filesystem state—like
add,status, anddiff—query fsmonitor and update the index accordingly. - Split index mode: instead of rewriting the entire index (sorted by file path) on every
git add, deltas are appended to a split index file and consolidated periodically. This makes index writes dramatically faster for large repositories. - Untracked cache: caches directory mtimes so Git can skip traversing directories that haven't changed.
An initial deployment of fsmonitor and Watchman to developer machines hit both performance bugs (Git sometimes ignoring fsmonitor data) and correctness issues (wrong results from git status). Some bugs were fixed upstream, but the effort stalled when team members with Git expertise moved on.
Enabling Optimizations by Default
By the second half of 2019, the repository had grown past 250,000 files, and Dropbox renewed its focus on performance. Upstream bugfixes and a fix for a major performance issue made the difference this time. The team's principle was that developer tools should be configured correctly by default. So they shipped a wrapper around Git that automatically enabled fsmonitor and relevant configs—but only for a whitelisted set of repositories. Unlike Microsoft's Scalar tool, which was similar in concept, this required no extra user action or new commands to learn.
The results showed substantial reductions in p50 and p90 durations for common Git operations. While not as fast as Git in a small repository, the improvements were significant and, importantly, no longer grew linearly with repository file count. The entire effort required fewer than 200 lines of custom plumbing code and no additional services or large virtual filesystems to maintain.
Takeaways
With some advance preparation—blocking large files and setting the right client flags—Git can perform reasonably well on a large monorepo with almost zero ongoing maintenance. External factors like fsmonitor, Watchman, split index, and untracked cache are all available in open-source Git. The choice between a monorepo and multiple repositories involves many trade-offs, but version control scalability need not be the deciding factor.
The broader open-source community continues to explore performance improvements for large repositories, including work on push performance and protocol enhancements. Dropbox's experience suggests that a monorepo is viable at significant scale with relatively modest engineering investment.
which git
/opt/dropbox-override/bin/git
.png)
