Why the old sync engine had to go

Dropbox’s desktop client has always revolved around a single piece of code: the sync engine that keeps the local folder in step with the cloud. That engine, now dubbed “Sync Engine Classic,” dates back to the product’s 2008 beta. It served hundreds of millions of users well, but by 2016 its architectural limits had become a serious drag on progress. After four years of work, we’ve replaced it with a new engine, codenamed “Nucleus,” now rolled out to every Dropbox user.

A full rewrite is rarely the right call. It usually means stalling feature work, porting over hard-won fixes, and risking the stability that years of production hardening provided. But Sync Engine Classic had reached a point where incremental changes were no longer viable. The problems were deep, and they boiled down to a few fundamental issues with the system’s data model, its testability, and its concurrency architecture.

The hard parts of sync at scale

File sync looks deceptively simple. In practice, it’s a distributed systems problem with a twist: clients go offline for long stretches and reconcile when they return. For most distributed systems, network partitions are the anomaly. For Dropbox, they’re the norm. That requirement alone makes durability tricky. Moves, for example, were represented in Sync Engine Classic as a delete at the old location plus an add at the new location. If a transient network hiccup delivered the delete but not the add, users would see their file vanish from the server and all other devices—even though they had only moved it locally.

The difficulty multiplies across the wide range of environments we support. Dropbox runs on Windows, macOS, and Linux, over many different filesystems, each with subtle behavior differences. Kernel extensions and drivers can change how the OS behaves within a single platform, and applications above Dropbox all use the filesystem in ways that may not match its spec. Guaranteeing durability in every one of those environments means understanding each implementation, working around its bugs, and sometimes reverse-engineering it. Rare filesystem bugs surface only in large populations, so the promise of “just working” everywhere and the demand for strong durability guarantees are fundamentally in tension.

Testing file sync properly is just as hard. The state space is astronomical: a shared folder can have thousands of members, each with a different view of the filesystem, different pending uploads, and different download progress. The set of valid actions from any state is enormous too—uploads and downloads run concurrently, and each file transfer can involve parallel content chunks, disk writes, and local file reads. Catching regressions before they hit production is vital, but the testing problem is huge.

Even defining correct behavior is not straightforward. Consider two users working offline in a folder structure with three nested directories: “Archives” inside “Drafts” inside “January.” Alberto moves “Archives” into “January.” Beatrice moves “Drafts” into “Archives.” When they both come back online, applying those moves directly creates a cycle. The old engine resolved this by duplicating each directory and merging the trees. Nucleus keeps the originals, and the final state depends on which user’s move uploads first. So the same input can produce different valid outcomes depending on timing—a specification problem with no simple answer.

What went wrong with Sync Engine Classic

Despite years of hardening and a strong team of sync experts, Sync Engine Classic was unhealthy in three ways that mattered most.

First, the data model. It was designed before sharing existed, and files had no stable identifier that survived a move. Consistency guarantees were loose, and we spent hours debugging issues that were theoretically possible but supposedly “extremely unlikely.” When the foundational nouns of a system can’t be changed incrementally, you run out of small improvements quickly.

Second, testability. Sync Engine Classic’s permissive data model meant we couldn’t write meaningful stress tests—there were too many undesirable but still legal outcomes to assert against. We relied on slow rollouts and field debugging instead. A strong data model with tight invariants makes testing straightforward, because you can always check whether the system is in a valid state.

Third, concurrency. Sync is heavily parallel, but Sync Engine Classic’s threading-based architecture handed scheduling decisions to the OS, making integration tests non-reproducible. In practice, we guarded against that with coarse-grained locks held for long stretches, sacrificing parallelism to keep the system understandable.

The result: shipping any change to sync behavior required an arduous rollout, complex inconsistencies kept surfacing in production, and onboarding new engineers took years. Incremental performance work failed to meaningfully scale the number of files the engine could manage. The rewrite wasn’t a gamble—it was the only path forward.

When Incremental Work Isn’t Enough

Before committing to a rewrite, it’s worth asking whether targeted improvements can get you where you need to go. Refactoring poor code into clearer modules is usually possible without starting over, and we did a lot of that with Sync Engine Classic, adding MyPy annotations incrementally to catch more bugs at compile time. But refactoring alone can’t change the underlying data model, and eventually that becomes the bottleneck.

Performance work is similar. A profiler will often show that most time is spent in a small amount of code, and optimizing those hotspots can yield big wins without a rewrite. We had a team working on performance and scale for months with strong results on file content transfer. But reducing memory footprint—say, increasing the number of files the system could manage—proved much harder to achieve incrementally.

Checklist for a Rewrite Decision

These questions helped us decide whether a rewrite was justified, and they’re worth considering for any system in a comparable position.

Have You Exhausted Incremental Improvements?

  • Refactor into better modules? Code quality alone isn’t a strong reason to rewrite. Untangling modules and renaming variables can be done step by step, but if the core primitives stay the same, you’re still stuck with the original design.
  • Optimize hotspots? Many performance issues aren’t fundamental. Profiler-driven optimization can deliver meaningful gains, as it did for us on content transfer. But some problems, like memory growth that limits scale, are inherent to the architecture.
  • Deliver incremental value? Even if a rewrite is necessary, look for ways to ship intermediate value. That validates early decisions, keeps momentum, and reduces the cost of slower feature development.

Can You Actually Pull Off a Rewrite?

  • Deeply understand the old system. Writing new code is easier than reading existing code, but you can’t rewrite what you don’t understand. The old system runs in production and carries years of accumulated wisdom; treat it as a resource, not an obstacle.
  • Have the engineering hours. A rewrite to full feature completeness takes serious time. You need both the capacity and the domain experts who know the current system, plus an organization that can sustain a long project.
  • Accept slower feature development. We didn’t freeze feature work on Sync Engine Classic, but every change there pushed the new engine further out. We chose a few projects to ship and allocated resources carefully, while investing in telemetry to keep maintenance overhead low.

Do You Know What You’re Building Toward?

  • Why will it be better the second time? A rewrite should be driven by changing requirements, not just past pain. Our decision was forward-looking: supporting collaborative work at scale demanded a more flexible engine.
  • What are your design principles? Starting fresh is a chance to reset technical culture. Given our experience, we emphasized testing, correctness, and debuggability from day one, and encoded those values in the data model. Writing these principles down early paid off repeatedly.

What Nucleus Looks Like

We ultimately built Nucleus in Rust, which proved to be a force multiplier. More than raw performance, Rust’s ergonomics and emphasis on correctness helped manage sync’s complexity. We encode invariants directly in the type system, letting the compiler catch mistakes we’d previously have found only in production.

  • Almost all code runs on a single “Control thread,” using the Rust futures library to schedule concurrent actions. Network I/O goes to an event loop, hashing and other expensive work to a thread pool, and filesystem I/O to a dedicated thread. This narrows the scope developers must consider when adding features.
  • The Control thread is fully deterministic when inputs and scheduling are fixed. We exploit that for pseudorandom simulation testing: with a seed, we generate random filesystem states, schedules, and perturbations, then run the engine to completion. Any correctness failure can be reproduced from the original seed, and we run millions of these scenarios daily.
  • We redesigned the client-server protocol for strong consistency. The protocol guarantees the client and server agree on the remote filesystem before any mutation. Shared folders and files have globally unique identifiers, and clients never see transient duplicated or missing states. Folders and files support atomic moves regardless of subtree size. Any discrepancy between client and server views is treated as a bug.