Designing for Testability

When we set out to rewrite the Dropbox sync engine, we knew that a robust testing strategy would be critical—but that strategy would only work if the new system was built with testability in mind from the start. Our legacy system, Sync Engine Classic, offered clear lessons on what made a system difficult to test and what we needed to do differently in Nucleus, our new engine.

Protocol and Data Model

Sync Engine Classic’s server-client protocol was designed when Dropbox was a simpler product, long before sharing, comments, or enterprise-scale teams. Over time, that protocol allowed the client to enter sync states that were far too permissive for effective testing. For example, the client could receive metadata for a file at /baz/cat before receiving its parent directory metadata at /baz. The local SQLite database had to represent this orphaned state, and any component processing filesystem metadata had to handle it. As a result, it was impossible to distinguish serious inconsistencies—like orphaned files—from acceptable transient states.

Nucleus takes a different approach. One of our core architectural principles is "Design away invalid system states." The protocol now reports a critical error before the client can enter such an orphaned state, so the persisted data model and higher-level components never need to consider that possibility. This gives us a strict, testable invariant: no file or folder can exist, even transiently, without a parent directory in the database.

The data models of the two systems are fundamentally different. Sync Engine Classic persisted the outstanding work required to sync each file—for instance, whether a file needed to be created locally or uploaded to the server. Nucleus instead persists observations. Rather than representing outstanding sync activity directly, it maintains three trees, each an individually-consistent filesystem state from which the correct sync behavior can be derived:

  • The Remote Tree represents the latest state of the user’s Dropbox in the cloud.
  • The Local Tree represents the last observed state of the user’s Dropbox on disk.
  • The Synced Tree represents the last known "fully synced" state between the Remote Tree and Local Tree.
Sync Engine Classic data model
Nucleus data model

The Synced Tree is the key innovation. You can think of each node in it as a merge base, similar to version control. It lets us derive the direction of a change: did the user edit the file locally or was it edited on dropbox.com? Without it, we couldn’t distinguish a remote addition from a local deletion. This data model is highly testable because it makes the system’s goal easy to express: converge all three trees to the same state. No matter how the trees are configured at the start of a test, they must still converge. That’s an invariant we can enforce.

Nucleus also represents nodes by a unique identifier, whereas Sync Engine Classic keyed nodes by their full path. In the legacy system, a rename was a delete at the source path and an add at the destination—exploding a folder move into O(n) operations for all descendants and causing the user to see two inconsistent subtrees during the process. In Nucleus, a move is just an update to the moved node’s attributes in the database, replicated to the filesystem with a single atomic move. Testing enforces another invariant: a moved folder is visible in exactly one location.

Concurrency Model

Sync Engine Classic’s concurrency model made testing extremely challenging. Components freely forked threads internally, leaving execution order to the OS. Coordination relied on global locks, with hard-coded timeouts and backoffs. Tests frequently resorted to arbitrary sleeps or invasive mocking to serialize execution manually—resulting in flakiness and frustrating debugging.

In Nucleus, nearly all code runs on a single "control" thread. Network I/O, filesystem I/O, and CPU-intensive work like hashing are offloaded to dedicated thread pools. But for testing, we can serialize the entire system, running asynchronous requests on the main thread instead of the background. This design is the key to determinism and reproducibility in our randomized testing systems.

Randomized Testing

Dropbox runs on hundreds of millions of user machines, each a different environment. Even the most obscure corner cases appear in the wild. Human-written unit and integration tests can’t anticipate every edge case—randomized testing is what gives us confidence that the system is truly robust.

We’re aware that "randomized testing" often brings to mind systems that fail intermittently and can’t be reproduced. Sync Engine Classic had those. So with Nucleus, we set a challenging requirement: all randomized testing frameworks must be fully deterministic and easily reproducible.

All our randomized testing systems share the same structure:

  1. Generate a random seed at the start of a test run.
  2. Instantiate a pseudorandom number generator (PRNG) with that seed.
  3. Run the test using that PRNG for all random decisions—initial filesystem state, task scheduling, network failure injection.
  4. If the test fails, output the seed.

Every night, tens of millions of randomized test runs execute. Generally, they’re 100% green on the latest master. When a regression slips in, CI automatically creates a tracking task for each failing seed, along with the commit hash. An engineer can then add logging inline and re-run the test locally—it’s guaranteed to fail again.

This guarantee requires Nucleus itself to be fully deterministic for a given PRNG input. For example, Rust’s default HashMap uses a randomized hashing algorithm to resist denial-of-service attacks via hash collisions. Nucleus doesn’t need that protection—an adversarial user could only degrade their own performance—so we override with a deterministic hasher. The commit hash is also an essential input: a change in code changes the execution path.

Two randomized testing systems protect Nucleus: CanopyCheck tests our ability to bring the three trees into sync, and Trinity tests the engine’s concurrency at large. We’ll examine both in turn.

Planner coverage through randomized testing

CanopyCheck is a randomized testing framework built to exercise the planner — the component at the heart of Dropbox’s sync engine. The planner takes three trees as input (the server state, the local disk state, and a merge-base representing sync progress so far) and outputs operations that incrementally converge those trees. Those operations are batched by the planner into groups that can run concurrently, respecting dependencies like creating a parent directory before its children.

Correctness of the planner is critical. Handwritten tests cover hundreds of tree configurations, but the space of possible inputs is effectively unbounded. CanopyCheck fills that gap by generating random inputs and asserting stronger invariants than a typical end-to-end sync test would allow.

A basic handwritten unit test for the planner already demonstrates how heavily the sync engine relies on Rust’s macro system. The planner_test! macro initializes the trees, runs the planner, updates the trees to reflect each operation’s result, and checks that the final state satisfies both the intended convergence property and internal consistency rules.

#[test]
fn test_remote_add() {
    planner_test! {
        initial synced, local: {
            /foo: 1 = Directory,
            /foo/bar: 2 = File contents: hello,
            /baz: 4 = Directory,
        }
        initial remote: {
            /foo: 1 = Directory;
            /foo/bar: 2 = File contents: hello,
            /foo/fum: 3 = File contents: world,
            /baz: 4 = Directory,
        }
        final remote, synced, local: {
            /foo: 1 = Directory,
            /foo/bar: 2 = File contents: hello,
            /foo/fum: 3 = File contents: world,
            /baz: 4 = Directory,
        }
    }
}

Note: The above test verifies that the planner emits an appropriate plan to download a remotely added node.

Generating meaningful random inputs

Simply picking three unrelated random trees would fail to exercise interesting scenarios — if the trees have disjoint file sets at non-overlapping paths, the planner never has to handle deletes, edits, or moves. Instead, CanopyCheck starts with one randomly generated tree and then applies random perturbations to derive the other two. This keeps the test cases rooted in realistic sync conflicts while still covering a broad space of configurations.

Run loop and invariants

A single CanopyCheck test follows a straightforward loop:

  1. Ask the planner for a batch of concurrent operations.
  2. Randomly shuffle the operations to verify that order doesn’t matter.
  3. Update the trees as if each operation succeeded — no actual I/O or concurrency involvement.
  4. Repeat until the planner returns no further operations.

If all goes well, the three trees converge after a finite number of iterations. That simple loop lets CanopyCheck verify a set of invariants that go well beyond “the planner doesn’t crash.”

Termination

The framework uses a heuristic cutoff of 200 planning iterations to detect potential infinite loops. Because the trees only change through the planner’s operations, any cycle that keeps producing new operations is immediately visible as a non-terminating run.

No panics

The planner (and Nucleus in general) is heavily annotated with assert! calls as a defensive measure. CanopyCheck exercises those assertions across a wide range of inputs, catching assumptions that would otherwise lead to runtime panics in production. In fact, CanopyCheck was able to reproduce the Archives/Drafts/January directory cycle bug described in the previous post on rewriting the sync engine — applying a local move and a remote move together created a cycle, which triggered an assertion in the tree data structure and failed the test.

Sync correctness

Simply requiring all three trees to be equal at the end of a test would allow catastrophic bugs to slip through: for example, a planner that always deletes everything on both the server and disk would converge trivially. So CanopyCheck enforces additional invariants that are strong enough to matter but simple enough to apply universally.

Some invariants are directly derived from the initial tree configuration. For instance, if a file exists only in the Remote Tree and not in the other two, it must be present in all three trees after the test run — the planner may not delete unsynced data from the server. The symmetric invariant applies for locally added files that must be uploaded. Another invariant guards Smart Sync behavior: a locally added file must remain downloaded as long as it hasn’t been moved into an “online only” folder, preventing the planner from prematurely evicting local contents.

Minimization of failing cases

The framework’s name references QuickCheck, the Haskell property-based testing library, and like QuickCheck it minimizes failing cases. Because the input format is just three trees, CanopyCheck can shrink a failing test by iteratively removing nodes from the initial trees and re-running the planner. The minimized input often reveals the bug at a glance — for example, missing handling for adding a node under a parent that was moved remotely.

Randomly generated inputs tend to be convoluted, so this minimization step is crucial for debugging. Tracking dozens of nodes across three trees is impractical for a human, and the real problem can easily be obscured by unrelated noise. Removing that noise makes it feasible to diagnose issues quickly.

Trinity: Probing Nucleus for Race Conditions

CanopyCheck validates the planning algorithm well, but the rest of Nucleus needs scrutiny too. The most insidious sync bugs are race conditions that only surface when operations collide in just the wrong order. A typical scenario:

  • Ada deletes foo in a shared folder.
  • Grace's sync engine learns foo should be deleted.
  • Grace writes new data into foo at the same time.
  • Grace's engine deletes foo, losing her change.

Trinity is the framework that catches these races before they reach users. It works by initializing the external backend state (the user's Dropbox on dropbox.com) and the filesystem state (the local Dropbox folder), then instantiating Nucleus just as a real client would when linking an existing folder.

How Trinity Drives Execution

Trinity and Nucleus alternate on the main thread. Until Nucleus reports a synced state, Trinity aggressively agitates the system: it modifies local and remote filesystems, intercepts Nucleus's asynchronous requests and reorders responses, injects filesystem errors and network failures, and simulates crashes. Once Nucleus reports synced, Trinity verifies the system is consistent. It also re-runs the same test with the same seed to confirm the final state is reproducible.

Nucleus is parameterized over compile-time dependencies, which lets Trinity inject wrapped versions of the filesystem, network, and timer. These wrappers intercept asynchronous requests and serialize responses, proxying through to concrete implementations when Trinity decides to satisfy a request.

Filesystem mock. Trinity swaps the native platform filesystem for an in-memory mock. This lets Trinity inject failures into any operation, reorder requests, and even simulate crashes by snapshotting and restoring filesystem states. The in-memory approach yields roughly a 10x performance boost, enabling far more random exploration.

Network mock. The entire server backend—metadata database, content storage, notification services—is replaced with a Rust mock. Trinity can arbitrarily reorder, delay, or fail any RPC. The mock emulates all server-side services Nucleus depends on, closely mimicking production behavior.

Timer mock. Nucleus uses a generic, mockable timer. If Nucleus requests a 5-minute timeout for a placeholder download, Trinity can intercept that request, fast-forward time, and fire the timeout at will.

Concurrency Via Futures

Because Trinity sees all asynchronous activity, it buffers intercepted requests from each mocked component. When Nucleus yields control back, Trinity randomly decides which requests to satisfy, which to fail, or whether to perturb the external state itself.

Nucleus itself is a Rust Future, and Trinity acts as a custom executor that interleaves the future's execution with its own logic. Nucleus is composed of worker futures in a tree structure; the upload worker, for instance, manages an unordered set of futures representing concurrent network requests. Each poll() call lets subsystems make progress, and the top-level future has type impl Future<Output = !>, meaning it can only ever return Poll::Pending—but each poll still advances the system.

Trinity polls that top-level future while also polling mocked filesystem and network requests it intercepted. When Nucleus blocks on outstanding requests, Trinity uses its PRNG to choose which succeed or fail. This both simulates production concurrency and amplifies the probability of rare execution orderings.

What Trinity Can't Cover

Mocking away external nondeterminism comes with tradeoffs.

Native filesystem interactions. The in-memory mock means no coverage of OS-specific logic for permissions, extended attributes, or Smart Sync placeholder hydration. Trinity has a "native" mode for this, but it runs roughly 10x slower, limiting how many seeds it can test. Native mode also serializes filesystem calls to preserve reproducibility, which real users don't experience. And Trinity can't reboot a machine mid-test, so crash durability via fsync placement remains untested.

Network protocol. A mock server may drift from production behavior, hiding client-server protocol bugs. Heirloom, a separate suite, tests against a real Dropbox server using the same deterministic seed principle. It runs about 100x slower than Trinity, so it sacrifices some determinism and throughput—but it covers the protocol layer Trinity can't.

Minimization. Because Trinity mocks less than CanopyCheck, it can't minimize failing cases as easily. Perturbing a test's initial state may change request scheduling and invalidate a hard-won failing seed. The team is exploring options like decoupling the global PRNG into independent ones; for now, developers analyze scenarios manually, adding logging and filtering traces by hand.

Trinity can't cover everything, so other higher-level CI suites provide weaker but valuable end-to-end coverage. But for Nucleus itself, Trinity delivers a confidence level that the Sync Engine Classic rewrite never could have achieved without it.