Streaming File Synchronization

How Dropbox Sync Works Under the Hood

Dropbox's sync architecture relies on a clear division between metadata and content. On the server side, files are organized into namespaces—each user has a root namespace, and every shared folder is a namespace that can be mounted in one or more root namespaces. Within a namespace, every file and directory is uniquely identified by its relative path.

File contents are split into 4MB blocks (the last block may be smaller). Each block is hashed with SHA-256, and a file is represented by a list of those hashes, called a blocklist. The blocklist is what the metadata layer stores; actual content lives elsewhere.

Two server types handle these responsibilities:

• Block data servers maintain a key-value store mapping hashes to encrypted content. They have no knowledge of users, files, or how blocks fit together.
• Metadata servers hold the database of users, namespaces, and the Server File Journal (SFJ)—an append-only record where each row is a version of a file, keyed by namespace, path, blocklist, and a monotonically increasing Journal ID (JID).

The Original Two-Phase Protocol

Each desktop client tracks its position in the SFJ per namespace using a cursor (a JID). When a file appears on an uploading client, the sync process follows a strict sequence:

1. The client attempts to commit the blocklist for the (namespace, path) pair. If the hashes are unknown or access is denied, the server responds with need blocks, indicating which blocks are missing.
2. The client uploads the missing blocks directly to a block server via store_batch calls. These may be multiple requests due to byte limits per request.
3. The client retries the commit. On success, the metadata server appends a new row to the SFJ—the file officially exists.

Only after the commit succeeds can other clients learn about the file. A downloading client, notified via longpoll connections, issues a list call with its cursors to fetch new SFJ entries. It then checks whether blocks exist locally (in existing files or a deleted-file cache), and if not, downloads them from a block server using retrieve_batch, again potentially across multiple requests.

Finally, the blocks are reassembled into a file on the local filesystem. The full pipeline—sniffing, hashing, committing, storing, listing, retrieving, and reconstructing—runs in separate threads, and compression plus rsync minimize the size of batch transfers.

Removing the Commit Bottleneck

For large files, sync latency is dominated by network time in store_batch and retrieve_batch. Critically, the uploading client must finish all store_batch calls before the SFJ commit, and the downloading client's list call only returns meaningful results after that commit. But the downloading client's block retrieval doesn't actually depend on the commit—it only needs to know which blocks to fetch.

That insight led to Streaming Sync, an optimization that lets a downloading client begin prefetching blocks while the uploading client is still transmitting them. Ideally, the downloader stays just one block-server network call behind the uploader.

Implementation Changes

Protocol. The uploading client's behavior is unchanged. The downloading client, however, now receives more than just new SFJ rows on list—the response also includes streaming-sync prefetchable blocklists, which correspond to not-yet-committed files.

Metadata server. The server retains state from an initial failed commit, but not in a persisted table—this data lives in memcache. The entry mirrors an SFJ row except it is versionless and has no JID. Memcache writes occur on failed commit calls; reads occur on list calls; deletes happen on successful commits (or memcache evictions).

Client. Clients maintain a prefetch cache for blocks that don't yet correspond to SFJ rows. After a list, the client queues prefetches into this cache, which on new client versions is stored in the `.dropbox.cache/prefetch_cache/` directory.

Handling Failure Cases

Streaming Sync must remain robust when an upload never completes. Since memcache entries can be mutated, expire, or be evicted mid-prefetch, the client needs fallback behavior when the server cannot verify that a block is eligible. This required adding special return codes to the store protocol to signal that condition. Server-side memcache entries must expire without thrashing, and the client's prefetch cache must be purged periodically to keep it from growing without bound.

Measured Impact

Streaming Sync only helps files large enough to require multiple store/retrieve requests, so the feature is limited to large new files. Theoretical improvement approaches 2x for very large files with symmetric upload and download bandwidth, but in practice, the slower side of the connection is the limiting factor. In tests across two machines on the same network (~1.2 Mb/s upload, ~5 Mb/s download), sync time improved by approximately 25%.

Rollout metrics will track the number of prefetched blocks, prefetch cache size, and memcache hit/miss rates.

Client-side support for Streaming Sync appears in beta version 2.9 and stable version 2.10 of the Dropbox desktop client; server-side support rolled out gradually after that.