Using Dropbox's Delta API: Lessons Learned From Site44

Keeping a Local Copy in Sync

Site44 hosts static websites directly from Dropbox, but it deliberately avoids fetching files from Dropbox on every request. Instead, the service maintains a local mirror of each user's content and serves from that copy. The challenge is keeping that mirror accurate as users edit files in Dropbox. The solution is the /delta endpoint, which Site44 has relied on since it was in beta in February 2012 and which has been in production since March of that year.

The key design point of the delta API is that it returns a set of instructions for updating local state, not a log of user actions. Each call returns delta entries containing a path and metadata; a null metadata value means the path was deleted, while a present metadata object describes the current state of that path. The first call with no cursor returns entries for every accessible file in the user's Dropbox (ideally just an app folder). Subsequent calls return only what has been created, modified, or deleted since the cursor was issued.

To make that work, every response includes a cursor, which you pass on the next call to ask, effectively, "What has changed since this cursor?" There is also a reset field. When set to true, it signals that all local state must be discarded before processing the returned entries. This is normal for the initial call but otherwise rare; still, production code must handle it explicitly.

Polling Without the Latency Penalty

Because Site44's value depends on noticing file changes quickly, the delta API's polling model presents a tradeoff. Polling infrequently keeps API load low but means a user's edits could take a while to appear on their site. Polling aggressively enough to catch changes within a second of them happening is impractical, both in terms of request volume and load on Dropbox's servers.

Site44's approach is hybrid. The service polls as frequently as is reasonable, but it also triggers an on-demand poll whenever a visitor refreshes a site in a browser. That way, a user who edits a file in Dropbox and then reloads their website immediately sees their changes, without Site44 needing to maintain a constant high polling rate. In practice, this has kept latency low enough that users generally see changes right away.

Simpler API Usage Elsewhere

The delta API is slightly more complex than other Dropbox endpoints, but it simplifies the rest of an application's integration. Without it, keeping a mirror in sync would require periodically walking the entire contents of a user's Dropbox and comparing metadata to detect changes. With delta, Site44 only needs to retrieve files that actually changed, using the /files endpoint to fetch new content when a delta entry indicates a change. The result is a small number of API calls that still maintain a fully consistent local copy.

This pattern, combined with the timed and on-demand polling strategy, gives Site44 the two properties it needs most: low latency for reflecting user edits and minimal load on Dropbox's API. For a hosting service built on top of a cloud storage provider, that kind of synchronization support is what makes the product practical at all.