Evolving our infrastructure through the messaging system model in Dropbox

From bespoke queues to a unified async model

Dropbox’s asynchronous platform handles more than 400 product use cases and routes over 30 million tasks every minute. It also powers change data capture (CDC), relaying changes in the underlying storage system to downstream lambdas and processes. A few years ago, that platform was buckling under its own weight: multiple bespoke systems developed for specific domains overlapped in function but diverged in reliability, operability, and developer experience.

The fix was not another rewrite, but a standardized abstraction. Inspired by the OSI model, Dropbox built an open messaging system model (MSM) that divides the platform into five logical layers. This separation lets frontend interfaces, lambda functions, event schedulers, and event routers work uniformly across use cases with different delivery guarantees and data sources—including CDC. The model is central to Dropbox’s push into AI features like Dropbox Dash.

Where the old platform fell short

As of 2021, the async landscape was a patchwork of systems, each tuned to a specific product or process. They handled streaming events for file uploads and edits, security, abuse prevention, machine learning, and search indexing, plus CDC events generated by any mutation in the storage layer. These systems were built and maintained separately, and the costs of that fragmentation showed up in several distinct ways.

Developer efficiency was poor. Product engineers faced a steep learning curve and were expected to own operational concerns—capacity planning, releases, support—which slowed feature development dramatically.

Reliability was uneven. Each system carried its own service-level objectives (SLOs) for availability, latency, and recovery, yielding inconsistent behavior. The systems were not multi-homed, exposing many business use cases to serious risk in the event of a data center failure.

Operability was expensive. A mix of Kafka, Redis, and Amazon SQS underpinned different components, creating a stack that was costly to run and complex to support.

Scalability was tapped out. By early 2021 the system dispatched jobs to lambda functions across over 30 billion requests daily, and hitting the SLOs was becoming impossible. The delayed event scheduler had already hit its throughput ceiling, which meant every new use case had to pass strict screening before onboarding to avoid destabilizing production.

Lambda infrastructure did not align with Dropbox’s SOA. The consumer-side lambda architecture diverged from Dropbox’s service-oriented architecture guidelines. Release procedures lacked uniformity, compute clusters ran below peak efficiency, these was no autoscaling (engineers had to manually add capacity), and diagnostics did not fit existing Dropbox methodologies for investigation.

Extensibility was limited. The existing setup could not easily absorb new workflows, which became a blocker with the rollout of Cypress, the new filesystem architecture: the CDC pipeline could not distribute Cypress events to multiple subscribers within Dropbox.

A staged rebuild, not a clean slate

With more than 400 use cases already in production, a from-scratch platform was off the table. Instead, Dropbox rebuilt existing systems incrementally, reducing migration risk. The process targeted three goals.

Development velocity: simplify the async interface so engineers do not need to understand the full landscape before building; reduce operational burden with release practices that detect regressions and automatically roll back; and enable automatic compute scaling when a lambda falls behind on its event backlog.

A robust async foundation: unify the common patterns across the existing async systems into a single interface, and support new use cases without building whole infra from scratch by exposing extensible components and flexible APIs.

Cost and operational efficiency: retire redundant underlying systems where viable, and move lambda infrastructure onto the Dropbox SOA stack to gain compute efficiency, autoscaling, multihoming, and built-in monitoring.

The chief success metric became time to launch for product engineers—how quickly a new use case reaches production—while the platform team tracked oncall time as its own operational KPI.

Opening the model

The MSM played the key role in reorganizing the platform into a unified event-driven system capable of covering a wide range of asynchronous work, including CDC flows. Its layering gives Dropbox a common grammar for describing event pipelines—so that, regardless of the delivery guarantee or data source involved, developers work against the same abstractions rather than bespoke machinery.

Decomposing the Asynchronous Stack

To rebuild the asynchronous platform with the flexibility and extensibility it required, Dropbox first decomposed the system into its fundamental components. The resulting architecture maps cleanly onto what the team calls the messaging system model (MSM), a layered abstraction similar in spirit to the OSI model for networks. At the highest level, the system splits into three buckets: a customer-facing frontend, an internal orchestration tier, and a compute tier where work actually executes.

Drilling down, those three buckets expand into five operational layers: frontend, scheduler, flow control, delivery, and execution. Some responsibilities overlap between the customer and orchestration tiers, but the five-layer model gives engineers a precise vocabulary for discussing where work enters the system, how it is prioritized and routed, and where it ultimately lands.

Frontend

The frontend layer is the entry point for everything the asynchronous system handles. Two distinct groups of users interact with it. Product engineers call a publish RPC to enqueue events programmatically for one or more subscribers. Separately, systems such as databases or event sources push changes to objects, entities, or files into the queue to drive internal and external workflows.

Schema management lives here. The frontend maintains the schema registry and validates every event schema that passes through the system, ensuring published events match the contract subscribers expect. It also converts incoming formats — JSON, Proto, Avro — into a standardized internal format, typically protocol buffers. Finally, the frontend layer owns durability guarantees for every published event.

Scheduler

The scheduler is the engine that coordinates events for consumers. Its duties depend on the use case. For change data capture (CDC), it calls external data source APIs to fetch the relevant payload ranges for subscribers. For delayed execution, it holds events separately and triggers them at the desired timestamp through a process that watches the schedule and publishes when the time arrives.

Critically, the scheduler also preserves ordering guarantees. It maintains the execution order of events and ensures tasks reach subscribers in that sequence.

Flow Control

Flow control determines how work is distributed to subscribers, factoring in subscriber availability, task priority, and throttling conditions. In a CDC scenario, for instance, the layer dynamically adjusts the query rate dispatched to subscribers when it detects that a subscriber cannot handle the throughput or that the source backing CDC has signaled the scheduler client to slow down.

State management is also handled here. The layer maintains data structures tracking each event's status — pending, running, complete — and includes retry mechanisms for transient failures.

Delivery

The compute tier splits into two parts. The first is delivery (the routing layer), which directs the event out of the messaging infrastructure to the domain where a process or lambda function will handle it. That destination may sit within the same VPC as the messaging infrastructure or in a public cloud such as AWS or Azure. In push-based models, routing is the equivalent of last-mile delivery: small, precise, and easy to get wrong if any link is weak.

The routing layer is responsible for several critical functions.

• Message filtering based on subscriber preferences
• Delivery retries for transient failures
• Continuous health monitoring of subscriber execution hosts, routing events only to healthy ones
• Dispatching event execution status back to the orchestration layer for state machine management
• Event delivery concurrency management

Execution

Execution is the second half of the compute tier: the actual processing of the event. Work typically happens in a lambda function (serverless code) or a remote process, possibly another system entirely. The compute layer, then, is routing the event and then processing it.

Lambda infrastructure is the framework that runs events. When triggered, a process starts and returns either a success or retriable failure status. If no status comes back — or an error occurs — the default assumption is a retriable failure. The router acts as the client in this push-model interaction.

For reliability, execution processes ideally run across multiple cloud environments. The router can push events to different clouds depending on the locality preference the lambda or process owner configures. For example, some operators pin processes to specific clouds to keep them near backend storage dependencies and minimize cross-data center latency.

Autoscaling is a core feature of the lambda infrastructure at Dropbox. The platform is backed by Atlas, which provides autoscaling as well as release-time hooks for validating and rolling back code changes that would degrade service uptime or harm features.