Introducing Cape

Handling a Billion Daily Events: Inside Dropbox’s Cape Framework

Dropbox processes over a billion file saves every day, each of which can trigger a range of asynchronous jobs: indexing content for search, generating file previews, or notifying third-party apps via the developer API. To manage this scale, Dropbox built Cape, a framework for real-time asynchronous event processing that has been in production since late 2016.

Cape replaced Livefill, the company’s previous event-processing system. The new framework was designed to handle the demands of a platform that has moved beyond pure file sync into real-time collaboration.

Core Requirements

Cape’s design targets several specific engineering challenges:

• Sub-second processing: Features like searching or viewing a newly shared file require processing to happen with latencies under a second.
• Support for multiple event types: Unlike Livefill, Cape is not limited to file events. It also processes metadata changes like sharing actions, permission updates, and comments.
• High throughput: The system handles tens of thousands of events per second, with multiple workloads potentially triggered by a single event.
• Flexible job durations: Processing workloads can range from milliseconds to minutes, depending on the event and the task.
• Isolation: A failure in one user’s processing logic should not degrade the framework for everyone else.
• At-least-once delivery: Every event must be processed at least once to ensure a consistent product experience.

Events and Domains

In Cape, each event stream is called a Domain. Events within a domain have a Subject (the entity the event affects, identified by its Subject ID) and a Sequence ID, a monotonically increasing number that orders events per subject.

The first version of Cape supports two primary event sources. The first is SFJ (Server File Journal), Dropbox’s metadata database for files. Each file change is uniquely identified by a Namespace ID (NSID) and a Journal ID (JID). The second source is Edgestore, the metadata store for non-file data. Changes there are tracked via a combination of a GID and a Revision ID.

The abstraction is extensible. For example, a stream of events flowing into a Kafka cluster could similarly be mapped onto Cape’s model.

Architecture Overview

Cape’s architecture is built around four main components:

• Cape Frontend: SFJ and Edgestore send asynchronous RPC “pings” containing event metadata. These pings are outside the critical path for the source services, minimizing any availability impact on core Dropbox functionality. The Frontend publishes these pings to Kafka queues for persistence.
• Cape Dispatcher: Subscribes to the Kafka queues, applies the business logic for event routing, and dispatches events to the appropriate workers via Redis. The Dispatcher is responsible for ordering guarantees and dependencies between lambdas.
• Lambda Workers: Execute the user-defined business logic in response to events, then report status back to the Dispatcher.
• Cape Refresh: Since pings are sent asynchronously, they are not guaranteed for every event. Refresh workers continually scan the SFJ and Edgestore databases to catch missed events and send the necessary pings. They also help detect permanent failures in application code across the billions of Subjects Cape processes.

Integrating with Cape

Developers follow three steps to process events with Cape:

1. Write a lambda: Implement the business logic using a defined interface, currently available in Python or Go.
2. Deploy: Deploy Cape workers using Dropbox’s standard deployment and monitoring tooling.
3. Configure events: Specify the Domain and Lambda that should handle the events in Cape’s config.

Once these steps are complete, the deployed workers will receive events, run the lambda, and return status responses to the Dispatcher. Users also get access to automatically generated monitoring dashboards that track key metrics for their lambda’s processing.

Production Usage

Cape currently processes several billion events per day. The 95th-percentile latency—measured from when the Cape Frontend receives an event to when processing starts—is under 100 milliseconds for SFJ events and under 400 milliseconds for Edgestore events. End-to-end latency for SFJ, including batching within the SFJ service itself, ranges from roughly 500 milliseconds to one second.

Cape is currently powering several production features at Dropbox:

• Audit Logs: Cape indexes events in real time for Dropbox Business admins to search.
• Search: Cape triggers real-time indexing whenever a file changes, making the contents searchable.
• Developer API: Cape sends real-time file-change notifications to third-party apps via the Dropbox developer API.
• Sharing Permissions: Cape handles expensive asynchronous tasks like propagating permission changes across deeply nested shared folders.

As a generic framework supporting the majority of Dropbox’s key event streams, Cape is expected to serve as the backbone for many future features.