How we designed Dropbox’s ATF - an async task framework

Building Dropbox’s Async Task Framework

When we set out to build a centralized service for asynchronous computing at Dropbox, the requirements were steep: handle 10,000 tasks per second from day one, support nearly 100 distinct task types, and serve more than two dozen engineering teams across the codebase. Existing open-source options and commercial products didn’t fit our use case or scale, so we designed and deployed our own system—the Asynchronous Task Framework (ATF).

ATF has been in production for over a year. It currently supports more than 100 use cases for immediate or delayed task scheduling, serving roughly 9,000 async tasks per second for 28 internal teams. The system replaces several bespoke async mechanisms that individual teams had previously built, reducing redundant development and eliminating reliance on legacy software.

Core concepts

A few terms appear throughout this discussion:

• Lambda: A user-defined callback implementing business logic.
• Task: A unit of execution for a lambda; each scheduled async job is a task.
• Collection: A labeled subset of tasks within a lambda (e.g., password reset vs. marketing emails under a "send email" lambda).
• Priority: A label defining execution precedence for tasks within a lambda.

Feature set and guarantees

ATF offers four key features to clients:

• Task scheduling: Tasks can be scheduled for immediate or delayed execution at a specified time.
• Priority-based execution: Higher-priority tasks run before lower-priority ones once ready.
• Task gating: Tasks can be paused, or completely dropped, at the lambda or collection level until appropriate.
• Status tracking: Clients can query the current status of a scheduled task.

The system makes several explicit guarantees. Each task is executed at least once, with execution considered complete only when the user callback signals so. At most one instance of a task runs concurrently, so developers don’t need to design for duplicate simultaneous execution. Tasks from different lambdas are fully isolated—in worker capacity and scheduling resources—as are tasks at different priority levels within the same lambda. For delivery latency, 95% of tasks begin executing within five seconds of their scheduled time. The scheduling API itself maintains 99.9% availability.

These guarantees impose requirements on lambda developers. Callbacks must be idempotent, since a single task may execute multiple times. They must be resilient to abrupt interruption and retry on different hosts. And they must signal terminal states—success, fatal failure, or retriable failure—appropriately to avoid infinite retries.

System architecture

ATF uses gRPC for remote calls and our in-house distributed metadata store, Edgestore, for task persistence. The architecture comprises seven components:

• Frontend
• Task Store
• Store Consumer
• Queue
• Controller
• Executor
• Heartbeat and Status Controller (HSC)

The Frontend accepts task scheduling RPCs from clients and writes to the task store. The Task Store persists all tasks and is implemented on top of Edgestore, which supports indexed querying on association attributes. Tasks are stored as entities, while scheduling metadata—such as the next timestamp at which a task should be polled—is stored as associated attributes.

The Store Consumer polls the task store periodically to find tasks ready for execution—both new tasks and older ones being retried after failures. It pushes eligible tasks onto the appropriate queue. Here’s a simplified view of that poll-and-push loop:

repeat every second:
  1. poll tasks ready for execution from task store
  2. push tasks onto the right queues
  3. update task statuses

ATF uses AWS Simple Queue Service (SQS) for internal queues, which buffer work between the Store Consumer and the controllers. Each <lambda, priority> pair gets its own SQS queue, so the total number is the product of the lambda count and priority levels.

Worker hosts run the system’s execution layers. Each host has one Controller process that polls its assigned SQS queues in a background thread and buffers tasks locally. The Controller only handles the lambdas it serves, and it maintains separate process-local queues per priority level, enabling fine-grained execution prioritization.

The Executor is a multi-threaded process that actually runs task callbacks. Each thread operates in a straightforward loop:

while True:
  w = get_next_work()
  do_work(w)

Worker hosts run one Controller alongside multiple Executor processes. Both components use a pull model, continuously long-polling for new work rather than receiving push notifications.

The Heartbeat and Status Controller (HSC) handles RPCs related to task lifecycle: ClaimTask to claim a task for execution from a Controller, Heartbeat to keep an in-flight task alive from an Executor, and SetResults to record completion. The HSC updates task state in the task store accordingly.

Why we built it

Before ATF, async processing at Dropbox was fragmented. Multiple engineering teams ran their own bespoke systems, leading to redundant development, incompatibilities, and growing technical debt from legacy software. We found little publicly published material from other companies that had built async infrastructure at our scale, which made design validation harder—but now that ATF is serving its target load across the company, we’re documenting the architecture in detail as a reference for others facing similar challenges.

Inside Task Execution

Every task in ATF moves through a well-defined pipeline managed by several distinct components:

1. The client calls Schedule on the Frontend, passing task details including the desired execution time.
2. The Frontend creates the task entity and its association in Edgestore.
3. At the scheduled time, the Store Consumer retrieves the task from Edgestore and places it on the relevant SQS queue.
4. The Executor issues a NextWork RPC to the Controller, which pulls from SQS, claims the task via HSC, and returns it to the Executor.
5. Execution begins. Throughout processing, the Executor sends Heartbeat RPCs to the Heartbeat and Status Controller (HSC), finishing with a TaskStatus RPC.
6. The HSC reflects every heartbeat and status change back into the Edgestore entity and association.

Critically, each state transition also updates the association's next trigger timestamp. If a task's state hasn't changed by that timestamp, the Store Consumer will pull it again. This mechanism underpins ATF's at-least-once delivery guarantee, ensuring no task is ever silently lost.

The Store Consumer polls for work using this simple state-based query:

assoc_status= && next_timestamp<=time.now()

Task lifecycle transitions are governed by the state machine below:

Reliability in Practice

At-least-once execution. ATF retries every task until it reaches a terminal state—either Success or FatalFailure. System-level failures are always treated as retriable, and lambda owners can also explicitly mark a task with a RetriableFailure state. Transient RPC errors or brief outages in dependencies like Edgestore or SQS could cause a task to be dropped from the immediate pipeline, but the timeout and re-polling logic in the Store Consumer guarantees it will be picked up again.

Preventing concurrent execution. ATF prevents two workers from running the same task simultaneously through two complementary safeguards. First, tasks must be explicitly claimed via an exclusive Claimed state before execution starts. Only tasks in the Enqueued state can be claimed—retried tasks return to Enqueued when re-pushed to SQS.

However, a long-running task whose heartbeats keep failing could look dead to ATF even while its original executor continues processing. The Store Consumer would eventually repoll and re-claim it, causing double execution. To close this loophole, the Executor terminates itself after three consecutive heartbeat failures. Since the heartbeat timeout window is longer than the cumulative time of three failures, the task cannot be re-polled before the original executor shuts down.

Isolation. Each lambda runs on its own dedicated worker clusters, queues, and per-lambda scheduling quotas. Within a single lambda, different priorities get their own queues and dedicated scheduling bandwidth, preventing any one workload from starving others.

Delivery latency. ATF is not built for ultra-low-latency delivery; latencies of a few seconds are perfectly acceptable. The primary tuning knob is the Store Consumer's polling interval—more frequent polls mean lower latency. ATF is currently calibrated to poll once every two seconds.

Operational Ownership

ATF is designed as a self-service framework. Lambda owners are responsible for all aspects of operating their workers. Each lambda's worker clusters belong to the owning team, which controls code deployments and capacity management. Every Executor process serves exactly one lambda, and teams can host multiple lambdas on the same cluster by launching additional Executor processes.

Future Extensions

The scheduling foundation ATF provides opens the door to several natural enhancements:

Periodic execution. ATF currently handles one-time scheduling. Adding recurring task support would unlock new client use cases without rearchitecting the core system.

Task chaining. Tasks can already schedule downstream tasks during execution, but the framework offers no visibility or control over these chains. Promoting chaining to a first-class concept would improve monitoring and management.

Dead letter queues. Lambdas with logic bugs can spin tasks into infinite retry loops, consuming significant scheduling bandwidth and often requiring manual intervention. Introducing a maximum retry threshold, after which tasks are pushed to a dead letter queue, would let owners pause or inspect problematic workloads autonomously. Tooling to reschedule tasks from the dead letter queue after fixing the underlying bug would further reduce operational overhead.