Prioritizing hundreds of billions of background requests
Every interaction with Facebook, Instagram, and WhatsApp generates work for our systems. Many of these resulting requests — delivering a notification, updating a follower feed, or computing a recommendation — do not need to block the user’s original action. They can be handled asynchronously, in the background. As the volume of these asynchronous requests has grown into the hundreds of billions per day, the system that manages them, called Async, has had to evolve from a simple queue into a multitenant platform with fine-grained prioritization and capacity controls.
Why the first design stopped scaling
The original Async stored all requests in a centralized database. Dispatchers queried that database, sorted by a desired execution time, and pulled jobs to send to worker servers. With only three priority levels and first-come, first-served behavior inside each, the system worked at lower traffic volumes. New use cases simply meant adding more workers.
That approach collapsed under growth. Large, high-volume use cases dominated the queue, starving smaller ones. A single stalled use case — due to a bug or a misconfiguration — could block everything behind it indefinitely. Adding workers did not fix the underlying fairness problem, and it created a new one: at peak hours, servers were overloaded while many sat idle at off-peak times. Engineers, frustrated by the bottleneck, started building their own point solutions, which multiplied maintenance burden and alert noise. Async needed a redesign, not more capacity.
The three core problems
Designing a new Async meant answering three fundamental questions about how a shared background system should behave.
Which request runs first
Not all requests have the same urgency. A Safety Check notification during a crisis is more time-sensitive than a notification about a like on an old photo, but the early system treated them as equal within a priority tier. To capture this nuance, Async introduced the concept of delay tolerance — the maximum time a job can wait before it must be executed. Each job is scheduled as close to its delay tolerance as possible, giving the system a precise ordering signal beyond simple priority.
Using capacity efficiently
Async traffic is not uniform. It falls into three broad categories, each with different predictability:
- Daily traffic is relatively predictable, such as people checking News Feed in the morning or churches streaming services on Sundays.
- Major events (New Year’s Eve, World Cup matches, public figure Live streams) are semi-predictable — known in advance, with anticipated spikes.
- Incident response is unpredictable; traffic from a crisis spikes dramatically and recedes within minutes or hours.
The challenge is to keep utilization high across all of them without over-provisioning for peaks.
Enforcing fair share
With many use cases sharing the same server pool, a single malfunctioning job could consume disproportionate CPU and memory. Async needed policies to ensure each use case gets its fair share of resources.
Architecture of Async today
Async now runs on a multitenant architecture: a shared pool of servers supporting many use cases with heterogeneous traffic patterns. Internal products such as the Voting Information Center and COVID Information Center depend on it. The redesign added three capabilities to keep the shared infrastructure reliable and fair.
Queueing with deadlines
The original Async stored its queue in a single MySQL table. During peak hours, a dispatcher would pull jobs, keep a dynamic number, and push the excess back — a costly back-and-forth that left little time for actual dispatch. The new design replaced this with a multiple-queue system:
- Each use case gets its own queue, sorted by job deadline.
- The dispatcher compares the headers of all queues and selects the most urgent job based on priority and deadline.
- When a job is selected from a queue, a background thread refills that queue from storage. Queues with no activity incur no overhead.
Queueing also helps during overload: jobs with long delay tolerances are deferred, spreading the workload over a longer window and letting the most urgent work proceed.
Time shifting
Two complementary time-shifting strategies smooth the load curve:
- Predictive compute runs at off-peak hours based on what users needed the day before, storing precomputed results in cache. During peak hours, requests read from that cache, trading occasional cache misses for much lower on-demand compute.
- Deferred compute schedules work as part of user request handling but executes it much later. For instance, “people you may know” lists are processed at off-peak times and load from cache when users are online at peak hours.
Both shift load away from peak periods and improve utilization of otherwise idle off-peak servers.
Batching
Every job has a life cycle: scheduled to a queue, picked up by a dispatcher, sent to a worker, and deleted. This requires one slot in the queue and one round-trip between dispatcher and worker, regardless of whether the job runs for milliseconds or minutes. During major events with 10x traffic, that overhead alone could overwhelm Async even when workers had spare capacity.
Batching mitigates this: Async accumulates multiple small jobs into one mega job. On the service side, one mega job costs the same as one individual job — one queue slot, one round-trip — but carries many jobs to the worker. This sharply reduces load on queue and dispatcher components. An unexpected side benefit emerged on the worker side: running multiple jobs in a single request improved efficiency because of higher cache reuse and code warmup.
Regulating capacity with quotas and rate limits
The multitenant architecture meant that a bug or misconfiguration in one use case could degrade Async for all others. Two mechanisms control this risk, both precise enough to track per-use-case resource use.
Async now measures queued jobs by CPU instruction utilization and memory. Each use case is assigned a quota, and engineers can request additional servers to raise it. When a use case exceeds its quota, its execution is throttled and an alert is sent. This prevents a runaway job from disrupting executions across the platform.
Ingestion is separately protected. A misconfiguration upstream could overload the submitter — the component that accepts new async requests — and temporarily take it out of service. Rate limits on accepted requests prevent unexpected spikes from breaking the pipeline.
One exception is required for incident response, whose traffic is inherently irregular and unpredictable. To keep those use cases running, quotas carry flexibility: a use case can exceed its limit for short bursts without being cut off.
Complexity Demands Better Tooling
The architectural evolution closed the original gaps, but it did not come for free. The system now handles far more traffic and has grown in complexity, introducing edge cases that make it significantly harder to understand. This is a direct cost to the team: a steeper learning curve reduces how quickly members can troubleshoot incidents and delays the onboarding of new use cases. For a system that is fundamentally a tool for human interaction, the user interface must be intuitive. In practice, this means that investing in troubleshooting tools is not a side quest but a prerequisite for continued growth.
Prioritizing Accountability and Flexibility
A central focus moving forward is strengthening resource accountability for both the system's users and its maintainers. This is a dual-purpose effort: improved visibility into resource consumption yields more accurate capacity forecasting, while also ensuring fairness in allocation between different use cases. This fairness is what allows the platform to absorb the inevitable spikes and surges in traffic without penalizing individual consumers.
Managing the system at this scale brings a unique set of challenges, but it also provides a fertile ground for innovation. These operational lessons are broadly applicable. Environments operating at Facebook's scale and those just beginning to architect a large-scale system can both apply these principles to build more reliable services.
