From a queue to a multi-source platform
Meta's internal asynchronous computing platform, “Async tier,” executes tasks in the background on a large worker fleet. Engineering teams register functions and submit workloads through an SDK. The platform then adds load balancing, rate limiting, quota management, downstream protection and other operational features. Today it processes trillions of workloads daily.
The core architecture rests on three building blocks: ingestion and storage, transport and routing, and computation. Ingestion stores workloads reliably while responding to the client as quickly as possible. Transport decides how many workloads to move into computation — too few underutilizes workers and delays processing, too many risks overwhelming them. Computation is the actual runtime where functions execute.
What the system looked like in 2020
Originally the platform was built around an in-house distributed priority queue called FOQS (Facebook Ordered Queuing Service), developed on MySQL. Workloads are enqueued with a timestamp indicating when they become available for consumption. The consumer dequeues an item and holds a lease on it. A successful processing step is acknowledged with an ACK; a failure or lease timeout triggers a NACK, making the item available again. Dequeuing is non-blocking: even if the oldest item is unresolved, consumers can lease newer available items.
A lightweight “Submitter” service provided the ingestion path. It validated incoming workloads, enforced overload protection, and inserted them into FOQS. The transport role fell to a component called “Dispatcher,” which pulled items from FOQS and forwarded them to worker runtimes.
Why the architecture had to change
Dispatcher overload
The dispatcher accumulated too much responsibility. It not only managed item lifecycle and pulled from FOQS; it also protected the queue by adapting dequeue rates, enforced rate limits and quotas, handled workload prioritization and downstream protection, dispatched to multiple runtimes, and managed cross-regional load balancing. Having most logic concentrated in one component made parallel feature work hard and impaired operational scaling.
Supporting external data sources
Customers increasingly wanted to run workloads based on data already sitting in streams, data warehouses, blob storage, or other pub/sub systems. It was technically possible before, but had significant downsides.
Two limitations stood out. First, customers had to write their own plumbing to move that data into the platform via the Submitter API, producing recurring duplication across use cases. Second, copying data from its home into FOQS was inefficient at scale, and some workloads are cheap to leave in their original storage but expensive to stage in a queue.
Rebuilding with clear layers
The fix was splitting the system into smaller components with single responsibilities and treating every data source as a first-class citizen.
A generic scheduler
First, the platform removed storage-specific logic from the transport layer. Workload reading moved upstream, leaving the transport service as a data-source-agnostic scheduler that manages execution, applies rate limits and quotas, and balances load. It exposes a single generic API.
Tailers for reading
Different data sources have different semantics — immutable versus mutable payloads, fast-moving streams versus large-batch files. New “tailer” adapters encapsulate the per-source read mechanisms. Like the UNIX tail command, they watch their source for new workloads. When a customer presents a data source, the platform launches matching tailer instances to read it.
Push changed the trade-offs
That reorganization changed the interaction model. Previously the transport pulled data from FOQS; now tailers push workloads to the scheduler as RPC calls.
Push mode made the scheduler API source-agnostic. Tailers no longer need ACK/NACK or lease timeout semantics; they get a direct success or failure indication. Cross-region load balancing also improved because it is now centrally controlled by tailers rather than by each region independently pulling from the queue.
The switch eliminated data duplication from intermediate queuing (data no longer had to be buffered in FOQS) and improved end-to-end latency, while making other data sources equally accessible as FOQS. On the downside, RPC-style pushing does not fit long-running workloads well. Keeping a connection open for the full function duration ties up resources on both client and server. It also increases the chance that a transient failure makes an entire long execution start again from zero. In practice the vast majority of workloads completed in seconds, so this was not a blocker — but it matters if many functions run for minutes or tens of minutes.
What the re-architecture bought us
The shift away from copying workloads merely for buffering produced several measurable improvements:
- No redundant workload duplication in FOQS.
- Customers no longer need to build and maintain bespoke buffering layers.
- The system is now split into components with clean contracts, letting engineering teams scale operations and develop features independently.
- Push-mode delivery improved end-to-end latency and cross-regional load distribution.
With native support for multiple data sources, the platform can now match workloads to the most efficient storage. In practice, two options dominate customer choices: FOQS (queue) and Scribe (stream). Having run both at scale, we have a clear picture of their tradeoffs for asynchronous workloads.
When a queue makes sense
Queues offer the most flexibility. The lease model and arbitrary ordering let customers fine-tune retry policies, access individual items, and run functions with variable execution times. Failed workloads can be NACKed back into the queue and retried after any desired delay.
That flexibility comes with overhead:
- Leases require an internal item-lifecycle management system.
- Priority-based ordering demands a secondary index on items.
The result is a broadly applicable, moderately expensive storage choice.
Streams: cheaper, but stricter
Streams are more limited. They provide immutable data in batches, with no granular retries or per-item random access. What they deliver instead is fast, sequential access to high-volume incoming traffic. For workloads that only need that pattern, streams cost less at scale precisely because they give up flexibility.
Retrying failures inside a stream
Giving up granular retries did not mean giving up At-Least-Once (ALO) delivery. We still had to offer source-agnostic retries for failed workloads running from streams.
A tailer reads a stream in batches, then advances a checkpoint once the whole batch is processed. If a single item in the final batch fails, forward progress stalls until that item succeeds after retries. On a high-traffic stream, lag builds quickly and the platform struggles to catch up. Dropping the failed workload would avoid the blockage but break the ALO guarantee.
The controlled-delay service
To break this deadlock, we built a companion service that stores failed items and retries them after an arbitrary delay, freeing the stream from any blockage. The service accepts workloads with their intended delay intervals (exponential backoff works well) and releases items to computation once the interval expires.
Two designs looked viable:
- Use a priority queue as intermediate storage, betting that only a small fraction of traffic fails and the main stream stays healthy. The risk: if 100% of jobs start failing, the queue fills and we clog the main stream again — only this time, instead of copying failures into the delay service, the stream itself backs up.
- Maintain several predefined delay-streams, each blocked by a fixed interval (e.g., 30 seconds, 1 minute, 5 minutes, 30 minutes). Any item entering a delay-stream waits its fixed time before being read. Combination of these fixed delays covers arbitrary retry windows. Since delay-streams are just sequential-access streams, this design can scale larger at lower cost.
Takeaways
No single architecture fits every asynchronous workload at scale. The decision constantly weighs tradeoffs:
- High-traffic, short-running workloads pair well with streams plus RPC.
- Long-running executions and granular retry needs favor queues, at the price of maintaining ordering and lease management.
- A stream-based design with high ingestion rates that cannot compromise delivery guarantees should budget for a separate retry-handling service.
