Stream processing at Facebook scale
Facebook has published details on Turbine, the service management platform that runs its stream processing workloads. The system has been in production for several years and currently manages thousands of streaming pipelines across clusters of tens of thousands of machines, processing hundreds of gigabytes of data per second.
The platform exists because Facebook's general-purpose cluster management frameworks, such as Tupperware, don't fully cover the requirements of stream processing. Many of Facebook's streaming applications need strict service-level objectives, including a 90-second end-to-end latency guarantee. Turbine is designed to meet those SLOs while minimizing human oversight, using a loosely coupled set of microservices that separately decide what to run, where to run it, and how to run it.
Placement and scheduling
Turbine's task scheduler uses a two-level placement mechanism. Shards are first placed onto containers via Facebook's general-purpose shard manager; stream processing tasks are then assigned to those shards using hashing. Shards are periodically load balanced, and Turbine provides mechanisms that let stream processing tasks be safely rescheduled during rebalancing. The system also has failure handling that prevents data corruption, loss, or duplicate processing when nodes fail.
Predictive horizontal and vertical scaling
The autoscaler adjusts resources along multiple dimensions—CPU, memory, and disk—to keep jobs within their service-level objectives. It estimates required resources for a job, then scales the number of tasks or the per-task resource allocation up or down as needed. Because the autoscaler monitors the effect of its own scaling decisions and considers historical workload patterns, it can iteratively revisit and refine its original resource estimates.
Update atomicity under concurrency
A notable challenge Turbine addresses is that a provisioning service, the autoscaler, and human operators can all issue updates to the same streaming job concurrently. Turbine's update mechanism provides fault tolerance, atomicity, consistency, isolation, and durability in that environment.
The design relies on a hierarchical job configuration and a precedence scheme for merging updates from multiple actors. Planned updates are kept separate from actual updates, and a state sync service continuously reconciles the expected job configuration with the running one, rolling back and retrying whenever an update fails.
Operational experience
Turbine's production deployment has validated the approach in practice. According to Facebook, the system balances workload fluctuation across clusters, handles unplanned load spikes predictively, and completes large-scale updates consistently within minutes. Beyond its internal utility, Facebook notes the platform may offer lessons for other large-scale stream processing frameworks and hopes its experience helps advance the state of the art.
