Behind the Streams: Building a Reliable Cloud Live Streaming Pipeline for Netflix. Part 2.

Building the Ingest Path

With no existing broadcast infrastructure to lean on, the first major decision was where to receive and process live signals. The team chose to build acquisition directly in the cloud, assuming feeds from venues would be delivered into that environment. Prior experience with transmission protocols and their reliability characteristics shaped that initial approach.

For the first live comedy special, the team engineered connectivity at the venue itself. Two diverse dedicated internet access (DIA) circuits carried UHD mezzanine feeds at 50 Mb/s using HEVC, staying within the bandwidth budget while maximizing quality.

Direct venue-to-cloud links worked for early shows but did not scale. The architecture shifted to a hub-and-spoke model: production feeds are aggregated at a well-connected broadcast facility before being sent to the cloud.

Cloud ingest itself is built with multiple layers of redundancy. It runs in two separate regions, each receiving feeds through two distinct managed direct network paths, avoiding the open internet. This yields four independent streams of the same live feed, protecting against network issues, outages, or equipment failures.

The core of that redundancy is SMPTE 2022–7 seamless protection switching. It merges the dual paths at the packet level and selects the best packet in real time. If one path suffers packet loss or degradation, the system switches to the alternate path without any effect on downstream encoding. In practice this has worked flawlessly, even during complete path failures. AWS Elemental MediaConnect provides the managed video transport for this, handling 2022–7 processing, redundancy, and failover, and it integrates cleanly with the cloud encoding services while offering monitoring tools.

Encoding and Segmenting for Distribution

Once the contribution feed lands in the cloud, the live streaming pipeline takes over. The cloud encoder produces segmented streams with adaptive bitrate ladders, tuned by the Netflix Video Algorithms team for resolution, frame rate, and content complexity. High-motion events like WWE get higher bitrates, matching the quality approach used for SVOD.

Device compatibility drives format support. The encoder outputs both AVC and HEVC for video, and HE-AAC and Dolby Digital Plus 5.1 for audio at various bitrates. Multiple audio languages are streamed for global reach, and closed captions are produced in both WebVTT and TTML/IMSC.

Reliability in the pipeline comes from a dual-pipeline architecture with seamless switchover enabled by a feature called “epoch locking.” The design follows ISO/IEC 23009–9, Dynamic Adaptive Streaming over HTTP (DASH) — Part 9: Redundant Encoding and Packaging for Segmented Live Media (REaP).

A key property of the REaP design is that the two pipelines produce interchangeable segments without communicating with each other directly. This lets the origin or downstream components choose between them freely. To make that possible, both the encoder and packager are configured with a constant average segment duration. The contribution encoder embeds timecode into each video frame using SEI (Supplemental Enhancement Information) messages, carrying UTC timing to the cloud encoding pipeline. The cloud encoder then deterministically maps frames with timecodes to output segments, making switchover seamless.

Why Build a Custom Live Packager

The team chose to build the live packager in-house rather than adopt a third-party solution. The primary reason was to reuse Netflix’s existing streaming system components and maintain stream-level compatibility with devices, which was essential for the ambitious launch timeline and broad device coverage goals.

Netflix’s streaming service supports millions of devices across hundreds of categories, vendors, and hardware and software versions. Achieving format compatibility with those devices took years of adjustments to streaming formats and extensive large-scale field tests. The packager is responsible for system-level bitstream compatibility, so the live packager was built to replicate the battle-hardened packaging formula of the SVOD packager. This allowed live streaming to achieve device compatibility without repeating time-consuming field tests.

Caption support was another deciding factor. Netflix devices consume plain text segments using either IMSC or WebVTT, and third-party packaging solutions did not support those customized caption formats.

DRM protection also pushed the decision toward an in-house build. The SVOD packager is fully integrated with the Netflix security backend for encryption key generation, segment encryption, and packager security authentication. A live packager could leverage those existing DRM components, accelerating time to market.

As introduced in part 1 of this series, Netflix uses personalized manifests with segment templates and a constant 2-second segment duration. The live packager generates the segment template metadata and submits it to the PlayAPI backend systems, which create client-facing manifests without requiring a major architecture change.

The Live Origin: From Static Buckets to Media-Aware Serving

Origin servers are the first stop for content flowing into a CDN, and for live streaming they must deliver low-latency, scalable, and reliable storage and distribution. Netflix's initial live pipeline used static storage buckets in AWS, with redundancy built in by configuring two buckets — one in each AWS region. For extra resilience, packagers cross-published segments to both buckets, producing four potential candidates per segment. CDN nodes then applied their own prioritization and fallback logic to pick the best option for delivery.

That static approach quickly exposed problems. Generic storage was not tuned for live streaming and lacked the availability and performance SLAs the use case demanded. With a constant 2-second segment duration, publishing latency variation in the order of seconds severely degraded performance. Concurrent live events pushing request rates above 100 RPS per event frequently triggered throttling failures. Dual publishing by the packager helped when one bucket was available, but simultaneous failures across both regions still occurred.

A deeper issue emerged from the dual-pipeline design itself. Each pipeline produces its own version of every segment and manifest. When one pipeline hiccups, the CDN can serve the alternative — but it cannot tell which candidate is defective. If one version causes client playback disruptions, the CDN has no way to distinguish it from the healthy one using static buckets.

This drove the design of an origin layer with media-aware capabilities, allowing the selection of optimal candidates from the streaming pipelines. Embedding intelligence in the live origin also opened new operational possibilities, such as advanced TTL cache control and efficient propagation of streaming metadata. Working with the Data Storage team, Netflix built the live origin on its key-value datastore platform, enhancing the storage solution to cover the live streaming use case.

Orchestration for Live Events

With a small engineering team backing the initial live launch, workflow automation was a priority. An orchestration system was developed to dynamically configure live encoding pipelines based on content and production requirements. This system powered the first live comedy special and later enabled a range of events — from Chris Rock: Selective Outrage in UHD at 30 fps to back-to-back NFL Christmas Gameday streams in HD at 59.94 fps with four audio languages and ad support. The latter required automatically transitioning users between games.

The cloud pipeline spans ingest, encoders, packagers, origin, and monitoring. The orchestration service provisions each of these components, reducing human error and allowing teams to focus on monitoring, rapid response, and feature development. To mitigate single points of failure, redundant encoding pipelines are created in distinct geographic regions for every event. The service provisions and deprovisions encoding instances based on the event schedule, optimizing resource usage and cost. It also spins up test events with varying configurations — new bitrate ladders, video codecs, and new components like a closed-captions service — to validate formats and features before production rollout.

The orchestrator also manages real-time operations such as setting live lookback and DVR windows, timing transitions between live events, and manual encoder operations used for incident mitigation.

Architecturally, the orchestration system uses a hub-and-spoke model. The central entity manages resources, permissions, and storage; each component — encoder, packager, origin, and so on — acts as a spoke responsible for specific tasks. State is managed through a combination of pub-sub and polling mechanisms, and the control plane spans multiple geographic regions, with each regional deployment capable of orchestrating resources across the global infrastructure.

Operating Live: The Control Room

In the early days, the engineers developing the live systems also operated them during events, before formal Live Operations teams existed. Off-the-shelf tooling could not scale to production live events, so the team built its own solution: a UI called Control Room that interacts with the orchestration layer to control and poll event resource status.

Control Room provides a unified dashboard for monitoring and controlling the redundant, dual-region architecture. Each event progresses through pre-event setup, encoding (including live on-air), and post-event cleanup. While SVOD metadata was mature and could be leveraged early on, the live event schedule was a new concept. Until that concept matured, setup and teardown were performed manually through Control Room before and after each event.

During the transmission window — covering the broadcast plus additional time for signal validation — operators initiate the encoding pipeline with a single action coordinating resources across both regions. The dashboard populates with live feeds from venue contribution encoders and cloud encoders in each region, showing critical metrics. Manual failover controls are readily accessible; if one region degrades, operators can redirect traffic to the healthy pipeline in seconds.

Control Room also governs the state of the live title on Netflix throughout its lifecycle, from start to finish or transition to another show. Because live events don't have predetermined endpoints, the conclusion requires manual intervention: the operator initiates the end sequence, which signals clients to exit the player and accurately marks the end of the DVR window for future on-demand viewing.

While orchestration handles configuration and provisioning automatically, Control Room provides the human visibility and control needed to ensure each event delivers the expected quality. As the content slate has scaled from a handful of comedy specials to a full live programming schedule, Control Room has evolved from a simple button interface into a comprehensive operations tool.