Reliability engineering for broadcast-scale Live video
Supporting Facebook Live and Facebook Watch means handling two very different types of content: user-generated video captured on arbitrary devices over the public internet, and professionally produced broadcast content from partners like La Liga, CONMEBOL, and UEFA. Broadcast audiences expect a near-flawless experience comparable to traditional TV, which raises the bar for reliability across the entire pipeline — ingest, processing, delivery, and playback.
For UGC, the public internet introduces inherent risk: network congestion can degrade quality, and failures can cause buffering or stalls. For marquee events like the UEFA Champions League Final, even a few seconds of audio or video loss carries significant impact. The system now in production supports millions of Live streams daily across both content types, and it proved itself in August when the UEFA Champions League Final broadcast peaked at 7.2 million concurrent viewers across the Brazil and Spanish-speaking Latin America streams.
Bridging broadcast and internet ingestion
Traditional TV broadcasters keep everything on private, dedicated paths with guaranteed bandwidth, typically using two geographically diverse fiber-optic paths or a fiber path with satellite backup. To support high-viewership broadcasts with similar guarantees, Facebook leverages a broadcast facility with managed encoders, providing access to private fiber and satellite links. That facility connects to data centers over dedicated, diverse connections, keeping video and audio quality as high as possible.
The ingestion system accepts both main and backup streams and supports manual or automatic failover between them if issues arise, all while preserving a flawless viewing experience. This shift — from supporting broadcasts in aggregate to supporting a single broadcast end-to-end — means that previously acceptable events like machine restarts, network failures, and congestion now require careful mitigation. Those mitigations, including coordinated failover and dedicated link capacity over multiple geo-distributed paths, must hold for the full duration of games, which can run longer than four hours with pre- and post-show content.
Protecting delivery from demand spikes
On the delivery side, major events trigger notifications to millions of followers, producing a spike of simultaneous viewers. When edge cache servers don't hold the requested content, they forward requests to underlying services, creating a thundering herd problem — too many requests can overwhelm the system, causing lag, dropouts, and disconnections.
To protect back-end services, Facebook developed novel request coalescing and cache sharding mechanisms. These allow reliable video delivery while maintaining realtime playback under high concurrency.
When serving video at this scale, Facebook works with ISPs to avoid overloading the network. In Rio de Janeiro, degraded viewer experience caused by congestion between Facebook and one ISP led to an increase in local interconnection capacity from 20 Gbps to 120 Gbps. That change kept nearly all Live video traffic within the viewer's ISP network, staying in-country and off congested international links — a benefit for both the ISP, which reduces costs, and viewers, who see fewer stalls and higher resolution.
Last-mile constraints and future work
The UEFA Champions League Final stream validated the ingest-to-edge pipeline, but it also exposed last-mile radio delivery as a significant bottleneck for popular broadcasts, given current cellular technology limits. Facebook is exploring LTE-Broadcast as a way to share a single copy of the stream among all viewers on the same cell tower, potentially overcoming that constraint.
Ongoing connectivity investments include a new submarine cable nearing completion that should improve redundancy and resilience in Latin America, plus continued evaluation of new edge locations to support healthy interconnection ecosystems and keep traffic in-country.
The 7.2 million concurrent viewer milestone is not the end of the road. The next challenge is New Year's Eve, which inverts the UEFA Champions League scenario: millions of people simultaneously streaming individual live videos of fireworks and celebrations as midnight strikes around the world. Instead of optimizing caches for a small number of extremely popular broadcasts, that event requires optimizing processing, I/O, and storage for a massive number of simultaneous smaller-reach streams.
