Shedding Smart Load: How Netflix Keeps Playback Alive During Outages

Not all traffic to a service is created equal. Some requests are essential to a core function—like pressing play on Netflix—while others, such as background logs, are expendable. Yet infrastructure failures don't discriminate; a spike in low-priority traffic can degrade a system just as quickly as a critical one.

Netflix historically relied on binary circuit breakers to manage this load: the system was either on or off, with no nuance for what was being dropped. To improve resilience for its members, the engineering team introduced priority-based progressive load shedding within its API gateway, Zuul. This allows the system to throttle low-priority requests first, preserving the experience for critical actions like streaming playback.

Building a Traffic Taxonomy

To categorize incoming requests, Netflix engineers focused on three dimensions: throughput, functionality, and criticality. This analysis led to a three-tier classification system:

  • NON_CRITICAL: High-volume traffic that has no impact on playback or the member experience. This includes logs and background requests, which often constitute a large percentage of system load.
  • DEGRADED_EXPERIENCE: Features that affect the user experience but not the ability to play content. This can include viewing history, language selection, and pause/stop markers.
  • CRITICAL: Requests that directly impact the ability to play. If these fail, members will see an error message.

Zuul assigns a priority score from 1 to 100 for each request based on its attributes, categorizing them into these buckets. In normal operation, this score is unused. However, when a backend service or Zuul itself becomes overloaded, the gateway treats requests with higher priority more favorably, effectively implementing a dynamic priority threshold. If a request's priority falls below the current threshold, it gets dropped.

Where to Throttle

Zuul can shed load at two distinct points in the request lifecycle, depending on where the bottleneck exists.

Service Throttling

Zuul monitors the error rates and concurrent request counts to individual backend services. When these metrics cross a defined threshold, it indicates the service is struggling with failures or latency. Zuul then throttles traffic aimed specifically at that service to relieve pressure.

Global Throttling

If Zuul itself is in trouble, the stakes are higher. A failure here would block all traffic to all backend services. Zuul uses key internal metrics—CPU utilization, connection count, and concurrent requests—to detect its own distress. Crossing these thresholds triggers aggressive throttling of all traffic to keep the gateway alive while the wider system recovers.

Progressive Throttling with a Cubic Curve

The next step was combining the priority scoring with the load-shedding mechanism to manage stress progressively. Instead of cutting off all traffic at once, Zuul drops requests in order of lowest priority first. The level of throttling is governed by a cubic function.

This curve ensures that lower-priority traffic is eliminated early on without impacting critical streaming requests. As the system becomes more severely overloaded, the throttle tightens. The design protects the core playback experience: a relatively small portion of traffic impacts streaming availability, so shedding low-priority requests may disable some features but will keep the system stable and members watching.

Dealing with Retry Storms

When Zuul drops a request, it communicates directly with the client device, telling it to back off. It does so by sending a signal indicating the number of retries permitted and the time window in which they can occur:

{ “maxRetries” : <max-retries>, “retryAfterSeconds”: <seconds> }

These retry dials are adjusted automatically based on the request priority. Higher-priority requests are allowed to retry faster and more aggressively, further increasing the likelihood of successful playback while preventing a retry storm on the backend.

Validating the Taxonomy

To ensure that requests classified as NON_CRITICAL really were safe to drop, the team created a failure injection point within Zuul. This allowed them to simulate the effects of load shedding by blocking entire ranges of priorities for specific devices or members. This manual testing gave them a direct look at what a "load-shedded" experience would be, confirming which requests could be shed without disrupting the user.

Because Netflix evolves rapidly, assumptions about what is non-critical might not remain true. To catch regressions, the team uses ChAP, an experimentation platform. It runs an A/B test over a small population of production users for 45 minutes. One group gets normal service while the other has a specific priority range throttled, allowing the team to measure deviation in playback KPIs across device types.

During the first experiment, this approach detected a race condition on Android and iOS devices where a low-priority request caused playback errors under throttling conditions. After the bug was fixed, the tests were scheduled to run periodically to catch future problems before they affect a wider audience.

Real-World Impact

In 2019, before this system was in place, Netflix experienced an outage that prevented a sizable percentage of members from playing content. In 2020, shortly after deploying progressive load shedding, the platform faced a similar issue with the same potential severity. This time, however, Zuul recognized the challenge and began shedding traffic by priority level automatically. Backend services stabilized, and the streaming availability metric, measured in streams per second (SPS), remained steady. Members continued watching in real-time while the infrastructure self-recovered.

The team sees room to expand on this work. Future efforts include refining retry policies between different devices and backends, tuning the load-shedding thresholds dynamically, and continually adjusting request priorities based on insights from chaos testing.