Prioritizing Playback Over Prefetch

In 2020, Netflix introduced prioritized load shedding at the Zuul API gateway to ensure that critical playback requests get preference over less important telemetry traffic. Now, the company has pushed similar logic deeper into its architecture, applying it at the individual service level. The motivation is straightforward: service teams can own finer-grained prioritization, non-edge services can participate, and mixed workloads can share clusters rather than requiring separate infrastructure for failure isolation.

PlayAPI, a backend service on the video streaming control plane, handles device-initiated manifest and license requests needed to start playback. Those requests fall into two categories:

  • User-initiated requests (critical): Triggered when a user presses play; a failure directly prevents playback.
  • Pre-fetch requests (non-critical): Made optimistically during browsing to cut latency if the user selects a title; failure only adds delay between pressing play and video appearing.

The Concurrency Problem

PlayAPI previously relied on a concurrency limiter to manage traffic spikes, high backend latency, and under-scaled services. That limiter throttled both request types equally, which created two failure modes:

  1. Pre-fetch traffic spikes reduced availability for user-initiated requests.
  2. Higher backend latency degraded both request types equally, even when the system had spare capacity to serve all critical traffic.

Sharding the two request types across separate clusters would have solved both issues and added failure isolation, but at a higher compute cost and added operational overhead for CI/CD, auto-scaling, metrics, and alerts on the new cluster.

A Partitioned Limiter

Rather than shard, Netflix implemented a concurrency limiter inside PlayAPI with two partitions, built on the partitioning features of the open-source Netflix/concurrency-limits Java library:

  • User-initiated partition: Guaranteed 100% throughput.
  • Pre-fetch partition: Uses only excess capacity.

The partitioned limiter is configured as a preprocessing Servlet Filter that determines request criticality from HTTP headers, avoiding body reads for rejected requests. That keeps the limiter from becoming a bottleneck and lets it reject requests with minimal CPU. In steady state, no throttling occurs and pre-fetch handling is unaffected; prioritization only activates when the server hits its concurrency limit.

Validation via Failure Injection

Netflix validated the mechanism with Failure Injection Testing, injecting 2 seconds of latency into pre-fetch calls whose normal p99 is under 200 ms. The failure was applied to one baseline instance with regular load shedding and one canary instance with prioritized load shedding. Some internal downstream services already run separate clusters for the two request types, with pre-fetch clusters hotter, so the test simulated a realistic downstream latency problem.

Without prioritized load shedding, availability for both request types dropped under injected latency. With it, user-initiated requests stayed at 100% availability while only pre-fetch requests were throttled.

Production Validation

Several months after rollout, an infrastructure outage impacted streaming. Once the outage was fixed, Netflix saw a 12x spike in pre-fetch requests per second from Android devices, attributed to a backlog of queued requests. The system was not scaled for such a spike, risking a second outage.

Prioritized load shedding in PlayAPI held up. Pre-fetch availability dropped as low as 20%, and at one point more than 50% of all requests were being throttled, but user-initiated request availability remained above 99.4% throughout.

Generic Priority Buckets

Based on that success, Netflix built an internal library for services to apply prioritized load shedding based on pluggable utilization measures with multiple priority levels. While the API gateway handles widely varying priorities, most microservices see only a few distinct ones. To keep things consistent, services can choose from four predefined buckets inspired by Linux tc-prio levels:

  • CRITICAL: Core functionality; never shed unless in complete failure.
  • DEGRADED: Affects user experience; progressively shed as load rises.
  • BEST_EFFORT: No user impact; answered best-effort, shed progressively under normal operation.
  • BULK: Background work; expected to be routinely shed.

Services can either adopt the upstream client’s priority or map incoming requests to these buckets by examining attributes like HTTP headers or the request body.

CPU-Based Shedding

Since most Netflix services autoscale on CPU utilization, that becomes the natural load signal. Shedding does not begin until CPU passes the target utilization, preserving the autoscaling signal. As load increases, progressively more critical traffic is shed to maintain user experience. For instance, a cluster targeting 60% CPU for autoscaling can start shedding when CPU exceeds that threshold, buying time for additional instances to come online. Once capacity catches up, low-priority traffic resumes.

Experiments and Anti-Patterns

Experiments ran with a service normally targeting 45% CPU for autoscaling but prevented from scaling, configured to shed noncritical traffic after 60% CPU and critical after 80%. At request rates past 6x the autoscale volume, the service shed noncritical then critical requests while keeping latency within reasonable bounds and preserving successful RPS throughput.

Netflix also warns against two load-shedding anti-patterns:

  • No shedding: Without shedding, latency rises for all requests rather than failing fast on some that can be retried; this can trigger a death spiral where unhealthy instances dump load onto others before autoscaling can react.
  • Congestive failure: Shedding too aggressively drops successful RPS even when the trigger is a traffic increase, indicating the limiter is rejecting requests it should serve.

Netflix’s implementation avoids both pitfalls by holding latency low and sustaining as much successful RPS during load-shedding as before it.

Latency-Based Shedding for IO-Bound Services

Services that depend on external datastores or backing services face a different overload profile than CPU-bound ones. When those dependencies become overloaded, they signal distress through increased latency rather than CPU saturation. For these IO-bound cases, Netflix reuses the same prioritized shedding framework but swaps in new utilization signals that better reflect downstream health. In addition to adaptive concurrency limiters — which infer load from average latency — the initial implementation offers two latency-based shedding modes:

  1. Per-endpoint target and maximum latency thresholds, letting a service shed when it is abnormally slow regardless of which backend is causing the delay.
  2. Observed storage latency utilization returned by the Data Gateway platform that runs Netflix storage services, telling clients when they are approaching their allocated storage capacity limits.

These measures act as early warnings. A service can shed low-priority work before it pushes a backend into congestive failure. The key advantage over concurrency limits alone is tuning effort: services already maintain strict latency SLOs (for example, p50 under 10ms and p100 under 500ms). Recasting those SLOs as utilization percentages means the system sheds early to protect high-priority traffic, while still accepting as much work as the backend can handle within SLO bounds.

To build the utilization signal, the service counts requests that finish slower than its target or maximum latency objectives and reports the failing percentage. Netflix's KeyValue storage service, for instance, advertises a 10ms target and 500ms maximum per namespace. Clients receive per-namespace utilization data like this:

utilization(namespace) = {
  overall = 12
  latency = {
    slo_target = 12,
    slo_max = 0
  }
  system = {
    storage = 17,
    compute = 10,
  }
}

Here, 12% of requests miss the 10ms target, none exceed the 500ms timeout, and storage utilization sits at 17% of the allocated amount. Different workloads consult different signals: a daily batch writer might shed when storage utilization nears capacity, since additional writes would worsen instability.

Latency utilization proved useful for a critical file origin service. This service accepts writes of new files in AWS and serves as the read origin for Netflix's Open Connect CDN. Writes are the most important workload and must never be shed. But when the backing datastore is under pressure, it is acceptable to progressively shed reads, especially for lower-priority files that the CDN can retry without degrading the product experience.

The origin service configured a KeyValue latency limiter that starts shedding lower-priority CDN reads once the datastore reports target latency utilization above 40%. A stress test generated over 50Gbps of read traffic across both high- and low-priority files:

Press enter or click to view image in full size

Press enter or click to view image in full size

The test ramped low-priority reads to 2,000 reads/second of roughly 4MiB files until the backend store overloaded at over 50Gbps. As overload hit, the origin shed only low-priority reads, protecting high-priority writes and reads. Before this change, a breaking point took down critical writes and reads alongside low-priority traffic. During the test the file-serving service itself stayed under 10% CPU, so IO-based limiters were the only mechanism able to defend the system. The origin also kept serving additional traffic whenever the backend responded with low latency — avoiding the brittleness of concurrency limits, which either cut traffic prematurely when nothing was wrong, or too late once the system had already entered congestive collapse.

What's Next

Service-level prioritized load shedding has become a core part of keeping Netflix available during unexpected stress, preserving the viewing experience when SLOs come under pressure. The engineering teams continue building on this framework as new failure modes emerge.

Acknowledgements

The design, implementation, and testing of these prioritization techniques involved many engineers across Netflix's consumer product, platform, and Open Connect teams, including Xiaomei Liu, Raj Ummadisetty, Shyam Gala, Justin Guerra, William Schor, Tony Ghita, and others.