A container throttling problem at scale

Most CPU-bound services running on Twitter's infrastructure begin violating their latency SLOs at around 50% of their reserved container CPU quota. Almost all services start failing at only slightly higher utilization. This holds even though CPU-bound workloads should theoretically approach much higher utilization levels. Because load is not evenly balanced across shards, and shard-level degradation becomes severe past that 50% mark, the practical ceiling is often lower still.

The root cause is an interaction between how services size their thread pools and how the Linux CFS scheduler enforces CPU quotas. Services reserve some number of cores through the cluster scheduler. The CFS bandwidth control mechanism limits the amortized CPU usage of each container over a time window, but it does not prevent a container from using many more cores than its reservation for short bursts. When a heavily threaded service has work to do, its threads all become runnable, CFS lets them run on whatever physical cores are available, the container exhausts its quota early in the period, and the whole process group is then throttled — effectively put to sleep until the next quota period starts. That forced sleep is catastrophic for tail latency.

The consequence is systematic overprovisioning: services are sized based on load tests or observed latency under load, and those numbers reflect the throttling penalty. So services either request more CPU per shard than they truly need or run more shards than necessary.

A concrete case: service-1

service-1, the largest and most expensive service at Twitter, provides a clear illustration. Under a load test at approximately its peak sustainable request rate, CPU utilization histograms show the service hovering mostly between 0 and about 20 cores — its reservation — with the average near 8 cores, or 40% of quota. The service fails its SLO anyway because of brief excursions well above the 20-core reservation. Those spikes exhaust the quota and trigger throttling, which causes the latency blow-up. The sampling period for the histogram is 10ms versus a 100ms CFS quota period, so the graph overstates the extent of true quota-exceeding parallelism, but the pattern is obviously correlated with throttling.

After reducing thread pool sizes so the service could not request far more parallelism than its reservation, the same service sustained 1.6x the previous load in testing. The load generator itself became the bottleneck before service-1 failed. Later testing indicated roughly 2x capacity after the thread pool changes. The same exercise applied to service-2 produced similar gains. Where latency, rather than cost, is the priority, a service can be left at its original capacity and the reduced throttling yields roughly a 20% reduction in latency.

The cost savings for hand-tuning the largest services are significant — mid-seven figures per year for service-1 itself, low-eight figures if clones of it are included. But fixing services one at a time does not scale, so the real question is how widespread the problem is.

Why so many services hit the same wall

Fleet-wide data for moderately sized services — those with at least 100 shards — shows nearly all of them maintain far more active threads than reserved cores. Tens of runnable threads per reserved core are not unusual. service-1, by comparison, only ran 1.5 to 2 runnable threads per reserved core under load.

Much of this oversubscription traces back to conventional wisdom about thread pool sizing. It is common, both at Twitter and in the broader industry, to see advice recommending thread pools of 2x the number of logical cores on the host. That heuristic makes sense for a single throughput-oriented pool with cheap context switches, where occasional blocking should not leave cores idle. It works less well for latency-sensitive services where multiple independent thread pools exist: 2x per pool, multiplied across pools, quickly produces tens of runnable threads per core. With CFS quota enforcement, those threads do not simply queue politely — they run, exhaust quota, and cause throttling.

Mitigations and fixes

Several approaches can reduce throttling, with varying scope, difficulty, and risk. Not all of them require application changes.

Right-sizing thread pools by default

The least invasive step is correcting oversized defaults in shared libraries. Netty defaults thread pools to 2x reserved cores; some internal libraries spin up eventbus thread pools at 2x the host's logical core count, resulting in over 100 threads where 1-2 would suffice for most use cases. Better defaults do not solve the entire problem, but they reduce its impact across the fleet and also cut lock contention and context switches. This work can proceed in parallel with more comprehensive solutions.

A negotiated thread pool API

A shared library could standardize the way applications size thread pools, with the JVM bridging package util-jvm as a natural host. The API could be as simple as globally capping total threads per process, though that does not distinguish application threads from I/O threads. A finer-grained design could introduce thread pool "quality of service" notions — marking some pools as nonblocking I/O threads and others as blocking application threads — to enable more precise negotiation. The main drawback is that service owners must opt in by adopting the new API.

Scheduler tuning

CFS exposes a number of knobs with the potential to reduce throttling's impact.

  • Shorter CFS period: The default period is 100ms, during which a container may consume its quota in slices as coarse as 5ms. In the worst case, a highly parallel application exhausts its entire quota in the first slice and sits throttled for the remaining 95ms. Shrinking the period proportionally reduces that worst-case sleep. The tradeoff is higher scheduler overhead, and total throttling could actually increase if bursts merely exceeding quota within the smaller window become more common.
  • Smaller bandwidth slices: CFS transfers runtime from a global pool to per-CPU pools in chunks currently defaulting to 5ms. If a process blocks or completes before consuming its full slice, the remainder is lost to the group. Smaller slices allow finer-grained accounting but also increase scheduler overhead, and no single value will fit all workloads. Mesos has previously rejected exposing this as a per-application tunable.
  • Other scheduler parameters: kernel.sched_tunable_scaling, kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns, and kernel.sched_autogroup_enabled all influence preemption behavior and resource sharing, but their net effect on throttling at Twitter's scale has not been measured.

All scheduler-level tuning shares a fundamental limitation: it does not address the core issue of parallel thread pools exhausting quota. It only changes the shape of the damage.

CPU pinning and isolation

Instead of policing CPU usage through time-sliced quota, pinning restricts a container to a fixed set of physical cores. With a 1:1 mapping between an application's mental model of CPU and physical cores, throttling disappears: the process group simply consumes its dedicated CPUs. There are also cache efficiency and determinism benefits, since the container no longer contends with other workloads for the same cores.

Mesos has historically made CPU pinning difficult, but in Kubernetes the k8s CPU Manager — alpha in 1.8, beta since 1.10 — provides a usable implementation via the kernel's cpuset cgroup functionality. The project has stalled somewhat in beta with relatively few users, though it is usable enough to validate the approach. Some work on the k8s side remains outstanding, and oversubscription becomes more complicated, though a multitiered scheduling model could reserve pinned CPUs for latency-sensitive pods while allowing secondary, less sensitive workloads to float across slack capacity. Facebook's ongoing kernel scheduler work on this concept could provide a path forward. Experiments at Twitter have shown performance gains from pinning nearly as large as the oversubscription factor the company currently relies on.

Cluster scheduler level oversubscription

Recovering idle CPU by scheduling more work onto each host is conceptually independent of the throttling fix. Data-driven oversubscription could raise machine utilization directly, without per-service tuning. The risk is that shard performance degrades on highly loaded hosts; unless the scheduler bases placement on measured utilization rather than reservations, some hosts will become overloaded and the shards on them will suffer.

Disabling or loosening quotas

Removing CFS quotas entirely and relying on the kernel's shares mechanism would let services use all available cores on an empty box while falling back to proportional shares under contention. This maximizes raw utilization but reintroduces the unpredictability that led Twitter to enable quotas in the first place: badly behaved services could interfere severely with neighbors, and service owners found performance too difficult to estimate under load. At least one company that tried this approach experienced severe incidents under load. Loosening quotas — setting the CFS quota to, say, twice the Mesos reservation — bounds the damage while still allowing cushion under load, but this is roughly equivalent to secretly doubling every service's reservation, which may be operationally confusing.

Twitter's current load testing framework — which injects unrealistic request mixes at individual shards without reproducing box-level contention from other services — makes the no-quota regime particularly risky, since the largest factor in shard-level latency variance is the overall load on the host.

Outcomes since the original analysis

The thread pool defaults work produced minor improvements. Two more substantial efforts followed from this analysis:

Finagle Offload Filter makes it easy for service owners to move application work off I/O threads onto a separate, appropriately sized pool. Combined with proper pool sizing, this yielded latency reductions of 15% to 60% depending on the service, enabling those services to cut provisioned capacity while still meeting SLOs.

A kernel patch takes the more direct approach of preventing containers from ever using more cores than their quota permits, rather than allowing a burst of parallelism and then sleeping the container to amortize. This eliminates quota-exhaustion throttling. In experiments on hosts running major services, the patch yields roughly a 50% cost reduction for a typical service with untuned thread pools. The naive version has a minor flaw — a container that stays idle for part of a period cannot "catch up" by consuming its full quota later in the period — but variant patches address this, and the effect is secondary to eliminating throttling.

The patch's impact exceeds what the original document anticipated, for two reasons. First, preventing a service from briefly grabbing all cores on a host also prevents the severe interference that throttling causes for its neighbors: a throttling service toggles between attempting to use all cores and using none, which is far more disruptive than sustained moderate usage. Second, throttling creates a death spiral under load: a shard that throttles accumulates requests while asleep, wakes to even more work, throttles harder, and approaches a metastable state where Finagle's load shedding only pushes the problem onto other shards. Removing throttling removes that failure mode.

A separate, independent effort consolidated larger services into fewer, larger shards to reduce per-shard overhead. A side benefit was that larger per-shard quotas are less sensitive to random load noise, contributing 0% to 20% CPU savings and 10% to 40% memory savings across large services.

Container CPU throttling is not unique to Twitter. Indeed has published on a regression fix in this area. Kernel developers have explored adding burst capacity to CFS quota accounting, which would permit short excursions beyond allocation before throttling — a welcome margin but not a cure for the fundamental problem, since a sustained burst still gets throttled. Uber worked around the issue for its Go services with automaxprocs, limiting goroutine parallelism at the runtime level; that trick works because Go services typically use a single thread pool, which is not the case for Twitter's multi-runtime services. The .NET runtime, meanwhile, has included adaptive thread pool sizing for a decade.