Why CPU utilization is a performance lever
Experiments by GitHub’s Performance Engineering team consistently show that CPU utilization is strongly correlated with latency. As utilization climbs, so does request latency—and that relationship creates both a risk and an opportunity. Understanding precisely where and how performance degrades allows for more informed provisioning decisions, reducing the need to spin up additional machines when existing ones are overloaded.
The team’s goal was to quantify these effects across different CPU families and instance types. The findings help identify the utilization range where resources are neither wasted nor bottlenecked—what the team describes as the “Golden Ratio” of CPU utilization.
Running controlled load tests
To collect accurate data, the team used an internal environment codenamed Large Unicorn Collider (LUC). Built a year ago, LUC runs within a small portion of GitHub’s Kubernetes clusters and mirrors the architecture and configuration of production workloads. It can be hosted on dedicated machines to avoid interference from other workloads. Traffic to LUC is adjustable and can be activated or deactivated within seconds, enabling safe experimentation.
The experiment used moderate production traffic to establish a baseline on a LUC pod. Request volume stayed constant throughout the test, ensuring a consistent CPU load. CPU utilization was then increased incrementally using the Linux stress tool, which occupies CPU cores by running random processing tasks. Steps were adjusted per instance type to account for differing core counts, while total CPU utilization remained the common factor across all instances.
It’s important to note that this setup does not replicate production workloads exactly. stress continuously performs mathematical operations, while real workloads include I/O and interrupts, which impose different demands on system resources. Still, the results provide valuable insight into how CPUs respond under load.
Latency rises across all instance types
As expected, CPU time per request increased for every instance type as CPU utilization rose. The differences in absolute latency are attributable to the varying CPU models used across instance types, so examining the percentage increase in latency provides more meaningful comparison:
One instance in the data stood out, deviating more than others. That case is examined below.
Turbo Boost and frequency scaling
All instances in the experiment use Intel CPUs, and the frequency data shows the clear effect of Intel’s Turbo Boost Technology. At lower CPU utilization—around 30% or below—core frequencies are higher, yielding faster CPU times and lower overall latency. As utilization rises and more cores are demanded, the CPU approaches its thermal and power limits, and frequencies drop:
In practice, this means a workload running at roughly 30% CPU utilization will report faster response times than the same workload on the same VM once utilization exceeds 50%.
Hyper-Threading overhead
Frequency scaling is not the only factor at play. All nodes have Hyper-Threading enabled, which allows a single physical core to appear as two virtual cores to the Linux kernel. The kernel schedules work across these logical CPUs, ideally keeping only one hardware thread active per physical core. Up to a certain utilization level this works well, but beyond that threshold it is no longer possible to fully use both virtual cores on a physical core. Performance then declines relative to normal operation.
Finding the utilization sweet spot
Underutilized nodes waste power, space, and hardware. Overutilized nodes, however, can become less efficient as performance degrades. The risk is that rising latency makes it appear that more capacity is needed, leading to a cycle of over-provisioning. This is especially problematic for blocking workloads that do not use an asynchronous model: as CPU performance worsens, each process handles fewer tasks per second, making the existing capacity insufficient.
The optimal balance is a utilization level that is high enough to avoid waste yet low enough to avoid significant performance degradation. Since the experimental data already shows how CPU time increases with utilization, a mathematical model can be built to find this threshold. The first step is to decide what percentage of CPU time degradation is acceptable, which may depend on user expectations or performance SLAs. Once defined, that threshold can be mapped back to a specific utilization level.
Plotting CPU utilization versus CPU time (latency) makes the trade-off explicit. In this example, the goal is less than 40% CPU time degradation, which corresponds to 61% utilization on the instance tested:
Investigating the outlier: a C-state problem
One instance type produced outlying data, and the experiment confirmed a previously known issue: certain instances were not reaching their advertised maximum Turbo Boost frequency. Instead, frequency remained below the advertised maximum even at low CPU utilization. The example below shows an instance whose CPU family advertises Turbo Boost above 3 GHz, yet it reports a maximum frequency of only 2.8 GHz:
The root cause was a disabled CPU C-state. Without C-states, CPU cores could not halt when idle and were perceived as “busy” by the turbo driver, preventing the CPU from entering higher Turbo Boost frequencies. After enabling the C-state—which allows cores to power down when idle—expected Turbo Boost behavior returned, with immediate improvements in latency and CPU frequency:
After the C-state adjustment, the percentage change in CPU time across instances became consistent:
Validating the cost of utilization
The experiments confirmed that performance declines as CPU utilization rises across different CPU families. With the data, the team can now identify optimal utilization thresholds that balance performance and efficiency, guiding resource provisioning strategies and making better use of existing hardware investments. The findings reinforce the importance of checking not just software load, but also underlying hardware states such as CPU C-states and Turbo Boost settings, which can have a measurable effect on throughput and latency.
