The Case of the Slower-Selling Server: Chasing a 10% Performance Gap
In Cloudflare’s global network, every server runs the entire software stack, which means maximum hardware performance is non-negotiable. To maintain supply chain flexibility, the company purchases servers from multiple vendors with identical configurations. But after deploying Gen X AMD EPYC Zen 2 (Rome) systems, a consistent anomaly emerged: servers from one vendor (SKU-B) were perpetually delivering 5-10% fewer requests per second (RPS) than servers from another (SKU-A).
Average RPS for SKU-B was roughly 10% below SKU-A, based on 24-hour observation windows.
Starting with Compute: DGEMM and Power Limits
Initial debugging focused on raw compute performance using AMD’s DGEMM high-performance computing tool, which measures sustained floating-point computation rates—specifically, real matrix-matrix multiplication with double precision. A modified version armed with AMD libraries for the EPYC instruction set was used for the tests.
DGEMM surfaced discrepancies tied to the processor’s Thermal Design Power (TDP), the maximum power draw a processor can sustain during a thermally significant period under load. Under stress tests, the underperforming servers only drew between 215 to 220 watts, whereas the processors supported a maximum TDP of 240 watts. The GFLOPS data told the same story: underperforming systems were logging approximately 100 gigaflops behind their healthier counterparts.
A DGEMM run on a well-performing system.
A DGEMM run on an underperforming system.
As a first attempt, idle power saving modes (C-states) were disabled via the CPU BIOS configuration. That action produced expected GFLOPS numbers and allowed the processors to reach max TDP. Convinced this was the root cause, the team returned the servers to the production test environment for data collection. The performance delta remained.
Network Path and BIOS Settings
Running a fresh comparison of BIOS settings logs between SKU-A and SKU-B didn’t reveal the culprit either, so the spotlight moved to networking. The open-source tool iPerf found no bottlenecks there.
While reviewing configuration, the team spotted that both SKUs were ignoring the AMD Preferred I/O function, which enables improved DMA write performance for devices on a single PCIe bus. Enabling this option on SKU-B caused no measurable performance movement in production. That steered the investigation toward memory configuration and operating speed.
HSMP Uncovers the Memory–Infinity Fabric Divide
Gen X systems run DDR4 modules rated for a maximum of 2933 megatransfers per second (MT/s). With the BIOS setting for memory clock frequency left on Auto, the system reports 1467 MHz—the actual clock frequency being half of the MT/s rate because DDR samples twice per clock cycle. Memory bandwidth checked out fine under a Stream benchmark test.
Out of internal options, the engineers consulted AMD and received the HSMP debug tool, which provides access to the Host System Management Port. It grants granular visibility into processor TDP limits, temperatures, and clock frequencies. When run on the lagging servers, the tool exposed a clock mismatch: the memory clock was set to 1467 MHz while AMD’s Infinity Fabric—the interconnect linking memory to the processor core—was running at 1333 MHz.
Memory (1467 MHz) and Infinity Fabric (1333 MHz) clock frequencies are misaligned on the slower systems.
For latency-sensitive, throughput-heavy workloads, these two frequencies must be equal. A snapshot from SKU-A showed both clocks matched perfectly.
Properly configured systems show memory and Infinity Fabric running at identical 1467 MHz clocks.
Resolution: A BIOS Compile-Time Fix
The Infinity Fabric clock wasn’t exposed as a tunable BIOS parameter, so the team requested a new firmware image from the vendor that hardcoded the fabric frequency to 1467 MHz at compile time. After deploying that BIOS to the underperforming systems in production, RPS numbers aligned—and in some instances, SKU-B now edges out SKU-A on the same-day, 24-hour benchmark.
Post-fix RPS for four SKU-A and four SKU-B machines shows performance parity and occasional outperformance from SKU-B.
