Getting to the Core: Benchmarking Cloudflare’s Latest Server Hardware

Refreshing Cloudflare’s Core Servers

Cloudflare’s core data center fleet processes logs, analyzes attacks, and computes analytics across a range of workloads. In 2020, the servers running these tasks were due for a hardware refresh, prompting a redesign that borrowed heavily from the company’s Gen X edge server platform. Two distinct server variants emerged from this effort: Core Compute 2020, an AMD-based system for general-purpose compute and analytics with solid-state storage, and Core Storage 2020, an Intel-based system with twelve spinning disks for database workloads.

Core Compute 2020: Converging Configurations

The Core Compute servers run Kubernetes, Kafka, and a variety of smaller services. The previous generation relied on Intel-based platforms, with the Kubernetes server using Xeon 6262 processors and the Kafka server using Xeon 4116 processors. A key goal of the refresh was to converge these configurations, simplifying spare parts and firmware management across the fleet.

As the configurations show, the two server types have been unified, with the sole difference being the number of NVMe drives installed based on the workload. Both servers moved from a dual-socket to a single-socket configuration, while the number of cores and threads per server either increased or stayed the same. In every case, base frequencies improved significantly, and the storage tier shifted from SATA SSDs to NVMe SSDs.

Synthetic Disk Benchmarking

Kafka is the heaviest user of the SSDs, and it spends most of its time sequentially writing 2MB blocks. A simple FIO script was created with 75% sequential writes and 25% sequential reads, scaling block sizes from 4096B to Kafka’s typical write size of 2MB. The results matched expectations for NVMe-based drives.

Production Workload Results

Many Core Compute services run in Kubernetes containers, some of which are multi-core. The transition to a single socket eliminates dual-socket NUMA problems, guaranteeing that all cores allocated to a container reside on the same socket.

Another constantly running workload on Compute hosts is the CSAM Scanning Tool. Systems engineering isolated a Core Compute 2020 host and a previous-generation host, ran only this workload, and measured the time to compare fuzzy hashes for images against the NCMEC hash lists. The tool is highly compute-intensive, so hardware improvements matter even after significant software optimization.

In the heatmaps, the X axis shows time, and the Y axis buckets the time taken to verify a non-match. The dominant red points on the Compute 2020 graph fall in the 5 to 8 millisecond bucket, while the previous-generation host’s red points sit in the 8 to 13 millisecond bucket. The Compute 2020 host is verifying hashes significantly faster on average.

Core Storage 2020: Single-Socket Economics

ClickHouse, which performs analytics over large datasets, was another major workload identified for the refresh. The last upgrade for ClickHouse servers was in 2018.

Processor and Memory Choices

ClickHouse runs on a 1U chassis with 12 x 10TB 3.5” hard drives. At design time, the server vendor had no AMD version of this chassis, so the platform remained Intel-based. Core Storage 2020 uses a single 20-core/40-thread Xeon processor instead of the previous generation’s dual-socket 10-core/20-thread arrangement. This move to a single Xeon 6210U retained the same core count but delivered a 17% higher base frequency and a 26% higher max turbo frequency, while the total CPU TDP dropped from 165W to 150W.

In a dual-socket server, remote memory accesses—accesses by a process on socket 0 to memory attached to socket 1—incur a latency penalty.

Putting all 20 cores on one socket eliminates these remote accesses, which take 76% longer than local memory accesses.

The memory in the Core Storage 2020 host is rated for 2933 MHz, but the Xeon 6210U clocks the installed 8 x 32GB configuration at 2666 MHz. That still provides a 13% boost over the previous generation’s memory speed. A balanced 6-DIMM configuration would allow a slightly higher clock, but the 8 x 32GB setup’s additional RAM capacity wins out.

Data capacity remains 12 x 10TB SATA drives in RAID 0 for maximum throughput. Unlike the previous generation, the new drives are helium-filled, which reduces drag compared to air and can lower latency.

Synthetic Storage Benchmarking

Lab testing used fio for four-corner benchmarking: random reads and writes at 4k block size, and sequential reads and writes at 128k block size. Random read performance improved by 10% in latency and 11% in IOPS. Random writes fared better, with 38% lower latency and 60% higher IOPS. Sequential throughput improved by 23% for writes and 90% for reads.

Production ClickHouse Performance

The higher base and turbo frequencies of the Xeon 6210U translated to higher sustained average frequencies during production ClickHouse workloads. A snapshot of two production hosts showed the Core Storage 2020 host sustaining an average CPU frequency 31% higher than the previous-generation hardware while running ClickHouse.

ClickHouse hosts continuously perform merge operations to optimize database structures. Though individual merges take only seconds, their constant execution consumes significant host resources. Merge times were sampled every five minutes over seven days, and the average, minimum, and maximum times were compared between a Compute 2020 host and its predecessor.

Merge Operation Improvements

The lab-measured gains in CPU frequency and storage performance have translated into significantly reduced times for this database operation in production.

Bottom Line

The Core 2020 servers delivered substantial performance improvements in both synthetic and production benchmarks. Cloudflare can now run the same workloads on fewer servers, saving capital expenditures and rack space. The near-identical configurations of the Kubernetes and Kafka hosts have also simplified fleet and spare parts management. Future redesigns will push toward further convergence across the major core workloads.