A Second Shot at ARM Servers
Cloudflare's dalliance with ARM server processors began over three years ago with the Qualcomm Centriq. That evaluation of the Falkor cores showed meaningful power savings at comparable performance, but the project stalled when Qualcomm exited the server market shortly after Cloudflare finished porting its software stack to arm64. The company has been waiting since for another server-grade ARM part that could cut power draw across its network, which now spans more than 200 cities in over 100 countries.
The wait may be over. ARM's Neoverse N1 blueprint is a licensing template for power-efficient processors that institutions can adapt to their own needs. Ampere took that blueprint and built the Altra, a chip aimed squarely at companies—like Cloudflare—that own and operate their own server fleets. To see if Altra could power the first generation of ARM edge servers, Cloudflare has been working with Ampere on evaluation. The only other publicly accessible Neoverse N1-based processor is the AWS Graviton2, which lives exclusively inside Amazon's cloud. A head-to-head comparison was the obvious move: Ampere's single-socket Mt. Snow server with an Altra Q80-30 against an EC2 instance running a Graviton2.
The Mt. Snow 1P server equipped with the Ampere Altra Q80-30
Two Chips from the Same Blueprint
Both the Ampere Altra and the AWS Graviton2 are built on ARM's Neoverse N1 reference design and fabricated on a TSMC 7nm process. The N1 core itself features an 11-stage out-of-order execution pipeline.
| Pipeline Stage | Width |
|---|---|
| Fetch | 4 instructions/cycle |
| Decode | 4 instructions/cycle |
| Rename | 4 Mops/cycle |
| Dispatch | 8 µops/cycle |
| Issue | 8 µops/cycle |
| Commit | 8 µops/cycle |
Each N1 core carries 64KiB of L1 instruction cache, 64KiB of L1 data cache, and a dedicated L2 cache that scales up to 1MiB per core. The shared System Level Cache (L3) can be sized up to 256MiB, broken into slices of up to 4MiB each.
| Cache Hierarchy | Capacity |
|---|---|
| L1i Cache | 64 KiB |
| L1d Cache | 64 KiB |
| L2 Cache | Up to 1 MiB |
| L3 Cache | Up to 256 MiB |
N1 cores can be grouped in pairs via the Component Aggregation Layer. Those pairs then attach to the Coherent Mesh Network (CMN-600), a mesh interconnect that scales to 64 pairs—128 cores total—using the Coherent Hub Interface for component communication.
Component Aggregation Layer (CAL) supports up to two N1 cores
Coherent Mesh Network (CMN-600) can be scaled up to 8x8 XPs supporting up to 128 cores
Verifying the shared lineage is straightforward: CPU part in cpuinfo returns 0xd0c on both systems, the designated part number for the Neoverse N1.
sung@ampere-altra:~$ cat /proc/cpuinfo | grep 'CPU part' | head -1
CPU part : 0xd0cKey Differences
Ampere backported speculative side-channel attack mitigations—specifically Meltdown and Spectre variants 1 and 2—from the ARMv8.5 specification into the Altra. That backport earns the "+" designation in the Altra's ISA, which is listed as ARMv8.2+.
The Altra holds two notable hardware advantages over the Graviton2: 25% more physical cores and a 20% higher operating frequency. Taken together, those numbers suggest the Altra could perform up to 50% better in compute-bound workloads.
Both processors implement 1MiB of L2 cache per core and 32MiB of L3 cache. That 32MiB figure is far below the 256MiB ceiling allowed by the CMN-600 spec, and it leaves the Altra with a lower L3 cache-per-core ratio than the Graviton2. In workloads where the working set exceeds the L3 cache, that disparity could put the Altra at a disadvantage.
Benchmark Setup
Cloudflare imaged both systems with Debian Buster and ran its open-source cf_benchmark suite, which executes 49 different workloads in roughly 15 to 30 minutes. Each workload exercises a library that Cloudflare has either used in its stack or seriously considered. The suite functions as a smoke test for any new hardware, giving technology partners a quick way to confirm workloads compile and run correctly.
Multiple runs showed no significant run-to-run variation. Overall scores were calculated using the geometric mean across all 49 workloads, with category scores computed the same way over their respective subsets. In single-core scenarios, the suite spawns a single application thread. In multi-core scenarios, it spawns threads until all hardware threads are occupied. Neither processor implements simultaneous multithreading, so each physical core executes exactly one hardware thread.
Overall Performance Findings
Given the Altra's advantages in both frequency and core count, results broke as expected. In single-core tests, the Altra outperformed the Graviton2 by 16%, with frequency alone providing up to a 20% edge in more than half the workloads. In multi-core tests, the Altra extended its lead to 31%. The theoretical advantage for the Altra sits between 20% and 50%, accounting for scaling overhead in the mesh network; the majority of multi-core workloads indeed landed within that range.
Workload Category Results
Across OpenSSL and LuaJIT, the Altra scaled proportionally with its clock speed in single-core performance and continued scaling with its core count in multi-core runs. The same held for the vast majority of compression and Golang workloads, with two exceptions: Brotli level 9 compression and several Golang multi-core workloads that failed to keep either processor fully busy.
OpenSSL
Cloudflare uses a fork of OpenSSL called BoringSSL for TLS handshake handling among other duties. OpenSSL's built-in speed benchmark measures both single- and multi-core performance, and the asymmetric and symmetric crypto workloads it runs saturated both processors at 100% CPU utilization. The Altra held a consistent 20% advantage over the Graviton2 in single-core tests and averaged 47% better in multi-core tests, scaling cleanly with its additional cores.
LuaJIT
LuaJIT has been described as the glue that holds Cloudflare together, and it remains widely used in gaming for the same reason: performance and responsiveness are non-negotiable. Though Cloudflare has been shifting LuaJIT workloads toward Rust, interest in the Ampere Altra's instruction cache coherency is high. That feature addresses a known LuaJIT weakness: the just-in-time compiler's self-modifying code can force instruction fetches to miss the instruction cache because the needed instruction sits in the data cache. With coherency enabled, the L2 cache becomes inclusive of the L1 instruction cache, and the L2 intercepts invalidations or stores to propagate changes up to the L1.
Single-core LuaJIT results were strong for the Altra except in the fasta benchmark, where both processors stalled at the front-end of the pipeline. In the multi-core version of fasta, the Altra spent fewer cycles stalled, likely thanks to the coherency implementation. Other workloads scaled well; binary trees showed a 73% multi-core gain. The spectral workload was omitted entirely because its multi-core variant failed to run on both servers.
Compression: Brotli and Gzip
Brotli and Gzip are Cloudflare's two primary compression algorithms, chosen to balance CPU cost against storage capacity and transfer speed. The Altra performed well on both algorithms, with two exceptions: Brotli level 9 and Gzip level 4 in multi-core mode.
Brotli level 9 produced the largest gap between the two processors anywhere in the benchmark suite—notable even though Cloudflare doesn't use Brotli level 7 or above for dynamic compression. The severity warranted further investigation.
Historically, most processors degrade noticeably at level 9. Compression workloads carry a high mix of branch instructions—roughly 15% to 20% of dynamic instructions. The first suspect was misprediction rate, since high miss rates accumulate penalty cycles quickly. But the misprediction rate at level 9 turned out to be surprisingly low.
Deeper analysis pointed to a different culprit: a high number of page faults at level 9. Ampere has shown that increasing page size from 4K to 64K bytes alleviates the bottleneck, bringing the Altra to parity with the Graviton2. Cloudflare plans to experiment with larger page sizes as Altra evaluation continues.
Gzip
Golang
Go is one of Cloudflare's primary languages alongside LuaJIT, Rust, and C++. The Golang test set covers cryptography, compression, regular expressions, and string manipulation. In the vast majority of single-core workloads, the Altra performed proportionally to its frequency advantage over the Graviton2. Multi-core performance followed suit except for the workloads that could not push both processors to 100% utilization; those did not scale proportionally to the hardware differences.
Power Draw and Thermal Behavior
The AWS Graviton2 instance exposed no frequency or power sensors, so we could not collect telemetry on it. The Ampere Altra model we tested was the Q80-30, rated at a 3.0GHz operating frequency and a 210W TDP.
During cf_benchmark, the Altra sustained its full operating frequency, though its dynamic voltage and frequency scaling (DVFS) mechanism occasionally dropped the clock between workloads to save power. Package power varied by workload, but the Altra never approached its rated TDP while the default fan curve kept temperatures below 70°C. At the system level, querying the baseboard management controller over IPMI showed the Ampere server drawing no more than 300W. Lower system power translates directly into more flexibility for rack population, and we hope this trend carries into production deployments.
What We Learned
In our top-down assessment, the Ampere Altra behaved largely as expected across the benchmark suite. The two Neoverse N1 processors consistently operated at their rated frequency and core count, and the Altra outperformed the AWS Graviton2 in the key areas we measured. Its higher operating frequency delivered better single-core results, and the larger core count multiplied that advantage in multi-core workloads. We also found the Altra evaluation server consumed less power than anticipated.
Comparing the Altra with contemporary x86 parts surfaced several notable design decisions, including the absence of simultaneous multithreading and dynamic frequency boost. Which philosophy proves better suited to the cloud remains to be seen. We are currently evaluating Altra’s performance against our existing server fleet and working with Ampere to define an ARM edge server based on Altra for potential large-scale deployment.
