Inside Gen 13- how we built our most powerful server yet

Picking the silicon: AMD Turin 9965

Cloudflare’s Gen 13 server design is built around the AMD EPYC Turin 9965, a 192-core processor. That choice came after lab and production evaluation of three Turin SKUs: the 9755, 9845, and 9965. All three offer more cores than the Genoa-X 9684X used in Gen 12, but with less L3 cache per core. That trade-off matters less now because the shift to FL2, Cloudflare’s Rust-based rewrite of its request-handling layer, has made workloads scale nearly linearly with core count rather than depending on large last-level caches.

The three candidates target different use cases. The 9755 favors per-core performance, the 9845 trades cores for lower socket power, and the 9965 balances efficiency with a high core count. Production testing settled the question in favor of the 9965 for four reasons:

• FL2 removed the L3 bottleneck. The 9965 offers only 2 MB of L3 per core versus 12 MB on Gen 12’s 9684X, an 83.3% drop. But the new software stack uses memory more efficiently, so the 192-core count — double Gen 12’s hardware threads — delivers up to 2x throughput.
• Best performance per TCO. At 500W TDP, the 9965 produced the highest aggregate requests per second of the three candidates, with favorable performance-per-watt at the rack level.
• Operational simplicity. Fewer, denser servers mean less provisioning, patching, and monitoring per unit of compute across the fleet.
• Forward compatibility. The platform supports DDR5-6400, PCIe Gen 5.0, and CXL 2.0 Type 3 memory. The 9965 also gives longer AMD security support, extending the server’s useful life.

Memory: 768 GB at 4 GB per core

The Turin 9965 supports twelve memory channels, and Gen 13 populates all of them with 64 GB DDR5-6400 ECC RDIMMs in a one-DIMM-per-channel (1DPC) layout. That yields 614 GB/s of peak bandwidth per socket, 33.3% more than Gen 12. Fully populating all channels in a balanced configuration — same DIMM type, capacity, and rank — enables interleaving across the full channel set, which spreads contiguous memory access and raises effective bandwidth.

Cloudflare’s target memory-to-core ratio was between 4 GB and 6 GB per core. With 192 cores, that range translates to 768 GB to 1152 GB. DIMM granularity at higher capacities comes in 16 GB increments, so the realistic 1DPC options were:

• 12x 48 GB (576 GB total, 1.5 GB/thread): too little capacity for memory-hungry workloads, violating the lower bound.
• 12x 96 GB (1152 GB total, 3.0 GB/thread): a 50% capacity increase per core, but with higher power draw and a substantial cost premium — memory prices are currently 10x what they were a year ago.
• 12x 64 GB (768 GB total, 2.0 GB/thread): preserves Gen 12’s 4 GB/core ratio while doubling total server capacity. FL2’s more efficient memory use means this headroom supports several years of workload growth without overprovisioning.

Gen 13 also sticks with dual-rank (2Rx4) DIMMs, which Gen 12 testing showed can deliver up to 17.8% higher throughput than single-rank modules at a 1:1 read-write ratio. The configuration provides 1.6 GB/s of memory bandwidth per thread against a ~1 GB/s requirement, leaving margin since current workloads are not memory-bandwidth-bound.

Storage: PCIe 5.0 and a third drive

Gen 13 moves storage to PCIe Gen 5.0 NVMe drives for lower latency and higher bandwidth. Capacity per server grows from 16 TB to 24 TB by populating three of the four E1.S slots with 8 TB drives. The platform keeps the E1.S 15 mm form factor introduced in Gen 12; its high-surface-area design is needed to cool Gen 5.0 controllers that can pull upwards of 25W under sustained heavy I/O, and the 2U chassis provides ample airflow.

The chassis also adds a front drive bay supporting up to ten U.2 PCIe Gen 5.0 NVMe drives. That lets Cloudflare use the same chassis across compute and storage platforms and convert a compute SKU to a storage SKU when needed.

Both the Samsung PM9D3a and Micron 7600 Pro meet the 1 DWPD endurance spec over the server’s 5-year operational life, with roughly 7% hardware over-provisioning. If future workloads demand more endurance, additional user capacity can be held back to increase effective over-provisioning. Both drives adopt the NVMe 2.0 specification and OCP NVMe Cloud SSD Specification 2.0, bringing Zoned Namespaces (ZNS) for write-amplification management, Simple Copy Command for intra-device data movement, enhanced Command and Feature Lockdown, and deeper telemetry for datacenter operations.

Network: From 25 GbE to 100 GbE

For over eight years, dual 25 GbE links have been the standard across our fleet. That backbone served us well, but the compute gains in Gen 13 have finally outgrown it. With 192 modern cores driving more requests and products scaling, the per-port bandwidth is being quadrupled to match.

Why the jump to 100 GbE

NIC bandwidth needs to scale in lockstep with compute performance. On Gen 12, production data across our global co-locations shows P95 bandwidth per port consistently exceeding 50% of available capacity. Since Gen 13 doubles per-server throughput, sticking with 25 GbE links risks saturating the NIC entirely.

The choice of 100 GbE over an intermediate 50 GbE step came down to supply chain economics. 50 GbE transceivers have low industry volume, making them an unreliable bet. Dual 100 GbE ports deliver 200 Gb/s aggregate bandwidth per server and provide headroom for several years of traffic growth.

Hardware and compatibility

Gen 13 maintains a dual-vendor strategy for NICs. Single-sourcing the Gen 11 NIC during the pandemic created supply chain pain we do not intend to repeat.

Both NIC options conform to the OCP 3.0 SFF/TSFF form factor with an integrated pull tab. This keeps chassis commonality with Gen 12, so field technicians need no new tools or training for replacements.

The OCP 3.0 NIC slot is allocated PCIe 4.0 x16 lanes on the motherboard. That provides 256 Gb/s of bidirectional bandwidth, comfortably exceeding the 200 Gb/s aggregate requirement of dual 100 GbE ports.

Management: Staying with DC-SCM 2.0

Gen 13 continues the architectural separation introduced in Gen 12, moving management and security components off the motherboard and onto the Project Argus Data Center Secure Control Module 2.0.

The DC-SCM 2.0 module houses the most security-critical server components:

• BIOS
• Baseboard Management Controller (BMC)
• Hardware Root of Trust (HRoT) and TPM (Infineon SLB 9672)
• Dual BMC/BIOS flash chips for redundancy

Why keep this architecture

The decision to stay with DC-SCM 2.0 is backed by proven security gains from the previous generation. Key benefits include:

• Rapid recovery: Dual image redundancy enables near-instant restoration of BIOS/UEFI and BMC firmware after accidental corruption or a detected malicious update.
• Physical resilience: The Gen 13 chassis moves the intrusion detection mechanism further from the flat chassis edge, making physical tampering harder.
• PCIe encryption: Building on the TSME CPU-to-memory encryption enabled since Gen 10, the AMD Turin 9965 processor extends encryption to PCIe traffic, protecting data in transit across every system bus.
• Operational consistency: Reusing the Gen 12 management stack keeps security audits, deployment, provisioning, and operational procedures fully compatible.

Power: Scaling to 1300W

Upgraded compute and networking naturally expand the server power envelope. Gen 13 ships with larger power supplies to match.

Gen 12 nodes ran comfortably on 800W 80 PLUS Titanium CRPS units. Gen 13 requires a step up to a 1300W 80 PLUS Titanium CRPS. Typical operational power consumption has risen to 850W, a 250W increase over Gen 12's 600W. The primary drivers are the 500W TDP CPU (up from 400W), doubled memory capacity, and the additional NVMe drive.

Why 1300W rather than 1000W? The PSU ecosystem lacks viable high-efficiency options at 1000W. Moving to the next industry-standard tier of 1300W ensures supply chain reliability.

The Titanium grade also keeps Gen 13 compliant with EU Lot 9, a regulation requiring servers deployed in the European Union to maintain minimum power supply efficiency at 10%, 20%, 50%, and 100% load thresholds that match 80 PLUS Titanium certification.

Thermal management in 2U

The 2U1N form factor introduced with Gen 12 continues to pay off. Gen 13 uses 5x 80mm fans (up from 4x in Gen 12) to handle the thermal load from the 500W CPU. The larger fan volume and 2U chassis airflow characteristics mean fans run well below maximum duty cycle at typical ambient temperatures, keeping fan power under 50W per fan.

Accelerator Support

Modularity remains a core fleet requirement. It enabled the rapid global deployment of GPUs to over 100 cities in 2024. Gen 13 maintains high-performance PCIe add-in card support with an updated 2U chassis layout configured for more demanding power and thermal requirements.

While Gen 12 was limited to a single double-width GPU, the Gen 13 architecture supports two double-width PCIe cards.

Final Specifications

Every server generation balances competing constraints: performance versus power, capacity versus cost, flexibility versus simplicity. Gen 13 delivers 2x core count, 2x memory capacity, 4x network bandwidth, and 1.5x storage capacity over Gen 12, while future-proofing accelerator deployments, improving total cost of ownership, and maintaining a robust management and security posture.

Gen 13 servers are fully qualified and will be deployed across Cloudflare's global network in more than 330 cities. As Gen 13 deployment begins, architecture planning for Gen 14 is already underway.