How we reduced core unit boot time from hours to minutes

When Core Servers Take Four Hours to Boot

Cloudflare's core data centers run the control plane, billing, and analytics — a workload distinct from the globally distributed edge that handles user traffic. Core servers are bare metal, and when something goes wrong during a reboot, the impact spreads quickly. The boot sequence is orchestrated by UEFI, the firmware standard that initializes hardware and hands control to the operating system. Small quirks in that handoff can have outsized consequences.

After a routine firmware update, some core servers began taking four hours to come back online, instead of the minutes they had taken before. A fleet-wide rollout that should have taken one day stretched into a multi-day effort. New nodes faced the full gauntlet of timeout delays on their first boot, maintenance windows expanded, and engineering teams had to watch over upgrades that were designed to run unattended.

The issue surfaced when nodes that had been powered off for an extended period were brought back online. Their firmware was out of date and needed multiple updates, combined with newer boot protocols that had been rolled out to servers in certain locations. The result was unacceptable boot times.

Why Network Boot Was the Bottleneck

Network boot lets a server load its operating system over the network rather than from local storage, which is essential for centralized control of machines across a globally distributed fleet. Cloudflare uses two primary network boot interfaces: PXE and UEFI HTTPS boot. In practice, most reboots go through PXE for automation reasons, relying on the open-source iPXE firmware, which supports modern protocols like HTTP and HTTPS.

iPXE effectively turns booting into a programmable workflow, with scripting that lets teams automate complex provisioning and diskless operations. Some hardware also supports native UEFI HTTPS network boot, allowing the motherboard firmware to securely fetch OS files.

The problem began after that firmware update, when monitoring showed machines stuck in a pre-OS state for far too long. The initial hypothesis was a regression in the firmware itself. But viewing the serial console during a boot cycle told a different story: POST completed, hardware initialized fine, and then the server sat idle. Instead of quickly reaching the correct network boot stage, it tried an IPv4 HTTPS boot, timed out after several minutes, then tried IPv4 iPXE, timed out again, and repeated both before finally reaching the IPv6 HTTPS interface that would succeed.

Each failed attempt burned roughly five minutes waiting for a timeout. With four failed attempts stacking up, a single boot cycle wasted around twenty minutes. For routine reboots that was painful; for firmware automation requiring multiple sequential reboots per server, the compounded delays pushed boot time to roughly four hours.

Declaring the Boot Interface Instead of Searching for It

The root cause was straightforward: the servers were blind-searching every available network boot interface, waiting for each to fail before moving to the next. The fix was to declare the correct boot interface upfront so the system never wastes time on interfaces that will never respond. Implementation, however, proved more complicated, with obstacles in the boot automation workflow, vendor-locked settings, and inconsistent naming conventions across NIC vendors.

Boot Automation Sequence and Firmware Upgrades

Boot automation runs in three stages: firmware initialization, pre-boot, and kernel startup. The PXE pre-boot stage sets up the network card and executes the bootloader that starts the kernel. On first boot, firmware upgrades are integrated into this workflow, and because each upgrade requires a reboot with the full network boot attempt sequence, boot time swelled to nearly four hours.

Restructuring the automation to declare the network boot interface order early in the pre-boot PXE stage, per hardware configuration and use case, cut the total time by about an hour. The boot process no longer spent 20 minutes probing for each firmware upgrade.

Two Constraints: Legacy UEFI and Configuration Persistence

Declaring the network boot interface order introduced two specific constraints: legacy UEFI versions don't support boot ordering, and UEFI firmware upgrades often reset configuration settings. To address both, the firmware automation now includes a state validation step that re-applies the configuration and triggers a reboot if settings have been modified.

Although the first boot may take slightly longer, future startups drop from about 20 minutes to less than a minute.

Vendor Locked Settings

The Network Boot settings data structure, EFI_IFR_REF3, was being lazy loaded — not instantiated until explicitly accessed through a GUI callback.

typedef struct _EFI_IFR_REF3 {
  EFI_IFR_OP_HEADER          Header;
  EFI_IFR_QUESTION_HEADER    Question;
  EFI_QUESTION_ID            QuestionId;
  EFI_GUID                   FormSetId;
} EFI_IFR_REF3;

While this is standard practice to speed up BIOS boot times, it made the "Network Boot Interface" invisible to programmatic scans, so automation couldn't discover the priorities. Cloudflare worked with vendors to enable specific tokens inside the fixed "Boot Order Module," which forces the network boot interface to be discovered during boot without manual GUI interaction.

The manufacturers' UEFI also had an immutable setting, Force Priority Httpv4 Httpv6 Pxev4 Pxev6, which prevented changing the boot order. Resolving that required a new BIOS version from the vendor and a debug session when setting the boot order.

Inconsistent NIC Vendor Strings

Network interface card vendors used different strings to identify same-function interfaces, which broke the configuration matching when iPXE tried to set boot order:

UEFI: HTTPS IPv4 Ethernet Network Adapter XXX-XXX-Y for OCP 3.0 P1
UEFI: HTTPS IPv4 Network Adapter - 50:00:E6:8F:4F:32 P1

A feature was added to the CfHIIConfig_App tool to allow setting config without the full string, using a pattern like .*HTTP.*IPv4.*P1 to match against accepted config strings and select the correct boot order. Cloudflare is working with UEFI vendors to standardize these strings to include only relevant details — protocol, transfer type, port number, and physical slot index — and drop product details like the MAC address. Those can be read from the NIC's embedded vital product data, which would eliminate both configuration drift and wildcard matching.

Verifying Configuration via iPXE

iPXE reads the network boot variable as hex, so checking whether the setting was modified required extra steps. A boolean flag, uefi-same-hex, was implemented to indicate whether a configuration changed, allowing iPXE to run a single set command instead of first running show to compare, then set only if config differed.

# construct path to read the update variable
set buffer-var-guid 91468514-75bc-4bb5-8f33-91efff9e9b1f
set var-upd-path efivar/CfHIIVarUpd-${buffer-var-guid}

#Run the config change command
imgexec <signed CF UEFI configuration App> set ${uefi-setting}=${uefi-value}

#Compare the update variable with the expected value if it has changed.
#If it has changed, set the local variable to reboot the system
iseq ${uefi-same-hex} ${${var-upd-path}} || set has-changed ${uefi-diff-hex}

From Four Hours to Three Minutes

Eliminating the guesswork from the network boot sequence turned a four-hour process back into a three-minute one. Changes are now dynamic, no manual BIOS interactions are needed, a single BIOS firmware image serves all SKUs, and configuration updates deploy through the existing release pipeline — all operating from iPXE.

The fix required deep work with UEFI internals, collaboration with OEM vendors to unlock programmatic boot order control, and use of open-source tools like iPXE to build scalable automation. The result is a clear framework for identifying and eliminating unnecessary delays in network boot sequences across bare-metal infrastructure.