Deploying firmware at Cloudflare-scale: updating thousands of servers in more than 285 cities

Firmware updates without the forklift

Keeping server firmware current is a security and reliability requirement, but at Cloudflare’s scale — servers from multiple vendors spread across more than 285 cities — the usual manual update flow does not scale. Waiting for someone to navigate a vendor utility on each box would stall patching for months. Instead, the SRE team relies on automation built around iPXE, an open-source network boot loader, to flash BIOS, BMC, and NIC firmware as part of the ordinary reboot cycle.

Every server in the fleet boots UEFI, and each machine is rebooted monthly as a routine maintenance window. That reboot is the natural moment to perform firmware work: the box is already out of production. The entire process is driven by iPXE scripts that check the current firmware version, and only if a mismatch is found, boot into a dedicated flashing environment.

Flashing BIOS from the UEFI shell

For vendors that ship a UEFI-compatible flashing binary, the update can happen before any operating system loads. iPXE exposes the current BIOS version through its built-in variable ${smbios/0.5.0}, allowing a boot script to compare it against the desired version and decide whether to flash.

# Check whether the BIOS version is 2.03
iseq ${smbios/0.5.0} 2.03 || goto biosupdate
echo Nothing to do for {{ model }}
exit 0

:biosupdate
echo Trying to update BIOS/UEFI...
echo Current: ${smbios/0.5.0}
echo New: 2.03

imgfetch ${boot_prefix}/tools/x64/shell.efi || goto unexpected_error
imgfetch startup.nsh || goto unexpected_error

imgfetch AfuEfix64.efi || goto unexpected_error
imgfetch bios-2.03.bin || goto unexpected_error

imgexec shell.efi || goto unexpected_error

The downloaded files include a startup.nsh script, which the UEFI shell executes automatically. It contains the exact command needed to run the vendor’s flashing utility.

%homefilesystem%\AfuEfix64.efi %homefilesystem%\bios-2.03.bin /P /B /K /N /X /RLC:E /REBOOT

After the flash completes and the machine reboots, the new BIOS version is already in place, so the same iPXE script now sees a match, skips the flashing path, and boots the server into production.

Handling vendors without UEFI flash tools

Most vendors do not provide UEFI flashing binaries for components like BMCs or NICs. They do, however, almost always support flashing from Linux. Cloudflare exploits that by booting a minimal Linux environment dedicated solely to the firmware update task.

The key security design choice: the flashing environment never configures a network interface. If it is not reachable, it is not exploitable. That also means all necessary tools and firmware must be injected during the boot itself, which is done via an initial ramdisk (initrd). iPXE can load multiple initrd images into the boot process, and constructing one is simply a matter of archiving the needed files using cpio in the newc format.

For a Broadcom NIC update, for example, the initrd would contain the bnxtnvm utility, the firmware package firmware.pkg, and a shell script named flash that runs the actual update.

cd broadcom
find .
./opt/preflight
./opt/preflight/scripts
./opt/preflight/scripts/flash
./opt/broadcom
./opt/broadcom/firmware.pkg
./opt/broadcom/bnxtnvm

Those files are compressed into a single image called broadcom.img, which is then presented to iPXE as an additional initrd.

find . | cpio --quiet -H newc -o | gzip -9 -n > ../broadcom.img

Since reading the NIC’s current firmware version from the UEFI shell is not practical, the fleet uses UEFI variables as the source of truth. After a successful flash from Linux, the boot environment writes a version marker via efivars, the UEFI variable file system. On the next boot, iPXE reads that variable and decides whether another flash is needed.

declare -r fw_path='/sys/firmware/efi/efivars/broadcom-fw-9ca25c23-368a-4c21-943f-7d91f2b76008'
declare -r efi_header='\x07\x00\x00\x00'
declare -r version='1.05'

/bin/mount -o remount,rw,nosuid,nodev,noexec,noatime none /sys/firmware/efi/efivars

# Files on efivarfs are immutable by default, so remove the immutable flag so that we can write to it: https://docs.kernel.org/filesystems/efivarfs.html
if [ -f "${fw_path}" ] ; then
    /usr/bin/chattr -i "${fw_path}"
fi

echo -n -e "${efi_header}${version}" >| "$fw_path"

An iPXE configuration then loads the flashing kernel, its userland, and the injected initrd.

set cf/guid 9ca25c23-368a-4c21-943f-7d91f2b76008

iseq ${efivar/broadcom-fw-${cf/guid}} 1.05 && echo Not flashing broadcom firmware, version already at 1.05 || goto update
exit

:update
echo Starting broadcom firmware update
kernel ${boot_prefix}/vmlinuz initrd=baseimg.img initrd=linux-initramfs-modules.img initrd=broadcom.img
initrd ${boot_prefix}/baseimg.img
initrd ${boot_prefix}/linux-initramfs-modules.img
initrd ${boot_prefix}/firmware/broadcom.img

The actual vendor flashing utility is not run directly. Instead, the initrd contains scripts that are copied into /opt/preflight/scripts inside the ramdisk. A systemd service named preflight.service is configured to run those scripts on boot using run-parts.

[Unit]
Description=Pre-salt checks and simple configurations on boot
Before=salt-highstate.service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/run-parts --verbose /opt/preflight/scripts

[Install]
WantedBy=multi-user.target
RequiredBy=salt-highstate.service

The whole update path is automated from the moment the server reboots, with no SRE touching the machine or waiting on a vendor console. The same mechanism that keeps operating systems patched monthly also keeps low-level firmware aligned.