Is this thing on? Using OpenBMC and ACPI power states for reliable server boot

Boot reliability in a fleet managed by OpenBMC

Cloudflare runs thousands of servers across its global network, and each one carries a Baseboard Management Controller (BMC)—a dedicated management processor that runs independently of the host CPU. Unlike proprietary BMC firmware, Cloudflare builds and deploys its own stack based on OpenBMC, the Linux Foundation project for open-source BMC firmware. That choice gives Cloudflare transparency into a security-critical subsystem and the ability to fix or extend firmware quickly across the fleet.

That control came with a learning curve. After moving servers to OpenBMC, Cloudflare's engineering team started seeing boot failures that were hard to diagnose using existing BMC tooling. The problems fell into a few recurring categories, all rooted in how the BMC and the host's UEFI (BIOS) firmware coordinate—or fail to coordinate—during the power-on sequence.

Reading power state through ACPI and IPMI

The BMC exposes management interfaces over IPMI and Redfish, and at Cloudflare, engineers commonly use ipmitool to query chassis power state and issue power commands. But in some failure modes, those interfaces returned misleading information.

In one scenario, a request to power on a host appeared successful—IPMI reported the chassis power state as ON—but there was no power reaching the CPU. The System Event Log (SEL) showed the host transitioning through standard ACPI states. The reported a soft-off (S5_G2) condition, then indicated the host was in a working state (S0_G0), followed by a system restart event. According to the logs, the host looked fully operational. A sensor called CPU_POWER told a different story: it read 0 watts, meaning the CPU was not powered at all.

That kind of discrepancy points to a known limitation of ACPI states: they describe power states of the whole system, not of individual subsystems. All those log entries came from UEFI, and they were not necessarily accurate to what the hardware was actually doing. For the servers involved, the UEFI did not implement sending boot progress details to the BMC, making the SEL an incomplete source of truth.

The missing memory module problem

A more subtle failure involved memory. Servers would occasionally boot with fewer DIMMs than physically installed. The root cause was not loose hardware or faulty memory, but contention on the I2C bus between the BMC and the UEFI.

During host boot, the UEFI reads the Serial Presence Detect (SPD) from each memory module over I2C to determine what memory is installed. At the same time, the BMC reads DIMM temperature sensors over the same I2C bus. When the two controllers tried to access the bus simultaneously, one of them was blocked—and if that one was the UEFI, it could fail to see a DIMM and simply skip it during initialization.

BMC journals during such boots showed interrupt failures at I2C address 1e78a300, which connects to the SPD of the memory modules. The failure of an interrupt request prevented the UEFI from completing its read, resulting in a server that came up with less memory than it had.

Fan failure and thermal telemetry

The same I2C contention issue surfaced thermal problems as well. DIMM temperature sensors could appear as failed during boot, prompting fans to enter a fail-safe Pulse Width Modulation (PWM) mode. In that mode, fan speeds are set to preset values regardless of thermal optimization, which can degrade cooling efficiency and increase power draw across the server.

When the BMC cannot read a temperature sensor, it cannot safely adjust fan speed, so the fail-safe behavior is protective by design. But when the underlying cause is a race condition between two subsystems rather than an actual thermal event, the result is unnecessary power overhead and a server that is not running as efficiently as it could.

A stateful gap at boot

What the three failure modes have in common is the asynchronous nature of subsystem initialization. The CPU, memory controller, fans, and BMC all power up and begin their own initialization sequences at different times. Without clear handshakes between the entities that manage those subsystems—primarily UEFI and the BMC—race conditions emerge.

Standard power-state reporting through ACPI and SEL describes the chassis or the host as a whole. It is not granular enough to reveal that the CPU has no power or that the UEFI missed a DIMM. For reliable boot at fleet scale, both the UEFI and the BMC need to be able to track state transitions below the level of "the host is on."

UEFI boot progress, if implemented, should be forwarded to the BMC: this delivers the kind of detailed SEL events needed to distinguish a working host from one that is failing to come up. But UEFI is developed by the OEMs and can be a black box—there are no guarantees that such logging is turned on.

Why ACPI state tracking matters for reliable boot

Debugging boot failures led us to examine how power states change across the subsystems inside a server chassis. ACPI, the Advanced Configuration and Power Interface specification, gives operating systems a standardized way to manage power. It supports OS-directed power management and hardware features such as a real-time clock to wake a server from sleep, and it defines the global "Gx" states, sleeping "Sx" states, and device "Dx" power states used across the industry.

For our purposes, three states are significant:

• S0_G0_D0: The working state, where the host is running normally.
• S2_D2: Memory context is preserved but CPU context is lost. This is the state we use to detect when the host's UEFI is doing platform firmware initialization.
• S5_G2: The soft-off state. Power still reaches the chassis, but neither processor nor DRAM context is retained, and the OS has no power-management context.

Because our problems appeared when we asked the host to reboot or power on, we needed to observe the host's power-state transitions between these conditions. The OpenBMC community had already built an ACPI daemon for this purpose. We extended it, adding the S2_D2 state so the BMC knows when UEFI firmware initialization is under way, and assigned management tasks to each ACPI power state.

One example is re-exporting Voltage Regulator (VR) sensors when the host enters S0_G0_D0:

cat /lib/systemd/system/Re-export-VR-device.service 
[Unit]
Description=RE Export VR Device Process
Wants=xyz.openbmc_project.EntityManager.service
After=xyz.openbmc_project.EntityManager.service
Conflicts=host-s2-state.target

[Service]
Type=simple
ExecStart=/bin/bash -c 'set -a && source /usr/bin/Re-export-VR-device.sh on'
SyslogIdentifier=Re-export-VR-device.service

[Install]
WantedBy=host-s0-state.target

OpenBMC's phosphor-host-ipmid exposes the ipmiSetACPIState Net Function, which the host invokes with NetFn=0x06 and Cmd=0x06. For immediate power cycles where the host cannot send S5_G2, we patched x86-power-control so the BMC itself recognizes when the host has entered soft-off. Once the host powers back on, UEFI runs POST and reports S2_D2; after booting the OS, the host signals S0_G0_D0.

What was actually going wrong

Most of the boot problems traced back to devices sharing a power domain with the CPU and interfering with UEFI/platform firmware initialization. Using the ACPI state as a trigger, we applied three fixes.

Servers not booting: We identified which devices disrupted the POST stage and used the host ACPI state to delay switching those devices into their active power mode until after firmware initialization completed.

Memory modules missing: DIMMs are powered and initialized during S2_D2. At that point UEFI reads the Serial Presence Detect (SPD) on each DIMM for enumeration, while the BMC may concurrently be reading DIMM temperature sensors. That caused SMBUS collisions, which could fail either the temperature reads or the UEFI enumeration. A failed enumeration made the server boot with reduced memory capacity, looking like faulty DIMMs. We resolved the race by disabling BMC temperature reads during S2_D2 and setting fixed fan speeds, so UEFI gets unobstructed access to the DIMM subsystems.

Thermal telemetry: In S0_G0, if sensors stop reporting, the BMC assumes overheating and puts fans into fail-safe mode at maximum speed. In S5_G2, however, some thermal sensors (CPU, NIC, and others) are unpowered by design. Rather than wait for read errors, we mark those sensors non-functional in their exported configuration while in S5_G2 and during the transition to S2_D2, which keeps fans out of fail-safe mode.

Beyond debugging

Tracking ACPI states has also improved our automated firmware regression testing. Tests that require hundreds of power cycles now assert against the actual ACPI power-state transitions instead of a simple running/not-running or pingable/not-pingable check.

Over time, implementing ACPI states is moving us toward three goals: all components are enabled by the end of the boot sequence; BIOS and BMC can both retrieve component information; and the BMC knows when thermal sensors are non-functional.

We have also started integrating the BootProgress object from the Redfish ComputerSystem schema into our systems. That gives us observability into pre-OS boot progress, which gives us a clearer debug starting point if UEFI fails during platform initialization.