Turning server repair into an unattended workflow
Cloudflare runs thousands of servers across more than 310 cities in over 120 countries. Hardware fails somewhere in that fleet constantly; the systems built on top of it are designed to absorb those failures, but finding and fixing the bad machines is a problem of its own. For a long time, that problem was handled by humans.
When a server failed, its state was marked Repair in the internal tracking database. From there, a Data Center Operations engineer had to manually locate the server, run diagnostics, interpret the results, decide whether it could go back to production, file the tickets to re-enable it, and execute the recovery. One server could consume hours of an engineer's day, and machines waiting for repair sat powered on in racks, burning energy while idle.
The fleet was growing faster than the operations team could keep up with. Repair and recovery work was competing with data center expansion for the same limited engineering attention—and losing. The Infrastructure Software Systems and Automation team took on the problem with an autonomous system designed to run the entire diagnostics-and-recovery loop without a human in the middle.
Phoenix: an autonomous repair loop
Phoenix is an automation that runs on a regular schedule to find data centers with broken servers, run diagnostics on those servers, recover the ones that pass through reprovisioning, and re-enable them in production. If a server fails at any stage, Phoenix updates the relevant tickets, identifies the cause of failure, and reverts the server state as needed. No human intervention is required at any point in the flow.
The full process is shown below:
Discovery
Discovery runs every 30 minutes and selects up to two data centers—both configurable values—that have servers in a broken or repair state. Phoenix can discover and process all broken servers in the fleet in roughly three days at that rate. Each run also checks for data centers that already have broken servers queued for recovery and executes the recovery phase for those immediately.
Diagnostics
Diagnostics runs a battery of tests across the broken servers in a selected data center in a single pass, validating hardware components and identifying which servers are candidates for recovery.
The diagnostic operation comprises:
- Out-of-band connectivity check — verifies the device is reachable via IPMI, confirming physical connectivity and accessibility. Only devices that pass move on to node acceptance testing.
- Node Acceptance Tests — runs an internally built tool called INAT (Integrated Node Acceptance Testing), which executes test suites for hardware validation, performance, and related checks.
For every server under diagnosis, Phoenix instructs it to boot into a custom Linux image built around INAT. The image runs its tests on boot and publishes results in HTML for humans and JSON for Phoenix to consume. When the boot diagnostics finish, the server is powered back off to avoid wasting energy. The tests cover benchmarks, CPU/memory/storage checks, drive wiping, and more.
Phoenix writes a summarized result to the server's tracking ticket immediately, including the specific cause of any failure.
Recovery
Recovery executes an expansion operation composed of two phases: first provisioning the servers that passed diagnostics, then re-enabling them back into production. Only servers that are successfully re-enabled begin receiving traffic again.
Servers that pass diagnostics move from Repair state to Pending Provision. If recovery stalls—due to configuration errors, service enablement issues, or failed diagnostics indicating a needed component replacement—Phoenix returns the server to Repair state and notifies the operations team for manual handling. Faulty machines are not re-selected for diagnostics until the required part replacement is complete.
Making recovery autonomous required building in enough intelligence to trust the system with live infrastructure. Phoenix is automation-aware: it knows when other automations are running operations like expansions, and it will only execute a recovery when no other provisioning operation is ongoing in the target data center. It also tolerates misbehaving hardware gracefully, letting a server drop out of the recovery candidate list the moment its behavior threatens to block the operation.
Visibility
Phoenix logs every operation and state transition, posting progress updates to chat rooms and Jira tickets. That historical data feeds dashboards that give different teams a view into automation activity and success rates, and answers common operational questions about repair and recovery.
Putting a bound on risk
The team knew from the start that not every recovered server would stay healthy. A reprovisioned machine can fail again and end up back in Repair, generating monitoring noise for SREs. To keep that from becoming a burden, Phoenix applies the concept of an Error Budget.
The Error Budget limits how many recovered servers may fail and return to Repair state within a given period. The time unit is set to 24 hours, divided into three consecutive buckets that map to the three “follow the sun” SRE shifts. Phoenix can only execute recoveries if failures in a bucket number no more than two. Any excess failures are deducted from the following buckets' budgets, and Phoenix stops recoveries immediately if the budget is exhausted before the time unit ends. The budget fully replenishes at the start of each 24-hour window.
The Error Budget gives Infrastructure Engineering and SRE teams a shared, explicit tolerance for post-recovery failures, letting them tune the balance between pushing recovered hardware back into service and keeping the noise level acceptable.
Outcomes and direction
Phoenix has taken over a workload that previously consumed full engineering days per server. Beyond freeing engineers from repetitive work, the system keeps broken devices powered off in racks, cutting unnecessary energy consumption. The automation has also pushed the team to rethink older processes for handling hardware failures.
The success of Phoenix has made autonomous automation a template for future infrastructure tooling at Cloudflare, a standard the team plans to keep investing in.
