How Cloudflare Automates Kernel Patching to Keep Pace With CISA’s Secure-by-Design Goals
When Cloudflare signed CISA’s Secure-by-Design pledge in May 2024, it committed to a set of security goals that prioritize customer safety. Among those goals is a requirement to demonstrate concrete actions that increase the installation of security patches. For a company running tens of thousands of servers at the edge, meeting that requirement means solving one of the harder operational problems in systems engineering: applying Linux kernel updates without degrading user experience.
The Reboot Tradeoff
In 2024, more than 1,000 CVEs were logged and patched against the Linux kernel. Cloudflare tracks the longterm stable branch of the kernel, pulling each new release into its internal repository where it is automatically built and tested. But testing is only the beginning. The kernel is a special piece of software: no matter how carefully it is compiled or configured, it only takes effect after a full system restart.
That restart presents a problem at Cloudflare’s scale. A single server in its network may be processing hundreds of thousands of requests at any moment. Simply rebooting it to apply a kernel patch would interrupt that traffic. Before a reboot can happen, the server must be safely drained of all work. The process begins with marking the server for maintenance, which signals the load balancer unimog to stop sending it public traffic. Once that flow has terminated, the server disables internal traffic—the routing, service discovery, and health check traffic that keeps the network coordinated. Only when the server is serving no traffic at all is it safe to restart with the new kernel.
This restart cycle is not merely tolerated; it is an operational tool. Frequent restarts exercise the reboot process itself, apply firmware updates, and refresh filesystem state. Cloudflare builds this expectation into its infrastructure so that when a critical patch arrives, the system is already prepared to cycle through its fleet.
A Staged Path to Production
The lifecycle of a kernel at Cloudflare begins upstream. When a new kernel version is released, it is automatically pulled into Cloudflare’s internal build and test pipeline. Once those tests pass, multiple kernel variants are built and prepared for deployment. The rollout proceeds through defined stages, each with stricter requirements than the last.
The first deployment target is an internal environment that does not receive live traffic. After confirming there are no crashes or anomalous behavior, the kernel moves to a production environment where the “eyeballs” are Cloudflare employees connected through Zero Trust. Telemetry from this stage captures CPU utilization, memory usage, and filesystem behavior, which are analyzed for deviation from the previous kernel. This stage is the first time a new kernel touches live traffic in a Cloudflare context.
Once kernel performance is validated, deployment expands to customer traffic—first as a small percentage of traffic across multiple data centers, then scaled to a single large regional data center. That qualification phase gathers data on real-world conditions before a candidate release is approved for the entire network.
Full rollout happens through an automated cycle called the Edge Reboot Release. This cycle runs every 30 days, guaranteeing that the infrastructure runs an up-to-date kernel each month. For vulnerabilities that require action faster than the standard cadence allows, Cloudflare can close the gap using live patching techniques built on eBPF and LSM.
Bringing the Control Plane Up to the Same Standard
The network edge was not the only part of the infrastructure with aging kernels. The Control Plane runs a different set of workloads than the network—about 80 clustered workloads responsible for persistent state and decisions that feed the edge. Until 2024, kernel maintenance in the Control Plane was performed ad hoc, which caused its running kernel to fall behind on patches. That inconsistency did not satisfy the Secure-by-Design pledge, so Cloudflare set out to make Control Plane patching as reliable as the network’s.
The challenge with clustered workloads is that each one requires a distinct methodology for safely restarting a member. Consider a relational database: to restart the primary instance, a replica must be available to take its place. That replica is promoted to primary, leaving the original instance free for restart. If a replica itself needs a reboot, another replica is required to fill in during the downtime.
To manage this complexity, Cloudflare built an internal tool called Reboau (short for reboot automation). It handles the custom reboot logic for Control Plane workloads, with added efficiencies at the rack level rather than for individual servers. For clustered workloads, draining and rebooting an entire rack of servers can be efficient, and Reboau monitors metrics to determine when it is safe to lose a cluster member, executes the reboot, and confirms the system recovers health.
In 2024, Cloudflare migrated its Control Plane workloads to Reboau, bringing them onto the same kernel upgrade cadence as the network. The result is uniform patching across the full infrastructure, helping close the gap between edge and core systems.
Patching as a Default Practice
Cloudflare’s approach to kernel maintenance is one part of the broader CISA Secure-by-Design effort, which is organized around seven security goals. The company states it has delivered five of those seven goals so far and aims to complete the remaining ones by May 2025. For other organizations building software, the takeaway is less about the specific tooling and more about the pattern: automation that supports frequent restarts, staged rollout with telemetry at each phase, and explicit design for secure patching rather than leaving it as an afterthought.
