How the Cloudflare global network optimizes for system reboots during low-traffic periods

Rebooting Without the Ruckus

Cloudflare's network spans more than 300 cities across over 100 countries, with roughly 30 locations in Mainland China. Keeping that fleet updated without disrupting traffic requires a delicate choreography. While the load balancer Unimog seamlessly shifts traffic when a single server goes down, some updates still necessitate a full reboot of a machine. To manage this safely, each data center operates within a designated "maintenance window"—typically a couple of hours—during which reboots are permitted.

The underlying goal is simple: schedule these disruptive actions for the quietest possible moments. However, determining those low-traffic periods was a manual, laborious process. The SRE team was tasked with reviewing historical metrics to define the windows, a responsibility that became increasingly toilsome as traffic patterns shifted and new data centers came online. The system was static, inflexible, and in need of automation.

Finding the Trough with Sine Fitting

The solution was to move away from manual review and toward a mathematical model. By applying sine-fitting to a data center's CPU usage pattern over a configurable period (for example, two weeks), the team could transform real-world traffic data into a theoretical sinusoidal wave. The troughs of this wave represent the lowest-traffic periods, which then become the candidates for the maintenance window.

This approach is appealing because most data centers exhibit a daily cyclical pattern. The math behind it is standard: a sine wave is defined by y(t) = A*sin(2πft + φ), where A is amplitude, f is frequency, t is time, and φ is the phase. In Python, the scipy.optimize.curve_fit function can fit an arbitrary dataset to a curve, but its accuracy improves significantly with good initial guesses for the wave's parameters. In this case, the amplitude is approximated as the standard deviation of the CPU data multiplied by √2. The frequency is locked to one cycle per day, or 1/86400 seconds. The phase requires no guess, and the wave is shifted vertically by the mean of the CPU values.

# Example of the initial parameters for curve_fit
from scipy.optimize import curve_fit
import numpy as np

# Assuming x is time in seconds, y is CPU usage
# f(x) = A * sin(2 * pi * freq * x + phase) + offset
def sine_func(x, amplitude, freq, phase, offset):
    return amplitude * np.sin(2 * np.pi * freq * x + phase) + offset

# The initial guess for the curve_fit
initial_guess = [np.std(y) * np.sqrt(2), 1 / 86400, 0, np.mean(y)]
popt, pcov = curve_fit(sine_func, x_data, y_data, p0=initial_guess)

Implementing the Automated Window

With the theory validated, the logic was integrated into the reboot system. When a new window needs to be determined, the system queries Prometheus for a data center's CPU data and attempts to fit a curve to it. If the fit is accurate enough, the resulting window is cached in Consul along with its metadata. If a data center is too new to have sufficient data for a meaningful fit, fallback logic is applied to set a window.

This process was designed with several practical considerations in mind:

• Caching: The computed window is stored in a Consul key-value store. The value is protected by a session lock with a predetermined validity period, ensuring that the calculation is not repeated needlessly.
• Pre-fetching: Rather than calculating on demand, the reboot system fetches and caches the maintenance window at startup. This makes the information readily available when a server requests a reboot.
• Observability: The system exports metrics to Prometheus, providing a clear view of decisions made and errors encountered. The maintenance window itself is also exported for other internal systems and teams to consume.

When a server is ready to reboot, it first checks whether the current time falls within the maintenance window before running other pre-flight checks. If the window isn't available locally—say, due to a session expiry—it is computed on the fly using the CPU data from Prometheus.

Measuring the Goodness of Fit

Not all data centers conform neatly to a sine wave. Some, like test environments, have constant CPU usage. Others have patterns that are recognizable but slightly skewed. To make intelligent decisions about whether to trust a computed window, the team needed a way to measure fit accuracy. This is the concept of "goodness of fit," often quantified by the chi-squared test.

The chi-squared value compares observed data points to the values predicted by the fitted curve. A chi-squared value close to the number of data points suggests a good fit. A value much smaller indicates an overestimated uncertainty, while a much larger value points to a poor model.

# Calculating fit accuracy
chi_squared = np.sum(((y_data - sine_func(x_data, *popt)) ** 2) / y_data)
fit_accuracy = max(0, (1 - (chi_squared / len(x_data)))) * 100

This calculation yields a fit percentage, allowing the system to classify patterns into three categories:

• Great Fit: Data centers with clear, predictable daily traffic patterns. These are the ideal candidates for automatic window selection and are the most common case.
• Skewed Fit: Data centers that follow a daily cycle but have smaller, secondary troughs or other deviations. The main troughs are still valid for maintenance, but the accuracy score is lower.
• Bad Fit: Data centers that show no sinusoidal pattern, typically those without customer traffic. For these, load-based scheduling is disabled, and an arbitrary default window is used. A different, often faster, reboot schedule is common here to catch issues early.

This accuracy metric is key to making the system smarter. It allows for the automatic acceptance of good windows, enables regression tracking, and identifies data centers with unexpected patterns that might need further investigation.

Looking Ahead

This automated, data-driven approach to maintenance windows reduces SRE toil and improves the safety of reboots. By exporting the windows as a shared data source, other teams can also use this information. For example, downstream services could schedule compute-intensive background tasks to run only during these defined low-traffic periods, making even better use of the network's idle capacity.