Centralized maintenance control
Cloudflare operates data centers in more than 330 cities, but that geographic redundancy does not make disruptive maintenance planning simple. As the network grew, manual coordination between infrastructure and network operations teams became unmanageable. Tracking overlapping maintenance windows and customer-specific routing requirements in real time exceeded what human oversight could reliably handle.
The solution was a centralized scheduling system built on Cloudflare Workers that could evaluate the entire network state programmatically. This scheduler enforces safety constraints so that routine hardware updates cannot inadvertently take down critical infrastructure paths or violate customer configuration requirements.
Identifying dangerous maintenance scenarios
Consider an edge router that is one of a small redundant group of gateways connecting the public Internet to multiple Cloudflare data centers in a metro area. If all routers in that cluster go offline simultaneously, those data centers lose connectivity. The Zero Trust product Dedicated CDN Egress IPs — formerly called "Aegis" — presents another challenge: customers select specific data centers for their user traffic to egress toward origin servers. If every selected data center is offline at once, customers suffer higher latency and potential 5xx errors.
The maintenance scheduler addresses both scenarios. It ensures at least one edge router stays active in a region and flags when overlapping maintenance events would take all of a customer's Aegis data centers offline simultaneously. Previously, such simultaneous disruptive events could cause downtime; now the scheduler alerts operators of conflicts and suggests alternative times that avoid overlapping with related maintenance.
Defining constraints
Each maintenance constraint starts with a proposed set of maintenance targets, such as routers or servers, and identifies all calendar events overlapping that proposed time window. The system then aggregates relevant product data — for example, Aegis customer IP pools that map egress IP ranges to specific data center IDs.
If data center 21 and data center 45 both serve the same Aegis customer pool, taking both down simultaneously would break that customer's egress traffic. The coordinator flags this conflict before the maintenance proceeds.
Graph processing on Workers
An initial naive implementation loaded all server relationships, product configurations, and health metrics into a single Worker to compute constraints. Even the proof of concept hit "out of memory" errors. Workers platform limits required a more selective data loading strategy: a maintenance request for a router in Frankfurt does not need data about Australian data centers with no overlapping dependencies.
Examining the constraints revealed a recurring pattern of objects and associations — in graph terms, vertices and edges. A router object, for instance, has associations to the Aegis pools in the data centers it serves. Drawing inspiration from Facebook's TAO research paper, the team built a typed graph interface over product and infrastructure data:
DATACENTER_INSIDE_AEGIS_POOLretrieves the Aegis customer pools a data center belongs to.AEGIS_POOL_CONTAINS_DATACENTERretrieves the data centers a pool needs to serve traffic.
These associations are inverted indices of each other. The access pattern is identical to before, but the graph implementation controls how much data gets queried. Instead of loading all Aegis pools into memory and filtering inside business logic, the system fetches only relevant data directly. This interface allows performance improvements to happen behind the scenes without complicating constraint logic, leveraging Workers' scalability and the CDN for fast internal data retrieval.
Fetch pipeline optimization
Switching to targeted graph queries reduced response sizes by 100x overnight — from a few massive payloads to many tiny requests. But this created a subrequest problem: instead of a handful of large HTTP requests, the Worker now issued an order of magnitude more small requests, consistently breaching subrequest limits.
A smart middleware layer between the graph implementation and the fetch API solved this. The pipeline includes three caching mechanisms:
- Request deduplication — inspired by Go's
singleflightpackage, it makes inflight HTTP requests wait on the same Promise instead of spawning duplicate requests. - LRU cache — a lightweight in-memory cache for previously seen requests.
- Edge cache — Cloudflare's
caches.default.matchcaches all GET requests at the region where the Worker runs.
TTL values vary by data source: real-time data caches for 1 minute, relatively static infrastructure data for 1–24 hours, and manually changed power management data for longer periods at the edge.
The pipeline also applies standard exponential backoff, retries, and jitter. This prevents wasted fetch calls when downstream resources are temporarily unavailable; without backoff, origin 5xx errors would trigger a flood of requests that breaches subrequest limits.
The combined layers achieved roughly 99% cache hit rate. Querying from cache in the Worker is an order of magnitude faster than fetching from origin servers in different regions. Real-time workloads mean the rate will never reach 100%, since fresh data must be requested at least every minute.
Real-time metrics with Thanos
Maintenance coordination requires reacting in real time to network degradation and machine failures. The system uses Thanos, the distributed Prometheus query engine, to deliver edge metrics into the coordinator.
To analyze edge router health, the system sends targeted queries. The original approach asked Thanos for each edge router's current health status and filtered for relevant routers inside the Worker. This returned multi-MB responses requiring decode and encode cycles; the Worker then cached and parsed these large payloads only to discard most data while processing one maintenance request. Since TypeScript is single-threaded and JSON parsing is CPU-bound, two large HTTP requests would block each other during parsing.
The graph approach instead queries targeted relationships like interface links between edge and spine routers via EDGE_ROUTER_NETWORK_CONNECTS_TO_SPINE. Average response size dropped from multiple MB to ~1 KB — roughly 1000x smaller. This also reduced CPU usage inside the Worker by offloading deserialization to Thanos. While this creates more small fetch requests, load balancers spread them evenly across Thanos instances, increasing throughput.
Historical analysis challenges
The graph implementation and fetch pipeline handle real-time request herds well. Historical analysis presents a different I/O challenge: scanning months of data to find conflicting maintenance windows. Previously, Thanos issued massive amounts of random reads to R2, Cloudflare's object store, creating a significant bandwidth penalty. A new approach developed internally by the Observability team addresses this without losing performance — details that complete the picture of how the maintenance scheduler handles both live coordination and long-range planning workloads.
Replaying the past against the network graph
With so many maintenance operations in flight at any given time, historical data is the only reliable way to validate that our scheduling logic is both accurate and ready for the network’s growth. The goal is twofold: avoid creating incidents, and avoid blocking legitimate physical maintenance. To walk that line, we run scheduled maintenance events from two months—or even a year—ago through the same constraint checks we use today, such as edge router availability or Aegis protections, and measure how often those historical events would have violated a constraint.
This approach echoes the work we described earlier this year on using Thanos to automatically release and revert software across the edge.
Thanos fans queries out to Prometheus, but when the data is older than the Prometheus retention window, it has to pull the blocks from object storage—R2 in our case. Prometheus TSDB blocks were built for local SSDs and assume random access patterns; those same patterns become a serious bottleneck over object storage. When our scheduler needs to look at months of maintenance history to find conflicting constraints, random reads from object storage create a massive I/O penalty.
To get around that, we built a conversion layer that rewrites the TSDB blocks as Apache Parquet files. Parquet is a columnar format designed for big data analytics: it stores data by column rather than by row, and it carries rich statistics that let us fetch only the data a query actually needs. Because we are the ones doing the rewrite, we can also lay out the files for a few large sequential reads instead of many small random ones.
sum by (instance) (hmd:release_scopes:enabled{dc_id="45"})In the example above, we select the tuple (__name__, dc_id) as the primary sort key. That keeps metrics like hmd:release_scopes:enabled adjacent when they share the same dc_id value.
The Parquet gateway then issues precise R2 range requests, pulling only the columns relevant to the query. That takes the payload from megabytes down to kilobytes. Better still, these file segments are immutable, so we can cache them aggressively on the Cloudflare CDN.
This setup effectively turns R2 into a low-latency query engine. We can now backtest complex maintenance scenarios against long-term trends instantly, without the timeouts and tail latency that came with reading native TSDB blocks. A recent load test showed the Parquet path delivering up to 15x the P90 performance of the old system on the same query pattern.
For a deeper look at the Parquet implementation, see the talk Beyond TSDB: Unlocking Prometheus with Parquet for Modern Scale from PromCon EU 2025.
Keeping pace with the network
Moving the scheduler onto Cloudflare Workers took us from a system that ran out of memory under load to one that handles data caching intelligently and uses efficient observability tooling to analyze infrastructure and product data in real time. The maintenance scheduler now balances network growth against the constraints of keeping customer traffic flowing.
Balance, though, is a moving target.
Hardware is added around the world every day, and the maintenance logic required to keep it running without disrupting traffic grows exponentially more complex as both the number of products and the variety of maintenance operations expand. The first set of challenges is behind us, but the problems ahead are more subtle—the kind that only surface once a system operates at this scale.
Those are the problems we are looking for help with. If that sounds like the kind of work you want to do, take a look at open roles on our Infrastructure team.
