How Cloudflare ships code globally without breaking the internet
When Cloudflare returns an 
The result is Health Mediated Deployments (HMD), a data-driven system that automates software rollouts across Cloudflare's global network. HMD queries Thanos, which stores and scales Prometheus metrics collected from Cloudflare services. Based on those metrics, HMD decides whether new code should continue rolling out, pause for evaluation, or be automatically reverted to limit damage.
Engineers configure signals from their services, such as alerting rules or Service Level Objectives (SLOs). A typical Service Level Indicator (SLI) might check the rate of HTTP 500 errors over a 10-minute window:
sum(rate(http_request_count{code="500"}[10m])) / sum(rate(http_request_count[10m]))An SLO pairs an SLI with an objective threshold—for instance, that a service returns 500 errors less than 0.1% of the time. If the success rate drops unexpectedly where new code is running, HMD reverts the change before humans even know which service broke:
Testing the safety net
Cloudflare's network handles millions of requests per second across diverse geographies, so HMD must react quickly when a buggy release slips through. To verify this, HMD runs backtests outside the release process, using historical incident data to measure how fast it would detect degrading signals in a future release.
Thanos joins thousands of small Prometheus deployments into a single query layer. For historical metric data beyond Prometheus's retention period, Cloudflare backfills from its R2 object storage. Today that means 4.5 billion distinct time series, roughly 8 petabytes of data in 17 million objects distributed globally.
Making it work at scale
A batch of backtests is expensive. Each run evaluates multiple SLOs per service, each SLO spans multiple queries across batches of data centers, and each data center issues tens to thousands of requests to R2. In aggregate, a batch can translate to hundreds of thousands of PromQL queries and millions of R2 requests. Initially, batch runs took about 30 hours; optimization has cut that to around 2 hours.
Recording rules
HMD slices the fleet of machines by dimensions referred to as "tier" and "color." To find machines matching a given pair, HMD originally ran a PromQL expression like this:
group by (instance, datacenter, tier, color) (
up{job="node_exporter"}
* on (datacenter) group_left(tier) datacenter_metadata{tier="tier3"}
* on (instance) group_left(color) server_metadata{color="green"}
unless on (instance) (machine_in_maintenance == 1)
unless on (datacenter) (datacenter_disabled == 1)
)Most of these series have cardinality roughly equal to the number of machines in the fleet—a substantial amount of data to fetch from object storage, transmit for query evaluation, and decode and join. Since this query runs in every HMD batch, Cloudflare precomputes it with Prometheus recording rules:
hmd:release_scopes:info{tier="tier3", color="green"}Beyond cleaner syntax, this reduces query-time load significantly. Because all joins can only match within a data center, the rules can be evaluated directly inside each data center's Prometheus instances. Cardinality now scales with the release scope instead of the whole fleet, which is cheaper and less vulnerable to network issues that would otherwise force retries.
Distributed query processing
HMD and the Thanos Querier are stateless components running in highly available deployments in North America and Europe. When HMD sends an SLI expression to the Thanos Querier:
sum(rate(http_request_count{code="500"}[10m]))
/
sum(rate(http_request_count[10m]))The querier requests raw time series data for the http_requests_total metric from connected Thanos Sidecar and Thanos Store instances worldwide, waits for all data to arrive, decompresses it, and computes the result:
This centralized approach has downsides: raw data from thousands of sources must reach one location before processing can begin, and a single instance handles all of it. Doubling the number of data centers means doubling the memory needed for query evaluation.
Many SLIs are simple aggregations that boil down service health to a number, like an error percentage. These aggregations are often distributive—they can be computed inside each data center and then coalesced. A per-data-center recording rule would allow queries like:
sum(datacenter:http_request_count:rate10m{code="500"})
/
sum(datacenter:http_request_count:rate10m)Requesting pre-aggregated results instead of raw high-cardinality time series cuts network transfer and processing by roughly an order of magnitude. But recording rules carry a steep write-time cost across thousands of production Prometheus instances, and scaling them alongside a growing set of SLIs would be unsustainable.
What Cloudflare needed was runtime evaluation of data center-scoped queries with coalesced results, for arbitrary queries:
(sum(rate(http_requests_total{status="500", datacenter="dc1"}[10m])) + ...)
/
(sum(rate(http_requests_total{datacenter="dc1"}[10m])) + ...)Thanos's distributed query engine does exactly that. Instead of requesting raw time series, it requests data center-scoped aggregates that are sent back and coalesced into the full query result:
To keep expensive data paths short, Cloudflare uses R2 location hints to specify primary access regions.
Measurements used Cloudprober probes evaluating the relatively cheap but global query count(node_uname_info). The distributed execution deployment responded 3–5 times faster on average than the centralized one:
More complex queries sometimes time out or crash the centralized deployment but are comfortably handled by the distributed one. For count(up) across about 17 million scrape jobs, the centralized querier had to be scoped to a single region and still took about 42 seconds:
The distributed queriers returned the full result in about 8 seconds.
Congestion control
HMD batch processing creates spiky load patterns that are hard to provision for. Batch queries also have lower priority than queries from on-call engineers triaging production issues. Cloudflare addressed both with an adaptive, priority-based concurrency control mechanism, inspired by Netflix's work on adaptive concurrency limits. A proxy dynamically limits batch request flow when Thanos SLOs degrade—for example, the cloudprober failure rate over the last minute:
sum(thanos_cloudprober_fail:rate1m)
/
(sum(thanos_cloudprober_success:rate1m) + sum(thanos_cloudprober_fail:rate1m))Jitter, a random delay, smooths query spikes. Batch processing prioritizes overall throughput over individual query latency, so jitter lets HMD send bursts while Thanos processes queries gradually over minutes. This reduces instantaneous load and improves overall throughput, even at the cost of higher individual query latency. HMD encounters fewer errors, minimizes retries, and boosts batch efficiency.
The solution mimics TCP's additive increase/multiplicative decrease congestion control. When the proxy receives a successful response from Thanos, it allows one more concurrent request. When backpressure signals breach thresholds, the proxy limits the congestion window proportionally to the failure rate:
As failure rate rises past the "warn" threshold toward "emergency," the proxy exponentially approaches allowing zero additional requests. A configured minimum request rate caps the loss so bad signals can't halt all traffic.
Columnar experiments
Prometheus TSDB blocks were never designed for reading over slow object storage, so Thanos does significant random I/O. Inspired by a talk on columnar formats, Cloudflare began storing time series data in Parquet files with promising preliminary results. The project is too early for firm conclusions, but the experimental object storage gateway is published as parquet-tsdb-poc on GitHub for the Prometheus community.
What's next
HMD has enabled safe, reliable software releases while pushing Cloudflare's observability infrastructure. Thanos batch runtimes are down 15x, with distributed query processing providing 3–5x faster response times on global probes. The company continues working with its observability, resiliency, and R2 teams, with an eye toward optimizing time series storage for object storage. The Parquet-based proof of concept is available for anyone exploring large-scale observability.
