Analytics at the Right Resolution

Cloudflare’s analytics products answer questions about traffic by parsing the enormous volume of events — HTTP requests, Workers invocations, Spectrum connections — logged every day. But the scale of that data varies wildly between customers and over time. A query that touches a few hundred records for one site might need to scan billions for another; a report that loads a few rows in normal conditions might require millions during an attack.

This disparity mirrors a pattern found across many domains. The useful answer to “how many HTTP requests hit my site last week?” doesn’t require an exact count — a figure correct to a few significant digits is plenty. The same holds for measurements like the world population or the length of Great Britain’s coastline: precision beyond a certain point adds little value. Cloudflare traffic follows a Pareto distribution, so what constitutes “good enough” accuracy shifts with the query and the moment.

That observation drove the design of ABR (Adaptive Bit Rate) for Cloudflare analytics. The name borrows from video streaming, where the server selects a resolution matching the client’s connection. Here, the system chooses a data resolution appropriate to the query: a request for firewall events by country over the past week can be answered from a lower-resolution sample than one covering the last hour. The answer is the same, but the lower-resolution path consumes less time and fewer resources.

Under the hood, analytics data lives in ClickHouse, Cloudflare’s columnar store. ABR writes the same data into separate tables at multiple resolutions, spanning seven orders of magnitude — from 100% down to 0.0001% of original events. This costs an additional 12% in disk storage, but enables fast ad hoc queries against the reduced datasets.

Sampling vs. Rollups

The usual alternative to this approach is a rollup system, which pre-computes partial or complete aggregations as data arrives. Rollups conserve space but carry hidden costs. Consider counting the world’s lentil production — 6.3M tonnes per year. A distributed system would count each lentil across hundreds of machines, tabulating dozens of attributes broken down by color, origin, distributor, and time slices, producing millions of aggregations per minute. The computation is expensive, and information not explicitly aggregated is lost. Moreover, much of that work is wasted: the target figure has only two significant digits, which a sample could provide at a fraction of the cost.

ABR preserves the original events, so every attribute remains queryable. It also allows cross-checking: because multiple resolutions of the same data exist independently, one can be validated against another, and against the full-resolution table, to confirm the system is working correctly.

Beyond Simple Counts

Counting and summing aren’t the only aggregations ABR supports. Two more complex operations — topK and count-distinct — are handled differently under sampling.

topK returns the K most frequent items in a set, such as the most common IP addresses or countries. Computing it entails counting the frequency of each element and returning the top entries. Under ABR, this is done on the sample for the matching resolution, which speeds the calculation considerably. There is a known bias: the estimate can overstate the significance of elements relative to their true frequency in the full set. In practice, this only becomes noticeable at very high cardinalities, and Cloudflare dashboards don’t exhibit visible differences across resolutions for typical metrics like top URLs or browser types. For large sets, proportion estimates remain quite accurate.

count-distinct — the number of unique values, like distinct cache keys — cannot be derived from a uniform random sample of events. The solution is an alternate sampling strategy based on the value in question: instead of sampling requests randomly, sample IP addresses randomly. This distinct reservoir sampling approach estimates the true number of distinct IPs from the cardinality of the sampled set. Further refinements to improve these estimates are planned.

Resilience as a Side Effect

The most significant payoff of ABR isn’t visible on the dashboard. Large distributed systems suffer from tail latency: any single slow component can delay a dependent query. In Cloudflare’s data pipeline, servers may falter, networks may degrade, or background load may spike. Before ABR, such conditions could prevent a dashboard from rendering. Now the system adapts, delivering the best results available at that moment without noticeable degradation, even during attacks or simulated cluster failures.

ABR will underpin several upcoming dashboards and analytics products. The goal is straightforward: analytical answers that are both useful and reliably fast, whatever the load.