Handling 706M Events Per Second Without Losing the Plot
Cloudflare’s data pipeline now ingests up to 706 million events per second (as of December 2024), a 100x increase since our 2018 pipeline write-up. At peak, that translates to moving 107 GiB/s of compressed data. These streams feed Logs, Analytics, and billing, plus machine-learning models for bot detection. Any point in that pipeline can fail — hardware faults, misconfiguration, overload — and, at this scale, you cannot buffer your way out of a stall. Data will be dropped. The engineering question is how to drop it deliberately, keep the information content useful, and still provide trustworthy answers.
The pipeline stages look like this:
Controlled data loss via downsampling
Uncontrolled loss is like receiving only random chunks of a high-resolution image: you get gaps and no context. Downsampling is the opposite — it trades resolution for completeness, delivering a blurry but coherent view of the whole picture.
Since failures can occur at any stage, downsampling must be available everywhere. Some services downsample at the source before data reaches the first forwarding component (Logfwdr). That introduces noise you cannot control, but you can estimate it robustly using statistics on the sampled data.
Logfwdr itself can downsample when its buffer overflows. Because it handles many streams at once, it assigns each a weight and applies max-min fairness: healthy streams using little buffer get what they need; lagging streams split the remaining memory fairly when the buffer saturates. The Go implementation runs each stream as a goroutine, communicating via channels and consulting a single tracker object on every memory allocation and deallocation. The tracker uses a max-heap to identify the heaviest user and sends a “shed some load” signal until total usage is back under the limit. Downsampling picks the least-downsampled batches first (via a min-heap) and merges them, keeping batch sizes roughly constant.
Downsampling is cheap, but it forces recompression of compressed buffer data — the most expensive operation in the pipeline. To avoid burning CPU exactly when the system is overloaded, streams in “shed load” state also downsample fresh, uncompressed data before it is compressed the first time.
This is a “bottomless buffer”: you can keep pushing data in and it is automatically thinned. It resembles reservoir sampling, but with two differences — the input never ends, and the output sample also never ends.
Adaptive sampling at the queue and the API
The next stage, Logreceiver, fronts a distributed queue. It partitions each stream by a key that downstream consumers such as Logpush and analytics inserters can use. Logreceiver also performs adaptive sampling: small customers (around 10 events/day) are effectively unsampled, while large customers (millions of events/sec) are sampled more aggressively. The same data is pushed at multiple resolutions — 100%, 10%, 1% — into different queue topics. If the queue is overloaded, Logreceiver skips writing high-resolution samples rather than nothing at all.
The same idea cascades downstream. Inserters can skip reading or writing high-resolution data. Analytics APIs can skip reading it. And the analytical database can fall back to lower resolutions under overload, degraded cluster state, or very large read ranges. The result is that the API returns some answer in every case.
Making sense of the thinned data
Sampled data lacks rows. To make it useful, every event x_i carries its sample interval — the reciprocal of its inclusion probability π_i. For example, with 1-in-1,000 sampling, each included event has w_i = 1/π_i = 1,000. When batches are downsampled again, the intervals multiply. This weight also approximates how many original events a sampled event represents.
The Horvitz-Thompson (HT) estimator provides two outputs: an estimate of a population total (SUM) and an estimate of that estimate’s variance. The variance lets you build confidence intervals — ranges covering the true value with a chosen confidence level (typically 0.95). For SUM, the HT estimator is:
sum(x_i / π_i) = sum(x_i * w_i)
Its variance depends on π_ij, the probability that events i and j are both sampled. We use Poisson sampling, where each event is an independent Bernoulli trial, so π_ij = π_i * π_j, which collapses the variance estimator to:
sum(x_i² * w_i * (w_i − 1))
COUNT uses the same estimator with x_i = 1, giving:
sum(w_i) and variance sum(w_i * (w_i − 1)).
AVG is trickier. The natural formula uses the population size N, which is not stored and cannot be known at query time because of dynamic filtering. Substituting the COUNT estimate works for the mean itself but not for its variance — confidence intervals become unusable. The workaround is simultaneous confidence intervals: construct intervals for SUM and COUNT independently at a stricter level (half the “inconfidence”), then combine them via the Bonferroni method.
In SQL, the estimators look like this:
WITH sum(x * _sample_interval) AS t,
sum(x * x * _sample_interval * (_sample_interval - 1)) AS vt,
sum(_sample_interval) AS c,
sum(_sample_interval * (_sample_interval - 1)) AS vc,
-- ClickHouse does not expose the erf⁻¹ function, so we precompute some magic numbers,
-- (only for 95% confidence, will be different otherwise):
-- 1.959963984540054 = Φ⁻¹((1+0.950)/2) = √2 * erf⁻¹(0.950)
-- 2.241402727604945 = Φ⁻¹((1+0.975)/2) = √2 * erf⁻¹(0.975)
1.959963984540054 * sqrt(vt) AS err950_t,
1.959963984540054 * sqrt(vc) AS err950_c,
2.241402727604945 * sqrt(vt) AS err975_t,
2.241402727604945 * sqrt(vc) AS err975_c
SELECT t - err950_t AS lo_total,
t AS est_total,
t + err950_t AS hi_total,
c - err950_c AS lo_count,
c AS est_count,
c + err950_c AS hi_count,
(t - err975_t) / (c + err975_c) AS lo_average,
t / c AS est_average,
(t + err975_t) / (c - err975_c) AS hi_average
FROM ...Applied to each timeslot in a time series, these intervals form a confidence band around the estimate.
A sampling bug that broke the math
Confidence bands on internal dashboards eventually revealed a systematic error. For one site, the “total bytes served” estimate was consistently higher than the true value from rollups, and the band never contained the truth.
Stored data was clean; the SQL math checked out. The root cause was in Logreceiver’s sampling method. Instead of independent random Bernoulli trials, it performed systematic sampling — picking events at equal intervals starting from the first event in a batch.
Systematic sampling breaks two assumptions. It invalidates π_ij = π_i * π_j, so the simplified variance estimator is wrong. Worse, the total estimate itself becomes biased. A Python repro confirmed it:
import itertools
def take_every(src, period):
for i, x in enumerate(src):
if i % period == 0:
yield x
pattern = [10, 1, 1, 1, 1, 1]
sample_interval = 10 # bad if it has common factors with len(pattern)
true_mean = sum(pattern) / len(pattern)
orig = itertools.cycle(pattern)
sample_size = 10000
sample = itertools.islice(take_every(orig, sample_interval), sample_size)
sample_mean = sum(sample) / sample_size
print(f"{true_mean=} {sample_mean=}")The culprit pattern: a person loading a large HTML page triggers one big response followed by a burst of small cached resources. For low-traffic sites, those responses cluster at the start of a batch. Logreceiver concatenates but never cuts batches, so the first event stayed first — and was always selected, skewing the estimate upward.
Raw unsampled data from a Logs product confirmed the hypothesis: the first response in a time-grouped batch is disproportionately larger than the rest.
The fix was simple — shuffle data before sampling. After rollout, estimates converged to the true values.
Sampled data in the product APIs
Most analytics datasets now run on sampled data. The Workers Analytics Engine exposes the sample interval in SQL so customers can build their own confidence bands. In the GraphQL API, nodes with “Adaptive” in the name use sampled data, and the sample interval is exposed as a field — though not enough to build intervals. As an experiment, confidence(level: X) is now available on count and edgeResponseBytes (sum) for httpRequestsAdaptiveGroups nodes.
A sample query:
query HTTPRequestsWithConfidence(
$accountTag: string
$zoneTag: string
$datetimeStart: string
$datetimeEnd: string
) {
viewer {
zones(filter: { zoneTag: $zoneTag }) {
httpRequestsAdaptiveGroups(
filter: {
datetime_geq: $datetimeStart
datetime_leq: $datetimeEnd
}
limit: 100
) {
confidence(level: 0.95) {
level
count {
estimate
lower
upper
sampleSize
}
sum {
edgeResponseBytes {
estimate
lower
upper
sampleSize
}
}
}
}
}
}
Returns estimates, confidence intervals, and the sample size:
{
"data": {
"viewer": {
"zones": [
{
"httpRequestsAdaptiveGroups": [
{
"confidence": {
"level": 0.95,
"count": {
"estimate": 96947,
"lower": "96874.24",
"upper": "97019.76",
"sampleSize": 96294
},
"sum": {
"edgeResponseBytes": {
"estimate": 495797559,
"lower": "495262898.54",
"upper": "496332219.46",
"sampleSize": 96294
}
}
}
}
]
}
]
}
},
"errors": null
}
In the example response, the estimated count is 96,947, and the 95% confidence interval spans 96,874.24 to 97,019.76. The estimate is based on a sample of 96,294 rows — enough for the central limit theorem to apply.
The pipeline has doubled roughly every 1.5 years, and downsampling with proper statistical estimation has kept it scalable and resilient. It also demonstrated how easily sampling can silently corrupt results — and how careful validation against ground truth catches what code review misses.
