More than 20% of the web sits behind Cloudflare's network, and the Data team owns the pipeline that moves that data — and more — from a globally distributed fleet of tens of thousands of servers into the applicable central data centers. Cloudflare Logs, the SQL API, observability, and internal analytics all depend on it. In May of this year the pipeline was delivering roughly two billion events per second, customer traffic plus internal feeds.
The recurring problem is exponential traffic growth against linear hardware growth, so the team must keep reacting to hold resource costs down. Across five years, the accumulated changes are equivalent to deferring two and a half years of hardware expansion: breaking up a monolithic data schema in 2022, changing compression in 2024, rearchitecting pipeline replication in spring 2026, decentralizing sampling in summer 2026, and introducing aggregations in fall 2026.
Pipeline shape, and blowing up the single schema
Collection starts at an internal service called Logfwdr, which gathers data from highly distributed sources such as Workers and FL. Logfwdr forwards to a set of centralized receivers, which push onto the buffering system Buftee. Consumers fetch from Buftee and distribute to final destinations such as ClickHouse.
The 2022 change dismantled the one and only schema. A single stream had carried every product's fields, forcing internal consumers to ingest everything and then demux or filter out what they didn't need. At the time most data products were built on HTTP, Security and CDN metadata. The 2021 message schema looked like this:
Some fields belonged to the same product; others were unrelated to each other. The hypothesis was that no team needs 100% of the fields. Product-specific messages can carry only what each product requires. Duplicating some fields into every message still replaced the firehose with something cheaper: backbone bandwidth dropped 35–40%, roughly a year's worth of compressed traffic growth.
Compression: LZ4 to zstd
By 2024, traffic growth had made network congestion a daily problem, so the pipeline moved from LZ4 to zstd. zstd had been tried before and showed higher CPU usage at the time, so the change was rolled out carefully with metrics watching the trade-off. A few servers went first, and CPU usage showed no noticeable difference.
The fleet-wide rollout followed with little CPU impact. Compressed traffic throughput fell by nearly 40%. Uncompressed bytes kept growing at the same rate, meaning the same size pipe could carry more than 1.5x the traffic — or, measured as deferred growth, about 10 months of headroom.
In 2026 the focus shifted to reducing uncompressed bytes, and therefore compressed bytes, through three strategies.
Load balancing: send once, replicate late
Analytics needs redundant copies in multiple centralized regions to survive regional disasters; data under Customer Metadata Boundaries stays strictly within its designated region. The historical answer was double-writing: Logfwdr transmitted two full copies of every non-regional analytics event, one per regional path, and each region then independently processed a complete copy through to ClickHouse. Sampling was applied in each region, downsampling high-volume traffic while preserving rare events.
That was manageable while only a few streams used secondary regions. As disaster resilience spread to almost all analytics streams, Logfwdr traffic doubled across the board, and sending every event twice over wide-area transit became unsustainable against exponential growth.
Logpush had already solved a version of this: its delivery is stateless, since customer destinations don't care which centralized data center pushes the logs, so Logfwdr treats the central data centers as one shared pool of receivers, with each message sent to a single region and failing over to another destination if a machine or facility is unreachable. That model had already shifted traffic seamlessly during large incidents.
Analytics was harder. ClickHouse clusters in both regions still need a full copy of non-regional data for instant, independent query failover. The fix was to decouple pipeline transit from database redundancy:
- Shared ingest pool — Logfwdr dispatches each event exactly once across wide-area transit, to whichever centralized receiver is available.
- Late replication — replication happens as late as possible: data lands in Buftee and remote inserters stream cross-data-center copies downstream.
- Failure handling — if a centralized region goes down, Logfwdrs route all non-regional traffic to the surviving region; if the inter-data center link blips, each cluster keeps ingesting its local share without backpressure while remote inserters catch up from Buftee's retention buffer once the backbone recovers.
For the migrated datasets, the cutover removed approximately 43 GB/s of compressed transit traffic while adding only ~7 GB/s of cross-data-center replication — a net reduction of 36 GB/s across wide-area networks. Two factors explain the asymmetry. First, replication now happens downstream of sampling: previously two copies of raw, unsampled events crossed wide-area transit, whereas now Logfwdr sends a single copy, the receiving region samples immediately before buffering into Buftee, and only the compacted stream is carried. Second, centralization improves compression: source data arrives fragmented across thousands of servers, but once centralized it is aggregated and partitioned into uniform batches over wider time windows, and that structural uniformity lets compression algorithms reach far higher ratios than Logfwdr transit batches ever could.
The first two datasets were HTTP analytics and Ready-Analytics, the in-house one-click analytics product.
Migrating further datasets to this architecture has continued through the spring.
Sampling earlier in the pipeline
Sampling already ran in centralized services, particularly the receivers, easing pressure downstream. Pushing it further upstream — into Logfwdr, while keeping the receiver-side sampling — means less data leaves the origin without hurting data quality.
Each dataset has a key chosen by its producer team for indexing and sampling. Records for each key are shuffled to eliminate bias, then, per group, a sample size equal to the square root of the total count is selected, which yields a representative sample whether it is sampled once or twice. Estimates scale back against the sample rate: a 10-of-100 sample has a rate of 10/100, so the unbiased estimate of the original count is 10*10 = 100.
Validation compared rows written to ClickHouse against the estimated total count across many keys and traffic footprints, covering the spectrum from "least seen" to "most seen" keys.
Keys with more data end up with a lower sample rate, while low-volume keys are sampled comparably, so the sample stays representative while fewer events are delivered and stored.
In power-law terms, the double-sampled curve (red) grows at roughly x0.319 versus x0.485 for sampled-once/control (blue). A key with 100x more data than another would have stored roughly 10x the rows under single sampling; it now stores roughly 4x. Concretely, 11.6 million events are represented by 18k rows in ClickHouse instead of 79k. Double sampling attenuates storage for "most seen" keys while keeping the same detail per event for "least seen" keys.
The strategy was first applied to three of the top datasets. Each at least halved the compressed data sent to centralized receivers — 25 GB/s altogether, or over 2 PB/day.
Pushdown aggregation
Sampling answers any question approximately. Aggregation takes the opposite trade: a fixed set of questions answered exactly, which requires choosing dimensions and metrics up front. In the HTTP dataset, for instance, Cloudflare records the number of unique IP addresses seen in a given minute rather than the addresses themselves, and sums requests for a given domain per minute grouped by HTTP method. None of those computations require waiting for the tail end of the pipeline.
That work used to happen late, inside the inserters reading from Buftee and writing into ClickHouse. Moving it earlier spreads load and shrinks what has to be centralized. The aggregation functionality is now applied at every step, combining matching partial aggregates as data converges. Records are grouped by a key that determines which records can be combined, and aggregation happens on each Logfwdr instance, again when partially aggregated events reach the centralized receivers, again after the dedicated partitioned buffers feed batches to the ClickHouse inserters, and finally in ClickHouse via aggregating table engines.
HTTP traffic was the first dataset migrated. It fell from 9+ GB/s to <2 GB/s in compressed bytes on the wire — roughly an 80% reduction even though it remains the highest-cardinality dataset.
Aggregation suits datasets that collapse easily, and it matters most where sums and counts must be exact, such as usage metrics. It also gives producer teams a second lever besides sampling, packaged as a generic one-click product called Ready-Aggregates, mirroring Ready-Analytics.
Where the savings land
Traffic growth consumes headroom faster than anything else. Ready-Analytics had load balancing and sampling applied. In mid-May, load balancing took it from ~26 GB/s to ~13 GB/s, a 50% cut. Two months later it was already climbing back toward ~20 GB/s — about half the reduction erased by organic growth. Sampling then brought the dataset's throughput to single digits at ~8 GB/s.
Without both interventions, Ready-Analytics alone would have passed 60 GB/s by September. Processing less data in centralized data centers also returns CPU and memory: the graphs below show load balancing and sampling in Logfwdr cutting CPU and memory on the Ready-Analytics receivers by ~50%, recovering ~750 CPUs and 3.5 TiB of memory.
Across the whole pipeline from spring to fall, the three mechanisms removed ~70 GB/s of compressed traffic from the backbone: ~38 GB/s from load balancing, ~25 GB/s from sampling across the top datasets, and ~7+ GB/s so far from aggregation.
Over a five-year horizon, this year's changes matter as much as the immediate relief. The projected volume five years out is now half what it would have been. Structural and algorithmic work of this kind is what keeps hardware demand sustainable.
