Signal vs. Noise at Cloudflare Scale

There is a common belief that because Cloudflare handles a massive share of global Internet traffic — and publishes much of what it sees via Cloudflare Radar — the company must have all the answers. The reality is more nuanced. Volume does not automatically translate to clarity. More data means more noise, and it means extra work to eliminate alternative explanations before a pattern can be trusted.

That is the challenge that Ram Sundara Raman, now an assistant professor at University of California, Santa Cruz, took on during his 2022 internship at Cloudflare, while pursuing his PhD. He has returned to share what he learned about working with measurement data at this scale.

Ram’s project demonstrates how research perspectives from a university lab (the Breakerspace lab at UMD) can be adapted into production-grade tooling at Cloudflare — here, specifically, to detect and explain connection failures for customers. This is one of the reasons Cloudflare’s goal to hire 1,111 interns in 2026 is more than a headcount figure; that cohort, equivalent to about 25% of the full-time workforce, represents ample opportunity to ship working code into production and to observe aspects of the Internet that are otherwise hard to see. Fielding ideas from a group that large is a challenge all its own.

For prospective interns thinking about data- or measurement-oriented projects, a useful starting question is whether the idea would actually matter to Cloudflare — not just whether it is technically interesting. But that is up front. Here is what Ram has to say about the practice of measurement itself.

Passive Detection at Scale: Lessons from Cloudflare’s Data

Network tampering research has traditionally relied on active measurements: crafting specific probes and sending them to detect interference. This approach, used in projects like Censored Planet, successfully identifies connection tampering in over 200 countries. However, active measurements are limited by resources — Censored Planet can only evaluate the 2,000 most popular websites due to time and cost constraints.

Cloudflare’s infrastructure seems like an obvious solution: a global view of real traffic across nearly every network. But as research published at ACM SIGCOMM’23 demonstrates, using passive observation at this scale presents its own set of formidable challenges.

From Active to Passive Observation

The goal was straightforward: detect middlebox interference — packet drops or RST injections that abort TCP connections — using only data that naturally flows to Cloudflare. No custom probes, no controlled experiments, no mixing in external datasets. The constraints were strict: everything Cloudflare does must be scalable and sustainable.

The limitations became apparent quickly. As a passive observer, you lose the ability to choose where to look. The data spans continents, millions of users, and thousands of ways connections can be made and broken. Identifying tampering from this noise requires working with data full of obstacles and pitfalls.

Challenge One: Scale

At the time of the internship, Cloudflare was serving more than 45 million HTTP requests per second across 285 data centers. TCP connections to the 1.1.1.1 DNS server added another data stream, and Network Error Logging (NEL) data was also considered. Each dataset had its own sampling methods, making combination impossible. NEL was ultimately rejected because it is biased — only some clients support it and only some websites enable it.

To manage the volume, researchers built special IPTABLES rules to log incoming TCP connections across all Cloudflare points of presence. The data was sampled uniformly at one in every 10,000 connections, logging only the first 10 inbound packets per connection. This constraint meant certain infrequent tampering types, or interference occurring after the first 10 packets, would remain undetected.

Challenge Two: Noisy Data

Even with sampling in place, the data was remarkably noisy. Event records had millisecond time resolution, but multiple packets could arrive within a single millisecond, making packet ordering unreliable. Denial-of-service attack traffic and port scans could resemble tampering events. Even well-intentioned Internet standards like Happy Eyeballs introduced quirks that confused detection.

Researchers developed 19 tampering signatures — distinctive packet patterns indicating middlebox interference — but spent considerable effort analyzing noise sources and iterating on these signatures. Events were accepted as tampering only when supported by corroborating evidence, such as inconsistent changes in the Time-To-Live (TTL) field in the IP header.

Challenge Three: No Ground Truth

Without active experiments, confirming that a detected event was actually tampering — rather than one of the thousand other phenomena on the Internet — was extremely difficult. Researchers relied on prior work in the censorship measurement space to recognize known signals and patterns, helping confirm many cases of tampering.

The exercise revealed surprising limitations for a company handling a double-digit percentage of Internet traffic:

  • Cloudflare could see which connections were affected and where they originated, but not who performed the tampering.
  • Sometimes it was possible to determine which domains were blocked, but not always — packets could be dropped before reaching Cloudflare.
  • As a passive observer, researchers could only see affected user activity, not what could have been affected.

Finding Meaning in Imperfect Data

Despite these constraints, the research succeeded. Using the 19 tampering signatures, distinctive behaviors were identified across hundreds of networks, including tracking large increases in tampering rates during periods of nationwide protests in Iran in late 2022. The data from this project is now continuously published on Cloudflare Radar.

The key conclusion: large infrastructure providers are not omniscient. A global view is powerful but does not automatically translate into easy observations. With imperfect data, researchers must be creative — finding patterns in noise and designing methods that work despite what is missing. The protocols are tightly specified, so patterns can be enumerated, but they still require careful manual analysis.

Future work will likely require combining passive and active probing, using provider scale alongside targeted measurements to capture the full picture of Internet tampering. The question of how to automatically identify tampering as attacks happen or networks change remains open, and collaborations between academia and industry will be essential to addressing it.