Why the Internet resists being measured
The Internet is a system built from independently owned and operated networks and services that rarely share or report on how their systems behave. Each forwards what others produce while remaining entirely autonomous — which is precisely what makes the Internet work, and also what makes it opaque. Measurement must therefore exist as a scientific practice, with rigor, repeatability and reproduction, in a context that is opaque yet critical.
The limits of measurement are easy to illustrate. Consider the claim that 5 out of 6 scientists say that Russian Roulette is safe. It is absurd but also logical, and an experiment producing it is trivially easy to design — but only if the actor ignores everything that makes measurement credible:
- Methodology: the cycle of data curation, modeling and validation. Here the experiment could only "succeed" if participants never saw each other's injuries, and no measurement is needed at all since the probabilities are calculable from available numbers.
- Ethics: how we measure can have undue, undesirable consequences; the bare minimum is do no harm.
- Representation: statements and visualizations must be at least informative and ideally actionable, or they mislead. A participant answering yes to "are you safe?" is answering a different question than "is the game safe?"
A cautionary time series from Lviv
Observation without context is perilous. In the internal HTTP request graph for Lviv, Ukraine, leading up to the evening of 28 February 2022, traffic from the region rose 3-4X on that day:

Four days earlier, the Russian incursion into Ukraine had begun. The spike could have been mistakenly reported as a potential DoS attack, but counter-indications existed: no DoS defense or mitigation system flagged an attack, and the profile was atypical. Attack traffic tends to be either a single source in a single location or multiple sources across multiple locations; here the increase came from multiple source networks in one location. Cloudflare had the tools to avoid erroneous reporting and correctly attributed the rise to people converging in Lviv, the city with the last train station on the westward journey out of Ukraine. Yet nothing visible from Cloudflare's perspective could explain it — an employee saw a BBC report on the mass movement of people, which made sense of the shift.
The lesson: always look for alternative explanations. Observations alone can lead to wrong conclusions through missing information or unrecognized biases — and good numbers without bias can be misunderstood too.
Vocabulary: how and what we measure
Active versus passive
Active measurement means the actor initiates an action designed to trigger a response — data such as latency from a ping or a DNS answer to a query, or an observable change in a mechanism, e.g. crafted probe packets that prompt reactions from and expose middleboxes. Passive measurement means the actor only observes: no action, no triggered response, no alteration of the system. Logs are typically compiled from passive observations, and the vast majority of data in Cloudflare Radar derives from them.
The trade-offs differ. Active measurements are targeted and controllable but exceptionally difficult and often costly to scale, and can only observe the parts of a system where they are deployed. Passive measurements are lighter weight but succeed only if the observer is in the right place at the right time. The two complement each other and are most powerful when orchestrated together. Prior work on CDN performance, for example, interrogated passive request logs for insight, then used active pings via RIPE's Atlas to confirm the results; conversely, efforts to passively detect and understand connection failures were informed by — and arguably only possible because of — a large body of active research-community measurements of wide-scale connection tampering.
Direct versus indirect
Insight need not come from direct observation. Consider path capacity — bandwidth. The common direct method is a speed test, which has two problems: it consumes as much bandwidth as it can (an ethical dilemma revisited above), and it actually measures throughput between sender and receiver — the available bandwidth, or residual capacity, of the bottleneck link. If two speed tests share a bottleneck, each may observe half the actual bandwidth. The numbers show it: observations from a speed test range from 69-85Mbps, a range of nearly 20% around the median rather than a fixed value.

A 25+ year-old indirect alternative is the packet pair, or packet train. It transmits pairs of packets with no delay between them, recording transmission times and then arrival times; the change between the two indicates the bottleneck bandwidth. Repeating the probes and applying statistical analysis yields a good estimate of true bottleneck bandwidth. Rather than pushing bytes and counting them over time, the technique infers the metric from the timing between packets.

The measurement lifecycle
Measurements are most powerful when they lead to reasonable predictions — sometimes confirming our understanding of the world and the systems we deploy, occasionally revealing something new. Predictive measurements follow a simple pattern: curate data, construct a model from that data, then validate the model, ideally with different data. Together these form a measurement lifecycle.

A full exercise spans the whole lifecycle, but advances within any single step can be extremely valuable. High-quality datasets are so difficult to curate that each can be a valid contribution, as can modeling techniques or validation tools. Measurement spans expert domains and benefits from diverse skill sets.
Curation: the linchpin
Data gathering and curation is the most common and familiar measurement exercise — often treated as synonymous with measurement. Data alone can be fascinating and useful, as Cloudflare Radar demonstrates, and simple counting can place our environments in context.
Curation consumes more energy, time and resources than modeling or validation. The cyclical pattern explains why: validation requires a model, models are built from data, and without data there is no model, no validation, no insight or prediction. The quality of each step depends on the quality of the previous one. The Large Hadron Collider and the James Webb Telescope show how much we can and need to do in relentless pursuit of high-quality data; the Internet measurement community's always-on equivalents, such as CAIDA and RIPE's Atlas, are less glamorous but no less important.
High-quality curation is hard. Fortunately, "high-quality" means representative, not perfect: if counting distance or time, accuracy must reflect the true value, and large populations can be reasonably studied from far fewer samples. The global assessment of connection tampering, for instance, drew valuable insights from a 1 in 10,000 (0.0001%) sample — a rate that works in part because of the immense diversity of Cloudflare's customers, which attracts traffic for all kinds of content and purposes.
Another, arguably counterintuitive, misconception is that more data naturally reveals more detail and answers more questions. As Ram Sundaran writes in a guest post, there is sometimes so much noise that finding answers in large datasets can seem like a small miracle.
Models as testable hypotheses
A model need not be elaborate to be useful. Sometimes it is just a statement of belief that can be checked against reality — for instance, the assumption that an ISP or network will typically prefer a direct no-cost peering path to a CDN over transit paths that incur a cost, even when the direct route is longer. Stated this way, the assumption becomes a hypothesis that validation can confirm or reject.

Predictive models go further, aiming at system behavior that is not obvious, not directly observable, or hard to establish. They typically draw on statistical techniques — identifying underlying stochastic processes, or building machine learning classifiers — though much of the statistical work goes into characterizing the curated data itself. Simple probability distributions with means, medians, variance and confidence indicators can be remarkably powerful.
Modeling how networks interconnect
How networks choose to connect to one another has drawn particular attention, since understanding the Internet's formation and growth is essential to simulation and helps predict how networks might fail. The Barabási–Albert (B-A) model assumes preferential connectivity — "rich get richer": a new network attaches to an existing one with a probability proportional to that network's number of connections. Later models dropped the intelligent selection mechanism; one based on the sizes of networks is more general, resembling how celestial bodies form.

Knowing when not to reach for ML is itself a skill. A guest blog describes ML being ruled out for understanding anomalous TCP behavior: TCP is tightly specified, so a full enumeration of packet sequences was feasible — and proved correct. Domain knowledge likewise shapes model quality. In detecting multi-user IPs, ML models successfully identified them, but isolating carrier-grade NAT devices (CGNATs) stayed elusive until domain expertise was applied. CGNATs are unusual among large-scale multiuser IPs because, unlike VPNs and proxies, users neither choose them nor know they exist. The insight that they are typically deployed across contiguous IP ranges (e.g. a /24 block) turned out to be a very important model feature.

Validation makes or breaks the exercise
Validation carries nearly all the value of a measurement exercise: it tests model output against data. Predictions reflected in the data confer validity; predictions that conflict with it point to a flawed model or to bias in the curated data. This is where good measurement most often falls apart, in one of two ways.
The first is unrepresentative validation data. Curating daytime traffic data, building a model, then validating against nighttime traffic would be a mistake, and QUIC data cannot validate measurements about TCP unless the hypothesis is that the two share attributes. Differences between validation and initial data must never be allowed to corrupt the measurement.
The second risk is validating against the curated data directly. This does neutralize dataset differences, but the only conclusion it supports is that the model describes the data — not what the data represents. Machine learning is itself a measurement in this sense, following the same lifecycle: curate data, build a model through an ML algorithm, validate output against data. The early practice of splitting one dataset 70% for training and 30% for validation makes an unwarranted positive evaluation more likely and can mislead. At best, a model trained on data that amplifies or omits important characteristics reflects those biases — a potential source of algorithmic bias.
Confidence rises when models hold up against unrelated data: the same attributes from a different source, such as models built from passive RTT logs and validated against active pings, or entirely different signals, such as confirming connection tampering with distributions and header values left out of the model's construction.
Ethics come before the measurement
Network measurement is easily mistaken for a risk-free activity detached from human effects. Speed tests and the packet-pair technique show otherwise. A speed test estimates bandwidth by consuming all available bottleneck capacity, which may or may not sit within the tester's own network; the cost of that consumption can fall on others and certainly degrades performance for the network's users. Those risks motivated packet pairing, which infers bandwidth from a few packet pairs and some math — with some orchestration between sender and receiver.
Best practice scrutinizes risks and effects before measuring. The burden is real, but ethical constraints often spark creativity and produce new methodology: the search for alternatives to JavaScript injection led to Cloudflare's own efforts to estimate the performance of other CDNs from passive data. See "Ethical Considerations in Network Measurement Papers" (Communications of the ACM, 2016).
Representation at every stage
Visualization and representation matter at every stage of the lifecycle. Representations should at minimum improve understanding and ideally make follow-up actions clear; statements without context are poor ones. "30% greater chance" sounds significant but means little without a reference point — 30% of 0.5% is likely less a concern than 30% of a 20% chance.
Cloudflare's "closeness" statement is one example: the company is "approximately 50 ms from 95% of the Internet-connected population globally." It comes from a survey of logs: for every IP address connecting to Cloudflare, half the minimum RTT is used as a worst approximation of latency to Cloudflare, and in 95% of cases minRTT/2 is at or below 50 ms.
Visualizations can be powerful enough to mislead outright, as a later blog post on routing resilience evaluations discusses. Below, two bar charts rank US states by interconnection facilities. On the left, states are ordered by raw count; the top state has more than 140 facilities. On the right, counts are normalized by state population.

The state names are deliberately omitted from the x-axis as a distraction; bars are colored green above and yellow below the median facilities per person in the right-hand chart. Immediately visible: the two states with the most facilities fall below that median — the bottom half when ordered per person.
Some visualizations leave no doubt. The image below gives strong evidence that both data and models were correct. Each column is a type of connection anomaly observed, split proportionally by the country where the connection originated. The left-most column, SYN→∅ anomalies (a type of timeout), is dominated by China, India, Iran, and the United States. Putting the data first in this way mitigated any bias about explanations, mechanisms or locations.

The arrangement answered one question at a glance: "Are the failures expected behaviour?" Expected or Internet-normal failures would appear in roughly similar proportions, not such different ones. The visualization was strong validation (though not the only one) of the approach and intuition — and opened further avenues of investigation.
Where this leaves measurement
Cloudflare continues to explore new ways of using available passive data and welcomes ideas. Measurement is a community-wide endeavour that helps us understand the Internet we all depend on, and the hope is that this serves both as an introduction to its challenges and as a map for evaluating measurement work published at Cloudflare or anywhere else.



