How we built Network Analytics v2

Why a redesign?

When Cloudflare first launched Network Analytics in 2020, it gave Magic Transit and Spectrum customers the ability to see exactly how their traffic was being handled by our DDoS protection systems. Every packet sampled from the global network was analyzed against the list of mitigation rules deployed by dosd, our in-house DDoS detection engine, and the results were rolled up into one-minute aggregates and stored in ClickHouse for long-term querying via GraphQL APIs.

That system worked well for years, but its architecture had one critical limitation: it assumed that a packet's fate could always be determined by looking purely at its contents. That assumption held for dosd and stateless systems like Magic Firewall, but it completely breaks down for flowtrackd — our Advanced TCP Protection system — which determines whether a packet belongs to a legitimate connection by tracking the state of previous packets.

The consequence was that Network Analytics v1 reported only a subset of the mitigations we applied to customer traffic. Reporting on Magic Firewall actions would have been achievable, but adapting the existing design to stateful, connection-oriented systems like flowtrackd was not. The core assumption had to be revisited from the ground up.

Thinking about network functions

Traditional on-premise network topologies provide a useful mental model. A packet typically cascades through a sequence of specialized devices — first a firewall, then a router, then a load balancer — each operating as an independent "network function" with defined inputs and outputs. Cloudflare's software stack has the same shape, except each server hosts those functions as separate software components running in sequence.

This observation pointed us toward the standard monitoring patterns used on traditional networks: Netflow and sFlow. Nearly every hardware appliance can emit samples to a centralized collector. Operators take samples at multiple points in the network so that each device can be monitored independently. That was the critical difference from our design, which sampled packets only once, at the edge, before any processing took place.

Netflow and sFlow also carry far more than packet headers. Their samples include metadata about the interface a packet arrived on and exited from, whether it was passed or dropped, and which firewall rule or ACL caused the action. Better still, the metadata format is extensible, meaning each device can attach fields that are only meaningful in its own context, and flow collectors can still present a rich view of the network without understanding every device's internals.

That flexibility was exactly what we needed. We realized that the right design for Network Analytics v2 is for each individual software component in the processing chain to emit its own packet samples, with its own context-specific metadata attached. That way, an event logged by DDoS mitigation might include the mitigation rule ID, an event logged by a firewall might reference the firewall rule, and the analytics platform can surface the full chain of decisions without needing to decode how each system works internally.

Having each component emit its own stream of packet samples felt counterintuitive at first, in part because it meant we would no longer be producing a single canonical stream that described every packet in the network. But once we looked at how traditional flow collectors treat devices as independent producers of samples, the value became clear. The architecture that supports multiple products and future components we have not yet conceived requires that each piece of the pipeline be able to describe its own behavior, in its own terms, at the moment it makes a decision.

Rebuilding the sampling pipeline

The redesign of Network Analytics splits into two main efforts. The first was building a new data pipeline, called samplerd (the “sampler daemon”), that receives metadata-rich packet samples from different sources, normalizes them, and writes them to long-term storage in a ClickHouse database. The second, larger effort was adapting existing Cloudflare systems to send packet samples to samplerd.

Reworking l4drop

Incoming packets first pass through our XDP daemon, xdpd, which manages several XDP programs: a packet sampler, l4drop for attack mitigation, and L4LB for load balancing. Previously, these programs were chained in a fixed order:

A packet entering the pipeline would pass through the sampler, then l4drop, where a mitigation decision would be made, and finally L4LB. Sampling had to happen before the mitigation decision so that dropped packets were still visible in dashboards. This is critical for understanding what is being dropped and adapting mitigations as attacks evolve.

However, this ordering meant samples could not record the outcome of the mitigation decision or the reason behind it. For example, a packet may be dropped due to a matching attack signature, or it may pass because it fell under a rate-limiting threshold. All this context is needed in the samples shown to customers.

The initial solution was to move the sampler after l4drop: l4drop would mark each packet as drop or pass, along with metadata explaining why, and the sampler would then decide whether to emit a sample based on those marks. The problem: this requires copying all decision metadata for every packet, even those that will never be sampled. Given that every packet entering Cloudflare passes through xdpd, that copy cost was prohibitive.

The key insight was that we only need to copy metadata for packets that will actually be sampled. So we split the sampler in two and sandwiched the mitigation decision programs between them:

First, a pre-sampler makes the sampling decision and marks the packet accordingly. The mitigation programs then determine drop or pass, but only copy metadata when the sampling mark is present. A final sampler stage checks both marks, builds a sample when needed, and drops or passes the packet accordingly. This makes the sampler no longer a standalone component but tightly integrated with l4drop.

Adapting iptables rules

Some mitigations, like stateful connection tracking, run in iptables rather than l4drop. iptables rules are evaluated in order, and rules may include rate limiting that only drops packets beyond a threshold — for instance, 10 packets per second.

Previously, rules would match on packet characteristics and immediately decide whether to drop or pass. To emit annotated samples, we considered simply adding sampling to the drop rules, but this would miss packets that pass a rate limiter (which is also important to report) and would cause oversampling for packets that pass one rate limiter only to be dropped by another down the line.

Instead, we applied the same staged approach used in l4drop. The rules are split into three sets:

1. The first set makes the random sampling decision and marks packets that should be sampled.
2. The second set is the mitigation rules. When a rule decides to drop a packet, it jumps to the third set; otherwise, the packet falls through to it.
3. The third set emits a sample if the sampling mark is present, then drops or passes the packet.

Communication between rule sets uses Linux packet marks. With this staging, no packet can be double-sampled, and both passed and dropped packets from rate limiters are covered.

ClickHouse and the GraphQL query layer

After samplerd collates samples from mitigation systems, an inserter does light processing before writing to ClickHouse. The inserter enriches metadata (for example, mapping a destination IP to an account) and, where an ongoing attack is detected, adds a unique attack ID to each associated sample.

To sustain high write throughput, we designed inserters so that data never needs to be updated once written. We primarily use ClickHouse's MergeTree table engine, but also use the AggregatingMergeTree engine for better query performance. Each sample is stored in a table structured as follows:

The sample interval records the number of packets between samples, per our ABR sampling approach.

These tables back the GraphQL Analytics API, which the dashboard uses directly or indirectly. A common query identifies attributes of a specific attack — for example, whether it uses a fixed destination port or IP. Since attacks can span days or weeks, such queries risk being slow. A naive query estimating the number of distinct values for a port or IP might look like this:

SELECT if(uniq(dest_ip) == 1, any(dest_ip), NULL), if(uniq(dest_port) == 1, any(dest_port), NULL)
FROM samples
WHERE attack_id = ‘abcd’

That query retrieves far more state than needed. Since we only care whether there is one value or many, we can check if the maximum equals the minimum for ordered values:

SELECT if(min(dest_ip) == max(dest_ip), any(dest_ip), NULL), if(min(dest_port) == max(dest_port), any(dest_port), NULL)
FROM samples
WHERE attack_id = ‘abcd’

Storing a running minimum or maximum takes roughly the size of the column itself, versus the state that uniq() requires. This is exactly what the AggregatingMergeTree engine handles — it computes and stores aggregate results grouped by a key, in our case the attack ID:

This generalizes to other aggregate functions, like sum(), as long as the function is associative: applying it to the result from a subset and another value gives the same result as applying it to the full set.

Querying the small aggregating table is dramatically faster and simpler. In practice it amounts to roughly 0.002% of the original data size (though not all columns are present). We wrap it in a SQL view:

SELECT if(min_dest_ip == max_dest_ip, min_dest_ip, NULL), if(min_dest_port == max_dest_port, min_dest_port, NULL)
FROM aggregated_samples
WHERE attack_id = ‘abcd’

Implementation detail: rows in the aggregated table can occasionally span multiple partitions, so an attack ID may appear in several rows. Production queries therefore take the min or max across all matching rows in the aggregating table, which is usually only three or four rows — still far faster than scanning thousands of samples spread over multiple days.

SELECT if(min(min_dest_ip) == max(max_dest_ip), min(min_dest_ip), NULL), if(min(min_dest_port) == max(max_dest_port), min(min_dest_port), NULL)
FROM aggregated_samples
WHERE attack_id = ‘abcd’

What the rewrite delivered

The migration to Network Analytics v2 has been worthwhile. Customers see higher fidelity, more accurate traffic data, and we internally gain much better tooling for troubleshooting and tuning mitigations. As new mitigation systems are deployed in the future, the reporting layer is now positioned to support them without a major redesign.