Defending UDP services without drowning legitimate traffic
Cloudflare’s DDoS mitigation stack spans multiple layers. Some layers inspect high-level traffic patterns; others are protocol- or application-specific. But sometimes the best vantage point for stopping an attack is inside the service itself. During my summer internship at Cloudflare, I built an open-source framework that gives UDP services a self-defense mechanism against floods, drawing on the company’s experience running UDP-based services like Spectrum and the 1.1.1.1 resolver.
The core goal is simple to state: attackers should not be able to drown legitimate traffic. The method is equally direct — identify a group of packets that form an attack and rate-limit that group using the packet attributes available to us (source and destination addresses and ports). We want to drop only as much traffic as needed to stay within limits, because even an attack group may contain legitimate packets. Noisy neighbors should not be cut off completely.
Finding heavy hitters in the stream
Grouping packets is harder than rate-limiting them. The only attributes we can trust are the four-tuple of source and destination address and port, and even those can be spoofed. Moreover, tracking a rate per address does not scale — especially once IPv6’s address space is in the picture.
This is a classic Heavy Hitters problem from the data-stream literature: identify elements whose frequency exceeds a given fraction of the total stream. A naive per-element counter does not scale, but probabilistic algorithms such as the CountMin sketch or the SpaceSaving algorithm deliver estimated counts with constant memory. We store rates into a CountMin sketch rather than counts, so memory use is independent of the number of distinct elements we track.
Real attacks, however, do not always originate from a single address or port. A reflection attack may hit a service with random source addresses but a single source port; a flood may come from an entire /24 subnet. Tracking rates only for fully specified tuples would miss these patterns.
Hierarchical heavy hitters for grouping
The extension of Heavy Hitters that handles this is Hierarchical Heavy Hitters, which exploits the natural hierarchy in elements — an IP address can be generalized from a fully specified address to a /24 subnet to a /0 wildcard. Each generalization is assigned a level, from most specific (level 0) to least specific.
The data structure maintains a CountMin sketch per subnet level. Each sketch is updated when a packet arrives; to decide whether a packet may pass, we check the rate for the packet’s tuple in every level’s sketch against the configured rate limit (for example, 25 packets per second).
Tracking only one attribute wastes useful context. The academic work on Hierarchical Heavy Hitters with the SpaceSaving algorithm proposes a two-dimensional extension for addresses, and we extend it further to include ports. Ports lack a natural hierarchy, so they take only two states: specified (e.g., 8080) or wildcard.
Our traversal algorithm is simpler than the paper’s because we do not need to enumerate all current heavy hitters — we only need to know whether the packet in hand would be one. And because our goal is to prevent any heavy hitters from passing, the structure will converge toward a state with none left. At each level we update every node and track the maximum rate seen; we then compute a probability based on that maximum and the configured limit, which determines whether traffic proceeds to the next, less specific level.
There is a subtlety: we do not drop a packet outright when any rate exceeds the limit. Instead, we let it through with probability rate limit/maximum rate seen. Dropping all traffic that exceeds the limit would eliminate the whole group, rather than trim it down to compliance. Since more specific nodes continue to be updated even when a node hits its limit, the filter converges toward the attack’s actual pattern without manual intervention, leaving unrelated traffic minimally affected.
A socket filter in Go
To keep overhead minimal before a drop decision is made, we turned to BPF. Specifically, Socketfilters attach to a single socket and run before the kernel passes a packet to userspace. That placement has advantages: the filter sees full packet information, fires per socket, and can be attached without root privileges — any application can set its own rate limits as needed. BPF limitations are well documented; the one unique to this project is the absence of floating-point arithmetic.
The kernel does not support floats, so we implemented a fixed-point representation that dedicates part of the 64-bit value to the fractional part of a number. Addition and subtraction are safe; multiplication and division demand twice the bits to avoid precision loss. Since 64 bits is the largest available width, we instead convert one argument to an integer, losing the fractional part but keeping intermediate results within bounds. That is acceptable for large rate values, but it means every fixed-point operation requires care with intermediate precision.
The resulting library, rakelimit, is open source at cloudflare/rakelimit. It is a Go library that attaches to any UDP socket and is straightforward to configure. This is an early prototype — the development continues — and we presented the work at this year’s Linux Plumbers Conference.
