The UDP restart problem

Graceful restarts for TCP servers are well understood: keep the old process alive while new connections go to the new instance, let existing connections drain, then shut down. UDP has historically been easier in one sense—stateless protocols like DNS don't care about restarts—but modern UDP-based protocols such as QUIC, WireGuard, and SIP maintain state across packets. Dropping that state means dropping connections.

At Cloudflare, where we deploy code frequently, restarting UDP servers without packet loss became a real requirement once HTTP/3 and QUIC became critical services. We've explored approaches like the established-over-unconnected method, but it has significant drawbacks: it is prone to race conditions with multi-packet handshakes and the kernel hash table used for dispatching packets can overflow when many inbound UDP sockets share the same local IP:port. We needed something better.

The result is udpgrm, a lightweight daemon that orchestrates UDP server upgrades by leveraging Linux's SO_REUSEPORT API and eBPF. It keeps old and new socket instances in the same reuseport group, tracks flows with an eBPF program, and routes packets to the correct instance so no packets are dropped during a restart. The source is available on GitHub.

How SO_REUSEPORT groups work

Linux's SO_REUSEPORT socket option—distinct from the more familiar SO_REUSEADDR—allows multiple sockets to bind to the same IP:port tuple. The kernel organizes these sockets into a reuseport group, which is essentially multiple packet queues behind one address. By default, the kernel distributes inbound packets across the group using a hash of the packet's 4-tuple. There's also SO_INCOMING_CPU, which steers packets toward sockets on the CPU that received the packet, but this offers limited flexibility.

For finer control, Linux provides SO_ATTACH_REUSEPORT_CBPF and its successor SO_ATTACH_REUSEPORT_EBPF, which let a process attach a BPF program to make socket selection decisions. With eBPF, developers can implement arbitrary routing logic. A typical program uses bpf_sk_select_reuseport to pick a socket from a map—either a SOCKHASH, SOCKMAP, or the older SOCKARRAY—indexed by a key. For a SOCKHASH, the map holds references to sockets even though the value size appears to be a scalar 8-byte value, which allows a simple number-to-socket mapping.

There's a catch: the socket map must be populated and maintained from user space, outside the eBPF program. Keeping this map accurate under restarts, crashes, and scaling events is difficult. udpgrm's purpose is to handle exactly that bookkeeping so server processes don't have to.

Socket generations and flow stickiness

To reason about restart workflows, udpgrm defines a socket generation: a set of sockets within a reuseport group belonging to one logical application instance. During a restart, the new process creates a new socket generation while the old process continues running with its own generation. Reuseport eBPF routing then must solve two problems: new flows should go to the active (new) instance, and existing flows should stay pinned to their original socket in the older generation until they drain.

Routing new flows is straightforward: udpgrm keeps a reference to the working generation—the generation that should receive new connections—and the eBPF program consults that pointer for each new packet. The harder part is distinguishing new packets from packets belonging to established flows, since that depends entirely on the protocol. QUIC has an initial packet concept similar to TCP's SYN, but other protocols define flows differently—some might use the 5-tuple, while QUIC uses a connection ID in the packet header to survive NAT rebinding.

Because flow semantics vary, udpgrm makes this configurable. Each reuseport group specifies a flow dissector that performs two tasks: recognize new packets versus packets from established flows, and for recognized flows, identify which specific socket the flow belongs to. udpgrm ships with three dissectors and is configurable for arbitrary UDP protocols.

Using udpgrm as an administrator

udpgrm is a stateful daemon that can be started with a simple command:

$ sudo udpgrm --daemon
[ ] Loading BPF code
[ ] Pinning bpf programs to /sys/fs/bpf/udpgrm
[*] Tailing message ring buffer  map_id 936146

Beyond running the daemon, udpgrm must hook into getsockopt, setsockopt, bind, and sendmsg syscalls, which are scoped to a cgroup. Hooks can be installed into the current cgroup:

$ sudo udpgrm --install --self

Or integrated into a systemd service configuration. Once running, the CLI can list reuseport groups, sockets, and metrics for debugging:

$ sudo udpgrm list
[ ] Retrievieng BPF progs from /sys/fs/bpf/udpgrm
192.0.2.0:4433
	netns 0x1  dissector bespoke  digest 0xdead
	socket generations:
		gen  3  0x17a0da  <=  app 0  gen 3
	metrics:
		rx_processed_total 13777528077
...

The programmer's view: registering sockets

Server code creates its own UDP sockets with SO_REUSEPORT enabled. Communication with the udpgrm daemon happens through setsockopt calls that are intercepted by eBPF hooks. A typical socket setup has three steps: retrieve the working generation number (and verify udpgrm presence—its absence is acceptable for non-production workloads), register the socket to a chosen socket generation, and then bump the working generation pointer. Registering to work_gen + 1 and confirming the registration is a common pattern.

That's the full API surface. Under the hood, udpgrm takes care of installing the reuseport eBPF program, maintaining the SOCKHASH, and collecting metrics.

Dealing with privileged ports and systemd

Production servers often bind to low ports like :443, requiring CAP_NET_BIND_SERVICE. It's cleaner to configure listening sockets outside the server process and pass them in via socket activation. However, systemd cannot create a fresh set of UDP SO_REUSEPORT sockets per server instance. udpgrm provides udpgrm_activate.py to fill this gap:

[Service]
Type=notify                 # Enable access to fd store
NotifyAccess=all            # Allow access to fd store from ExecStartPre
FileDescriptorStoreMax=128  # Limit of stored sockets must be set

ExecStartPre=/usr/local/bin/udpgrm_activate.py test-port 0.0.0.0:5201

This binds to the address, stores the socket in the systemd FD store under a named key, and the server inherits the socket with the usual FD_LISTEN environment variables.

Systemd's service model also assumes at most one instance of a service at a time, but graceful restart requires at least one—old and new running together during the drain. To reconcile this, udpgrm includes a decoy script that exits when systemd sends the stop signal while the actual old server instance keeps running in the background. The full pattern thus combines three elements: cgroup hooks via udpgrm --install --self, socket creation via udpgrm_activate.py, and mmdecoy to satisfy systemd's lifetime expectations.

How udpgrm decides which socket gets the packet

udpgrm ships with several built-in strategies for mapping an incoming packet to the correct socket generation. The simplest is the flow-based approach: a table, indexed by a hash of the standard 4-tuple, holds a target socket identifier for each active flow. Because the table size is fixed, the number of concurrent flows this mode can track is bounded. The table is only populated when a message actually goes out; udpgrm hooks the sendmsg syscall and records the flow at that moment, marking it "assured."

The second mode is cookie-based. Instead of maintaining state in a flow table, the target socket identifier — the udpgrm cookie — is embedded directly in the incoming packet. In QUIC, for instance, that identifier can live inside the connection ID field. The dissection logic itself is expressed as cBPF code, which makes this mode flexible but slower: the cBPF is interpreted within an eBPF program. It also demands that the protocol and server cooperate by carrying and recognizing the cookie.

The third mode is effectively a no-op: no state tracking at all. This is aimed at traditional UDP services — DNS is the canonical example — where dropping even a single packet during an upgrade is unacceptable, and where the connectionless nature of the traffic makes per-flow routing unnecessary.

Beyond these three, udpgrm exposes a template for custom dissectors. The included reference implementation is a QUIC dissector that parses the TLS Server Name Indication and can steer specific hostnames to particular socket generations. That template is where the real power lies: flow mode is the least effort for legacy protocols, the cBPF approach is a good fit for experimentation with custom connection IDs (the developers used it while building their own QUIC DCID scheme), and no-op mode is only suitable for very narrow server profiles. For arbitrary, high-performance logic, the custom dissector path is the intended route. Full details are in the project README.

The bigger picture

The rise of QUIC — and UDP-based transports in general — makes graceful restarts a real operational concern, yet until now there wasn't a reusable, configurable tool to handle it. udpgrm tackles the problem from several angles at once: a clean setsockopt() interface, hidden socket-stealing mechanics, expressive dissector configuration, and systemd integration that fits existing service management patterns.

Under the hood, this is a genuinely hard problem. The Linux Sockets API was not designed for the way modern UDP workloads need to migrate connections between processes. udpgrm works around that gap, but the authors are clear that this is really a feature the platform should provide. The longer-term hope is that systemd eventually absorbs the core ideas — including the "at least one" server instance requirement, automatic SO_REUSEPORT socket creation, installing the REUSEPORT_EBPF program, and managing the active generation pointer. For now, udpgrm gives the ecosystem the vocabulary and the working implementation to push in that direction.