Everything you ever wanted to know about UDP sockets but were afraid to ask, part 1

UDP servers are harder than they look

Cloudflare built its reputation on HTTP servers running over TCP. Lately the company has been running large stateful UDP services, driven by QUIC (and HTTP/3), WARP (WireGuard tunneling), and generic UDP traffic via Spectrum. UDP looks simple next to TCP, but scaling it takes domain knowledge that only shows up in production. The fundamentals matter: connected vs. unconnected sockets, wildcard binding, and graceful restarts.

Connected and unconnected sockets

UDP sockets come in two flavors. Connected sockets carry a full 4-tuple: {source ip, source port, destination ip, destination port}. Unconnected sockets carry only a 2-tuple: {bind ip, bind port}.

Historically, connected sockets were for outbound flows and unconnected sockets for inbound server-side traffic. That division is not fixed. Connected sockets can serve ingress, and unconnected sockets can send egress — but each choice has trade-offs.

Two code snippets both send a single DNS query packet. The first uses a connected socket; the second uses an unconnected one:

The unconnected version requires the programmer to verify the source IP of any received packet; otherwise random internet noise — port scans, for example — can confuse the logic. Reusing the same unconnected socket to query another server is tempting but dangerous for DNS, since DNS security assumes the client source port is unpredictable and short-lived.

For outbound traffic, connected sockets are generally the better choice. Linux can cache the route lookup result on the connection struct, saving CPU cycles per packet.

There's a way to reuse the descriptor with a fresh source port: the "dissolving socket association" trick, done via connect(AF_UNSPEC). It's obscure but valid Linux behavior.

Echo servers and the wildcard bind problem

Server-side UDP traditionally uses unconnected sockets. Writing a naive echo server looks straightforward, though in practice you should not deploy one: it can become a DoS reflection vector. UDP services should always reply with less data than they received, and should apply rate limiting. (Cloudflare has covered packet-receiving techniques elsewhere.)

That code raises questions that never come up with TCP:

• Packets can exceed 2048 bytes — over loopback, with jumbo frames, or via IP fragmentation.
• An empty payload is legal UDP.
• Inbound ICMP errors need handling.

TCP transparently deals with MTU, fragmentation, and ICMP errors. UDP services must do that work themselves, depending on protocol requirements.

There is a bigger flaw. The server binds to a specific address such as ::1 or 127.0.0.1. Binding to a wildcard exposes the sendto() problem: Linux picks the egress source IP itself, which may not match the destination the client used. If the machine has ::2 on loopback and the client sends to it with source ::1, the reply is sourced from ::1 — wrong from the client's perspective.

marek@mrprec:~$ sudo tcpdump -ni lo port 1234 -t
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
IP6 ::1.41879 > ::2.1234: UDP, length 2
IP6 ::1.1234 > ::1.41879: UDP, length 2

With a wildcard bind, a server may receive packets destined for multiple IPs. The response must use the appropriate source IP, but the BSD Sockets API does not make it easy to discover which address the packet was actually destined to.

Linux and BSD offer CMSG metadata via IP_RECVPKTINFO and IPV6_RECVPKTINFO. The improved server loop uses recvmsg() and sendmsg() instead of recvfrom()/sendto() to request and set that metadata:

The IPV6_PKTINFO CMSG carries this structure, holding the destination IP and interface number — note there is no port field:

Graceful restarts are awkward for UDP

Classic request-response UDP protocols like DNS don't keep state at a higher level, so servers can restart freely — configuration upgrades included. Socket activation via systemd removes even the brief downtime window.

Modern connection-oriented protocols need more care. In TCP, each connection is its own file descriptor: the old process can simply stop calling accept() and let long-lived connections drain while a new instance handles newcomers. NGINX documents exactly this upgrade pattern.

UDP has no accept(), so graceful restarts are genuinely hard.

Established-over-unconnected

Cloudflare sometimes uses a technique it calls "established-over-unconnected". On Linux, you can create a connected socket on top of an unconnected one:

It amounts to reproducing TCP accept() semantics for UDP:

• Start an unconnected socket.
• Wait for an inbound packet.
• Create a fully connected socket over the same local IP and port.

In ss output, the connected socket shares the port with the unconnected parent:

marek@mrprec:~$ ss -panu sport = :1234 or dport = :1234 | cat
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port    Process                                                                         
ESTAB     0         0                    [::1]:1234               [::1]:44592    python3
UNCONN    0         0                        *:1234                   *:*        python3
ESTAB     0         0                    [::1]:44592              [::1]:1234     nc

The approach has two race conditions. First, a client may send multiple packets before the connected socket exists; application code must hand off packets that belong to an already-established flow. Second, between bind() and connect() on the new socket, stray packets for the unconnected socket can arrive. Early packets on the connected socket need source filtering.

Is it worth it? For a modest number of long-lived flows, yes. For high rates of short-lived flows — DNS or NTP — it's overkill.

Graceful service restarts remain especially tricky in UDP. Established-over-unconnected is one workable approach; another based on SO_REUSEPORT with eBPF is left for a future write-up.