A UDP Connectivity Puzzle

Cloudflare Tunnel lets customers connect private services through the Cloudflare network without exposing public IPs or opening firewall ports. The cloudflared agent runs alongside those services, proxying traffic to and from Cloudflare's edge. Historically, this depended on long-lived TCP connections multiplexed over HTTP/2 frames. That design has two notable limitations: TCP over HTTP/2 suffers from Head of Line (HoL) blocking, and initiating communication from the cloudflared side is awkward with the current Go HTTP/2 stack.

Moving to QUIC addresses the HoL problem and would make future connection patterns—like cloudflared initiating traffic to another cloudflared—possible. Since QUIC runs over UDP, the edge needed a UDP listener for cloudflared to reach. On paper, that was a small change, but it surfaced a subtle networking issue unique to connectionless protocols.

Initial Failure

Adding a QUIC listener on the edge plus client-side support in cloudflared seemed straightforward. The first test run was not encouraging.

$  cloudflared tunnel run --protocol quic my-tunnel
2021-09-17T18:44:11Z ERR Failed to create new quic connection, err: failed to dial to edge: timeout: no recent network activity

Firewall rules for the new port were in place, and iptables was ACCEPTing the relevant traffic. The standard checks turned up nothing. Time to look at the packets themselves.

Following the Packets

A tcpdump on the server machine revealed both inbound and outbound UDP traffic on port 7844, but something was off. Packets arrived addressed to 198.41.200.200:7844, yet responses left from 203.0.113.0:7844 instead. (The latter is an example IP used for illustration.)

$  sudo tcpdump -n -i eth0 port 7844 and udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:44:27.742629 IP 173.13.13.10.50152 > 198.41.200.200.7844: UDP, length 1252
14:44:27.743298 IP 203.0.113.0.7844 > 173.13.13.10.50152: UDP, length 37

The source address in the replies came from an interface whose network couldn't route back to the public client, so those responses would never arrive. The issue was that the server had multiple IPs on the same interface, and the kernel was choosing the one associated with the default route rather than the one that actually received the incoming packet.

This behavior is fine for connection-oriented TCP. The kernel maintains connection state from the three-way handshake and can pick the correct source IP for replies. UDP, being connectionless, keeps no such state. The typical server pattern uses recvfrom, which tells you who sent the data but not which destination IP received it. When the server later calls sendto to respond, the kernel relies on routing heuristics to choose a source address—and those heuristics sometimes choose wrong, as in this case where the default route's IP won.

Teaching the Kernel the Source IP

The fix is to stop relying on kernel heuristics and set the source address explicitly from the application. Linux provides recvmsg and sendmsg, system calls that carry control messages alongside the actual payload. The msghdr structure's msg_control field can hold out-of-band data, including the destination address information obtained during receive.

ssize_t sendmsg(int socket, const struct msghdr *message, int flags)
ssize_t recvmsg(int socket, struct msghdr *message, int flags);
 
struct msghdr {
     void    *   msg_name;   /* Socket name          */
     int     msg_namelen;    /* Length of name       */
     struct iovec *  msg_iov;    /* Data blocks          */
     __kernel_size_t msg_iovlen; /* Number of blocks     */
     void    *   msg_control;    /* Per protocol magic (eg BSD file descriptor passing) */
    __kernel_size_t msg_controllen; /* Length of cmsg list */
     unsigned int    msg_flags;
};

The approach is to copy the control data from recvmsg back into the sendmsg call, giving the kernel the exact source address to use. The QUIC library in use—quic-go—had recently added support for this, so pulling in the update should have been enough.

It wasn't. A subsequent tcpdump showed the same wrong source IP in responses. Reading the library source suggested the correct values were being passed, so the next step was observing the system calls directly.

Strace to the Rescue

Using strace to trace system calls for the process quickly exposed the discrepancy. The recvmsg call was passing the right destination—the trace showed ipi_addr=inet_addr("172.16.90.131")—but the accompanying sendmsg call was not setting ip_spec_dst at all. Without that field populated, the kernel went back to guessing and got it wrong again.

17:39:09.130346 recvmsg(3, {msg_name={sa_family=AF_INET6,
sin6_port=htons(35224), inet_pton(AF_INET6, "::ffff:171.54.148.10", 
&sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, msg_namelen=112->28, msg_iov=
[{iov_base="_\5S\30\273]\275@\34\24\322\243{2\361\312|\325\n\1\314\316`\3
03\250\301X\20", iov_len=1452}], msg_iovlen=1, msg_control=[{cmsg_len=36, 
cmsg_level=SOL_IPV6, cmsg_type=0x32}, {cmsg_len=28, cmsg_level=SOL_IP, 
cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("eth0"),
ipi_spec_dst=inet_addr("198.41.200.200"),ipi_addr=inet_addr("198.41.200.200")}},
{cmsg_len=17, cmsg_level=SOL_IP, 
cmsg_type=IP_TOS, cmsg_data=[0]}], msg_controllen=96, msg_flags=0}, 0) = 28 <0.000007>
17:39:09.165160 sendmsg(3, {msg_name={sa_family=AF_INET6, 
sin6_port=htons(35224), inet_pton(AF_INET6, "::ffff:171.54.148.10", 
&sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, msg_namelen=28, 
msg_iov=[{iov_base="Oe4\37:3\344 &\243W\10~c\\\316\2640\255*\231 
OY\326b\26\300\264&\33\""..., iov_len=1302}], msg_iovlen=1, msg_control=
[{cmsg_len=28, cmsg_level=SOL_TCP, cmsg_type=0x8}], msg_controllen=28, 
msg_flags=0}, 0) = 1302 <0.000054>

The root cause turned out to be a bug in the library: the control message level was set to IPROTO_TCP instead of IPPROTO_IPV4 when making the sendmsg call. A small patch to correct the protocol constant, and the strace output showed the expected behavior.

18:22:08.334755 sendmsg(3, {msg_name={sa_family=AF_INET6, 
sin6_port=htons(37783), inet_pton(AF_INET6, "::ffff:171.54.148.10", 
&sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, msg_namelen=28, 
msg_iov=
[{iov_base="Ki\20NU\242\211Y\254\337\3107\224\201\233\242\2647\245}6jlE\2
70\227\3023_\353n\364"..., iov_len=33}], msg_iovlen=1, msg_control=
[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data=
{ipi_ifindex=if_nametoindex("eth0"), 
ipi_spec_dst=inet_addr("198.41.200.200"),ipi_addr=inet_addr("0.0.0.0")}}
], msg_controllen=32, msg_flags=0}, 0) =
33 <0.000049>

With that in place, cloudflared successfully established QUIC connections from anywhere out to the Cloudflare network.

Lessons Learned

Though the underlying defect was trivial, the debugging path was instructive. UDP leaves source-address selection to the kernel unless the application explicitly provides it, and the tools that brought this into focus were the classics: tcpdump for seeing what is on the wire and strace for confirming what the process actually asks the kernel to do. For anyone debugging elusive connectivity problems, both remain essential. The current cloudflared release supports QUIc tunnels via the protocol flag set to quic.