Closing the UDP Performance Gap for QUIC

QUIC’s decision to run over UDP is what makes it so deployable: it works through existing network infrastructure and lives entirely in user-space, letting browsers ship new protocol features without waiting on operating system updates. The flip side is that UDP has never received the same level of optimization as TCP. Decades of work have gone into TCP segmentation offload, hardware acceleration, and kernel tuning; UDP has largely been left to fend for itself, and QUIC inherits that disadvantage.

That gap matters because QUIC’s throughput is ultimately limited by how fast an implementation can push UDP datagrams. While the measurements below were taken on a single host running Linux 5.3 (client and server on the same laptop), they illustrate the relative impact of each technique rather than production-network performance.

Starting Point: One Packet Per Syscall

The baseline QUIC implementation in NGINX uses sendmsg() to transmit each UDP packet individually. The struct msghdr contains a struct iovec that can hold multiple buffers, but all buffers in a single iovec are coalesced into one UDP datagram, so the kernel still only sends one packet per system call.

sendmsg

The result is a ceiling of roughly 80–90 MB/s when h2load performs 10 sequential requests for a 100 MB resource. The bottleneck is the sheer number of system calls required—each one triggers a costly user-kernel context switch, and QUIC packet delivery demands many of them.

Batching with sendmmsg()

The straightforward remedy is sendmmsg(), a sibling of sendmsg() that accepts an array of struct mmsghdr entries, each representing one UDP datagram with its own struct msghdr and struct iovec. Swapping NGINX to use sendmmsg() collapses many system calls into one.

% sudo bpftrace -p $(pgrep nginx) -e 'tracepoint:syscalls:sys_enter_sendm* { @[probe] = count(); }'
Attaching 2 probes...
 
 
@[tracepoint:syscalls:sys_enter_sendmsg]: 2437
@[tracepoint:syscalls:sys_enter_sendmmsg]: 15676

The syscall count drops considerably, and throughput climbs accordingly—though the performance gain is smaller than the reduction in syscalls might suggest, since the kernel still has to walk each buffer individually.

sendmmsg-chart-1

UDP Segmentation Offload

Even with sendmmsg(), the application must split every QUIC packet into its own buffer. Linux’s Generic Segmentation Offload (GSO) removes that requirement by letting the application hand the kernel a single contiguous "super buffer," which the kernel then segments into packets as late as possible. GSO was originally TCP-only, but UDP support arrived in Linux 4.18. Control is via the UDP_SEGMENT socket option, or via ancillary data for per-call control.

setsockopt(fd, SOL_UDP, UDP_SEGMENT, &gso_size, sizeof(gso_size)))
cm = CMSG_FIRSTHDR(&msg);
cm->cmsg_level = SOL_UDP;
cm->cmsg_type = UDP_SEGMENT;
cm->cmsg_len = CMSG_LEN(sizeof(uint16_t));
*((uint16_t *) CMSG_DATA(cm)) = gso_size;

Here gso_size sets the segment length; the application supplies one large buffer made up of packets of that size (plus a trailing smaller one), and the kernel handles the segmentation. Hardware offload can push this further, but no capable NIC was available for testing. The UDP_SEGMENT option supports batching up to 64 segments.

Combining GSO with plain sendmsg() already yields significant gains over the baseline:

gso-chart

The syscall count drops accordingly compared to sendmsg() without GSO.

% sudo bpftrace -p $(pgrep nginx) -e 'tracepoint:syscalls:sys_enter_sendm* { @[probe] = count(); }'
Attaching 2 probes...
 
 
@[tracepoint:syscalls:sys_enter_sendmsg]: 18824

The biggest win comes from using GSO with sendmmsg(). Each struct msghdr in the batch can carry up to 64 segments by setting UDP_SEGMENT via ancillary data, so a single system call can transmit multiple "super buffers," each of which the kernel segments into dozens of packets. The improvement is substantial.

The Pacing Problem

Raw transmission speed is only part of the story. Blasting packets at maximum rate creates bursty traffic that invites congestion and packet loss on real networks, which defeats the purpose of the optimization. Packet pacing—inserting a brief delay between outgoing packets—smooths that burstiness and helps flows perform better. Linux handles this for TCP through the fq packet scheduler and the BBR congestion control algorithm’s built-in pacer.

QUIC’s user-space design creates a tension: pacing each packet individually prevents application-side batching, while batching sends all queued packets as fast as the kernel can manage. Linux offers two socket-level facilities that can offload pacing back to the kernel:

  • SO_MAX_PACING_RATE: sets a ceiling for the fq scheduler’s pacing of outgoing packets. It works on UDP sockets, but integrating it with QUIC is awkward because a single UDP socket can carry multiple QUIC connections—unlike TCP, where each connection owns its socket. It is also not flexible enough to replicate a BBR-style pacer.
  • SO_TXTIME / SCM_TXTIME: lets the application schedule specific packets for specific transmission times, giving fq a timestamp to hold packets until. This offers finer control and fits naturally into both sendmsg() and sendmmsg(), but it lacks support for per-packet timestamps when GSO is active—segmented packets all end up sent at the same moment.

The throughput gains from batching and segmentation are clear, but how they will interact with effective pacing remains an open area for experimentation.