Performance work on virtualized networking quickly runs into the tap device. For projects built on Firecracker, the KVM-based VMM that runs lightweight Micro-VMs, every instance talks to the host through a tap device. The API itself is small, but the semantics and tuning knobs are scattered and poorly documented. What follows are the notes worth having up front.

What a tap device actually is

A tap is a virtual network interface that presents itself like an Ethernet card. Rather than terminating in physical wiring, it exposes a file descriptor to an application that sends and receives packets. VPN clients were the historical consumers: traffic gets routed toward the tap interface, and the client picks packets up from the descriptor and processes them. More recently, VMMs such as Qemu, Firecracker and gVisor open the application side of the tap and hand packets to the guest, leaving the interface itself for the host kernel to route, forward or NAT as usual. That inverts the original pattern — during the VPN era the tap was a traffic destination, whereas behind a VM it looks like a traffic source.

Despite appearing trivial, tap devices are hard to make fast, because the Linux networking stack is tuned for packets handled by physical NICs rather than by a userspace application. The interface has grown enough features over the years that good performance is achievable; the rest of this article covers how to use the modern API.

BLOG-2123 Embedded Image - d5mfGT

Source: DALL-E

Choosing the variant

In the kernel the interface is called "the universal tun/tap". The tun variant, selected with IFF_TUN, behaves like a point-to-point link and carries no L2 Ethernet headers — awkward to configure for anyone used to modern Ethernet networks, and unsuitable for Firecracker or gVisor, both of which expect L2 headers. The tap variant, selected with IFF_TAP, does have Ethernet headers and has attracted most of the recent attention.

Creating and inspecting devices

iproute2 handles tun/tap natively: ip tuntap add creates a device and ip tuntap list enumerates them.

$ sudo ip tuntap add mode tap user marek group marek name tap0
$ ip tuntap list
tap0: tap persist user 1000 group 1000

The device flags are also visible via /sys/devices/virtual/net/<ifr_name>/tun_flags.

Opening the clone device

Everything starts by opening /dev/net/tun, the "clone device":

    /* First, whatever you do, the device /dev/net/tun must be
     * opened read/write. That device is also called the clone
     * device, because it's used as a starting point for the
     * creation of any tun/tap virtual interface. */
    char *clone_dev_name = "/dev/net/tun";
    int tap_fd = open(clone_dev_name, O_RDWR | O_CLOEXEC);
    if (tap_fd < 0) {
   	 error(-1, errno, "open(%s)", clone_dev_name);
    }

That descriptor is then used to instantiate a specific device by name:

    struct ifreq ifr = {};
    strncpy(ifr.ifr_name, tap_name, IFNAMSIZ);
    ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_VNET_HDR;
    int r = ioctl(tap_fd, TUNSETIFF, &ifr);
    if (r != 0) {
   	 error(-1, errno, "ioctl(TUNSETIFF)");
    }

An empty ifr_name, or a name that does not exist yet, creates a new tap device; otherwise the existing device is opened. When reopening, flags such as IFF_MULTI_QUEUE must match how the device was originally created, or the call fails with EINVAL. A practical pattern is to retry with the multi-queue setting flipped when EINVAL comes back.

Available ifr_flags bits:

IFF_TAP / IFF_TUN

Already discussed.

IFF_NO_CARRIER

Holding an open tap device file descriptor sets the Ethernet interface CARRIER flag up. In some cases it might be desired to delay that until a TUNSETCARRIER call.

IFF_NO_PI

Historically each packet on tap had a "struct tun_pi" 4 byte prefix. There are now better alternatives and this option disables this prefix.

IFF_TUN_EXCL

Ensures a new device is created. Returns EBUSY if the device exists

IFF_VNET_HDR

Prepend "struct virtio_net_hdr" before the RX and TX packets, should be followed by setsockopt(TUNSETVNETHDRSZ).

IFF_MULTI_QUEUE

Use multi queue tap, see below.

IFF_NAPI / IFF_NAPI_FRAGS

See below.

In practice you almost always want IFF_TAP, IFF_NO_PI and IFF_VNET_HDR, and sometimes IFF_MULTI_QUEUE.

IFF_NAPI and IFF_NAPI_FRAGS

According to the original patchset, IFF_NAPI and IFF_NAPI_FRAGS existed primarily to improve syzkaller's code coverage. Later work found that IFF_NAPI also brings performance benefits when running XDP on tap. It allocates a dedicated NAPI instance for packets an application writes into the tap, which permits both XDP and batching with GRO; otherwise the kernel falls back to a backlog NAPI.

Queue limits in both directions

Internally a tap is a pair of packet queues: exposed as a network interface toward the host and as a character device file descriptor toward the application. The queue running toward the application (the tap TX queue) holds txqueuelen packets, set through an interface parameter:

$ ip link set dev tap0 txqueuelen 1000
$ ip -s link show dev tap0
26: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ... qlen 1000
	RX:  bytes packets errors dropped  missed   mcast      	 
         	0   	0  	0   	0   	0   	0
	TX:  bytes packets errors dropped carrier collsns      	 
       	266   	3  	0  	66   	0   	0

In ip link statistics, the "TX dropped" column means the tap application was too slow and the queue filled up. In the opposite direction — from application to host, the interface RX queue — the limit is expressed in bytes and set with the TUNSETSNDBUF ioctl. Qemu's source comments on this setting; overflowing this queue is not easy to provoke.

VNET_HDR and virtio header layout

Once the device is open, the usual next step is configuring the VNET_HDR size and offloads. The conventional value for VNETHDRSZ is 12:

    len = 12;
    r = ioctl(tap_fd, TUNSETVNETHDRSZ, &(int){len});
    if (r != 0) {
   	 error(-1, errno, "ioctl(TUNSETVNETHDRSZ)");
    }

The sensible set is {10, 12, 20}, all taken from the virtio spec. Twelve bytes leave room for this little-endian header:

struct virtio_net_hdr_v1 {
#define VIRTIO_NET_HDR_F_NEEDS_CSUM  1    /* Use csum_start, csum_offset */
#define VIRTIO_NET_HDR_F_DATA_VALID  2    /* Csum is valid */
    u8 flags;
#define VIRTIO_NET_HDR_GSO_NONE      0    /* Not a GSO frame */
#define VIRTIO_NET_HDR_GSO_TCPV4     1    /* GSO frame, IPv4 TCP (TSO) */
#define VIRTIO_NET_HDR_GSO_UDP       3    /* GSO frame, IPv4 UDP (UFO) */
#define VIRTIO_NET_HDR_GSO_TCPV6     4    /* GSO frame, IPv6 TCP */
#define VIRTIO_NET_HDR_GSO_UDP_L4    5    /* GSO frame, IPv4& IPv6 UDP (USO) */
#define VIRTIO_NET_HDR_GSO_ECN       0x80 /* TCP has ECN set */
    u8 gso_type;
    u16 hdr_len;     /* Ethernet + IP + tcp/udp hdrs */
    u16 gso_size;    /* Bytes to append to hdr_len per frame */
    u16 csum_start;
    u16 csum_offset;
    u16 num_buffers;
};

Advertising offload support

Offloads are enabled with an ioctl:

    unsigned off_flags = TUN_F_CSUM | TUN_F_TSO4 | TUN_F_TSO6;
    int r = ioctl(tap_fd, TUNSETOFFLOAD, off_flags);
    if (r != 0) {
   	 error(-1, errno, "ioctl(TUNSETOFFLOAD)");
    }

The permitted bit values confirm what the userspace application is able to receive:

TUN_F_CSUM

L4 packet checksum offload

TUN_F_TSO4

TCP Segmentation Offload - TSO for IPv4 packets

TUN_F_TSO6

TSO for IPv6 packets

TUN_F_TSO_ECN

TSO with ECN bits

TUN_F_UFO

UDP Fragmentation offload - UFO packets. Deprecated

TUN_F_USO4

UDP Segmentation offload - USO for IPv4 packets

TUN_F_USO6

USO for IPv6 packets

Offloads denote extra packet features the tap application can handle. Which offloads the sender actually applied is recorded per packet in the vnethdr prefix.

Checksum offload

BLOG-2123 Embedded Image - YcThpe

Typical UDP packet layout for a tap-received packet.

TUN_F_CSUM moves checksum work off the kernel: applications setting this flag declare they can validate checksums. For a UDP IPv4 packet under this offload:

  • vnethdr flags carry VIRTIO_NET_HDR_F_NEEDS_CSUM
  • hdr_len is 42 (14+20+8)
  • csum_start is 34 (14+20)
  • csum_offset is 6 (the UDP checksum sits 6 bytes into L4)

Every further offload depends on checksum offload being supported.

Why TUN_F_CSUM is mandatory in practice

Consider the following code:

s = socket(AF_INET, SOCK_DGRAM)
s.setsockopt(SOL_UDP, UDP_SEGMENT, 1400)
s.sendto(b"x", ("10.0.0.2", 5201))     # Would you expect EIO ?

This snippet produces a packet. Sent at a tap device it fails in a surprising way, returning EIO ("Input/output error"), when the tap was opened without TUN_F_CSUM while the application transmits GSO / UDP_SEGMENT frames. The behavior could arguably be classified as a kernel bug and there is thought about fixing it, but for now anyone using tap should simply set the TUN_F_CSUM bit.

Segmentation offloads and packet trains

Linux lets an application handle many packets in one send/recv call provided they all have the same length (UDP_SEGMENT). Tap exposes the same capability through offloads. TUN_F_TSO4 and TUN_F_TSO6 tell the kernel the tap application can cope with long packet trains — which means it must be prepared for much larger buffers, up to 65507 bytes for IPv4 and 65527 for IPv6. The TSO4/TSO6 bits have long covered TCP; TUN_F_USO4 and TUN_F_USO6 are the more recent additions for UDP. Whenever these offloads are in play, the gso_type field reports the offload type and gso_size holds the segment size inside the GRO packet train. TUN_F_UFO, a UDP fragmentation offload, is deprecated.

TUNSETOFFLOAD declares which offloads the application can handle on the read() side of the tap. If the ioctl succeeds, the application may also assume kernel support for those offloads in the other direction.

A bug to watch for with TUN_F_USO4

ethtool is the tool for inspecting tap offload state:

$ ethtool -k tap0 | egrep -v fixed
tx-checksumming: on
    tx-checksum-ip-generic: on
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
tx-udp-segmentation: on
rx-gro-list: off
rx-udp-gro-forwarding: off

It shows which offloads are enabled and lets you turn them off. While experimenting with USO, packet trains forwarded from a tap to a real network interface were sometimes found to be badly packetized; see the linked netdev discussion and proposed fix. Until that lands, consider ethtool -K tap0 rx-udp-gro-forwarding off.

Socket options worth knowing

Setup

TUNGETFEATURES

Return vector of IFF_* constants that the kernel supports. Typically used to detect the host support of: IFF_VNET_HDR, IFF_NAPI and IFF_MULTI_QUEUE.

TUNSETIFF

Takes "struct ifreq", sets up a tap device, fills in the name if empty.

TUNGETIFF

Returns a "struct ifreq" containing the device's current name and flags.

TUNSETPERSIST

Sets TUN_PERSIST flag, if you want the device to remain in the system after the tap_fd is closed.

TUNSETOWNER, TUNSETGROUP

Set uid and gid that can own the device.

TUNSETLINK

Set the Ethernet link type for the device. The device must be down. See ARPHRD_* constants. For tap it defaults to ARPHRD_ETHER.

TUNSETOFFLOAD

As documented above.

TUNGETSNDBUF, TUNSETSNDBUF

Get/set send buffer. The default is INT_MAX.

TUNGETVNETHDRSZ, TUNSETVNETHDRSZ

Already discussed.

TUNSETIFINDEX

Set interface index (ifindex), useful in checkpoint-restore.

TUNSETCARRIER

Set the carrier state of an interface, as discussed earlier, useful with IFF_NO_CARRIER.

TUNGETDEVNETNS

Return an fd of a net namespace that the interface belongs to.

Filtering

TUNSETTXFILTER

Takes "struct tun_filter" which limits the dst mac addresses that can be delivered to the application.

TUNATTACHFILTER, TUNDETACHFILTER, TUNGETFILTER

Attach/detach/get classic BPF filter for packets going to application. Takes "struct sock_fprog".


TUNSETFILTEREBPF

Set an eBPF filter on a tap device. This is independent of the classic BPF above.

Multi-queue

TUNSETQUEUE

Used to set IFF_DETACH_QUEUE and IFF_ATTACH_QUEUE for multiqueue.

TUNSETSTEERINGEBPF

Set an eBPF program for selecting a specific tap queue, in the direction towards the application. This is useful if you want to ensure some traffic is sticky to a specific application thread. The eBPF program takes "struct __sk_buff" and returns an int. The result queue number is computed from the return value u16 modulo number of queues is the selection.

Scaling: batching does not save a single queue

Tap devices are neither network sockets nor true files; their semantics are closest to pipes, and the API shows it. Each packet requires its own read() or write() syscall. Batching seems like the obvious answer, but sendmmsg()/recvmmsg() do not work on tap file descriptors. The usual substitutes are the old io_submit AIO interface or modern io_uring, which gained tap support fairly recently. In practice syscall batching yields little — perhaps around 10%. The kernel simply cannot forward millions of packets per second for one flow on one CPU. The workable approach is to scale vertically for elephant flows using TSO/USO packet trains and horizontally across concurrent flows using multi-queue.

BLOG-2123 Embedded Image - yzP0N2

The chart makes the offload payoff plain. Without offloads, a sample "echo" tap application manages 320 to 500 thousand packets per second on a single core at an MTU of 1500. With offloads enabled it reaches 2.7Mpps while receiving only 56 thousand packet trains per second. Not every traffic pattern can fully exploit GRO/GSO, but offloads remain absolutely critical for decent tap — and general Linux — networking performance.

Scaling with multi queue

When a tap application handles several concurrent flows and needs more than one CPU, multi queue is the mechanism to reach for. A tap queue file descriptor is obtained by adding the IFF_MULTI_QUEUE flag at open time. Queues can also be detached and reattached through TUNSETQUEUE combined with IFF_DETACH_QUEUE/IFF_ATTACH_QUEUE, although the practical use cases for that are unclear.

Once created, a multi queue tap spreads load across several queues, each with its own unique file descriptor. The queue selection logic deserves close attention, because it can produce surprising behavior.

How the kernel picks a queue

By default, the Linux tap driver keeps a symmetric flow hash for each handled flow in a flow table, recording which queue the application used to transmit that traffic. On the receive side it honors that record and delivers subsequent packets of the flow to the same queue. If the userspace application transmits a TCP flow on queue #2, packets belonging to that flow arriving into the application land on queue #2 as well. This works well as long as the sender consistently selects a single queue. Should the sender switch its TX queue, new packets shift immediately, and packets within one flow may end up seen as reordered. The design also ignores CPU locality, which can cost some performance in very high throughput applications.

Flow-hash based selection can be overridden with the tc multiq qdisc and the skbedit queue_mapping filter:

tc qdisc add dev tap0 root handle 1: multiq
tc filter add dev tap0 parent 1: protocol ip prio 1 u32 \
        match ip dst 192.168.0.3 \
        action skbedit queue_mapping 0

Because tc is fragile, it is not a recommended solution. A better route is a TUNSETSTEERINGEBPF eBPF program that replaces the queue selection algorithm; with it the flow tracking code is no longer used, and a carefully written steering program can keep flow processing local to a single CPU for the best performance.

Performance recommendations

For the best performance:

  • enable vnethdr
  • enable offloads (TSO and USO)
  • consider spreading the load across multiple queues and CPUs with multi queue
  • consider syscall batching for additional gain of maybe 10%, perhaps try io_uring
  • consider customizing the steering algorithm

References