Performance work on virtualized networking quickly runs into the tap device. For projects built on Firecracker, the KVM-based VMM that runs lightweight Micro-VMs, every instance talks to the host through a tap device. The API itself is small, but the semantics and tuning knobs are scattered and poorly documented. What follows are the notes worth having up front.
What a tap device actually is
A tap is a virtual network interface that presents itself like an Ethernet card. Rather than terminating in physical wiring, it exposes a file descriptor to an application that sends and receives packets. VPN clients were the historical consumers: traffic gets routed toward the tap interface, and the client picks packets up from the descriptor and processes them. More recently, VMMs such as Qemu, Firecracker and gVisor open the application side of the tap and hand packets to the guest, leaving the interface itself for the host kernel to route, forward or NAT as usual. That inverts the original pattern — during the VPN era the tap was a traffic destination, whereas behind a VM it looks like a traffic source.
Despite appearing trivial, tap devices are hard to make fast, because the Linux networking stack is tuned for packets handled by physical NICs rather than by a userspace application. The interface has grown enough features over the years that good performance is achievable; the rest of this article covers how to use the modern API.

Source: DALL-E
Choosing the variant
In the kernel the interface is called "the universal tun/tap". The tun variant, selected with IFF_TUN, behaves like a point-to-point link and carries no L2 Ethernet headers — awkward to configure for anyone used to modern Ethernet networks, and unsuitable for Firecracker or gVisor, both of which expect L2 headers. The tap variant, selected with IFF_TAP, does have Ethernet headers and has attracted most of the recent attention.
Creating and inspecting devices
iproute2 handles tun/tap natively: ip tuntap add creates a device and ip tuntap list enumerates them.
$ sudo ip tuntap add mode tap user marek group marek name tap0
$ ip tuntap list
tap0: tap persist user 1000 group 1000
The device flags are also visible via /sys/devices/virtual/net/<ifr_name>/tun_flags.
Opening the clone device
Everything starts by opening /dev/net/tun, the "clone device":
/* First, whatever you do, the device /dev/net/tun must be
* opened read/write. That device is also called the clone
* device, because it's used as a starting point for the
* creation of any tun/tap virtual interface. */
char *clone_dev_name = "/dev/net/tun";
int tap_fd = open(clone_dev_name, O_RDWR | O_CLOEXEC);
if (tap_fd < 0) {
error(-1, errno, "open(%s)", clone_dev_name);
}
That descriptor is then used to instantiate a specific device by name:
struct ifreq ifr = {};
strncpy(ifr.ifr_name, tap_name, IFNAMSIZ);
ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_VNET_HDR;
int r = ioctl(tap_fd, TUNSETIFF, &ifr);
if (r != 0) {
error(-1, errno, "ioctl(TUNSETIFF)");
}
An empty ifr_name, or a name that does not exist yet, creates a new tap device; otherwise the existing device is opened. When reopening, flags such as IFF_MULTI_QUEUE must match how the device was originally created, or the call fails with EINVAL. A practical pattern is to retry with the multi-queue setting flipped when EINVAL comes back.
Available ifr_flags bits:
IFF_TAP / IFF_TUN | Already discussed. |
IFF_NO_CARRIER | Holding an open tap device file descriptor sets the Ethernet interface CARRIER flag up. In some cases it might be desired to delay that until a TUNSETCARRIER call. |
IFF_NO_PI | Historically each packet on tap had a "struct tun_pi" 4 byte prefix. There are now better alternatives and this option disables this prefix. |
IFF_TUN_EXCL | Ensures a new device is created. Returns EBUSY if the device exists |
IFF_VNET_HDR | Prepend "struct virtio_net_hdr" before the RX and TX packets, should be followed by setsockopt(TUNSETVNETHDRSZ). |
IFF_MULTI_QUEUE | Use multi queue tap, see below. |
IFF_NAPI / IFF_NAPI_FRAGS | See below. |
In practice you almost always want IFF_TAP, IFF_NO_PI and IFF_VNET_HDR, and sometimes IFF_MULTI_QUEUE.
IFF_NAPI and IFF_NAPI_FRAGS
According to the original patchset, IFF_NAPI and IFF_NAPI_FRAGS existed primarily to improve syzkaller's code coverage. Later work found that IFF_NAPI also brings performance benefits when running XDP on tap. It allocates a dedicated NAPI instance for packets an application writes into the tap, which permits both XDP and batching with GRO; otherwise the kernel falls back to a backlog NAPI.
Queue limits in both directions
Internally a tap is a pair of packet queues: exposed as a network interface toward the host and as a character device file descriptor toward the application. The queue running toward the application (the tap TX queue) holds txqueuelen packets, set through an interface parameter:
$ ip link set dev tap0 txqueuelen 1000
$ ip -s link show dev tap0
26: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ... qlen 1000
RX: bytes packets errors dropped missed mcast
0 0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
266 3 0 66 0 0
In ip link statistics, the "TX dropped" column means the tap application was too slow and the queue filled up. In the opposite direction — from application to host, the interface RX queue — the limit is expressed in bytes and set with the TUNSETSNDBUF ioctl. Qemu's source comments on this setting; overflowing this queue is not easy to provoke.
VNET_HDR and virtio header layout
Once the device is open, the usual next step is configuring the VNET_HDR size and offloads. The conventional value for VNETHDRSZ is 12:
len = 12;
r = ioctl(tap_fd, TUNSETVNETHDRSZ, &(int){len});
if (r != 0) {
error(-1, errno, "ioctl(TUNSETVNETHDRSZ)");
}
The sensible set is {10, 12, 20}, all taken from the virtio spec. Twelve bytes leave room for this little-endian header:
struct virtio_net_hdr_v1 {
#define VIRTIO_NET_HDR_F_NEEDS_CSUM 1 /* Use csum_start, csum_offset */
#define VIRTIO_NET_HDR_F_DATA_VALID 2 /* Csum is valid */
u8 flags;
#define VIRTIO_NET_HDR_GSO_NONE 0 /* Not a GSO frame */
#define VIRTIO_NET_HDR_GSO_TCPV4 1 /* GSO frame, IPv4 TCP (TSO) */
#define VIRTIO_NET_HDR_GSO_UDP 3 /* GSO frame, IPv4 UDP (UFO) */
#define VIRTIO_NET_HDR_GSO_TCPV6 4 /* GSO frame, IPv6 TCP */
#define VIRTIO_NET_HDR_GSO_UDP_L4 5 /* GSO frame, IPv4& IPv6 UDP (USO) */
#define VIRTIO_NET_HDR_GSO_ECN 0x80 /* TCP has ECN set */
u8 gso_type;
u16 hdr_len; /* Ethernet + IP + tcp/udp hdrs */
u16 gso_size; /* Bytes to append to hdr_len per frame */
u16 csum_start;
u16 csum_offset;
u16 num_buffers;
};
Advertising offload support
Offloads are enabled with an ioctl:
unsigned off_flags = TUN_F_CSUM | TUN_F_TSO4 | TUN_F_TSO6;
int r = ioctl(tap_fd, TUNSETOFFLOAD, off_flags);
if (r != 0) {
error(-1, errno, "ioctl(TUNSETOFFLOAD)");
}
The permitted bit values confirm what the userspace application is able to receive:
TUN_F_CSUM | L4 packet checksum offload |
TUN_F_TSO4 | TCP Segmentation Offload - TSO for IPv4 packets |
TUN_F_TSO6 | TSO for IPv6 packets |
TUN_F_TSO_ECN | TSO with ECN bits |
TUN_F_UFO | UDP Fragmentation offload - UFO packets. Deprecated |
TUN_F_USO4 | UDP Segmentation offload - USO for IPv4 packets |
TUN_F_USO6 | USO for IPv6 packets |
Offloads denote extra packet features the tap application can handle. Which offloads the sender actually applied is recorded per packet in the vnethdr prefix.
Checksum offload

Typical UDP packet layout for a tap-received packet.
TUN_F_CSUM moves checksum work off the kernel: applications setting this flag declare they can validate checksums. For a UDP IPv4 packet under this offload:
- vnethdr flags carry
VIRTIO_NET_HDR_F_NEEDS_CSUM - hdr_len is 42 (14+20+8)
- csum_start is 34 (14+20)
- csum_offset is 6 (the UDP checksum sits 6 bytes into L4)
Every further offload depends on checksum offload being supported.
Why TUN_F_CSUM is mandatory in practice
Consider the following code:
s = socket(AF_INET, SOCK_DGRAM)
s.setsockopt(SOL_UDP, UDP_SEGMENT, 1400)
s.sendto(b"x", ("10.0.0.2", 5201)) # Would you expect EIO ?
This snippet produces a packet. Sent at a tap device it fails in a surprising way, returning EIO ("Input/output error"), when the tap was opened without TUN_F_CSUM while the application transmits GSO / UDP_SEGMENT frames. The behavior could arguably be classified as a kernel bug and there is thought about fixing it, but for now anyone using tap should simply set the TUN_F_CSUM bit.
Segmentation offloads and packet trains
Linux lets an application handle many packets in one send/recv call provided they all have the same length (UDP_SEGMENT). Tap exposes the same capability through offloads. TUN_F_TSO4 and TUN_F_TSO6 tell the kernel the tap application can cope with long packet trains — which means it must be prepared for much larger buffers, up to 65507 bytes for IPv4 and 65527 for IPv6. The TSO4/TSO6 bits have long covered TCP; TUN_F_USO4 and TUN_F_USO6 are the more recent additions for UDP. Whenever these offloads are in play, the gso_type field reports the offload type and gso_size holds the segment size inside the GRO packet train. TUN_F_UFO, a UDP fragmentation offload, is deprecated.
TUNSETOFFLOAD declares which offloads the application can handle on the read() side of the tap. If the ioctl succeeds, the application may also assume kernel support for those offloads in the other direction.
A bug to watch for with TUN_F_USO4
ethtool is the tool for inspecting tap offload state:
$ ethtool -k tap0 | egrep -v fixed
tx-checksumming: on
tx-checksum-ip-generic: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
tx-udp-segmentation: on
rx-gro-list: off
rx-udp-gro-forwarding: off
It shows which offloads are enabled and lets you turn them off. While experimenting with USO, packet trains forwarded from a tap to a real network interface were sometimes found to be badly packetized; see the linked netdev discussion and proposed fix. Until that lands, consider ethtool -K tap0 rx-udp-gro-forwarding off.
Socket options worth knowing
Setup
TUNGETFEATURES | Return vector of IFF_* constants that the kernel supports. Typically used to detect the host support of: IFF_VNET_HDR, IFF_NAPI and IFF_MULTI_QUEUE. |
TUNSETIFF | Takes "struct ifreq", sets up a tap device, fills in the name if empty. |
TUNGETIFF | Returns a "struct ifreq" containing the device's current name and flags. |
TUNSETPERSIST | Sets TUN_PERSIST flag, if you want the device to remain in the system after the tap_fd is closed. |
TUNSETOWNER, TUNSETGROUP | Set uid and gid that can own the device. |
TUNSETLINK | Set the Ethernet link type for the device. The device must be down. See ARPHRD_* constants. For tap it defaults to ARPHRD_ETHER. |
TUNSETOFFLOAD | As documented above. |
TUNGETSNDBUF, TUNSETSNDBUF | Get/set send buffer. The default is INT_MAX. |
TUNGETVNETHDRSZ, TUNSETVNETHDRSZ | Already discussed. |
TUNSETIFINDEX | Set interface index (ifindex), useful in checkpoint-restore. |
TUNSETCARRIER | Set the carrier state of an interface, as discussed earlier, useful with IFF_NO_CARRIER. |
TUNGETDEVNETNS | Return an fd of a net namespace that the interface belongs to. |
Filtering
TUNSETTXFILTER | Takes "struct tun_filter" which limits the dst mac addresses that can be delivered to the application. |
TUNATTACHFILTER, TUNDETACHFILTER, TUNGETFILTER | Attach/detach/get classic BPF filter for packets going to application. Takes "struct sock_fprog". |
TUNSETFILTEREBPF | Set an eBPF filter on a tap device. This is independent of the classic BPF above. |
Multi-queue
TUNSETQUEUE | Used to set IFF_DETACH_QUEUE and IFF_ATTACH_QUEUE for multiqueue. |
TUNSETSTEERINGEBPF | Set an eBPF program for selecting a specific tap queue, in the direction towards the application. This is useful if you want to ensure some traffic is sticky to a specific application thread. The eBPF program takes "struct __sk_buff" and returns an int. The result queue number is computed from the return value u16 modulo number of queues is the selection. |
Scaling: batching does not save a single queue
Tap devices are neither network sockets nor true files; their semantics are closest to pipes, and the API shows it. Each packet requires its own read() or write() syscall. Batching seems like the obvious answer, but sendmmsg()/recvmmsg() do not work on tap file descriptors. The usual substitutes are the old io_submit AIO interface or modern io_uring, which gained tap support fairly recently. In practice syscall batching yields little — perhaps around 10%. The kernel simply cannot forward millions of packets per second for one flow on one CPU. The workable approach is to scale vertically for elephant flows using TSO/USO packet trains and horizontally across concurrent flows using multi-queue.

The chart makes the offload payoff plain. Without offloads, a sample "echo" tap application manages 320 to 500 thousand packets per second on a single core at an MTU of 1500. With offloads enabled it reaches 2.7Mpps while receiving only 56 thousand packet trains per second. Not every traffic pattern can fully exploit GRO/GSO, but offloads remain absolutely critical for decent tap — and general Linux — networking performance.
Scaling with multi queue
When a tap application handles several concurrent flows and needs more than one CPU, multi queue is the mechanism to reach for. A tap queue file descriptor is obtained by adding the IFF_MULTI_QUEUE flag at open time. Queues can also be detached and reattached through TUNSETQUEUE combined with IFF_DETACH_QUEUE/IFF_ATTACH_QUEUE, although the practical use cases for that are unclear.
Once created, a multi queue tap spreads load across several queues, each with its own unique file descriptor. The queue selection logic deserves close attention, because it can produce surprising behavior.
How the kernel picks a queue
By default, the Linux tap driver keeps a symmetric flow hash for each handled flow in a flow table, recording which queue the application used to transmit that traffic. On the receive side it honors that record and delivers subsequent packets of the flow to the same queue. If the userspace application transmits a TCP flow on queue #2, packets belonging to that flow arriving into the application land on queue #2 as well. This works well as long as the sender consistently selects a single queue. Should the sender switch its TX queue, new packets shift immediately, and packets within one flow may end up seen as reordered. The design also ignores CPU locality, which can cost some performance in very high throughput applications.
Flow-hash based selection can be overridden with the tc multiq qdisc and the skbedit queue_mapping filter:
tc qdisc add dev tap0 root handle 1: multiq
tc filter add dev tap0 parent 1: protocol ip prio 1 u32 \
match ip dst 192.168.0.3 \
action skbedit queue_mapping 0
Because tc is fragile, it is not a recommended solution. A better route is a TUNSETSTEERINGEBPF eBPF program that replaces the queue selection algorithm; with it the flow tracking code is no longer used, and a carefully written steering program can keep flow processing local to a single CPU for the best performance.
Performance recommendations
For the best performance:
- enable vnethdr
- enable offloads (TSO and USO)
- consider spreading the load across multiple queues and CPUs with multi queue
- consider syscall batching for additional gain of maybe 10%, perhaps try io_uring
- consider customizing the steering algorithm



