Day 41: What a Bridge Actually Does

Yesterday I spent most of the day trying to understand what a network bridge is. I’m only looking at bridges in the context of container and VM networking, because that’s the use case I’m working on — and honestly it’s the only context where I’ve ever seen a bridge used. Some of what follows is almost certainly wrong.

My motivation: I’ve been setting up VMs with AWS Firecracker (which is genuinely amazing — sub-second VM startup). I wanted the VM to reach the outside internet, but every guide I found mentioned “creating a bridge” and none of the copy-pasted snippets worked. So I decided to actually learn what a bridge is instead of blindly pasting. I’m using Docker examples throughout because it’s already set up on my machine and it does roughly what I want (though it uses veth pairs instead of tap devices).

What a Bridge Is

A bridge is a layer 2 (Ethernet) device. You can attach an arbitrary number of network interfaces to it, and it forwards Ethernet frames between them. My mental model is now:

  1. You send a packet to the bridge.
  2. If the bridge has an attached interface whose MAC address matches the destination MAC on your packet, it forwards the packet to that interface.
  3. If there’s no match (or it’s a broadcast), it floods the packet to all attached interfaces.

Tap devices and veth pairs are also layer 2 interfaces. They don’t care about IP addresses — they just pass frames along, and something else is responsible for setting the right MAC addresses.

A key question: if packets need the correct destination MAC to traverse the bridge, how does the host know which MAC to use? The answer is ARP, just like on a physical network. The host broadcasts “who has 172.17.0.8?” and the container replies with its MAC address. It’s a bit funny because all of this lives on the same machine — the information is already there — but the system just pretends it’s a physical network and goes through the discovery dance anyway.

The Two Route Table Entries That Matter

The most important piece of bridge setup is getting the routing tables right. You need two routes:

Route entry 1: on the host. Everything on the bridge’s subnet must be routed to the bridge interface. Docker’s default setup looks like this:

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

Route entry 2: inside the container or VM. The system’s default gateway must point to the bridge. In a Docker container you’ll see:

$ ip route list
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2

The line default via 172.17.0.1 dev eth0 means any packet leaving the container’s subnet goes to 172.17.0.1 — which is the docker0 bridge (technically a veth pair leading to the bridge).

SNAT and the Missing Third Piece

Routes alone don’t give you internet access. To let containers or VMs reach the wider internet, you also need an SNAT rule via iptables on the host. Docker’s default rule is:

$ sudo iptables-save
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

Putting It Together for a Firecracker VM

I did manage to get internet access from inside my Firecracker VM — I’m reusing the docker0 bridge for now, which works but isn’t a long-term plan. The full recipe for a Docker-like setup where you can SSH into a VM and the VM can reach the outside is:

  1. Create the VM’s network interface — either a tap device or a veth pair — with ip tuntap add dev "$TAP_DEV" mode tap.
  2. Attach it to the bridge with sudo brctl addif docker0 $TAP_DEV.
  3. Bring the interface up with ip link set dev "$TAP_DEV" up.
  4. Set the VM’s gateway to the bridge IP, passed via kernel boot args.
  5. Configure the host route table so the bridge’s subnet is routed to the bridge (normally done when creating the bridge — I skipped this since I reused Docker’s).
  6. Add the SNAT iptables rule on the host.
  7. Set nameserver 8.8.8.8 in /etc/resolv.conf inside the VM, since there’s no local resolver.

Here’s a snippet from my Firecracker startup script, adapted from the firecracker-demo code:

CONTAINER_IP=172.17.0.33
GATEWAY_IP=172.17.0.1
DOCKER_MASK_LONG=255.255.255.0
ip tuntap add dev "$TAP_DEV" mode tap
sudo brctl addif docker0 $TAP_DEV
ip link set dev "$TAP_DEV" up
# I'm not sure what these two are for exactly
sysctl -w net.ipv4.conf.${TAP_DEV}.proxy_arp=1 > /dev/null
sysctl -w net.ipv6.conf.${TAP_DEV}.disable_ipv6=1 > /dev/null

I also had to set the VM’s gateway via kernel boot args: ip=${CONTAINER_IP}::${GATEWAY_IP}:${DOCKER_MASK_LONG}::eth0:off.

I’m still not fully confident about bridges, but I can SSH into a Firecracker VM and browse the internet from inside it, which feels like a win. I’ll probably post my full script once I’ve cleaned it up and stopped borrowing Docker’s bridge.