When Linux Runs Out of Ephemeral Ports

Most engineers have assumptions about how systems behave that turn out to be wrong. Often this happens with basic, ubiquitous operations. A recent example from our production environment: a server that suddenly refused SSH connections to localhost while an SSH session to that same server was already active. Another time, a basic DNS query failed with an unusual networking error.

Both failures traced back to the same root cause: Linux had exhausted its ephemeral port range. When that happens, the system cannot establish any new outgoing connections. The failure is typically transient and can be hard to diagnose without knowing what to look for.

These incidents led us to investigate how Linux allocates source ports for outgoing connections. The problem is not just the size of the ephemeral range — it's how the sockets API is used. Some common patterns unnecessarily prevent port reuse, and some connection types are more limited than they appear.

How Source Port Allocation Works

When establishing an outbound connection, an application provides the destination address and port. The operating system must determine the source IP and source port to form a complete 4-tuple for the connection.

The source IP is selected based on routing configuration; you can see what will be chosen with ip route get. The source port comes from the configured ephemeral port range, controlled by two sysctls:

  • ip_local_port_range sets the inclusive low and high bounds for outgoing connection ports
  • ip_local_reserved_ports excludes specific ports that operators need for services

The default ephemeral range contains about 28,000 ports. The question is whether that means you can have at most 28,000 concurrent outgoing connections.

How TCP Handles Connection Reuse

For TCP, connections are identified by a full 4-tuple. In principle, source IP and port pairs can be reused across different destination addresses. With vanilla TCP code — let the kernel choose everything — the system can share a local two-tuple among established connections to different destinations. When the ephemeral range is exhausted against a single destination, you'll see EADDRNOTAVAIL. This is the intended behavior.

Manually selecting source IP

Problems appear when applications must explicitly choose a source IP. At Cloudflare, we need to separate outgoing traffic by product — a CDN origin might want to firewall off WARP traffic, for example. Our applications can't leave source IP selection to the OS because the automatic choice could be wrong.

Instead, our applications call bind() before connect(), a technique we call "bind-before-connect." This pattern looks innocent but has a serious drawback. When bind() is called, the kernel reserves an unused local two-tuple — it can't know the socket will eventually be used for connect() rather than listen(). Sharing the source pair with a connected socket would break listening sockets. As a result, each connection locks a source port permanently for the lifetime of the connection, no matter how many destination addresses are in use.

The IP_BIND_ADDRESS_NO_PORT solution

Linux introduced a proper fix for this in 2015: the IP_BIND_ADDRESS_NO_PORT socket option. Setting this option tells the kernel to delay reserving the source port until connect() is called. This restores the desired behavior — source two-tuples can be reused across distinct destinations — while still allowing explicit source IP selection. On modern Linux, any bind-before-connect TCP code should set this option.

Explicit source port selection

Some debugging scenarios require controlling the full 4-tuple. For example, cURL's --local-port option or troubleshooting ECMP routing. In those cases, IP_BIND_ADDRESS_NO_PORT isn't appropriate. To enable source two-tuple sharing with a manually selected port, you must set SO_REUSEADDR:

  • Bind to a specific source address and port
  • Set SO_REUSEADDR before binding
  • Connect to the destination
  • If connect() fails with EADDRNOTAVAIL, retry with a different source port

The application takes responsibility for handling conflicts when an established socket sharing the 4-tuple already exists.

A Userspace connectx() Implementation

These tricks allow implementing a common function, which we call connectx(). It performs what bind() + connect() should ideally do — create sockets that share local two-tuples as long as they go to distinct destinations — without tripping over ephemeral range limits. The API should support three use cases:

  • Vanilla egress: the OS selects both outgoing IP and port
  • Source IP selection: the user chooses IP, the OS picks the port
  • Full 4-tuple control: the user supplies everything

The name is intentional; macOS's Darwin has a real connectx() syscall, which is more powerful than our version since it supports TCP Fast Open. For TCP on Linux, the right set of socket options and syscall ordering achieves the same outcome. For UDP, however, the situation is considerably more difficult.

UDP Presents a Harder Problem

UDP sockets on Linux are more limited than TCP in surprising ways. By default, the total number of outbound UDP connections is constrained by the ephemeral port range, even if connections point to many destinations. During connect(), Linux seeks a source two-tuple that isn't in use — it doesn't consider whether a 4-tuple could be shared. There's no fundamental reason for this; it's simply suboptimal behavior.

Setting SO_REUSEADDR on UDP sockets also allows established sockets to share identical 4-tuples. This creates an "overshadowing" problem: with conflicting sockets, only one receives traffic, and a newer connected socket silently shadows an older one. Without producing any error, the old socket becomes dead. That's unacceptable for production use.

Detecting Socket Conflicts

Because conflict detection is essential for safe UDP port sharing, we explored several approaches.

The eBPF approach

Linux cgroups offer a BPF_CGROUP_INET4_CONNECT hook. An eBPF program attached at that point runs on every connect() syscall from processes in the cgroup. Our idea was to check for a 4-tuple conflict before the socket transitions from UNCONNECTED to CONNECTED states.

This approach has real limitations. It only works when a user supplies a full 4-tuple — not for automatic source IP or port assignment. More fundamentally, it's racy: no lock is held between the eBPF conflict check and the kernel's connect() machinery on another CPU. A conflicting socket could be created in that window.

Netlink socket traversal

Another approach is checking for conflicting connected sockets in userspace. The SO_DIAG_BY_FAMILY netlink interface (the same one ss uses) allows looking up sockets quickly via the kernel's __udp_lookup() routine — no iteration over all sockets needed. The code is simple, but shares the same race condition as the eBPF hook.

Using SO_REUSEADDR as a lock

We found a way to avoid the race: treat SO_REUSEADDR as a locking mechanism. The scheme:

  1. Set SO_REUSEADDR to allow port reuse, then bind() to the desired 4-tuple components
  2. Clear SO_REUSEADDR after binding — this blocks new sockets from claiming that source port
  3. Verify ownership of the 4-tuple within this critical section
  4. Perform connect(), knowing no socket can grab the same 4-tuple meanwhile

This is a cooperative algorithm — it assumes all tenants on the system use the same protocol. It's convoluted and relies on undocumented kernel behavior, but it eliminates the race condition.

The catch: this only works when the full 4-tuple is known in advance. When an application only names a destination, the kernel needs to fill in the source IP and port. That requires implementing source discovery in userspace — using routing table lookups and probing for available ports. It's not trivial, but it's achievable.

What We Learned

This investigation clarified why we were hitting ephemeral port exhaustion during normal operations. The immediate cause would appear to be "too many concurrent connections," but that framing was inaccurate. Insufficient reuse of source ports was the real problem, combined with bind-before-connect patterns that locked ports for each connection's lifetime.

To summarize, what's needed to avoid the ~28,000 connection per-protocol limit:

  • For TCP with explicit source IP: set IP_BIND_ADDRESS_NO_PORT before binding
  • For TCP with a full 4-tuple: use SO_REUSEADDR and handle EADDRNOTAVAIL retries
  • For UDP with a full 4-tuple: use the SO_REUSEADDR toggling trick to prevent overshadowing
  • For UDP with automatic source discovery: implement the full algorithm in userspace

The connectx() functionality should exist natively in Linux. The BSD API doesn't express the intent of "connect this socket to a specific destination" cleanly. Getting all the use cases right in userspace is involved — but it demonstrates that a well-designed approach could make this far simpler with proper kernel support.