Why connect() Slows Down When Ports Run Low

Cloudflare has been pushing for broader IPv6 adoption, partly to reduce reliance on increasingly expensive IPv4 addresses. In the course of that effort, we examined our cache service, one of the larger internal consumers of IPv4 addresses. Cache misses can spike when content goes viral, leading to bursts of TCP connections—potentially 50k unicast connections to a single destination. Currently, these are balanced across two source IPv4 addresses. Reducing that to a single address would cut IPv4 usage, but first we needed to understand the performance implications.

Testing One Source Address vs. Two

Using a modified version of the wrk benchmark tool to spread connections across source IPs, we ran 70k connections over 48 threads on an idle machine. We measured latency in tcp_v4_connect() with the BPF tool funclatency. Results showed a bimodal distribution in both cases.

Figure 2: A bimodal chart, given two IPv4 source addresses, depicting 20 thousand connections that were slow to find a port, and more than roughly 50 thousand connections that were fast to find a port.

With two source addresses, the majority of connections completed quickly, but roughly 20k fell into slower buckets. With a single source address, the pattern became more extreme:

Figure 3: A bimodal chart, given one IPv4 source address, depicting a split of roughly 25 thousand connections in the fast case, and over 40 thousand connections in the slow case.

Over half the connections fell into the slow case. The implication was clear: moving to one IPv4 address would add noticeable latency to connect() calls. To fix that, we needed to locate the bottleneck.

Flame Graph Points to Port Selection

Profiling a production machine with a flame graph revealed that most samples landed in __inet_hash_connect(), the kernel function responsible for picking a source port during a late bind. Many samples also appeared in __inet_check_established(), with lock contention in between. To get cleaner data, we built a homegrown benchmark that removed lock contention and fixed the connection count.

Figure 5: A detailed chart representing the bimodal distribution of port-finding speeds. Out of 56,512 connections the average time spent finding an even port is 0.025 milliseconds while we see 4.59 milliseconds on average for odd ports.

The latency scatter plot exposed an odd/even split. Green dots (even ports) were consistently fast, while red dots (odd ports) were dramatically slower and appeared only after even ports were exhausted. This distribution explains the bimodal behavior seen earlier: the algorithm cycles through all even ports before touching odd ones.

Inside __inet_hash_connect()

The port-selection algorithm is a variant of the Double-Hash Port Selection Algorithm from RFC 6056. Linux computes a time-based hash, adds randomness, and stores the result in an offset that is always even.

   offset &= ~1U;
    
other_parity_scan:
    port = low + offset;
    for (i = 0; i < remaining; i += 2, port += 2) {
        if (unlikely(port >= high))
            port -= remaining;

        inet_bind_bucket_for_each(tb, &head->chain) {
            if (inet_bind_bucket_match(tb, net, port, l3mdev)) {
                if (!check_established(death_row, sk, port, &tw))
                    goto ok;
                goto next_port;
            }
        }
    }

    offset++;
    if ((offset & 1) && remaining > 1)
        goto other_parity_scan;

The starting port is the configured low port plus that offset:

port = low + offset;

If the low port is even, the start is even; if odd, the start is odd. The loop then walks through every other port—first all even, then all odd (or vice versa). For each candidate, it calls __inet_check_established() to verify the TCP 4-tuple is unique. The socket list checked there grows as more unique tuples accumulate, which can further slow selection.

Figure 6: A step-by-step diagram showing how the function __inet_has_connect() finds a port for connections preferring late port-binding.

This diagram shows an 8-port example. Green marks the chosen port, arrows trace the progression through even and then odd ports, and the offset increments when crossing over. For workloads staying within half the port range, the algorithm is efficient; beyond that, the second half introduces the latency penalty.

Why the Split Exists

The even/odd split was added for security and compatibility reasons. Port selection has historically been used for device fingerprinting, so randomization was introduced via the offset. The split itself dates back to patches designed to prevent conflicts between connect()-heavy and bind()-heavy workloads, assigning even offsets to the former and odd to the latter. That design works well—until a connect() workload exceeds half the available port range.

Understanding this root cause is the first step. Next, we need a strategy to mitigate the penalty when ports run low—ideally one that lets us consolidate onto a single source IPv4 address without sacrificing connect() performance.

Mitigating slow connect() in user space

For kernels older than 6.8, there are two practical strategies that avoid large-scale architectural changes by addressing the problem directly in application code.

Select, test, repeat

The simplest approach is to loop through the system port range, randomly picking a candidate port each iteration and testing whether connect() succeeds:

sys = get_ip_local_port_range()
estab = 0
i = sys.hi
while i >= 0:
    if estab >= sys.hi:
        break

    random_port = random.randint(sys.lo, sys.hi)
    connection = attempt_connect(random_port)
    if connection is None:
        i += 1
        continue

    i -= 1
    estab += 1

This method works well up to roughly 70-80% port range utilization. As the range approaches exhaustion, you can expect eight to twelve attempts per successful connection. The primary drawback is the extra syscall overhead incurred on each conflict.

Randomly shifted port ranges

A more efficient approach leverages the IP_LOCAL_PORT_RANGE socket option, yielding significantly better performance:

Figure 7: A detailed chart representing a flatter-linear distribution of port-finding speeds. Out of 56,512 connections the average time spent finding an even port is 0.03 milliseconds while we see 0.031 milliseconds on average for odd ports. There are 868 connections that resulted in an error.

The error connections (black dots) tend to cluster at the end of the port range as exhaustion approaches, similar to the select-test-repeat pattern. The mechanism works as follows:

IP_BIND_ADDRESS_NO_PORT = 24
IP_LOCAL_PORT_RANGE = 51
sys = get_local_port_range()
window.lo = 0
window.hi = 1000
range = window.hi - window.lo
offset = randint(sys.lo, sys.hi - range)
window.lo = offset
window.hi = offset + range

sk = socket(AF_INET, SOCK_STREAM)
sk.setsockopt(IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, 1)
range = pack("@I", window.lo | (window.hi << 16))
sk.setsockopt(IPPROTO_IP, IP_LOCAL_PORT_RANGE, range)
sk.bind((src_ip, 0))
sk.connect((dest_ip, dest_port))

First, fetch the system's local port range. Then define a smaller custom window and randomly shift it within the system range. This randomization causes the kernel to begin port selection at a random even or odd port, and limits the search to the custom window rather than the entire range.

Testing with various window sizes revealed that 500 or 1000 ports work well:

Window size Errors Total test time Connections/second
500 868 ~1.8 seconds ~30,139
1,000 1,129 ~2 seconds ~27,260
5,000 4,037 ~6.7 seconds ~8,405
10,000 6,695 ~17.7 seconds ~3,183

Larger windows perform worse because they offer less random offset opportunity—a maximum window size of 56,512 is effectively indistinguishable from default kernel behavior. Conversely, too-small windows cause problems too; a window size of one is equivalent to the select-test-repeat approach.

Kernel-native solution (6.8+)

A patch scheduled for the 6.8 kernel eliminates the need for window shifting. Instead of passing a random sub-range to setsockopt(IPPROTO_IP, IP_LOCAL_PORT_RANGE, …), you simply provide the full system port range:

IP_BIND_ADDRESS_NO_PORT = 24
IP_LOCAL_PORT_RANGE = 51
sys = get_local_port_range()
sk = socket(AF_INET, SOCK_STREAM)
sk.setsockopt(IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, 1)
range = pack("@I", sys.lo | (sys.hi << 16))
sk.setsockopt(IPPROTO_IP, IP_LOCAL_PORT_RANGE, range)
sk.bind((src_ip, 0))
sk.connect((dest_ip, dest_port))

Setting the IP_LOCAL_PORT_RANGE option tells the kernel to randomize the starting offset as even or odd, then iterate incrementally rather than skipping alternate ports:

Figure 8: A detailed chart representing a flat-linear distribution of port-finding speeds. Out of 56,512 connections the average time spent finding an even port is 0.029 milliseconds while we see 0.029 milliseconds on average for odd ports. There are no errors.

This kernel-based approach performs comparably to the user-space implementation, with a slight edge coming from the ability to always search the full port space. No cycles are wasted on a potentially filled sub-range.

Other protocols and their quirks

IPv4 and IPv6 use largely the same algorithms, with differences mainly in how socket uniqueness is compared and where the port search occurs. Several other protocols have notable behaviors.

DCCP

DCCP shares the same port selection algorithm as TCP, so it inherits the kernel improvements. It might also benefit from the user-space approach, but that remains untested.

UDP and UDP-Lite

UDP uses a different mechanism in udp_lib_get_port(). When no port is pre-specified in bind(), the algorithm loops over the entire port range, but with one key difference: a random step variable determines the stride between successive candidates. The search relies on uint16_t overflow to eventually wrap back to the starting port. If all ports are exhausted, the starting port increments by one and the process repeats. There is no even/odd port splitting.

A representative UDP measurement setup looks like:

sk = socket(AF_INET, SOCK_DGRAM)
sk.bind((src_ip, 0))
sk.connect((dest_ip, dest_port))

With a single IPv4 source address, the results are unsurprising:

Figure 9: A detailed chart representing a flat-linear distribution of port-finding speeds. Out of 56,512 connections the average time spent finding an even port is 0.001 milliseconds while we see 0.001 milliseconds on average for odd ports. There are no errors.

UDP behaves fundamentally differently from TCP, with less work required for port lookups overall. The chart's outliers reflect worst-case scenarios where a particularly poor random number collision forces a more complete sweep of the ephemeral range.

UDP introduces another complication: with SO_REUSEADDR set, the function udp_lib_lport_inuse() skips the 2-tuple (source IP, source port) uniqueness check for UDP sockets. This can result in a new socket silently overwriting a previous one—a hazard that deserves careful attention.

Final observations

When balancing load over several IPv4 source addresses during peak times to avoid port exhaustion, the question naturally arises: what does connect() performance look like for heavily connection-bound workloads on a single source address? Measuring connect() latency through flame graphs and synthetic tests revealed that port selection is not just a correctness concern but a genuine performance bottleneck. Specifically, TCP's port selection loops across half the ephemeral range before considering the other half on each connect() call.

Aside from adding more IP addresses or other architectural changes, three mitigations stand out: the select-test-repeat pattern, randomly shifted port ranges via IP_LOCAL_PORT_RANGE, and the cleaner kernel-native option in 6.8+. Results will vary with your particular workloads, so it is worth measuring your own systems to determine the most appropriate strategy.