Boosting Dropbox upload speed and improving Windows’ TCP stack

An Unlikely Bottleneck: Windows TCP and NIC Firmware

Collaborative troubleshooting sessions with customers often reveal performance issues that are hard to spot in isolation. In one such session, we uncovered a significant discrepancy: upload speeds on Windows were consistently lower than on Linux or macOS, even after network peering was optimized. While latencies improved dramatically, Windows uploads remained stuck at a few hundred megabits per host—a fraction of the available bandwidth on the 10G connections used by our more demanding customers.

The root cause turned out to be an interaction between the Windows TCP stack and the firmware on the Network Interface Controllers (NICs) used on our Edge servers. The fix, which Microsoft has since rolled out, improves Windows' implementation of the TCP RACK-TLP algorithm and its resilience to packet reordering. It is available starting with Windows 10 build 21332 via the Windows Insider Program dev channel.

Why Upload Speeds Matter

Dropbox is widely used by creative studios, particularly in video and game production. These workflows often span offices in different time zones, relying on continuous uploads and downloads of very large files. The hardware ranges from dedicated servers with 10G internet connections to thousands of MacBooks on standard Wi-Fi.

The Dropbox Desktop Client is designed to be simple on the surface, but beneath that UI lies multi-threaded compression, chunking, and hashing, all written in Rust, backed by HTTP/2 and TLS. Balancing performance across heterogeneous hardware is tricky:

• Aggressive compression can starve upload threads on 10G servers, but pulling back wastes bandwidth on slower connections.
• More hashing threads speed up server uploads but can overheat laptops with high CPU and memory usage.
• Optimizing hashing code for speed can cause high I/O wait on spinning disks.

With so many competing scenarios, we aim for the client to adapt automatically rather than exposing a large number of tunables.

Isolating the Problem to Windows

During routine performance troubleshooting, our network engineers established direct peering with the customer, which significantly cut latency. Yet uploads from Windows hosts remained at a few hundred megabits per second each—too slow for workflows that move massive files continuously.

We quickly noticed that only Windows users were affected; macOS and Linux could saturate the network. To dig deeper, we set up a controlled test environment in a cloud provider and started investigating Windows networking from an application-layer perspective. Using API Monitor, a Windows analog of strace, we first suspected the application might be "app limited"—not holding enough data in the socket buffer. That was not the case. Instead, the application was waiting on I/O completion port (IOCP) events from the kernel, which usually points to TCP-level bottlenecks.

Wireshark confirmed long periods of upload inactivity, with the client waiting for ACKs while keeping only ~200 KB of data in flight. The tcptrace time-sequence graph showed that bytes in flight were not constrained by the receive window, indicating the sending side's TCP stack was preventing its congestion window from growing.

Digging Deeper with Microsoft's Tools

At this point, we engaged the Windows Core TCP team. The first suggestion was to move from our UNIX-style packet dumps to a Windows-native tracing approach:

> netsh trace start provider=Microsoft-Windows-TCPIP capture=yes packettruncatebytes=120 tracefile=net.etl report=disabled perf=no

The resulting ETL file can be converted to pcap using etl2pcapng, enabling deep packet analysis in Wireshark or tshark. This was an eye-opener for engineers accustomed to Linux tooling. While tcpdump shows what happens on the wire, it cannot correlate that with kernel-level events. netsh trace, however, ties together on-the-wire activity with TCP layer events, timers, buffer management, socket layer behavior, and even the Windows IOCP subsystem. (Microsoft Message Analyzer, which we used to confirm our theory of small congestion windows, has since been retired; Microsoft now recommends pktmon for log analysis along with packet dumps.)

Microsoft engineers also pointed us toward packet reordering on the link, suggesting we check for DSACKs:

> tshark -r http2-single-stream.pcapng "tcp.options.sack_le < tcp.ack" | find /c /v "" 131

This filter isn't direct proof of reordering, but it indicates likely reordering because spurious retransmissions often produce DSACKs. If a system has reordering resilience, this filter won't trigger.

Where Reordering Came From

Using the SACKs observed in captures, we traced the reordering to somewhere between our L4 and L7 load balancers. We noticed a reordering event for a single flow occurring every 1–10 seconds. Windows uses CUBIC congestion control, just like Linux, and reaching 10 Gbps over a 100 ms RTT requires a packet loss rate below 0.000003%. Even a tiny amount of perceived loss—triggered by the traditional "3 duplicate ACK" heuristic—can drastically reduce performance.

We went through each layer of the network path, ruling out common culprits like network flapping and ECMP load-balancing issues. Then we examined the NICs. The source of the reordering was a side effect of an Intel NIC feature called Application Targeted Routing (ATR). ATR is designed to reduce CPU usage by directing packets to the CPU that currently owns the TCP flow, thereby reducing cache misses. In practice, however, it can make the operating system believe there is packet reordering on the link.

The problem is especially severe when the flow director's filter table overflows and gets forcefully flushed:

$ ethtool -S eth0 | fgrep fdir
 port.fdir_flush_cnt: 409776
 port.fdir_atr_match: 2090843606356

This is a known issue, documented in academic literature. On Linux, the OS handles such NIC-induced reordering gracefully, but Windows' TCP stack was less resilient, interpreting the reordering as packet loss and throttling the congestion window.

The collaboration with Microsoft led to improvements in Windows' TCP RACK-TLP implementation, making it more tolerant of packet reordering. The long-term fix is now available, and we're pleased to have a workaround that helped our users immediately while the permanent solution was developed.

Why Windows uploads lagged behind

Dropbox’s Edge Network sits between the client and the storage backend, and for years a quirk in how we deployed network cards was quietly handicapping Windows uploads. The problem wasn't bandwidth or client code—it was packet reordering induced by a hardware feature on our server NICs.

Intel's i40e driver family enables Application Targeting Routing (ATR) by default. ATR programs the NIC's FlowDirector table to steer flows toward the CPU core that most recently handled them, reducing cache misses. While this improves throughput in many workloads, it introduces a subtle failure mode for TCP senders: ATR can deliver a SYN packet to a different core than the connection's data packets. On the receive side at our edge, the connections that went through the ATR path were thus more prone to having their SYN and data land on different queues in the receive ring. In environments using Receive Side Scaling with a single queue per flow, this could induce same-flow reordering.

Because Dropbox runs TCP connections to a single PoP at a time, all traffic for those flows passes through one switch and one NIC. With most uploads hitting a single receiver, any reordering significantly degraded throughput. The SACK-based loss detection in Windows TCP interpreted even slight reordering as congestion or loss, triggering excessive retransmissions and halving effective throughput: what should have been tens of connections per PoP instead appeared to be double the connection count, each competing from a starting window we wouldn't normally expect.

Evaluating the fixes

We weighed three possible solutions on the server side, with different costs and benefits:

• Pin proxy threads to CPUs—combined with IRQ pinning and XPS. This would eliminate cross-core thread migration and rule out reordering entirely, but required substantial engineering and rollout effort. We put it aside unless we get closer to hitting CPU utilization limits on the Edge within the next couple of years.
• Reconfigure FlowDirector—older 10G Intel NICs (ixgbe) had a FdirPballoc parameter for tuning FlowDirector memory. That tunable is absent from current i40e docs and modern kernel ixgbe docs, and we didn't want to spend time doing kernel archaeology to locate what happened to it.
• Turn ATR off—we don't rely on it for anything, so this became our choice. A simple ethtool command disables the feature:

# ethtool --set-priv-flags eth0 flow-director-atr off

Note that the flag names are driver- and firmware-specific; flow-director-atr applies to Intel i40e NICs.

Immediate results

We applied the change to a single PoP. Server-side packet reordering dropped immediately:

And Windows clients connecting to that PoP saw upload throughput jump proportionally:

As a follow-up, we also started per-platform tracking of client upload and download performance. After rolling the fix across the entire Dropbox Edge Network, Windows upload speeds reached parity with macOS:

Microsoft's complementing fix

Meanwhile, the Windows Core TCP team was independently working on making Windows TCP resilient to reordering. The resulting update, delivered in Windows 10 build 21332, implements the full Standards Track RFC 8985 ("The RACK-TLP Loss Detection Algorithm for TCP"), including the reordering heuristic. This lets TCP tolerate up to a round-trip-time's worth of reordering in the network before declaring a loss.

Microsoft's team explained why this is important: when a sender receives an ACK with a SACKed segment, it cannot immediately tell whether the missing gaps are due to reordering or actual loss. Only hindsight—seeing the gaps filled without a retransmission—reveals which it was. Therefore, a reliable loss detection algorithm must budget a reordering window to disambiguate the two cases. That's precisely what the new heuristic provides.

To confirm, we reran our tests against a single PoP using the newest available Windows 10 build (10.0.21343.1000), toggling ATR on and off. No upload performance degradation was observed in either configuration.

Appendix: not the first to hit this wall

We weren't pioneering this discovery. The problem is already documented in HPC literature. Fermilab's paper "Why Does Flow Director Cause Packet Reordering?" describes essentially the same issue:

Acknowledgments

This was a cross-team debugging effort. Thanks to:

• Dropbox Networking: Amit Chudasma.
• Dropbox Desktop Client Sync Engine: Geoffry Song, John Lai, and Joshua Warner.
• Microsoft Core TCP Team: Matt Olson, Praveen Balasubramanian, and Yi Huang.
• Our customers' engineers who joined the performance improvement sessions.