Dropbox traffic infrastructure: Edge network

Why Dropbox built an edge network

With more than half a billion registered users, an exabyte of data, and petabytes of metadata, Dropbox’s Traffic team handles millions of HTTP requests and terabits of traffic daily. To manage this load, Dropbox has deployed a global network of points of presence (PoPs), collectively referred to as the Edge.

The core motivation is latency. BGP chooses routes based on hop count, ignoring link latency, throughput, and packet loss. Simply placing a PoP close to a user can more than halve latency. This directly benefits file transfers: TCP’s congestion window grows faster on low-latency connections because slow start is RTT-bound. Those connections also reach a higher congestion-avoidance threshold, since packets spend less time on congested public internet links before entering Dropbox’s own backbone.

Beyond performance, the Edge gives traffic engineers control to experiment with new protocols, load-balancing algorithms, and techniques such as BBR congestion control—things that are difficult to test on infrastructure you do not own.

Choosing PoP locations

Dropbox currently operates 20 PoPs worldwide, with a newly announced PoP in Toronto and two more planned in Scandinavia by the end of the year. Expansion into LATAM, the Middle East, and APAC is under research for 2019.

Selecting PoP locations is no longer simple. Engineers must weigh backbone capacity, peering connectivity, submarine cable routes, and—critically—position relative to existing PoPs. The decision between, say, Brazil and Australia, or Vienna and Warsaw, is not obvious by intuition alone.

Dropbox alternates between adding PoPs that most benefit existing users and those that capture new ones. A script assists this process:

1. Split the Earth into 7th-level s2 regions.
2. Place all existing PoPs.
3. Compute the distance to the nearest PoP for each region, weighted by “population.”
4. Exhaustively search for the best location for a new PoP.
5. Add it and repeat from step 3.

The “population” metric can be any optimization target—total people, existing users, or potential users. A standard L1 or L2 loss function scores each placement, with Dropbox weighting latency effects on TCP throughput. Although gradient descent or Bayesian optimization could solve the problem, the search space is small (fewer than 100K seventh-level s2 cells), so brute force yields a definitively optimal answer without the risk of local minima.

GSLB: routing users to PoPs

Global Server Load Balancing (GSLB) is the most critical Edge component. It sends each user to the nearest PoP, unless that PoP is over capacity or under maintenance. If GSLB frequently misroutes users, the entire Edge network becomes useless—or even harmful. Two primary techniques exist: BGP anycast and GeoDNS.

BGP anycast

Anycast is the simplest method: advertise the same subnet from every PoP and let BGP deliver packets to the “optimal” one. It offers automatic failover and simple setup, but has serious drawbacks.

Performance. BGP’s route selection ignores latency, throughput, and packet loss, generally picking the path with the fewest hops. Anycast load balancing is mostly optimal but performs poorly at high percentiles. However, the probability of “critical” misrouting—sending a user to a different continent—may drop sharply as the number of PoPs grows. Dropbox will evaluate whether anycast begins to outperform GeoDNS as its PoP count increases.

Traffic steering. With anycast, control is limited. Moving traffic from one PoP to another requires MED attributes, AS_PATH prepending, and direct coordination with providers—none of which scale. Because AS_PATH sits in the middle of the N WLLA OMNI mnemonic, administrators can easily override it, meaning anycast picks the “cheapest” route, not the nearest or fastest. A graceful PoP drain is also impossible: BGP balances packets, not connections, so routing changes immediately redirect in-flight TCP sessions to the next best PoP, where users receive RST packets.

Troubleshooting. Reasoning about anycast routing requires knowledge of internet routing state at a given moment. Debugging performance issues involves traceroutes, looking glasses, and back-and-forth with providers. Any connectivity change on the internet can break user connections to anycasted IPs, and intermittent issues from misconfigured hardware are difficult to diagnose.

Tools. A unique request ID for every request, traceable through logs, is essential. Edge responses should echo a header identifying the PoP or embed it in the request ID. Debug sites that pre-collect troubleshooting data—such as github-debug.com, fastly-debug.com, and dropbox-debug.com—help users attach the right information to support tickets.

Despite these drawbacks, Dropbox still uses anycast for its apex domains like dropbox.com (without www) and as a fallback during major DDoS attacks.

GeoDNS

GeoDNS gives each PoP its own unicast IP space. DNS hands out different addresses based on the user’s geographical location, as inferred by the DNS resolver or EDNS Client Subnet data.

This approach provides explicit control over traffic steering and permits graceful drain. Unicast setups are also far easier to reason about and troubleshoot. However, accuracy depends on multiple layers of approximation: the DNS provider’s guess of the user’s IP, the GeoIP database’s guess of the user’s location, and the assumed correlation between physical proximity and latency.

Different DNS providers often make different decisions, complicating performance monitoring. DNS also suffers from stale data—TTL is not a reliable guarantee. Despite a one-minute TTL for www.dropbox.com, it takes 15 minutes to drain 90% of traffic and up to an hour to drain 95%.

From GeoIP to Real User Metrics

Dropbox's DNS routing has gone through several generations. It started with simple continent-to-PoP mappings, moved to country-level routing with per-state data for larger countries, and now uses a combination of LatLong-based routing with AS-based overrides to handle internet connectivity quirks and peering arrangements.

Hybrid Unicast/Anycast GSLB

One composite approach to GSLB combines unicast and anycast announces alongside GeoDNS mapping. A PoP announces both its own unicast subnet (for example, a /24) and one of its supernets (for example, a /19) from all PoPs, including itself.

This implies that every PoP should be set up to handle traffic destined to any PoP: i.e. have all the VIPs from all the PoPs in the BGP daemons/L4 balancers/L7 proxies configs.

This design allows fast switching between unicast and anycast addresses, providing immediate fallback without waiting for DNS TTL expiration. It also enables graceful PoP draining and retains all the benefits of DNS-level traffic steering. The trade-off is increased operational complexity and potential scalability issues once VIP counts reach the high thousands. On the positive side, PoP configurations become more uniform across the network.

Collecting Real User Metrics

All the GSLB methods described so far share a fundamental limitation: none of them use actual user-perceived performance as a signal. BGP routing relies on hop counts, while GeoIP relies on physical proximity. To address this, Dropbox built a Real User Metrics (RUM) collection pipeline based on performance data from its desktop clients.

Companies that do not have an app usually do latency measurements with the JS-based prober on their website.

The existing availability measurement framework in the Desktop Clients was extended to also log latency information. A sample of clients periodically runs measurements against all PoPs and reports back the results. Separately, a resolver_ip→client_ip submap is built by joining DNS and HTTP server logs for HTTP requests to random subdomains of a wildcard DNS record, with some post-processing applied for EDNS ClientSubnet-capable resolvers.

The final client_subnet→PoP map is produced by combining aggregated latencies, the resolver_ip→client_ip map, BGP fullview data, peering information, and capacity data from the monitoring system. Web server logs are also a candidate signal source, since TCP_INFO data is already available, including retransmit counts, cwnd/rwnd, and RTT measurements.

The map is packed into a radix tree and uploaded to a DNS server, then compared against both anycast and GeoIP solutions. Map generation approaches have ranged from a simple HiveQL query doing per-/24 aggregation to machine learning solutions like Random Forests, stacked XGBoosts, and DNNs. The sophisticated models yield slightly better results but at the cost of longer training and difficult reverse engineering. For now, the team prefers the solution that is easier to reason about and troubleshoot.

Data Hygiene and Extrapolation

All latency and availability data is anonymized and aggregated by /24 subnet for IPv4 and /56 for IPv6. The system does not operate directly on real user IPs, with strict ACL and retention policies enforced for all RUM data. Several common data quality issues were discovered during map construction:

• The standard GetTickCount64 timer on Windows is quantized to around 16ms; the Python client switched to time.perf_counter().
• TCP and HTTP probes are far less reliable than HTTPS due to IP and DNS hijacking in the wild, such as Wi-Fi captive portals.
• Even unique DNS requests can be received multiple times — from lost responses and proactive cache refreshes — so this duplication must be accounted for when joining HTTP and DNS logs.
• Timing results include negative and submicrosecond values as well as ones that are impossibly old, indicating data corruption from unknown sources.

Two extrapolation techniques are used to speculatively expand the resulting map:

• If all samples for an AS point to the same "best" PoP, the entire set of IP ranges announced by that AS is routed to that PoP.
• If an AS has multiple "best" PoPs, it is broken down into announced IP ranges; if all measurements for a single range agree on one PoP, that choice is extrapolated to the whole range.

This approach doubles map coverage, improves robustness to changes, and allows the map to be generated from a smaller dataset.

Evaluating the DNS Map

Once a RUM-based map is constructed, it needs a single evaluation metric — analogous to an F1 score for binary classification or BLEU for machine translation. Having one scalar value makes it possible to automatically block bad maps from going live and to compare iterations and construction algorithms numerically. A common validation approach is testing the map against a subset of data the training process never saw. For ad-hoc troubleshooting, subnets are mapped back to lat/long coordinates, aggregate stats are computed by h3 regions, and results are visualized with kepler.gl.

h3 was chosen over s2 primarily because Kepler has built-in support for it, and h3 generally offers a simpler Python interface for experimentation with visualizations. The same visualization approach works for showing current performance, week-over-week differences, or comparing GeoIP database versions.

An alternative visualization skips GeoDNS mapping entirely: plot IP addresses on a 2D plane using a space-filling curve such as a Hilbert curve, with additional data in height and color dimensions. This approach requires heavy regularization to be human-readable and considerable ColorBrewer2 work to be aesthetically pleasing.

RUM DNS Deployment Status

RUM-based DNS is still an active project and has not shipped for main VIPs yet. However, data collected from GSLB experiments indicates it is the only way to properly utilize more than 25–30 PoPs. Early map prototypes suggest it could improve Edge network effectiveness by up to 30%, making it one of the top priorities for 2019.

The project will also provide all the byproducts needed for the Explicit Loadbalancer.

Explicit Loadbalancing

Once a user's request actually arrives at a PoP, all the guesswork about resolver IPs, GeoIP effectiveness, and BGP decision optimality becomes unnecessary — the user's IP is known and there is a direct RTT measurement. At that point, routing can be handled at a higher level, such as embedding a link to a specific PoP in HTML or handing a different domain to a desktop client downloading files.

The same IP→PoP map built for RUM DNS can be reused here, but exposed as an RPC service instead. This method allows for very granular traffic steering based on per-resource information like user ID, file size, and physical location in distributed storage. It also enables almost immediate draining of new connections, although previously issued resource references may persist for extended periods.

Complex schemes are possible, such as handing off URLs where the domain name embeds external DNS routing information while the path or query arguments carry internal routing data, allowing the PoP to make more optimal routing decisions. Alternatively, that data could be placed as an opaque blob in an encrypted or signed cookie. Given these possibilities, careful attention is needed to avoid overcomplicating the system.

Internal Use Today

Dropbox currently uses explicit loadbalancing not as an external method but for internal re-routing. The traffic team is building the foundation for broader use of this approach.

A look inside a Dropbox point of presence

Network architecture

Each PoP is a mix of network gear and Linux servers with rich connectivity: backbone links, multiple transits, and public and private peering. The more traffic that stays on those interconnects, the less time packets spend traversing the public internet, which cuts packet loss and boosts TCP throughput. Roughly half of Dropbox’s traffic currently arrives via peering.

L4 load balancer

Within a PoP, nginx boxes act as both L7 proxies and L4 load balancers (L4LBs), spreading connections across the proxy tier. The L4LBs use standard building blocks for scale and fault tolerance: BGP ECMP, DSR, and consistent hashing. The dataplane is IPVS, a kernel-level load balancer with a netlink API that provides state tracking, pluggable scheduling algorithms, and IP-in-IP encapsulation for DSR.

There are two dominant approaches to building high-performance packet processors today:

Kernel

Processing packets early in the network stack lets you reuse in-kernel data structures and TCP/IP parsing. Linux has long shipped IPVS and netfilter modules for connection-level load balancing, and newer kernels add the eBPF/XDP combo for safer, faster in-kernel processing. The tradeoff is tight kernel coupling: upgrades may require reboots, kernel versions are pinned, and integration testing is harder. Facebook and Dropbox both follow this path.

Userspace

The alternative is to create a virtual NIC PCIe device with SR-IOV and bypass the kernel entirely with DPDK, netmap, or similar. This gives full control over networking, but TCP/IP parsing, data structures, and memory management all become the developer’s problem (or a third-party library’s). Testing is far easier. Google and GitHub use this model.

Dropbox currently runs a homegrown consistent hashing module on the L4LBs. Starting with Linux 4.18, there’s an in-kernel Maglev Hash implementation for IPVS (ip_vs_mh). Compared to Ketama, Maglev trades some hash resiliency for a more even load distribution across backends and better lookup speed.

Packets are hashed on the 5-tuple (protocol, source IP, dest IP, source port, dest port) to spread load further. The downside: any server-side cache becomes ineffective, since connections from the same client land on different backends. A 3-tuple hash on (protocol, source IP, dest IP) would preserve cache locality when needed.

L4LBs must also handle ICMP Packet Too Big replies specially, since those originate from a different host and can’t be hashed on the outer header. The alternative used by Cloudflare — pmtud — broadcasts incoming ICMP to every box in the PoP, which only works if there is no separate routing layer and packets are ECMP’d straight to the L7 proxies.

The L4LB control plane is written in Go, tightly coupled to Dropbox’s infrastructure, and owns online reconfiguration, BGP connectivity, and backend health checks. Health checking on an encapsulating DSR-based L4LB is tricky: probes must traverse the same encapsulation path as real data, otherwise you risk sending traffic to a box whose tunnel isn’t set up.

Key properties of the L4LBs:

• They are resilient and horizontally scalable — since no TCP connection is terminated and consistent hashing drives scheduling, L4LBs can be added or removed without interrupting existing flows.
• Graceful proxy churn — because the L4LBs keep a connection tracking table, changing the backend set doesn’t break existing connections, unlike plain ECMP.
• L7 proxies scale horizontally until bandwidth becomes the limiting factor; the L4LB is fast enough to be network-bound.
• Any IP-based protocol is supported.
• Any hashing algorithm (Maglev, Rendezvous, etc.) can be swapped in for experimentation.
• Any hashing policy works — 3-tuple, 5-tuple, or QUIC Connection ID.

As a side effect, nearly any production server can become a high-performance load balancer by just running the binary.

Future work on the L4LB front: replacing the routing dataplane with DPDK or XDP/eBPF, possibly by integrating an open-source project like Katran. Dropbox currently uses IP-in-IP for encapsulation and is considering GUE, which is more NIC-friendly in terms of steering and offload support.

L7 proxies: nginx

Edge PoPs close to users cut TCP and TLS handshake times, which directly improves TTFB. Owning this infrastructure rather than leasing a CDN lets Dropbox iterate on emerging technologies that squeeze latency and throughput. A few areas of focus:

TCP

Fair Queueing (FQ) is a packet scheduler that provides fairness between flows and adds pacing at the upper protocol level. Without it, packets are dumped to the network as they arrive from the TCP stack, creating head-of-line blocking down the stack. With FQ, packets from different flows interleave and no single flow blocks another. Pacing works similarly: instead of blasting thousands of packets out at once when the congestion window allows, the TCP stack hints the scheduler at a desired sending rate (derived from congestion window and RTT), and the scheduler paces packet submission to sustain that rate.

FQ costs about 5 percent CPU but makes routers and shapers along the path behave better, reducing packet loss and bufferbloat. In practice, deploying FQ eliminated all buffer drops on Dropbox’s top-of-rack switches, which had shallow buffers vulnerable to microbursts despite their high aggregate throughput.

Beyond FQ, newer Linux kernels offer Tail Loss Probe, TCP Small Queues, TCP_NOTSENT_LOWAT, RACK, and more. The Traffic team periodically dives into network- and transport-level optimizations, often involving Wireshark and packetdrill. One upcoming evaluation is BBR v2 once it’s publicly testable.

TLS

All external connections to Dropbox are TLS-protected; internal backbone connections are re-encrypted and mutually authenticated. Because the same TLS stack serves gRPC internally, performance matters twice: handshakes use the most efficient hardware instructions available, and large transfers minimize memory copies.

The TLS setup itself is simple: BoringSSL, TLS tickets with frequently rotated ephemeral keys, and a preference for AEAD ciphersuites with ChaCha20/Poly1305 on older hardware. RFC-version TLS 1.3 is rolling out across the Edge network. As boxes approach 100 Gbit, Dropbox’s future plans include exploring TCP_ULP and adding its support to the software stack.

HTTP

The edge nginx proxies main job is maintaining keepalive connections to data-center backends over the backbone. That means a set of hot connections that never hit congestion window limits on an almost lossless link.

Nginx itself is built with Bazel, producing a static binary hermetically; the configs are bundled, the whole thing is packaged into a squashfs, distributed over torrent, mounted read-only, symlinked, and then upgraded in place.

Because the nginx config is static, a separate Upstream Management Service (UMS) provides dynamic upstream reconfiguration without full redeploys. Possible approaches include regenerating configs and hot reloading (breaks connection reuse and spikes memory when used often), the nginx plus configuration API, a sidecar proxy on the same box (big CPU/memory cost), or custom Lua/C modules. Dropbox already uses Lua, so UMS’s dataplane is built there: a balancer_by_lua_block directive combined with an ngx.timer.every hook that periodically fetches config from the control plane over HTTPS.

A Lua-based balancer module allows quick experimentation with load-balancing algorithms before they’re written in C; the downside is that Lua is hard to test, particularly at a company where it isn’t a primary language. The UMS control plane is a Go service that gathers data from Dropbox’s service discovery, monitoring, and manual overrides, then exposes it as a REST endpoint that nginx polls.

The edge stack terminates HTTP, HTTP/2, and gRPC connections in nginx. Being able to proxy gRPC lets Dropbox experiment with apps speaking gRPC directly to application servers, streamlining development and unifying internal and external service communication. The long-term aim is gRPC for all APIs; for those that can’t migrate, such as web, Dropbox is considering converting every HTTP request into a gRPC method call at the edge.

The internal side of traffic

The external edge is only half of Dropbox's traffic story. The internal counterpart — a gRPC-based service mesh, scalable and robust service discovery, and a distributed filesystem for config distribution with notification support — will be covered in a subsequent series of posts.

Owning the edge

Dropbox operates a globally distributed edge network handling terabits of traffic and millions of requests per second, all managed by a small team based in Mountain View, CA.

The team is hiring software and site reliability engineers to work on TCP/IP packet processors and loadbalancers, HTTP/2 proxies, and the internal gRPC-based service mesh. Openings also exist across a broad set of engineering roles in San Francisco, New York, Seattle, Tel Aviv, and other global offices.

Credits

This work spans contributions from many current and former members of the traffic team over several years. Thanks to Ashwin Amit, Brian Pane, Dmitry Kopytkov, Dzmitry Markovich, Eduard Snesarev, Haowei Yuan, John Serrano, Jon Lee, Kannan Goundan, Konstantin Belyalov, Mario Brito, Oleg Guba, Patrick Lee, Preslav Le, Ross Delinger, Ruslan Nigmatullin, Vladimir Sheyda, and Yi-Shu Tai for making Dropbox faster and more reliable.