Rethinking "closest" PoP selection

The Dropbox Traffic team manages application networking for more than half a billion users. In a previous post on our edge network, we described how geolocation-based load balancing works: when a user performs a DNS lookup for www.dropbox.com, we respond with the IP address of what we determine to be the nearest point of presence (PoP). For most cases, this approach works well and has allowed our edge network to scale to more than 20 clusters.

But geolocation alone has a fundamental flaw: it ignores network topology. A user can be physically close to a PoP while their ISP has no interconnection with us at that location. Even on a map of user mappings, it's possible to spot cases where users are assigned to unrealistic locations based on their IP geolocation alone.

When geography and topology disagree

Consider an ISP in Berlin (IATA code TXL). From a purely geographic perspective, the closest PoP is in Berlin itself. But if we don't have a private network interconnection (PNI) with that ISP in Berlin, the ISP must use a transit provider to reach us. That transit connection might be in Frankfurt (FRA), where we do have a PoP. The result: the client's traffic hops from Berlin to Frankfurt and back to Berlin again (TXL→FRA→TXL→FRA→TXL).

One solution is to establish PNIs with such ISPs—Dropbox has an open peering policy and actively benefits from direct interconnects. But it's unrealistic to track every such case manually, and corner cases remain.

A more extreme example involves users in Vladivostok, Russia. Geographic routing sends them to our Tokyo PoP, which is right next door on a map. However, most Russian ISPs have no presence in Tokyo; the bulk of their transit connections are in western Russia. The 75th percentile of RTT for these users reaches 300–400 ms because their traffic travels across the globe (Vladivostok→Moscow→Atlantic→US→Pacific→Tokyo).

Teaching DNS about topology

The fix is to shift from geographic "closeness" to network latency "closeness"—the PoP with the lowest latency to the user is the one that matters. To make this work, your DNS infrastructure needs a custom route map: a mapping of user subnets to PoPs based on measured latency.

Dropbox collaborated with NS1 to add support for uploading and using such custom DNS maps on a per-record basis. The map format is simple: a JSON object that maps subnets (where DNS resolvers live) to PoPs, identified by configurable tags—we use IATA codes.

Building the initial latency map

We already had latency data between user subnets and every Dropbox edge cluster, gathered through tests in our desktop client. What we didn't have was the link between a user's subnet and the DNS resolver they use.

Here's how DNS resolution normally plays out:

  1. The user opens www.dropbox.com or starts the Dropbox desktop client, which sends a DNS query to the configured resolver (usually the ISP's).
  2. The resolver iterates through requests to the authoritative DNS server for the dropbox.com zone (unless it has a cached reply).
  3. The authoritative server responds with the IP address of www.dropbox.com.
  4. The ISP's recursive resolver returns that IP address to the end user.

Critically, the authoritative DNS server never sees the end user's IP address—it only sees the resolver's address. While some public resolvers support DNS ECS extensions to carry client subnet information, adoption is still limited.

To bridge this gap, we added a test to our desktop client that issues DNS queries to random subdomains of dropbox.com. Random names guarantee the resolver has no cache and must perform a full lookup. On our side, we log all requests to these subdomains, giving us a mapping between unique DNS names and resolver IP addresses. The client periodically reports back which unique queries it issued (aggregated by subnet, with no individual IP logging). Joining these two datasets yields: client subnet ↔ DNS resolver ↔ latency to the PoP.

From there, we calculate the best PoP for the majority of users behind each resolver—using the 75th percentile of latency. The resulting map contains no user-specific data, only resolver subnets and target PoPs. NS1 supports multiple maps and can match on DNS ECS information when present, falling back to the resolver's IP address otherwise. The final map is uploaded via the NS1 API.

Measurable gains right away

We started deploying the latency-based map on a test domain and compared results against the geo-based map. Almost immediately, we saw roughly 10–15% latency improvement at both the 75th and 95th percentiles, with no negative impact at higher percentiles. The improvements showed up across regions, including Africa and the US.

To quantify the benefit more precisely, we compared the latency-based map against our anycast setup (used for the apex record of dropbox.com). Probability density function graphs showed that about 10% of users gained ~2 ms, 1% gained ~20 ms, and the long tail saw improvements of 200 ms or more. The gains were nearly universal: on a hexbin map of the world, brighter colors represented larger latency reductions.

Fixing the corner cases

The biggest wins came from eliminating the pathological geo-routing failures:

  • Vladivostok: Users were previously sent to Tokyo with 300–400 ms RTT. The latency-based map now routes them to Frankfurt—one of Europe's largest internet exchange points and a major transit hub for Russian ISPs. RTT dropped to roughly 150 ms.
  • Iceland: Geo-routing sent users to Oslo, Norway. But there's no direct submarine cable between Iceland and Norway; cable routes go through Denmark, making Amsterdam the better choice. Latency-based routing reduced RTT for Icelandic users by 10–15%.
  • Egypt: Geo-routing pointed to Milan, but latency measurements favored Paris. Submarine cable maps confirm why: most cables from Egypt land in France. The shift produced another ~10% latency improvement.

Across Europe, the difference is striking when comparing geo vs latency-based maps. Geo-routing draws fairly straight borders; the latency-based map demonstrates that most central European ISPs favor Frankfurt for inter-provider traffic exchange. The new map also revived our Berlin PoP: geo-routing had degraded performance there because many ISPs lacked PNIs in Berlin, so we only used it for whitelisted ISPs. With latency-based routing, the Berlin PoP is now fully utilized.

Next steps

The latency-based map is just an initial version. Our top priority for the next iteration is incorporating BGP policy—for example, steering specific ISPs toward locations where we have direct peering relationships. We're also planning to move from static weights to dynamic user allocation based on real-time PoP load, using a feedback loop from actual usage data.

Open peering and operational notes

Dropbox’s Edge network is open for public peering in more than 30 locations, with ASN19679 registered on PeeringDB. The team also makes private peering available in most large peering facilities for customers needing guaranteed capacity. Details are published on Dropbox’s peering page.

The peering footprint is visualized in the figure below using Kepler.gl and Uber’s h3 hexagon indexing library.

DNS ECS (RFC 7871) remains an important but inconsistently deployed feature. It works by having a recursive resolver forward a subnet prefix on behalf of the user issuing the DNS query, which allows the authoritative server to tailor responses geographically. Adoption is still limited: Google and OpenDNS are the most notable resolvers that support ECS, and no other major providers do at scale.

For operators planning capacity or troubleshooting latency, the public data above — the peering DB entry, the map of public and private peering locations, and the ECS notes — should make it straightforward to reason about how DNS steering decisions are made and where that logic can break down.

What this means for network operators

Anyone running a geographically distributed service with IP anycast will recognize the tension this design addresses: the DNS answer that is best for the client is not always the one that is best for the network. The Dropbox system described in the main body of this article resolves that by coupling fine-grained client ECS data with per-PoP health metrics, and by pushing as much of the decision logic as possible to edge infrastructure that can react in real time.

The practical takeaways extend beyond Dropbox. First, anycast alone will not give precise client-to-data-center mapping in regions where the routing table is shallow. Second, collecting resolver ECS coverage data is a prerequisite for steering accuracy — resolvers that do not send ECS will fall back to coarser signals, so knowing which resolvers support it per region matters. Third, constant validation loops, not periodic snapshots, are what keep a steering system stable as PoP capacities fluctuate and ISPs change routing.