Chasing Down the Last 100ms

Two weeks after releasing extensive edge network benchmarks, we set out to fix the weak spots. Our initial data revealed 61 country/network pairs where our average latency was more than 100ms behind the leading provider. We committed to improving in at least 10% of networks where we weren't #1 — we've exceeded that goal, and here's how we did it.

After two weeks of targeted work, we are now the fastest provider in 29 of those 61 networks. Performance improved dramatically in nearly all of the remaining ones as well.

The fixes weren't a single sweeping change. Each problem network required its own diagnosis. Here are three case studies showing the different failure modes we encountered and the specific interventions that resolved them.

Route Advertisement in Brazil

One network in São Paulo, which we'll call Network-A, was known to our edge team for frequently congesting its dedicated private network interconnect (PNI) with Cloudflare. The initial diagnosis showed that a significant chunk of traffic destined for Network-A wasn't using that PNI at all — it was being routed through a transit provider instead.

That wasn't surprising; at some point, traffic had likely been shifted off the PNI due to the congestion issues. But the path through the transit provider turned out to be far worse than expected. Traffic was being sent all the way to the United States and then back to Brazil — classic tromboning. The transit provider was preferring a longer path, likely due to BGP configurations or peering preferences, which explained the elevated latency.

The solution involved using our existing infrastructure in Rio de Janeiro, where we also had a PNI with Network-A. São Paulo and Rio are connected via our backbone network, so we had a way to keep the traffic in-country.

  1. We started announcing Network-A's IP addresses from our backbone in Rio and accepting them in São Paulo.
  2. We modified BGP behavior using the LOCAL_PREF path attribute to prefer the backbone path over the PNI.
  3. We removed the transit provider as the preferred route for Network-A traffic.

The result was immediate: p95 TCP RTT dropped from 175ms to 90ms. Network-A moved us from #5 to #1 in their ranking.

Ingress Gaps in Spain

Network-B, a European ISP with global reach, showed high latencies in Spain and elsewhere. Traffic from its users in Spain was being served from our data centers in Milan and Marseille instead of staying local. The reason was straightforward: Network-B had no peering session with us in Madrid.

We checked our Peering Portal and stood up public peering sessions at an IXP in Madrid and other locations where we saw high latency. We then identified that several destination IP ranges weren't being advertised in-country — the anycast addresses weren't enabled in Barcelona. Once we enabled them, traffic from Network-B in Spain was handled locally.

We also investigated the peering status in Milan, where Network-B's public session was down. Our logs showed the session had been dropped because Network-B believed we were sending more IP prefixes than allowed. We contacted them and advised updating their configuration based on the data in PeeringDB.

The changes moved us from #2 to #1 for this network, though the latency chart still shows roughly 300ms. We can optimize our side of the path, but we can't control how every provider connects to the rest of the Internet.

Upstream Choice in Africa

The third case presented the opposite lesson: sometimes a transit provider path can be better than a private interconnection. For a network we'll call Network-C, traffic was served from our Western European data centers because we didn't peer with its parent ASNs in Africa.

In London, we had both a private peering connection with Parent-1 and a transit connection with Parent-2. Both parents advertised Network-C prefixes. Our policy preferred Parent-1 since it was a private peer — but we suspected Parent-2, which belonged to the same organization as Parent-1, might provide a more direct path.

The fix was a targeted BGP change. Our routing policies assign different LOCAL_PREF values based on connection type. To steer only Network-C traffic toward the transit provider, we set a higher LOCAL_PREF for those specific prefixes. We used a regular expression to match Network-C's ASN in the AS-path without affecting other traffic on the transit connection. The change launched us from #2 to #1 for this network.

Headline Metrics Update

These efforts also shifted our global aggregate numbers. At Speed Week, two top-level metrics showed us in the #2 slot: mean TCP connection time and mean TTLB among the top 1,000 networks by IP count.

We've since taken the #1 spot in mean TCP connection time. Our TTLB metric has also moved to #1, though less dramatically — the work of the past two weeks optimized network paths, not our software platform, so connection times improved more than time-to-last-byte.

The Long Tail of Network Performance

Each network latency problem requires a different solution because every network is unique. The Internet is a network of networks, and improving performance often requires cooperation with other operators — adjusting BGP routes, establishing new peering sessions, and rethinking which paths we prefer.

We're continuing beyond this two-week window. We're automating the discovery of poor-performing networks like these, and we're working toward automating remediation as well. We'll report again on our progress as we push further toward minimizing latency globally.