The Stack Rides on the Network
About two months ago we published benchmarking results of edge networks around the world. Those tests covered TCP connection time, time to first byte and time to last byte, and showed Cloudflare was the fastest provider in 49% of the top 1,000 networks measured. We then set a goal during Birthday Week in September 2021 to grow that share of networks by 10% every Innovation Week — and met it.
For Full Stack Week in November 2021, the target was blown away. Cloudflare is now the fastest provider in 79 additional networks among the top 1,000 by number of IPv4 addresses advertised, a 14% increase. It is not the endpoint, but it is worth unpacking how that progress was made.
Workers vs. Compute@Edge
Given that this is Full Stack Week, it's also worth updating our serverless performance numbers. Our previous comparisons put Cloudflare Workers at 210% faster than Lambda and 298% faster than Lambda@Edge. Fastly's Compute@Edge was not generally available at that time, so we could not include it.
With Compute@Edge now GA, we ran the same test methodology: a function returning the current time, measuring the time to first byte after DNS, connection, and TLS handshake. Tests were performed on November 8, 2021, using free tier accounts on both providers, executing identical code that returns all request headers:
addEventListener('fetch', event => event.respondWith(handleRequest(event)));
async function handleRequest(event) {
let requestHeaders = Object.fromEntries(event.request.headers)
return new Response(JSON.stringify(requestHeaders), {status: 200})
};
The results, based on tests from 50 nodes using Catchpoint data from across the world, show that Cloudflare Workers is 196% faster than Fastly's Compute@Edge.
Orange represents Cloudflare Workers, black represents Compute@Edge.
The raw data is available for independent review.
Improving Last Mile Rankings
To measure network performance, Cloudflare uses Real User Measurements (RUM) fetching a 100kb file from several providers. Users report performance from their networks, and the aggregate signal shows where each provider ranks. After Birthday Week, we identified 601 country/network pairs where Cloudflare was more than 100ms behind the leading provider — where a pair is defined as a network's performance within a specific country.
In the past two weeks we became number one in TCP Connection Time in 79 new networks.
Peru: Keep Traffic in the Country
A large number of users in Lima were being routed to Chile for service, despite Cloudflare having multiple locations in Peru. This pushed us to fourth place on that specific network. The root cause: the network provider was distributing traffic unevenly, overloaded one of our Lima locations, and the fail-safes shifted traffic out of the country.
The short-term fix was manual load balancing across Lima locations. We stopped advertising some prefixes from the most loaded location, expecting traffic to redistribute to other Lima locations rather than leave the country. The hypothesis held; after scaling the change, the P90 Client TCP RTT dropped from 240ms to 60ms.
Cloudflare is now number one in network performance in Peru.
Sri Lanka: From Colombo to Newark
In Sri Lanka, a network provider was sending user requests all the way to Newark. That resulted in severe latency and a fourth-place ranking. Although Colombo is a relatively small location, we advertised as much traffic as possible through it to minimize the volume that would reach Newark.
After making the change, P90 Client TCP RTT dropped from 150ms to 50ms.
1 * * *
2 100.85.0.1 3.061ms 2.522ms 2.728ms
3 198.51.100.146 AS29766 3.651ms 1.855ms 2.715ms
4 198.51.100.145 AS29766 3.438ms 3.225ms 2.805ms
5 222.165.177.150 AS9329 2.233ms 2.272ms 2.843ms
6 222.165.177.145 AS9329 2.703ms 2.862ms 2.291ms
7 103.87.125.253 AS45489 3.658ms 3.708ms 3.613ms
8 103.87.124.245 AS45489 120.027ms 120.665ms 120.471ms
9 103.87.124.146 AS45489 115.597ms 115.863ms 115.178ms
10 50.208.235.157 be-107-2008-pe01.60hudson.ny.ibone.comcast.net AS7922 249.884ms 249.475ms 250.063ms -> going from Sri Lanka to New York
11 96.110.41.145 be-4101-cs01.newyork.ny.ibone.comcast.net AS7922 267.839ms 267.979ms 268.719ms
12 96.110.34.34 be-3112-pe12.111eighthave.ny.ibone.comcast.net AS7922 262.647ms 261.272ms 262.272ms
13 66.208.233.106 AS7922 262.378ms 258.948ms 258.057ms
14 172.70.108.4 AS13335 268.974ms 280.475ms 268.158ms
15 172.67.182.209 AS13335 267.329ms 266.466ms 266.593ms
Even with all ranges advertised through Colombo, the provider was still routing some Cloudflare prefixes to Newark, so we contacted them directly about the user-impacting change. Cloudflare subsequently moved from fourth to number one in Sri Lanka.
Where We Stand After Full Stack Week
Compared to Birthday Week, our position improved in 79 more networks.
Improvement was not limited to the last mile — Time to Last Byte is also faster than it was in September.
Country-level performance remains a moving target, driven by daily access patterns and aggregation windows. Here is the map of fastest providers per country as of Birthday Week:
And two months later, as of Full Stack Week:
The Long Tail Remains the Work
The common theme of these network updates is the long tail of problems. Each slow network is usually slow for its own reason, as the Peru and Sri Lanka cases show. We have automated the discovery of such issues and are working on automation that can also remediate them.
Performance gains on the network layer benefit everything running on it, not just one metric. One more Innovation Week remains in 2021, and we will be back with another progress report.
