The hard part of benchmarking a CDN is deciding what to measure

During Speed Week, Cloudflare has been discussing services that accelerate web traffic. But any claims about speed need to be backed by evidence. To get that evidence, the company recently ran a measurement experiment using Real User Measurement (RUM) data from the standard browser API. The goal was to compare Cloudflare's performance against Akamai, Amazon CloudFront, Fastly, and Google across real-world conditions.

Commercial Internet measurement services such as Cedexis, Catchpoint, Pingdom, and ThousandEyes already exist and offer this sort of RUM data. Cloudflare subscribes to third-party monitoring services, but was not satisfied with what they showed. The sampling methods were often skewed by measuring from the server side rather than the eyeball side of the network, or were tainted by bots and automated traffic. The data was also not granular enough. It showed performance by country or region, but did not dive into individual networks, obscuring the details and outliers behind averages.

How the test was built

Cloudflare created a small internal team to do the measurements, separate from the team that manages and optimizes its network. To test other CDNs fairly, they used the same endpoints that commercial measurement services use, on the assumption that competitors would have already ensured those endpoints are optimized to show their networks' best performance.

The measurements in this analysis are based on four days just before Speed Week began (2021-09-10 12:25:02 UTC to 2021-09-13 16:21:10 UTC) and were taken by downloading exactly the same 100KB PNG file. This is a common industry test measurement because it allows measurement of network characteristics like connection time, but also total download time.

Getting data from the browser

Getting real user measurement data has been made easy for HTTP endpoints thanks to the Resource Timing API, supported by most modern browsers. This API allows a page to measure network timing data of fetched resources using high-resolution timestamps, accurate to 5 µs. The point is to capture how a real end-user experiences the Internet, not what a synthetic test might show from a single component of the web browsing process.

To measure each CDN, a browser visits one of Cloudflare's special pages, downloads an image from each CDN, and records the measurements using a Cloudflare Workers based API. To make sure the test replicates real-life conditions, Cloudflare made sure no local caching was involved in the request, checked if the object has been compressed by the server, took HTTP header size into account, and recorded whether the connection was pre-warmed.

The three key metrics

Three measurements were used to illustrate network performance:

  • TCP connection time – counts only the time taken for a machine to establish a connection to the remote host, calculated as connectEnd - connectStart. This shows how well-connected a network is.
  • TTFB (time to first byte) – the time for the first byte of data to return after an HTTP request is sent, calculated as responseStart - connectStart - (requestStart - connectEnd). This is a common measure of server responsiveness.
  • TTLB (time to last byte) – the time taken to send the entire response to the web browser, calculated as responseEnd - connectStart - (requestStart - connectEnd). This measures how long a complete download takes.

Two sets of data were produced for each metric: mean and p95 (95th percentile). The mean gives the average user experience, but can miss skewed distributions. The p95, which can be thought of as the "reasonable worst case," tells what performance 95% of measurements fall below.

Results: CDN performance by network

The Internet is a network of networks, identified by autonomous system numbers (ASNs). Large networks—ISPs, cloud providers, universities, mobile phone companies—operate autonomously and join the global Internet through BGP. There are roughly 70,000 ASNs globally, and during the measurement period Cloudflare saw traffic from about 21,000 of them. Not all networks are sources of traffic, so this is expected. For the analysis, results are grouped in three ways: all networks, the top 1,000 networks by IP address count, and the top 1,000 networks by number of observations. The first group includes major ISPs but also networks that have huge numbers of IP addresses that aren't necessarily used. The measurements-based group captures networks with heavy usage.

Taking an example: for TTFB at p95 among the top 1,000 networks by IPv4 addresses advertised, Cloudflare had the lowest time to first byte for the largest number of networks. Google was next, then Fastly, followed by Amazon CloudFront and Akamai.

Looking at two large US-based ISPs shows how granular this data is. Cox Communications (ASN 22773), with roughly 6.5 million customers, had these p95 TTFB results: Cloudflare 332.6ms, Fastly 357.6ms, Google 380.3ms, Amazon CloudFront 404.4ms and Akamai 441.5ms. Cloudflare was fastest, about 7% ahead of the next CDN.

Comcast (ASN 7922), with about 30 million customers, showed p95 TTFB of: Fastly 323.7ms, Cloudflare 324.2ms, Google 353.7ms, Akamai 384.6ms and Amazon CloudFront 418.8ms. Here Fastly edged out Cloudflare by 0.2%.

The full results comprise 18 charts covering all combinations of the three measurements, two aggregations (mean and p95), and three network groupings. In two of the 18 charts, Cloudflare was not #1 (it was #2), both in average times. Both of those figures are areas for improvement.

Where optimization is needed

Examining the top 1,000 networks by observations, Cloudflare is currently #1 or #2 for TTFB in 69.9% of networks, for TTLB in 65.0%, and for TCP connection time in 70.5%. To see how much work is needed to go from #2 to #1, the data shows that the median TTFB of the #1 network is 92.3% of Cloudflare's, median TTLB is 94.0%, and TCP connection time is 91.5%. That last figure is significant because it shows big gains can be made by optimizing network-level routing.

Why not a map?

World maps are a common way to present Internet performance data, but they obscure information. Maps show geography and land mass, not population or where people actually are. For example, Indonesia has double the population of Russia and about 1/10th of the land.

Focusing on networks allows optimization for actual users. For instance, Biznet, a major Indonesian ISP (ASN 17451), has the following p95 TTFB results: Cloudflare is fastest at 677.7ms, then Fastly 744.0ms, Google 872.8ms, Amazon CloudFront 1,239.9ms and Akamai 1,248.9ms.

With this granular view of every network connecting to it, Cloudflare can optimize routes. During Birthday Week, the stated goal is to improve performance in 10% of the networks where it is not #1 today. The measurements are ongoing, not a one-time effort.