A Broad CDN Outage: How a Small Tracing Change Broke Tiered Cache

Cloudflare's CDN hit a roughly six-hour incident on October 25, 2022, when a change to its Tiered Cache system caused a spike in HTTP 530 errors. At the peak, about 5% of all HTTP requests were failing. The root cause was an unassuming monitoring feature that wreaked havoc on internal request routing, exposing a testing blind spot that let the bug fly under the radar.

The outages were specific to customers using Tiered Cache, Cloudflare Images, or Bandwidth Alliance. Content that could be served directly from local data center cache was unaffected—issues arose only when requests had to reach further up the caching hierarchy.

How Tiered Cache Works

Cloudflare's CDN solves a latency problem: when a user's local data center doesn't have an asset cached, the request must travel to the origin server, which can be far away. Tiered Cache cleverly organizes data centers into a hierarchy—lower tiers near users, upper tiers closer to origins. A lower-tier miss first queries the upper tier; if the fresh asset lives there, the upper tier serves it without a costly fetch from origin. This both accelerates delivery and slashes how often Cloudflare queries a customer's origin server, but it opens up more moving parts that need to interoperate correctly.

Partial Cloudflare outage on October 25, 2022

What Actually Broke

Cloudflare routinely adds distributed tracing to monitor code performance. The recent addition to Tiered Cache wrapped the existing go() function with trace_fn(), which calls the function and reports its execution time—a classic, innocent-looking observability patch.

local trace_fn = require("opentracing").trace_fn

local function go()
  -- code to run here
end

function _M.go()
  trace_fn(ngx.ctx, "tiered_cache_rewrite", go)
end

But there was a poisonous interaction. The logic that injects the tracing function into the opentracing module clears control headers on every single request:

require("opentracing").configure_module(conf,
  -- control header extractor
  function(ctx)
    -- Always clear the headers.
    clear_control_headers()
    --

Users' requests have the critical data extracted from those headers and processed before the clear happens, but that code path wasn't equally populated for internal Tiered Cache traffic. There, the lower tier passes control headers through unfiltered, as needed upstream. With headers wiped clean, the logic lacked key data like hostname resolution for the internal DNS lookup to find the correct origin server IP address. The failures presented as DNS errors, returning 530 to the client.

Incident Trajectory

The bad CDN component began its deployment at 08:40 UTC to a small subset of data centers. Over the early phases, a reported increase in 500s pointed at a single small data center, which was pulled from production while engineering teams devoted their attention there. Around 12:30 UTC, the issue spread as more data centers got the release automatically, and by 14:22 UTC—when deployment reached the largest data centers—530 responses clearly escalated and multiple teams joined to trace the source.

A deciding moment arrived at 17:03 UTC when the release was rolled back in Atlanta: the errors dropped, confirming the release was to blame. The rollback was rushed out to all data centers configured as Tiered Cache upper tiers, marking the return to normal at 18:04 UTC, with the full rollback complete by 18:30 UTC.

Requests resulting in a response with status code 530

The most deceptive factor was timing. A separate internal DNS release had been under way during the same window, and the distributed-tracing error signaled wrong origin responses in a way that made DNS look like the guilty party. It was actually just a side effect. That red herring dragged out the investigation before the teams recognized the tracing component's role.

Timeline of Key Events

  • 08:40 — faulty release starts rolling out to a small subset of data centers
  • 10:35 — first customer alert fires for increased 500 error codes
  • 17:28 — peak impact: around 5% of all HTTP requests fail with 530
  • 17:03-18:04 — rollback in Atlanta confirms root cause; spread to all Upper Tiers, erasing impact

Prevention Steps

Cloudflare's announced follow-up measures center on both testing coverage and enhanced observability to catch bad releases sooner:

  • Risk exposure in staging: Including an upper-tier data center earlier in the gradual deployment would force exposure to the specific configuration caveats during a minimal rollout window instead of in full production.
  • Expanded acceptance tests: Broader topology coverage, especially across various Tiered Cache scenarios, addresses a testing gap that had hidden the side effect.
  • Better crisis telemetry: More aggressive alerts tied to missing request context—where host information in control headers is unavailable—would surface quickly.
  • Fail-fast behavior: Ensuring the system errors out loudly when resolver input is incomplete would reveal the bug during development, not just after deployment.

Nothing in the post-incident list offers ground-breaking foresight, but it targets the specific mechanical failure: the deployment of a lightweight tracing change into a deep hierarchical structure without validating each topology and failure mode.

The fix is underway, and the systems have been reverted. Cloudflare claims customer impact was confined to those using the featured Cloudflare products, but got a reminder that even observability tooling has to play by production's rules.