A Missing Shell Flag and a Five-Second Global Slowdown

For roughly 30 minutes on December 16, 2021, Cloudflare was slow. Between 20:10 and 20:40 UTC, web requests were artificially delayed by up to five seconds before being processed. The cause turned out to be a chain of unlikely failures, starting with a missing shell option called pipefail.

The Players

Cloudflare's Front Line service protects against large-scale attacks. Orchestrating that protection is dosd, a sidecar service running on every Cloudflare server. When dosd detects an attack, it provides Front Line with fingerprints describing how to match and block the malicious traffic.

dosd instances communicate peer-to-peer. This decentralized design requires very fast access to large amounts of configuration data, which comes from Quicksilver, Cloudflare's global configuration replication system.

One piece of configuration data dosd needs is addressing data from the Addressing API, which is source of truth for IP address management. Since addressing data changes infrequently, a Kubernetes cron job queries the API at 10 minutes past each hour and writes the result into Quicksilver.

The Change and Rollback

On December 16, the Front Line team released a fix for a bug in compression handling in the presence of a Cache-Control: no-transform header. The team quickly realized the fix broke customers who had come to depend on the old behavior, so they decided to roll back.

The rollback was manual, performed in two batches as a safety measure. The first batch targeted tier 2 and 3 data centers at 19:25 UTC and completed in about 30 minutes. After confirming no issues, SREs started the second batch at 20:10 UTC. Within minutes, alerts fired: traffic levels dropping, CPU dropping, and a P0 incident declared.

The rollback was paused, but that didn't help. About 10 minutes in, the DOS team got a critical alert: dosd was not running on numerous servers.

Empty Data, Down Service

Service logs showed dosd was panicking because the customer addressing data in Quicksilver was corrupted. Actually, the Quicksilver key was simply empty. Without valid data, dosd couldn't make correct choices, so it refused to continue.

The fix was simple: manually re-run the Kubernetes cron job that populates the key. At 20:40 UTC, the job completed, dosd started running again, and traffic returned to normal.

How the Key Went Empty

The cron job uses a small Bash script. The dos-make-addr-conf executable queries the Addressing API, serializes JSON data into a TOML document, and writes it into config.toml. That TOML is then piped into dosctl, which writes it to the Quicksilver key template_vars.

Here's the bug: if dos-make-addr-conf fails and exits with a non-zero error code, the shell pipeline ignores that error and continues executing by default. The output of the failed command — potentially empty — gets piped unconditionally into dosctl, clobbering the template_vars key.

The bash pipefail option, introduced decades ago to address this exact problem, changes pipeline behavior so that if any command fails, the entire pipeline fails. It is not enabled by default, which is why best practice recommends enabling it at the start of every script.

This bug was especially insidious because dosd did attempt to gracefully handle invalid TOML in the key. But an empty string is a perfectly valid TOML document. Had an error message been written instead, dosd would have rejected the update and continued using the previous value.

From Empty Key to Slow Traffic

Front Line doesn't depend on dosd directly to serve requests. Every few seconds, it asynchronously fetches fresh attack fingerprints and stores them in an in-memory cache, consulted while serving each request. If dosd fails, stale fingerprints continue to be used.

However, the rollback process required reloading Front Line's code, which flushed those in-memory caches, including the attack fingerprint data. The next request that consulted the cache got a "cache miss." The caching layer then tried to reach dosd, and while waiting for a reply, it blocked all pending requests from progressing.

Since dosd was down, the attempt timed out after five seconds. Every pending request was stuck waiting for that timeout. Once it fired, all the queued requests became unblocked, and the cycle repeated every five seconds on every server until the dosd failure was resolved.

Both conditions had to be met: dosd had to fail and Front Line's cache had to be flushed at the same time. Without the flush, stale fingerprints would have kept request processing running normally.

Why the First Rollback Was Safe

The rollbacks were performed by forcing servers to run a Salt highstate. Each batch ran highstate across thousands of servers simultaneously, contacting the Addressing API for customer addressing information.

The second batch started at 20:10, exactly when the highstate flood hit the Addressing API. All the requests were queued and eventually served, but the delay was long enough to time out the cron job, which runs only at 10 minutes past the hour. With pipefail not enabled, that timeout resulted in the empty value being written to Quicksilver.

If the second batch had been started even a few minutes earlier or later, the flooding wouldn't have coincided with the cron job's execution, and the bug would have stayed dormant.

Hardening the Chain

This incident involved several small or unlikely failures cascading into a severe outage. Cloudflare has hardened each link in that chain:

  • The thundering herd: The manual rollback flooded the Addressing API with requests. It has since been significantly scaled out to handle high request rates.
  • The cron job bug: It is no longer possible for the cron job to clobber the Quicksilver key when it fails.
  • dosd error handling: Additional error conditions when loading configuration data from Quicksilver are now taken into account so dosd gracefully degrades in the face of corrupt data.
  • Front Line's dependency: All unexpected dependencies on dosd have been removed, so Front Line survives dosd failures.

The broader lesson is that systems must remain resilient to failure, no matter how unlikely that failure may appear. Even a missing shell option can, under the right circumstances, slow down traffic for millions of users.