What Happened

On October 30, 2023, multiple Cloudflare services experienced outages for 37 minutes. The root cause was a misconfiguration in the deployment tooling used for Workers KV, Cloudflare's globally distributed key-value store. During the incident, KV returned HTTP 401 (Unauthorized) errors for all get, put, delete, and list operations against KV namespaces, instead of the requested data.

Because Workers KV is used by numerous Cloudflare products for configuration distribution, routing lookups, static asset serving, and authentication state, the blast radius was significant. Customers using affected products would have seen elevated error rates or been unable to access certain features for the duration of the incident. The specific impact varied by product, depending on how each service leverages KV.

Cloudflare also relies heavily on its own product suite internally. This dependency complicated the response effort, as tools required for the rollback themselves depended on the failing KV service.

Timeline and Detection

Automated alerting flagged the issue immediately at 19:54 UTC. However, a significant delay occurred between initial detection and the actual rollback, which did not happen until 20:15 UTC. This gap was due to the cascading nature of the failure. Cloudflare Access uses Workers KV for request verification, including the validation of user JWTs (JSON Web Tokens). With KV down, engineers could not access the internal dashboard used to revert the change, nor could they retrieve the necessary secrets from the continuous integration (CI) system to trigger an automated rollback.

Although automatic rollback procedures had been previously tested and had succeeded, the authentication failure—which stemmed from Access relying on the very service that was down—made those paths unusable. Engineers ultimately had to use break-glass mechanisms to bypass normal tooling and manually switch the production route to the previous known-good version of Workers KV. This immediately eliminated the failing request path and restored service.

Root Cause Analysis

The Workers KV team was progressively deploying a new release using a specialized deployment tool. This tool manages separate staging and production environments and typically upgrades production at progressive percentages until all environments run the latest build. However, the deployment tool contained a latent bug in how it returns releases and their respective versions. Instead of returning releases from a single environment, the tool returned a broader list, mixing production and staging releases together.

During the incident, the service was deployed and tested in staging successfully. But when promotion to production was attempted, the deployment automation incorrectly referenced a script that had been deployed to the staging account instead of the pre-production version on the production account. This caused the deployment mechanism to point the production environment to a version that was not running anywhere in production, effectively black-holing traffic. Calls to KV were directed to a version not authorized for production access, resulting in the HTTP 401 errors.

The fix was ultimately straightforward: a manual change of the production build path back to the previous, known-good deployment. This highlights that the core issue was not with the Workers KV product itself, but with the deployment tooling's assumption of a single-tenant model and its lack of validation that the deployment matched the target environment.

Post-Incident Actions

Cloudflare acknowledges that Workers KV now underpins a substantial portion of its products and services, a situation that developed organically as more teams built on the platform. This incident has underscored the need to reduce the blast radius of such critical dependencies. The company is prioritizing improvements to deployment tooling, its usability for internal teams, and product-level controls for these dependencies.

Key follow-up actions planned for the quarter include:

  • Migrating KV deployments to standardized Workers deployment models that include automated impact detection and recovery.
  • Ensuring the rollback process has access to a known-good deployment identifier and remains functional when Cloudflare Access is unavailable.
  • Adding pre-checks to deployments that validate input parameters to prevent version mismatches from propagating to production.
  • Hardening the progressive deployment tooling for multi-tenancy, as the current design assumes a single-tenant model.
  • Adding validation to progressive deployment scripts to confirm that the deployment matches the application environment (e.g., production vs. staging).