Workers KV: A Post-Mortem and a Path Forward
Workers KV has experienced a series of incidents over the last two months, culminating in three back-to-back events during the week of July 17th, 2023. This disrupted service for many external customers as well as internal teams at Cloudflare, who rely on KV for products like Cloudflare Access and Cloudflare Pages. The impact was not acceptable, and we're sharing the details of our investigation, the root causes we've identified, and the concrete steps we are taking to improve how we deploy changes to KV.
What Went Wrong
KV is currently transitioning to a new architecture designed to address customer feedback on latency for infrequently accessed keys and to tighten the upper bound on the eventual consistency model for writes. By mid-June, we had begun ramping up production traffic onto this new backend. Each increase in traffic volume, however, surfaced previously unseen problems, leading to a pattern of rolling back, fixing bugs, and retrying. This process proved unsustainable. The series of incidents stemmed from two challenging root causes that were difficult to track and observe due to an observability gap in our error reporting and a mutation of local state that unexpectedly impacted global state.
To understand the failure, it's important to know that KV is composed of two Workers scripts: the Storage Gateway Worker and SuperCache. SuperCache is an optional path in the Storage Gateway Worker workflow and is the foundation for KV's new, faster backend.
Here is a timeline of the events:
| Time | Description |
|---|---|
| 2023-07-17 21:52 UTC | Cloudflare observes alerts showing 500 HTTP status codes in the MEL01 data-center (Melbourne, AU) and begins investigating. We also begin to see a small set of customers reporting HTTP 500s being returned via multiple channels. It is not immediately clear if this is a data-center-wide issue or KV specific, as there had not been a recent KV deployment, and the issue directly correlated with three data-centers being brought back online. |
| 2023-07-18 00:09 UTC | We disable the new backend for KV in MEL01 in an attempt to mitigate the issue (noting that there had not been a recent deployment or change to the % of users on the new backend). |
| 2023-07-18 05:42 UTC | Investigating alerts showing 500 HTTP status codes in VIE02 (Vienna, AT) and JNB01 (Johannesburg, SA). |
| 2023-07-18 13:51 UTC | The new backend is disabled globally after seeing issues in VIE02 (Vienna, AT) and JNB01 (Johannesburg, SA) data-centers, similar to MEL01. In both cases, they had also recently come back online after maintenance, but it remained unclear as to why KV was failing. |
| 2023-07-20 19:12 UTC | The new backend is inadvertently re-enabled while deploying the update due to a misconfiguration in a deployment script. |
| 2023-07-20 19:33 UTC | The new backend is (re-) disabled globally as HTTP 500 errors return. |
| 2023-07-20 23:46 UTC | Broken Workers script pipeline deployed as part of gradual rollout due to incorrectly defined pipeline configuration in the deployment script. Metrics begin to report that a subset of traffic is being black-holed. |
| 2023-07-20 23:56 UTC | Broken pipeline rolled back; errors rates return to pre-incident (normal) levels. |
Our initial alert on July 17th was for HTTP 500 status codes in the MEL01 data-center (Melbourne). This coincided with three data centers coming back online after maintenance. It wasn't immediately clear if the issue was KV-specific or related to the data centers themselves. Alerts for similar failures in VIE02 (Vienna) and JNB02 (Johannesburg) followed. We made the decision to disable the new backend globally at 13:51 UTC that day.
After the initial incident, we attempted to reduce the scope of impacted accounts using an allow-list configuration. However, while rolling out a change for the Storage Gateway Worker on July 20th, an older configuration was inadvertently progressed, re-enabling the new backend. A subsequent attempt to manually fix the deployment at 23:46 UTC passed a malformed configuration value, sending traffic to an invalid Workers script configuration. All deployments and the broken configuration pipeline were rolled back by 23:56 that evening.
Over the following three days, we worked to identify the root cause. We found that KV's Worker script was throwing an unhandled exception very early in the request-handling process. This was exacerbated by a previous mitigation that disabled error reporting in a data-center due to noise generation, which had resulted in logs being rate-limited upstream. This prevented us from capturing meaningful logs or identifying the exception, as an uncaught exception terminates request processing.
We also discovered that the KV Worker script failed to re-enter its healthy state when a data-center was brought back online. The Worker was mutating an environment variable perceived to be in request scope, but it was actually in global scope, persisting across requests. This effectively left the Worker "frozen" with the previous, invalid configuration.
Furthermore, a new progressive release process designed to de-risk rollouts prolonged the incident. A bug in its deployment logic, which was incorrectly defined and untested at scale, caused us to drop a single-digit percentage of traffic until it was rolled back 10 minutes later. We need to spend more time hardening this process before using it as the default path in production.
Assessment of Impact
The root cause of the incidents was confined to three data-centers used as regional tiers for KV's new backend. Consequently, a portion of traffic across the Oceania, Europe, and African regions was affected. Only some keys from enrolled namespaces use any given data center as a regional tier to limit a single point of failure, so not all traffic in a given location was impacted. We estimate the affected traffic was 0.2-0.5% of KV's global traffic, but some customers saw error rates approaching 20% of their total KV operations.
KV's high total traffic volume and its role as a critical dependency for many customers amplify the impact of even small error rates. In all cases, once the changes were rolled back, errors returned to normal levels and did not persist.
A Framework for Safer Software
To address these failures, we're applying principles analogous to the "hierarchy of controls" used in traditional engineering. The hierarchy, from most to least effective, is: elimination, substitution, engineering, administration, and personal protective equipment (PPE). In this context, code review is the "PPE" — a useful last line of defense, but an insufficient control on its own. Engineering controls, such as automation, gradual rollouts, and detailed metrics, are vastly safer ways of shipping software.
We're taking a similar approach by prioritizing engineering and automation improvements over human processes. This includes eliminating unnecessary changes, building more resilience into our codebase, and improving our deployment tooling. We see tooling as the most powerful mechanism for preventing incidents, with process improvements serving as an additional layer of protection, not the sole mitigation.
Our Plan for Improvement
We have identified and prioritized several efforts to improve KV’s reliability and meet our availability SLOs. This list is not exhaustive, and we are continuing to expand on preventative measures.
- Improve observability tooling: Substantially improve existing tooling for unhandled exceptions, especially for high-volume services where traditional logging is too noisy. We have directly addressed the specific uncaught exception in KV's primary Worker script.
- Improve environment variable safety: Increase safety around the mutation of environmental variables in a Worker. These currently operate at global (per-isolate) scope but can appear to be per-request, which is unexpected. Any changes must consider backwards compatibility.
- Expand test coverage: Broaden KV's test coverage to simulate traffic from any source data-center, which would have allowed us to more quickly reproduce and identify the root cause.
- Improve release process: Enforce a higher level of scrutiny for reviews and approvals, reduce the number of changes deployed at once, and set a higher bar for new infrastructure dependencies.
- Enhance logging: Add sampling throughout the request-handling process to improve troubleshooting, especially for non-2xx requests.
- Review alerting thresholds: Evaluate and improve alerting thresholds around error rates to ensure that sub-percentage global errors, which can have a severe impact on specific users or locations, are not lost in the noise.
- Address tooling maturity: Resolve maturity issues with our new progressive deployment tooling for Workers before it becomes available to customers.
We recognize that KV hasn’t lived up to expectations. Because we depend on it so heavily internally, we've felt that pain as well. The work to fix the issues that led to this cycle of incidents is already underway, and it will improve the reliability not just of KV, but of any software written on the Workers platform. These changes require careful planning and execution to ensure we do not repeat the mistakes of the past.
