Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

On September 8, 2026, Cloudflare was alerted by security researcher Gadi Evron of Knostic to a tenant isolation issue in the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3). The exposure window ran from August 27, 2026 until remediation, and the model was restored on September 9. No other model hosted on Workers AI was affected.

During that window, a small number of customers received responses containing data from other customers' recent requests to the same model. In total, 155 responses were affected — all of them to prompts longer than 681,280 tokens. Excluding Cloudflare's own testing and the researcher's, the identified requests match the timing and volume of ordinary long-context use rather than repeated exploitation. No evidence of deliberate exploitation was found.

The report is published as a coordinated disclosure alongside a post from SGLang, the open-source inference engine Cloudflare runs. It covers the bug, the response timeline, the hardening applied to production SGLang deployments, and ongoing work with SGLang's upstream maintainers.

Root cause: a configuration mismatch and an engine bug

GLM is Z.ai's text generation model built for agentic coding workflows and offered through Workers AI. Evron submitted his report through Cloudflare's HackerOne program on September 8. The issue was reproduced the following morning, and the model was taken offline within 40 minutes of the incident being declared. A corrected configuration and a patched inference engine followed, with service restored the same day.

Two overlapping problems produced the cross-request KV cache exposure.

The first was a misconfiguration. GLM 5.3 was deployed with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs held only 681,280 tokens. Cloudflare knew about the mismatch before launch and expected the engine to return an error for any prompt it could not process. Long-context tests returned successful responses, so the gap was not treated as a problem.

The second was an SGLang bug in how over-length requests were handled in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and swapped for a one-token placeholder, but the request was not aborted and no client error was returned. Prefill generated the KV cache for the placeholder and transferred it to the decode node. Decode received a "transfer complete" message and assumed prefill had overwritten the KV cache from previous requests with the full prompt, when in fact most of the previous state was left intact. Decode then generated a response while attending over stale cache pages still holding data from earlier requests.

Fixes and upstream patches

With the model disabled, two changes were made before public access was restored.

The pool was resized to hold the full context. The GLM 5.3 configuration was aligned with the GLM 5.2 configuration, freeing enough GPU memory that the prefill KV pool now holds about 2.04 million tokens — roughly twice the 1,048,576-token context length. Any prompt that fits the context window now fits the pool, so the rejection path is never reached for a valid request.

The rejection path was made safe in the engine. Pool sizing alone is fragile: changes to quantization, tensor parallelism, CUDA graph sizes or a future model version can alter pool size in ways that are easy to miss, and other situations may let the prefill scheduler reject a request and trigger the same bug. The patch retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation or KV transfer occurs. It was proposed upstream as #38704, which also has the decode node verify that the number of tokens transferred matches the number it reserved. The same fix was already on SGLang main via #36988, merged September 6 for a related failure path but not yet released. Maintainers also merged #38935, which rejects these requests earlier following Cloudflare's approach.

Both changes were validated against the researcher's reproduction script and Cloudflare's own canary-marker tests before the model was restored.

Why the shared KV pool needs ownership tracking

Workers AI serves GLM 5.3 with SGLang in a disaggregated layout: prefill GPUs build the KV cache for a prompt and hand it over RDMA via Mooncake to decode GPUs that generate the response.

That cache is a shared pool of fixed-size pages in GPU memory. A request gets pages when admitted and releases them when finished. Pages are not zeroed between requests — the next request to receive a page simply overwrites it. This is common practice among inference engines and necessary for performance, but it makes the engine responsible for tracking page ownership. Ownership and lifetime tracking must handle shared ownership within a node (KV cache offload and token generation access the same memory with different lifetimes) as well as ownership technically shared with the remote prefill node, which holds a simultaneous mutable reference to the page.

Pool size is also not a configuration value: SGLang sizes it at startup from whatever GPU memory remains after model weights load. Context length is configured separately, and nothing forces the two to agree.

The failure path in detail

On the GLM 5.3 prefill nodes, the pool came to 681,280 tokens against a 1,048,576-token context. SGLang prints the chosen pool size in startup logs and exposes it as a metric. The mismatch was known, but pre-launch benchmarking showed successful HTTP status codes and a streaming body with no error frames, which was taken to mean the engine handled the difference gracefully through streaming or some other mechanism. Testing had only validated response codes and engine performance at long contexts, not output correctness and coherence. The check now runs directly against the running engine, comparing KV pool size to context length before any disaggregated deployment change ships.

SGLang addressed the handling of prompts longer than the pool in v0.5.21. Previously, the scheduler rejected an over-length prompt via Req.set_finish_with_abort, which recorded the error for the client and swapped the prompt for a one-token placeholder so the request could drain through the batch machinery without allocating real memory. On single-node deployments this worked: the placeholder ran, the batch finished, and the client got an error.

In disaggregated mode, the admission queues on prefill and decode checked whether a request was finished, not whether it was rejected, and a rejected request reported itself as live:

1. Bootstrap. The router told decode the prompt length (8 pages in the diagram), and decode reserved 8 KV pages.
2. Prefill admission. The queue admitted any request where req.finished() was False. A rejected request stored its rejection in to_finish, not finished_reason, so finished() still returned False, and the request was admitted.
3. Prefill forward pass. Rejection had already replaced the prompt with a one-token placeholder, so prefill ran a trivial forward pass on that token.
4. KV transfer. Prefill sent one page of KV to decode and signaled the transfer as complete.
5. Decode admission. Decode's queue had the same finished() check and accepted the handoff.
6. Decode attention. Decode began generating and attention read all 8 reserved pages. Only page P0 had been written by this request; P1 through P7 still held earlier requests' prompts and partial generations.
7. Response. The client received HTTP 200 rather than HTTP 400. prompt_tokens matched the long prompt and the reply was coherent, arriving faster than an 8-page prefill should take and containing data from other users' KV pages.

Every request above the threshold behaved this way. Another user had reported the bug to SGLang as issue #30233 on July 6, 2026; Cloudflare noticed it while investigating. Cloudflare has been flagging these kinds of issues with SGLang since v0.5.10, and this incident accelerated work already underway with the SGLang team. The patch shipped in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22. Operators depending on SGLang should update to v0.5.22 or later.

GLM 5.3 was the only model affected. Every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path. The engine fix and the hardening described above have been applied to every model Cloudflare serves with SGLang, and deployment checks require them for every new deployment.

Timeline

All times UTC.

Hardening the KV cache path

The incident shares a family resemblance with a class of KV cache memory-safety bugs that Cloudflare has spent several months closing. In each variant, a page is freed, reused, or read while another party still holds a claim on it.

One cause lives inside a single node: token generation, offload to host memory, the KV transfer receiver, and the abort and error paths can all hold the same page concurrently. If one of them frees it while another is still working, a fresh request may be handed that page mid-read or mid-write. Earlier fixes for decode offload (#24226, #30899) and for double release on grammar errors (#30937) fell into this category.

The other cause crosses nodes. RDMA writes that prefill has already dispatched cannot be recalled when a request is aborted, so if decode frees those pages immediately, a new request can claim them and the late writes land in its cache. Cloudflare contributed fixes (#31357, #38961) and a reproduction script (#32564) for that race. Both variants produce silent corruption: bad output, or one user's data surfacing in another user's response.

What set this incident apart is that the rejected request passed through the handoff reliably rather than only under unlucky timing, leaving decode to read pages the request had never written. Earlier work did not cover a failure in which prefill rejects the request and decode never hears about it.

Cloudflare has since:

• Broadened deferred release so it retains prefill's source pages too and applies to all failure types, not just aborts. Proposed upstream as #38961, it was superseded by #40645, #41402, and #41404.
• Deployed bounded Mooncake transfer batches (#32758, contributed by the SGLang community) so a single transfer call cannot exceed the release timeout.
• Reworked the load balancer so a client disconnect no longer aborts a request mid-transfer; it now holds the prefill connection until prefill completes and the decode connection until the first token, then closes both.
• Added a validation gate to every disaggregated deployment change: KV pool size against context length, deferred release on both roles, transfer batch bounds within the release timeout, and all required patches present.

These changes are live in Cloudflare's disaggregated production deployments.

Upstream coordination with SGLang

Cloudflare met the SGLang maintainers at RadixArk on September 10, the day after the incident. RadixArk agreed to investigate the reported races, upstream the integrity-checking concept, and build reproduction tests for the bug class.

Patches addressing this incident and related KV cache races have landed on SGLang main, and all but one ship in v0.5.22. Anyone running disaggregated SGLang should move to v0.5.22 or backport the patches.

Work that remains open:

• #39729 — further cleanup for --disaggregation-enable-kv-checksum. Merged, not yet released.
• Zeroing KV pages on release, draft proposal: #46435
• Extending KV checksum validation from the prefill-to-decode handoff to every decode step
• Wider model and feature coverage for KV Canary and KV Checksum

Operator guidance

For those running SGLang in disaggregated prefill/decode mode:

1. Check KV pool size against context length on every prefill and decode node, reading it from the engine's startup log, and repeat the check after any change to weights, quantization, parallelism, or memory fraction. If the pool is smaller, reduce context length to fit. Neither a proxy nor the engine can be assumed to reject what does not fit — this is the check that was missing.
2. Upgrade to a build containing #36988 and #38935 (v0.5.22 or later), so over-length requests are rejected before handoff regardless of pool size. This is the engine-side backstop for when the first check is wrong.
3. Enable deferred KV release on both roles (the default after #41023) and bound Mooncake transfer batches so a single timed-out call cannot leave an unbounded tail of writes.
4. Turn on corruption detection where supported. --disaggregation-enable-kv-checksum and --kv-canary do not cover every architecture yet; verify from the logs that requests are actually being checked. Cloudflare and the SGLang team are extending both to more architectures, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.
5. Stop your proxy from issuing needless aborts. Let requests run through to the first token so you do not trigger aborts that require cross-node coordination.

Longer-term defenses

Integrity by construction

Configuration checks and lifecycle fixes shrink the space for page misuse but cannot eliminate it; the next bug will take a shape not yet seen. The durable answer is for the engine to detect at read time that a request is about to attend over a page it does not own, and to fail that single request instead of returning another user's data.

Cloudflare prototyped this over the summer and described it in August. Each physical KV page carries a tag that changes on reallocation; each request records the tags it expects, and the mapping is verified before decode reads the cache. Testing showed under 1% cost in throughput and tail latency. The prototype was exercised on GLM 5.2 in mid-August but disabled shortly after due to reliability issues, and because it was tied to particular attention backends and model architectures it could not serve as a general fix. The goal now is a production version built into the engine across all models rather than Cloudflare-specific patches. The per-transfer checksum in #39500 validates the prefill-to-decode handoff; page-level tags go further by checking every read, including reads long after the transfer.

Integrity by mathematical impossibility

RDMA moves data directly between node memory, so the sender and receiver must agree on destination, size, and completion; any disagreement can make the receiver read the wrong data. This bug is a consistency failure of exactly that sort, with the two machines holding inconsistent views of system state.

Cloudflare already applies formal verification elsewhere: Topaz provides conflict-free IP selection for its authoritative DNS servers, and models of Meerkat's QuePaxa implementation exposed consistency bugs that were subsequently fixed.

That experience is now being directed at KV cache transfer engines, starting with modeling and proving the safety of an RDMA-based KV cache transfer protocol. The plan is to model production SGLang's distributed KV cache using Mooncake, then track and fix any divergence from the safety properties being guaranteed, aiming to prove as far as possible that such a bug cannot recur. Gaps in what the provers can establish are expected, but proving the remainder lets the team review and monitor those gaps closely.

Disclosure

This report accompanies a coordinated disclosure. Cloudflare thanks researcher Gadi Evron, who reported the issue through its HackerOne program with a clear, reproducible script and allowed time for fixes and coordination with the SGLang maintainers before publication. It also thanks the SGLang team at RadixArk for treating the issue as their own and landing upstream fixes with urgency.

The SGLang team's account is available at lmsys.org. Security issues in Workers AI can be reported at hackerone.com/cloudflare.