Smaller, faster, safer: running Kimi and GLM at scale

Memory is the bottleneck for long-context MoE models

Moonshot's Kimi K-series and Z.ai's GLM are among the most capable open models served on Cloudflare's Workers AI platform. Both are large, long-context mixture-of-experts (MoE) models that are difficult to serve efficiently because of GPU memory limits. Workers AI runs these models on disaggregated prefill and decode pools, and three additional layers of optimization make it possible to fit them into memory and keep latency low: quantizing the KV cache, compressing model weights, and adding integrity checks to protect the shared KV cache. All experiments and production traffic run on SGLang, an open-source inference serving framework.

Halving the KV cache with FP8

The KV cache stores attention keys and values for every token the model has already processed, letting it extend a long conversation without re-reading the entire context. For long-context models, the KV cache typically fills GPU memory before the weights do. The default storage is 16-bit (BF16) precision; Cloudflare stores it in 8-bit floating point (FP8 e4m3) instead. On Kimi K2.6, this roughly doubles the context that fits in memory, from about 686,000 tokens to around 1.37 million.

Quantizing the cache does not speed up decoding — the FP8 attention kernel converts values as it reads them, adding a small amount of work per token. The benefit is capacity: more requests can remain resident on a single GPU. Comparing attention kernels on Kimi K2.6 decoding with disaggregated H200 deployment:

At any given concurrency level, BF16 is a few percent faster per token. But BF16 exhausts the cache at 32 concurrent requests, while FP8 scales to 64 and reaches 2,192 tokens per second. That is about 41% higher than BF16's peak throughput, for roughly 30% less cost per token.

Because prefill and decode run in separate pools, the tradeoff is applied strategically. Prefill is compute-bound rather than memory-bound, so the cache stays in BF16 there to retain its slightly higher throughput. Decode, which is memory-bound, uses the FP8 cache.

Accuracy is unaffected. Across Cloudflare's evaluation suite, FP8 and BF16 caches are indistinguishable:

INT4 weights for GLM 5.2

On GLM 5.2, model weights are compressed from 8-bit floating point to 4-bit integers (INT4) with no accuracy loss. The checkpoint drops from 705 GB to 421 GB, roughly 40%. Memory per GPU across an 8-way tensor-parallel deployment falls from ~88 GB to ~52 GB, leaving room for about 1.18 million tokens of KV cache on the same hardware.

Across the evaluation suite, INT4 and FP8 weights are indistinguishable:

Smaller weights speed up decode because generating each token involves streaming weights from GPU memory, and decode is limited by memory bandwidth — moving less data means each token is emitted sooner. The effect is strongest at low concurrency, where per-request latency matters most:

Prefill behaves differently. It is compute-bound, and INT4 weights must be expanded before multiplication, making them slower there: GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. The disaggregated design lets Cloudflare run INT4 for decode and FP8 for prefill, each where it wins. Accuracy stays within 0.8 points of the FP8 model across every benchmark, making the quality indistinguishable.

Integrity checking for a shared cache

Quantization and compression both increase the number of requests sharing one GPU's memory. That efficiency relies on exact bookkeeping in paged attention, continuous batching, and cache reuse. At Cloudflare's request volumes, even a one-in-a-billion error would surface regularly, so the team built KV cache integrity checking as a defensive layer.

Every physical cache page receives a tag that changes on reallocation, and the server records which pages and tags each request expects to use. Before supported decode operations read from the cache, these mappings are checked; any mismatch aborts the affected request rather than allowing data from the wrong page to be returned.

The cost of the check was measured on a mid-sized production model in a two-prefill, two-decode configuration with 8,192-token inputs and 1,000-token outputs:

Overhead is under 1% on both throughput and tail latency, with the 95% confidence interval remaining near 1%. The validation runs as a separate batch check rather than fused into the attention kernel, avoiding a race between GPU thread groups. The feature is enabled per deployment; the default path uses a no-op tracker, so deployments that don't need integrity checks pay nothing.

Ongoing work

The team is expanding FP8 KV caches across more of the fleet, evaluating NVFP4 weights on Blackwell (NVIDIA's GPU architecture), and working toward leaving integrity checks on everywhere at negligible cost. These optimizations support more customers at lower cost at the same model accuracy.