Why extra-large models demand a rethink of inference infrastructure
Running a model like Moonshot’s Kimi K2.5—over a trillion parameters—is as much a hardware problem as a software one. With hundreds of gigabytes of weights to house and context windows that can stretch to millions of tokens, the conventional single-GPU serving path no longer applies. Cloudflare’s Workers AI has been moving these big open-source models into production, and the engineering behind that push is a mix of traffic shaping, cache management, and a bespoke inference engine built in-house.
Splitting prefill from decode
An LLM request has two distinct phases. Prefill ingests the input tokens and populates the KV cache; decode generates output tokens one by one. These stages have very different hardware profiles—prefill is compute-bound while decode is memory-bound—and running them on the same GPU means they block each other, leaving silicon idle. One answer is prefill-decode (PD) disaggregation, where separate inference servers handle each stage. A request lands on a prefill server first, populates its KV cache, and then moves to a decode server that pulls the cache over and starts generating.
This split lets each server be tuned and scaled for its specific workload, or even run on heterogeneous hardware. But it puts real pressure on the load balancer. Traffic must be steered from prefill to decode, and the balancer has to rewrite decode responses—including streaming SSE—so they include prefill metadata like cached token counts. Different inference backends also expect different information to start a KV cache transfer. Cloudflare extended its balancer to do token-aware routing: it estimates how many prefill and decode tokens are in flight per endpoint and distributes load accordingly. After launch, the team re-analyzed usage patterns and retuned the setup, which produced a notable drop in p90 time-to-first-token on the same GPU fleet.
Prompt caching and session affinity
Agentic workloads are context-heavy. Every turn resends the full conversation—system prompt, tool definitions, prior messages—so recomputing those input tensors each time is wasteful. Prompt caching avoids that, but only if requests land where the cache lives. Cloudflare exposes the x-session-affinity header so clients can steer requests back to the region holding the computed tensors. The company also added the header to popular agent harnesses like OpenCode, which showed a measurable throughput gain once adopted.
Internally, KV-aware routing already exists, but the explicit client header makes the intent unambiguous and comes with a pricing incentive: cached input tokens are discounted. Working with heavy internal users to adopt the header lifted input cache hit ratios from 60% to 80% during peak periods, which translates directly into more requests served per GPU.
Sharing the KV cache across GPUs
Larger models span multiple GPUs, which means the KV cache—populated on one GPU during prefill—has to be reachable from others. For Kimi K2.5, Cloudflare leverages Moonshot AI’s Mooncake Transfer Engine and Mooncake Store. The transfer engine moves data between GPUs over RDMA protocols like NVLink and NVMe over Fabric, bypassing the CPU entirely for memory-to-memory transfers. That speeds up cache sharing in multi-GPU, multi-node setups.
Paired with frameworks like LMCache or SGLang HiCache, the cache becomes cluster-wide. A prefill node can find and reuse a cache created for an earlier request on a different node, which removes the need for session-aware routing inside the cluster and lets the load balancer spread traffic more evenly. Mooncake Store also extends cache storage beyond VRAM onto NVMe, keeping sessions warm longer and improving hit ratios.
Speculative decoding with a draft model
Speculative decoding exploits the fact that a forward pass can predict several future tokens at once. A small draft model proposes a handful of candidates, and the large target model validates them in a single pass—cheaper than generating each token with the full model, while preserving quality because the target model still has final say.
This technique lands particularly well in agentic settings where tool calls and structured outputs dominate. A tool call follows a predictable shape: a name, arguments, a JSON envelope. Cloudflare pairs Kimi K2.5 with NVIDIA’s EAGLE-3 draft model and tunes the number of speculative tokens to balance acceptance rate against overhead. The result is a meaningful tokens-per-second improvement.
Infire: inference tuned for a global network
Cloudflare’s own inference engine, Infire, is written in Rust and built for the constraints of serving models across a distributed edge network. Supporting models of Kimi K2.5’s class required extending Infire in several directions.
Multi-GPU execution
A model like Kimi K2.5 has roughly 560 GB of weights, far beyond the ~80 GB VRAM of a single H100. Infire now runs across multiple GPUs using pipeline parallelism, tensor parallelism, and expert parallelism. Pipeline parallelism focuses on balancing stages so no GPU starves while others execute; tensor parallelism prioritizes minimizing cross-GPU communication. For most models, combining both offers the best throughput-latency trade-off.
Leaner memory footprint
Infire already used less GPU memory than vLLM; further optimization cut the internal state overhead—activations and related buffers—even more. The engine now runs Llama 4 Scout on two H200 GPUs with over 56 GiB free for KV cache (enough for 1.2 million tokens), and Kimi K2.5 on eight H100s with over 30 GiB left for cache. In both cases, vLLM would struggle just to boot.
Faster cold starts
Multi-GPU support surfaced boot time improvements as well. Even for Kimi K2.5, Infire can begin serving in under 20 seconds, with load time bounded mostly by drive speed.
Hardware efficiency gains
On unconstrained systems, Infire delivers up to 20% higher tokens-per-second throughput. Just as importantly, it lets lower-end hardware run models that previously would not have fit at all.
The work here is iterative. New models and research arrive weekly, and each one shifts the balance between cache, compute, and memory. The current setup is a point on that curve, not the end of it.
