How we built the most efficient inference engine for Cloudflare’s network

Inference is the engine behind chatbots, agentic AI, fraud detection, and autonomous systems. But on a hyperscaler, running those workloads means renting expensive GPUs in a few centralized data centers — a model that clashes with Cloudflare’s globally distributed edge, which sits within 50ms of 95% of the world’s Internet-connected population. To serve inference workloads efficiently at that scale, we can’t simply buy more hardware; we need to squeeze every bit of idle capacity out of the GPUs we already run.

That constraint becomes harder as models grow. Supporting the likes of the Llama 4 herd and gpt-oss pushed us to rethink our inference stack. Most of our models ran on vLLM, the widely used open-source serving engine. vLLM is excellent for dedicated inference servers on powerful hardware in large data centers, but it’s far less optimized for dynamic workloads, distributed networks, and the security constraints of running inference at the edge alongside other services. We concluded that we needed to build our own solution.

The result is Infire: an LLM inference engine written in Rust that maximizes memory, network I/O, and GPU utilization. It serves more requests with fewer GPUs and significantly lower CPU overhead. Our initial benchmarks show Infire completing inference tasks up to 7% faster than vLLM 0.10.0 on unloaded H100 NVL machines; under real load, the gap grows considerably. Infire now powers the Llama 3.1 8B model for Workers AI — test it at @cf/meta/llama-3.1-8b-instruct.

Why vLLM didn’t fit our edge

Inference has improved dramatically, largely thanks to vLLM’s V1 engine, with optimizations like an efficient KV cache, better batching, and Flash Attention 3. But vLLM is best tuned for dedicated, data-center-class deployments. Cloudflare’s needs diverge in a few critical ways.

First, vLLM doesn’t support co-hosting multiple models on one GPU without Multi-Instance GPU (MIG). We need to dynamically schedule several models on a single GPU to minimize downtime and utilize every idle cycle. Second, our infrastructure is increasingly written in Rust, whereas vLLM’s Python core leaves CPU performance on the table; multiple abstraction layers make low-level control difficult. Writing a server in Rust gives us direct control over those implementation details.

Third, security. Workers AI runs on our edge nodes alongside other services, and we won’t trust an un-sandboxed third-party Python process in that environment. Running vLLM under gvisor, our standard sandbox, exacts a heavy toll: under full edge load, vLLM via gvisor consumes up to 2.5 CPU cores, competing with critical services and driving down GPU utilization. vLLM also brings long startup and teardown times. Our priorities diverged from vLLM’s upstream goals, so we built a server tailored to our needs, with an in-house AI Research team exploring features that would be impractical to upstream.

Inside Infire’s architecture

Infire has three main components: an OpenAI-compatible HTTP server, a batcher, and the core engine.

Fast, parallel startup

When our auto-scaling service schedules a model to an edge node, Infire first fetches the weights from R2 object storage, caching them locally for future reuse. Model weights are large — most non-quantized models use BF16, which takes two bytes per parameter. Llama 3.1 8B needs roughly 16 GB; Llama 4 Scout, with 109B parameters, requires about 218 GB. Infire loads these efficiently using page-locked (pinned) host memory combined with CUDA asynchronous copies across multiple streams.

While the weights transfer, Infire just-in-time compiles the necessary CUDA kernels based on the model’s parameters — hidden state size, dictionary size, and the target GPU — and loads them onto the device. Parallelizing compilation with the weight transfer amortizes both latencies; loading Llama-3-8B-Instruct from disk takes just under four seconds.

The token pipeline

The HTTP layer, built on the hyper crate, handles hundreds of concurrent connections with modest CPU use. Requests arrive with prompts — text or a chat transcript — along with generation parameters like temperature. The server passes each prompt to a tokenizer, which converts raw text into tokens; we use HuggingFace’s tokenizers crate for byte-pair encoding. Those tokenized prompts head to the batcher for GPU scheduling.

Batching to feed the Tensor Cores

Inference is dominated by vector-by-matrix multiplication: huge weight matrices are fetched from memory, and the memory-transfer cost swamps the arithmetic. Nvidia GPUs have dedicated Tensor Cores for matrix-by-matrix operations, but those only pay off when operations are aggregated into large multiplications. Batching is how we make that happen.

Infire uses two complementary techniques. Prefill processes all prompt tokens in parallel, since they’re known in advance and need no decoding — this is why input tokens are cheaper and faster than output tokens. Batching aggregates multiple prompts into a single decode operation.

Infire runs continuous batching with chunked prefill: process as many prompts as possible in parallel, and use the remaining batch slots for prefill tokens from incoming requests. As tokens decode, the batcher retires prompts that emit an End of Stream token and returns tokens to the decoder for text conversion.

KV-cache management is another batcher responsibility. Attention must revisit all previously computed KV values for each decoded token; without a cache, long contexts would explode runtime. But pre-allocating the full context window for each request is wasteful: Llama’s 128K-token window would limit an H100 to only four concurrent prompts. Paged KV caching splits the cache into smaller pages and assigns new pages only as required, enabling near-unlimited parallelism under typical load. The batcher also drives the forward pass by scheduling kernels on the GPU.

Rust-precise CUDA kernels

Building our own engine lets us target our exact hardware — currently Nvidia Hopper GPUs. Infire compiles kernels with low-level PTX instructions tuned for that architecture, and for large matrix multiplications it falls back to cuBLASLt when that library proves faster.

Infire also relies on very fine-grained CUDA graphs, creating a dedicated graph for every possible batch size on demand and caching them for future launches. A CUDA graph replaces a series of individual kernel launches with a single construct, significantly reducing amortized kernel-launch cost — so back-to-back kernels execute faster as a graph than as separate launches. That kind of low-level control — from pinned memory to graph caching — is exactly why Infire is the right engine for our edge.

Measuring Infire against vLLM

To validate Infire's design, we ran synthetic benchmarks on an edge node equipped with an H100 NVL GPU. The test used the widely adopted ShareGPT v3 dataset, comprising 4,000 prompts at a concurrency of 200. We compared Infire against vLLM in two configurations: running directly on bare metal, and running under gvisor—the same sandboxing approach we use in production today. To reflect real-world conditions where an edge node shares resources with other traffic, the gvisor-based vLLM benchmark was constrained to a single CPU.

The results confirm our initial objective: Infire matches and slightly surpasses vLLM's throughput. More importantly, it achieves this with significantly lower CPU consumption, largely because Infire operates as a trusted bare-metal process. By not competing with other services for compute resources, inference no longer drains capacity from our edge infrastructure. We're seeing GPU utilization above 80%, which directly translates into reduced operational costs.

Future optimizations

This is only the first milestone. Several proven performance techniques remain on our roadmap for Infire, including integration with Flash Attention 3 and broader adoption of kernel fusion across our operator set. These enhancements promise to unlock further gains in inference speed.

Infire represents our approach to running AI efficiently, close to users worldwide. By leveraging continuous batching, a paged KV-cache, and low-level optimizations tailored to our specific hardware, we maximize GPU utilization while minimizing overhead. Compared to our previous vLLM-based deployment, Infire completes inference tasks faster and with a fraction of the CPU load—particularly under the strict security constraints our production environment demands. The net result is the ability to serve more requests with fewer resources, making every request handled through Workers AI faster and more cost-effective.

Looking ahead, we're planning multi-GPU support for larger models, quantization, and true multi-tenancy in the next iteration of Infire. These efforts align with our broader goal of positioning Cloudflare as the premier platform for developers building AI applications. To test whether your AI workloads benefit from our infrastructure, get started with Workers AI today.