Workers AI targets the inference bottleneck
Requests for faster large language model (LLM) generation have been a recurring theme in feedback on Workers AI. In response, the platform has rolled out three upgrades aimed at improving inference efficiency: next-generation hardware, KV cache compression, and speculative decoding. The hardware refresh, based on Cloudflare's 12th generation compute servers, supports newer GPUs that can handle larger models, including Meta Llama 3.2 11B with vision support and Meta Llama 3.1 70B. Under typical load, customers can expect two to three times the throughput for these models compared to the previous hardware generation.
KV cache compression goes open source
One of the primary constraints on LLM inference is GPU memory, or vRAM. Every token processed by a model generates vectors that capture its meaning relative to earlier tokens; these are stored in the KV cache. Memory requirements therefore scale linearly with the total number of tokens across all sequences being processed, which limits both the number of concurrent sequences and their maximum length.
The KV cache is structured hierarchically: each model layer has its own attention operation, and the vectors for a given layer are partitioned among that layer's attention heads. For a single sequence, the KV cache for one layer is an M x N matrix, where M is the number of attention heads and N is the sequence length. Each token's representation spans all KV vectors across every head and layer.
A common compression approach is eviction: identify KV vectors that are unlikely to be queried by future attention operations and remove them. This usually involves tracking past attention weights—a measure of how often each KV pair has been accessed—and evicting those with the lowest total attention, similar to a least-frequently-used cache policy.
The matrix layout creates a constraint, however. Because memory is allocated per head based on the maximum sequence length, removing an unequal number of KVs from different heads does not reduce the footprint; the freed slots simply become padding in the tensor. This forces a uniform compression rate across all heads, which limits how aggressively the cache can be shrunk without degrading quality.
The fix is to change how the KV cache maps to physical memory. PagedAttention uses an N x M block table to index into a series of blocks, allowing tokens to be stored without wasted space for padding. The table holds integer indices rather than high-dimensional vectors, so its own memory overhead is negligible. With this representation, different attention heads can be compressed at different rates, giving the eviction strategy far more flexibility.
Compression results
Testing on the LongBench suite with Llama-3.1-8B showed that most tasks retain over 95% of their performance with an 8x cache reduction. Further compression to 64x still preserves over 90% task performance. The practical effect is a higher number of concurrent requests: throughput increased by 3.44x at 8x compression and 5.18x at 64x compression. The implementation is available in a vLLM fork for those who want to experiment.
Speculative decoding: predicting more than one token
Standard autoregressive generation produces one token per model call. Prompt-lookup decoding, an implementation of speculative decoding, changes that. The technique works by matching the last n tokens of generated text against earlier text in the prompt or output. When a match is found, it predicts candidate continuation tokens based on the identified pattern. The model then verifies all candidates in a single forward pass, accepting or rejecting them in bulk.
This works well because much of natural language is highly predictable: idioms, common expressions, and standard grammar all present few high-probability continuations. For Workers AI, the method has shown measurable gains, increasing generation speed by up to 40% for llama-3.1-8b-instruct and up to 70% for the 70B model.
Speculative decoding is not free of tradeoffs. Quality typically drops relative to standard decoding, both on benchmarks like MMLU and in human evaluation, and more aggressive speculation amplifies the effect. Prompt-lookup decoding is positioned as a low-impact option, and it will be added to models including @cf/meta/llama-3.1-8b-instruct.
These changes are part of a broader push to make inference faster without requiring customers to manage their own ML research and infrastructure teams.
