Faster inference, batch workloads, and broader fine-tuning land on Workers AI
Since Workers AI launched in September 2023, the platform has focused on making inference broadly accessible. Recent work has centered on routing improvements, GPU optimizations, and capacity management. The common thread across these changes is an emphasis on engineering-led efficiency gains rather than simply adding more hardware.
Several updates are rolling out now: speculative decoding for faster responses, an asynchronous batch API for large-scale workloads, and expanded LoRA support. New models, pricing changes, and a redesigned dashboard round out the release.
2-4x faster generation on Llama 3.3 70B
Speed improvements are being applied to models in the catalog, starting with @cf/meta/llama-3.3-70b-instruct-fp8-fast. The optimizations include speculative decoding, prefix caching, and an updated inference backend, delivering 2-4x faster inference times without a meaningful change in output quality. More models will receive these improvements as they are released.
How speculative decoding works
LLMs generate text by predicting the next token given previous tokens, typically one token per forward pass. For a 70 billion parameter model like Llama 3.3 70B, each pass is computationally expensive.
Speculative decoding places a smaller draft model in front of the main model. The draft model proposes a set of candidate tokens, and the main model evaluates them in parallel. Since evaluating multiple tokens in a single forward pass is less costly than generating them one by one, inference can be 2-4x faster.
This approach also makes better use of GPU resources. LLMs tend to be memory-bound, leaving compute idle; speculative decoding fills that gap by running the draft model on otherwise unused capacity.
Prefix caching to cut pre-fill time
LLM generation has two phases: pre-fill, which processes input tokens like prompts and context, and decoding, which produces output. Prefix caching targets the pre-fill phase.
For example, a code generation request that includes a full file as context would normally re-process that file for each new request. With prefix caching, the pre-fill tokens are stored, so repeated context is only processed once. This is particularly useful for RAG, code generation, and chatbot applications that reuse the same context, reducing response times and resource usage.
Quality validation
Because this is an in-place update to an existing model, the team took care to avoid breaking existing applications. Internal blind A/B testing and feedback from internal and external customers confirmed output quality and response format compatibility. Note that LLMs are not perfectly deterministic even with identical inputs; any issues can be reported via Discord or X.
Async batch API for large workloads
A new asynchronous batch API is available for customers with substantial inference needs. Instead of erroring out under capacity pressure, requests are fulfilled at a later time and returned as a batch — useful for jobs like summarizing many documents at once.
Batch workloads are a challenging use case for serverless platforms. Summarizing a 70-page PDF might involve chunking and sending ~280 inference requests; multiplied across many documents and concurrent users, this becomes a sudden flood of thousands of requests. While these workloads aren't latency-sensitive, they do require completeness guarantees.
The async API addresses this by accepting a batch of requests and promising to fulfill them, returning results as a batch. It also isolates real-time traffic from background jobs, preventing queues from building up behind batch work.
Batch inference is currently supported for these models:
@cf/meta/llama-3.3-70b-instruct-fp8-fast@cf/baai/bge-small-en-v1.5,@cf/baai/bge-base-en-v1.5,@cf/baai/bge-large-en-v1.5@cf/baai/bge-m3@cf/meta/m2m100-1.2b
Batch requests are sent by adding a flag to the request:
let res = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-batch", {
"requests": [{
"prompt": "Explain mechanics of wormholes"
}, {
"prompt": "List different plant species found in America"
}]
}, {
queueRequest: true
});Currently, the API accepts an array of requests and returns an array of responses in a single HTTP transaction. Submission returns a request ID that can be polled for status. Future iterations plan to support queue-based inputs and outputs, integrating with Event Notifications and Workflows so subsequent actions can run when responses are ready. Developer docs and a template for deploying a Worker that uses the batch API are available.
Expanded LoRA support and new models
LoRA (Low Rank Adaptation) adapters allow a trained adapter file to modify a model's responses without full fine-tuning. Adapters are cheaper to train and far smaller than full model weights, making them a practical way to customize output style.
LoRA support now covers more models, with ranks up to 32 and adapter files up to 300 MB. Supported models include @cf/meta/llama-3.2-11b-vision-instruct, @cf/meta/llama-3.3-70b-instruct-fp8-fast, @cf/meta/llama-guard-3-8b, @cf/meta/llama-3.1-8b-instruct-fast, @cf/deepseek-ai/deepseek-r1-distill-qwen-32b, @cf/qwen/qwen2.5-coder-32b-instruct, @cf/qwen/qwq-32b, @cf/mistralai/mistral-small-3.1-24b-instruct, and @cf/google/gemma-3-12b-it. Some of these are marked as coming soon.
To use LoRA, you need a trained adapter or one from HuggingFace. Upload both adapter_model.safetensors and adapter_config.json to your account via wrangler commands or the REST API. LoRA files are private and scoped to your account, after which you can run fine-tuned inference.
const response = await env.AI.run(
"@cf/qwen/qwen2.5-coder-32b-instruct", //the model supporting LoRAs
{
messages: [{"role": "user", "content": "Hello world"}],
raw: true, //skip applying the default chat template
lora: "00000000-0000-0000-0000-000000000", //the finetune id OR finetune name
}
);A cleaner view of usage and pricing
Workers AI has also seen a wave of usability updates. A revised pricing model now expresses usage in familiar units — tokens, audio seconds, image size and steps — while billing continues in neurons behind the scenes. The new dashboard, built on Workers Observability components, shows both unit consumption and neuron totals. Model pricing is displayed directly on the models page in the developer docs, and AI Gateway users now see Workers AI usage surfaced as metrics.
New models and notable updates
More than ten models have joined the Workers AI catalog recently, with pricing listed on each model's docs page. Four of these are releasing today:
@cf/mistralai/mistral-small-3.1-24b-instruct— a 24B parameter model with vision and tool calling support, claiming capabilities comparable to larger models.@cf/google/gemma-3-12b-it— a 12B model for text generation and image understanding tasks like Q&A, summarization and reasoning, with a 128K context window and multilingual support across 140+ languages.@cf/qwen/qwq-32b— a medium-sized reasoning model positioned as competitive with DeepSeek-R1 and o1-mini.@cf/qwen/qwen2.5-coder-32b-instruct— a code-focused model whose coding ability is said to match GPT-4o.
Several other models were added earlier and are also now generally available or newly priced:
@cf/deepseek-ai/deepseek-r1-distill-qwen-32b— a Qwen 32B variant distilled from DeepSeek’s R1 that performs chain-of-thought reasoning.@cf/baai/bge-m3— a multilingual embeddings model supporting over 100 languages, with dense, multi-vector and sparse retrieval in a single pass, across inputs of varying granularity.@cf/baai/bge-reranker-base— the platform’s first reranker, scoring the similarity between a query and context; in RAG workflows it can reorder initial vector search results to surface the most relevant documents.@cf/openai/whisper-large-v3-turbo— a faster, more accurate speech-to-text model that is now out of beta with published pricing.@cf/myshell-ai/melotts— the first Workers AI text-to-speech model, generating MP3 voice audio from text input.@cf/meta/llama-4-scout-17b-16e-instruct— a natively multimodal 17B MoE model with 16 experts for text and image understanding.
In-place updates are also landing on existing entries. @cf/meta/llama-3.3-70b-instruct-fp8-fast runs with speculative decoding, prefix caching and an updated server backend for a speed lift on Llama 3.3 70B. The three BGE English embedding models (@cf/baai/bge-small-en-v1.5, @cf/baai/bge-base-en-v1.5, and @cf/baai/bge-large-en-v1.5) accept a new pooling input parameter accepting either cls or mean.
Older models will be deprecated as replacements arrive, with email notices going out ahead of the change; the Developer Docs changelog tracks both releases and removals.
