Workers AI Update: Hello, Mistral 7B!

Mistral 7B arrives on Workers AI

Workers AI now includes Mistral-7B-v0.1-instruct, a 7.3 billion parameter language model that delivers benchmark results that punch well above its weight class. The instruct-tuned variant deployed on Workers AI outperforms comparable 13B chat models in Mistral's benchmark suite. On broader evaluation, Mistral 7B beats 13B models across the board, holds its own against 34B models on many tasks, and approaches CodeLlama 7B on code generation without compromising English language capability.

The model is available now through the REST API:

curl -X POST \
“https://api.cloudflare.com/client/v4/accounts/{account-id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1” \
-H “Authorization: Bearer {api-token}” \
-H “Content-Type:application/json” \
-d '{ “prompt”: “What is grouped query attention”, “stream”: true }'

API Response: { response: “Grouped query attention is a technique used in natural language processing  (NLP) and machine learning to improve the performance of models…” }

Or directly from a Worker script:

import { Ai } from '@cloudflare/ai';
export default {
    async fetch(request, env) {
        const ai = new Ai(env.AI);
        const stream = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: 'What is grouped query attention',
            stream: true
        });
        return Response.json(stream, { headers: { “content-type”: “text/event-stream” } });
    }
}

One of the model's key architectural advantages is grouped-query attention, an inference optimization introduced in 2023 that maintains output quality while dramatically speeding up generation. For 7B-parameter models, the practical effect is significant: roughly 4x more tokens per second compared to Llama-class models using conventional attention.

How attention works

At its core, attention follows the Scaled Dot-Product formulation from the foundational paper Attention Is All You Need. Queries and keys of dimension d_k, plus values of dimension d_v, feed into a simple computation: dot products of each query with all keys, scaled by 1/sqrt(d_k), then normalized with softmax to produce weights on the values.

Intuitively, this computes similarity between each element of an input sequence and every other element, letting the model determine which parts deserve more focus. The scaling step prevents extreme values that would destabilize the softmax.

But raw attention has no learnable parameters. Real transformer models add three layers of complexity:

Three styles of attention

Modern language models diverge in how they arrange key (K) and value (V) vectors relative to query (Q) vectors:

• Multi-head attention: one K/V set per Q vector—the standard introduced in the original transformer paper, offering maximum flexibility at high memory cost.
• Multi-query attention: a single shared K/V set for all Q vectors, minimizing memory access during inference at some quality cost in large models.
• Grouped-query attention: Q vectors partition into G-sized groups, each sharing one K/V set. Mistral 7B uses this approach.

Grouped-query attention emerged from Google's 2019 multi-query attention research, which demonstrated equal performance on translation benchmarks while drastically reducing memory footprint. That technique excelled in early, task-specific models, but as models grew larger and more general, quality degradation became more apparent.

The 2023 GQA paper (Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints) refined the approach: instead of all-or-nothing K/V sharing, grouped-query attention uses one K/V set per fixed group of Q vectors. This produces better tradeoffs between inference speed, memory usage, and output quality.

Therein lies the practical relevance for production deployments. Choosing a model requires balancing model quality against memory consumption, batch size constraints, and hardware costs. Understanding these attention variants helps developers see why Mistral hits a sweet spot in that optimization problem.

Putting it to use

By pairing grouped-query attention with sliding window attention, Mistral achieves low latency and high throughput while maintaining benchmark competitiveness against much larger models—making it an unusually efficient choice at its parameter scale.

Test it alongside other models in the Workers AI catalogue at ai.cloudflare.com, or read the full integration details in the text generation model documentation. Workers AI deployment also supports streaming responses for interactive use cases.