Llama 3 lands on Workers AI

Cloudflare has added Meta's Llama 3 to Workers AI, serving the 8B variant as a launch partner alongside Meta. The serverless inference platform now lets developers run the newest open Llama model through the same model catalog they already use for other LLMs, without provisioning or deploying the model themselves. The instruction fine-tuned version of Llama 3 8B is among the supported models, and inference takes only a few lines of code.

Continuity with the Llama 2 era

Workers AI entered beta with Llama 2 support, added because it was one of the most requested open source models from the developer community. Builders have since used the platform for knowledge-sharing chatbots, creative content generation, and workflow automation. Cloudflare frames its goal around flexibility: developers should be able to work with multiple models and weigh accuracy, performance, and cost without handling the complexity of hosting and deploying them.

That position is helped by Meta's commitment to an open AI ecosystem, which Cloudflare says allows customers of all sizes to use AI at scale in production.

What the new model brings

Meta's evaluation results place Llama 3 as the most advanced open LLM currently on benchmarks including MMLU, GPQA, HumanEval, GSM-8K, and MATH. The model was trained on 15T tokens, an increased volume that gives it a firmer grasp of language intricacies. Its larger context window doubles Llama 2's capacity, letting the model better handle lengthy passages with rich contextual data. The model itself supports an 8k context window, though Workers AI currently caps support at 2.8k and expects to reach 8k through quantized models soon.

Tokenization has changed as well: a new tiktoken-based tokenizer carries a vocabulary of 128k tokens, encoding more characters per token and improving results on English and multilingual benchmarks. Because of that larger vocabulary, the embedding and output layers hold 4 times as many parameters, making this generation larger than Llama 2.

Architecturally, Llama 3 relies on grouped-query attention (GQA) for more efficient inference on longer sequences, which also makes the 8B model equivalent in architecture to Mistral-7B. For tokenization it uses byte-level byte-pair encoding (BPE), similar to OpenAI's GPT tokenizers, so tokens can represent arbitrary byte sequences, including ones without valid utf-8 encoding. That flexibility in representing language contributes to the model's improved performance.

Safety tooling and availability

Meta shipped Llama 3 with supporting tools beyond the base models: Llama Guard 2, Code Shield, and CyberSec Eval 2. Cloudflare hopes to release these on Workers AI shortly.

Llama 3 8B is listed in the Workers AI Model Catalog now, with documentation available for the instruct model. Developers can also share experiences or ask questions in the Developer Discord.