Workers AI is now generally available, following an open beta. Alongside GA, the platform has a published price list, a larger model catalog, and changes to the routing layer and rate limits.

Inference routing and limits

Load balancing inside Workers AI now routes requests to more GPUs across more cities, and each city tracks the total inference capacity available to it. A request that would otherwise queue in its local city can instead be sent to another location when traffic is high.

Rate limits have increased across all models as a result. Most LLMs now allow 300 requests per minute, up from 50 during the beta; smaller models run at 1500-3000 requests per minute. Per-model figures are listed in the Developer Docs.

Pricing

A pricing calculator for the 10 non-beta models was published earlier in the month, and several models were re-optimized to lower the cost of inference. Llama 2 is now over 7x cheaper and Mistral 7B over 14x cheaper to run than the pricing first published on March 1.

Billing for non-beta models began April 1st; beta models remain free and unlimited. Every customer receives 10,000 neurons per day. Workers Free customers hit a hard rate limit after that daily allotment, while Workers Paid customers pay $0.011 per 1000 additional neurons.

Leveling up Workers AI: general availability and more new capabilities

Tooling and GPU footprint

The Workers AI dashboard now reports usage analytics across models, including neuron calculations to help with cost forecasting, and the AI playground has been rebuilt for testing and comparing models and for configuring prompts and parameters.

GPUs are currently deployed in over 150 cities. The plan announced at the September 2023 launch is to have inference-tuned GPUs in nearly all data centers by the end of 2024.

BLOG-2301 Embedded Image - eOJ0Ov

Next-generation compute servers with GPUs arrive in Q2 2024, with gains in performance, power efficiency and reliability; the Gen 12 design was previewed in a December 2023 post. These and later hardware generations are intended to support larger models and fine-tuning on the platform, raising inference throughput, cutting latency and improving availability for production workloads.

Hugging Face models

Popular models on Hugging Face now carry a click-to-run option that launches them on Workers AI where supported. Four models were added through the partnership: Mistral 7B v0.2 with improved context windows, Nous Research's Hermes 2 Pro (fine-tuned from Mistral 7B), Google's Gemma 7B, and Starling-LM-7B-beta (fine-tuned from OpenChat). That brings the Hugging Face-curated set available for serverless GPU inference on Workers AI to 14 models, all served through Hugging Face's TGI backend, with the two teams working together to curate, optimize and deploy them.

"We are excited to work with Cloudflare to make AI more accessible to developers. Offering the most popular open models with a serverless API, powered by a global fleet of GPUs is an amazing proposition for the Hugging Face community, and I can't wait to see what they build with it."- Julien Chaumond, Co-founder and CTO, Hugging Face

The full set is listed in a Hugging Face Collection, and the "Deploy to Cloudflare Workers AI" button sits at the top of each model card. Model requests can go through Discord.

Bring Your Own LoRA

Fine-tuned inference was among the most requested Workers AI features, and BYO LoRAs is the first step toward it. Low-Rank Adaptation adapts only some of a model's parameters to a task instead of rewriting all of them as full fine-tuning does, so fine-tuned outputs arrive without the compute cost of full fine-tuning. Workers AI applies a trained LoRA adapter to a base model at runtime, giving fine-tuned inference at a fraction of the cost, size and speed penalty of a fully fine-tuned model. Fine-tuning jobs and fully fine-tuned models hosted on the platform are planned for later.

BYO LoRAs is in open beta for Gemma 2B and 7B, Llama 2 7B and Mistral 7B. Supported adapters are up to 100MB in size with a maximum rank of 8, and each account can hold up to 30 LoRAs. Use is subject to the Developer Platform Terms of Service, including model-specific license restrictions.

const response = await ai.run(
  "@cf/mistralai/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
  {
      messages: [{"role": "user", "content": "Hello world"],
      raw: true, //skip applying the default chat template
      lora: "00000000-0000-0000-0000-000000000", //the finetune id OR name 
  }
);

Python Workers in open beta

Cloudflare Workers can now be written in Python, in open beta. Python is the second most popular programming language after JavaScript and the usual choice for AI applications. Python Workers support every Cloudflare resource binding, including Vectorize, D1, KV and R2.

The Python LangChain library runs on Python Workers, as it does in JavaScript, along with other packages such as FastAPI. A Python Worker is configured by pointing wrangler.toml at a .py file:

name = "hello-world-python-worker"
main = "src/entry.py"
compatibility_date = "2024-03-18"
compatibility_flags = ["python_workers"]

No extra toolchain or precompilation step is required: the Workers runtime provides the Pyodide Python execution environment directly, mirroring how JavaScript Workers already run.

from js import Response

async def on_fetch(request, env):
    return Response.new("Hello world!")

Wider provider coverage in AI Gateway

AI Gateway gives developers analytics, caching, rate limiting and other controls over AI applications. Support for Anthropic, Google Vertex and Perplexity is being added, and Azure and Amazon Bedrock were rolled out quietly in December 2023. Between those and Workers AI itself, the most popular providers are now reachable through the gateway. Setup instructions are in the Developer Docs.

BLOG-2301 Embedded Image - bMAxLu

Persistent logs and the roadmap toward control-plane routing

Persistent logs are slated for Q2 of 2024. The feature will let developers push logs, including prompts and responses, to object storage; attach custom metadata so requests can be tagged with user IDs or other identifiers; and manage API keys through a secrets mechanism.

The goal is for AI Gateway to act as the control plane for AI applications, dynamically evaluating and routing requests across models and providers. Logged data is intended to feed a one-click fine-tuning flow, with the fine-tune job and resulting fine-tuned model eventually running directly on Workers AI.

Vectorize: metadata filtering now, multi-million vector indexes at GA

Vectorize has been in open beta since September 2023 and persists embeddings, such as those produced by Workers AI text embedding models, so that queries can return the closest match for similarity search or recommendations. Without a vector database, model output is forgotten and recalling it means paying to re-run the model.

Metadata filtering has been added since the open beta began: vector search can now be combined with filtering on arbitrary metadata, which covers the query complexity AI applications need. General availability is targeted for June 2024 and will include support for multi-million vector indexes.

// Insert vectors with metadata
const vectors: Array<VectorizeVector> = [
  {
    id: "1",
    values: [32.4, 74.1, 3.2],
    metadata: { url: "/products/sku/13913913", streaming_platform: "netflix" }
  },
  {
    id: "2",
    values: [15.1, 19.2, 15.8],
    metadata: { url: "/products/sku/10148191", streaming_platform: "hbo" }
  },
...
];
let upserted = await env.YOUR_INDEX.upsert(vectors);

// Query with metadata filtering
let metadataMatches = await env.YOUR_INDEX.query(<queryVector>, { filter: { streaming_platform: "netflix" }} )

A full stack for AI on the Developer Platform

The stated position is that any developer should be able to build and ship full-stack applications quickly, AI experiences included. Workers AI reaching GA, Python support in Workers, AI Gateway, Vectorize and the Hugging Face partnership together widen what can be built with AI on the platform. The Developer Docs are the starting point, and feedback is being collected on Discord.