Workers AI: New GPUs, Bigger Models, New Pricing

Workers AI is being upgraded with more powerful GPUs across Cloudflare's network. The company now has GPU capacity in over 180 cities, double what it had a year ago. These new GPUs enable inference on significantly larger models, including Meta Llama 3.1 70B, which was previously out of reach for a platform limited to 8B parameter LLMs. The upgrade also brings faster response times and larger context windows, such as the full 128K context window now supported for Llama 3.1 and 3.2 models.

Performance expectations are now published per model in the catalog, with metrics including Time To First Token (TTFT), throughput in tokens per second (TPS), context window size, and pricing. Generally, users can expect throughput of 80+ TPS for 8B models with a TTFT of 300 ms, depending on geographic location.

Cloudflare served as a Meta launch partner for the Llama 3.2 collection, offering Day 0 support for the 11B vision model and the 1B and 3B text-only models on Workers AI. The company has also published a technical post detailing methods behind these performance gains, including an open-source approach to KV cache compression and speculative decoding.

Run Any* Model and a Two-Part Catalog

Cloudflare is opening up the Workers AI model catalog with a new feature called Run Any* Model, currently in closed beta. This expands beyond manually curated open-source models to support models compatible with Cloudflare's GPUs and inference stack. The catalog is now split into two parts:

  • Static catalog: Curated by Cloudflare with popular open-source models that have guarantees on availability and speed, kept warm to avoid cold starts. Pricing remains serverless, with charges only for requests.
  • Dynamic catalog: Models launched via Run Any* Model. Public models can be shared by users; private model instances are planned for the future.

This is positioned as a first step toward supporting custom or private models broadly, a capacity currently available only for select customers.

Neurons Are Out, Unit-Based Pricing Is In

The "neurons" pricing model introduced with Workers AI last year is being retired. Cloudflare acknowledges that neurons were difficult to understand and hard to compare with other providers' pricing structures. The new pricing is unit-based and depends on model task and size:

  • LLMs: priced on parameters and input/output tokens
  • Image generation: priced on output resolution and number of steps
  • Embeddings: priced on input tokens
  • Speech-to-text: priced on seconds of audio input

The free tier has been redone to fit these metrics, offering a monthly allotment across all task types. New pricing takes effect October 1, 2024, and applies to all models graduating from beta.

AI Gateway: Persistent Logs, Logpush, and Evaluations

AI Gateway has proxied over 2 billion requests as of September 2024, less than a year after launch. The platform is now moving further into ML Ops territory with several new features in open beta.

Persistent logs allow developers to store and analyze up to 10 million logs per gateway. Each request through AI Gateway generates a log with timestamp, request status, model, and provider. The revamped logging interface includes cost and duration details, and supports human feedback annotation with thumbs up/down. Logs can be filtered, searched, and tagged with custom metadata.

Persistent logs are available on all plans. The Workers Free plan includes storage for up to 100,000 logs total; the Workers Paid plan raises that to 200,000 logs. Additional storage costs $8 per 100,000 logs per month. Billing for this feature will be implemented when it reaches general availability.

For users who want to export logs, AI Gateway now supports Logpush, allowing automatic transfer to Cloudflare R2, Amazon S3, Google Cloud Storage, and other providers. Logpush uses its existing pricing model and is available to users on paid plans.

AI Gateway is also introducing AI evaluations with human-in-the-loop feedback, now in open beta. Users can create datasets from logs to score and evaluate model performance, speed, and cost, initially for LLMs only. Support has been added for cost analysis across many models and providers, including the ability to define custom costs. Future features will include automated LLM-based scoring and model comparison.

Vectorize Goes GA With Major Improvements

Vectorize has been completely redesigned since its initial announcement and is now generally available. The v2 version supports indexes up to 5 million vectors (up from 200,000), returns up to 100 results per query (up from 20), and has cut median latency by 95%, from 500 ms to 30 ms. A migration path from v1 to v2 will be available in early Q4.

Pricing has been cut substantially: query prices are down 75% and storage costs are down 98%. A free tier is also being introduced, which includes:

  • 30 million stored vector dimensions per month
  • 5 million queried vector dimensions per month

Cloudflare conducted benchmarking on Vectorize v2 with two datasets: dbpedia-openai-1M-1536-angular, containing 1 million vectors with 1,536 dimensions, and Laion-768-5m-ip, containing 5 million vectors with 768 dimensions. Tests used a concurrency of 300 via a Cloudflare Worker binding on warm caches. The company reports that running Vectorize close to inference and user interaction minimizes end-to-end latency when combined with Workers AI.

An Integrated AI Platform

The broader vision for these products is tighter integration across the AI portfolio. Connecting Workers AI to AI Gateway via a binding enables analytics, logging, caching, and rate limiting on AI requests. The future roadmap includes automated use of Vectorize for context in inference calls, cutting down orchestration needed for RAG applications, and full observability with traces that show how retrieval and generation components work together.

Future work will focus on making AI Gateway aid in dataset creation for fine-tuning models on Workers AI, with seamless redeployment on the same platform, and tighter Vectorize-Workers AI integrations for automated context supply and conversation memory.