Why the network edge is where inference belongs
AI workloads split into two fundamentally different phases. Training a generative model is a long-running, compute-intensive process that can take months and is best done in centralized clouds with large GPU clusters. Inference, by contrast, happens on every single invocation of a model — every prompt, every image generation, every translation. That makes it the far more common AI workload, and it changes the infrastructure calculus entirely.
Inference has a user on the other end, which makes it latency-sensitive. Running it on-device is appealing for responsiveness, but device hardware is nowhere near powerful enough for most models, and battery life imposes hard limits. Running it in a centralized cloud gives you horsepower, but at the cost of hundreds of milliseconds of network distance — sometimes across borders, which adds its own complications. The network edge splits the difference: compute that is close enough to users to feel instantaneous, and powerful enough to run real models.
Cloudflare is expanding its developer platform to make that edge the default place for inference, with a set of announcements covering serverless GPU inference, vector storage, AI traffic management, and partnerships across the AI ecosystem.
Workers AI: serverless GPU inference
Workers AI, formerly known as Constellation, brings serverless GPU inference to Cloudflare’s global network. The pitch is straightforward: no machine learning expertise required, no hunting for GPU capacity, no infrastructure to manage. Developers pick one of the provided models and start making inference calls, paying only for what they use.
The model catalog launched with broad appeal in mind, covering common inference tasks out of the box. That includes LLMs, which have become the most common type of model developers are deploying in 2023.
Vectorize: giving AI access to state
Running a model is only one part of an AI application. Chatbots that answer questions about a product catalog or a custom corpus need to convert that information into embeddings and store them somewhere queryable. That requires a vector database — a state layer for AI in the same way Workers KV provided state for early Workers compute.
Vectorize is Cloudflare’s answer: a vector database designed to make it easy, fast, and affordable to index and store vectors. It sits alongside Workers AI so developers can build the full pipeline — parse source material, generate embeddings via a model, store them, and query them at inference time — without assembling infrastructure from multiple vendors.
AI Gateway: cost control and visibility
Cloudflare’s typical approach to improving something starts with measurement, and AI deployments are no different. AI Gateway is positioned as the observability and control layer for AI applications, regardless of where the underlying model is running.
It provides real-time monitoring to help debug and fine-tune deployments, plus two mechanisms that directly address runaway AI costs. Caching frequently used AI responses reduces latency and offloads repeated work, while rate limiting ensures resources are allocated efficiently. For organizations struggling to reign in AI spend before anything reaches production, the gateway gives them both visibility and levers to pull.
AI Gateway applies whether the inference runs on Workers AI or elsewhere, making it a neutral control plane for AI traffic that may span multiple providers.
Llama 2 arrives on the network
Until recently, access to large language models meant calls to proprietary APIs. Meta’s open-source Llama 2 release changed that, letting developers run and deploy their own LLMs — provided they had GPU access, which remained a significant barrier.
By adding Llama 2 to the Workers AI catalog, Cloudflare removes that barrier. Developers get access to an open-source LLM with no configuration and no GPU procurement required, running on Cloudflare’s global network.
Writing once, running anywhere
The edge is well-suited to latency-sensitive inference, but not every AI workload belongs there. Applications will continue to span the spectrum from device to edge to centralized cloud. Self-driving cars need on-device decision-making where milliseconds matter; hundred-billion parameter models will still head to centralized clouds.
Cloudflare’s position is that the edge should not just serve inference, but route between all three locations. The ONNX runtime is central to that vision: it standardizes the environment for running models, making it possible to move the same model across device, edge, and cloud. Cloudflare sees its network as a routing layer that can direct workloads to the best location based on what an application is optimizing for — latency, accuracy, cost, compliance, or privacy.
Partnerships across the model lifecycle
Three partnerships round out the ecosystem play.
With Hugging Face, Cloudflare is integrating serverless inference into the platform where developers already discover and explore models, reducing friction between finding a model and deploying it.
The Databricks collaboration brings MLflow, an open-source platform for managing the machine learning lifecycle, to Workers AI. Developers using MLflow to package and track models will be able to deploy those models directly onto Cloudflare’s serverless platform.
Cloudflare also lists NVIDIA, Microsoft, and Meta among its partners in this push, signaling that the AI infrastructure buildout will be collaborative rather than solitary.
Compliance, cost, privacy
Rapid AI adoption has a governance problem. Governments are increasingly requiring data to be processed locally or stored in-country, and inference workloads must respect those constraints. A network spanning 300 cities, combined with Cloudflare’s Data Localization Suite, provides the granularity to keep AI deployments within required jurisdictions.
The cost problem is simpler to state: AI is expensive, and spend can spiral before anything reaches production. The stated intent is to keep costs affordable and, more importantly, charge only for actual usage. Both Workers AI and AI Gateway include the visibility and controls meant to prevent budget overruns.
Privacy is addressed with a firm commitment: customer data passing through Cloudflare for inference will not be used to train large language models.
These announcements position Cloudflare’s network — described as Region Earth — as the first inference cloud, a layer that sits between underpowered devices and distant data centers, and one that will only become more relevant as AI applications move from demos to production.
