Cloudflare Builds an Inference Layer For Agentic Workloads
Agentic applications are transforming the economics of AI inference. A simple chatbot makes a single inference call per user prompt, but an agent can chain together a dozen calls to complete one task. That means a slow provider isn't adding 50 milliseconds to a response; it is adding half a second. A single failed request isn't a simple retry; it becomes a cascade of downstream failures. And the model landscape shifts fast enough that the best model for a given job today may be a completely different one from a different provider in a few months.
To address these realities, Cloudflare is positioning its AI stack as a unified inference layer: a single API for accessing models from any provider, designed for speed and reliability across multi-step agent workflows. The company has seen significant adoption of AI Gateway and Workers AI since their launch, and recent updates have added zero-setup default gateways, automatic retries on upstream failures, and finer-grained logging controls. The company also announced the Replicate team has officially joined its AI Platform team.
One API for a Multi-Provider Catalog
Developers using Workers can now call third-party models through the existing AI.run() binding. Switching a workload from a Cloudflare-hosted model to one from OpenAI, Anthropic, or another provider is a one-line change. REST API support is slated to follow in the coming weeks for those working outside the Workers environment.
const response = await env.AI.run('anthropic/claude-opus-4-6',{
input: 'What is Cloudflare?',
}, {
gateway: { id: "default" },
});The unified catalog expands to 70+ models across 12+ providers, all accessible via a single API and billed against one set of credits. The new roster includes providers such as Alibaba Cloud, AssemblyAI, Bytedance, Google, InWorld, MiniMax, OpenAI, Pixverse, Recraft, Runway, and Vidu. The catalog also extends beyond text to include image, video, and speech models for multimodal applications.
Consolidating access also means consolidating spend visibility. With most companies calling an average of 3.5 models from multiple providers, no single vendor can offer a complete view of AI usage. AI Gateway provides a centralized place to monitor and manage those costs. Custom metadata attached to requests allows teams to break down spend by free versus paid users, individual customers, or specific workflows.
Bringing Custom Models to Workers AI
The hosted catalog covers a lot of ground in a single gateway, but some workloads require models that have been fine-tuned on proprietary data or optimized for a specific use case. Cloudflare is preparing to let users bring their own models to Workers AI, leveraging Replicate's Cog technology for containerizing machine learning workloads.
Cog is designed around a simple workflow: dependencies are declared in a cog.yaml file, and inference code lives in a Python file. Cog abstracts away packaging concerns like CUDA dependencies, Python versions, and weight loading.
build:
python_version: "3.13"
python_requirements: requirements.txt
predict: "predict.py:Predictor"A companion predict.py file contains a setup function for the model and a handler that runs when an inference request, or prediction, comes in.
from cog import BasePredictor, Path, Input
import torch
class Predictor(BasePredictor):
def setup(self):
"""Load the model into memory to make running multiple predictions efficient"""
self.net = torch.load("weights.pth")
def predict(self,
image: Path = Input(description="Image to enlarge"),
scale: float = Input(description="Factor to scale image by", default=1.5)
) -> Path:
"""Run a single prediction on the model"""
# ... pre-processing ...
output = self.net(input)
# ... post-processing ...
return outputThe container image is built with cog build and then pushed to Workers AI, where it is deployed and served through the usual Workers AI APIs. Cloudflare reports that the majority of its traffic already comes from dedicated Enterprise instances running custom models, and future work includes customer-facing APIs, Wrangler commands for pushing containers, and GPU snapshotting to speed up cold starts.
The Fast Path to First Token
For live agents, user-perceived speed is measured by time to first token, not total response time. A model might take three seconds to complete a full response, but delivering the first token 50 milliseconds faster is the difference between a responsive agent and a sluggish one.
AI Gateway sits on Cloudflare's network spanning 330 cities, placing it close to both users and upstream inference endpoints and minimizing network round-trips before streaming begins. For open-source models hosted on Workers AI itself—including agent-focused models like Kimi K2.5 and real-time voice models—there is no extra public Internet hop when called through AI Gateway, since code and inference run on the same global network.
Failover and Resilience for Multi-Step Chains
Reliability is a structural requirement for agents, where each step depends on the previous one. AI Gateway provides automatic failover when a model is available from multiple providers: if one provider goes down, traffic is routed to another available option without developer-written failover logic.
Long-running agents built with the Agents SDK also gain resilience to disconnects during streaming inference. AI Gateway buffers streaming responses as they are generated, independent of the agent's lifetime. If an agent is interrupted mid-inference, it can reconnect and retrieve the buffered response without starting a new inference call or paying twice for the same output tokens. Combined with the Agents SDK's built-in checkpointing, the interruption remains invisible to the end user.
With the Replicate team now part of the platform group, integrations are underway to bring Replicate models onto AI Gateway and to replatform hosted models onto Cloudflare infrastructure. Documentation for getting started is available for AI Gateway, Workers AI, and the Agents SDK.
