Announcing AI Gateway: making AI applications more observable, reliable, and scalable

AI Gateway: A Single Control Point for AI API Traffic

Cloudflare has announced the beta release of AI Gateway, a service designed to sit between your application and the AI APIs it calls—such as OpenAI, Hugging Face, or Replicate—to add a layer of caching, request management, and analytics. The goal is to handle the common operational overhead of AI applications so that development teams can focus on the core product rather than plumbing.

Connecting Your Application

Getting started requires a simple configuration change: swap the endpoint URL in your existing API calls for a dedicated AI Gateway URL. For example, an OpenAI integration would change its base URL from "https://api.openai.com/v1" to "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai". The request payload and authentication tokens remain unchanged, allowing most applications to work immediately once the endpoint is configured.

The service currently supports model providers like OpenAI, Hugging Face, and Replicate, including their various endpoints and response streaming. Each provider gets a dedicated gateway endpoint, so the one-line change is all that is needed. For scenarios requiring more flexibility, a universal endpoint is available. This endpoint supports fallback definitions and automatic retries. For instance, if a request to OpenAI GPT-3 fails, you can configure Hugging Face GPT-2 as a fallback and have the gateway resend the request there without manual intervention. This approach helps improve resilience during outages or rate limits, and can also help diversify usage across multiple model providers to manage costs or bills. With the universal endpoint, the payload must be adjusted to include the provider and endpoint so the gateway can route the request correctly.

Visibility, Cost, and Control

Once traffic flows through AI Gateway, the service aggregates and logs request metadata. This provides visibility into metrics that are often difficult to extract directly from model providers, including request counts, number of users, request duration, and the operational cost of running the application. The analytics are consolidated even when requests are spread across multiple backend providers.

To help manage scaling and costs, AI Gateway includes two control mechanisms: caching and rate limiting. Developers can define rules to cache API responses, allowing subsequent identical requests to be served from Cloudflare's cache rather than calling the original—and potentially costly—API again. Rate limiting offers another lever to throttle request volume and shut down abusive or unexpected traffic patterns. Each of these mechanisms gives developers the flexibility to set rules that align with their budget and application performance targets.

Integration with the Workers Platform

AI Gateway is designed to work alongside other Cloudflare AI offerings, including the Workers AI inference platform and the Vectorize vector database. This allows developers to build a full-stack AI application entirely within the Workers ecosystem: deploying with Workers, running model inference at the edge, storing vector embeddings, and now tracing the activity with AI Gateway.

Roadmap: From Application to Enterprise Use

The initial release is focused on the core features of logging, caching, and rate limits, but Cloudflare's roadmap points to more advanced capabilities. Future plans include usage alerts, jailbreak protection, dynamic model routing with A/B testing, and more sophisticated cache rules.

The longer-term vision for AI Gateway moves beyond an individual application development tool into an enterprise governance layer. The proposed model would allow organizations to route all outgoing AI-related traffic through Cloudflare, enabling them to log user requests, apply access policies, and enforce rate limiting and data loss prevention (DLP) strategies. For example, if an employee pasted an API key into a public AI tool, the gateway could be configured to redact that information or block the request entirely before it reached the external provider. Logging and alerts would also allow security teams to flag suspicious behavior and enforce usage policies enterprise-wide.

This approach acknowledges a growing tension in enterprises: the desire to adopt AI tools for their efficiency while being cautious about data privacy and user error. The intent behind AI Gateway is to make that adoption more manageable by giving organizations clear visibility into how their systems and users interact with AI services—making it easier to decide not just how AI is being used, but how it should be used.