A persistent connection for AI Gateway
AI Gateway is processing over 3 billion logs and growing rapidly. For developers working with the platform, managing persistent connections to AI Gateway required either implementing HTTP/2 keep-alive or repeatedly negotiating new TLS handshakes. Since WebSocket libraries are more widely supported across modern programming languages than HTTP/2, Cloudflare built a WebSockets API for AI Gateway using Durable Objects.
Through this API, every AI provider supported by AI Gateway can be accessed over a single persistent TCP connection. Even when the underlying inference provider doesn't support WebSockets natively, AI Gateway handles the protocol translation and proxies the request using the provider's supported protocols.
The WebSockets API in beta requires authentication. You need a Cloudflare API token with the AI Gateway: Run permission, sent in the cf-aig-authorization header.
The authentication logic is straightforward:
- When Authenticated Gateway is enabled and a valid token is present, requests pass through.
- When Authenticated Gateway is enabled but the
cf-aig-authorizationheader is missing or invalid, requests fail. - When Authenticated Gateway is disabled, the header is ignored entirely.
How it works
Because Cloudflare previously used Durable Objects to scale AI Gateway's logging infrastructure, extending those same Durable Objects to handle WebSocket connections was a natural fit.
When a Worker receives a new WebSocket connection, authentication happens in one of two ways. The primary method validates a Cloudflare API token sent in the cf-aig-authorization header. But because browser WebSocket clients don't support custom headers in their standard API, AI Gateway also accepts a token via the sec-websocket-protocol header. Cloudflare doesn't recommend storing API keys in browser environments, but this flexibility exists for WebSocket clients that can't set custom headers.
After verification, the connection is upgraded and handed off to a Durable Object, which generates a random UUID to identify the connection among all messages it receives. Any AI Gateway settings sent via headers — such as cf-aig-skip-cache — are stored and applied to every request in the session, though they can still be overridden on a per-request basis.
Once the connection is established, the Durable Object listens for messages in the AI Gateway universal format. This means existing applications can switch from HTTP to WebSocket communication without changing their message structure. Non-streaming requests return a JSON envelope that includes extra metadata such as the AI Gateway log ID for the request.
Streaming requests behave differently. AI Gateway sends an initial message containing request metadata, then relays streaming chunks in real time as they arrive from the inference provider. The streaming chunks only include the eventId field in their metadata, since all other request metadata was already provided in the initial message.
Handling concurrent requests
WebSocket connections are fully asynchronous, allowing clients to send multiple streaming requests without waiting for responses. But this creates an identification problem: when several streaming responses arrive simultaneously, the client can't tell which chunks belong to which request.
To solve this, the universal format now includes an eventId field. Clients can attach a custom ID to each request, allowing the client to correctly match streaming chunks to their originating request. When all chunks for a request have been sent, AI Gateway signals completion, including the full metadata again for flexibility.
Connecting to the beta
The real-time WebSocket API for AI Gateway is now in beta and open to everyone. To get started, take your gateway's universal endpoint URL and swap the protocol:
wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/Then open a WebSocket connection using that endpoint, along with a Cloudflare token that has the AI Gateway Run permission. The platform also works with WebSocket clients like the ws npm package as well as the built-in browser WebSocket client.
In Q1 2025, Cloudflare plans to support WebSocket-to-WebSocket connections using Durable Objects, enabling direct access to OpenAI's real-time API through AI Gateway. In the meantime, a Worker is available on GitHub to proxy those requests. There is also a Cloudflare Discord channel for questions and support.
