Real-time voice AI needs an infrastructure rethink
Text-based AI interfaces have proven themselves, but they only capture part of how humans actually communicate. We speak, interrupt, clarify, and point — and voice AI promises to bring those natural interaction patterns into applications. The challenge is that building production-grade voice AI is genuinely hard. It requires coordinating speech-to-text, language model inference, and text-to-speech while managing audio pipelines, handling interruptions, and staying within a strict latency budget of under 800 milliseconds for conversation to feel natural.
That budget is unforgiving: roughly 40ms for microphone input, 300ms for transcription, 400ms for LLM inference, and 150ms for text-to-speech. Any additional latency from poor infrastructure choices or distant servers turns a delightful experience into a frustrating one. Cloudflare is addressing this by shipping a set of new capabilities designed to make real-time voice AI as easy to deploy as a static website, leveraging its network of 330+ datacenters to keep processing close to users.
Cloudflare Realtime Agents: a runtime for voice pipelines
Cloudflare Realtime Agents is a new runtime for orchestrating voice AI pipelines on Cloudflare's global network. Instead of managing complex infrastructure, developers can focus on building conversational experiences with composable building blocks rather than a rigid pipeline. The system supports configurable data flows, tee and join operations, and granular control over agent behavior.
When a user connects, audio streams from their device to the nearest Cloudflare location via WebRTC using Cloudflare's RealtimeKit mobile or web SDKs. A pre-configured pipeline then runs — speech-to-text, followed by the LLM, then text-to-speech — with support for interruption detection and turn-taking. Your configured runtime options, callbacks, and tools execute, and the generated audio streams back to the user with minimal latency.
The core abstraction is simple: your agent is a JavaScript class extending RealtimeAgent, where you initialize a pipeline of text-to-speech, speech-to-text, text-to-text, or even speech-to-speech transformations. A text handler, for example, is just a function that takes text and returns text, inserted between speech-to-text and text-to-speech stages.
class MyTextHandler extends TextComponent {
env: Env;
constructor(env: Env) {
super();
this.env = env;
}
async onTranscript(text: string) {
const { response } = await this.env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
prompt: "You are a wikipedia bot, answer the user query:" + text,
});
this.speak(response!);
}
}Realtime Agents is designed for flexibility rather than lock-in. Developers can use models from Workers AI, OpenAI, Anthropic, or any provider through AI Gateway; accept audio and/or text inputs and respond with audio and/or text; and maintain stateful context across a conversation without managing it manually. You can work with RealtimeKit for WebRTC session management and UI, or connect directly with any standard WebRTC client or raw WebSockets for full control. The runtime also integrates with the Cloudflare Agents SDK.
The runtime is free to use during its open beta starting today, and it works with speech and audio platforms including ElevenLabs and Deepgram. For LLM inference, you can use models through Workers AI and AI Gateway, connect to third-party models like OpenAI, Gemini, Grok, or Claude, or bring your own custom models.
Raw WebRTC audio as PCM in Workers
For developers who need more flexibility than Realtime Agents provides, Cloudflare is now exposing the raw WebRTC audio pipeline directly to Workers. This works by leveraging Cloudflare's Realtime SFU, which converts WebRTC audio in the Opus codec to PCM and streams it to any WebSocket endpoint you specify.
This opens up several use cases:
- Live transcription — stream audio from a video call directly to a transcription service
- Custom AI pipelines — send audio to AI models without setting up complex infrastructure
- Recording and processing — save, audit, or analyze audio streams in real-time
The choice between WebSockets and WebRTC matters for voice AI. WebSockets are well-suited for server-to-server communication and testing scenarios where ultra-fast responses aren't critical. WebRTC, however, has clear advantages for live audio: it uses UDP instead of TCP, avoiding head-of-line blocking delays; the Opus codec adapts to network conditions and handles packet loss gracefully; and it includes built-in echo cancellation and noise reduction that would otherwise need separate implementation.
With this feature, you get the best of both: WebRTC for client-to-server communication, with Cloudflare converting to familiar WebSockets for server-to-server communication and backend processing. Once audio is available as PCM at the original sample rate, you can resample and send it to different AI providers, run WebAssembly-based audio processing, build with Durable Objects, Alarms, and other Workers primitives, or deploy containerized processing pipelines with Workers Containers.
The WebSocket is bidirectional — data sent back on it becomes available as a WebRTC track on the Realtime SFU, ready for consumption within WebRTC. A demo application using the ElevenLabs API for text-to-speech illustrates the setup.
WebSocket support in Workers AI
Real-time voice AI depends on persistent, low-latency connections to inference servers — HTTP works for chat and batch inference but isn't suited for live audio applications. Workers AI now supports WebSocket connections on select models to address this.
The first model with WebSocket support is PipeCat's smart-turn-v2 turn detection model. Turn detection determines when a speaker has finished talking and it's appropriate for the AI to respond — getting this right is the difference between an AI that constantly interrupts and one that feels natural in conversation.
"""
Cloudflare AI WebSocket Inference - With PipeCat's smart-turn-v2
"""
import asyncio
import websockets
import json
import numpy as np
# Configuration
ACCOUNT_ID = "your-account-id"
API_TOKEN = "your-api-token"
MODEL = "@cf/pipecat-ai/smart-turn-v2"
# WebSocket endpoint
WEBSOCKET_URL = f"wss://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/run/{MODEL}?dtype=uint8"
async def run_inference(audio_data: bytes) -> dict:
async with websockets.connect(
WEBSOCKET_URL,
additional_headers={
"Authorization": f"Bearer {API_TOKEN}"
}
) as websocket:
await websocket.send(audio_data)
response = await websocket.recv()
result = json.loads(response)
# Response format: {'is_complete': True, 'probability': 0.87}
return result
def generate_test_audio():
noise = np.random.normal(128, 20, 8192).astype(np.uint8)
noise = np.clip(noise, 0, 255)
return noise
async def demonstrate_inference():
# Generate test audio
noise = generate_test_audio()
try:
print("\nTesting noise...")
noise_result = await run_inference(noise.tobytes())
print(f"Noise result: {noise_result}")
except Exception as e:
print(f"Error: {e}")
if __name__ == "__main__":
asyncio.run(demonstrate_inference())Deepgram speech models on Workers AI
Deepgram's speech-to-text and text-to-speech models are now available on Workers AI, running in Cloudflare locations worldwide. This delivers lower latency since speech recognition happens at the edge, close to users on the same network as Workers. It also enables WebRTC audio processing without leaving the Cloudflare network and gives access to Deepgram's audio models directly through Workers AI. Global scale is handled automatically by Cloudflare's network across 330+ cities.
All of these features are available now. Cloudflare Realtime Agents is in open beta, WebRTC audio as PCM in Workers is documented and ready for integration, smart-turn-v2 is available for testing through Workers AI, and Deepgram's @cf/deepgram/aura-1 and @cf/deepgram/nova-3 models are live.
export class MyAgent extends RealtimeAgent<Env> {
constructor(ctx: DurableObjectState, env: Env) {
super(ctx, env);
}
async init(agentId: string ,meetingId: string, authToken: string, workerUrl: string, accountId: string, apiToken: string) {
// Construct your text processor for generating responses to text
const textHandler = new MyTextHandler(this.env);
// Construct a Meeting object to join the RTK meeting
const transport = new RealtimeKitTransport(meetingId, authToken, [
{
media_kind: 'audio',
stream_kind: 'microphone',
},
]);
const { meeting } = transport;
// Construct a pipeline to take in meeting audio, transcribe it using
// Deepgram, and pass our generated responses through ElevenLabs to
// be spoken in the meeting
await this.initPipeline(
[transport, new DeepgramSTT(this.env.DEEPGRAM_API_KEY), textHandler, new ElevenLabsTTS(this.env.ELEVENLABS_API_KEY), transport],
agentId,
workerUrl,
accountId,
apiToken,
);
// The RTK meeting object is accessible to us, so we can register handlers
// on various events like participant joins/leaves, chat, etc.
// This is optional
meeting.participants.joined.on('participantJoined', (participant) => {
textHandler.speak(`Participant Joined ${participant.name}`);
});
meeting.participants.joined.on('participantLeft', (participant) => {
textHandler.speak(`Participant Left ${participant.name}`);
});
// Make sure to actually join the meeting after registering all handlers
await meeting.rtkMeeting.join();
}
async deinit() {
// Add any other cleanup logic required
await this.deinitPipeline();
}
}
