Opening the mic: adding AI participants to global real-time media

OpenAI added WebRTC support to its Realtime API on December 17, 2024. Combined with Cloudflare Calls, this opens a path to multimodal applications that were previously impractical: multiple remote users can simultaneously see and converse with a voice or video AI over a live media session.

Before this, audio and video AI interactions were largely single-player. Only one person at a time could talk to the model, or everyone had to be in the same room. With Cloudflare Calls routing media between participants and the Realtime API, an AI can now join a distributed conversation just like any other attendee.

Use cases beyond the demo

Cloudflare demonstrated the concept in its Orange Meets video-conferencing app. The working example lets you invite ChatGPT into a meeting, but the underlying pattern generalizes well:

  • Enterprise meetings: A secure "corporate AI" with access to internal data could be pulled into calls to answer questions like "Do we have any open Jira tickets about this?" or "How much more did customer X spend with us versus last year?"
  • Consumer entertainment: Global livestreams and interactive games, such as a live murder mystery played across cities, become feasible when every viewer can talk directly to an AI host.

WebRTC vs. WebSockets for real-time media

Real-time products have historically leaned on WebSockets, which run over a single persistent TCP connection. That works well for text chat or game state sync and, notably, eliminates the need for clients to poll a server for updates. Cloudflare supports WebSockets both across its network and in AI Gateway.

For voice and video, though, WebSockets create a latency problem. Audio must be chunked into 100–500 ms segments, and the chunk size plus head-of-line blocking becomes the floor on how fast you can deliver a real-time multimodal experience. As the number of users and geographies grows, that delay quickly becomes unacceptable.

WebRTC takes a different approach: native audio and video tracks flow over UDP-based channels directly between participants. No chunking is required, so you can send multiple users' audio and video to an AI and get the model's responses back in real time without the extra buffering.

How Cloudflare Calls handles the infrastructure

Standing up WebRTC infrastructure manually — media routers, TURN relays, global coverage — is a well-known chore. Cloudflare Calls abstracts that away: it acts as a single mesh network that routes each user to the closest Cloudflare data center via anycast. It can interconnect with other WebRTC services, such as OpenAI's Realtime API, and fan out the AI's response to hundreds or thousands of listeners with near-zero added latency.

Media traffic through Cloudflare Calls is encrypted by default. The Orange Meets demo goes further with a button that lets participants explicitly decide when the AI may listen and interact, giving meeting hosts granular control over when the model is in the conversation.

Wiring ChatGPT into a session

Cloudflare Calls is built around three primitives: Applications, Sessions, and Tracks. A Session corresponds directly to a WebRTC PeerConnection between a client and the nearest Cloudflare data center. Within a session, one or more Tracks carry audio, video, or data, aligning with the MediaStreamTrack concept.

To bring ChatGPT into the Orange Meets conference, the integration treats the AI as another track within an existing session. The routing logic in the demo sets up bidirectional connections: humans can hear ChatGPT, and ChatGPT can hear the humans.

// Connect Cloudflare Calls sessions and tracks like a switchboard
async function connectHumanAndOpenAI(
	humanSessionId: string,
	openAiSessionId: string
) {
	const callsApiHeaders = {
		Authorization: `Bearer ${APP_TOKEN}`,
		'Content-Type': 'application/json',
	}
	// Pull OpenAI audio track to human's track
	await fetch(`${callsEndpoint}/sessions/${humanSessionId}/tracks/new`, {
		method: 'POST',
		headers: callsApiHeaders,
		body: JSON.stringify({
			tracks: [
				{
					location: 'remote',
					sessionId: openAiSessionId,
					trackName: 'ai-generated-voice',
					mid: '#user-mic',
				},
			],
		}),
	})
	// Pull human's audio track to OpenAI's track
	await fetch(`${callsEndpoint}/sessions/${openAiSessionId}/tracks/new`, {
		method: 'POST',
		headers: callsApiHeaders,
		body: JSON.stringify({
			tracks: [
				{
					location: 'remote',
					sessionId: humanSessionId,
					trackName: 'user-mic',
					mid: '#ai-generated-voice',
				},
			],
		}),
	})
}

All code for the demo is publicly available in the Orange Meets GitHub repository.

Trying it yourself

The Cloudflare Calls + OpenAI Realtime API demo is live, and the full implementation is on GitHub. To build your own real-time interactive AI features, start with the Cloudflare Calls documentation.