ElevenLabs on AI Gateway

Vercel has added 14 ElevenLabs models to AI Gateway through a partnership with ElevenLabs. For the first 14 days of the launch, through November 9th, every one of these models is available at 50% off.

The model lineup

The catalog covers six kinds of audio work, all reachable with a single gateway API key:

• Speech to text: transcribe recordings or caption live speech.
• Streaming text to speech: begin playback while the language model is still generating its answer.
• Realtime voice agents: a spoken back-and-forth in one live session, for support lines, tutors or in-app assistants.
• Music: generate a complete song or instrumental from a text description, at a requested length.
• Sound effects: produce interface or game sounds and ambience that loops cleanly.
• Multi-voice dialogue: voice an entire conversation in one call, with a separate voice per speaker.

A pipeline can therefore stay entirely inside the gateway: transcribe input, reason over it with any of the hundreds of available models, then answer via ElevenLabs, optionally alongside generated music and effects. Usage, cost and trace data land in the same views as the rest of your AI Gateway traffic.

Transcription into speech

Scribe v2 Realtime transcribes speech as it is spoken, and your chosen language model composes the reply. Eleven Flash v2.5, a low-latency voice model, then speaks that reply as it streams, so the user hears the start of the answer before generation has finished. The steps are independent models, so any one of them can be swapped without reworking the rest of the pipeline.

The reference scenario is a traveler asking about a connecting flight; the text model weighs flight status against gate distance before the speech model responds, working from a 16 kHz microphone stream.

import {
  experimental_streamSpeech as streamSpeech,
  experimental_streamTranscribe as streamTranscribe,
  streamText,
} from 'ai';
const heard = streamTranscribe({
  model: 'elevenlabs/scribe-v2-realtime',
  audio: microphoneStream,
  inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
});
const { textStream } = streamText({
  model: 'anthropic/claude-haiku-5.5',
  prompt: await heard.text,
});
const { audioStream } = streamSpeech({
  model: 'elevenlabs/eleven-flash-v2.5',
  text: textStream,
  voice: 'hpp4J3VqNfWAUOO0d1Us',
});

Two-way conversations

Eleven Realtime v1 Max and Eleven Realtime v1 Mini handle the full spoken exchange in a single model, suited to support calls and in-app assistants. Both accept user audio and speak a reply within one WebSocket session, eliminating separate transcription and speech steps; a sample exchange — “Can I change the delivery address for my order?” answered with a request for the order number — runs over that session.

import { gateway } from 'ai';
const { token, url } = await gateway.experimental_realtime.getToken({
  model: 'elevenlabs/eleven-realtime-v1-max',
});

The browser owns the WebSocket connection, and the server only mints a short-lived token for the selected model, keeping the API key off the client. On the client side, the useRealtime hook from @ai-sdk/react manages the WebSocket, microphone capture and audio playback. Its sessionConfig needs an ElevenLabs voice ID and turnDetection: { type: 'semantic-vad' }. These models support semantic voice activity detection but not server-vad; the realtime guide covers full browser setup.

Music, effects and dialogue

Music v2.5 and Music v2 compose tracks from a description — genre, mood, instruments — returning a finished piece of the requested length, with or without vocals, and sections can be laid out individually with their own style and duration. Sound Effects v2 does the same for short sounds such as footsteps, a door slam or looping rain.

These three models all use generateSpeech, where text describes the sound rather than supplying words to read. A request for a 30-second track:

import { generateSpeech } from 'ai';
const { audio } = await generateSpeech({
  model: 'elevenlabs/music-v2.5',
  text: 'A mellow lo-fi beat with soft piano',
  providerOptions: { elevenlabs: { musicLengthMs: 30000 } },
});

For scripted material, Eleven v4, Eleven v4 Turbo and Eleven v3 perform a multi-speaker script in one request. Each line is assigned an ElevenLabs voice ID, a voice change starts a new turn, and the result is one continuous recording with no clips to join. Lines can also stream in from a language model writing the script.

import { experimental_streamSpeech as streamSpeech } from 'ai';
const host = 'hpp4J3VqNfWAUOO0d1Us';
const guest = 'CwhRBWXzGAHq8TQ4Fs17';
const { audioStream } = streamSpeech({
  model: 'elevenlabs/eleven-v4',
  text: [
    { type: 'text', text: 'Did the deploy go out?', voice: host },
    { type: 'text', text: 'It shipped an hour ago.', voice: guest },
  ],
  providerOptions: { elevenlabs: { dialogue: { voices: [host, guest] } } },
});

Gateway terms

AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover and performance optimizations aimed at uptime above what a single provider offers. It includes custom reporting, Zero Data Retention support, budgets for API keys and routing rules. Provider pricing is passed through with no markup and no platform fee on inference, BYOK requests included.

The AI Gateway audio docs carry examples for each ElevenLabs model, and all audio models can be browsed on AI Gateway.