AI SDK 3.4 adds middleware, a new stream protocol, and multi-step tool calls
The AI SDK, an open-source toolkit for building AI applications with JavaScript and TypeScript, has reached version 3.4. The release focuses on making the SDK more flexible for different backends and use cases, introducing language model middleware, a new data stream protocol, structured output modes, and multi-step generation support.
Language model middleware
The headline feature is language model middleware, a modular mechanism for intercepting and modifying model calls. It provides three methods: transformParams for changing calls before they go out (useful for retrieval-augmented generation), and wrapGenerate/wrapStream for implementing logic around model interactions, such as logging, caching, or guardrails.
Because middleware is packaged as a self-contained unit, it can be shared and reused across projects. The AI SDK 3.4 middleware is experimental, so its API may shift in patch releases.
A typical RAG implementation might use transformParams to fetch relevant content and append it to the prompt based on the user's latest message, keeping that logic separate from the core application code:
import type {
Experimental_LanguageModelV1Middleware as LanguageModelV1Middleware
} from "ai";
export const yourRagMiddleware: LanguageModelV1Middleware = {
transformParams: async ({ params }) => {
const lastUserMessageText = getLastUserMessageText({
prompt: params.prompt,
});
// do not use RAG when the last message is not a user message
if (lastUserMessageText == null) return params;
// find relevant sources for the last user message:
const sources = findSources({ text: lastUserMessageText });
const instruction =
"Use the following information to answer the question:\n" +
sources.map((chunk) => JSON.stringify(chunk)).join("\n");
// return params with the instruction added to the last user message:
return paramsWithUpdatedLastUserMessage({ params, text: instruction });
},
};
Middleware attaches directly to a model instance:
import {
streamText,
experimental_wrapLanguageModel as wrapLanguageModel,
} from 'ai';
import { openai } from '@ai-sdk/openai'
const result = await streamText({
model: wrapLanguageModel({
model: openai('gpt-4o'),
middleware: yourLanguageModelMiddleware,
}),
prompt: 'What is founder mode?',
});
Vercel's RAG template demonstrates middleware applied to an internal knowledge base use case, with the source code available on GitHub.
Data stream protocol for any backend
Previously, the AI SDK UI—including the useChat and useCompletion hooks—required AI SDK Core on the backend. Version 3.4 decouples the frontend from that dependency by introducing a Data Stream Protocol: a specification any backend can implement to send data that AI SDK UI hooks can consume.
Using the protocol requires two steps:
- Send data from the backend following the protocol specification.
- Point the AI SDK UI hooks at the custom backend endpoint via their
apiconfiguration.
Reference implementations are available for Python FastAPI as well as JavaScript frameworks including Express, Fastify, Hono, and Nest.js. The protocol also enables fully custom chat frontends that still use AI SDK Core for the model interactions.
New structured output modes
The generateObject and streamObject functions gain an output parameter with a default value of object. New modes let you control the shape of the result more precisely:
- Object mode (default): forces the model to return a single object conforming to the provided schema.
const { object } = await generateObject({
model: openai('gpt-4o'),
schema: z.object({
product: z.object({
name: z.string(),
description: z.string(),
price: z.number(),
}),
}),
prompt: 'Generate a description for a new smartphone.',
});
- Array mode: generates an array of objects matching a given element schema. When paired with the React
useObjecthook, it streams complete array elements as they become available, avoiding layout shifts in the UI.
const { elementStream: destinations } = await streamObject({
model: openai('gpt-4o'),
output: 'array',
schema: z.object({
city: z.string(),
country: z.string(),
description: z.string(),
attractions: z.array(z.string()).describe('List of major attractions.'),
}),
prompt: 'What are the top 5 cities for short vacations in Europe?',
});
for await (const destination of destinations) {
console.log(destination); // destination is a complete array element
}
- Enum mode: constrains output to one value from a preset list, which suits classification tasks like sentiment analysis. This mode works only with
generateObject, given the short outputs involved.
const { object: movieGenre } = await generateObject({
model: openai('gpt-4o'),
output: "enum",
enum: ["action", "comedy", "drama", "horror", "sci-fi"],
prompt:
`Classify the genre of this movie plot:` +
`"A group of astronauts travel through a wormhole ` +
`in search of a new habitable planet for humanity."`,
});
- No-schema mode: allows omitting a schema entirely, useful for dynamic user-driven requests where the output shape isn't known ahead of time.
Multi-step generation in streamText
The steps parameter, previously added to generateText, is now available for streamText. This enables real-time streaming responses that automatically handle multiple rounds of tool calling and result feeding without manual orchestration.
import { z } from 'zod';
import { streamText, tool } from 'ai';
import { openai } from '@ai-sdk/openai';
const result = await streamText({
model: openai('gpt-4o'),
messages,
tools: {
listOrders: tool({
description: "list all orders",
parameters: z.object({ userId: z.string() }),
execute: async ({ userId }) => getOrders(userId)
}),
viewTrackingInformation: tool({
description: "view tracking information for a specific order",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) => getTrackingInformation({ orderId })
}),
},
maxSteps: 3,
onStepFinish({ toolResults, usage }) {
// your own logic
// e.g. saving chat history or recording usage for each step
},
onFinish({ steps }) {
// your own logic
// e.g. saving chat history or recording usage for the entire generation
}
});
Intermediate tool calls and results from each step are accessible through the steps property on the result object. An onStepFinish callback fires when each step completes, and the full set of steps is available in the onFinish callback after the whole generation finishes. The SDK handles the looping and orchestration automatically for multi-tool scenarios.
Telemetry and observability updates
AI SDK 3.4 brings tracing improvements that align with the OpenTelemetry Semantic Conventions for GenAI operations. Since the SDK added OpenTelemetry support in version 3.3, observability platforms have published integration guides for SDK users, including Braintrust, Langfuse, and Arize AI's OpenInference package.
The newest release adds telemetry attributes for response ID, response model, and response timestamp, along with performance metrics including time to first chunk and average output tokens per second. These additions support more accurate cost tracking, user feedback collection, and detailed performance dashboards. Full details are in the telemetry documentation.
Deterministic testing with mock providers
Language models are non-deterministic, slow, and expensive to call, which makes unit testing difficult. AI SDK 3.4 introduces mock providers and test helpers to address this. For example, developers can create a mock response with generateText like this:
import { generateText } from 'ai';
import { MockLanguageModelV1 } from 'ai/test';
const result = await generateText({
model: new MockLanguageModelV1({
doGenerate: async () => ({
rawCall: { rawPrompt: null, rawSettings: {} },
finishReason: 'stop',
usage: { promptTokens: 10, completionTokens: 20 },
text: `Hello, world!`,
}),
}),
prompt: 'Hello, test!',
});
These utilities let you control SDK output and test code repeatably without contacting a model provider. See the testing documentation for more.
Provider additions and enhancements
The 3.4 release expands provider support with new features for existing providers and several new embedding models:
- Anthropic: Prompt caching support for cheaper and faster applications.
- Amazon Bedrock: Support for Bedrock Guardrails to implement safeguards and modify model responses.
- Cohere: Embedding model support added.
- OpenAI: Structured outputs that always generate JSON adhering to a schema, plus reasoning token support for o1-preview and o1-mini.
- Google Generative AI: Search grounding for Gemini, giving the model access to up-to-date information, plus embedding model support.
- Mistral: Image support for Pixtral.
- LlamaIndex: A new adapter for using LlamaIndex tools and abstractions with the AI SDK.
- New community providers: Cloudflare Workers AI, Portkey, Anthropic Vertex, and FriendliAi.
What's next
AI SDK 4.0 is planned as a maintenance release that promotes experimental features to stable and removes deprecated ones. In the meantime, developers can explore the latest guides, browse the Template Gallery for inspiration, or discuss projects in the GitHub Discussions community.
