Vercel AI SDK 3.3 adds tracing, multimodal attachments, and structured output streaming
The Vercel AI SDK, a TypeScript/JavaScript toolkit for building AI applications across frameworks like Next.js and Svelte, has reached version 3.3. The release introduces experimental support for OpenTelemetry-based tracing, file attachments in chat applications, and client-side streaming of structured objects, alongside several smaller additions.
OpenTelemetry tracing for model calls
Debugging AI applications requires insight into individual model calls, including timing, token usage, prompts, and generated content. SDK 3.3 adds an experimental telemetry layer built on OpenTelemetry, letting developers feed trace data into observability platforms such as Datadog, Sentry, Axiom, LangFuse, Braintrust, or LangSmith.
Vercel recommends configuring telemetry with @vercel/otel. Next.js projects deploying on Vercel can enable it by adding an instrumentation.ts file:
import { registerOTel } from '@vercel/otel';
export function register() {
registerOTel({ serviceName: 'your-project-nameapp' });
}
Recording is opt-in per function call via the experimental_telemetry option, which also accepts function IDs and custom metadata to identify call locations and enrich the recorded data:
const result = await generateText({
model: anthropic('claude-3-5-sonnet-20240620'),
prompt: 'Write a short story about a cat.',
experimental_telemetry: {
isEnabled: true,
functionId: 'my-awesome-function',
metadata: {
something: 'custom',
someOtherThing: 'other-value',
},
},
});
The AI SDK documentation covers the full telemetry setup, and Vercel offers a deployable Next.js template to get started.
File attachments in useChat
Chat interfaces often need to send more than text. The useChat() React hook's handleSubmit() now supports an experimental experimental_attachments parameter, allowing users to attach images, PDFs, and other media for both model input and inline preview.
Attachments can be provided in one of two forms. Passing a FileList object from a file input sends multiple files, which the hook converts into data URLs before forwarding them to the AI provider:
const { input, handleSubmit, handleInputChange } = useChat();
const [files, setFiles] = useState<FileList | undefined>(undefined);
return (
<form
onSubmit={(event) => {
handleSubmit(event, {
experimental_attachments: files,
});
}}
>
<input
type="file"
onChange={(event) => {
if (event.target.files) {
setFiles(event.target.files);
}
}}
multiple
/>
<input type="text" value={input} onChange={handleInputChange} />
</form>
);
Alternatively, a list of URLs can attach external resources or media content:
const { input, handleSubmit, handleInputChange } = useChat();
const [attachments] = useState<Attachment[]>([
{
name: 'earth.png',
contentType: 'image/png',
url: 'https://example.com/earth.png',
}
]);
return (
<form
onSubmit={event => {
handleSubmit(event, {
experimental_attachments: attachments,
});
}}
>
<input type="text" value={input} onChange={handleInputChange} />
</form>
)
Vercel provides a live example, a deployable template, and a multi-modal chatbot guide covering this workflow.
Streaming structured objects to the client
Generating structured data from natural language is a recurring pattern, and the new experimental useObject hook for React streams such objects directly to the client, enabling interfaces to render JSON incrementally as it is produced.
A typical implementation shares a schema between server and client code:
import { z } from 'zod';
export const expenseSchema = z.object({
expense: z.object({
category: z
.string()
.describe(
'Category of the expense. Allowed categories: ' +
'TRAVEL, MEALS, ENTERTAINMENT, OFFICE SUPPLIES, OTHER.',
),
amount: z.number().describe('Amount of the expense in USD.'),
date: z
.string()
.describe('Date of the expense. Format yyyy-mmm-dd, e.g. 1952-Feb-19.'),
details: z.string().describe('Details of the expense.'),
}),
});
export type PartialExpense = DeepPartial<typeof expenseSchema>['expense'];
export type Expense = z.infer<typeof expenseSchema>['expense'];
The server route uses streamObject to invoke the language model and stream the generated object:
import { anthropic } from '@ai-sdk/anthropic';
import { streamObject } from 'ai';
import { expenseSchema } from './schema';
// Allow streaming responses up to 30 seconds
export const maxDuration = 30;
export async function POST(req: Request) {
const { expense }: { expense: string } = await req.json();
const result = await streamObject({
model: anthropic('claude-3-5-sonnet-20240620'),
system:
'You categorize expenses into one of the following categories: ' +
'TRAVEL, MEALS, ENTERTAINMENT, OFFICE SUPPLIES, OTHER.' +
// provide date (including day of week) for reference:
'The current date is: ' +
new Date()
.toLocaleDateString('en-US', {
year: 'numeric',
month: 'short',
day: '2-digit',
weekday: 'short',
})
.replace(/(\w+), (\w+) (\d+), (\d+)/, '$4-$2-$3 ($1)') +
'. When no date is supplied, use the current date.',
prompt: `Please categorize the following expense: "${expense}"`,
schema: expenseSchema,
onFinish({ object }) {
// you could save the expense to a database here
},
});
return result.toTextStreamResponse();
}
On the client, the hook previews the partial expense while streaming and appends the completed item once generation finishes:
'use client';
import { experimental_useObject as useObject } from 'ai/react';
import {
Expense,
expenseSchema,
PartialExpense,
} from '../api/expense/schema';
import { useState } from 'react';
export default function Page() {
const [expenses, setExpenses] = useState<Expense[]>([]);
const { submit, isLoading, object } = useObject({
api: '/api/expense',
schema: expenseSchema,
onFinish({ object }) {
if (object != null) {
setExpenses(prev => [object.expense, ...prev]);
}
},
});
return (
<div>
<form onSubmit={e => {
e.preventDefault();
const input = e.currentTarget.expense as HTMLInputElement;
if (input.value.trim()) {
submit({ expense: input.value });
e.currentTarget.reset();
}
}}
>
<input type="text" name="expense" placeholder="Enter expense details"/>
<button type="submit" disabled={isLoading}>Log expense</button>
</form>
{isLoading && object?.expense && (
<ExpenseView expense={object.expense} />
)}
{expenses.map((expense, index) => (
<ExpenseView key={index} expense={expense} />
))}
</div>
);
}
Rendering handles partial objects through optional chaining and nullish coalescing:
const ExpenseView = ({ expense }: { expense: PartialExpense | Expense }) => (
<div>
<div>{expense?.date ?? ''}</div>
<div>${expense?.amount?.toFixed(2) ?? ''}</div>
<div>{expense?.category ?? ''}</p></div>
<div>{expense?.details ?? ''}</div>
</div>
);
A sample expense-tracker app demonstrates the pattern, and a supporting template is available for deployment. Full details are in the object generation documentation.
Extended model call controls
Several new options give developers more control over model interactions:
JSON schema support: The
jsonSchemafunction provides an alternative to Zod schemas for tools and structured object generation. It supports type annotations and an optional validation function, which Vercel says is useful for dynamic tool and structure definitions.Stop sequences: The
stopSequencesoption onstreamTextandgenerateTextdefines text sequences that halt generation, giving more command over where output ends.Custom headers: A
headersoption on most SDK functions allows sending additional HTTP headers, useful for tasks like carrying tracing information or enabling provider-specific beta features.
The release also adds providers for AWS Bedrock and Chrome AI (community-contributed), with a full changelog available in the migration guide. As with any experimental SDK feature, note that APIs may change between patch versions. Given that the experimental capabilities are opt-in, teams can adopt version 3.3 incrementally without risking instability in existing production applications.
