Build a local and offline-capable chatbot with the Prompt API

On-device LLM inference without a model picker

Chrome's built-in Prompt API is part of a family of proposed browser AI interfaces currently going through the standards process. Unlike WebLLM, the Prompt API ships with a fixed foundation model that the browser downloads and manages. That model is shared across all origins, so no per-site download or model selection step is needed before you can start generating text.

The API is available through the Early Preview Program, and there is also an origin trial for Chrome Extensions so you can test it with real extension users. A GitHub issue exists for developers who want the ability to choose between different models.

Starting a session with a system prompt

You initialize a conversation with the create() method on the LanguageModel interface. The configuration object can specify a system prompt, which keeps working the same way as it does in other LLM libraries, but the Prompt API also offers a shorthand field for the same purpose:

const session = await LanguageModel.create({
  initialPrompt: [
    {
      type: 'system',
      content: `You are a helpful assistant. You will answer questions related
        to the user's to-do list. Decline all other requests not related to the
         user's todos. This is the to-do list in JSON: ${JSON.stringify(todos)}`,
    },
  ],
});

Once the session object exists, you can send prompts to it. The Prompt API splits its response methods into two distinct paths instead of using a single streaming configuration flag:

• prompt() resolves to the complete response string at once.
• promptStreaming() returns an async iterable that yields the growing response text.

This differs from WebLLM in an important way: the streaming method on the Prompt API gives you the full accumulated response on each iteration, so you don't have to concatenate the chunks yourself. You can render the streamed output directly as it comes in.

Note that the very first prompt may take a long time to answer if the model has not yet been downloaded to the browser. On subsequent visits—or any time the model is already present because another origin triggered the download—inference starts immediately.

Here is the core request/response loop in practice:

const stream = session.promptStreaming("How many open todos do I have?");
for await (const reply of stream) {
  console.log(reply);
}

Demo and summary

A working to-do application using the Prompt API is available as a sample, and the full source is on GitHub.

Both WebLLM and Chrome's Prompt API demonstrate the practical advantage of running LLMs locally: fully offline capability, stronger privacy, and no per-request cloud costs. Cloud providers may offer larger models and consistent performance on weak hardware, but the on-device tradeoff is increasingly compelling for many use cases.