Running an LLM entirely in the browser

WebLLM is a JavaScript runtime that executes large language models directly on the user's device. Inference happens locally, so prompts and responses never cross the network, and your data cannot be used for model training. The runtime combines WebAssembly for CPU work with WebGPU for low-level access to the device's GPU.

WebLLM is distributed as an npm package. Install it with:

npm install @mlc-ai/web-llm

The package is part of the Machine Learning Compilation project. A standalone application is also available for trying out the technology.

Choosing a model

WebLLM supports a range of open models. The model name encodes its key specifications. For example, Llama-3.2-3B-Instruct-q4f32_1-MLC indicates:

  • The base model is Llama 3.2.
  • It has 3 billion parameters.
  • The “Instruct” suffix means it is fine-tuned for instruction-following assistants.
  • q4 refers to 4-bit quantization of weights.
  • f32_1 specifies uniform quantization with full-precision, 32-bit floating-point numbers.
  • The MLC suffix marks it as a build from Machine Learning Compilation.

Quantization reduces the number of bits used to represent model weights, trading precision for lower memory usage. Higher bit counts improve accuracy but consume more resources. Floating-point formats (F32 vs. F16) carry the same trade-off: full-precision 32-bit numbers are more accurate, while 16-bit numbers are faster and lighter but require compatible hardware.

File size scales with parameter count and precision. A 3-billion-parameter model at 4 bits per parameter can already require around 1.4 GB of storage. Smaller models are workable, but for translation and general knowledge, models with 7 billion parameters deliver better results—at roughly 3.3 GB or more.

Instantiate the engine and trigger the model download with the following code:

import {CreateMLCEngine} from '@mlc-ai/web-llm';
const engine = await CreateMLCEngine('Llama-3.2-3B-Instruct-q4f32_1-MLC', {
  initProgressCallback: ({progress}) =>  console.log(progress);
});

CreateMLCEngine takes the model identifier and an optional configuration object. The initProgressCallback option lets you observe and surface the download progress while the user waits.

Storing the model for offline use

The downloaded model is held in Cache API storage, the same storage mechanism used to make Service Worker–powered sites work offline. Unlike HTTP caching, the Cache API is programmable and fully under application control.

Once the model files are cached, WebLLM reads them locally and no longer makes network requests for inference. The application becomes fully offline-capable. Cache storage is isolated per origin: two different origins cannot share a cached model and would each need to download it separately.

In Chrome DevTools, you can inspect the cached files under Application > Storage > Cache storage.

Defining the conversation

The engine can be primed with an initial message history. Standard LLM chats have three roles:

  • System prompt: Defines the model's behavior and role. It can also feed domain-specific context that was not part of the model's training data. Only one system prompt is allowed.
  • User prompt: The user's input.
  • Assistant prompt: Prior model replies, which are optional.

Multiple user and assistant messages can provide the model with few-shot examples before it generates its own reply. A minimal chat setup for the to-do app would look like this:

const messages = [
  { role: "system",
    content: `You are a helpful assistant. You will answer questions related to
    the user's to-do list. Decline all other requests not related to the user's
    todos. This is the to-do list in JSON: ${JSON.stringify(todos)}`
  },
  {role: "user", content: "How many open todos do I have?"}
];

Sending messages and consuming the stream

With the engine ready, calls go through the engine.chat.completions property. Calling create() starts the inference. For the to-do app, responses are streamed so that text appears incrementally and the perceived waiting time is reduced:

const chunks = await engine.chat.completions.create({  messages,  stream: true, });

The method returns an AsyncGenerator. Iterate with a for await...of loop and read chunks as they arrive. Each chunk contains only the new tokens (the delta), so the complete reply must be assembled on the client:

let reply = '';

for await (const chunk of chunks) {
  reply += chunk.choices[0]?.delta.content ?? '';
  console.log(reply);
}

Streaming is well-supported on the web; HTML might be updated efficiently by leveraging DOMImplementation for progressively appended text. The streamed results are plain strings, so interpreting them as JSON or any structured format requires a separate parsing step.

Constraints and next steps

WebLLM works well but ships with limitations. The first run requires a very large download, and cached models cannot be shared across origins, forcing each web app to store its own copy. WebGPU inference is close to native speed but does not fully match it.

A proposed alternative is the Prompt API, an experimental browser feature that would reuse a single centrally downloaded model across multiple web applications, eliminating the per-origin download cost and enabling full execution speed.