Sizing up client-side AI
Running AI models directly in the browser avoids server round-trips, cuts infrastructure costs, removes the need for API keys, and keeps user data on the device. JavaScript libraries such as TensorFlow.js, Transformers.js, and MediaPipe GenAI make this possible across browsers. The tradeoff is that you are asking users to download more files and their devices to work harder.
Before committing to a client-side approach, be honest about your use case. Not every feature needs on-device inference, and if AI is part of a critical user journey, you need a fallback when the model cannot run or download cleanly.
Choosing and shrinking your model
Library and model size need the same scrutiny you would give any dependency. The numbers involved are wide-ranging: BudouX, a model for character breaking in Asian languages, is just 9.4KB gzipped, and MediaPipe’s lightweight language detection model is about 315KB. Handpose, a vision model, totals roughly 13.4MB with all related resources—larger than typical frontend bundles but comparable to the median web page of about 2.2MB (2.6MB on desktop). By contrast, Gen AI models like DistilBERT weigh in at 67MB, and even compact LLMs such as Gemma 2B can be around 1.3GB.
Use your browser’s developer tools to measure the exact download size of any model you plan to ship. Two practical strategies help: compare accuracy across models of different sizes (fine-tuning and shrinking techniques can cut a model down dramatically while retaining acceptable quality), and prefer task-specialized models over generic LLMs when the job is narrow, such as sentiment or toxicity detection.
Pre-download checks
Not every device can run a given model, even when hardware specs look adequate under ideal conditions. Until the platform provides a unified solution, you can reduce surprises with a few checks today:
- Detect WebGPU before loading anything if your chosen library depends on it. Several client-side AI libraries, including Transformers.js v3 and MediaPipe, use WebGPU without always falling back to Wasm when it’s unavailable. A feature-detection check is the mitigation.
- Filter out underpowered devices using
Navigator.hardwareConcurrency,Navigator.deviceMemory, and the Compute Pressure API. These APIs are intentionally imprecise to prevent fingerprinting and lack universal support, but they can still identify devices that are very unlikely to handle inference.
Large model downloads warrant an explicit warning. Desktop users tolerate sizable downloads far better than mobile users; detect the platform via the mobile property from the User-Agent Client Hints API, falling back to the User-Agent string when needed.
Keep the transfer lean in three ways:
- Delay the download until there is reasonable certainty the model will actually be used. A type-ahead suggestion feature, for example, should defer the fetch until the user begins typing.
- Cache explicitly with the Cache API instead of relying solely on the implicit HTTP cache, which avoids re-downloading the model on every return visit.
- Fetch in chunks with a helper like
fetch-in-chunks, which splits a large download into smaller parts that are more resilient to interruptions.
Download and preparation phase
Do not block the user interface while a model downloads or prepares. Core page features should remain functional even if the model is not ready yet, so prioritise smooth interaction over model availability.
Progress indicators should show both what is complete and what remains. If your AI library manages the download internally, check whether it exposes progress events you can surface; if not, consider requesting the feature. For your own download code, chunked fetching libraries like fetch-in-chunks support progress callbacks for this purpose.
Network interruptions are inevitable, and large model downloads make them more likely. Plan for the user going offline mid-transfer: let them know the connection is broken and try to resume the download when connectivity returns. Chunked downloads make this recovery far simpler.
After the bytes arrive, model preparation can still be a heavy CPU-bound task that janks the main thread. Move this work, and any other expensive AI-related processing, into a web worker:
During inference
Inference itself can be computationally heavy. When it executes through WebGL, WebGPU, or WebNN, the GPU handles it in a separate process, so the UI thread stays responsive. CPU-based fallbacks like Wasm do not have this property: move inference to a web worker in that case to keep the page interactive. Keeping all AI code (fetch, preparation, inference) in the worker simplifies the implementation, even when the GPU handles inference.
Runtime failures still happen after all your device checks pass. The user might start resource-intensive work elsewhere in the browser, for example. Wrap inference calls in try/catch and handle runtime errors accordingly. For WebGPU specifically, listen for both uncapturederror and GPUDevice.lost events, since the latter can fire when the GPU actually resets under pressure.
When inference will take longer than the few hundred milliseconds users perceive as instantaneous, communicate that the model is working with a visible status indicator and a light animation to ease the wait.
Finally, make inference cancellable. If the user adjusts their input mid-generation, the system should not waste CPU or GPU cycles producing an answer that will be discarded.
