Understand LLM sizes

The real meaning of model size

When a model is described as "Gemma 2B" or "Mistral 7B," the number refers to parameters, not download size. A parameter is a learned weight inside the model, and the count reflects how many of these weights the model uses. Gemma 2B has approximately 2 billion parameters; Mistral 7B has around 7 billion. The approximation matters: size classes like "2B" or "7B" group models by rough scale rather than exact counts.

Parameter count correlates with capability. Larger models can capture more complex language patterns and handle nuanced prompts, and they tend to be trained on larger datasets. Within the same model family, a larger variant will generally outperform a smaller one for the same task, meaning you may need to do more prompt engineering with a smaller model to get comparable results.

Smaller models, real use cases

Very large LLMs—such as GPT-4 or Gemini Ultra—are believed to run into the hundreds of billions or even trillions of parameters, though exact numbers are often undisclosed. These models require dedicated servers with specialized hardware like TPUs. For web developers, the more relevant question is whether a model is small enough to run on a user's device.

That question is harder to answer than it sounds. There is no standard baseline device for the web; a low-end Android phone and a laptop with a dedicated GPU are wildly different targets. Optimization varies by hardware and framework, and there's no programmatic way to know whether a given model will fit on a given device. Download feasibility depends on factors like available VRAM.

Still, empirical evidence from early 2024 shows that models ranging from a few million to a few billion parameters can run in the browser on consumer hardware. Examples include:

• Gemma 2B via the MediaPipe LLM Inference API, which can work on CPU-only devices.
• DistilBERT via Transformers.js.

Expect the landscape to shift quickly. WebAssembly and WebGPU support is expanding across libraries, and new optimizations are making on-device inference more practical. Smaller, highly performant models are also becoming more common thanks to shrinking techniques.

Checking download size

More parameters mean a larger download. While there are ways to estimate download size from parameter count, they tend to be complex, and as of early 2024 model documentation rarely includes download size. For on-device work, the practical approach is to check the Network panel in Chrome DevTools or equivalent browser tools and measure the actual transferred bytes.

Shrinking an LLM

Several techniques can significantly cut a model's memory footprint:

• LoRA (Low-Rank Adaptation): A fine-tuning approach that freezes the pre-trained weights, reducing the number of parameters that need to be updated or stored.
• Pruning: Removing less important weights from the model to shrink its size.
• Quantization: Reducing weight precision from, say, 32-bit floating point down to 8-bit representations.
• Knowledge distillation: Training a smaller model to imitate a larger, pre-trained one.
• Parameter sharing: Reusing the same weights in multiple parts of the model so fewer unique parameters are needed.

When is a model an LLM?

The boundary is fuzzy. Some in the AI community reserve the term "LLM" for models with billions of parameters, classifying something like DistilBERT as a simpler NLP model. Others use the label more broadly to include smaller but still capable models. Either way, the definition matters less than whether a model fits your use case and your target hardware.