Unweight: Compressing LLM Weights for Faster Inference
Running inference within 50ms of 95% of the world's Internet-connected population demands ruthless efficiency with GPU memory. Our Rust-based inference engine, Infire, improved memory utilization, and our model scheduling platform, Omni, eliminated cold-starts. Now we're tackling the next bottleneck: model weights themselves.
Generating a single token from an LLM requires reading every model weight from GPU memory. On the NVIDIA H100 GPUs used across our datacenters, tensor cores process data nearly 600 times faster than memory delivers it — the bottleneck is memory bandwidth, not compute. Every byte crossing the memory bus is a byte that could have been avoided if weights were smaller.
We built Unweight, a lossless compression system that reduces model weights by 15–22% while preserving bit-exact outputs, requiring no special hardware. The breakthrough: decompressing weights in fast on-chip memory and feeding them directly to tensor cores avoids an extra round-trip through slow main memory. Depending on the workload, Unweight's runtime selects from multiple execution strategies, with an autotuner choosing the best one per weight matrix and batch size.
We're publishing a technical paper and open sourcing the GPU kernels to encourage innovation in this space.
Initial results on Llama-3.1-8B show ~30% compression of Multi-Layer Perceptron (MLP) weights alone. Because Unweight works selectively on decoding parameters, overall model size drops 15-22%, saving ~3 GB VRAM.
Unweight allows more models to fit on a single GPU
Why lossless compression matters
Recent research explores creative compression approaches, most commonly quantization — converting 32- or 16-bit floats to 8- or 4-bit integers. But quantization is lossy: multiple 16-bit values can map to the same 4-bit integer, degrading output quality unpredictably. For production inference serving diverse use cases, we needed lossless compression that preserves exact model behavior.
Existing systems like Huff-LLM, ZipNN, and ZipServ don't fit our requirements. ZipNN compresses weights for distribution and storage, decompressing on CPU. Huff-LLM requires custom FPGA hardware. ZipServ fuses decompression with GPU inference but targets consumer GPUs, incompatible with our H100s. None provide lossless inference-time decompression on Hopper GPUs integrated with our Rust-based inference engine.
The real challenge isn't basic compression — exponent bytes in BF16 weights are highly redundant and entropy coding works well. The difficulty is decompressing fast enough to avoid slowing inference. On an H100, tensor cores idle waiting for memory, but that idle capacity can't simply be repurposed. Each GPU compute unit runs either the decompression kernel or matrix multiplication, not both, due to shared memory constraints. Decode latency not perfectly overlapped with matrix multiplication becomes directly additive to token latency.
Unweight's solution: decompress weights in fast on-chip shared memory and feed results directly to tensor cores. Making this work efficiently across batch sizes and weight shapes is where the engineering complexity lives.
Understanding BF16 weight structure
Every number in an AI model is stored as a 16-bit "brain float" (BF16), with three parts:
- Sign (1 bit): positive or negative
- Exponent (8 bits): the magnitude
- Mantissa (7 bits): the precise value within that magnitude
Here's how one of these weights breaks down:
Sign and mantissa vary unpredictably across weights, resembling random data that can't be meaningfully compressed. The exponent tells a different story.
The exponent's exploitable redundancy
Across trained LLMs, of 256 possible exponent values, only a handful dominate. The top 16 exponents cover over 99% of all weights in a typical layer. Information theory suggests only ~2.6 bits are needed to represent this distribution — far less than the 8 bits allocated.
Exponent value distribution in a typical LLM layer
Unweight exploits this redundancy by leaving sign and mantissa untouched and compressing only the exponent byte using Huffman coding, which assigns short codes to common values and longer codes to rare ones. Given the skewed exponent distribution, this achieves roughly 30% compression on the exponent stream.
This applies selectively to MLP weight matrices (gate, up, and down projections), which comprise roughly two-thirds of a model's parameters and dominate memory traffic during token generation. Attention weights, embeddings, and layer norms remain uncompressed. Overall, this translates to about 20% reduction in MLP weight size.
The few weights with rare exponents are handled separately: if any weight in a row of 64 has an exponent outside the top-16 palette, the entire row is stored verbatim. This eliminates per-element branching in the hot path — one decision per row instead of checking every weight for edge cases.
The GPU memory bottleneck
An NVIDIA H100 GPU has two relevant memory types:
- High Bandwidth Memory (HBM): large but relatively slow; model weights live here
- Shared memory (SMEM): tiny but extremely fast; stages data immediately before computation
Generating each token requires reading the full weight matrix from HBM. The memory bus between HBM and SMEM is the bottleneck — not the math itself. Fewer bytes across the bus means faster token generation.
Compression reduces bytes crossing the bus, but the GPU can't compute on compressed data. Weights must be decompressed first. Most prior work decompresses entire weight matrices back into HBM before standard matrix multiplication. This helps storage capacity but not bandwidth — the full uncompressed matrix still reads from HBM for every token.
Four approaches to using compressed weights
No single strategy for using compressed weights works best. The right approach depends on batch size, weight matrix shape, and available GPU time for decompression. Unweight offers four compressed execution pipelines with different tradeoffs between decompression effort and computation complexity:
Four different execution pipelines
At one end, full decode completely reconstructs the original BF16 weights and hands them to NVIDIA's cuBLAS library for standard matrix multiplication. This is the simplest path, running cuBLAS at full speed on ordinary data, but writes the most bytes back to main memory. It works well at small batch sizes where matrix multiplication is tiny and custom kernel overhead dominates.
At the other end, direct palette skips preprocessing entirely. Weights are pre-transcoded to a compact 4-bit format at model load time, and the matrix multiplication kernel reconstructs BF16 values on the fly from these indices. This incurs zero preprocess cost, but the kernel does more work per element.
Two independent paths sit in between: one decodes only exponent bytes (halving preprocess traffic), and another transcodes to 4-bit palette indices at runtime (quartering it). Both use a reconstructive matrix multiplication custom kernel that loads compressed data, reconstructs BF16 in fast shared memory, and feeds it directly to tensor cores without a round-trip through main memory.
Why no single pipeline wins
Less preprocessing means less data written to HBM, freeing the memory bus sooner. But it shifts reconstruction work onto the matmul kernel. Whether this tradeoff pays off depends on the situation:
- Small batch sizes (1-64 tokens): Matmul is tiny with limited computation to overlap, and custom kernel fixed costs dominate. Full decode + cuBLAS often wins due to cuBLAS's lower overhead.
- Large batch sizes (256+ tokens): Matmul runs long enough to absorb additional reconstruction work. Lighter preprocessing finishes faster, and freed bus bandwidth with compute overlap pays off. The palette or exponent pipelines pull ahead.
Different weight matrices within the same layer can favor different pipelines. The "gate" and "up" projections have different dimensions than the "down" projection, changing matrix multiplication operation order and performance tradeoffs.
Throughput vs. pipeline strategy
Unweight doesn't hard-code a single strategy. The runtime picks the best pipeline for each weight matrix at each batch size, informed by autotuning that measures actual end-to-end throughput on target hardware.
How reconstructive matmul works
Three of the four pipelines use a custom matrix multiplication kernel that fuses decompression with computation. This kernel loads compressed data from HBM, reconstructs original BF16 values in shared memory, and feeds them directly to tensor cores in one operation. Reconstructed weights never exist in main memory.
Traditional decompression vs. Unweight
With Unweight, ~30% fewer bytes cross the memory bus for MLP weight matrices
Inside this kernel, GPU thread groups split into two roles:
- Producer groups load compressed inputs from HBM into shared memory using dedicated memory-copy hardware (TMA). They stage sign+mantissa bytes, exponent data (or palette indices), and verbatim exponent rows for rare exponents. They run ahead of consumers, filling a circular buffer so data is ready before needed.
- Consumer groups reconstruct BF16 values by combining exponents with sign+mantissa bytes, then immediately feed results into Hopper's WGMMA tensor-core instructions. Reconstructed weights go from assembly to computation without leaving shared memory.
The reconstructive matmul comes in variants differing in output tile handling per compute unit and circular buffer depth. Wider output tiles improve data reuse at large batch sizes; deeper buffers hide memory latency at small batch sizes. The autotuner selects the best variant per workload.
GPU Resource Contention and Scheduling
Unweight’s fused execution pipelines run a preprocess kernel (Huffman decoder or palette transcoder) concurrently with the reconstructive matmul. Both compete for the same GPU resources, however.
On Hopper, each compute unit (SM) has 228 KB of shared memory. The matmul alone needs roughly 227 KB for its buffers and accumulator tiles, while a decode kernel needs about 16 KB for its Huffman lookup tables. Because 227 + 16 exceeds the 228 KB budget, the two kernels cannot co-reside on a single SM—every SM dedicated to decoding is one less for matrix multiplication.
This forces a tunable balance: more SMs for decoding speeds up preprocessing but slows the matmul, and vice versa. The autotuner resolves the tradeoff by measuring real throughput rather than relying on static heuristics.
Overlapping Work Across Transformer Layers
Unweight exploits transformer structure to hide much of the decompression cost, even with the SM partitioning constraint. Layers are classified as "hard" (needing Huffman decoding) or "easy" (using palette data that the matmul consumes directly), and the runtime alternates between them.
While computing an easy layer—which requires no preprocessing—a separate set of CUDA streams decodes the next hard layer’s weights in the background. By the time the easy layers finish, the hard layer’s preprocessed weights are ready. Double-buffered preprocess slots prevent one hard layer’s decode output from being overwritten while still in use.
The down projection benefits most: consumed last in the MLP sequence (after gate, activation, and up), it gets the longest runway to complete its decode.
Autotuning the Configuration Space
With four pipelines, multiple matmul kernel variants, and a tunable SM split, the configuration space is large. Instead of hard-coding a strategy, Unweight autotunes by measuring end-to-end throughput on the target hardware. The sweep optimizes the gate projection while holding up and down fixed, then repeats for up, then down, iterating until no improvement remains. The output is a per-model configuration file specifying pipeline, matmul variant, and SM allocation for each projection at each batch size—all based on measured performance.
One Format, Two Purposes
Encoding format, execution pipeline, and scheduling are independent choices. A single Huffman-compressed bundle serves both scenarios:
- Distribution: Huffman encoding maximizes compression (~22% total size reduction), cutting network transfer times for model shipping.
- Inference: Huffman-encoded projections transcode to the palette format at load time, enabling efficient runtime execution without constraining the distribution format.
Packaging doesn’t force one strategy. The runtime picks the best execution path per projection and batch size on the fly.
Measured Results
On Llama 3.1 8B, Unweight achieves:
- ~13% footprint reduction for inference bundles (gate/up MLP projections only) or ~22% for distribution bundles (all MLP projections including down)—both 100% bit-exact lossless. For Llama 70B, extrapolation suggests roughly 18–28 GB saved.
- 30–40% throughput overhead end-to-end on H100 SXM5, largest at batch size 1 (~41%) and narrowing to ~30% at batch 1024. Three sources—small-batch fixed costs, redundant weight-tile reconstruction, and excluded down projection—are under active optimization.
These are intermediate, single-model numbers. Compression ratios should generalize across SwiGLU architectures since exponent statistics are consistent across scales, but throughput results depend on the current kernel implementations. Attention weights, embeddings, and layer norms are not yet compressed, diluting overall reduction.
Tradeoffs and Rationale
GPUs are costly across dimensions: hardware price, high-bandwidth memory, and power draw. Prior work shows promising ~30% compression on full models, but targets consumer GPUs and research stacks that don’t operate at production scale. Unweight follows a different logic: MLPs hold most model weights and account for significant inference compute. It compresses only MLP weights where benefit is clear, targets datacenter H100s with balanced compute and memory, and adapts with four execution pipelines rather than one approach.
Unweight isn’t free. On-chip reconstruction adds work uncompressed weights wouldn’t require. For Llama 3.1 8B, the inference config saves ~13% of total memory at roughly 30% throughput cost at typical serving batch sizes. Larger batches narrow the gap through better preprocess overlap; further optimization is expected—the down projection, about one-third of compressible weights, and kernel improvements are pending.
For Cloudflare’s network, the payoff is capacity: serving modern models with less GPU memory per instance, enabling cost savings and broader deployment. Distribution benefits are larger, with ~22% smaller bundles cutting transfer times to edge locations.
Ongoing and Future Work
Three directions guide near-term progress:
Down projection compression. Gate and up projections are compressed today; down projection, roughly one-third of compressible weights, needs a different kernel variant due to transposed dimensions. This should push total size reduction beyond 22%.
Kernel optimization. The 30–40% overhead traces to small-batch fixed costs, redundant reconstruction at large batches, and the missing down projection—each with a known mitigation path described in the technical paper.
Model coverage. Current results target Llama 3.1 8B, but exponent statistics are consistent across SwiGLU architectures. Unweight is moving toward larger models served via Workers AI.
Longer term, the architecture may fit Mixture-of-Experts models, where cold experts fetched on demand would benefit directly from reduced storage costs.
The work is open-sourced to contribute to the growing field of compression and GPU efficiency research.
