Weight-only quantization without the calibration tax
Deploying large language models on a single GPU usually means quantizing them first. The two dominant strategies are calibration-free methods like bitsandbytes, which work purely from weights, and calibration-based methods like GPTQ and AWQ, which fit quantization parameters to an external dataset. The latter typically produce better results but carry two costs: sensitivity to the choice of calibration data and a slow, compute-heavy fitting process that scales poorly with model size.
Half-Quadratic Quantization (HQQ), developed by the Mobius team, aims to close that gap. It needs no calibration data and runs orders of magnitude faster than GPTQ, while producing quality competitive with calibration-based approaches. On Llama-2-70B, HQQ completes quantization in under five minutes—more than 50x faster than GPTQ—and the resulting 2-bit model beats full-precision Llama-2-13B in perplexity at comparable memory usage.
Outlier-aware optimization without gradients
Plain quantization degrades accuracy because model weights span a wide range, and extreme values—outliers—are disproportionately distorted by rounding. GPTQ and AWQ mitigate this by minimizing error on layer activations, using calibration data to guide the optimization.
HQQ instead minimizes error on the weights themselves. To handle outliers properly, it uses a sparsity-promoting loss such as the lp-norm with p < 1. This loss better reflects the heavy-tailed distribution of outlier errors than a squared-error term does. The objective is non-convex, so HQQ applies a Half-Quadratic solver, introducing an auxiliary variable We to split the problem into efficiently solvable sub-problems. The scale parameter s is fixed, and optimization focuses on the zero-point z.
The two sub-problems have closed-form solutions. Sub-problem 1 is a proximal operator; with the lp-norm it resolves to the generalized soft-thresholding operator. Sub-problem 2 reduces to averaging over the quantization group axis. Since the solver uses closed forms rather than gradients, HQQ runs entirely in inference mode with half-precision arithmetic. The solver converges in about 20 iterations, whereas gradient-based approaches like AdamW need thousands of steps and cannot handle p < 1 at all. The result: quantizing Llama-2-7B is over 100x faster than with autograd-based methods.
Speed: from hours to minutes
HQQ performs quantization on the GPU in half-precision, transferring data only once the solver has finished. This yields dramatic speedups. The practical consequence: models like Llama-2-70B, which take GPTQ hours to process, can be quantized in minutes on commodity hardware.
Benchmark results
Llama-2
Quality is measured via perplexity (PPL) on the wikitext2 dataset, alongside runtime GPU memory in GB. HQQ is compared against bitsandbytes (BNB), GPTQ via AutoGPTQ, and AWQ via AutoAWQ.
Solver parameters were fixed at p=0.7, beta=1, kappa=1.01, with 20 iterations and early stopping. Unless otherwise noted, weights are quantized into groups of size 128 (indicated by _g128), and the zero-point is quantized to 8-bit without additional grouping or optimization.
| Method | nBits | Llama-2-7B | Llama-2-13B | Llama-2-70B | |||
|---|---|---|---|---|---|---|---|
| PPL ↓ | MEM ↓ | PPL ↓ | MEM ↓ | PPL ↓ | MEM ↓ | ||
| FP | 16 | 5.18 | 13.5 | 4.63 | 25.6 | OOM | OOM |
| BNB | 8 | 5.22 | 7.9 | 4.67 | 14.4 | 3.17 | 68.15 |
| GPTQ_g128 | 8 | 5.19 | 7.8 | 4.63 | 14.8 | 3.12 | 74.87 |
| HQQ_g128 | 8 | 5.19 | 7.6 | 4.63 | 14 | 3.12 | 69.32 |
| BNB_g64 | 4 | 5.43 | 4.7 | 4.79 | 8.2 | 3.29 | 39.11 |
| GPTQ_g128 | 4 | 5.41 | 5 | 4.74 | 8.9 | 3.24 | 40 |
| GPTQ_g64 | 4 | 5.38 | 5 | 4.73 | 9.1 | 3.23 | 41.13 |
| AWQ_g128 | 4 | 5.32 | 4.6 | 4.71 | 8.2 | 3.21 | 35.78 |
| AWQ_g64 | 4 | 5.28 | 4.6 | 4.7 | 8.5 | 3.2 | 37.08 |
| HQQ_g128 | 4 | 5.35 | 4.6 | 4.74 | 7.9 | 3.21 | 35.97 |
| HQQ_g64 | 4 | 5.3 | 4.6 | 4.7 | 8.2 | 3.19 | 37.52 |
| GPTQ_g128 | 3 | 6.3 | 3.9 | 5.25 | 7 | 3.85 | 33.7 |
| GPTQ_g64 | 3 | 6.1 | 4 | 5.16 | 7.3 | 3.7 | 33.47 |
| HQQ_g128 | 3 | 6.2 | 3.8 | 5.15 | 6.8 | 3.58 | 30.11 |
| HQQ_g64 | 3 | 5.82 | 4.5 | 4.98 | 7.4 | 3.45 | 33.46 |
| GPTQ_g64 | 2 | nan | 3.5 | 13 | 6 | 9.44 | 24.5 |
| HQQ_g32 | 2 | 15.61 | 3.5 | 7.63 | 5.9 | 4.82 | 24.2 |
| HQQ_g16 | 2 | 7.3 | 4.1 | 6.36 | 6.9 | 4.12 | 30.27 |
| HQQ_g16_s* | 2 | 7.31 | 3.7 | 6.37 | 6.1 | 4.13 | 26.37 |
*: Scale is also quantized to 8-bit with a group size of 128.
Without using any calibration data, HQQ holds its own against calibration-based methods across model sizes. The most striking result is at the 2-bit extreme: the quantized Llama-2-70B achieves a lower perplexity than the full-precision Llama-2-13B, while requiring roughly comparable memory.
Vision Transformers
HQQ extends cleanly to vision models. The evaluation uses OpenCLIP ViT models trained on LAION, with top-1 and top-5 accuracy reported on ImageNet. Since GPTQ and AWQ calibration require text inputs, HQQ is compared only against bitsandbytes (BNB), applied by quantizing all linear layers in the transformer blocks.
Two benchmarks are reported. Zero-shot classification uses OpenAI prompt templates averaged over text features; it directly measures quantized model quality since no training is involved. Linear probing instead freezes the quantized backbone and trains a linear Softmax classifier on top, evaluating the model's utility as a feature extractor.
| Method | nBits | Model | Linear (top-1) | Linear (top-5) | 0-shot (top-1) | 0-shot (top-5) |
|---|---|---|---|---|---|---|
| FP | 16 | ViT-B-32 | 0.764 | 0.941 | 0.664 | 0.896 |
| FP | 16 | ViT-L-14 | 0.82 | 0.964 | 0.731 | 0.93 |
| FP | 16 | ViT-H-14 | 0.841 | 0.973 | 0.772 | 0.949 |
| BNB | 8 | ViT-B-32 | 0.762 | 0.94 | 0.663 | 0.896 |
| HQQ | 8 | ViT-B-32 | 0.763 | 0.941 | 0.663 | 0.896 |
| BNB | 8 | ViT-L-14 | 0.82 | 0.964 | 0.731 | 0.93 |
| HQQ | 8 | ViT-L-14 | 0.82 | 0.964 | 0.731 | 0.93 |
| BNB | 8 | ViT-H-14 | 0.84 | 0.972 | 0.771 | 0.949 |
| HQQ | 8 | ViT-H-14 | 0.841 | 0.973 | 0.772 | 0.95 |
| BNB | 4 | ViT-B-32 | 0.733 | 0.925 | 0.608 | 0.859 |
| HQQ | 4 | ViT-B-32 | 0.75 | 0.933 | 0.639 | 0.881 |
| BNB | 4 | ViT-L-14 | 0.815 | 0.961 | 0.718 | 0.925 |
| HQQ | 4 | ViT-L-14 | 0.815 | 0.962 | 0.721 | 0.926 |
| BNB | 4 | ViT-H-14 | 0.837 | 0.971 | 0.766 | 0.947 |
| HQQ | 4 | ViT-H-14 | 0.839 | 0.973 | 0.769 | 0.948 |
| HQQ | 3 | ViT-B-32 | 0.664 | 0.881 | 0.481 | 0.753 |
| HQQ | 3 | ViT-L-14 | 0.799 | 0.954 | 0.689 | 0.909 |
| HQQ | 3 | ViT-H-14 | 0.831 | 0.969 | 0.755 | 0.943 |
| HQQ | 2 | ViT-B-32 | 0.318 | 0.551 | 0.04 | 0.106 |
| HQQ | 2 | ViT-L-14 | 0.731 | 0.917 | 0.559 | 0.815 |
| HQQ | 2 | ViT-H-14 | 0.808 | 0.96 | 0.716 | 0.924 |
HQQ outperforms 4-bit BNB by wide margins on zero-shot accuracy: +3.1% top-1 with ViT-B-32. At aggressive low-bit settings, the gap over full-precision models also grows with size—the quantized ViT-H-14 at 3-bit beats full-precision ViT-L-14 by +2.4% top-1 zero-shot accuracy, and the 2-bit ViT-H-14 surpasses the full-precision ViT-B-32 by +5.2%.
The upshot: data-free quantization need not be a quality compromise. HQQ's speed enables rapid testing across models and bit-widths, and its outlier-aware objective keeps accuracy within striking distance of—or, in some extremes, better than—methods that require calibration.
A Calibration-Free Route to Low-Bit Quantization
Weight quantization typically costs you either accuracy or time: data-dependent methods such as GPTQ and AWQ deliver strong results but require calibration datasets, while simpler calibration-free schemes degrade more quickly at low bit-widths. HQQ (Half-Quadratic Quantization) sits in a different quadrant — no calibration data, yet accuracy that stays competitive with the data-dependent approaches across model scales.
Those quantization runs stay practical at the high end. Llama-2-70B, among the largest openly available models, can be quantized in a few minutes using HQQ's core optimization routine. That speed comes from iterative Half-Quadratic splitting, which avoids slow full-model passes over the weight matrix that dominate other quantization cost models. The method's practicality is underscored by its availability for public use: reproduction code is provided, and ready-to-use quantized models can be downloaded for immediate testing and integration.
More concretely, HQQ's reported strengths are:
- Competitive quality down to extreme low bits — across bit-widths, model sizes, and downstream applications, HQQ output tracks methods that tune against calibration data.
- No data dependence — quantization starts from the weights alone, removing dataset selection and preprocessing from your pipeline.
- Fast execution — even flagship-scale models finish in a handful of minutes thanks to the Half-Quadratic optimization strategy at the core of the method.
HQQ's actual code support reflects these design decisions. The full implementation enabling reproduction of reported results is distributed on GitHub, with optimized model checkpoints made available separately for direct deployment.
