How low-bit inference enables efficient AI

The Cost of Running Modern Models

At Dropbox, nearly all in-house models are attention-based architectures for tasks like understanding text, images, videos, and audio—core capabilities behind Dropbox Dash's ability to search, summarize, and reason over large collections of user content. As models grow in size and complexity, efficiently serving them in production is a central challenge for delivering responsive user experiences. In attention-based models, most compute comes from repeated matrix multiplications in two main parts of the model:

Linear layers compute embeddings throughout the model, including layers inside attention blocks that determine how different parts of the input relate to one another, MLP layers that process and refine representations, and the model's final output stage where representations become concrete results, such as predictions or responses.

The attention mechanism itself evaluates relationships across the input to determine relevant information, a step that significantly increases compute cost with longer context sizes.

On GPUs, these matrix multiplications are handled by specialized hardware. NVIDIA GPUs use Tensor Cores and AMD GPUs use Matrix Cores. These dedicated processors are accessed through matrix multiply-accumulate (MMA) instructions and are designed specifically to accelerate matrix operations—delivering substantial performance gains compared to general-purpose CUDA Cores.

These cores have a notable scaling behavior: as numerical precision is reduced, they can perform more matrix operations per second, typically yielding higher FLOPS. In practice, halving the precision often allows the cores to roughly double throughput.

Quantization and Precision Trade-offs

Lowering numerical precision is accomplished through quantization, which reduces the number of bits used to represent numerical values. Quantizing tensors from 16-bit to 8-bit or 4-bit reduces memory footprint since each element needs fewer bits. This is done by rescaling data to fit within a smaller representable range. For instance, 8-bit quantization maps values to 256 bins, restricting each tensor element to one of these discrete levels while approximating original floating-point values. Quantization to below 8 bits usually requires an additional process called bitpacking, where multiple low-bit elements are combined into native data types such as uint8 or int32, because 4-bit formats are not natively supported.

Lowering precision improves speed, memory usage, and energy efficiency, since lower-bit data needs less power for both memory transfer and computation. For example, with FP4 support, Blackwell offers significant energy savings compared to the H100.

Lower bits have also been explored—binary and ternary weights restrict values to two or three discrete levels, promising greater theoretical energy efficiency. However, such schemes don't suit modern GPUs well because they can't fully leverage Tensor/Matrix Cores. While experimental efforts with custom hardware or specialized accelerators exist, this approach hasn't seen broad industry adoption due to limited ecosystem support and concerns about model quality. Real-world gains depend heavily on how well formats are supported by existing hardware and software ecosystems.

Quantization in practice: Two broad families

Quantization encompasses a range of techniques that differ in how values are represented, scaled, and executed. These choices have direct consequences for model accuracy, inference speed, and how well a workload can exploit modern GPU hardware. At Dropbox, this matters because the AI stack spans multimedia understanding and other diverse workloads running on multiple generations of hardware. Some of these workloads are latency-sensitive, favoring fast per-request execution, while others are throughput-oriented and optimized for processing large batches. The format chosen determines whether a workload is bound by software overhead, memory bandwidth, or specialized units such as Tensor Cores.

A useful dividing line is the introduction of the MXFP microscaling standard, which brought native hardware support for low-bit data types. Quantization methods for large language models can be grouped into pre-MXFP formats, which rely on explicit dequantization and software-managed scaling, and MXFP formats, which push these operations into Tensor Core hardware. The practical differences between these approaches are significant for real-world inference.

Pre-MXFP formats: Explicit scaling, software overhead

Before MXFP, sub-byte quantization relied primarily on integer data types. Common configurations include A16W4 (16-bit activations, 4-bit weights) for weight-only quantization, and A8W8 (8-bit activations, 8-bit weights) when activations are quantized as well. Maintaining model quality with A16W4 typically requires calibration or advanced algorithms like AWQ or HQQ. Lower-bit formats such as A16W3, A16W2, and BitNet demand increasingly sophisticated training-time quantization methods.

When activations and weights have different data types, the standard approach is to explicitly dequantize the lower-bit tensor to the higher precision before the matrix multiplication (MMA). This can help in memory-bound scenarios where reducing data movement is the priority. In compute-bound workloads, however, the extra dequantization arithmetic can offset those gains and even slow down execution.

This trade-off is especially clear in weight-only quantization: it cuts data transfer but does not accelerate the matrix multiplication itself. The right choice between activation quantization (A8W8) and weight-only quantization (A16W4) depends on the workload. Weight-only quantization tends to win in local deployments with small batch sizes and reasoning-heavy tasks, where memory bandwidth is the limiting factor. Activation quantization is often more effective for large-context prefills and high-throughput serving, where compute becomes the bottleneck.

Popular methods such as AWQ and HQQ rely on linear quantization with grouping. Symmetric linear quantization expresses dequantization as a simple scaling operation. A more flexible variant, asymmetric linear quantization, adds an offset, allowing dequantization to be implemented as a fused multiply-add that maps well to modern GPU hardware.

Grouping assigns shared scale parameters to small contiguous blocks of tensor elements, typically of size 32, 64, or 128. Despite its simplicity, this approach substantially reduces quantization error at low bit widths and has become a core component of practical low-bit schemes.

On the activation side, two 8-bit approaches dominate. Channel-wise quantization is straightforward and efficient, well suited for on-the-fly inference, since rescaling can be applied directly after matrix multiplication. Per-block quantization, popularized by JetFire and DeepSeek V3, divides tensors into small tiles with independent scales per block, limiting the impact of outliers and reducing error. It is particularly effective in quantization-aware training, while still delivering practical Tensor Core speedups.

Beyond linear approaches, non-linear methods such as QuiP# and GPTVQ push precision lower but face practical hurdles. Linear 4-bit quantization already delivers strong accuracy and can often be applied on the fly with techniques like HQQ, avoiding expensive offline passes. Non-linear formats require custom fused kernels and deep integration into inference frameworks. Even then, low-bit weights must be converted to a format compatible with Tensor Cores, making linear quantization simpler and more practical on current architectures.

Quantization also extends to the attention module. Methods such as Flash Attention 3 and Sage Attention use 8-bit quantization to accelerate attention-related matrix multiplications, improving throughput and memory efficiency with minimal accuracy loss.

MXFP formats: Native hardware quantization

The MXFP microscaling format changes how quantized models run by providing native hardware support for quantization. Tensor Cores can operate directly on quantized activations, weights, and their scaling factors in a single fused operation. Pre-MXFP formats required explicit dequantization before or after MMA operations, adding overhead and limiting performance.

MXFP quantizes both activations and weights using micro-scaling, similar in spirit to AWQ and HQQ but implemented in hardware. It uses symmetric quantization with a fixed block size of 32 and shared scaling factors stored in the E8M0 format. Some hardware also supports mixed-precision MMA operations, such as MXFP8 × MXFP4, allowing activations to use MXFP8, MXFP6, or MXFP4 while weights stay in MXFP4. A breakdown of MX types is shown below (source: Open Compute Project, OCP Microscaling Formats (MX) Specification, Version 1.0, Table 1).

The E8M0 format represents positive powers of two in the range [2⁻¹²⁷, 2¹²⁷]. Scales are typically quantized as scale = weight.amax(axis=1, keepdim=True) / max_val, limiting values to at or below 1. Extremely small magnitudes are rarely needed; values as small as 2⁻¹⁵ often suffice for near-zero weights. This suggests scales could theoretically use fewer bits than E8M0, although that would add complexity.

While E8M0 is hardware-friendly and flexible, restricting scale values to powers of two causes a noticeable accuracy drop with MXFP4. Simple post-training adjustments can largely mitigate this loss, as demonstrated in our blog post.

To address remaining numerical limits, NVIDIA introduced NVFP4 as an alternative to MXFP4. NVFP4 uses a smaller group size of 16 rather than 32 and E4M3 FP8 scaling factors for higher precision. Since FP8 has a relatively large minimum representable value, a global per-tensor floating-point multiplier normalizes the scaling range and improves numerical stability.

Although MXFP4 and NVFP4 are standardized, implementation depends on the GPU architecture. Different compute capabilities use different Tensor Core instructions: sm_100 architectures use the tcgen05.mma instruction, while sm_120 architectures use mma.sync, both with the block_scale modifier. Kernels compiled for sm_100 are therefore not portable to sm_120. While the mainstream AI software stack focuses on server-grade GPUs like the B200 and B300, portability is improving. Notably, Triton has added MXFP support for sm_120 devices, enabling broader compatibility for low-bit Triton kernels.

Where low-bit inference goes next

The quantization techniques covered here are already delivering real efficiency gains across the industry, and they map directly onto choices we make at Dropbox when optimizing model deployment. Products like Dash depend on large-scale models for conversational AI, multimodal search, document understanding, and speech processing — workloads that have to meet strict latency, reliability, and cost targets in production. Quantization is one of the primary levers we pull to hit those targets and make the most of modern accelerator hardware.

Adoption still lags in a few important areas. MXFP and NVFP formats are not yet standard in real-world deployments, and FP4 quantization has incomplete support across popular frameworks and model stacks. Many open-source runtimes do not fully support these formats on all GPU architectures, and FP4 models remain scarce.

These gaps will become more significant as hardware pushes toward lower-bit compute. Practical low-bit inference for production systems will depend on tighter software design, more mature framework support, and new quantization methods that preserve model quality as precision drops. This is an active area of exploration for us — work that will directly shape how we deliver fast, reliable, and cost-efficient AI experiences to Dropbox users.