Why compression needs an intermediate representation
At Dropbox’s scale, even a 1% improvement in lossless compression efficiency has a measurable impact on storage costs. The current default, zlib, saves roughly 8% of disk space across the file types Dropbox stores. To push beyond that, the company has open-sourced DivANS, a new framework that restructures compression pipelines into modular stages. The goal is to let researchers improve individual components without having to redesign an entire compressor.
DivANS borrows a concept from language compilers: the intermediate representation (IR). Just as compilers decouple source languages and target architectures, DivANS separates compression into distinct layers. A raw file is first converted into a text-based IR, that IR can be optimized independently, and finally the IR is serialized into a compressed bitstream. Decompression reverses only the last two steps. This split enables parallel work on front-ends (file-specific algorithms), optimizers, and back-ends (entropy coders), a structure the authors argue helped advance compiler research decades ago.
Three instructions and helpful hints
The DivANS IR is built from Lempel-Ziv (LZ) commands, and uses only three instruction types:
insert: emit fresh literal bytes, entropy-coded (e.g., Huffman), when no prior data matchescopy: reference previously seen bytes elsewhere in the filedict: pull bytes from a predefined dictionary, with optional small alterations
Any losslessly compressed input can be represented as a sequence of these commands. But to actually minimize file size, the coder must predict upcoming bytes from context. DivANS supports optional "hints" to carry this predictive information:
prediction: the probability tables themselves, each corresponding to a situation (such as "a consonant was just seen, so a vowel may follow")btype{l,c,d}: pointers that switch between probability tables as context changes
These hints are entirely optional—an interpreter that ignores them can still reconstruct the original data—but they allow optimizers to tune probabilities against the cost of describing them. If done correctly, the balance between table size and prediction accuracy yields better compression; if done poorly, direct inserts can be more efficient.
For the test string “It snowed, rained, and hailed the same morning,” only the initial “It,” the first “ed,” and the “l” in “hailed” require fresh insertions. Everything else is copied from earlier in the sentence or pulled from the Brotli dictionary with slight modifications, as shown in the IR and interactive demo.
Try DivANS yourself
An interactive demonstration is available at https://dropbox.github.io/divans, where any file up to 2MB can be processed. Files larger than one megabyte tend to see the greatest benefit.
Serializing with ANS and adaptive probabilities
Once the IR is finalized, it must be written out efficiently. DivANS relies on Asymmetric Numeral Systems (ANS), introduced by Jarek Duda, which achieves arithmetic-coding-level compression at considerably higher speed. These entropy coders assign shorter bit sequences to more probable symbols. In practice, actual compression ratios depend more on the accuracy of probability estimates than on the coder itself.
DivANS feeds its ANS coder with dynamically updated probabilities. The decoder works in 4-bit nibble chunks, and each nibble is treated as a symbol. A table of cumulative distribution functions (CDFs)—the "prior table"—is indexed by bits from recently observed bytes. Each CDF entry corresponds to a situation, such as a recent period and space predicting a capital letter, or recent copy commands referencing nearby offsets. The same logic applies to the IR itself, enabling predictions about command types in addition to byte values.
Probabilities start uniform and are updated after each nibble is read. As similar situations recur during encoding, the CDFs converge toward accurate predictions without any hardcoded linguistic assumptions. Alternative rule sets, including those learned by deep models, could replace the default lookup logic. Encoding runs in reverse while decoding runs forward, which keeps both directions fast and ensures the encoding and decoding probability tables stay in sync.
Measured gains and design targets
In preliminary experiments, DivANS yielded a 2% compression ratio improvement over the best available methods for many file types, while also meeting three design requirements:
- Security, determinism, and reproducibility
- Streaming decode speeds above 100 Mbit/s
- Better compression ratios than other methods meeting those same two criteria
By making the IR explicit and standardized, DivANS opens the field to incremental contributions. A front-end developer can emit the same IR from any algorithm, optimizer researchers can experiment independently with IR transformations, and back-ends can be swapped without touching the rest of the pipeline. The source code is available under an open-source license on GitHub, as part of Dropbox’s ongoing investment in compression engineering.
Testing DivANS Against Real Workloads
To evaluate DivANS, Dropbox ran benchmarks against the Silesia compression corpus and a sample of 130,000 random chunks uploaded to Dropbox. The Silesia corpus is a classic benchmark set, but its mix of medical images, HTML files, and books doesn't reflect the characteristics of cloud storage data. The sampled uploads provide a more realistic picture. Tests ran on 2.6 GHz Intel Xeon E5 2650v2 servers in Dropbox datacenters, comparing DivANS and Brotli at quality levels 9 and 11 against zlib, 7zip, bz2, and Zstd at maximum settings. Since Dropbox already uses Lepton for image compression, the benchmark excluded files where Lepton or zlib achieved less than 1% compression, focusing on the one-third of uploads where zlib is effective.
Compression Gains on Dropbox Data
On the Dropbox dataset, DivANS at q11 delivers a 12% space saving over zlib and more than 2.5% over the other algorithms at their maximum settings. When the benchmark includes all non-JPEG files—two-thirds of uploads—DivANS still comes out ahead.
DivANS generally can't extract much additional compression from files that are already compressed, but it does achieve meaningful gains on some of them. Notably, 88% of DivANS's overall savings comes from the one-third of files where zlib is most effective.
At these settings, DivANS and Brotli encode at similar speeds, but DivANS decodes about five times slower. The gap exists because Brotli decompresses bytes with just two table lookups and minor bookkeeping, while DivANS maintains probability models for each nibble and uses vector instructions to update CDFs and ANS state. Dropbox expects a few straightforward bookkeeping optimizations to push decode speeds above 200 Mbps.
Results on Silesia and a Hybrid Proof of Concept
On the Silesia corpus, DivANS q11 achieves 2.7% better compression than Brotli and Zstd but trails 7zip. The gap is explained by LZMA's intermediate representation (IR) fitting the mozilla binary file particularly well—that file occupies 20% of the archive. Dropbox demonstrated the IR's flexibility by adding three print statements to the LZMA codec to emit the standard DivANS IR, then wrote a script to interleave Brotli hints with LZMA IR. This mashup improved DivANS's compression on Silesia beyond all other algorithms tested, suggesting that automating such hybrid approaches could boost both speed and ratio.
What's Next for DivANS
The current DivANS system already achieves better compression ratios than alternatives on Dropbox data, albeit at a performance cost. The more significant goal is opening the code and the DivANS Intermediate Representation to the broader community. Lowering the barrier to creating new compression algorithms lets researchers focus on three independent directions:
- Producing efficient IR representations of raw files faster than DivANS
- Optimizing IR representations themselves
- Encoding IRs into efficient bitstreams
Why Rust
DivANS is written in Rust, which provides safety, security, and determinism guarantees in its safe subset. Rust matches the speed of hand-tuned C, requires no garbage collector, and embeds well in any language with a C foreign function interface—including runtime allocator selection through that FFI. These properties make it straightforward to run the DivANS codec in a browser via WASM.
Rust's recent SIMD intrinsic support lets DivANS manipulate CDFs with vector instructions, and its safe multithreading model—"fearless concurrency"—enabled decoding IR commands and literals on separate threads. The multithreading work began after the decompressor was fully written and relied on rustc guidance to refactor the code into independent halves.
An Open Invitation
The ideal outcome is a compression ecosystem where multiple disparate algorithms generate DivANS IR—not just Brotli, and not just in Rust—and where other codecs consume DivANS IR to produce efficient bitstreams. Pull requests are welcome.
