VMAF Gets Faster, Friendlier, and More Codec-Aware

Since Netflix open-sourced VMAF, the video quality metric has become a common yardstick for encoding work far beyond its original streaming use case. It now shows up in live sports pipelines, video chat systems, gaming captures, 360-degree video, and user-generated content platforms. The metric’s design—fusing human visual system modeling with machine learning trained on subjective scores—gives it an edge over traditional metrics like PSNR or SSIM when predicting quality across resolutions, shots, and content genres.

As adoption has widened, Netflix has kept working on the tool’s practical usability. Recent changes include a more permissive license, a rewritten C API, significant speed gains through fixed-point math and vectorization, and a new mode designed to separate the perceptual effects of compression from those of image enhancement.

Fixed-Point Conversion and Vectorization Deliver Another 2x

VMAF’s performance has already benefited from several rounds of low-level optimization, including frame-level multithreading and frame skipping that allow real-time 4K evaluation. The latest gains come from a collaboration with Intel and Facebook, carried out in two stages. First, working with Ittiam, VMAF’s data pipeline was converted from floating-point to fixed-point representation. Then Intel applied vectorization on top of that fixed-point pipeline using AVX2 and AVX-512 intrinsics.

The result is another average 2x speedup while maintaining numerical accuracy to the first decimal digit of the final VMAF score. These improvements apply across 4K, full HD, and SD resolutions, and further work is planned.

libvmaf v2.0.0 Introduces a Modernized API

To make VMAF easier to drop into other projects, Netflix has announced libvmaf v2.0.0, which ships with a redesigned C API. Previously, libvmaf offered a minimal interface primarily useful for FFmpeg integration. The new API is designed with extensibility in mind:

  • New features can be added without breaking the existing API.
  • Adding a new feature extractor is straightforward, supporting the future evolution of VMAF algorithms.
  • Memory allocation and incremental frame-level VMAF computation are now more flexible.

That last capability is significant: it allows VMAF to be integrated directly into an encoding loop, guiding encoding decisions iteratively on a frame-by-frame basis.

A New “No Enhancement Gain” Mode

One thing that distinguishes VMAF from PSNR and SSIM is its ability to reflect visual gains from image enhancement operations like sharpening or histogram equalization. These improvements show up as higher VMAF scores. This property has proven useful in perceptual encoding optimization—for example, the tune=vmaf mode in libaom achieves BD-rate gains by sharpening frames before AV1 compression.

But this sensitivity creates a problem for codec evaluation. When an encoder includes pre-processing steps as part of its pipeline, it becomes hard to tell whether a VMAF improvement comes from compression or from enhancement. Since separating an encoder from its pre-processing is often impractical—especially with proprietary encoders—measuring pure compression gain with VMAF gets murky.

Community feedback reflects this tension: some users appreciate that VMAF captures enhancement gains, while others worry those gains could be overused or gamed. Netflix’s response is a new VMAF NEG mode (“neg” for “no enhancement gain”). The mode detects the magnitude of VMAF gain attributable to image enhancement and subtracts it from the score, effectively muting the enhancement’s contribution.

The guidance from Netflix:

  • Use NEG mode for codec evaluation to isolate the pure effect of compression.
  • Use default mode when assessing compression and enhancement combined.

What’s Next for VMAF

VMAF’s NEG mode is an initial step toward quantifying perceptual quality without enhancement. In regular mode, VMAF can overpredict quality when enhancement operations such as oversharpening actually degrade the viewing experience. Future versions will address this by imposing limits on the attainable enhancement.

Netflix has also identified other areas for algorithm improvement, including better prediction under challenging conditions like banding and blockiness in shaded regions, more accurate temporal masking effects in high-motion sequences, and improved handling of video encoded from noisy sources. The work continues to draw on human visual system modeling, subjective testing, and machine learning.