What actually drives codec comparison results

Published codec comparisons frequently appear to contradict one another: one report credits codec A with 15% savings over codec B, another finds the relationship reversed. The discrepancy rarely comes from the codecs themselves. Testing methodology, encoder maturity, tuning choices and source content all shift the outcome far more than the underlying standard does. For adaptive streaming in particular, the framework used to measure one codec against another can matter as much as the codec.

Encoder implementation and settings

The same standard can behave very differently depending on which implementation is being measured. Reference encoders — the software written during standardization, primarily for research — generally lag production encoders, which face stricter speed and resource constraints but benefit from real-world tuning. For new standards with immature implementations, the gap can be substantial.

Encoding settings add another layer of variability. Parameters such as the number of coding passes, rate-control strategy and visual tuning are largely application-driven. Standardization bodies typically disable rate control and perceptual tuning so that individual coding tools can be compared in isolation. A streaming service, by contrast, is free to favor any tool that improves visual quality, even at the cost of encoding speed or footprint.

Methodology: staying locked to one resolution changes the answer

Traditional codec testing follows the “common test conditions” approach used in standardization: a small set of short test sequences, each encoded at its input resolution with a fixed set of quality parameters. That arrangement has proven suitable for developing standards, but it codifies a constraint that does not exist in adaptive streaming. There, the same video can be encoded at multiple resolutions and bitrates, and content resolution need not equal encoding or display resolution.

When per-resolution rate-quality curves are plotted together, they cross. For any given bitrate there is one resolution that delivers the best quality, and the optimum changes across the rate range. Selecting the best curve at each rate yields a “convex hull” of encoding points, and BD-rate savings are then computed on that hull rather than on single-resolution curves. The flexibility this offers leads to quality improvements over fixed-resolution encoding.

This logic generalizes beyond a single scene. The dynamic optimizer (DO) methodology takes the convex hull across all the shots in a multi-shot video and jointly optimizes compression across qualities, resolutions and shots. DO is codec agnostic, can use any quality metric, and eliminates the need for high-level rate control within the encoder — lower-level adaptive quantization inside a frame still helps, since DO operates at the shot level and above. Because it performs an exhaustive search, it is computationally expensive and unsuitable for real-time use, but it represents an upper bound on what high-level rate-control algorithms can achieve.

Testing content and metric choice

Content selection can skew results for or against any codec. A fair comparison uses balanced material that was not part of the codec’s development corpus. Netflix has released multi-shot sequences such as El Fuente and Chimera specifically to help researchers avoid the problem of conflating training data with test content.

Metric choice changes conclusions as well. PSNR has long been the default since it is simple and correlates reasonably with subjective opinion, but perceptual metrics like VMAF track human scores better across a wider quality range. Two additional choices matter when using any metric:

  • Scaled metrics: metrics may be computed at the encoding resolution or after scaling the encode up to the display resolution. The latter approach better reflects the viewing experience.
  • Temporal averaging: the arithmetic mean is the conventional way to average per-frame scores. The harmonic mean — which yields HVMAF when applied to VMAF — weighs outliers more heavily, reflecting the reality that a few badly distorted frames in a shot spoil the experience regardless of how good the rest of it is.

Measuring three standards two ways

To show how much the comparison framework matters, both approaches were applied to three widely used standards: H.264/AVC and H.265/HEVC (ITU-T and ISO/MPEG) and VP9 (Google). Each was tested with its reference encoder and with a production encoder.

The traditional comparison encoded 14 sequences from the MPEG common test conditions and 14 from the Alliance for Open Media set — all 1080p, mostly single-shot clips of 1 to 10 seconds — at five fixed quality settings per sequence at original resolution, and computed BD-rate savings using luma PSNR. Neither content set appears to have favored its “own” codec: HEVC encoders gained no advantage from the MPEG sequences, nor VP9 encoders from the AOM set.

The adaptive streaming comparison changed several factors at once. Perceptual tuning was enabled, encoding was done at ten resolutions per shot from 1920x1080 down to 256x144, and DO drove the final optimization using HVMAF computed at the display resolution of 1080p. A third test set was added with eight full Netflix titles — about eight hours of 1080p, 30 fps, 8-bit material spanning varied content.

Results were split into two operating ranges: low HVMAF (30–63) and high HVMAF (63–96), a range strongly associated with high subjective quality. The high-range numbers are the meaningful figures for an adaptive streaming service delivering content to paying subscribers. The positioning between the newer standards changes noticeably depending on which framework you use — reflecting the codecs at least as much as it reflects the recipes used to cook up the comparison.

What the Results Mean

The single most important lesson from any codec shootout is that the outcome is inseparable from the methodology. Encoder selection, encoding settings, testing content, and the metrics used all shape the final verdict. A narrow choice of testing conditions can flip the ranking entirely.

Our own measurements illustrate the point clearly. Under the traditional full-ladder PSNR comparison, EVE-VP9 lands about 1% behind x265. Shift to the HVMAF high-range scenario, however, and EVE-VP9 pulls roughly 12% ahead. Neither number is wrong; each simply answers a different question about the same encoders.

Given the massive volume of video compressed and delivered daily by streaming services, and the fact that traditional offline comparison methods and adaptive streaming workflows often diverge in their conclusions, the video coding community would benefit from routinely adopting an adaptive streaming perspective. The good news is that this does not require a heavy lift. Computing metrics on the convex hull of the rate-distortion curve or simply adding HVMAF results alongside PSNR and SSIM would go a long way toward making codec evaluations more representative of real-world delivery.

Practical Recommendations

For those planning their own codec comparisons, the following practices will make results more robust and reproducible:

  1. Describe the full encoding pipeline. Encoder version, parameter sets, and tuning options must be disclosed, as they can alter relative performance by more than the differences you are trying to measure.
  2. Use content that reflects the target distribution. A handful of clips biased toward high or low complexity will skew the aggregate numbers.
  3. Report multiple metric perspectives. PSNR, SSIM, and VMAF each rank encoders differently; HVMAF adds an adaptive-streaming-relevant view that should not be omitted.
  4. Evaluate on the convex hull. For adaptive streaming use cases, per-ladder comparisons miss the efficiency of dynamic optimization; the hull reflects what a rate controller can actually achieve.

Further Reading and Credits

Detailed technical information and full result tables are available in the paper “Video codec comparison using the dynamic optimizer framework” by Ioannis Katsavounidis and Liwei Guo. The experimental work described here was run on the Archer platform built by the Netflix media infrastructure team.