Scaling Media Machine Learning at Netflix - Netflix TechBlog

Building the Foundations of Media ML at Netflix

Netflix began offering streaming in 2007 alongside its DVD service, and as both the catalog and member base grew, so did the complexity of recommendations. With thousands of shows and millions of accounts, surfacing the right content depends on more than just matching titles — it relies on visuals like trailers and artwork that give members a reason to click. Machine learning has already been used to personalize artwork and help creatives produce promotional media efficiently.

The media-focused ML infrastructure at Netflix addresses a core goal: reducing the time from idea to product for media ML practitioners. This infrastructure provides clear paths for accessing and processing media data, training large-scale models efficiently, productizing models to run on both existing and new assets, and storing model outputs for use in content creation workflows.

Building this infrastructure required tackling a set of unique challenges and creating components designed specifically for media workloads.

Media Access with Jasper

Early on, researchers found it difficult to access media data. Even after gaining access, they encountered issues with inconsistency across assets — differences in decoding performance, file size, metadata, and formatting could stall progress. To address this, the team at Netflix standardized media assets through pre-processing steps that generate quality-controlled derivatives with snapshotted metadata. A unified library now gives ML practitioners seamless access to video, audio, image, and text assets.

Feature Storage with Amber Feature Store

Media feature computation is expensive and time-consuming. Many ML practitioners independently computed the same features against the same assets, wasting resources. The Amber Feature Store was built to memoize features and embeddings tied to media entities. It includes a data replication system that supports copying data to different storage solutions tuned for varied access patterns.

Orchestration with Amber Compute

Productized models need to score newly arriving assets. In the past, ML practitioners had to build bespoke triggering and orchestration components for each pipeline — a maintenance burden that became a source of downstream errors. The Amber suite of infrastructure components offers triggering capabilities that initiate algorithm computation with recursive dependency resolution, removing the need for custom orchestration glue.

Training Performance

Training media models creates demands on storage, network, and GPUs. Netflix runs a large-scale GPU training cluster on Ray that supports multi-GPU and multi-node distributed training. Datasets are precomputed, preprocessing is offloaded to CPU instances, and model operators are optimized within the framework. A high-performance file system resolves data-loading bottlenecks, increasing overall training throughput by 3–5 times.

Serving and Searching

Media feature values can be optionally synchronized to other systems based on query needs, such as Marken — a scalable annotation service that persists feature values as versioned, strongly typed constructs associated with media entities like videos and artwork. Marken offers a user-friendly query DSL for filtered searches, unique query capabilities on temporal and spatial data by time frames or region coordinates, and vector searches that scale across the full catalog.

Although most ML practitioners interact with this infrastructure through Python, the underlying systems span many technologies, including Conductor, Dagobah, Metaflow, Titus, Iceberg, Trino, Cassandra, Elastic Search, Spark, Ray, MezzFS, S3, Baggins, FSx, as well as Java/Scala applications built on Spring Boot.

Case Study: Scaling Match Cutting

Match Cutting is a video editing technique where a transition between two shots uses similar visual framing, composition, or action to connect scenes. It is a powerful storytelling tool. A previous post described how machine learning identifies candidate pairs; the focus here is the engineering needed to deliver it at scale.

The initial match cutting implementation handled matches within a single title — either a movie or an episode of a show. The average title has around 2,000 shots, requiring about 2 million pair comparisons. This work was originally bundled in a single Metaflow flow, with each step mapped to a Metaflow step to control resource allocation individually.

Step 1: Shot detection and clip creation

The pipeline downloads a video file and generates shot boundary metadata. Here is an example of that data structure:

SB = {0: [0, 20], 1: [20, 30], 2: [30, 85], …}

In the SB dictionary, each key is a shot index, and its value is the frame range for that shot. For shot index 1, the range is [20, 30] — start frame 20 is inclusive, and end frame 29 is the last frame (the range endpoint is exclusive). With this information, the next step is to materialize individual clip files (such as clip0.mp4 or clip1.mp4) for each shot for later processing.

Step 2: Embedding extraction and deduplication

This step operates on the individual clips produced in Step 1, along with the shot boundary list. Each clip is passed through a video encoder to produce a fixed-size embedding, which is then used to identify and remove duplicate shots. Take this deduplicated version as an example:

# the second shot (index 1) was removed and so was clip1.mp4
SB_deduped = {0: [0, 20], 2: [30, 85], …}

In this case, SB_deduped is the outcome of removing duplicates from the original SB. The deduplicated shot boundaries and the surviving files are then sent along to the next step.

Step 3 and beyond

From Prototype to Production

The five-step pattern we described works for a single match-cutting flavor focused on one title. As soon as we tried to generalize it — matching across titles or adding new flavors — the architecture started to crack in predictable places.

Standardization gaps

The representations extracted in steps 2 and 3 depend on the input video's encoding characteristics. In some cases — instance segmentation being a notable example — the output representation is a function of the input file's dimensions. Without a standardized encode (consistent recipes and resolution), matching quality deteriorated whenever titles with different source files had to be processed together.

Redundant computation

Shot-level segmentation and near-duplicate shot removal are common prerequisites across many media ML pipelines. We were repeating these expensive steps per pipeline instead of computing once and reusing. Memoizing them would reduce waste and, just as importantly, guarantee coherence: if algorithmic pipeline A and pipeline B both depend on the same shot boundary detection step, then shot index i in both pipelines refers to the same frame range. Without that shared dependency, verifying equivalence requires extra work downstream.

Orchestration mismatch

Editors at Netflix need to start working on a title as soon as its video files land. That drove us to trigger match-cutting computation on file arrival, but the triggering logic had its own problems. Metadata changes sometimes re-triggered computation for an unchanged video file. Meanwhile, different pipelines had built bespoke triggering components, each slightly different, leading to inconsistencies.

Existing workflow orchestrators like Conductor and Meson didn't naturally fit this domain. Media ML needs a tighter coupling between asset metadata, media access, feature storage, feature computation and triggering — with well-defined interfaces so new algorithms can be plugged in.

That's the gap Amber fills. It's a media ML feature development and productization suite that treats each algorithmic component as an Amber Feature, with its own computation scope, storage, and triggering semantics. Dependencies between features are explicit, which allows composing a complex graph of interrelated algorithms.

Match cutting across titles is expensive

Step 4 is quadratic in the number of shots. Matching across a 10-episode series with roughly 2K shots per episode means 200M comparisons. Scaling that to 1,000 files — across multiple shows — balloons to roughly 200 trillion pairwise computations.

Editors may also want to consider arbitrary subsets of the catalog. A naive pre-compute-everything strategy fails immediately: for just 1,000 files, there are 2¹⁰⁰⁰ possible subsets — more than the number of atoms in the observable universe. We needed an approach that sidesteps both restraints.

The Production Pipeline

The Media Machine Learning Infrastructure team had already built most of the building blocks we needed. We assembled them into a new pipeline.

Standardized video encodes. The entire Netflix catalog is pre-processed and stored for ML reuse. Match cutting relies on homogeneity across videos, and this standardization provides it. (

Canonical shot segmentation. Rather than running shot detection ourselves, we depend on the infrastructure team's canonical shot boundary feature. Because it's memoized, we reuse its outputs — saving compute and ensuring shot segments are consistent across all algorithms that share the dependency.

Embedding orchestration via Amber. We use Amber's feature dependency semantics to tie embedding computation to shot deduplication. Amber's triggering starts scoring as soon as standardized encodes are available and recursively walks the dependency chain to compute everything that's needed.

Feature value storage. Embeddings live in Amber, which provides immutability, versioning, audit trails and metric tracking. That's not just bookkeeping — it lets future algorithms build on top of both the match-cutting output and its intermediate representations.

Replication to Marken. For serving, we use Amber's synchronization mechanisms to replicate feature values to Marken.

Media Search Platform. High-scoring shot pairs are served to editors inside internal apps through Marken, backed by the Media Search Platform. The diagram above captures the end-to-end flow.

What's Next

We're extending this infrastructure beyond promotional media workflows like match cutting. Active directions include ML-assisted VFX tooling, richer content-understanding models for recommendations, and combining personalization signals with content understanding to improve creative tooling. Future posts will detail the solutions for each component mentioned here.