MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

A Foundation Model for Watching Netflix’s Catalog

Netflix’s recommendations and production workflows depend on a granular, machine-readable understanding of every title in its catalog. That understanding must extend beyond visual frames to audio cues and dialogue, especially as the service expands into live events and podcasts. Rather than training separate models for each narrow media task, Netflix has built a single multimodal encoder — the Netflix Media Foundational Model (MediaFM) — pretrained on its own entertainment-specific dataset of shots.

MediaFM accepts audio, video, and timed text as input and produces contextual shot-level embeddings. The pretraining data consists of tens of millions of individual shots across multiple titles, which gives the model a scale of long-form narrative context that general-purpose video datasets lack. The result is a reusable backbone for tasks like ad relevancy, clip popularity prediction, and clip tagging, as well as cold-start support for newly launched titles in recommendations and tools for analyzing internal content.

From Raw Shots to Fused Embeddings

The input unit for MediaFM is the shot, extracted by a shot boundary detection algorithm that segments a movie or episode. For every shot, the pipeline produces three separate embeddings:

• Video: frames sampled at uniform intervals are embedded with SeqCLIP, an internal CLIP-style model fine-tuned on video retrieval datasets
• Audio: the shot’s audio samples are embedded with Meta FAIR’s wav2vec2
• Timed Text: corresponding subtitles, closed captions, or audio descriptions are encoded with OpenAI’s text-embedding-3-large

These three embeddings are concatenated and unit-normed into a single 2304-dimensional fused vector per shot. Training examples are temporally ordered sequences of these fused embeddings drawn from the same title, up to 512 shots per sequence. Title-level metadata, such as synopses and tags, is also passed through text-embedding-3-large to create a global context embedding.

Transformer Encoder with Global Context

The architecture is a Transformer encoder similar to BERT. A sequence of fused shot embeddings flows through four stages:

1. Input Projection: a linear layer reduces each fused embedding to the model’s hidden dimension
2. Sequence Construction: a learnable [CLS] token is prepended, followed by the projected title-level embedding inserted as a [GLOBAL] token, so every shot attends to the title’s global context
3. Contextualization: positional embeddings are added and the sequence passes through the Transformer stack, producing shot representations conditioned on their surrounding narrative
4. Output Projection: a final linear layer maps hidden states back to the 2304-dimensional fused embedding space

Training relies on a self-supervised Masked Shot Modeling (MSM) objective. In each sequence, 20% of shot embeddings are randomly replaced with a learnable [MASK] embedding. The model’s task is to predict the original fused embedding for each masked position, and it is optimized by minimizing the cosine distance between predictions and ground-truth embeddings. This forces the encoder to learn temporal and semantic relationships among shots, using both local cues and the broader narrative arc of the title.

MediaFM: A Multimodal Foundation Model for Media Understanding

The key to the biggest wins from MediaFM came from the researchers' novel application of Muon to the model's hidden parameters, with AdamW reserved for the remaining parameters. The team noted that swapping in Muon produced noticeable improvements to the overall training process, though the architecture itself—combining multiple modalities of input and then contextualizing those representations over longer sequences—is where the largest gains in downstream performance emerged.

Evaluating a Media Foundation Model

To measure the quality of what MediaFM learns, the team trained lightweight, task-specific linear classifiers on top of the frozen representations—a standard practice known as "linear probing" but applied atop a base model that is itself frozen to truly isolate the strengths of the learned embeddings.

Most clip-level tasks involve short segments ranging from a few seconds to a minute, typically what a member might see while being recommended a title. A crucial insight from testing is that the way you embed that clip matters: extracting a shot's features "in context"—pooling them from within a larger surrounding sequence, like an entire episode—performed significantly better than treating a clip as a standalone, isolated set of shots.

These foundational embeddings have value across a surprisingly broad set of applications at the streaming service. A representative set of clip-level evaluations includes:

• Ad Relevancy: Multi-label classification for ad placement, scored with average precision. Here, the embeddings drive the retrieval stage, generating the candidate set of relevant ads early in the funnel.
• Clip Popularity Ranking: A ranking task predicting whether a specific clip will generate higher CTR than other clips from the same show or movie, scored using Kendall's tau.
• Clip Tone: Multi-label classification of clips into 100 emotional tones (like creepy, scary, or humorous), scored via micro average precision.
• Clip Genre: Multi-label classification of clips into eleven core genres (including Action, Anime, Comedy, and Documentary) that honors the underlying mode of the feature, scored via macro average precision.
• Clip Retrieval: A binary classifier identifying human-vetted "clip-worthy" moments from a 1:3 positive-to-negative ratio of clips, scored with average precision.

The clip retrieval task proved an excellent indicator of the system's potential: a natural progression of roughly 15% improvement appeared at each layer of added sophistication. This holds true particularly for tasks that require deep narrative comprehension.

Benchmarking Against Commercial Baselines

To establish a fair comparison, researchers benchmarked MediaFM against commercial encoders, in addition to the open-source SeqCLIP baseline that also provides the video embedding for MediaFM itself. The competitive field included Google's VertexAI multimodal embeddings and TwelveLabs' Marengo 2.7 embeddings.

MediaFM outperformed every baseline on all five tasks measured. Interestingly, the magnitude of those wins scaled with the task's need for nuanced interpretation of the story or message — proving the platform's embeddings aren't just good at surface-level pattern matching. The largest margins of victory showed up where the context of the narrative mattered most for the label at hand.

What Actually Drives the Gains

MediaFM represents an architectural improvement over prior TV-scale work in two key ways: modality fusion (combining raw image, audio, and text inputs into a single shot-level representation) and the higher-level contextualization of those shots relative to the episodes they come from.

In head-to-head tests against a stripped variant that uses only the fused multi-modal raw input without the additional contextualization step, the source of improvement varies by job. For a task like classifying hook clips by their tone, fusing raw audio and video did offer some utility, but the bigger leap in accuracy came primarily from the contextualization itself.

Ironically, the sequence of upcoming shots may not be just optional—it's actually essential when it comes to popularity ranking. Having a full, raw representation of a clip from all three modalities at once weakens the retrieval model on preference ranking, yet that same representation is rescued and notably improved the minute it passes through the contextualization step, using the broader directory of program data. This stresses that reasoning over time and plot shifts information is more worthy of strengthening than the simple core inputs.

Charting a Path to Multimodal LLMs

The project represents an effective strategy for self-supervised fusion of shot-level detail across an entire content catalog: capturing relationships between voiceover-to-scene, tones, and dialogue moments no annotation scheme can cover fully. The research community is already looking ahead to even stronger foundational starting points for the next iteration of MediaFM.

In particular, pretrained multimodal architectures that unify audio, vision, and text modalities with built-in ability to walk across all three are well positioned to inject domain context from the world directly. The team specifically cites forward research leveraging Qwen3-Omni as a known candidate foundation to set the benchmark for future model generations.

The remaining series will detail how that knowledge is being decoded at a totally distinct level: embedding the underlying the titles and synopses themselves with custom context-aware adaptations—covering work done well past what the raw clip feature harvest has produced so far.