The cold-start problem in asset selection

Every artwork tile and autoplaying preview on Netflix exists to connect a member with a title, and picking the right one is its own personalization problem. Historically, models answered which asset works well but handled the launch window poorly: assets were treated as opaque IDs, so an asset with no interaction history could only be served through heavy exploration or popularity heuristics that ignore individual taste.

Multimodal embeddings change that. Because a new asset arrives with its embedding, taste signals learned from related assets transfer to it immediately, and personalization can begin close to a title's launch rather than after interactions accumulate. Three production systems are covered below — artwork personalization, query-aware artwork ranking, and video preview personalization — along with a low-cost method for screening candidate embeddings before end-to-end integration and A/B testing.

Artwork personalization: letting the model see the image

Artwork is often a member's first touchpoint with a title, and a diverse set is produced per title to appeal to different tastes. The existing personalized system relies on interaction history, which is exactly what new titles and assets lack.

The fix is to encode each artwork with CLIP, a pretrained image-text embedding model, and fold that into the asset representation alongside the learned ID embedding:

e_id(a) is the asset’s learned ID embedding, and e_a is its CLIP image embedding. The two are concatenated and passed through an MLP layer to give h_a, the representation the model scores against a member.

A brand-new artwork is no longer an unseen ID. The model receives a CLIP embedding the moment the asset is created, so preferences over visual themes, talent, and color palettes apply immediately, expressed in image-embedding space rather than bound to specific asset IDs. That space transfers across titles: engagement with artwork featuring a particular comedian carries over to their next title, prioritizing the asset that puts them front and center.

Knowledge transfer through CLIP embeddings. A member who has interacted with a comedian’s past stand-up artwork (left) leads the model to favor the new-title asset that features that comedian prominently (green check) over one that does not, even though it has never seen that specific image before.

From five per-canvas models to one

Artwork spans multiple canvases with different croppings — billboard, vertical-box, horizontal-panel, short-panel, landscape-panel — and each canvas previously got its own model, since ID-based models cannot relate the cropped and resized renderings of one scene. Signal stayed trapped within a canvas, and each canvas faced its own cold-start.

CLIP embeddings are largely invariant to crop, resize, and aspect ratio, so near-identical renderings land at nearly the same vector:

One source image, many canvases. The same Running Point artwork is cropped and resized across billboard, TV, mobile, and out-of-home placements, each with a different asset ID. Because CLIP embeddings barely change under crop and resize, a single unified model can personalize all of them.

A single unified model can therefore pool signal across every canvas, so affinity learned on a high-traffic canvas informs selection on a sparse one. One model replaces five, with the largest gains where interaction data is thinnest.

Weighting the mixed training data

Pooling raises a question the per-canvas models never had: how to mix data across canvases of very different impression volume, whose logged interactions are not equally valuable to long-term experience. Raw pooled counts would let the highest-volume canvas and most frequent interaction types dominate. Hand-tuned per-canvas weights would only swap one arbitrary choice for hyperparameters requiring endless online sweeps.

Instead, training examples are weighted by reward-based weighting, building on long-term reward modeling, where each example takes the long-term reward score of its interaction type:

a_ti is a training example, a positive interaction on asset i of title t. Its weight is set by the interaction type e observed on it, scored by ρ, that type’s long-term reward.

Here e(·) is the observed positive interaction type and ρ its long-term reward score. Because interaction types are not evenly distributed across canvases, this rebalances the mixture automatically: a canvas contributes in proportion to the long-term value of the interactions it drives, not its impression count.

Offline evaluation with logged propensities

Results must clear an offline metric before a large-scale A/B test. The offline bar is the subtle one. Logs from the current production policy are biased — some assets are shown far more often than others — so logged rewards describe what the policy preferred, not what members would have chosen from the full candidate set. A model that disagrees with the logging policy looks worse than it is.

The mitigation is inverse propensity scoring (IPS) over a dedicated slice of exploration traffic, where a randomized policy samples a title's candidate assets from a known distribution and the propensity of showing a given asset in a given context is logged exactly at serving time:

D is the exploration slice and r(x, a) the observed reward, such as a play. Rare impressions are upweighted, making the estimator unbiased for the reward a candidate policy would have earned in deployment. Known-by-construction propensities, rather than post-hoc modeled ones, are in the authors' experience the main reason offline numbers track online outcomes. IPS is reported as a ratio to the production baseline, and a candidate must win there before receiving A/B traffic.

Ablating the two ideas separately

Image embeddings and model consolidation were ablated independently against the old five-model production system:

  • V1 — the five per-canvas models unchanged, each augmented with image embeddings.
  • V2 — one model over all five canvases with learned ID embeddings alone.
  • V3 — one model over all five canvases with image embeddings in the asset representation.
Relative offline IPS lift by canvas for the three variants, each measured against the prior per-canvas model on that same canvas. Both ideas help where interaction data is scarcest, and V3 is strongest. Hatched bars fall inside the ±1% band, where the change in the offline metric is not significant; V3 values are labeled on the plot.

Each idea helped where expected — the data-starved short-panel and landscape-panel canvases. V3 was the clear winner; a change within ±1% is not significant for this metric, and most standalone V1 and V2 effects fall in that hatched band.

The online A/B test across all device platforms, run for at least four weeks, drew a clearer line: neither idea moved core member metrics on its own. V1 and V2 were flat and non-significant; only V3 produced a statistically significant lift, and it runs in production today.

The two ingredients depend on each other. V1 shows a per-canvas model what an asset looks like, but a sparse canvas has too few examples to teach the model how to use that. V2 supplies abundant data, but only ID-based data, which new assets lack. V3 has both: mature canvases teach the shared model how CLIP embeddings map to preference, and that mapping transfers to sparse canvases. Effects compound rather than add — the V3 short-panel lift of 5.691% exceeds V1 and V2 combined. The takeaway is to look for a second blocking factor before concluding content features do not help.

The new-UI cold-start test

The motivating product change was Netflix's largest TV home-screen redesign in a decade, which would make short-panel the dominant artwork canvas effectively overnight. That is cold-start in its sharpest form: the canvas about to receive the most impressions held the least historical data. Consolidation lets short-panel selection draw on pooled signal from every canvas, and CLIP embeddings let the unified model personalize an asset with few interactions of its own.

V3 shipped ahead of the launch and was measured with a month-long holdback A/B test keeping a small control group on the prior per-canvas model. It absorbed the shift immediately, with statistically significant gains on the core discovery metric and on streaming hours, larger than in the steady-state ablation — expected, given that a sudden shift in canvas dominance is where V3 helps most.

Query intent on the search page

General taste is the wrong signal while searching. A member querying a specific actor wants artwork featuring them, whatever their broader taste says, and the member's intent is stated explicitly in the query.

CLIP's shared text-image space provides this almost for free. Query match is measured directly as the cosine similarity between the CLIP text embedding of the query and the CLIP image embedding of the asset, blended with the personalization score:

The first term is the score the artwork model already produces for a member and asset — "what we think you like." The second compares the query's text embedding against the asset's image embedding — "what you just asked for." The weight α, between 0 and 1, is tuned through online A/B testing.

No extra modeling work was needed: the CLIP embeddings already present in the asset representation carry the text-image alignment, so a query-aware ranker comes from adding one similarity term at scoring time.

Query-aware artwork for a search for a specific actor. Each result surfaces an asset that visually features the searched actor, aligning the artwork with the member’s explicit intent.

MediaFM for video previews

Video previews are harder than stills: their appeal comes as much from motion, pacing, dialogue, and soundtrack as from any frame. Older video preview models treated each preview as an opaque ID. The first content-aware attempt, SeqCLIP, encoded each frame with a CLIP embedding and averaged them into one vector — capturing what a preview looks like but not what it sounds like.

MediaFM, Netflix's first in-house multimodal foundation model, was trained on 80 million shots and fuses three per-shot signals into a single embedding:

  • Visual: SeqCLIP
  • Audio: a pretrained speech and audio embedding model
  • Text: captions encoded via a large-scale text model

Integration required no new infrastructure — the shot embeddings go into the asset representation exactly as CLIP embeddings did for artwork.

Both embeddings were compared against an ID-only baseline, first offline with IPS and then in a five-week online A/B test across all device platforms. Both signals produced the same ordering, MediaFM > SeqCLIP > ID-only, with each step of added content awareness helping and gains largest on TV.

Relative offline IPS lift for the two content-aware video preview embeddings, each measured against the ID-only baseline at the zero rule. Adding visual content awareness helps, and adding audio and timed text on top of it helps further.

MediaFM delivered a statistically significant lift in the core streaming metric over ID-only and outperformed SeqCLIP, confirming that audio and timed-text signals unavailable to a visual-only encoder add real value. It has since shipped as the default video preview embedding across all platforms.

A linear probe as a cheap gate

New embeddings appear constantly, while end-to-end trials cost data engineering, retraining, and weeks of A/B traffic. To avoid running the full pipeline for every candidate, the funnel starts with one question: from the content embedding alone, can you predict which asset wins under a plain, unpersonalized policy?

The method fixes a set of titles, uses exploration data to find each title's debiased popularity winner — the asset with the highest interaction rate after adjusting for how often it was shown, via its propensity score — and labels that asset 1, everything else 0. A linear probe then recovers the label from the asset embedding alone, using no title, cast, or metadata, minimizing binary cross-entropy loss:

Linear and embedding-only is deliberate, isolating how much of an asset's popularity is actually encoded in the embedding. A simple linear classifier should identify likely winners if the semantic drivers are present; otherwise the probe performs no better than random guessing, the baseline it is scored against.

The probe screened and pruned a broad candidate set before any production pipeline changed, narrowing the field to two finalists, SeqCLIP and the leading MediaFM variant. The linear probe accuracies, offline IPS lifts, and online A/B results all ranked MediaFM ahead of SeqCLIP, and the probe now gates every new MediaFM version before release.

Linear probe Δaccuracy, offline IPS lift, and online A/B metric lift for the two finalists. All three agree that MediaFM beats SeqCLIP. The online panel is measured against the ID-based baseline, with its values withheld.

Shared serving: the Netflix Embedding Store

All of these embeddings — CLIP for artwork, SeqCLIP and MediaFM for video previews — live in the Netflix Embedding Store, part of Netflix's AI Platform, which hosts dense embeddings for titles, games, member profiles and multimedia assets. A foundation model encodes raw asset content into a dense vector once, and the store serves it to every downstream system — artwork model, query-aware ranker, video preview model, and others — through one interface, with the identical embeddings at training and online inference time, so there is no skew between what a model learns from and what it sees in production.

Its key property is decoupling foundation-model updates from personalization-model deployments. A new embedding, or a new version of one, can be registered, backfilled across the catalog, and validated independently, without touching the training or serving code of any consumer. Once registered, it is available to every ranking and personalization model through configuration alone, with no downstream code changes or coordinated release. That is what allowed CLIP to be swapped into the artwork model, the query-aware ranker to be built on the same vectors, and MediaFM to roll through the video preview model — each an independent change rather than a cross-team migration.

Foundation-model embeddings (CLIP, SeqCLIP, MediaFM) are stored once and consumed by every downstream system: artwork, query-aware artwork, video previews, and other rankers.

Lessons and next steps

Three points stand out from the work:

  1. Pretrained CLIP embeddings consolidated five artwork models into one while improving data-starved canvases — a benefit that became especially clear during the TV home-screen rollout.
  2. Multimodality wins decisively for video: audio and text signals a purely visual encoder cannot access pushed MediaFM past SeqCLIP.
  3. A cheap proxy task yields large savings, pruning candidates before full end-to-end experiments and online A/B tests.

The next goal is extending the Embedding Store toward a single shared semantic space for image, text, and video. That would enable cross-modal retrieval — matching a video preview to a search query, or a static artwork to the preview derived from it — along with unified asset ranking across surface types.