Two Routes to Scene Boundary Detection
Movies and TV episodes are not monolithic pieces of content; they are assembled from frames, shots, scenes, sequences, and acts. Recognizing where these boundaries fall is essential for downstream work like video summarization, content-based retrieval, dubbing quality assessment, and editing. At Netflix, such processes run hundreds of times daily across multiple teams, creating a strong incentive for algorithmic tooling. Low-level segmentation, such as shot boundary detection, can be handled with pixel-based approaches. Scene boundary detection—identifying transitions between continuous sequences of shots sharing a time, location, and theme—demands a more nuanced understanding of narrative and emotional arcs, often better inferred from screenplay or audio tracks than from video alone.
Engineers at Netflix developed two complementary methods for scene boundary detection. The first is a weakly supervised approach that aligns screenplay text with timed text (captions and audio descriptions) to timestamp screenplay scene headers, or sluglines. The second is a supervised sequential model—a bidirectional GRU—that consumes pretrained shot-level embeddings and outperforms current state-of-the-art baselines on internal benchmarks.
Timestamping Scripts with Dynamic Time Warping
Screenplays follow a strict format: every scene starts with a scene header that specifies location and time of day. That consistency allows reliable parsing into structured data. However, screenplays are not always faithful to the final product. Changes made on set by directors or actors, as well as significant post-production editing, rarely make it back into the written script.
To use this noisy source, the team aligned timed text from the finished video (closed captions, audio descriptions) with the screenplay's dialogue and action lines—descriptive lines capturing non-dialogue aspects of a scene. Two challenges arise. First, on-the-fly changes yield line pairs that are semantically similar but not identical. That is addressed by embedding lines from both sources with pretrained sentence-level models optimized for paraphrase identification. Second, post-shoot rewrites can reorder, remove, or insert whole scenes. Dynamic time warping (DTW) handles this by measuring similarity between sequences that vary in speed or timing. DTW assumes monotonic alignment—if shot a aligns to shot b and later shot c aligns to shot d, then d must follow b in the other sequence. This condition is frequently violated, but the method proves robust enough to recover from local misalignments, and the vast majority of salient events, including scene boundaries, align well.
Once DTW completes, scene headers carry timestamps indicating potential scene boundaries in the video. The alignment outputs can also augment audiovisual machine learning models with screenplay-derived information, such as scene-level embeddings, or transfer labels from video content to train screenplay prediction models.
Sequence Modeling on Shot Embeddings
The screenplay alignment approach requires a high-quality script and provides a fast path to a working scene change detector. It can also serve as a feature for the team's complementary approach: training a sequence model on annotated scene change data, captured by existing Netflix workflows or drawn from public datasets.
The architecture is straightforward: a bidirectional GRU (biGRU) ingests shot representations and predicts whether a given shot ends a scene. The discriminative power comes from the rich, pretrained multimodal embeddings—a design choice driven by the scarcity of labeled scene change data and the relative abundance of unlabeled material for pretraining.
Video embeddings come from an in-house model pretrained on video clips aligned with text. Audio embeddings start with source separation to isolate foreground speech from background music, sound effects, and noise. Each separated waveform is embedded separately with wav2vec2 and the results are concatenated. Fusion strategies were explored in two forms. In early fusion, audio and video embeddings are concatenated and fed into a single biGRU. In late fusion, each modality is encoded by its own biGRU, and the hidden states are concatenated just before the output layer.
Press enter or click to view image in full size
The evaluation produced several notable findings:
- The model matches and sometimes exceeds the state-of-the-art benchmark, which was tested on video modality alone against the same evaluation data. F-1 scores were measured on the positive label, with a relaxed "off-by-n" variant that counts predictions within n shots of the true boundary—a fairer metric given the human-in-the-loop deployment setting.
- Audio features improve results by 10–15%, consistent with prior work. The primary driver of performance variation is the fusion strategy.
- Late fusion consistently outperforms early fusion by 3–7%. The likely reason is that temporal dependencies between shots are modality-specific and are better captured when each modality has its own encoder.
Next Steps: Fusing Signals for General-Purpose Models
The two approaches—screenplay alignment and multimodal sequence modeling—are complementary. Logical future work includes combining them so screenplay features feed the unified model. A further extension is generalizing outputs across multiple shot-level inference tasks, such as shot type classification and memorable moments identification, with the hypothesis that such multi-task training produces better general-purpose video understanding models for long-form content. As these models tackle more complex narrative structures, scene boundary detection is expected to be the first of several projects integrating narrative understanding into Netflix's multimodal machine learning stack.
