Automated reframing for 9:16, 4:5 and 1:1
Netflix needs vertical variants of its content for two growing channels: a social following of 1.2B and Clips, the vertical feed recently rolled out in the Netflix mobile app. Manual reframing in editing tools cannot keep pace with catalog growth, so the team built an automated multi-ratio cropping system inside its internal production tools. It generates high-quality vertical and other aspect-ratio variants with minimal manual effort.
The system treats resizing as a tracking and temporal optimization problem. Casting the crop as an intelligent “virtual camera,” it follows key action and honors each scene's composition rather than falling back on a center crop. All three formats are produced in a single pass without extra computational overhead.
Shots, sub-shots and the focus region
The first stage segments footage into atomic shots with a neural-network-based shot boundary detector. Since many shots contain several characters conversing across the frame, an Active Speaker Detection step further splits a continuous shot into logical sub-shots, letting the virtual camera cut and re-focus as the conversation moves. The detector relies on an internal face detector and LipFlap, and works on both live-action and animated titles.
With sub-shots defined, the pipeline locates the most important region in every frame. Face detections are combined with visual saliency maps to filter and prioritize the faces in focus; where several characters compete, faces are ranked by signals such as active speaker status and relative size. Shot-level content understanding aggregates these cues into a continuous focus region for the camera.
Smoothing the crop over time
Tracking that region frame by frame would yield shaky footage, so the crop's motion is governed by a set of heuristics over the course of a shot. Small displacement locks a static crop window covering the relevant area; significant motion triggers a dynamic tracking window whose path is smoothed so panning reads as intentional and cinematic.
Editor controls and XML handoff
The algorithm ships inside the internal Clips tool, where editors and producers generate multiple aspect-ratio versions of a clip with smart reframing. A unified in-app module provides shot-reframing controls for reviewing the crops and making targeted corrections.
For complex shots or custom campaigns, producers export the resizing parameters as XML and load them into external timeline editors such as Adobe Premiere. On a curated evaluation set in internal testing, the algorithm outperformed Premiere's Auto Reframe; the export lets agencies and external partners begin from the algorithm's framing decisions instead of recreating crops manually. Automated resizing and manual refinement in one place cut production time and effort while preserving creative control.
The project is part of a broader push to scale content creation without compromising creative intent. Pairing algorithm-driven reframing with editor-friendly tools has reduced time spent on repetitive resizing and enabled faster experimentation across mobile feeds and marketing campaigns.
