Prioritizing Video Encoding At Scale
Facebook processes hundreds of millions of video uploads daily. Delivering each at optimal quality — maximum resolution with minimal buffering — requires careful selection of codecs and encoding parameters. Advanced codecs like VP9 offer superior compression compared to older ones like H264 but demand significantly more compute. Encoding every upload with the most advanced codec would be computationally prohibitive, so Facebook must prioritize which videos get those expensive encodes.
To address this, Facebook combines a benefit-cost model with machine learning to predict which videos will be highly watched. These high-demand videos are then encoded with advanced codecs and more compute-intensive “recipes” first, reducing buffering and improving visual quality for the majority of viewing. The system is designed to be fair to creators of all sizes — popularity cannot be determined simply by friend or follower counts.
The Limitations Of Hard-Coded Rules
Previously, Facebook’s video encoding system assigned priorities based on a list of hand-crafted rules considering factors like whether a video was licensed music, a product video, or from an account with many followers. This approach had significant drawbacks. As new codecs emerged, the rules expanded and required constant manual tuning. Moreover, the coarse-grained nature of rule-based priorities made it impossible to fully optimize the end-user experience given the varying compute, quality, and compression trade-offs of different codecs.
The skewed nature of video consumption further complicated simple prioritization. While a large publisher like Disney may typically generate more watch time than a small vlogger, any video can go viral. A static rule set cannot capture this dynamic landscape, potentially leaving popular content from smaller creators without the advanced encodings it deserves.
A Benefit-Cost Approach
Facebook’s new model retains initial H264 ABR encodings for immediate delivery but calculates the priority of subsequent encoding jobs differently. The approach rests on a few key observations: encoding is a one-time compute cost, roughly one-third of videos generate most watch time, and data center energy is limited. Therefore, the best way to maximize overall experience within power constraints is to apply expensive codecs to the most-watched videos — but only once the lower-resolution “lanes” have been produced so the video is deliverable.
To implement this, the system defines three concepts:
- Benefit: the encoding family’s compression efficiency relative to H264 fast preset, times the effective predicted watch time.
- Cost: the normalized compute cost to produce missing encodings needed to make a family deliverable.
- Priority: Benefit divided by Cost.
Compression efficiency is measured using Facebook’s Minutes of Video at High Quality per GB (MVHQ) metric. For example, a video that achieves 153 minutes of high-quality playback per GB with H264 fast, 170 minutes with H264 slow, and 200 minutes with VP9 would assign relative benefit values of 1.0, 1.1, and 1.3, respectively. Because MVHQ can only be calculated post-encoding, historical data is used to estimate values for new videos.
Effective predicted watch time adjusts raw ML predictions for codec support. For instance, roughly 20 percent of video consumption occurs on devices that cannot decode VP9. Thus a video predicted to have 100 hours of watch time gets an effective VP9 score of 80 hours, while the H264 family scores a full 100 hours.
Compute cost sums the CPU usage to produce all missing resolutions in a family. For example, if a video is missing all four VP9 resolutions, the cost is the sum of all four, applied equally to each job. This structure prevents the system from excessively prioritizing advanced codecs without first ensuring basic playability.
Predicting Watch Time with Machine Learning
Determining which videos benefit most from advanced encoding relies on an ML model that predicts next-hour watch time. The model incorporates numerous features: the uploader’s follower count, average watch time of their prior videos, video duration, dimensions, privacy settings, post type (Live, Stories, Watch, VOD), and content age. It also considers past watch time trajectory, which emerged as the single strongest predictor of future traffic.
Several challenges shaped the model’s design:
- High variance and a long tail: Next-hour watch time can range from zero to over 50,000 hours, requiring the model to quantify scale, not just binary popularity.
- The trajectory problem is temporal: New videos lack historical watch time data. The model must handle missing features gracefully since early identification of future hits maximizes the opportunity for high-quality encoding.
- Popular videos dominate training sets, potentially making patterns from hit content less applicable across the broader catalog.
- Traffic patterns differ: Stories have shorter lifespans, live streams attract concentrated viewing, and VOD content can gain popularity long after upload.
- Offline ML metrics don’t ensure product wins: Reductions in RMSE or Huber Loss do not directly translate to better user experience or compute utilization, so offline model optimization alone isn’t sufficient.
Choosing between RMSE and MAPE during model selection
During development, candidate models were evaluated using both Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE), because each metric has a different sensitivity: RMSE is more affected by outliers, while MAPE is more sensitive to errors on small values. Given the high variance in watch time labels, the team uses MAPE to judge predictions for popular and moderately popular videos, and RMSE for less-watched content. Evaluation also includes per-category breakdowns to ensure the model generalizes across video types, ages, and popularity levels.
Since summary metrics don't always translate directly into product experience, the team also converts predictions into classification problems when two models have similar RMSE and MAPE. This reveals important trade-offs. For instance, if a video accrues 1,000 minutes of watch time but Model A predicts 10 minutes, its MAPE is 99 percent; Model B predicting 1,990 minutes also yields a MAPE of 99 percent — yet Model B's result is far more likely to lead to high-quality encoding.
To capture the balance between over- and under-applying advanced encoding, the model's classifications are checked at multiple thresholds (e.g., 10 seconds). The team counts videos where the actual watch time and prediction are both below the threshold, and the inverse, to derive false positive and false negative rates. This shows whether the model leans toward scheduling too many advanced encoding jobs or missing opportunities, across different popularity levels.
What the two-model design changes in practice
To compensate for videos without prior watch time trajectories, two separate models are in use. The view-time model relies on the three feature sets described earlier. The upload-time model handles new uploads, borrowing performance signals from other videos by the same content creator as a substitute for a watch time history; once a video accumulates enough real trajectory data, it is switched over to the view-time model.
Operational gains beyond new uploads
The new model also identifies older videos on Facebook that could benefit from more advanced encodings, shifting computing resources toward them. That has moved a significant portion of watch time onto advanced encodings, reducing buffering without extra compute cost. Improved compression similarly helps viewers with limited data plans, including users in emerging markets, watch more at higher quality.
Introducing new encoding recipes no longer requires manually assigning priorities. Depending on a recipe's cost and benefit, the model automatically sets a priority to maximize overall throughput benefit — so a very compute-intensive recipe meant only for extremely popular videos can be targeted correctly without curation. This simplifies ongoing investment in newer codecs and improves the video experience for viewers.
