Instagram’s path to 1,000+ production models

Instagram’s recommendation systems now span far beyond Feed, Stories, and Reels. Comment ranking, notification importance, and even tagging suggestions are all driven by ML. Each of these surfaces contains multiple layers of the ranking funnel—sourcing (retrieval), early-stage ranking (ESR), and late-stage ranking (LSR)—with progressively fewer candidates as operations become more expensive.

Figure 1: The ranking funnel.

The constant experimentation across surfaces and layers creates a serious infrastructure challenge. ML engineers need room to tweak weights and test changes, which results in a large and growing number of models serving live traffic.

Figure 2: An expression of the factors behind the fleet’s numerical growth.

Recognizing the infrastructure gap

As the model count climbed past 1,000, Instagram’s infrastructure team identified three categories of risk that were holding back both productivity and reliability.

  • Discovery: Even a team focused on one app could not keep up with growth. Product ML teams maintained their own, often incomplete, records of production models.
  • Release: There was no consistent mechanism for safely launching new models. The slow, ad-hoc process hurt ML velocity and product innovation.
  • Health: No standard definition of prediction quality existed. With such a diversity of surfaces and subtle degradation patterns, quality problems could go unnoticed.

Three solutions addressed these gaps: an authoritative model registry, automated launch tooling, and a new health metric called model stability. Together, they formed a foundation for scaling from a handful of models to over 1,000 without losing control.

The model registry

Before the registry existed, investigating a model issue was a slow, error-prone process. On-call engineers had to manually ask model owners about the model’s context: what it did in the ranking stack, which surface it supported, and how critical it was to the business. This context determined the operational response. For an experimental model serving a small percentage of traffic, the right move might be to shut it down and reroute to the baseline. But for a core model, “just turn it off” was rarely possible, and the on-call engineer had to page the model owner—defeating the purpose of dedicated on-call coverage.

Figure 3: A fictional but typical non-productive investigation.

The model registry solves this by serving as a system of record built on top of Meta’s distributed configuration suite, Configerator. The registry is a schematized ledger that provides structured, read-and-write access to operational metadata for every production model. It standardizes the collection of business function and importance, ensuring operational resources are focused on the most critical models.

Figure 4: An abridged example of what a model registry entry looks like.

As Instagram scaled its AI investment in content recommendations, the number of models and AI assets grew rapidly. Maintaining a minimum standard for all models became harder without an authoritative source of business context and importance. The registry provides a structured interface for capturing that context via two key concepts: model types and criticality.

Model types

A model type describes the purpose of an ML workload—its place in the ranking funnel. For example, ig_stories_tray_mtml breaks down as:

  • ig: An Instagram model, as opposed to fb or whatsapp.
  • stories: Serves IG Stories.
  • tray: Serves the main Stories tray, not another surface.
  • mtml: A multi-task-multi-label model, commonly used in late-stage ranking.

These type strings tag AI assets across training flows, model checkpoints, and inference services. They serve as proxies for business context, enabling asset management, policy enforcement, and analytics. The registry’s metadata entries anchor on two main types: ModelMetadata for instances and ModelTypeMetadata for categories. Core attributes are universally applicable, while extended attributes allow teams to encode operational preferences—for instance, Instagram encodes baseline and holdout model IDs to orchestrate ranking funnel execution.

Criticality tiers

For importance, Instagram adopted the same criticality system used across Meta’s services. The Global Service Index (GSI) records a criticality from TIER0 to TIER4, aligned with the maximum incident severity a service can cause (SEV0 being most critical, SEV4 a “heads up”). Because this system had established social proof at the company, infra engineers were already familiar with it. Models are now annotated with criticality at both the model type and model levels.

This replaced a situation where each team could raise its own service to TIER1, overburdening supporting teams. Now, to qualify for elevated monitoring, teams must provide immediate 24/7 on-call response and demonstrate meaningful contribution to critical business metrics.

Automation built on configuration

Once a critical mass of Instagram models was onboarded, the registry integrated fully with Meta’s monitoring and observability suite through Configerator. This enabled fully automated model performance monitoring and alerts, integrated with the existing SLICK tooling for SLIs. Dashboards now allow monitoring models across many time series dimensions, with model-specific alerting driven directly by registry entries. These integrations gave every team confidence that monitoring coverage was complete and automated, without manual setup per model.

Faster, safer model launches

The registry also paved the way for rethinking the model release process. The previous flow was slow and inconsistent, with no standard gate for safety. Instagram developed an automated flow for launching new models that includes estimation, approval, prep, scale-up, and finalization. What once took days now takes hours, allowing ML engineers to iterate and ship more quickly without sacrificing operational safety.

Model stability as a health signal

Finally, Instagram defined a pioneering metric called model stability to measure the accuracy of model predictions. This consistent definition of health—which had previously been lacking—now feeds SLOs for all models registered in the system. With SLOs in place, the entire product surface’s ML health can be understood at a glance, even as the model count grows well past a thousand.

Removing launch friction

Once we had visibility into the models running in production, the next bottleneck was the process of getting new models out the door. The time-to-launch for new models was a major drag on ML velocity, and the old manual workflow was neither reliable nor efficient.

The cost of manual capacity planning

In the past, adding a new service to production was a heavy lift. The standard approach was to clone the entire service, stand up a shadow deployment to test with cloned traffic, and then run repeated overload tests to find a peak throughput. This method was inexact; real-world traffic varies throughout the day, making it difficult to determine a consistent load limit. Debugging unexpected performance issues often stretched this process to as long as two days. Once a capacity number was finally established, we had to run it through a set formula to estimate the final cost.

Figure 5: A formula calculating capacity estimations for a new launch.

The actual traffic shift was equally delicate. Even after determining that a service required, say, 500 replicas, we rarely had that many spare instances sitting idle. Launching meant gingerly scaling up the new service by roughly 20%, shifting 20% of the live traffic over, and then scaling down the old service by a matching 20% to reclaim the capacity. This incremental, manual shuffle had to be repeated until the migration was complete, and each team also had to remember to set up their own alerts for the new baseline in time to avoid false alarms from their old models.

This arduous process created a competition for finite compute resources. Launch prioritization was effectively first-come, first-served, with no mechanism for reserving capacity that would be freed up in the future. To fix this fairness problem, we switched from individual engineer limits to allocating bandwidth quotas by team. This made sense because not all engineers on a team experiment at the same time—one might be training a model while another is in the testing phase. With clear quotas, teams could define their own launch policies and run launches in parallel without needing org-level prioritization.

Automating the launch lifecycle

We knew that relying on autoscaling alone wasn't the answer. Small configuration differences between services or architectural changes could silently alter performance, and we didn't have infinite capacity as a buffer. If we ran out of supply mid-shift, reverting was costly and slow.

The solution was to invest in pre-planning. We partnered with the AI Infra team to build two major pieces of tooling:

  • Offline performance evaluation: We used a benchmarker that read pre-recorded traffic from a data warehouse to replay different load levels against a clone of the experimental service. By setting thresholds for latency and error rates, the tool automatically converged on a stable maximum load figure.
  • An automated launch platform: This tool took the capacity numbers from our tests, collected the live demand data, and ran the cost formula automatically. It then executed the upscaling and downscaling cycles as traffic shifted, removing the manual guesswork.
Figure 6: Load tests converging on an accurate measure of load.

Finally, we tied the launch directly to the model registry, allowing teams to land model changes in code. This not only made the fleet of 1000+ models easier to maintain and understand, but also strengthened trust in the registry as the single source of truth tied to the model’s actual lifecycle.

Figure 7: A theoretical model registry change during launch.

This suite of launch automation substantially reduced the class of SEVs related to model launches. It also improved our pace of innovation significantly—we went from a few launches to over 10 per week—and slashed the time engineers spent on a single launch by more than two days.

Defining model health beyond uptime

Once models were launching smoothly, we faced a new challenge: ensuring their output remained relevant. While ranking models behave like typical distributed services at the request/response level, a universal SLO based on request success rates is insufficient. The accuracy of the recommendation matters as much as the uptime of the system. A user who loves golf but dislikes cooking content might still receive a "successful" response recommending a cooking video—but that response is wrong for that user. We needed a dedicated "model stability" metric to capture this distinction.

Figure 8: Different types of responses that can be provided to an end user.

Moving beyond binary success

Ranking models are not idempotent; they generate prediction scores for a set of candidates to determine which content a user is most likely to engage with. If these scores don't accurately reflect user interests, engagement drops. To measure that accuracy, we define stability across a model’s multiple output predictions. For a multi-task-multi-label (MTML) model, we evaluate each underlying prediction against two core metrics:

  • Calibration: The ratio of predicted click-through-rate (CTR) to empirical CTR. This tells us if we are over- or under-predicting user action. A perfect predictor sits at 1.
  • Normalized entropy (NE): A measure of how well the model separates action from inaction, calculated by comparing average log-loss per impression against a baseline of always predicting the empirical CTR. Lower is better, with 1 being equivalent to random guessing.
Figure 9: A simplified MTML model predicting three actions.

To claim a model is stable, every underlying prediction must be stable. Observing these metrics in real time allows us to define healthy ranges and set thresholds for breaches. When any single prediction is flagged as unstable, the model stability indicator defaults to 0.

Figure 10: Example predictions of calibration and NE over a period of time.

Operationalizing quality alerts

With a unified definition in place, we recorded the stability metadata and expected thresholds for every model in the registry. This portable definition of health enabled us to build generic alerting across all ranking models. Instead of hunting for issues manually, ML teams are now notified the moment a model becomes unstable.

Figure 11: Model stability definitions stored in the model registry.

This shift to measuring predictive health has uncovered previously hidden issues within models, allowing teams to remediate them faster than before. The result is a direct improvement in the quality of the recommendations our systems deliver to users.

Lessons from the field

Scaling Instagram’s recommendation system to over 1000 models produced several key takeaways that continue to guide our infrastructure strategy.

Understanding infrastructure is a prerequisite for building the right tools

We couldn't solve the discoverability or launch problems until we had a unified footprint of what was actually running. This foundational understanding drove the creation of tools like the model registry, which in turn boosted our reliability posture and operational efficiency.

Enabling teammates accelerates the entire organization

By removing the bottlenecks in the model iteration cycle, we gave client teams the autonomy to own their workflows end-to-end. This self-service approach reduced the operational load on the central infra team and increased agility and productivity across the board.

Reliability must include a quality dimension

An always-on service that returns bad results is not reliable from the user's perspective. Operationalizing model stability gave us a clear, measurable way to manage model quality proactively. This focus on accuracy is critical to maintaining user engagement and satisfaction as our ML systems continue to grow.