Sequence Learning at Ads Scale: A Two-Stage Architecture
Meta’s recommendation platforms process billions of daily user interactions, and a 2024 post described how modeling the order and timing of user actions — rather than relying on static engineered sparse features — yields richer, sequence-aware representations of user interests. That approach worked, but scaling it into production posed a new problem: temporal sequences and the transformers that process them grow together, and the legacy hybrid architecture — one model for user event sequences, another for sparse feature interactions — became the bottleneck.
The hybrid design carried three specific tradeoffs: lossy knowledge transfer between components, continued dependence on manual feature engineering, and scaling ceilings caused by interference between ranking and sequence components. To resolve the tension between model complexity and serving efficiency, Meta introduced a multi-stage sequence model that decouples offline user modeling from online ranking, paired with a learning paradigm built on dense tokenization and target-aware attention.
Decoupling Offline Modeling from Online Ranking
The multi-stage architecture splits sequence processing into two complementary stages with distinct operational profiles. The first stage, an offline user model, runs asynchronously and processes long user histories as a deep transformer. It scales to several transformer layers with sequence lengths in the thousands, producing user-level embeddings that are precomputed and cached. The upstream model strictly separates user features from ad and context features, which keeps user embeddings independent of any particular ad candidate.
The second stage, an online ranking model, combines those cached representations with fresh user signals and real-time ad candidate information to produce a final ranking. This stage is optimized for speed and operates within strict latency budgets.
Separating the system into two distinct stages changes the economics of scaling. Model complexity can grow along a scaling curve for the offline user model without proportionally increasing serving costs for the online ranking models.
Architecture Innovations: Dense Tokenization and Target-Aware Attention
Two architectural changes drive the learning capability of the sequence model. Dense tokenization integrates sparse features with sequential behavioral data into a single dense vocabulary, allowing attention mechanisms to discover feature interactions independently. Traditional recommendation systems relied on manually engineered representations to capture sparse cross-feature interactions; here, the model learns those interactions directly from data.
Target-aware multi-head attention fuses tokenized sparse features and ad candidate information with user behavior sequences, then processes them through a memory-efficient attention mechanism that lets each layer weigh past user behaviors against the specific ad being scored. Stacking multiple aligned attention blocks with stable attention distributions enables each layer to capture higher-order interactions between the target ad and historical behavior, progressively distilling long sequences into compact representations.
Scaling Properties and Levers
On real-world ads traffic, the multi-stage model demonstrates a predictable, LLM-style scaling law. Performance improvements follow a log-linear relationship with respect to compute, with a marked improvement in scaling efficiency over other transformer-based sequence models. The relationship between compute (FLOPs) and performance (measured by normalized entropy, NE) holds across model depth, content and semantic enrichment, model width, and sequence length.
That LLM-style scaling emerged despite structural differences is notable: ads systems must integrate sparse ID features with temporal user sequences, whereas LLMs process dense continuous text. Four levers are anticipated to unlock further gains along the scaling frontier:
- Balanced model shape: Performance requires balanced growth across model depth, width, and sequence length. Scaling along a single axis leaves the others as bottlenecks, leading to diminishing returns — a finding mirroring LLM scaling research, described as the scaling synergy principle.
- Multi-stage tunability: The architecture provides a tunable lever to scale either stage independently. Scaling the online ranking model delivers steeper improvements per unit of compute, bounded by serving and request time requirements. Scaling the offline model follows a more gradual curve, but its asynchronous inference avoids latency constraints and allows unhindered scale-in.
- Sequence composition: Performance improves as sequences lengthen, but sequence diversity beats homogeneity. A balanced mix of action types (views, clicks, conversions) yields better results than single-action sequences, suggesting that diverse engagement types and broad temporal coverage produce richer behavioral representations.
- Semantic feature representation: Semantic content features from foundation models complement traditional collaborative filtering signals. They are especially valuable in cold-start scenarios — new ads or advertisers with limited historical data — improving signal coverage for the sparse problem at the heart of recommendation systems.
Measured Impact
The multi-stage sequence modeling architecture has delivered results across three dimensions. By modeling thousands of user event sequences, the offline model generates nuanced user representations that improve ad relevance and conversion rates. Together with broader modeling innovations, these sequence-derived representations drove a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook, and 3.5% in ad clicks on Facebook.
The two-stage design also delivers performance improvements with greater compute efficiency than hybrid approaches. Initial evaluations improved ranking quality with minimal impact to serving resources, confirming that model complexity and production efficiency can scale together. As a core component of Meta’s Generative Ads Recommendation Model (GEM), the architecture is designed for generalization: the same multi-stage backbone and scaling properties extend to any ads ranking task with minimal adaptation and overhead.
Current Work and Outlook
The sequence model scaling law shows no signs of saturation. With architectural parity achieved, scaling model complexity can draw on techniques proven in the LLM domain — mixture-of-experts, cross-user compute sharing, and advanced attention mechanisms — potentially allowing continual scaling at the optimal performance and efficiency tradeoff.
The technical details of the architecture and its scaling properties are published in the paper “LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation.”
