Why handcrafted features hit a wall
Meta’s ad recommendation engine has long run on deep learning recommendation models (DLRMs) that consume thousands of human-engineered sparse features. Those features capture a person’s interactions with entities such as Facebook pages, and DLRMs proved adept at learning high-dimensional embeddings from them. But the approach has structural weaknesses that no amount of feature tuning can fix:
- Sequential information is lost. Aggregating events over time windows discards the order in which they happened, even though order matters for understanding intent.
- Granular detail disappears. When events are merged across an aggregation window, fine-grained signals — like which attributes co-occurred in a single event — are gone.
- Human intuition is the bottleneck. Complex, non-obvious patterns in massive datasets are exactly what handcrafted features fail to anticipate.
- The feature space becomes redundant. Overlapping aggregation schemes add compute and storage cost without proportionate value.
The shift that addresses these limits is a move from engineered features to learning directly from event sequences, borrowing techniques from natural language understanding and computer vision.
Event-based features
The building blocks of the new architecture are event-based features (EBFs), which standardize heterogeneous inputs to sequence learning models along three dimensions:
- Event streams — the data source, such as the sequence of ads a person recently engaged with or pages they liked.
- Sequence length — how many recent events are drawn from each stream, determined by the stream's importance.
- Event information — semantic and contextual details for each event, like ad category and timestamp.
Each EBF is a single coherent object holding all key event information, replacing legacy sparse features as the models' primary input. Combined with an event model, EBFs eliminate the need for manual feature aggregation.
Sequence modeling with EBFs
The event model synthesizes event embeddings from event attributes. It learns embeddings for each attribute, compresses them linearly into a single attributed-based embedding, and combines that with a timestamp encoding that captures recency and order. The result translates an EBF sequence into a sequence of event-level representations.
This is conceptually similar to how language models embed words — except that EBFs draw from a vocabulary orders of magnitude larger than natural language, spanning heterogeneous event streams with millions of entities.
The event embeddings feed into the sequence model at the core of the recommendation engine. That model, an event-level summarization architecture, uses attention mechanisms to synthesize event embeddings into a predefined number of embeddings keyed to the ad being ranked. Multi-headed attention pooling reduces the self-attention cost from O(N*N) to O(M*N), where M is tunable and N is the maximum event sequence length.
Scaling sequence learning
Productionizing this paradigm required scaling in two directions: the architecture itself, and the length and richness of the sequences it consumes.
Scaling the architecture
A custom transformer architecture was developed to handle the feature encoding schemes needed for full sequential modeling. The core challenge is performance: a single ads request must rank thousands of ads within a few hundred milliseconds. The prior sum-pooling approach, optimized for fixed-length pooled embeddings, doesn't translate directly to variable-length event sequences, which create jagged embedding tensors and higher compute, communication, and variance costs.
The solution relies on hardware codesign to handle jagged tensors efficiently:
- Native PyTorch support for jagged tensors.
- Kernel-level GPU optimizations for processing them.
- A Jagged Flash Attention module, enabling Flash Attention directly on jagged tensors.
Scaling length and richness
Longer event sequences give the model deeper context on a person's evolving interests, while richer semantic signals make each event more informative. Sequence length grows via multi-precision quantization and value-based sampling. Semantics expand through multimodal content embeddings — for example, embedding attributes of each event — encoded efficiently with customized vector quantization.
Results and direction
The event sequence learning paradigm is now widely deployed across Meta's ads systems. Since launch it has improved ads prediction accuracy, yielding 2-4% more conversions on selected segments. The architectural shift also brought infrastructure efficiency gains and faster research iteration.
Future work targets three areas: scaling event sequences by 100X; developing more efficient sequence modeling architectures such as linear attention and state space models; and KV cache optimization plus multimodal enrichment of event sequences.
