Why Meta Is Reworking Interest Representation for Deep Funnel Ads

Meta’s ads system faces a fundamental representation problem: how to connect sparse, high-intent user signals to an enormous and constantly shifting catalog of advertiser offerings. Hierarchical Interest Representation is an upstream representation layer designed to solve this by learning unified embeddings across the full universe of ads entities — users, advertisers, products, and services. The goal is to map latent user interests onto what advertisers actually offer, improving deep funnel optimization across systems like GEM, Andromeda, and the Adaptive Ranking Model.

The core innovation is an in-house transformer-based graph learning approach with bias-aware attention and self-supervised cross-view distillation. It learns multi-hierarchical interest representations across one of the largest graph networks in the industry, trained end-to-end on billions of real interactions.

Three Structural Hurdles

User engagement with ads entities is naturally graph-structured: users and entities (advertisers, products, campaigns) are nodes; activities and events are edges. At Meta’s scale, that graph presents three distinct challenges.

Sparse Connections in a Massive Vocabulary

Millions of advertisers serve ads to billions of people monthly, but impression opportunities are finite and deep funnel feedback is rare. The raw graph is vast yet thinly connected, making it difficult to learn meaningful relationships from direct edges alone.

Inferred Signal Dynamics

Meta supports explicit feedback like "Interested/Not interested" on content, but inferred interests from engagement patterns remain critical. These signals are noisy and ever-shifting, requiring representations that remain stable enough to be useful for ranking.

Long-Range Relationship Capture

Because individual deep funnel connections are sparse, patterns are best observed across long-range, graph-connected entities. Capturing these global relationships at scale demands memory-efficient attention kernels and high-performance learning algorithms — a computationally intensive pursuit even with modern hardware.

Four Design Principles

Hierarchical Interest Representation projects the raw graph into a configurable super-graph driven by four properties.

Dimension Reduction

The system collapses the raw graph into a super-graph where each super-node is a learned latent interest primitive. Sparse user-ad edges become meaningfully denser when projected onto primitive interests. This primitive vocabulary is inherently more stable and stationary than the dynamic raw entity space.

Knowledge Enrichment

Advertiser and product entities are enriched with multimodal content — text, images, and video — pulled from page metadata and catalog attributes. These features are processed through vision and language models, so the system understands what an entity is, not just how users interact with it. This enables generalization to rare or unseen entities.

Unified Relational Representation

Users, entities, and latent interest primitives are learned together in a single metric space. Embedding operations allow the system to compute primitive-to-primitive and cluster-to-cluster relationships, estimate user proximity to interest primitives, determine how closely an ad serves a given interest, and identify nearest-neighbor users, ads, and products across types.

Multi-Hierarchical Granularities

There is an inherent trade-off in projecting to primitive interests: coarser abstractions capture dense, stable relationships, while finer levels preserve sparse, specific connections. The system learns cascading super-graphs across multiple hierarchical layers, allowing downstream ranking, retrieval, and personalization architectures to select the appropriate granularity.

The output is a set of universal embeddings for ads entities plus "Bag-of-Meaning" interest tokens. These representations are designed to power new personalization, retrieval, supervision, and ranking architectures across Meta’s ads stack — bridging the distance between high-level stable interest anchors and the specialized, sparse signals of deep funnel intent.

Building Representations Over a Heterogeneous Engagement Graph

Hierarchical Interest Representation (HIR) learns embeddings that combine advertiser and product knowledge with a user's direct engagement history. The system applies an LLM-inspired transformer architecture to large-scale graphs, using sparse attention to model long-range relationships, and is designed to serve applications across the ads stack, from retrieval to final ranking.

The foundation is a typed, weighted, time-decayed graph that unifies users, ads, advertisers, campaigns, products, and pixels. Typed engagement edges—such as a business creating an ad or a user interacting with one—carry action type and timestamp so that recency is balanced against long-term interest. This structure allows a single representation to reason about user lifestyle, advertiser catalog reach, and connecting products in one space.

The graph spans billions of monthly entities and interactions, running both online for production freshness and offline for iteration. Node embeddings fuse pretrained semantic features with behavioral statistics. Frequent nodes receive a learnable ID embedding via deep hash embeddings: a small shared network over hashed node IDs that keeps memory bounded as the vocabulary scales toward tens of billions.

Encoder Design and Graph-Aware Attention

The encoder assembles several complementary input signals. World knowledge—multimodal summary text, images, and video from relatively stable entities like advertisers—is processed through a customized LLM engine to produce encoded features. A node encoder joins node type, hashed ID, these world knowledge features, and per-type metadata so each node carries both semantic identity and real content. A position encoder adds local topology via random-walk and importance-prioritized position encoding over the sampled graph view. An edge encoder feeds edge type, weight, and temporal signals into attention.

Graph relationships translate naturally to transformer attention. Structural signals—node-type transitions along event edges and shortest-path distances—enter as attention biases that augment the query-key dot product in every layer. This keeps the model topology-aware rather than treating a subgraph as an unordered bag of nodes, capturing long-range patterns that message-passing typically over-smooths.

Structural bias usually forces materializing a full pairwise bias matrix, sacrificing memory-efficient attention. FlexAttention instead computes each bias term on the fly, so the matrix never exists. Variable-length subgraphs pack into a single block-masked sequence, avoiding padding waste and cross-graph leakage. New biases become small scoring rules with no low-level kernel changes.

Training Objectives and Temporal Safety

Two objectives train the encoder. Cross-view distillation operates as self-supervision: for each anchor node, a broad teacher view and narrow student view both pass through the encoder, and the student must predict the teacher's interest cluster. Since the teacher sees more context, its confidence provides a target for the student—extending supervision far beyond the sparse fraction of users with deep-funnel conversions. Sinkhorn-Knopp balanced assignment prevents all clusters from collapsing into one.

Engagement prediction is the supervised counterpart. Given two node representations, an engagement type, and a time, the model predicts whether a real edge exists at that time. Together the two objectives blend view-invariant structural priors with observed behavior, turning HIR into a general scoring function for the delivery stack.

Training and evaluation strictly respect event ordering. The graph engine imposes a cutoff timestamp per call and masks any node or edge dated later. Edge splits are chronological across train, validation, and test windows, and batch shuffling happens only inside fixed time chunks. The model sees only information that would have existed at that moment, so gains reflect learning rather than future leakage.

The raw graph lives on Meta's online graph engine, feeding both training and serving from the same data source. During training, subgraph fetches and node-feature reads pipeline with GPU compute: workers prepare upcoming batches while the model trains on the current one, hiding data-loading latency. Testing showed a 30x wall-clock speedup over synchronous baselines with high GPU FLOPs utilization, and bit-exact reproducibility is preserved through migrations and checkpoint recovery.

Tokenization and Downstream Activation

Continuous universal embeddings serve ranking well but are ill-suited to inverted-index retrieval, set aggregation, or human interpretation. HIR discretizes them via composite quantization into Bag-of-Meaning (BoM) tokens—a compact, unordered vocabulary of interest concepts. Users are described by tokens for their interests; ads and advertisers by tokens for the interests they serve.

HIR plugs into delivery through existing components such as GEM, Andromeda, and the Adaptive Ranking Model. BoM tokens let engagement-based personalization incorporate latent user interests and enable compact inverted-index recall for retrieval. The hierarchy also supports specialized architectures, including Mixture of Experts generative distribution routing organized by super-interest categories and hierarchical reasoning over stable interest anchors for user-ad affinity.

Ongoing Work

Future efforts target training scaling efficiency, embedding freshness, knowledge compression, and more expressive memory-efficient attention kernels. The team is also exploring parameter-efficient, objective-conditional fine-tuning of the upstream encoder, which would enable segment-level specialization for heterogeneous deep-funnel objectives on a shared base model.