Meta’s Generative Ads Model (GEM): The Central Brain Accelerating Ads Recommendation AI Innovation

GEM: Meta’s Foundation Model for Ads Ranking

Meta has introduced the Generative Ads Recommendation Model (GEM), its most advanced ads foundation model to date. Built on an LLM-inspired paradigm and trained across thousands of GPUs, GEM is designed to increase ad performance and advertiser ROI by enhancing the broader ads recommendation model fleet’s ability to serve relevant ads. Since launching earlier this year, GEM has driven a 5% increase in ad conversions on Instagram and a 3% increase on Facebook Feed in Q2. In Q3, architectural improvements doubled the performance benefit derived from added data and compute.

Architectural Innovations

GEM addresses several core challenges inherent to large-scale ads recommendation: a vast and imbalanced feature space with sparse meaningful signals, heterogeneous data across apps, surfaces, and creative formats, and the need for efficient training at scale. Its architecture introduces customized attention mechanisms for sequence and non-sequence features, cross-feature learning, and multi-domain optimization. This design now delivers four times the efficiency of previous ads ranking models in driving performance gains for a given amount of data and compute.

Modeling User-Ad Interactions

GEM enhances the Wukong architecture with stackable factorization machines and cross-layer attention connections. This allows the model to learn which feature combinations matter most. Each Wukong block can scale vertically for deeper interactions or horizontally for broader coverage, enabling discovery of increasingly complex user-ad patterns.

Scaling Sequence Learning

User behavior sequences — histories of clicks, views, and interactions — contain rich preference signals but are difficult to process at length. GEM uses a pyramid-parallel structure that stacks parallel interaction modules to capture relationships at scale. A new offline feature infrastructure processes sequences of up to thousands of events at minimal storage cost, allowing the model to draw on extended histories of organic and ad interactions.

Cross-Feature Learning

Existing approaches compress sequences into compact vectors, risking loss of engagement signals. GEM instead preserves full sequence information via InterFormer, an architecture that alternates between sequence learning and cross-feature interaction layers. This interleaving allows progressive refinement of sequence understanding without sacrificing access to the full user journey.

Multi-Domain Learning

Different surfaces — Facebook, Instagram, Business Messaging — exhibit distinct user behaviors. GEM learns from cross-surface interactions while keeping predictions tailored to each domain’s characteristics. For example, insights from Instagram video ad engagement can improve Facebook Feed predictions while each domain optimizes for its own objective.

Scaling With Efficiency

GEM’s training stack combines multi-dimensional parallelism, custom GPU kernels, and memory optimizations. This delivers a 23x increase in effective training FLOPS with only a 1.43x boost in model FLOPS utilization (MFU), achieved by scaling to 16x more GPUs. This makes it feasible to train an LLM-scale ads foundation model and to continue scaling training capacity at an attractive return on investment.

Post-Training Knowledge Transfer

The value of GEM hinges on transferring its knowledge to hundreds of user-facing vertical models (VMs). Meta employs both direct and hierarchical transfer strategies. Direct transfer applies to major VMs within GEM’s training data spaces; hierarchical transfer distills knowledge into domain-specific foundation models, which then teach VMs. The combined approach uses knowledge distillation, representation learning, and parameter sharing, achieving twice the effectiveness of standard knowledge distillation.

Fresh Supervision via Student Adapter

VMs commonly suffer from stale supervision caused by delays in FM training and domain mismatches with surface-specific objectives. GEM addresses this with a Student Adapter, a lightweight component that refines the teacher’s outputs using recent ground-truth data. It aligns teacher predictions with observed outcomes throughout student training, ensuring more up-to-date supervision.

Complementary Techniques

Representation learning generates semantically aligned features from raw data, supporting efficient teacher-to-student transfer without adding inference overhead. Parameter sharing lets VMs selectively incorporate FM components, enabling smaller, latency-sensitive models to reuse rich representations without incurring the full computational cost of the foundation model itself. Together, these techniques propagate GEM’s learnings across the ads model fleet, driving the paradigm shift in Meta’s ads recommendation system.

Training a Model at LLM Scale

GEM pushes recommendation systems to a scale usually associated with large language models. Getting there required rebuilding the training stack from the ground up. The overhauled stack delivers a 23x increase in effective training FLOPs while using only 16x more GPUs. Model FLOPs Utilization (MFU), a standard measure of hardware efficiency, rose by 1.43x, indicating better use of available GPU resources. Achieving that kind of throughput growth alongside efficiency gains is essential for training an ads foundation model of this size.

Handling the multimodal workloads demanded strategies like multi-dimensional parallelism, custom GPU kernels, and co-design between the model architecture and the underlying system. Those techniques scale nearly linearly across thousands of GPUs, improving compute throughput, memory usage, and overall hardware efficiency.

Parallelism Strategy

GEM contains both dense and sparse components, which presents different scaling challenges. Dense parts rely on Hybrid Sharded Distributed Parallel (HSDP) to cut communication costs and manage memory efficiently across thousands of GPUs. The sparse side—primarily large embedding tables containing user and item features—uses a two-dimensional approach mixing data parallelism and model parallelism, tuned for synchronization efficiency and memory locality.

System-Level Optimizations

Beyond parallelism, the team implemented a range of techniques designed to saturate GPU compute throughput and eliminate training bottlenecks:

• A custom in-house GPU kernel built for variable-length (jagged) user sequences with computation fusion, taking advantage of the latest GPU hardware features.
• PyTorch 2.0 graph-level compilation that automates activation checkpointing for memory savings and operator fusion for faster execution.
• Memory compression via FP8 quantization of activations and unified embedding formats to shrink the memory footprint.
• GPU communication collectives operating without Streaming Multiprocessor (SM) resources through NCCLX, Meta's fork of NVIDIA's NCCL. This removes contention between communication and compute workloads.

Startup Time and Continuous Efficiency

Improving effective training time (ETT)—the portion of training spent processing new data—was also a priority. Job startup time dropped by 5x through tweaks to trainer init, data reader setup, checkpointing, and PyTorch 2.0 compilation. PyTorch 2.0 compilation alone dropped by 7x via caching.

Hardware efficiency is managed across the full model lifecycle. During the exploration phase, lightweight model variants accelerate iteration at a fraction of full-sized training cost; these variants support over half of all experiments. In the post-training stage, GEM runs forward passes to generate knowledge, such as labels and embeddings, for downstream models. Unlike language models, Meta also performs continuous online training to refresh the foundation model. Traffic sharing is optimized between training and post-training knowledge generation, and between the foundation model and its downstream users, reducing total computational demand across the end-to-end system.

Scaling GEM Across the Ecosystem

Future improvement hinges on deeper understanding of user preferences and intent. For advertisers, the goal is one-to-one connections at scale, driving stronger engagement and output.

Plans call for GEM to learn from Meta's entire ecosystem—user interaction on both organic and ads content across text, images, audio, and video—and to extend those learnings to major surfaces across Facebook and Instagram. That stronger multimodal foundation helps GEM parse the nuances behind clicks, conversions, and long-term value. It sets the stage for a unified engagement model that ranks organic content and ads intelligently, delivering maximum value on both sides.

Ongoing research will scale GEM further by advancing both architecture and training recipes on new AI hardware, helping it learn efficiently from more diverse data. GEM will also evolve toward inference-time scaling to allocate compute dynamically, power intent-centric user journeys, and enable agentic, insight-driven advertiser automation built for higher return on ad spend.