One Model Across the Ads Ecosystem
The 2026 generation of GEM replaces per-domain optimization — Facebook, Instagram, Reels and others handled separately — with a single central foundation model trained on data and signals from all of Meta's ads platforms. It bends the scaling curve at LLM-scale computation and uses post-training techniques that widen the latency budget without requiring real-time evaluation of every ad candidate, all on a more efficient training stack. Gains therefore arrive simultaneously across surfaces and funnel stages rather than one surface at a time.
Measured results: 18.9% conversion improvement and 12% CTR improvement across Meta's ad surfaces and objectives, plus 5.8%, 4.1%, and 1.6% conversion uplift on Facebook, Instagram, and off-site conversion for targeted users over the past year. The uplift covers both upper-funnel surfaces and deep-funnel objectives.
Architecture: Kunlun and the Bending of the RecSys Scaling Curve
Frontier LLMs scale predictably with compute, parameters and data in transformer architectures; recommender systems historically have not, because of massive heterogeneous features and shifting distributions. GEM's answer is Kunlun, an LLM-style architecture co-designed with current GPU hardware for cheaper, deeper models and long-context learning over user engagement history. At GEM's largest pre-training scale, Meta observed an LLM-like scaling law alongside 2.5x growth in model complexity over earlier GEM versions: more capacity and longer context produce emergent ranking-quality gains.
Layer composition
Each Kunlun layer combines three elements:
- Attention — customized attention and feed-forward networks that capture long dependencies in user behavior and adapt to data dynamics.
- Summarization component — a simplified summarization architecture that extracts user insights from sequences and bridges sequence and non-sequence structure.
- Cross interaction — expanded into a mixture-of-experts (MoE) design for large-scale modeling of interactions between sequence and non-sequence features.
Two feature families frame the design: non-sequence features such as user demographics and ad content representations, and sequence features made of event sequences, each event carrying its own feature set.
Against the previous Interformer architecture, Kunlun delivers 2x performance gains at the same computation scaling. Three innovations account for this.
Deep hardware co-design
- Block attention replaces full self-attention with block-aligned sparse attention, reducing complexity in sequence length from quadratic to linear relative to window size, using only window sizes that are multiples of the block size so no partial blocks reach the kernel.
- Architecture reformulation turns the Per-Example Feed-Forward Network (pFFN), which adapts traditional FFNs to per-example distributions, into attention operations via Generalized Dot-Product Attention (GDPA). Unifying the model's attention-like operations into a single GPU kernel yields up to 2x forward speedup.
- Expert parallelism gives the cross-interaction component a scalable, MoE-like framework in which each GPU processes a subset of expert sub-networks in parallel.
Sequence scaling
Two components make ultra-long sequences tractable. Component Skip Connection (CompSkip) selectively activates sequence modules per layer and routes outputs only through residual connections; deactivating components on the assumption that the network recovers quickly on re-activation cuts FLOPs by 43% with negligible quality loss. Sequence Unification (OneSeq) unifies user sequences chronologically after event feature aggregation so a single unified sequence layer can process long sequences even when events are heterogeneous — a purchase event described by ad ID and account ID, for instance, cannot be merged at the data-source level with a post event described by actors and object ID. Post-aggregation unification maximizes GPU efficiency through large tensor operations and reduced operator fragmentation.
 |
 |
What Long Context Buys
Kunlun makes ultra-long sequences feasible; GEM then scaled the long engagement context by 2x in production and demonstrated infrastructure capable of scaling long context up to 25x. Three observations emerged:
- Prediction quality follows a power law with sequence length, predictably and consistently across lengths.
- Larger models benefit disproportionately from longer sequences, creating a cycle in which more compute justifies longer sequences and longer sequences justify more compute.
- The two scaling laws reinforce each other, pointing to investment directions beyond simply adding compute.
Transfer, Serving and Training Efficiency
Serving a model at GEM's scale in real time is prohibitively expensive and would degrade latency, hurting both user experience and advertiser ROI. LoopFM, a structured representation and state-of-the-art post-training technique, transfers GEM's internal representations and sequential historical memory into downstream models. Computation sharing doubles the intelligence transfer relative to the knowledge distillation used in the previous version, at no additional compute cost or latency. Elastic pre-computation (Solaris) harvests the upstream model's gains.
On the training side, data center GPUs and their software stacks are largely tuned for LLM workloads, while recommendation workloads have a different data profile that makes high compute utilization at scale difficult. Meta developed custom low-precision kernels and new 5D parallelism strategies co-designed with current GPU hardware and its training cluster, and optimized model architecture together with infrastructure. The result is 4x total training FLOPs at doubled end-to-end training efficiency — 20-25% model FLOPS utilization (MFU) — enough for GEM to train at LLM scale (see the companion blog on Training GEM at LLM Scale).
The Central Compute Engine
Meta's apps cover dozens of surfaces with 3.6 billion people using at least one daily, and every surface and objective previously ran its own prediction system. GEM instead drives one unified compute engine from the central foundation model, jointly predicting clicks and conversions across all surfaces, including Facebook and Instagram, and across every key advertiser objective from off-site conversions to mobile app installs. Representations learned on data-rich click tasks transfer to data-sparse conversion tasks. Cost-free data mixing and a domain expert framework keep this heterogeneous learning tractable, with predictable scale and improved reliability. Post-training, in turn, extends the latency budget without requiring real-time evaluation of all candidates.
Transferring the Central Model's Intelligence
GEM's value rests on how well its intelligence moves into the hundreds of vertical models serving ads in real time. The approach pairs precomputed structure representation — carrying deep historical memory from the central brain — with knowledge distillation, and reports 2x transfer effectiveness beyond distillation alone. Async computation is also used for a few domain-specific foundation models to move knowledge into vertical models.
Precomputed Structure Representation
Distillation transmits only a compressed view of GEM's intelligence at each training process, which caps its bandwidth. The precomputed structure representation, for example LoopFM, compresses the central brain into a compact form and structures it with deep historical memory so downstream models can learn long context. Its parts:
- Understand and compress intelligence: intelligence from GEM is compressed into compact representations, summarized from the important layers, that describe the event or data processed by the foundation model.
- Structure with deep historical memory: that compact representation is sequenced with representations GEM previously produced for the same user in a context window, capturing intent dynamics across all surfaces and traffic via GEM's memory.
- Learn from long context intelligence: specialized architecture lets downstream models learn the structure representation with long historical context at both inference and training time.
Generation is precomputed and shared with the computation that produces soft labels for distillation, so no additional foundation model cost is incurred.
Speculative Precomputation with Solaris
Evaluating a large-scale model inside a live ad request is not affordable. Solaris moves the evaluation in time instead: it speculatively precomputes foundation model embeddings for user-ad pairs a future request is likely to need and delivers them just-in-time on the serving hot path, borrowing from speculative decoding in LLM serving. Downstream models thus get near-real-time foundation model intelligence without a latency constraint.
- Deliver fresh runtime knowledge: the user-ad embedding is generated directly, giving vertical models point-to-point knowledge of how a specific user is likely to act on the exact ad candidate.
- Speculate ahead of the request: as multi-stage ranking narrows a request to its top candidates, Solaris evaluates those user-ad pairs with the foundation model in asynchronous background jobs — no latency budget, elastic capacity, richer knowledge.
- Enrich coverage hierarchically: when an exact user-ad embedding is missing, the system degrades rather than drops, substituting an aggregate of the user's own recent embeddings across the same ad groups or borrowing from behaviorally similar users.
Cross-Domain Optimization at Meta Scale
GEM was further trained on ads data at Meta scale: all-surface data such as Facebook and Instagram Feed and Reels/video, plus deep funnel optimization covering off-site conversion and mobile app install. It powers the first optimized model that jointly predicts CTR across all surfaces and CVR across the major optimization goals; previously each surface and objective had its own system. Engagement patterns from across platforms can now inform conversion behavior, such as purchases, across ad surfaces, making GEM the central model and compute engine for ads recommendation on Meta's advertising surfaces.
RecSys models, unlike LLMs with static data, must handle data dynamics, freshness and diverse distribution across surfaces and traffic. GEM's training recipe for large-scale cross-surface, cross-domain unification consists of:
Cost-free data mixing. Virtual data loading pulls from combinations of data pipelines without physically merging data, avoiding added storage cost, while remaining flexible for an ecosystem where downstream models still train on surface- or traffic-specific data. A data mixture mechanism controls mixture proportions during offline and online training, tracking the real datastream order received in real time and mitigating instability at domain switches and overmemorization of recent domain data.
A domain expert framework. A shared expert learns a unified representation across all domains while domain-specific experts capture surface-specific patterns, each expert drawing on both cross-domain and specialized knowledge. The experts are built on the Kunlun architecture framework.
Training Efficiency at LLM Scale
GEM now trains on several thousand of the latest-generation GPUs. Data center GPUs and their software stacks are largely optimized for LLM workloads, but recommendation workloads have a different profile — distinctive data characteristics and rich user and ads signal interaction patterns — which makes high GPU compute utilization hard to reach for a foundation recommendation model of GEM's size.
Training efficiency is measured through E2E MFU, decomposed as E2E MFU = Local MFU (compute efficiency) × Scaling Ratio (scaling efficiency). Local MFU captures how close a single GPU runs to the hardware roofline; Scaling Ratio captures how much single-GPU performance survives distribution across thousands of GPUs. Keeping them separate treats compute and scaling as related but distinct optimization problems, each with its own techniques — both must be addressed to maximize end-to-end MFU.
- Compute efficiency: a customized recommendation kernel library — Jagged Flash Attention (JFA), GDPA, Block-Attention and others — plus mixed ultra-low precision training including MXFP8 attention and MLP, purpose-built for the latest generation GPU architecture.
- Scaling efficiency: topology-aware 5D parallelism with Streaming Multiprocessor (SM)-free collectives — 2D FSDP + Expert Parallelism for dense parameters, combined with Fully Sharded 2D Model Parallelism for sparse parameters — co-designed with Meta's multi-tiered network hierarchy to cut communication overhead.
Over the past 12 months, GEM's E2E training efficiency doubled to 20-25% MFU while total training FLOPs scaled 4x. Training a foundation model at the intersection of LLMs and recommendation systems is a co-design problem rather than a software or hardware problem alone: the 2x gain required kernels, precision, parallelism, networking and memory to move together. The companion blog, GEM Training: Train Meta's LLM-Scale Ads Recommendation Model Efficiently, covers the full technical story.
Roadmap
Meta intends to extend GEM to learn from all of its platform data, capturing behavior across organic and ads engagements spanning text, images and video. The work targets LLM-scale long context so the model can reason over longer spans of engagement context, which is expected to improve ad relevance and help advertisers reach the right audiences more efficiently.
GEM will continue scaling on larger compute clusters, opening new scaling opportunities. AI-driven optimizations — ecosystem enhancement and architecture co-design — are being introduced to accelerate future evolutions, iteratively refining model architecture in lockstep with hardware so each generation delivers more intelligence per dollar of compute. The post-training paradigm is to evolve with reasoning capability, transferring more of that intelligence to downstream models.
Acknowledgements
Thanks to Sandor Abad, Amit Anand Amlesahwaram, Carmen Au, Yasmine Badr, Balaji Balasubramanian, Habiya Beg, Ayush Bihani, John Bocharov, Raghav Boinepalli, Bopa Bovet, Mandeep Chadha, Shuo Chang, Dan Chanpuriya, Hansey Chen, Laming Chen, Ping Chen, Ruilin Chen, Wenlin Chen, Xian Chen, Jae-Woo Choi, Sagar Chordia, Cathy Cui, Yan Deng, Lisang Ding, Wentao Duan, Eric Falconer, Qi Fan, Xiaorui Gan, Nikhil Garg, Chenxiao Guan, Saket Gur, Brian Hansen, Rustam Hashimov, Bingjie He, Jack Hsueh, Yusuo Hu, Angie Huang, Qin Huang, Yuzhen Huang, Subodh Iyengar, Qinjin Jia, Shali Jiang, Qiang Jin, Pawel Kadluczka, Donghyun Kim, Santanu Kolay, Hairong Kuang, Peter Lamar, Seunghan Lee, Max Leung, Jazlyn Li, Mingda Li, Qianru Li, Chen Liang, Haowen Lin, Wei Ling, Boyang Liu, Rocky Liu, Serena Liu, Xi Liu, Xiaolong Liu, Xin Liu, Zikun Liu, Hank Lo, Kenny Lov, Anuj Madan, Shuby Mao, Paolo Massimi, Santosh Mohan, GP Musumeci, Jade Nie, Enes Palaz, Ke Pan, Sandeep Pandey, Jerry Peng, Tianshu Peng, Patrick Phelps, Oana Platon, Richard Qiu, Damian Reeves, Jason Rudy, Purbesh Sahoo, Ke Sang, Rikin Shah, Devashish Shankar, Jingyi Shen, Min Si, Vibha Sinha, Jake Siso, Matt Steiner, Bor-Yiing Su, Anamaya Sullerey, Musharaf Sultan, Chonglin Sun, Vivienne Sung, Qiuling Suo, Kevin Tang, Robert Tang, James Teng, Ritwik Tewari, Lei Tian, Sandeep Tiwari, Viral Vimawala, Ernest Wang, Kai Wang, May Wang, Zhaojin Wen, Rupert Wu, Xiaozhen Xia, Huaqing Xiong, Han Xu, Ivonne Xu, Jiayi Xu, Liangbei Xu, Hao Yan, Tak Yan, Angel Yang, Fan Yang, Jiyan Yang, Justin Yang, Wen-Chi Yang, Hongtao Yu, Li Yu, Zhichen Zeng, Hanbin Zhang, Jiasheng Zhang, Larry Zhang, Lei Zhang, Zhengyu Zhang, Feipeng Zhao, Qian Zhao, Hua Zheng, Jie Zheng, Peng Zhong, Jacky Zhou, Qinghai Zhou, Song Zhou, Zhehui Zhou, Rich Zhu, Wankun Zhu, and the entire team behind the development and productionization of the largest foundation model in Meta's ads recommendation system.
