GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Bridging the LLM-Recommendation Gap

Meta’s Generative Ads Recommendation Model (GEM), the foundation model powering ad recommendations across Instagram and Facebook, has reached LLM-scale training on several thousand of the latest-generation GPUs. The central achievement: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4× in 12 months.

The engineering challenge stems from GEM’s hybrid architecture — trillions of sparse embedding parameters alongside billions of dense parameters — combined with recommendation-domain data properties that diverge sharply from typical LLM workloads. Standard AI infrastructure optimized for LLM training does not transfer directly. The solution required co-designing kernels, precision, parallelism, networking, and memory.

Why GEM Strains Conventional GPU Stacks

GEM applies customized attention mechanisms to two feature categories independently while enabling cross-feature learning: sequence features (e.g., user activity history) and non-sequence features (e.g., user location, ad creative representation). The interaction between this hybrid design and recommendation-domain data properties creates two distinct bottlenecks.

Challenge 1: Per-GPU Compute Underutilization

Data center GPUs and their software ecosystems are optimized for LLM-shaped workloads, but GEM’s training profile differs fundamentally:

• Jagged inputs: User activity history varies wildly in sequence length. Padding to max length wastes up to 50% compute.
• Asymmetric attention shapes: Self-attention operates on long sequences with short attention windows; cross-attention pairs long queries with short key/value; pooled multi-head attention (PMA) compresses history into short queries over long key/value. These shapes undermine intra-kernel pipelining efficiency.
• Memory-bound operations: Small embedding dimensions in MLPs and various normalization layers leave compute units idle.
• Numerical sensitivity: CTR/CVR prediction tasks are highly sensitive to precision changes, making naive low-precision training impractical.

Challenge 2: Scaling Bottlenecks Across Thousands of GPUs

E2E latency per training step follows the formula Max across GPU Rank (Max(Local Compute Time, Communication Time)). Near-linear scaling demands four conditions: compute time dominates communication, communication hides behind compute without contention, minimal recomputation from memory pressure, and consistent load balance across ranks. GEM threatens each one:

• Trillion-scale sparse parameters with billion-scale dense parameters drive heavy communication amid mixed compute patterns.
• Layer-wise architecture diversity creates uneven overlap windows; resource contention between communication and computation complicates hiding.
• Long sequences generate large activations that push memory limits, forcing recomputation that erodes efficiency.
• Jagged sequences induce data-driven load skew across ranks.

An Efficiency Framework With Two Levers

Training efficiency decomposes into a product: E2E MFU = Local MFU (compute efficiency) × Scaling Ratio (scaling efficiency).

Local MFU measures single-GPU utilization against the hardware roofline. It is governed by kernel design, numerical precision, and how workload compute patterns map onto GPU architecture — Tensor cores, memory hierarchy, and streaming multiprocessor scheduling.

Scaling Ratio measures retained single-GPU performance across thousands of GPUs, where communication overhead, load imbalance, straggler effects, and memory-driven recomputation erode it. Isolating local MFU is done by running layers individually on one GPU without recomputation or communication exposure; scaling ratio derives from the ratio between local and E2E MFU.

Two Optimization Tracks

Compute Efficiency: Custom Kernels and Ultra-Low Precision

At the kernel level, a customized recommendation kernel library was developed — including Jagged Flash Attention (JFA), Generalized Dot-Product Attention (GDPA), and BlockAttention — alongside mixed ultra-low precision training with MXFP8 attention and MLP, purpose-built for recommendation workloads and designed to exploit the latest GPU architectures.

Scaling Efficiency: Topology-Aware 5D Parallelism

For distributed scaling, Meta implemented topology-aware 5D parallelism with SM-free collectives: 2D FSDP combined with Expert Parallelism for dense parameters, alongside Fully Sharded 2D Model Parallelism for sparse parameters. This design is co-optimized with Meta’s multi-tiered network hierarchy to minimize communication overhead.

The distinction matters because it separates two related but independent problems: compute efficiency is a kernel-level and precision problem targeting the per-GPU roofline; scaling efficiency is a distributed-systems problem addressing the gap between single-GPU and multi-GPU throughput. Both levers were required to reach the 20–25% E2E MFU at 4× scaled FLOPs that defines GEM’s current training capability.

Kernel-Level Rewrites for Recommendation Workloads

GEM's training efficiency gains come from two complementary efforts: a custom kernel library built for the jagged, asymmetric shapes typical of recommendation data, and an ultra-low-precision training recipe. The kernel work targets three specific bottlenecks, each with its own dedicated solution:

• JFA — eliminates up to 50% compute waste from padding jagged inputs.
• BlockAttention — reduces long user-history self-attention cost from O(L²) to O(L) while preserving model quality.
• GDPA — unifies and accelerates GEM's diverse, asymmetric attention modules where FlashAttention's dense long-sequence assumptions break down.
• MXFP8 attention + MLP — converts lower-precision Tensor Core throughput into real end-to-end speedups without regressing precision-sensitive CTR/CVR objectives.

Jagged Sequence Flash Attention (JFA)

FlashAttention assumes dense, fixed-length sequences as found in LLMs. Recommendation user sequences are inherently jagged — ranging from hundreds to tens of thousands of tokens per sample. Padding to max length wastes up to 50% of compute, while naive approaches that skip padding leave SMs idle when short sequences finish early. JFA is a custom FlashAttention implementation that operates directly on variable-length jagged tensors, eliminating padding overhead while supporting rec-specific features like custom attention biases, asymmetric query/key-value lengths, and efficient backward passes.

JFA went through four generations of refinement to close the gap from being slower than padded SDPA to matching SOTA CUDA/Cutlass performance on latest-generation GPUs:

• Jagged masking via subtraction scheme: Traditional 2D masking for jagged boundaries (marking invalid positions with -inf) consumes significant non-tensor-core instructions (~28% of executed instructions). The replacement masks Query/Key with zeros via the Tensor Memory Accelerator (TMA) and subtracts the extra exponents, producing numerically equivalent results without masking overhead.
• Backward parallelization: FlashAttention's backward pass requires accumulating dQ across sequence tiles, typically via costly atomic adds. For recommendation workloads with high batch x heads, a non-seq-parallel scheme with split dQ computation delivers 21-40% backward speedup by eliminating both atomic writes and redundant recomputation.
• Warp specialization and persistent kernels: Upgrading to Triton Low-Level Extensions (TLX) enabled explicit warp specialization, TMA usage, and persistent kernel scheduling — unlocking 30-100% TFLOPS improvement by leveraging the latest hardware features.

JFA v4 (TLX) achieves 40-140% TFLOPS improvement over JFA v2 under production jagged distributions (sparsity 0.5), contributing to an 18.5% relative local MFU gain and 12% QPS gain.

Generalized Dot-Product Attention (GDPA)

GEM relies on diverse attention-like interaction patterns — self-attention, PMA, and cross-attention — that share a common structure: two matrix multiplications with an element-wise activation in between, but replacing softmax with activations like GELU or SiLU instead. These modules are unified under a single GDPA kernel optimized for production RecSys training on latest-generation GPUs.

Existing FlashAttention kernels, designed for LLM-style dense long-sequence inputs, perform poorly under real production traffic. A 2.6x forward performance gap and up to 4x worst-case gap was observed between real-world workloads and synthetic benchmarks, driven by short/asymmetric K/V sequences, jagged inputs, and large batch sizes that break pipeline occupancy assumptions.

The kernel pipeline, scheduling, and math were redesigned to close the gap between real-world performance and the hardware roofline:

• Pipeline redesign for non-softmax activations: Eliminating the softmax correction stage frees four warps and their registers. For short K/V sequences, outer-loop software pipelining recovers ~10% performance lost to inner-loop pipelining when the inner loop runs only 1–2 iterations.
• Software-level tile scheduling for jagged tensors: Valid tiles are precomputed on CPU, empty tiles skipped entirely, and zigzag assignment applied across SMs — reducing workload skew from 6x to near-balanced.
• ALU-only activation approximation: GELU's SFU-bound tanh is replaced with a 6th-order Taylor expansion (ALU-only), accurate within the bounded input range enforced by QK-norm. This eliminates SFU contention in both forward and backward passes.

BlockAttention

For GEM self-attention, the core efficiency challenge was scaling long user sequences without paying the quadratic cost of full attention. The first step was moving from full self-attention to sliding-window attention, limiting each token to nearby events and reducing complexity from O(L²) to O(L * window). The Sliding Window Attention (SWA) kernel skipped off-window tiles in JFA and reduced long-sequence self-attention latency by up to 68% with neutral NE (normalized entropy, a model-quality metric).

The structure was then pushed further with block-aligned attention. Since GEM can safely use fixed 64-token blocks, each Q block only attends to its corresponding K/V block, turning attention into independent 64×64 problems. This removes the partial-window masking and multi-tile iteration still present in SWA, and lets a dedicated TLX kernel eliminate FlashAttention overheads such as online softmax correction, logsumexp HBM traffic, and separate Di preprocessing.

Fusing RoPE backward into the attention epilogue removes another memory-bound kernel and keeps gradients in FP32 registers. Together, TLX block attention + fused rotary improves self-attention layer MFU by +30.6% over Triton block attention, or roughly +44% over the SWA baseline.

Mixed Ultra-Low-Precision Training

Lower precision directly translates to higher Tensor Core throughput on GPU. On the latest generation hardware, FP8 delivers 2x peak FLOPS over FP16, and FP4 delivers 4x. As hardware vendors scale low-precision FLOPS faster than FP16, low-precision training becomes increasingly attractive — provided numerical stability and quantization overhead are handled.

MXFP8 Flash Attention

An extension of the FA4 kernel adds end-to-end MXFP8 blockscaled MMA for both forward and backward passes, leveraging the GPUs' native low-precision support. Low-precision attention is not simply a datatype swap: scale factors must be generated along each GEMM's K dimension, staged through shared memory (SMEM) / tensor memory (TMEM) despite FA4's already full TMEM footprint, and computed online for intermediates such as softmax P and backward dS.

To keep the Tensor Core speedup at module level, quantization was fused into upstream normalization and projection kernels — emitting FP8 activations and tensor-core-friendly scale layouts directly while avoiding extra BF16 global-memory traffic. For jagged recommendation workloads, FP8 data stays at unpadded positions with only compact scale factors scattered/padded for TMA. Three kernel-level innovations were required:

• TMEM scale factor placement: FA4 fully utilizes its 512-column TMEM for accumulators, leaving no room for block-scale factors. The solution overlaps scale factors with temporarily-unused TMEM regions (placing S(i) scale factors in the S(1-i) accumulator region), requiring one additional lightweight barrier hidden behind existing GEMM latency.
• Online P-to-MXFP8 conversion: Softmax output (P) is quantized to MXFP8 in-place within the softmax warp, reusing the row-max already computed for softmax normalization. Scale factors are derived via optimized PTX bit-manipulation sequences instead of expensive log2/round/clamp operations.
• Block-wise quantization: A [32, 32] square quantization computes one scale factor per 32×32 block via redux.sync.max.abs.f32 warp-wide reduction. This makes quantization transpose-invariant so each tensor is quantized only once — useful for the backward pass, where transposed Q,K values are needed.

On GEM representative shapes (measured on power-capped latest-generation GPUs), >1.3x speedup was achieved for the forward kernel with MXFP8, and >1.5x for the backward kernel.

Handling Quantization Overhead

Quantization overhead comes from two sources: model parameters (weights) and intermediate tensors (activations). Handled naively, extra casting, scaling, and data movement can offset the compute speedup from low-precision Tensor Cores.

• Weight quantization on FSDP shard: Each rank's local shard is quantized before the FSDP all-gather to amortize quantization cost across ranks, avoiding re-quantization of the fully gathered weight on every rank. The all-gather itself then communicates low-precision payloads instead of BF16, reducing volume and latency.
• Activation quantization via kernel fusion: For linear modules, quantization is fused into the preceding normalization (PreNorm fusion) to avoid a separate quantization step with its kernel launch and HBM traffic. For attention modules, quantization is also fused into the preceding projection so the attention kernel consumes low-precision activations directly with no extra quantization step.

Addressing Numerical Stability

Quantization errors, outliers, and rounding bias can make low-precision training numerically fragile, especially for gradient computation. These challenges were addressed through a three-pronged strategy:

• Outlier mitigation: Random Hadamard Transforms spread outliers and smooth distributions prior to low-precision quantization.
• Recipe tuning: Stochastic rounding eliminates deterministic rounding bias. Selective skipping or higher-precision weight-gradient (WGrad) computation is applied where activations and gradients exhibit sevoutlier behavior, materially improving model quality.
• Mixed precision: Ultra-low precision is used where it benefits most (e.g., large GEMMs), with fallback to BF16 where ultra-low precision is insufficient — for example, later model layers that are more sensitive to quantization errors.

Pushing Past the Scaling Bottlenecks

Training efficiency at thousands of GPUs degrades unless every layer of the stack is coordinated. The optimization work behind Meta's GEM model tackles four specific bottlenecks: collective communication volume, SM contention from data movement, activation memory, and workload skew. Each maps to a distinct technique, and together they move the training pipeline closer to ideal scaling.

5D Parallelism Matched to Network Topology

GEM’s hybrid architecture mixes dense and sparse parameters with very different communication profiles. The training system uses 5D parallelism: 2D FSDP combined with Expert Parallelism (EP) for dense weights, and a fully sharded 2D model-parallel scheme for sparse tables. The guiding rule is to match each collective’s message volume to the bandwidth available at the corresponding level of the network hierarchy.

The cluster has a three-tier topology: NVLink for the eight GPUs inside a host, RoCE within an AI zone, and oversubscribed RoCE between zones. When a collective saturates a tier, the design introduces a new parallelism dimension to shrink the message size or group count at that tier.

Dense Parameters: Three Dimensions, Two Tiers

The billions of dense parameters are sharded with FSDP, meaning weights are all-gathered before compute and gradients reduce-scattered after. Two extra dimensions improve bandwidth utilization. First, 2D FSDP splits the full job into two topology-aware groups: a shard group (typically 128–256 GPUs) handles the all-gather and reduce-scatter with high effective bandwidth, while a replica group handles gradient synchronization via all-reduce. Because weights are already sharded, the replica group messages are small enough to tolerate slower cross-zone links.

Second, large modules like DHEN experts expose communication that pipelining cannot hide. Adding EP assigns one expert per rank, cutting both the FSDP group size and its message size. The EP-related communication runs only over intra-node NVLink, where bandwidth is ample:

• Forward: FSDP all-gathers expert params (inter-node, 16-way) → EP all-gathers activations (intra-node NVLink, 2-way) → local expert compute → EP reduce-scatter of outputs.
• Backward: FSDP all-gather expert params (inter-node, 16-way) → EP all-gather of output gradients (intra-node NVLink, 2-way) → expert gradient compute → EP reduce-scatter of input gradients → FSDP reduce-scatter of parameter gradients.

Parameter all-gathers are aggressively prefetched so communication overlaps with the compute of the preceding module.

Sparse Parameters: From 1D to Fully Sharded 2D

The trillion-parameter embedding tables are too large to replicate, so they rely on model-parallel sharding and all-to-all communication for feature distribution. Three generations of parallelism brought sparse training to near-zero overhead.

Initial 1D model parallelism fails at scale for two reasons. First, shard distribution across thousands of ranks causes severe workload imbalance. Second, all-to-all collectives spanning all ranks degrade quickly, particularly across oversubscribed zones. The V2 design partitions ranks into smaller model-parallel groups (e.g., 256 GPUs) with data-parallel replicas, cutting latency and improving balance.

V2’s tradeoff is memory: each replica group holds a full copy of its assigned shard. The fully sharded 2D approach (V3) shards each replica’s parameter copy across its group, keeping only fractions local and reconstructing on demand:

• Forward: all-gather table shards → all-to-all feature distribution → embedding lookup → all-to-all embedding return.
• Backward: all-gather table shards → all-to-all gradient exchange → local update → reduce-scatter parameters.

The extra collectives travel over NVLink and overlap with dense compute via pipelining. Reconstructed copies are released before peak activation memory, making sparse scaling nearly overhead-free at training scale.

Removing Compute Kernels from Data Movement

Pipelining hides most communication latency, but the collectives themselves still occupy SMs. Kernels like all-gather and reduce-scatter can consume ~24 SMs that would otherwise run compute, and wave scheduling can amplify the loss beyond the raw occupancy hit. The measured cost reaches 15% efficiency under some conditions.

For data-only movements, Meta’s NCCLX extension to NCCL eliminates SM involvement entirely. Hardware copy engines handle the intra-node NVLink transfers and RDMA handles inter-node traffic. SM usage for an all-gather drops from 24 to 1, recovering ~23 SMs for compute kernels. This yields about a 5% end-to-end QPS gain at full scale.

For reduction-heavy collectives like all-reduce, NVLink SHARP shifts the reduction work off the SMs and onto network switch hardware.

Lowering the Activation Memory Bill

After sharding weights and optimizer state, activations dominate per-GPU memory. Two techniques shrink that footprint so training can use large local batches — up to 1,000+ samples — without paying full recompute costs.

PyTorch’s compiler-based activation checkpointing already decides per-node whether to recompute or save, but it applies one global memory budget. GEM instead schedules per-region budgets across the compiled subgraphs between graph breaks, directing memory to regions with the best recompute ROI. This pushes the memory-latency tradeoff beyond any uniform budget.

On top of that, activation quantization compresses the checkpointed tensors that AutoAC decides to stow. Saving intermediate activations at lower precision (e.g., BF16 to FP8/MX4) occurs at the boundary between forward and backward graphs, further cutting memory.

These techniques make large local batch sizes (1K+ samples) practical, which matters because small batches and heavy recomputation both drag down model FLOP utilization.

Rebalancing Jagged Workloads Without Cross-Rank Traffic

LLM training sidesteps load balancing by padding sequences to a fixed length. GEM’s recommendation sequences are naturally ragged, and padding would waste over half the compute. Jagged kernels avoid per-rank waste but make the remaining compute skew data-dependent and dynamic — the heaviest rank typically exceeds the average by roughly 15% every iteration.

Global rebalancing redistributes the shards every step, but the required collective communication cancels the gains. Local strategies avoid that cost but leave notable skew on the table.

The solution — Base Batch Shuffling — orders small sub-batches of 128 samples by total sequence length and interleaves them (heaviest paired with lightest) when assembling the full training batches of 1k+ samples per rank. This approximates the ideal global balance while demanding zero cross-rank communication.

BBS delivers a 4% efficiency gain on GEM training, composed of a 4% QPS improvement and 4% peak memory reduction. The maximum-over-average workload gap drops immediately when the technique is activated.

Co-Design as the Path Forward

The 2x efficiency improvement came from optimizing kernels, precision, parallelism, networking, and memory as one system rather than a set of independent layers. Training a foundation model at the intersection of LLMs and recommendation systems is a co-design problem, and future gains are expected to follow the same pattern — with agents automating parts of the optimization cycle to speed iteration. Scaling the GEM model further will depend on continuing this style of cross-stack co-design.

Engineering a 2x Efficiency Gain in Large-Scale Ads Training

Meta’s ads ranking models have grown to LLM scale—hundreds of billions of parameters—and training them pushes the limits of conventional practices. The team behind the GEM foundation model set out to double training efficiency, and their work hinges on a simple but stubborn observation: the training job was bottlenecked by embedding communication, not compute. Their fixes—both algorithmic and systems-level—collectively deliver roughly a 2x throughput improvement without sacrificing model quality.

The Bottleneck Was Communication, Not Compute

GEM is trained on a hybrid of two model families: a large embedding-backed ranking model and a generative retrieval model. Scaling it exposed a stark asymmetry. The embedding tables, which hold hundreds of billions of parameters, consume enormous memory, but the actual computation they require is modest. Standard dense-data-parallel sharding forces all of those parameters to sit in GPU memory and all of the associated gradients to be exchanged across the network. The result: high communication overhead and memory pressure that limit how much model capacity can fit on a single accelerator.

Meta’s solution was to grade the communication and compute needs of each parameter and treat them differently. Sparse embeddings that receive frequent updates but require little compute are sharded and replicated only where needed, cutting memory and network traffic. Dense layers, by contrast, remain in data parallelism with traditional all-reduce. The job orchestrates these heterogeneous sharding strategies across the cluster.

Pinpointing the Memory Wasters

To make room for denser models, the team audited memory usage during training. They found several sources of waste. Activation memory peaked during the embedding layers rather than the transformer stack. Gradient statistics—specifically the sum of squared gradients used by the Adam optimizer—were stored redundantly across replicas. And the practice of aligning buffers to 128 bytes, though invisible to the user, consumed a surprisingly large fraction of memory.

Their mitigations included fusing the embedding lookup and the first MLP layer into a single kernel, recomputing rather than storing intermediate activations, and avoiding full-buffer gradient copies by carefully managing momentum state. Taken together, these optimizations cut memory consumption by a substantial margin, freeing capacity for a larger model dimension.

The Sparse-Dense Tradeoff

One of the more consequential findings was the interplay between the embedding dimension (a model hyperparameter) and the training architecture. Because embeddings are stored in a distributed fashion, the cost of fetching them scales with the per-embedding dimension. Halving that dimension from 128 to 64 reduces communication cost dramatically—by about half—but it also trims model quality. GEM’s designers compensated by coupling the embedding dimension to the number of experts in the mixture-of-experts (MoE) layers and to the MLP’s hidden width. By shrinking the embedding while widening the dense MLP, they held quality constant while cutting the embedding-related communication in half.

They further introduced a meta-architecture knob to flex the tradeoff between compute and communication. When the gradient computation is synchronized for dense parameters while embedding communication runs asynchronously, the two can overlap, hiding the communication latency behind useful compute.

Efficient Sparse All-to-All With Token Grouping

Sparse communication between workers relies on an all-to-all operation. Naive all-to-all is inefficient when tokens cluster around a small number of “hot” IDs—common in real-world ads data. The fix was token grouping: within each training micro-batch, the sequential data is split into groups, and the system checks how likely tokens are to share the same gather or scatter destination. Then it shuffles or redistributes the tokens within a group so that each group’s data is more likely to head to the same worker. This converts many small messages into fewer, larger transfers. The result: a significant reduction in all-to-all time, linear speedup with more groups, and only a marginal increase in end-to-end latency due to the extra data shuffle.

Two more refinements rounded out the communication toolkit:

• Post-shuffle packing: Instead of issuing many separate all-to-all calls after shuffle, the system concatenates the post-shuffle buffers together, performs one bulk all-to-all for the combined buffer plus one small all-to-all for the IDs, and then unpacks on the receiving side. This reduces the number of operations by an order of magnitude.
• Payload splitting: rather than storing a large payload table in GPU high-bandwidth memory, the oversized payload is split into smaller shards that each memory tier can hold, pulling them together only when needed.

Nearly 2x End-to-End Gain

The cumulative effect of these changes is close to a 2x end-to-end training throughput improvement over the baseline, measured in terms of large-batch, long-sequence training steps per second. The win comes from four quarters: reduced embedding communication, fewer wasted memory buffers, the ability to fit larger model dimension per accelerator, and more efficient network use through token grouping. This matters because, at this scale, even a 1% throughput gain translates into enormous GPU-hours saved—and Meta now has a playbook for training the next generation of retrieval and ranking models without letting communication overhead dictate the model’s shape.

The complete system design, including the attention across embeddings, the multi-granularity model architecture, and the full training recipe, is described in the team’s paper, “GEM: A Foundation Model for Retrieval and Ranking at Meta.”

Acknowledgements

The authors thank Tianshu Peng, Jiasheng Zhang, Angel Yang, Rikin Shah, Ke Sang, Kevin Tang, Pawel Kadluczka, Jacky Zhou, Han Xu, Enes Palaz, Hao Yan, Jake Siso, Rupert Wu, Liangbei Xu, Yusuo Hu, Serena Liu, Hongtao Yu, Bor-Yiing Su, Santosh Mohan, Min Si, Shali Jiang, Laming Chen, Boyang Liu, Qinghai Zhou, Xiaozhen Xia, Jason Rudy, Jiayi Xu, Dan Chanpuriya, Justin Yang, Mandeep Chadha, Carmen Au, Hairong Kuang, Subodh Iyengar, Balaji Balasubramanian, Anamaya Sullerey, Viral Vimawala, Saket Gur, May Wang, Vibha Sinha, Rustam Hashimov, Ernest Wang, Max Leung, Shuo Chang, Musharaf Sultan, Oana Platon, Jade Nie, Eric Falconer, Ping Chen, Damian Reeves, Xian Chen, Ellie Wen, Chonglin Sun, GP Musumeci, Reva Srinivasan, Brian Hansen, Vivienne Sung, Patrick Phelps, Paolo Massimi, Jie Zheng, Anuj Madan, Nikhil Garg, Xiaorui Gan, John Bocharov, Ritwik Tewari, Wenlin Chen, Rocky Liu, Tak Yan, Santanu Kolay, Sandeep Pandey, Matt Steiner, and the entire v-team behind training Meta’s largest ads recommendation workloads at scale and efficiently.