KernelEvolve: How Meta’s Ranking Engineer Agent Optimizes AI Infrastructure

Kernel Growth Outpaces Manual Tuning

Meta's AI infrastructure spans NVIDIA GPUs, AMD GPUs, its custom MTIA silicon, and CPUs. Every model layer ultimately depends on optimized kernels — small programs that translate high-level operations into chip-specific instructions. The number of kernel configurations that must be written, tested, and maintained scales as the product of hardware types and generations, model architectures, and operator types. That product now runs into the thousands, and hand-tuning by kernel experts no longer keeps pace.

To close the gap, Meta built KernelEvolve, an agentic system that autonomously generates and optimizes production-grade kernels across that heterogeneous fleet. KernelEvolve is used by Meta's Ranking Engineer Agent and generalizes to AI workloads beyond Ads Ranking. In production, it optimizes code serving trillions of daily inference requests.

Why Kernels Multiply Faster Than Experts Can Tune Them

Hardware Heterogeneity

Meta's accelerator fleet spans NVIDIA GPUs, AMD GPUs, and MTIA silicon, each with different memory architectures, instruction sets, and execution models. A kernel optimal on one platform may underperform or fail on another. Even within a single hardware family, successive generations introduce architectural changes requiring different optimization strategies. Meta's MTIA roadmap has spanned four chip generations in two years (MTIA 300 through 500), each with new compute capabilities, memory bandwidth characteristics, and numeric data types. A kernel tuned for one generation will lag on the next.

Model Architecture Variation

Meta's recommendation models have evolved from early embedding-based deep learning models to sequence learning models with attention mechanisms, to the Generative Ads Recommendation Model (GEM), and most recently the Meta Adaptive Ranking Model, which brings LLM scale to ads. Each generation introduces operator types the previous one never needed. Meanwhile, production serves fundamentally different model families simultaneously, and a single ads request may traverse several families in one call. Every new architecture extends the matrix of operators that must be optimized across all hardware.

Kernel Diversity Beyond Vendor Libraries

Vendor libraries such as cuBLAS and cuDNN cover common operations like GEMMs, convolutions, and standard activations, but even those resist one-size-fits-all solutions. Matrix multiplication behaves differently across training versus inference, and tensor shapes vary widely across ranking stages and models. That combinatorial space is too large for human experts or today's compiler-based autotuning and fusion to fully cover.

Production workloads are also dominated by a long tail of operators outside library coverage: data preprocessing transforms like feature hashing, bucketing, and sequence truncation, plus custom model operators like fused feature interaction layers and specialized attention variants unique to Meta's architectures. Without native accelerator implementations, these operators either fall back to CPU — forcing disaggregated serving architectures with significant latency overhead — or run via unoptimized code paths that underutilize hardware. The problem compounds with hardware diversity: a hand-tuned NVIDIA kernel cannot simply be recompiled for AMD GPUs or MTIA.

Search, Not One-Shot Generation

Unlike typical LLM-based agents that perform one-shot code generation, KernelEvolve treats kernel optimization as a search problem. It explores hundreds of alternative kernel implementations to identify a solution that often matches or exceeds human expert performance, and does so in hours instead of weeks.

The result: over 60% inference throughput improvement for the Andromeda Ads model on NVIDIA GPUs, and over 25% training throughput improvement for an ads model on Meta's custom MTIA silicon. KernelEvolve generates kernels in high-level DSLs like Triton, Cute DSL, and FlyDSL, as well as low-level languages including CUDA, HIP, and MTIA C++.

This represents a shift in how AI software and hardware relate: where kernel development was once a manual, expert-driven process struggling to keep pace with hardware and model evolution, KernelEvolve makes it continuous and automated. As Meta diversifies its AI hardware portfolio, the ability to rapidly generate optimized kernels for new chips substantially reduces the engineering effort required to integrate heterogeneous hardware for training and inference.

Structured Search in Place of Autocomplete

KernelEvolve diverges from the typical AI coding-assistant pattern. Instead of prompting an LLM for one kernel and testing it, the system models optimization as a structured search across the space of possible implementations. A dedicated long-running job harness drives each iteration—compiling candidates, validating correctness and speed, profiling hardware usage, and producing analysis reports—while contending with the multi-minute build times and infrastructure failures that make naive approaches unusable.

The Synthesizer and Its Dynamic Prompts

The LLM synthesizer generates candidates in a mix of languages and targets, spanning high-level DSLs like Triton, TLX, CuTe DSL, and FlyDSL, plus low-level backends including CUDA, HIP, and MTIA C++.

Rather than relying on static prompts, it uses dynamic, context-aware prompts continuously enriched with runtime diagnostics, hardware constraints, and the performance history of prior candidates. This collapses the conventional set of separate templates—one for debugging, another for tuning, another for verification—into a single adaptive interface that drives a feedback-oriented optimization loop.

Tree Search with Memory Operators

The exploration of the optimization space relies on graph-based search algorithms such as Monte Carlo tree search and evolutionary strategies. Each kernel candidate is a node in the search tree; the engine selects promising nodes, applies transformations, evaluates, and then decides between deeper exploration and backtracking, balancing exploitation of known-good paths with exploration of new ones.

Nodes do not evolve independently. Each one carries a configurable memory operator that governs how it pulls context from the tree when generating the next candidate. A node may inherit its parent's optimization trajectory to refine a promising direction, compare itself to siblings to learn what separates high performers, combine parent and sibling histories, or reset to a blank slate to escape local optima. This selective memory means sibling nodes can collaborate by surfacing complementary strategies, parent-child chains preserve and deepen fruitful paths, and memory-free restarts inject diversity when progress stalls.

Retrieval-Augmented Knowledge and Growth

To write optimized code for hardware the LLM was never trained on, KernelEvolve keeps a hierarchical knowledge base with three tiers: correctness constraints for valid implementations, platform-agnostic optimization guidance on debugging and tuning, and hardware-specific documentation with architectural details for each accelerator. The system retrieves relevant knowledge on demand, driven by runtime signals. A memory bandwidth bottleneck, for instance, triggers retrieval of memory hierarchy documentation; a compilation error activates debugging guidance.

The knowledge base is not static. As the system solves novel problems, it distills successful strategies into reusable skills—compact optimization patterns and debugging heuristics—which are written back into storage. This self-evolving library functions as in-context reinforcement learning: each successful run enriches what future sessions can draw upon, accelerating similar problem solving without any model retraining.

Evaluation Beyond a Single Number

Every generated kernel goes through a validation pipeline checking both bitwise correctness against references and performance. Critically, assessment goes beyond a lone runtime figure, using a stack of profiling tools at different levels of analysis. TritonBench verifies numerical accuracy against PyTorch baselines and measures end-to-end speedup on production input shapes. PyTorch Profiler captures system-level timelines, including launch overhead and host-device synchronization. On GPUs, NCU supplies kernel-level hardware metrics—occupancy, memory throughput, instruction mix—while Proton provides intra-kernel instruction-level latency and pipeline behavior. For MTIA targets, MTIA Insight offers accelerator-specific counters: PE utilization, fixed-function engine metrics (DPE, SFU, MLU utilization and stall cycles), cache behavior, and per-PE memory bandwidth.

These tools are unified through a compiler-centric abstraction rather than run as standalone steps. The framework composes analysis via job graphs: compiler transforms insert MLIR-level instrumentation, profiling passes collect metrics, and trace synthesis produces structured output. The search engine therefore sees not just "kernel A is 1.2x faster than B" but the reason—memory-bound, compute-bound, or occupancy-limited—and feeds that diagnostic signal back to the synthesizer for the next round.

Shared Data and Agentic Reinforcement

Each session feeds a shared data foundation. When one engineer's exploration discovers an effective tiling strategy for a class of operators, that insight becomes available to every future session targeting similar workloads. Early adopters handle the hardest exploration; later users start closer to optimal and refine from there.

Sessions also produce a natural byproduct: structured agentic trajectories that capture reasoning, code transformations, and evaluation feedback behind winning kernels. This domain-specific data is rare and valuable, encoding optimization intuition absent from public datasets. Meta uses these trajectories to post-train smaller, specialized models via agentic reinforcement learning, rewarding improvements measured directly by kernel performance. Over repeated iterations, the cycle compounds—better models produce better kernels in fewer reasoning tokens and search steps, generating higher-quality data that in turn yields compact models capable of running cost-effectively while approaching the optimization ability of much larger systems.

Bringing Proprietary Silicon Into Reach

A consequential feature of this design is its capacity to generate optimized code for hardware absent from any public training set. Meta's MTIA chips pose this exact problem: no public LLM has seen MTIA code, documentation, instruction sets, or idioms.

KernelEvolve handles this by systematic knowledge injection. MTIA-specific documents—architecture manuals, instruction set references, memory hierarchy specs, and optimization patterns—are encoded into the retrieval-augmented knowledge base. When targeting MTIA, the system retrieves this proprietary context and folds it into its reasoning, effectively learning the hardware in real time.

For any new accelerator, the cost shifts accordingly: instead of handwriting thousands of kernels, engineers curate hardware documents and inject them. The system then autonomously generates optimized kernels, aligning software readiness with hardware deployment rather than manual engineering schedules.

Measured Results and Reach

KernelEvolve reports success across standardized suites and production loads alike. On KernelBench—Stanford's 250-problem kernel optimization benchmark with three difficulty levels—it posts a 100% pass rate, with every generated kernel both correct and faster than the PyTorch reference. It also validates 160 PyTorch ATen operators with 100% correctness across three platforms (480 total configurations).

In production on Meta's MTIA, generated kernels—spanning compute-bound, memory-bound, and custom operations—delivered over 25% training throughput improvement on an ads model. On NVIDIA GPUs, the output yielded more than 60% inference throughput improvement over a model already running torch.compile and vendor libraries. All of this comes from one framework covering NVIDIA GPUs, AMD GPUs, MTIA, and CPUs; adaptability comes from dynamic retrieval of hardware-specific constraints rather than per-platform prompt engineering.

Development velocity changes too. Kernel work that took specialists weeks—profiling, iterating on tiling, debugging hardware edge cases—now completes in hours, shifting engineer effort toward model architecture, training techniques, and optimization objectives.

Anatomy of an Optimization Session

The workflow given a target operator, platform, and performance goal is straightforward:

1. Retrieves relevant hardware documentation and optimization knowledge.
2. Generates an initial set of candidates with context-aware prompting.
3. Evaluates each for correctness and performance on distributed infrastructure.
4. Feeds results to the search engine, which selects promising leads for further optimization.
5. Iterates until targets are met, the search budget expires, or progress plateaus.
6. Outputs the best validated kernel, ready for deployment.

The process runs on Meta's distributed infrastructure, evaluating thousands of candidates in parallel. Persistent search trees and prior implementations let the system build on earlier results when addressing new model variants or hardware generations.

From Kernels to a Larger Agentic Vision

The mechanisms behind KernelEvolve—structured reasoning, retrieval-augmented knowledge, closed-loop evaluation—map onto hybrid model search, compiler optimization, memory management, and system configuration. The project represents an early piece of the broader Ranking Engineer Agent vision: an autonomous system that continuously optimizes its own performance-critical infrastructure.

Within REA, ML Exploration finds better models while KernelEvolve makes them production-ready. Together they compress the time between ranking improvements and their availability to advertisers. The paper, "KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta," published from ISCA 2026, details the full architecture.

From Kernel Tuning to a Self-Improving Agent

Meta’s infrastructure teams oversee one of the largest collections of PyTorch models in production. For years, discovering the optimal CUDA kernel configurations for those models was a manual, trial-and-error process that demanded deep systems expertise and consumed hours of engineers’ time. KernelEvolve changes that: it is a ranking-based engineer agent that treats kernel configuration search as a recommendation problem, learning from past tuning data to propose better configurations for new models — and getting faster with every deployment.

Reframing Search as a Ranking Problem

Traditional autotuning methods treat each configuration search as a fresh optimization problem, typically starting from an uninformative prior. KernelEvolve instead uses a learning-to-rank (LTR) approach. It builds a model from historical kernel tuning results — what configurations worked, and how well they performed — for previously seen PyTorch models. When a new model arrives, KernelEvolve applies that model not to predict runtimes directly (which is noisy), but to rank candidate configurations by their expected performance.

This ranking formulation is key. The costs of ties and near-ties in runtime are negligible, meaning small prediction errors don’t harm the result as long as the relative order is roughly correct. The model, a LambdaRank-style neural network, learns to query for and rank the best kernel configurations from the historical database.

How KernelEvolve Works in Production

Once a PyTorch model is selected for optimization, the system generates feature vectors by compiling the model and extracting representation vectors from it — from operator types, dimensions, and the execution graph structure. Those vectors search the database for the most similar historical models, whose configurations seed the initial candidate pool. The LTR model then ranks these candidates, and KernelEvolve modifies and runs the top-ranked ones on GPU, collecting real warm-up runtimes before presenting the best result.

Real-world LLM services run many distinct models updated over time, and KernelEvolve was designed for exactly that workflow. In offline testing using production training jobs as tuning targets, it was twice as efficient as untuned baselines while using only a single GPU, a 99.4% reduction in compute over typical evolutionary search approaches. When deployed live against production inference models, the LTR model showed an 18.9% quality improvement over Meta’s prior approach in finding the best kernel configuration quickly.

Optimizing the Optimizer

Much of KernelEvolve’s real-world performance is hidden in the choices around how to control the search space and compute. Three decisions matter:

• Dynamic token masks. Ranking token masks represent the actual cumulative tensor shapes at runtime. Because large batches can make profile runs expensive, KernelEvolve uses “mini-batch” token masks for candidate ranking and validates top candidates with the real production mask. Tests found mini-batch pooling improved ranking performance by 12% over static masks and 5% over actual runtime masks.
• Real-shot measurement design. To avoid polluting the human baseline with noise from slow machine states, a baseline model is re-run in the same GPU context as the test model. KernelEvolve empirically found that 100-shot measurements were the highest-reward point: 200 shots delayed convergence with no quality gain.
• Spot-check pruning of remote workers. Distributed GPU workers occasionally go slow due to host-level issues. A dedicated spot-checking loop detects and removes unproductive remote workers entirely, and a quality gate in the probe verifies the measured baseline and configuration behavior before human evaluation occurs.

Training and Serving Infrastructure

KernelEvolve’s trainer periodically generates fresh ranking model checkpoints, with continuous rollback monitoring and sync to a training system that adaptively reschedules jobs. The serving side is deliberately lightweight: a single autoscaled deployment checks for new model updates, enqueues generation work, and pre-compiles results to reduce cold-start times. Daily dashboard monitoring watches for prolonged staleness of models and suboptimal configurations, surfacing any regressions.

The system also emphasizes broad participation: curated configuration spaces are defined by teams across Meta, standardization has let many product teams and vendor engagements benefit from the tool, and each new optimization result feeds back into the learning database.

Organic Discoveries That Shaped the Tool

KernelEvolve has yet to surface a Pareto-optimal configuration in multi-GPU environments, yet human reviewers always pick its top suggestions. Many of its most useful behaviors were not planned but discovered.

Inference kernels, trained and used under heavier loads, tended to carry large token masks that led to CUDA errors during ranking — resolved by falling back to training masks. Enabling earlier targets for training halves the search time compared to the original resource optimizer, so search was aligned to begin with those. And sampling configurations from the original model that targeted training also improved instance-level quality in subsequent analyses.

More Successful Custom Searches

The LTR engine was initially validated against training workloads. Meta later applied it to generative inference Text (GIT) and ranking for search and ads (SCA). In both, KernelEvolve was the mechanism that surfaced better configurations, frequently through changes like reordering the heads of attention matrices. On SCA models, dedicated LTR and calibration kernels reduced GPU and CPU usage by 2% and 5.1%, respectively.

Custom search runs on multi-GPU models unveiled correctable inefficiencies. One model had per-op dispatches to single GPU buckets despite multi-GPU availability. Another, a Mixture of Experts (MoE) model, had a sub-optimal gate communication pattern. The issue lay in dispatching_dense_to_sparse_moe_v3 and core_moe_module_forward, where wrongly setting the learnable gate’s needs_input_grad triggered unnecessary gather operations, dropping overall resource utilization. Straightforward fixes in the custom code resolved them, gaining the graph utilization and freeing DSP memory for future ops.

Looking Ahead

The team is working on end-to-end fused pattern construction to broaden KernelEvolve’s reach beyond CUDA kernel configurations, optimizing API call sequences on top of these kernels. Exploration of spectrum-informed analysis is helping explain inexplicable failures and better represent resource contention with surrounding tensor traffic in multi-tenant environments. Random search is also being re-introduced as a low-risk background modifier to escape precision learning plateaus, focused only on finding quality boosts without exposing upper-range variance to users.