AI Flame Graphs: Profiling the Accelerator, Not Just the CPU
AI workloads are expensive to run, and profiling them has historically been difficult. Intel has been developing a tool called AI Flame Graphs to help developers see exactly where time and resources go—from the CPU code that launches a workload down to the individual instructions executing on the AI accelerator or GPU. The first version is now available as a preview in the Intel Tiber AI Cloud for the Intel Data Center GPU Max Series.
The visualization extends the classic CPU flame graph concept to cover the full software stack plus the hardware profile of the accelerator itself. In practice, this gives developers a single view of the entire path their code takes, from host-side frameworks and runtimes down to the kernel operations on the GPU.
What the Flame Graph Shows
In the example above, a SYCL matrix multiply microbenchmark runs three different implementations. The colors distinguish layers: green frames represent actual instructions running on the accelerator, aqua frames map those instructions back to source code, and red, yellow, and orange frames show the CPU-side code paths that launched the accelerator work. The gray dash frames just mark the boundary between CPU and accelerator code. The x-axis is proportional to cost, so the widest frames are the best targets for optimization.
Looking at the output, multiply_basic()—the unoptimized implementation—dominates the graph at 72% of samples. Adding a local variable accessor brings that down to 21% for multiply_local_access(), and adding matrix tiling on top of that drops it further to 6%. The towers shrink as optimizations stack up, which is exactly the kind of quick read these graphs are meant to give.
Profiling Approach: EU Stalls and eBPF
The profiler combines two techniques: Intel EU stall profiling for hardware sampling on the Data Center GPU Max Series, and eBPF for software instrumentation of the CPU side. The design goal is that generating a flame graph should be as easy and low-overhead as running a standard CPU profiler—no application restarts, no interposer libraries, no code changes. Just profile an existing workload and get a visualization.
It is worth noting that EU stall samples measure time when the hardware is not making forward progress. This focuses attention on the true pain points in the code, but it does not show resource usage from instructions that retire without stalling. Timer-based sampling would give a different view, and the team hopes to support that in future versions.
Why This Is Harder Than It Looks
The work started as an internal Intel project about a year ago. The team found that while other AI profilers exist, they typically trace CPU stacks and time accelerator segments without looking at instruction offsets running on the accelerator itself. Some do profile accelerator instructions, but only through high-overhead binary instrumentation, which kills the "use it daily" use case.
There were mature options for building a harder-to-use, higher-overhead version months ago, for example using Intel GTPin for binary instrumentation. But for widespread adoption the tool needs negligible overhead and ease of use so developers will actually run it regularly and include it in deployment pipelines.
The engineering challenge goes deeper than just sampling the right hardware counter. AI accelerator programs often do not exist in the file system. Once loaded, they can be deallocated from main memory entirely and only reside in accelerator memory, beyond the reach of standard OS debuggers and profilers. There is no /proc representation for these programs, so tools like ps(1) cannot see them. Getting stack traces and symbols to work requires understanding each runtime, framework, language, and driver involved. PyTorch, for example, requires considerably more work to get clean CPU stacks than simpler frameworks.
On the plus side, some interfaces do exist. Kernel drivers and runtimes offer debug hooks to list running accelerator programs (as intel_gpu_top(1) uses), kill hung workloads via intel_gpu_abrt(1), and even generate basic ELF files that can be inspected with gdb(1). But overall, the team describes OS-style debugging for GPUs and AI accelerators as being roughly "two years old"—usable but early, with a lot of refinement still needed.
Searching Within the Graph
Flame graphs include a built-in search feature. Matching terms from a search (via Ctrl-F or clicking "Search") are highlighted in magenta on the graph, and the bottom-right corner shows the percentage of sampled stacks that contain the term. This is a simple way to quantify how often a given instruction, source path, or function name appears across the profile. In one example, searching for sbid (software scoreboard dependency) reveals it appears in 78.4% of samples, pinpointing the stall as the dominant bottleneck.
Interpreting the Visualization Takes Practice
Feedback from Intel developers seeing the tool for the first time is that it can be disorienting. The graph exposes entire layers of the stack that most AI developers never interact with directly: runtime internals, kernel logic, user-mode drivers. Their own code may appear as only a small tower in a sea of unfamiliar frames.
This mirrors early reactions to CPU flame graphs, which similarly surfaced system libraries and kernel internals most application developers never see. The payoff comes from learning to read a dozen or so functions that dominate the profile. That takes a few hours to a few days, but it is time that "well spent" when it leads to double-digit percentage cost reductions. Experienced users eventually read a flame graph in under a minute: find the widest tower, zoom in, identify the responsible function, done.
The team expects AI developers to get there too. Publishing case studies of performance wins from AI Flame Graphs should help build the mental model for what these visualizations show and how to act on them.
With CPU flame graphs seeing over 80 implementations and mainstream adoption, the ambition behind AI Flame Graphs is to bring that same level of easy, daily profiling to AI development. There is no simpler way to show a developer exactly what is slowing down their accelerator code, and that is the first step to fixing it.
PyTorch Support Arrives
PyTorch posed a steeper challenge because it required walking Python stacks. Prior work on CPU profiling provided a foundation, and after considerable effort, the first PyTorch AI Flame Graph is now available. In the visualization, PyTorch frames appear in pink at the bottom of the stack:
The example runs the Llama 2 7B model using the Intel Extensions for PyTorch (IPEX). The trace shows GPU kernel execution back to Python source, with most samples leading to a gemm_kernel matrix multiply shown in aqua. That kernel exhibits many stalls from software scoreboarding, similar to the earlier example. The JIT-generated oneDNN kernels lack a source path, so that layer simply reads "jit". Two instructions (0xa30 and 0xa90) together account for 27% of the entire profile. Getting every layer included was difficult but marks an important milestone: if PyTorch can be handled, other frameworks should follow.
Current State: Works, With Caveats
Some AI workloads profile easily; others require moderate overhead and significant setup. Like CPU profiling, difficulties vary by workload and language. Many AI software dependencies need tweaks and recompilation—for example, enabling frame pointers for stack walking—which makes preparation time-consuming. PyTorch is the hardest case, potentially requiring over a week of OS-level work before it can be flame-graphed. The necessary changes need to be pushed upstream across multiple repositories, involving teams inside and outside Intel, a process expected to take at least a year. During that period, AI workloads should gradually become easier to profile and the overhead will drop.
This mirrors the early days of eBPF, which once required patching and recompiling the kernel, LLVM, and Clang—a process that could consume days when errors occurred. Those patches have since been merged upstream and defaults changed so eBPF "just works." AI Flame Graphs are still in their early stage, but the improvements they require are fundamentally about better debugging support, which is also a prerequisite for the "Fast by Friday" vision of root-causing any issue in five days or less.
Release and Outlook
AI Flame Graphs will first appear as a preview feature on the Intel Tiber AI Cloud for the Intel Data Center GPU Max Series. Current users can request early access through Intel's service channel. Support for other hardware, inclusion in other Intel products, official launch details, and open-sourcing decisions rest with other Intel teams, who will make their own announcements.
The potential payoff is significant: small efficiency gains across AI data centers could translate into substantial savings in electricity, water, and cost. If AI Flame Graphs prove as successful as CPU flame graphs have been, improvements exceeding 10% should become routine, with 50% or greater eventually achievable. The early period will be difficult, with many software components to tune and recompile, but the long-term trajectory is clear. Other implementations of AI flame graphs will likely emerge, possibly even as commercial products. If those use higher-overhead, harder-to-use technologies, they risk discouraging adoption and preventing the field from capturing much-needed wins. The technology is too important to ship poorly—AI flame graphs should be easy to use, production-safe, low-overhead, and reveal the full stack. Intel has demonstrated that this is achievable.
Personal predictions here are not guarantees of specific performance improvements; official publication and public launch from Intel will provide firmer ground.
