GPU Profiling Meets Doom: Full-Stack Flame Graphs

Intel's iaprof, the open-source profiler that powers AI Flame Graphs, has added support for Intel Battlemage GPUs. That makes it possible to generate full-stack GPU flame graphs—showing not just GPU instructions but the CPU code paths that spawned them—for gaming workloads. Combined with FlameScope, Brendan Gregg's heatmap-based profile viewer, you can correlate CPU and GPU activity at subsecond resolution to spot exactly where bottlenecks form and why.

Gregg's demonstration target is GZDoom, an open-source Doom engine. Side-by-side FlameScope views for CPU and GPU utilization reveal the kind of correlation that's normally painful to dig out of raw profiles: gaps in GPU activity line up precisely with heavy CPU periods, and vice versa.

Reading the CPU and GPU Pictures

FlameScope breaks a profile into a heatmap where each column is one second, built from 50 x 20ms samples. Selecting a region isolates the flame graph for that time range. A red stripe in the CPU heatmap resolves into roughly 180 ms of shader compilation, dominated by NIR optimization passes—the intermediate representation Mesa uses internally—and GPU shader compilation itself.

The interactive CPU flame graph makes the usual advice concrete: look for the widest towers, optimize those first.

GPU flame graphs layer on extra context. The GPU FlameScope view lets you drill into a particularly busy stretch, such as Doom's "room 3," packed with hundreds of enemies. In the resulting flame graph, green frames are actual GPU instructions, aqua frames show the source functions for those instructions, and red (C) and yellow (C++) frames expose the CPU code that initiated the GPU work. Gray "-" frames mark the CPU/GPU boundary. Cost is proportional to width, so the widest segments are where optimization effort pays off most.

The interactive GPU flame graph breaks the workload into four stall categories: rendering walls (41.4%), postprocessing effects (35.7%), stenciling (17.2%), and sprites (4.95%). The CPU stacks below each segment identify the individual shaders causing stalls and the stall reasons.

Making GZDoom Hard Enough to Profile

GZDoom is open source and Linux-native—both prerequisites, since the profiler doesn't yet support Windows. But the Intel Battlemage GPU handles the classic maps with such ease that a stall-based profile yields almost no samples. Rather than switch games, Gregg and colleagues made GZDoom harder: they built custom maps with Slade and tuned Battlemage resource limits to amplify utilization. The test map has an empty starting room, a corridor of torches, and an open third room with a large skybox full of enemies and Sergeants that spawn continuously. Walking between rooms produces distinct workload signatures to compare.

Getting iaprof Set Up: Nightmare Difficulty

If you're on Intel's Tiber AI Cloud, the software stack is preconfigured—everything works out of the box, with support for the Intel Max Series GPU. Off the cloud, Battlemage support demands real effort:

  • A Linux system with root access for eBPF and Intel's eustall interfaces.
  • A recent kernel with matching Intel drivers: Linux 6.15+ with the Xe driver for Battlemage; Linux 5.15 with i915 for the Max Series.
  • A kernel built with Intel's driver-specific eustall and eudebug interfaces, some of which are upstreamed and others still in progress. These are enabled by default on the Tiber AI Cloud.
  • Frame pointers compiled into every library and application profiled—glibc, GZDoom itself, Intel's oneAPI and graphics libraries. Recent Fedora and Ubuntu 24.04 LTS ship system libraries with frame pointers by default, but any dependency without them shows up as shallow, stubby stacks one or two frames deep.

Custom kernels and library recompilation make this a serious project. Once it's working, profiling is a single command modeled on Linux perf:

git clone --recursive https://github.com/intel/iaprof
cd iaprof
make deps
make
sudo iaprof record > profile.txt
cat profile.txt | iaprof flame > flame.svg

What's Still Ahead

The inevitable question about NVIDIA support has an answer: Nsight Graphics offers GPU flame graphs, but they're shallow (GPU code only), require an interposer, and are onerous for continuous profiling. This eustall-based approach, by contrast, aims for the same always-on, everything-visible behavior CPU profilers provide. Future work includes binary releases on GitHub, broader hardware support, and reducing overhead from current levels to the stated target of under 5%, particularly on the i915 driver.

What the Doom Test Shows

The same tooling that was developed for AI workloads turns out to be just as useful for a very different kind of graphics-intensive task: gaming. Testing with GZDoom on Intel's new Battlemage hardware produced a unified view of CPU and GPU activity at millisecond resolution. The resulting flame scope heat maps show visual patterns that can be selected to drill down into flame graphs and identify the exact code responsible for any observed behavior.

In the GZDoom session, GPU pauses were traced back to the corresponding CPU burst. Selecting the relevant section in the heat map and reading the flame graph explained why the GPU stalled at that moment. The same technique exposes GPU code usage for arbitrary time windows, not just isolated incidents.

Getting the Tools Running

The flame graph tools have been open sourced, but running them is not a plug-and-play experience. You need Intel hardware and a willingness to tinker with Linux kernel and library configuration. For some, actually playing Doom on Nightmare! difficulty might be the easier path. The setup friction is expected to decrease over time as the project matures.

In the meantime, the hope is that enough people work through the installation process to validate the tools in more environments and surface new performance problems the authors haven't encountered. The contribution list for this work includes Brendan Gregg, Ben Olson, Brandon Kammerdiener, and Gabriel Muñoz.