The Return of the Frame Pointers

Why Your Flame Graphs May Be Lying to You

Profiler and debugger failures are often subtle. A flame graph might show a stack trace that looks plausible at first glance, but on closer inspection, a significant portion of samples are grouped under "[unknown]" and appear disconnected from the rest of the graph. This is a classic symptom of a system compiled without frame pointers.

Consider a CPU flame graph with about 15% of samples sitting in a stack tower above the label "[unknown]" on the left edge. At a glance, the graph appears normal, but those samples are actually incomplete—they are missing application frames because the stack walker stopped dead at the libc layer. Here's what happened: the profiler interrupted the kernel during execution, walked down through kernel frames like vfs* and ext*, crossed the syscall boundary via sys_write() into userspace, and reached the libc syscall wrapper __GI___libc_write(). At that point, it tried to resolve the next frame's symbol and failed, recording "[unknown]" instead.

The failure occurs because the compiler optimization "-fomit-frame-pointer" frees up the frame pointer register for data storage. The profiler still treats that register as a frame pointer and tries to match its value to a function symbol. Since the value is no longer a valid frame address but essentially arbitrary data, the walk halts before reaching the application frames above. In unlucky cases where the random value happens to be a valid pointer, the profiler may add a bogus frame, or even loop infinitely if the data points to itself, producing a tower of junk frames until the maximum frame limit is hit.

Other profiling techniques suffer even more. Off-CPU flame graphs, which are dominated by libc read/write and mutex calls, can be rendered nearly useless without frame pointers. If your application itself is also compiled without them, the problem extends to every single stack trace.

The good news: Fedora and Ubuntu are shipping versions compiled with frame pointers enabled by default. This fixes not only flame graph accuracy but also makes off-CPU analysis far more practical, and eases adoption of continuous profiling tools.

Frame Pointers and the x86-64 ABI

The x86-64 ABI specification defines how the %rbp register—the "base pointer"—can be used to navigate a call stack. External profilers and debuggers, including Linux perf and eBPF tools, rely on this mechanism to walk stack traces and visualize them as flame graphs.

However, the ABI explicitly states this usage is optional. A footnote notes that %rsp (the stack pointer) can index the stack frame instead, saving two instructions in the function prologue and epilogue while freeing up %rbp as a general-purpose register. Compilers took that option, and the result was years of broken stack walking.

The 2004 Fallout

In 2004, gcc's Roger Sayle changed the i386 backend to default to "-fomit-frame-pointer -ffixed-ebp" for 32-bit targets. The motivation was clear: 32-bit processors have only four general-purpose registers, so freeing %ebp was a meaningful win—a roughly 20% increase in register count. There was also a desire to beat Intel's icc compiler, and a belief that contemporary debuggers supported other stack-walking techniques anyway.

The problem: the change was applied to x86-64 as well, which already had over a dozen registers and gained far less from reclaiming one more. Within months, complaints surfaced. Eric Schrock wrote in late 2004 that on amd64, adding a 17th general-purpose register "isn't going to open up a whole new world of compiler optimizations"—you're just saving a few instructions that are already highly optimized on x86—and the cost in loss of debuggability is steep. He warned that the real trouble starts "when people start compiling /usr/bin/ without frame pointers," and that's exactly what happened: not just /usr/bin but also libraries and user applications followed suit. When that all happened, profilers that once worked without OS or runtime configuration broke.

Netflix and the Java Problem

When Brendan Gregg arrived at Netflix in 2014, Java's lack of frame pointer support broke application stacks entirely. The result was a Java flame graph and a system without proper visibility for application code. A fix for the JVM's c2 compiler followed, which Oracle reworked and released as the -XX:+PreserveFramePointer option in JDK 8u60.

Solving the Java issue exposed deeper problems: libc was still breaking a portion of samples, and most off-CPU stacks were effectively unusable. The interim solution involved compiling a custom libc with frame pointers for production use, then working with Canonical to provide a prebuilt variant. For a time, Canonical's libc6-prof package—libc6 with frame pointers—served this need.

The Overhead Question

Production rollouts of frame-pointer-enabled libraries have produced overhead numbers ranging from less than 1% in most cases to roughly 10% in extreme workloads. That 10% outlier deserves some nuance. One such case was an unusual application that generated stack traces over a thousand frames deep, so extreme that it broke Linux's perf profiler. The fix required adding the kernel.perf_event_max_stack sysctl. But this instance also ran on a virtual machine lacking low-level hardware profiling, so a clean confirmation that all of the overhead came from frame pointer instructions alone wasn't possible.

Microbenchmarks can hit the 10% mark as well. That often comes down to a small hot function whose performance hinges on fitting into L1 cache. Adding just a few instructions—any instructions—is enough to push the code over a cache line or affect instruction cache warmth. A well-designed experiment would compare the frame-pointer build against a build without frame pointers, then replay the added prologue and epilogue as inline NOP instructions in the non-frame-pointer build to isolate the instruction-cost effect. If a similarly sized performance drop appears, the cause is cache behavior rather than frame pointers per se—the same "straw that broke the camel's back" type of effect. Removing frame pointers fixes the immediate problem, but so would shrinking any other code in the hot path.

One other reported case involved the Python benchmark scimark_sparse_mat_mult, also near 10% overhead. It traced back to a quirk in a single large function where gcc chose %rbp-relative offsets instead of %rsp-relative ones, inflating code size. The issue was analyzed and allegedly fixed upstream. As a result, the typical 1% to 2% cost—often less than 1%—across real-world workloads is wildly outweighed by the performance wins to be found via profiling, which can range from 5% to 500% improvements. The cost is even easy to justify in most enterprise settings; the minor expense of true observability is trivial compared to the speedups it reveals.

Still, it's worth remembering the trade-off isn't universal. Not every deployment profits from profiling. Embedded devices with no debugging ambitions can certainly be compiled without frame pointers and used as-is. This change matters specifically to enterprise Linux and server back-ends, where observability pays long-term dividends.

Making It the Default

By the time Meta, Google, and Netflix had all quietly run frame-pointer-enabled libc builds for years (Google's early adoption was part of pioneering continuous profiling), the industry had a chasm between the haves and have-nots. In-house platform builds from those firms enjoyed excellent profiling capabilities that outsiders, lacking dedicated OS teams and vendor ties, couldn't realize.

The road back upstream for that set of changes proved rocky. Attempting to set this as a default for Fedora descended into a 116-post thread with strong opinions and competing demands. It may be a reasonable suggestion to ask for side repositories with benchmark data and thorough code-size measurement, but for organizations like Netflix, a company that doesn't even use Fedora, that becomes burdensome. LWN summed up the debate as "Fedora's tempest in a stack frame" for its detail and drama. There's undeniable care in the careful vetting process, but it also pushes the industry to pour energy into convincing the fragile political machinery of a standards change, all while better technologies for function graph introspection are being incrementally developed and may one day make frame pointers obsolete altogether.

Distros Turn the Page on Frame Pointers

After years of debate, major Linux distributions are reversing course and re-enabling frame pointers by default. Fedora accepted the proposal on its second attempt, becoming the first distro to bring frame pointers back. Ubuntu followed with its 24.04 LTS release, announcing frame pointers by default. Arch Linux has also begun enabling them.

This restores reliable stack walking through OS libraries. Applications may still lack stack tracing support, but that’s typically easier to address per language — Java offers the -XX:+PreserveFramePointer option, for example, and Go has had frame pointers as the default for years.

What Comes After Frame Pointers

Frame pointers aren’t the only way to unwind a stack. Several alternatives exist, each with tradeoffs:

• LBR (Last Branch Record): Intel’s hardware feature supports only 16 or 32 frames — too shallow for most application stacks to build flame graphs, but useful as a last resort for partial insights.
• BTS (Branch Trace Store): Another Intel mechanism without the depth limit, but it incurs overhead from memory load/stores and BTS buffer overflow interrupt handling.
• AET (Architectural Event Trace): A JTAG-based tracer capable of capturing low-level CPU, BIOS, and device events, and apparently stack traces too. It’s not practical for cloud environments where hardware-level access is unavailable.
• DWARF: Long used by debuggers for stack unwinding. Its overhead is high since it wasn’t designed for real-time use. While some JIT-to-DWARF work exists, it’s impractical for busy production JVMs constantly in C2 compilation. Experimentation with eBPF-based DWARF walkers has tried to cut costs, but Java remains a hurdle.
• eBPF stack walking: Mark Wielaard (Red Hat) demonstrated a Java JVM stack walker at LinuxCon 2014 using SystemTap, where an external tracer walked a runtime with no runtime assistance. Similar approaches work with eBPF, but performance overhead can spike from user-space reads of runtime internals, and the approach is brittle unless the walker ships with and is maintained alongside the language runtime.
• ORC (oops rewind capability): The Linux kernel’s lightweight unwinder, introduced by Josh Poimboeuf (Red Hat), allowed newer kernels to drop frame pointers while keeping stack walking intact. Many users run ORC without knowing it—the transition was smooth because kernel profiling code was updated in tandem.
• SFrames (Stack Frames): A user-space take on ORC, providing lightweight stack unwinding. Talks by Indu Bhagat (Oracle) and Steven Rostedt (Google) have covered the design.
• Shadow Stacks: A newer Intel and AMD security feature that pushes return addresses onto a separate hardware stack for verification at function return. That hardware stack could also serve as a source for stack traces, eliminating the need for frame pointers entirely.

Daan De Meyer (Meta) has also consolidated details on different stack walkers on the Fedora wiki.

Predictions and Outlook

Speculating on the future, it’s plausible that by 2029 Ubuntu and Fedora ship releases with SFrames for OS components, including libc, and drop frame pointers again—having reaped five years of performance wins and new stack-based tooling in the meantime. By 2034, shadow stacks could be enabled by default for security and serve as the basis for all stack tracing.

The original arguments for omitting frame pointers—significant i386 performance gains, compatibility with the day’s debuggers, and competition against Intel’s compiler—no longer hold in 2024. Eric Schrock contended it didn’t make sense even in 2004 when applied to x86-64, a view that hindsight supports. Profiling has effectively been broken for two decades and is only now being repaired.

Running the 2024 releases of Fedora and Ubuntu should make CPU flame graphs far more meaningful, enable Off-CPU flame graphs for the first time, and open the door to new observability capabilities. Continuous profilers also benefit, as they no longer need to convince customers to change their OS to get complete profiles.

The effort behind this change extends well beyond online threads—it includes extensive discussions, meetings, and work from contributors across Meta, Red Hat, Intel, Oracle, Canonical, Polar Signals, and others. In particular, Andrii Nakryiko (Meta), Daan De Meyer (Meta), Davide Cavalca (Meta), Neal Gompa (Velocity Limitless), Ian Rogers (Google), Steven Rostedt (Google), Josh Poimboeuf (Red Hat), Arjan Van De Ven (Intel), Indu Bhagat (Oracle), Mark Shuttleworth (Canonical), Jon Seager (Canonical), Oliver Smith (Canonical), Javier Honduvilla Coto (Polar Signals), Mark Wielaard (Red Hat), Ben Cotton (Red Hat), and Eric Schrock all played notable roles.

I enabled frame pointers at Netflix, for Java and glibc, and summarized the effect in BPF Performance Tools (page 40):

"Last time I studied the performance gain from frame pointer omission in our production environment, it was usually less than one percent, and it was often so close to zero that it was difficult to measure. Many microservices at Netflix are running with the frame pointer reenabled, as the performance wins found by CPU profiling outweigh the tiny loss of performance."

I've spent a lot of time analyzing frame pointer performance, and I did the original work to add them to the JVM (which became -XX:+PreserveFramePoiner). I was also working with another major Linux distro to make frame pointers the default in glibc, although I since changed jobs and that work has stalled. I'll pick it up again, but I'd be happy to see Fedora enable it in the meantime and be the first to do so.  

We need frame pointers enabled by default because of performance. Enterprise environments are monitored, continuously profiled, and analyzed on a regular basis, so this capability will indeed be put to use. It enables a world of debugging and new performance tools, and once you find a 500% perf win you have a different perspective about the <1% cost. Off-CPU flame graphs in particular need to walk the pthread functions in glibc as most blocking paths go through them; CPU flame graphs need them as well to reconnect the floating glibc tower of futex/pthread functions with the developers code frames.

I see the comments about benchmark results of up to 10% slowdowns. It's good to look out for regressions, although in my experience all benchmarks are wrong or deeply misleading. You'll need to do cycle analysis (PEBS-based) to see where the extra cycles are, and if that makes any sense. Benchmarks can be super sensitive to degrading a single hot function (like "CPU benchmarks" that really just hammer one function in a loop), and if extra instructions (function prologue) bump it over a cache line or beyond L1 cache-warmth, then you can get a noticeable hit. This will happen to the next developer who adds code anyway (assuming such a hot function is real world) so the code change gets unfairly blamed. It will only regress in this particular scenario, and regression is inevitable. Hence why you need the cycle analysis ("active benchmarking") to make sense of this.

There was one microservice that was an outlier and had a 10% performance loss with Java frame pointers enabled (not glibc, I've never seen a big loss there). 10% is huge. This was before PMCs were available in the cloud, so I could do little to debug it. Initially the microservice ran a "flame graph canary" instance with FPs for flame graphs, but the developers eventually just enabled FPs across the whole microservice as the gains they were finding outweighed the 10% cost. This was the only noticeable (as in, >1%) production regression we saw, and it was a microservice that was bonkers for a variety of reasons, including stack traces that were over 1000 frames deep (and that was after inlining! Over 3000 deep without. ACME added the perf_event_max_stack sysctl just so Netflix could profile this microservice, as the prior limit was 128). So one possibility is that the extra function prologue instructions add up if you frequently walk 1000 frames of stack (although I still don't entirely buy it). Another attribute was that the microservice had over 1 Gbyte of instruction text (!), and we may have been flying close to the edge of hardware cache warmth, where adding a bit more instructions caused a big drop. Both scenarios are debuggable with PMCs/PEBS, but we had none at the time.

So while I think we need to debug those rare 10%s, we should also bear in mind that customers can recompile without FPs to get that performance back. (Although for that microservice, the developers chose to eat the 10% because it was so valuable!) I think frame pointers should be the default for enterprise OSes, and to opt out if/when necessary, and not the other way around. It's possible that some math functions in glibc should opt out of frame pointers (possibly fixing scimark, FWIW), but the rest (especially pthread) needs them.

In the distant future, all runtimes should come with an eBPF stack walker, and the kernel should support hopping between FPs, ORC, LBR, and eBPF stack walking as necessary. We may reach a point where we can turn off FPs again. Or maybe that work will never get done. Turning on FPs now is an improvement we can do, and then we can improve it more later.

For some more background: Eric Schrock (my former colleague at Sun Microsystems) described the then-recent gcc change in 2004 as "a dubious optimization that severely hinders debuggability" and that "it's when people start compiling /usr/bin/* without frame pointers that it gets out of control" I recommend reading his post: [0].

The original omit FP change was done for i386 that only had four general-purpose registers and saw big gains freeing up a fifth, and it assumed stack walking was a solved problem thanks to gdb(1) without considering real-time tracers, and the original change cites the need to compete with icc [1]. We have a different circumstance today -- 18 years later -- and it's time we updated this change.

[0] http://web.archive.org/web/20131215093042/https://blogs.oracle.com/eschrock/entry/debugging_on_amd64_part_one
[1] https://gcc.gnu.org/ml/gcc-patches/2004-08/msg01033.html