Fedora now has frame pointers

Fedora now has frame pointers. I don’t want to dwell on the how of this, it was a somewhat controversial decision and you can read all about it here. But I do want to say a bit about the why, and how it makes performance analysis so much easier.

Recently we’ve been looking at a performance problem in qemu. To try to understand this I’ve been looking at FlameGraphs all day, like this one:

FlameGraphs rely on the Linux tool perf being able to collect stack traces. The stack traces start in the kernel and go up through userspace often for dozens or even hundreds of frames. They must be collected quickly (my 1 minute long trace has nearly half a million samples) and accurately.

Perf (or actually I think it’s some component of the kernel) has various methods to unwind the stack. It can use frame pointers, kernel ORC information or DWARF debug information. The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.

For example, here is a broken stack trace from Fedora 37 (with full debuginfo installed):

Notice that we go from the qemu-img program, through an “[unknown]” frame, into zlib’s inflate.

In the same trace we get completely detached frames too (which are wrong):

Upgrading zlib to F38 (with frame pointers) shows what this should look like:

Another common problem with lack of frame pointers can be seen in this trace from Fedora 37:

It looks like it might be OK, until you compare it to the same workload using Fedora 38 libraries:

Look at those beautiful towering peaks! What seems to be happening (I don’t know why) is that stack traces start in the wrong place when you don’t have frame pointers (note that FlameGraphs show stack traces upside down, with the starting point in the kernel shown at the top). Also if you look closely you’ll notice missed frames in the first one, like the “direct” call to __libc_action which actually goes through an intermediate frame.

Before Fedora 38 the only way to get good stack traces was to recompile your software and all of its dependencies with frame pointers, a massive pain in the neck and a major barrier to entry when investigating performance problems.

With Fedora 38, it’s simply a matter of using the regular libraries, installing debuginfo if you want (it does still add detail), and you can start using perf straight away by following Brendan Gregg’s tutorials.