Linux Systems Performance in 40 Minutes

Systems performance analysis doesn’t have to be a niche discipline reserved for kernel engineers. With the right tools and methodologies, application developers and system administrators can uncover meaningful performance wins without investing months in study. At USENIX LISA 2019, a 40-minute session distilled the essentials of Linux systems performance into six core areas: observability tools, methodologies, benchmarking, profiling, tracing, and tuning.

The full presentation, including the video and slide deck, is available online:

The talk is an update to material that many companies already use as recommended or required reading for new engineers, drawn from the Systems Performance book. Work on a second edition is underway, following completion of the BPF performance tools book.

The Six Pillars of Linux Performance

The session organizes the practice around six practical areas, each addressing a different stage of the performance lifecycle. Observability tools provide the raw data—counter statistics, metrics, and event logs—needed to understand system state. Methodologies offer structured approaches for turning that data into actionable analysis, such as the USE method (utilization, saturation, and errors) for resource checks or the TSA method (thread state analysis) for application-level issues.

Benchmarking establishes a baseline for expected performance, while profiling identifies where CPU time is spent. Tracing goes further, following code paths and kernel events to explain why a particular bottleneck exists. Finally, tuning applies targeted changes based on the evidence gathered from the previous steps.

The presentation notes that the different background colors used in the screenshot examples have no technical significance—a purely cosmetic choice to avoid monotony.

Workshop Expansion and Availability

Beyond the talk itself, a BPF performance tools workshop was run at LISA with over 200 attendees. The 90-minute format was tight, but participants still found it valuable. A longer, three-hour version is scheduled for the rescheduled USENIX SREcon Americas West conference in Santa Clara in June.

Workshops have not been video-recorded so far, but there is interest in capturing one for remote viewing, particularly in the context of supporting work-from-home arrangements.