Why Have Crisis Tools Ready Before You Need Them?
Performance-related outages are the worst time to discover your diagnostic tooling isn't installed. When a system is already degraded, package managers run slowly, repositories may be unreachable, and every minute spent installing software is a minute the service stays down. A practical safeguard is to pre-install a set of "crisis tools" on every Linux server.
These essentials were consolidated in Table 4.1, "Linux Crisis Tools," in Brendan Gregg's Systems Performance (2nd edition). The recommended suite covers basic statistics, tracing, and analysis capabilities:
| Package | Provides | Notes |
|---|---|---|
| procps | ps(1), vmstat(8), uptime(1), top(1) | basic stats |
| util-linux | dmesg(1), lsblk(1), lscpu(1) | system log, device info |
| sysstat | iostat(1), mpstat(1), pidstat(1), sar(1) | device stats |
| iproute2 | ip(8), ss(8), nstat(8), tc(8) | preferred net tools |
| numactl | numastat(8) | NUMA stats |
| tcpdump | tcpdump(8) | Network sniffer |
| linux-tools-common linux-tools-$(uname -r) | perf(1), turbostat(8) | profiler and PMU stats |
| bpfcc-tools (bcc) | opensnoop(8), execsnoop(8), runqlat(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), profile(8), etc. | canned eBPF tools[1] |
| bpftrace | bpftrace, basic versions of opensnoop(8), execsnoop(8), runqlat(8), biosnoop(8), etc. | eBPF scripting[1] |
| trace-cmd | trace-cmd(1) | Ftrace CLI |
| nicstat | nicstat(1) | net device stats |
| ethtool | ethtool(8) | net device info |
| tiptop | tiptop(1) | PMU/PMC top |
| cpuid | cpuid(1) | CPU details |
| msr-tools | rdmsr(8), wrmsr(8) | CPU digging |
This list is a starting minimum. Servers with specialized hardware should have their corresponding vendor tools installed too, for example intel-gpu-tools on Intel GPU systems or nvidia-smi on NVIDIA systems. Debuggers like gdb are also worth pre-installing so they are immediately available should a crisis require them.
These tools change infrequently, so the list should stay relevant for years. The main downside to adding them is on-disk size, which can add time to cloud instance deployment. That said, most of the packages are small, and the concern has prevented ~1 Gbyte debuginfo packages from being included by default in practice — but that's a trade-off many would accept for quicker diagnosis.
The Cost of Waiting: A Lesson in Obstacles
Installing software mid-incident is rarely straightforward. Beyond a slow system and a lengthy apt-get install, you can hit firewall rules that block outbound repository traffic, immutable file systems set by security teams that prevent writing executables, and misconfigured package sources. An estimate that "it'll take a few minutes" can easily stretch an hour, all while you could have been analyzing the problem. The real-world failure modes include:
- Repositories unreachable: Outbound HTTP/HTTPS/FTP to package repositories may be blocked by the network security team, and relaxing those rules is rarely a quick request.
- Immutable systems: Even as root, you may be unable to install binaries if the platform security team has made parts of the file system read-only. Disabling this can require spinning up a new server image.
- Workarounds are crude: Without
iostat, you may end up parsing data manually from/proc/diskstatsand relearning file format details from kernel documentation. - The failure repeats: The moment you launch new images with an installable system and restore the service, the original issue remains. The outage returns as soon as you are off-call and, in a worst-case scenario, the temporary security exceptions you enabled become a vector for an attack.
The lesson isn't that these scenarios happen every time, but that any one of these blockers, in a production system under load, is enough to make installation infeasible. In a larger organization, a "traffic team" might have already failed over regions by the time a tool is installed, leaving the faulty servers idle and you without answers.
Make the Default Install the Prepared Install
Companies with custom OS images already include these tools in their build. But many are still running a stock distribution and will only realize the gap at the worst possible moment. A good practice is for Linux enterprise variants to ship with this crisis toolkit as part of the default package set. The small size penalty is a low price for having the iostat, top, perf, and tracing tools you need already on the box when a performance incident hits.
If you maintain a base server image, include these diagnostics from the start. When the alert page goes off at 4pm, the best debugging tool is the one that's already installed.
