strace in practice: what system administrators actually use it for

strace is one of those tools that seems deceptively simple—it just traces system calls—but it turns out to be useful for an enormous range of debugging problems. A recent informal poll of 200 strace users surfaced nine recurring categories of problems, and most of them fall into three buckets: finding files a program depends on, figuring out why a program is stuck or slow, and diagnosing why a program is failing.

Finding files and configuration

The single most common use case is answering a deceptively simple question: “where is the config file?” A program may document its configuration location in a man page, on a website, or in --help, but strace shows you the ground truth—the actual open() system call that reveals which path the program really tries.

Beyond config files, strace is frequently used to discover all sorts of other file dependencies:

  • Shared libraries: diagnosing why a program loads the wrong version of a .so file
  • Language runtime files: locating where a program searches for Ruby gems or Node.js modules
  • SSL root certificates
  • Game save files and closed-source data files
  • Unused node_modules files, useful for shrinking package size

Diagnosing hangs and liveness

When a program appears stuck, strace is often the fastest way to understand why. Attaching with strace -p PID shows the currently executing system call, and the answer is usually one of a handful of possibilities:

  • Blocked forever in a select() poll
  • Waiting in wait() for a subprocess
  • Stalled on a network request to an unresponsive peer
  • Blocked in write() because the output buffer is full
  • Waiting in read() on stdin for input

One concrete example: running strace df -h can reveal which mount point is hung, allowing you to unmount it and unblock the command.

A closely related use is determining whether a long-running program is actually making progress or just spinning. Since strace shows you system calls as they happen, attaching to a process and watching for new calls is a quick liveness check—as long as the program is doing any kind of I/O.

Performance troubleshooting

strace doubles as a rough profiler. Using strace -t to timestamp each system call helps you spot large gaps that point to the bottleneck. Real-world examples from the field include:

  • A Java application server reading class files one byte at a time—a classic missing-BufferedReader mistake that crippled page load times
  • Slow startup caused by opening and reading the same config file thousands of times
  • PHP session file reads taking ~60 seconds due to flock() calls
  • A program re-initializing its PRNG on every request by reading from /dev/random, exhausting the entropy pool
  • A server hitting its open-file limit, failing accept() with EMFILE, and retrying endlessly without reporting the error
  • A workflow making 30-second HTTP requests that timed out and retried 5 times before failing, invisible without tracing
  • Slow DNS resolution—you can’t see gethostbyname() directly, but the DNS packets show up clearly in strace output

Diagnosing failures

Many mysterious failures are, at their core, hidden permission errors. A program might fail with an inscrutable message when the real problem is that it cannot open some filesystem node. strace shows the exact open() call and the EACCES (permission denied) error, even when the application itself reports something vague. A representative case: a pen plotter tool emitted a cryptic error, and strace revealed the user simply lacked permission to open the USB device.

strace also shines in comparing success and failure cases. A common pattern is a binary that works when run interactively but fails when launched by systemd, as a cron job, or via su - user /some/script. Tracing both invocations lets you diff the system call sequences and find the difference. Environment variables are often worth checking first for these cases, but strace gives you the definitive answer.

Another class of failure involves command line arguments. When one script spawns another, strace reveals the exact flags being passed such as detecting overlong command lines or determining which compiler flags a build actually uses.

Network debugging and reverse engineering

For network issues, strace can determine which domain or IP a connection targets by inspecting DNS requests for the domain or the connect() call for the address. This is particularly handy when tcpdump is unavailable, or simply when strace is the tool the engineer knows better.

For understanding kernel APIs like netlink, io_uring, or I2C, strace lets you observe how an existing application interacts with the kernel. Documentation for these interfaces is often sparse or confusing, and tracing a real working program provides concrete examples that man pages can’t match. The same approach scales to general reverse engineering—figuring out how any program works by observing its system call activity.

What this tells us

The breadth of uses is striking, but so is the repetition. Dozens of respondents reported using strace for the exact same purpose: finding config files. The power of strace comes from the fact that no matter how a program documents its behavior, the system calls are ground truth. Any file access, network connection, or permission check ultimately shows up in the trace.