Had a process making mysterious open() calls on files that didn’t exist. strace told me what was being opened but not which code path was doing the opening.

strace -k adds a backtrace for every syscall:

open("/etc/weird-config", O_RDONLY) = -1 ENOENT
 > /usr/lib/x86_64-linux-gnu/libc.so.6(open+0x2d) [0xf6a7d]
 > /app/bin/myservice(load_config+0x5a) [0x4a2e1]
 > /app/bin/myservice(main+0x120) [0x4b000]

Now I know: load_config is the culprit, and I can look at that function. Without -k, I’d have had to guess.

Caveats:

  • -k needs libunwind. Debian: apt-get install libunwind8. Most distros ship strace without it; you may need a newer build.
  • It adds meaningful overhead. A few percent in normal use, more for syscall-heavy code. Not for “attach this to prod” use.
  • Symbols need to be readable. Stripped binaries will show hex offsets only. Combine with a debuginfo package for pretty output.

When paired with -e trace=openat or -e trace=file, you get a really nice “who is trying to open what, and why” tool. Next time I have a process that can’t find its config, this is the move.