Mastering Strace: Advanced Debugging and Performance Analysis

Strace is a potent Swiss Army knife for systems troubleshooting on Linux. It shines a light on the black box of program execution by tracing the underlying system calls between running processes and the kernel.

While the basics of strace are easy to use, mastering strace for advanced debugging and performance tuning requires some practice. But the effort pays dividends the next time you need to diagnose those pesky "works-on-my-machine" bugs or analyze why an application is running slow.

In this comprehensive 3500+ word guide, you‘ll gain expert techniques for unlocking strace‘s full potential.

Strace Refresher

Let‘s quickly recap how strace does its magic before diving into advanced usage.

When you execute a program in Linux, behind the scenes it makes many system calls to the kernel to perform tasks like:

• Opening files
• Creating sockets
• Allocating memory
• Creating processes/threads

Strace intercepts these system calls, along with arguments and return values. This grants complete visibility into the application‘s runtime activity.

Some examples of common system calls traced by strace:

System Call	Description
read()	Read data from a file descriptor
write()	Write data to a file descriptor
open()	Open a file descriptor
close()	Close a file descriptor
socket()	Create a new network socket
connect()	Connect a socket to an address
clone()	Create a new process (fork)

So when you run:

strace ls -l

You‘ll see the sequence of underlying system interactions used to initialize the ls process and query directory contents before outputting results.

This low-level visibility makes strace invaluable for all kinds of Linux debugging scenarios. Let‘s explore some advanced techniques.

Filtering Specific System Calls

One easy way to tame lengthy strace output is using filters to only show specific system calls of interest.

For example, to trace just file I/O activity:

strace -e trace=read,write,open,close ls /somedir 

The -e trace=set option takes a comma-separated list to control which system calls get traced. Useful sets might include:

File I/O: read, write, open, close
Process management: clone, execve, exit, kill
Memory allocation: mmap, mremap, munmap, brk
Network activity: socket, connect, sendto, recvfrom

Refer to the full list of system calls to build customized filters for your debugging needs.

Syscall Statistics & Summaries

Another handy strace feature is system call summaries via the -c flag:

strace -c ls /somedir

This prints aggregate statistics when the traced process exits:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 29.63    0.000100           3        30           read
  8.33    0.000028           3        10           close
  7.41    0.000025           3         8           fstat
  7.41    0.000025           3         8           mmap
  4.94    0.000017           3         5           open
  4.94    0.000017           3         5        5 stat

The summary breaks down:

• Relative time spent in each system call
• Number of calls per syscall
• Errors encountered

This reveals hot spots for optimization like slow I/O routines.

You can even filter the statistics by specific calls to drill down. Say we want memory allocation stats:

strace -c -e trace=mmap,munmap,brk,... ls /somedir

Decoding Opaque Arguments

A frequent challenge when reading strace output is decoding opaque arguments and return codes.

Calls like socket(), ioctl(), fcntl() etc. involve complex flag values and binary data structures. For example:

ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or 0, 0x7ffd27bc1d90) = -1 ENOTTY (Inappropriate ioctl for device)

What is that mystery IOCTL code? And the pointer argument?

This is where the -v verbose flag comes in handy. It tries to symbolically decode obscure arguments into human readable values:

ioctl(3, SNDRV_TIMER_IOCTL_NEXT_DEVICE or TMR_TIMEBASE, {timer_id=0}) = -1 ENOTTY (Inappropriate ioctl for device)

Much clearer!

For truly baffling calls, your best bet is to lookup the function prototype in the relevant system header file or API documentation to decipher each argument. Then mapping strace‘s raw memory addresses to defined constants.

It takes some sleuthing, but uncovering that obscure value can sometimes be the key to unlocking a gnarly bug.

Injecting a Fault

Ever seen buggy code handle errors gracefully in the development environment, only to crash unceremoniously in production? This "Heisenbug" is the bane of developers everywhere.

Strace provides a devious but effective way to simulate system call errors via fault injection.

The --inject-error=syscall,errno parameter randomly throws specified error codes when invoked:

strace --inject-error=open,ENOENT grep pattern /somefile

This will randomly fail open() calls used by grep to simulate file errors.

Devious! But incredibly useful for resilience testing, hardening fault handling logic, and smoke testing recovery code.

Customize the syscall and errno per the kinds of issues you want to simulate. Common ones include:

• Network: inject read/write errors on sockets
• File failures: inject open/read/write ENOENT, EBADF, EIO
• Memory issues: inject mmap/brk ECORRUPT, ENOMEM
• Permissions : inject open/access EACCESS, EPERM

Unleash fault injection on your programs before attackers do!

Performance Profiling & Optimization

In addition to debugging, strace provides valuable application performance profiling:

1. Find hot code paths

Frequent syscalls reveal where a process spends execution time – whether productive or wasted cycles.

For example, excessive stat() calls indicate lots of filesystem queries slowing things down. You may optimize with caching or alternate data structures.

2. Measure I/O bottlenecks

Does your app grind to a halt doing big data crunching? Check if it‘s I/O bound by evaluating syscall counts/latency:

$ strace -c my_calc_app myfile

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
45.63    1.234567         120     10256           read 
23.12    0.623971          85      7346           write

This shows heavy disk I/O limiting performance. Consider faster storage, I/O parallelism, mmap instead of read/write, etc.

3. Identify wasted resources

Runtime bloat? Are too many memory maps alive, temporary files open, processes running?

Summarize resource usage via:

strace -c -e trace=mmap,open,clone,... my_app

Then optimize your architecture.

Integrations

While strace focuses specifically on system calls, integrating it with other tools expands your investigative powers:

GDB – The GNU debugger, allows attaching strace to a running process AND inspecting source code and variables. A powerful combo!

gdb -p <PID>
(gdb) call system("strace -p %d -o /tmp/log &") 

This enables simultaneous strace logging on a process you are debugging with gdb.

Perf – Linux performance analyzer can provide correlated hardware performance counters – cpu cycles, cache misses etc. – alongside strace system calls.

perf stat -e cycles,page-faults,context-switches strace -c <cmd>

Syscall proxies – Tools like goss interpose on strace to collect richer OS interaction detail including arguments parsing.

Containers – Strace a container process to inspect interaction with the container runtime, network bridge, and host virtualization.

And many other combinations abound! Strace is designed for easy integration.

Digging Into Source Code

As a bonus, strace itself serves as a great learning tool for developers.

The strace source code provides heavily documented examples of:

• Raw Linux syscall mechanics
• Solid system programming techniques
• Architecting trace/debug tools
• Parsing procfs/syscall interfaces

Beyond usage, engineers can leverage strace for kernel and low-level education.

Conclusion

Strace is a Swiss army knife for systems developers, administrators, SREs and other Linux professionals. It runs on every Linux box and requires no special access since it simply traces public kernel interfaces.

With so many capabilities this article only scratched the surface of strace‘s potential. We explored filters, statistics, decoders, error injection, profiling, integrations and internals.

Yet strace has many more capabilities we didn‘t cover like custom output formatting, decoding network protocols, tailing logs, tracing child processes, container integration, decoding IOCTL parameters, and other exotica described in the exhaustive strace man pages.

Although strace output admittedly leans towards the arcane side, a little practice goes a long way. Mastery pays off handsomely the next time you face a gnarly systems conundrum. Mousing around a GUI can‘t match the precision diagnostics of watching raw syscalls execute.

Add strace to your toolkit if you haven‘t already. Its power and ubiquity across systems make it a must-learn for anyone working on Linux platforms. The highest paid engineers wield all the debugging tools – don‘t miss out on the Swiss Army knife!