strace for System Call Debugging on Linux

What Is strace?

strace is a diagnostic tool that intercepts and records every system call made by a process. System calls are the interface between your application and the Linux kernel — every file open, network connection, memory allocation, and process creation goes through a system call.

When a program fails silently, hangs mysteriously, or behaves unexpectedly, strace reveals exactly what it is doing at the kernel level. It is one of the most valuable debugging tools in Linux.

Your First strace

# Trace a simple command
strace ls /tmp

# You'll see output like:
# execve("/usr/bin/ls", ["ls", "/tmp"], ...) = 0
# brk(NULL)                               = 0x5629a8a33000
# openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
# ...
# openat(AT_FDCWD, "/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
# getdents64(3, [...], 32768)             = 480
# write(1, "file1.txt\nfile2.txt\n", 20)  = 20
# close(3)                                = 0
# exit_group(0)                           = ?

Each line shows: the system call name, its arguments, and the return value after =. A return value of -1 indicates an error, followed by the error name and description.

Tracing a Running Process

You don’t need to start a process with strace. You can attach to a process that is already running:

# Attach to a running process by PID
sudo strace -p 12345

# Attach to all threads of a process
sudo strace -p 12345 -f

# Detach with Ctrl+C (the traced process continues normally)

This is especially useful for debugging production issues on running servers.

Filtering by System Call

The raw output of strace can be overwhelming. Use -e to filter for specific calls:

# Only show file-related calls
strace -e trace=file ls /tmp
# Shows: open, openat, stat, access, etc.

# Only show network calls
strace -e trace=network curl https://example.com
# Shows: socket, connect, sendto, recvfrom, etc.

# Only show process-related calls
strace -e trace=process bash -c 'echo hello'
# Shows: execve, fork, clone, wait4, exit_group, etc.

# Only show specific system calls
strace -e openat,read,write cat /etc/hostname

# Only show calls that failed
strace -e trace=file -Z ls /nonexistent
# -Z filters to only show calls with errors

Available trace categories:

• file — File operations (open, stat, chmod, etc.)
• network — Networking (socket, connect, send, recv, etc.)
• process — Process management (fork, exec, wait, etc.)
• signal — Signal handling
• memory — Memory mapping (mmap, brk, etc.)
• ipc — Inter-process communication
• desc — File descriptor operations (read, write, close, etc.)

Debugging Common Problems

File Not Found

When a program says “config file not found” but you know it exists:

strace -e openat myapp 2>&1 | grep config
# openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
# openat(AT_FDCWD, "/opt/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
# openat(AT_FDCWD, "./config.yaml", O_RDONLY) = 3

Now you can see exactly which paths the program searches and where it expects the file.

Permission Denied

strace -e openat,access myapp 2>&1 | grep -i "perm\|denied\|EACCES"
# openat(AT_FDCWD, "/var/log/myapp.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = -1 EACCES (Permission denied)

Hanging Process

When a process appears stuck, attach to it and see what it is waiting on:

sudo strace -p $(pgrep myapp)
# If you see something like:
# futex(0x7f2a1c000900, FUTEX_WAIT, ...) -- it's waiting on a lock
# read(5,                                -- it's blocked reading from fd 5
# poll([{fd=3, events=POLLIN}], 1, -1)   -- it's waiting for I/O on fd 3

# Find out what file descriptor 5 refers to
ls -la /proc/$(pgrep myapp)/fd/5
# lrwx------ 1 root root 64 ... 5 -> socket:[12345]

Slow Application

When a program is unexpectedly slow, use timing options:

# Show time spent in each system call
strace -T myapp
# openat(AT_FDCWD, "/data/large.db", O_RDONLY) = 3 <0.000015>
# read(3, "...", 4096) = 4096 <0.523412>  # <-- This read took 0.5 seconds!

# Show wall-clock timestamps
strace -t myapp       # HH:MM:SS
strace -tt myapp      # HH:MM:SS.microseconds
strace -r myapp       # Relative timestamps (time since last call)

Network Connection Issues

# Trace a curl request to see DNS resolution and connection details
strace -e trace=network -f curl https://api.example.com 2>&1 | grep -E 'connect|sendto'
# connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = -1 EINPROGRESS

Summarizing System Calls

For performance analysis, use -c to get a summary table:

strace -c ls /tmp
# % time     seconds  usecs/call     calls    errors syscall
# ------ ----------- ----------- --------- --------- ----------------
#  32.14    0.000261           5        46           mmap
#  19.87    0.000163           6        23           openat
#  12.44    0.000102           4        25           close
#  10.13    0.000082           3        24           fstat
#   6.57    0.000053           5        10           mprotect
#   5.37    0.000044           4        10           read
# ...

# Combine summary with full trace
strace -c -S time myapp    # Sort by time spent
strace -c -S calls myapp   # Sort by number of calls

Following Child Processes

Many applications fork child processes. Use -f to trace them all:

# Trace a shell script and all commands it spawns
strace -f -e trace=process,file bash script.sh

# Trace a web server and its worker processes
sudo strace -f -p $(pgrep nginx) -e trace=network

# Write each process's trace to a separate file
strace -f -ff -o /tmp/trace myapp
# Creates /tmp/trace.12345, /tmp/trace.12346, etc.

Output Options

# Write trace to a file (stderr stays clean)
strace -o /tmp/trace.log myapp

# Increase string length (default truncates at 32 chars)
strace -s 1024 myapp

# Show full paths for file operations
strace -y myapp
# read(3</etc/passwd>, "root:x:0:0:...", 4096) = 1523

# Decode file descriptor numbers to paths
strace -yy myapp
# sendto(3<TCPv4:[10.0.0.1:54321->10.0.0.2:443]>, ...)

Real-World Debugging Scenarios

Why Is My Docker Container Failing to Start?

# Run the container's entrypoint under strace
docker run --entrypoint strace myimage -f -e trace=file,process /original/entrypoint.sh

Which Config Files Does an Application Load?

strace -e openat -f myapp 2>&1 | grep -v ENOENT | grep -E '\.(conf|cfg|yaml|yml|json|ini|env)'

Is My Application Using DNS Correctly?

strace -e trace=network -f myapp 2>&1 | grep -E 'connect|sendto.*53'

Memory Allocation Patterns

strace -e brk,mmap,munmap myapp 2>&1 | head -50

strace Alternatives

Tool	Use Case
strace	General system call tracing
ltrace	Library call tracing (e.g., libc functions)
perf trace	Lower overhead system call tracing
bpftrace	Programmable tracing with eBPF (advanced)
sysdig	System-wide tracing with container awareness

Performance Warning

strace uses ptrace to intercept every system call, which adds significant overhead — typically slowing the traced process by 10-100x. For production systems under load, consider using perf trace or eBPF-based tools instead.

# Lower-overhead alternative for production
sudo perf trace -p $(pgrep myapp)

Summary

strace is an essential debugging tool that shows you exactly what system calls your programs make. The key techniques are: filtering with -e trace=file|network|process to focus on relevant calls, using -T for timing analysis, -c for call summaries, -f for following child processes, and -p for attaching to running processes. When a program fails silently, hangs, or behaves unexpectedly, strace is often the fastest path to understanding why.