Understanding System Calls and Libraries

1. Tracing System Calls

Tool	Description	Usage	Notes
strace	Tracks system calls made by a process	`strace -p <PID>`	Useful for debugging and performance analysis
ltrace	Tracks library calls made by a process	`ltrace -p <PID>`	Tracks dynamic library calls, not system calls
sysdig	System-level system call tracer and analyzer	`sysdig -p "{proc.name}: {syscall.name}"`	Supports filtering and real-time analysis

Example: Using strace to trace a command

strace -e trace=openat,close,read,write ls /etc

Example: Tracing a running process

strace -p 1234

Note: Use -e trace=... to limit the system calls being traced for better performance.

Warning: Tracing system calls can significantly slow down the target process.

2. Tracing Library Calls

Tool	Description	Usage	Notes
strace	Tracks system calls and signals	`strace -e trace=openat,read,write ./program`	Useful for debugging and performance analysis
ltrace	Traces library calls made by a program	`ltrace -e openat,read,write ./program`	Tracks calls to shared libraries like libc
perf	Performance analysis tool with syscall support	`perf record -e syscalls:sys_enter_openat ./program`	Can trace specific syscall categories or events

Example: Using ltrace to trace library calls

ltrace -e openat,read,write ./myapp

Note: ltrace requires programs to be compiled with debug symbols for detailed output.

Warning: Tracing can significantly slow down execution due to overhead.

Option	Description	Example	Notes
-e	Specify which calls to trace	`-e openat,read`	Filter by function names or syscall categories
-o	Output to file instead of stdout	`-o trace.log`	Useful for large or long-running traces
-c	Collect call counts and timing	`-c ./program`	Provides summary statistics without full trace

3. Viewing Shared Libraries

Command	Description	Example
ldd	Lists shared library dependencies of an executable	`ldd /bin/ls`
ldconfig -p	Displays the list of cached shared libraries	`ldconfig -p \| grep libc`
objdump -p	Shows program headers and dynamic section info	`objdump -p /usr/bin/python3 \| grep NEEDED`
readelf -d	Displays dynamic section information from ELF files	`readelf -d /lib/x86_64-linux-gnu/libc.so.6 \| grep NEEDED`

Example: Using ldd to view dependencies

ldd /usr/bin/python3
	linux-vdso.so.1 (0x00007fff...)
	libpython3.10.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0 (0x00007f...)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f...)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f...)

Note: The ldd command uses the LD_TRACE_LOADED_OBJECTS environment variable to determine dependencies.

Warning: Not all systems support ldd for dynamically linked executables; it may not work in some containerized environments.

4. Managing Library Cache

Command	Description	Usage
ldconfig	Manages the library cache	`sudo ldconfig` updates the cache
ldconfig -p	Displays the current library cache	`ldconfig -p` lists all cached libraries
ldconfig -v	Verbose output when updating cache	`sudo ldconfig -v` shows detailed updates

Example: Updating Library Cache

sudo ldconfig

Note: The ldconfig command updates the cache used by the dynamic linker to find shared libraries.

Warning: Running ldconfig without sudo will not update system libraries.

5. Setting Library Path

Environment Variable	Value	Description
LD_LIBRARY_PATH	`/usr/local/lib:/opt/myapp/lib`	Defines directories to search for shared libraries at runtime
LD_PRELOAD	`libmylib.so`	Forces loading of specified libraries before others

Example: Setting Library Path Temporarily

export LD_LIBRARY_PATH=/usr/local/lib:/opt/myapp/lib:$LD_LIBRARY_PATH

Example: Using LD_PRELOAD to Override System Library

LD_PRELOAD=/opt/myapp/lib/libm.so ./myapp

Note: Changes to LD_LIBRARY_PATH only affect the current shell session and any child processes spawned from it.

Warning: Setting LD_LIBRARY_PATH incorrectly can lead to library version conflicts or application crashes.

6. Viewing Open Files

Command	Description	Example
lsof	Lists open files and the processes that opened them	`lsof -p 1234`
/proc/<pid>/fd	Directory containing file descriptors for a specific process	`ls -l /proc/1234/fd`
fuser	Identifies processes using specified files or file systems	`fuser -v /tmp/file.txt`

Example: Using lsof to view open files for a process

lsof -p 1234
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
node    1234   user   12r    REG    8,1   102400   567890 /home/user/app.js
node    1234   user   13w    REG    8,1   204800   567891 /home/user/log.txt

Example: Inspecting file descriptors via /proc

ls -l /proc/1234/fd
total 0
lrwx------ 1 user user 64 Apr 5 10:30 0 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 1 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 2 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 3 -&gt; /tmp/file.txt

7. Finding File Descriptors

Command	Description	Usage
lsof	List open files and their associated processes	`lsof -p <PID>`
/proc/<PID>/fd	Directory containing symbolic links to open file descriptors	`ls -l /proc/<PID>/fd`
fuser	Identify processes using specific files or filesystems	`fuser -v /path/to/file`

Example: Viewing file descriptors for a process

# View file descriptors for process with PID 1234
ls -l /proc/1234/fd

# Output example:
lrwx------ 1 user user 64 Apr 5 10:30 0 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 1 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 2 -&gt; /dev/tty
lrwx------ 1 user user 64 Apr 5 10:30 3 -&gt; /tmp/logfile.txt

Note: File descriptors are integers used by the kernel to track open files, sockets, and other I/O resources. Each process has its own set of file descriptors.

8. Debugging with gdb

Command	Description	Usage
gdb <program>	Start gdb with a program	`gdb ./myapp`
break <function>	Set breakpoint at function	`break main`
run	Start program execution	`run`
continue	Continue execution after breakpoint	`continue`
step	Step into function calls	`step`
next	Step over function calls	`next`
print <variable>	Print variable value	`print x`
backtrace	Show call stack	`backtrace`
quit	Exit gdb	`quit`

Example: Basic gdb session

gdb ./myapp
(gdb) break main
(gdb) run
(gdb) print x
(gdb) continue
(gdb) quit

Note: Use gdb --args <program> <arguments> to pass arguments to the program under debug.

Warning: Ensure your binary is compiled with debug symbols (-g flag) for better debugging experience.

9. Analyzing Core Dumps

Core dumps are binary files created when a program crashes due to a segmentation fault or other critical error. They contain the memory state of the process at the time of the crash, which can be invaluable for debugging.

Command	Description	Notes
ulimit -c	Check the maximum size of core dumps	Set to 0 to disable core dumps
ulimit -c unlimited	Enable unlimited core dump size	Use with caution as it can fill up disk space
gdb <executable> <core_file>	Debug the core dump using GDB	Useful for inspecting stack traces and variables

Example: Analyzing a Core Dump with GDB

$ gdb /usr/bin/myapp core
(gdb) bt
(gdb) info registers
(gdb) print variable_name

Note: Core dumps are typically stored in /var/crash or /tmp depending on system configuration.

Warning: Core dumps may contain sensitive data. Ensure proper handling and access control.

10. Using valgrind for Memory Analysis

Example: Running valgrind on a simple C program

valgrind --leak-check=full ./my_program

Option	Description	Default
--leak-check=full	Reports all memory leaks with detailed information	full
--leak-check=summary	Reports only summary of memory leaks	full
--show-leak-kinds=all	Shows all types of memory leaks	full
--track-origins=yes	Tracks origins of uninitialized values	no

Note: Valgrind is a powerful tool for detecting memory leaks and errors in C/C++ programs. It simulates a virtual machine to analyze program behavior.

Warning: Valgrind can significantly slow down program execution, so it's best used during development and testing phases.