Optimizing System Performance
1. Identifying Performance Bottlenecks
| Tool | Description | Use Case |
|---|---|---|
| top | top command shows real-time system processes and resource usage. |
Monitor CPU and memory usage of running processes. |
| htop | htop is an interactive process viewer with better UI than top. |
Enhanced visualization of system load and process hierarchy. |
| iostat | iostat reports CPU and I/O statistics for devices and partitions. |
Analyze disk I/O bottlenecks. |
| vmstat | vmstat reports virtual memory statistics including swap usage. |
Detect memory pressure and swapping activity. |
| sar | sar (System Activity Reporter) logs system activity over time. |
Historical analysis of performance trends. |
| perf | perf is a performance profiling tool for kernel and userspace programs. |
Deep-dive analysis of function-level performance. |
Example: Using top to Identify High CPU Processes
top
# Press '1' to view per-CPU stats.
# Sort by CPU usage using 'Shift+<'.
Example: Using iostat to Analyze Disk I/O
iostat -x 1 5
# -x: extended statistics.
# 1 5: update every second, 5 times.
Note: Use
htop for a more user-friendly interface than top, especially in environments where interactivity is preferred.
Common Performance Bottlenecks
- CPU saturation: Processes consuming too much CPU time.
- Memory pressure: Excessive swapping or allocation failures.
- Disk I/O limits: Slow reads/writes due to hardware or configuration.
- Network contention: High latency or bandwidth saturation in networked systems.
2. Tuning Swap Usage
| Parameter | Values | Description |
|---|---|---|
| vm.swappiness | 1-100 |
Controls tendency to swap out memory pages. Lower values (e.g., 1) prefer keeping pages in RAM; higher values (e.g., 60) favor swapping. |
| vm.vfs_cache_pressure | 0-1000 |
Adjusts how aggressively the kernel reclaims memory used for caching directory and inode entries. Higher values increase reclaiming. |
| vm.dirty_ratio | 0-100 |
Maximum percentage of system memory that can be filled with "dirty" pages before synchronous writes begin. |
| vm.dirty_background_ratio | 0-100 |
Percentage of system memory at which background writeback begins, even if dirty pages are below dirty_ratio. |
Example: Adjusting Swap Settings
# View current swap settings
cat /proc/sys/vm/swappiness
cat /proc/sys/vm/vfs_cache_pressure
# Temporarily set swappiness to 10
echo 10 > /proc/sys/vm/swappiness
# Permanently set swappiness to 10 in sysctl.conf
echo 'vm.swappiness = 10' >> /etc/sysctl.conf
# Apply changes without reboot
sysctl -p
Note: A value of
vm.swappiness = 1 is often recommended for desktop systems to minimize swapping, while vm.swappiness = 60 is typical for servers where swapping might be acceptable under memory pressure.
Warning: Setting
vm.swappiness too high can lead to excessive swapping, which degrades performance. Setting it too low may cause memory exhaustion under high load.
3. Optimizing File System Performance
| Option | Values | Notes |
|---|---|---|
| io.scheduler | none, noop, cfq, deadline, bfq |
IO scheduler to use for SSDs or HDDs. NEW in kernel 5.10 |
| mount.options | noatime, nodiratime, async |
Mount options to reduce metadata writes. DEPRECATED in favor of noatime |
| tmpfs.size | 1G, 2G |
Size of tmpfs mount in bytes or with suffixes (K, M, G) |
Note: Use
noatime to prevent access time updates on files, which reduces disk I/O.
Warning: Changing
io.scheduler can impact performance on different hardware types.
| File System | Optimization | Notes |
|---|---|---|
| ext4 | data=writeback |
Improves performance by reducing data consistency checks |
| xfs | logbsize=64k |
Increases log buffer size for better performance |
| btrfs | compress=zstd |
Enables Zstandard compression for better space usage |
4. Tuning I/O Scheduler
| Property | Value | Description |
|---|---|---|
| io.scheduler | none |
Disables I/O scheduler (not recommended for general use) |
| io.scheduler | noop |
Simple FIFO queue; suitable for SSDs and devices without seek time |
| io.scheduler | deadline |
Ensures I/O requests complete within a time limit; good for databases |
| io.scheduler | cfq |
Completely Fair Queuing; balances throughput and latency (deprecated in newer kernels) |
| io.scheduler | bfq |
Bucket Fair Queuing; designed for desktop systems with high responsiveness needs |
| io.scheduler | none |
Disables I/O scheduler (not recommended for general use) |
Note: The I/O scheduler can be changed dynamically without rebooting the system.
Warning: Changing I/O scheduler settings may affect system performance and stability. Ensure proper testing before deploying in production.
5. Configuring CPU Governor
| Property | Value | Description |
|---|---|---|
| cpu.governor | ondemand |
Dynamic scaling governor that adjusts CPU frequency based on system load |
| cpu.governor | performance |
Fixed performance governor that runs the CPU at maximum frequency |
| cpu.governor | powersave |
Fixed powersave governor that runs the CPU at minimum frequency |
| cpu.governor | userspace |
Userspace governor that allows users to set the CPU frequency manually |
| cpu.governor | conservative |
Conservative governor that increases frequency slowly when load increases |
Example: Set CPU governor to performance
echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
Note: Changes made via
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor are not persistent across reboots. Use cpupower or systemd services for persistence.
Warning: Setting the governor to
performance will increase power consumption and heat generation.
6. Managing Process Priorities
| Priority Level | Numeric Value | Description | Usage |
|---|---|---|---|
| Idle | 19 |
Lowest priority, runs only when no other processes are active | For background tasks that should not interfere with system responsiveness |
| Normal | 0 |
Default priority level for most processes | Standard execution without special scheduling requirements |
| High | -5 |
Higher than normal priority, useful for time-sensitive tasks | For processes requiring more CPU time than typical |
| Real-time | -20 |
Highest priority, can preempt other processes | For real-time systems or critical tasks needing immediate processing |
Example: Setting Process Priority with nice
# Run a process with low priority (idle)<br> nice -n 19 command && echo "Process started with low priority"<br><br># Run a process with high priority<br> nice -n -5 command && echo "Process started with high priority"Example: Changing Priority of Running Process with renice
# Find the PID of a process<br> ps aux | grep "process_name"<br><br># Change the priority of a running process to normal (0)<br> renice 0 <PID>
Note: Setting real-time priorities requires root or special permissions. Use with caution as it can affect system stability.
7. Using Huge Pages
| Property | Description | Values | Notes |
|---|---|---|---|
| /proc/sys/vm/nr_hugepages | Number of huge pages to allocate | 0 to max_hugepages |
Set via sysctl or by writing to file |
| /proc/sys/vm/hugetlb_shm_group | Group ID allowed to use huge pages in shared memory | 0 to 65535 |
Default is 0, meaning no group restrictions |
| /proc/sys/vm/nr_overcommit_hugepages | Number of huge pages to overcommit | 0 or 1 |
Enables overcommitting huge pages (default: 0) |
Note: Huge pages must be enabled in the kernel and supported by hardware.
Warning: Allocating too many huge pages can lead to memory exhaustion.
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| vm.nr_hugepages | Number of huge pages to allocate at boot | 0 |
Can be set in /etc/sysctl.conf |
| vm.hugetlb_shm_group | Group ID allowed to use huge pages in shared memory | 0 |
Set to 1000 for group users |
8. Optimizing Network Stack
| Parameter | Value | Description |
|---|---|---|
| net.core.rmem_max | 134217728 |
Maximum receive socket buffer size in bytes |
| net.core.wmem_max | 134217728 |
Maximum send socket buffer size in bytes |
| net.ipv4.tcp_rmem | 4096 87380 134217728 |
Min, default, and max TCP receive buffer sizes |
| net.ipv4.tcp_wmem | 4096 65536 134217728 |
Min, default, and max TCP send buffer sizes |
| net.ipv4.tcp_congestion_control | bbr |
Enabled congestion control algorithm (BBR recommended) |
| net.ipv4.tcp_mtu_probing | 1 |
Enable MTU probing to avoid fragmentation |
Example: Apply network optimizations
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sudo sysctl -w net.ipv4.tcp_wmem="4<code> 65536 134217728"
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.ipv4.tcp_mtu_probing=1
Note: These settings can significantly improve performance for high-bandwidth or low-latency applications. Apply with caution and test in a staging environment.
9. Disabling Unnecessary Services
| Service | Description | Status | Command |
|---|---|---|---|
| bluetooth | Bluetooth service for wireless device connectivity | DEPRECATED | sudo systemctl disable bluetooth |
| cups | Common Unix Printing System for printer management | NEW | sudo systemctl disable cups |
| avahi-daemon | Service discovery for local network services | DEPRECATED | sudo systemctl disable avahi-daemon |
| nfs-server | Network File System server for shared directories | NEW | sudo systemctl disable nfs-server |
Note: Disabling services reduces system overhead and improves security by minimizing attack surface.
10. Analyzing Performance with perf
| Command | Description | Options | Example |
|---|---|---|---|
| perf record | Captures performance data | -o <output-file>, -e <event> |
perf record -o perf.data -e cpu-cycles ./myapp |
| perf report | Displays performance data in a readable format | -i <input-file>, --show-total-period |
perf report -i perf.data |
| perf stat | Collects performance statistics | -e <event>, -r <runs> |
perf stat -e cpu-cycles,cache-misses ./myapp |
Note: The
perf tool is part of the Linux kernel's performance analysis suite and requires root privileges or appropriate capabilities to run on some systems.
11. Profiling Applications
| Tool | Description | Usage | Notes |
|---|---|---|---|
| perf | Linux kernel performance profiler | perf record -g <command> |
Can profile CPU, memory, and context switches |
| strace | Trace system calls made by a process | strace -c <command> |
Useful for identifying I/O bottlenecks |
| htop | Interactive process viewer | htop |
Real-time monitoring of CPU and memory usage |
| top | System monitor displaying processes | top |
Basic utility for process monitoring |