Troubleshooting Runtime Issues

1. Resolving Port Conflicts

Port	Service	Conflict Resolution
80	HTTP Default	Stop process using port: `sudo lsof -i :80` and kill with `kill -9 <PID>`
443	HTTPS Default	Identify and terminate conflicting service: `sudo netstat -tulnp \| grep :443`
8080	Alternative HTTP	Change Apache config to use another port in `httpd.conf`: `Listen 8081`

Example: Change Default Port

sudo nano /etc/httpd/conf/httpd.conf

# Find and change:
Listen 80
# to:
Listen 8080

Note: Ensure firewall rules allow traffic on new port.

2. Fixing Permission Issues

Issue	Description	Solution
File/Directory Permissions	Apache cannot read or write files due to restrictive permissions.	Ensure the web server user (e.g., `www-data`) has appropriate read/write access to files and directories.
SELinux/AppArmor	Security modules may block Apache from accessing resources.	Check and adjust SELinux/AppArmor policies to allow Apache access. Use `sestatus` or `aa-status` to verify.
User/Group Ownership	Files are owned by a different user than the one running Apache.	Change ownership using `chown` to match the web server user and group (e.g., `www-data:www-data`).

Example: Setting Correct Permissions

sudo chown -R www-data:www-data /var/www/html
sudo chmod -R 755 /var/www/html

Note: Always test file permissions with ls -l to ensure the web server user can read and write as needed.

Warning: Avoid setting overly permissive permissions (e.g., 777) as they pose a security risk.

3. Debugging SSL/TLS Certificate Issues

Issue	Description	Solution
Certificate not trusted	Certificate is not issued by a recognized CA	Ensure certificate is signed by a trusted CA or install custom CA in browser
Certificate expired	SSL certificate has passed its validity date	Renew and install a new certificate before expiration
Hostname mismatch	Certificate does not match the server's domain name	Ensure certificate includes correct Subject Alternative Names (SANs)
Intermediate certificate missing	Server does not send intermediate certificates	Configure Apache to include the full certificate chain in `SSLCertificateChainFile`
Self-signed certificate	Using a self-signed certificate without trust setup	Install the self-signed certificate in client trust stores or use a CA-signed one

Example: Check certificate validity using OpenSSL

openssl x509 -in /path/to/cert.pem -text -noout

Example: Verify certificate chain in Apache

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /path/to/server.crt
    SSLCertificateKeyFile /path/to/server.key
    SSLCertificateChainFile /path/to/intermediate.crt
</VirtualHost>

Note: Use openssl s_client -connect example.com:443 to inspect the full certificate chain from the server.

Warning: Misconfigured certificate chains can lead to mixed content errors or browser warnings.

4. Resolving Virtual Host Conflicts

Conflicting Directive	Description	Solution
ServerName	Defines the primary server name for a virtual host	Ensure each virtual host has a unique `ServerName` or `ServerAlias`
Listen	Specifies the IP address and port to listen on	Use unique `Listen` directives for each virtual host if binding to different ports
DocumentRoot	Points to the directory from which files are served	Assign distinct `DocumentRoot` paths to avoid file access conflicts

Example: Conflicting Virtual Host Setup

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/example
</VirtualHost>

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/another
</VirtualHost>

Note: The second virtual host will override the first one due to matching ServerName and port.

Example: Resolved Virtual Host Setup

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/example
</VirtualHost>

<VirtualHost *:80>
    ServerName another.example.com
    DocumentRoot /var/www/another
</VirtualHost>

Steps to Resolve Conflicts

Identify overlapping ServerName or ServerAlias entries
Assign unique server names to each virtual host
Verify that Listen directives do not conflict unless intended
Ensure DocumentRoot paths are distinct per virtual host

5. Debugging URL Rewrite Rules

Directive	Description	Example
RewriteLog	Specifies the path to the rewrite log file	`RewriteLog /var/log/apache2/rewrite.log`
RewriteLogLevel	Sets the verbosity level of rewrite logging (0-9)	`RewriteLogLevel 3`

Example: Enable Rewrite Logging

RewriteLog /var/log/apache2/rewrite.log
RewriteLogLevel 3

Note: Rewrite logging can significantly impact performance. Use with caution in production.

Log Level	Description	Use Case
`0`	No logging	Production environments
`1`	Basic info	General troubleshooting
`3`	Verbose info	Detailed rule debugging
`5`	Full debugging	Complex rewrite rule issues

Example: Debugging a Rewrite Rule

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteLogLevel 5
    RewriteLog /var/log/apache2/rewrite.log

    RewriteRule ^old-page\.html$ /new-page.html [R=301,L]
</IfModule>

Warning: Ensure proper permissions on the log file to prevent security issues.

Debugging Steps

Enable RewriteLog and set appropriate RewriteLogLevel
Check the rewrite log for rule matching and processing steps
Verify rule patterns match expected URLs
Use RewriteCond to test conditions before rules

6. Analyzing Core Dumps

Example: Enabling core dump generation

# Enable core dumps
ulimit -c unlimited

# Set core dump location
echo '/tmp/core.%e.%p.%t' &gt; /proc/sys/kernel/core_pattern

# Restart Apache to apply settings
systemctl restart apache2

Directive	Description	Default	Notes
CoreDumpDirectory	Directory to store core dumps	`/var/crash`	Must be writable by Apache user
ServerLimit	Maximum number of server processes	`256`	Used in core dump naming

Note: Core dumps are generated when Apache crashes due to segmentation faults or other critical errors.

Example: Analyzing core dump with GDB

# Load core dump with GDB
gdb /usr/sbin/apache2 /tmp/core.apache2.12345.1234567890

# Get backtrace
(gdb) bt

# Examine specific frame
(gdb) frame 5
(gdb) info registers

Core Dump Analysis Steps

Enable core dump generation using ulimit -c unlimited
Set core pattern to capture relevant information
Restart Apache to apply configuration changes
Use GDB to analyze the core dump file
Examine backtrace and register state for crash root cause

7. Using strace for System Call Tracing

Example: Basic strace usage to trace Apache process

# Trace system calls made by Apache process
strace -p &lt;apache_pid&gt;

# Trace with output to file
strace -p &lt;apache_pid&gt; -o /tmp/apache_trace.log

# Trace specific system calls
strace -p &lt;apache_pid&gt; -e trace=open,close,read,write

Option	Description	Notes
-p <pid>	Attach to running process by PID	Must have sufficient permissions
-o <file>	Write trace output to file	Useful for post-analysis
-e trace=<syscalls>	Trace specific system calls	Reduce noise in output

Note: strace can significantly impact performance and should be used with caution in production environments.

Warning: Ensure you have appropriate permissions to trace processes and that sensitive data is not exposed in the trace output.

8. Debugging Performance Issues

Issue	Description	Solution
High CPU Usage	Server consumes excessive CPU resources during peak load.	Use `mod_status` to identify slow requests and optimize code or configuration.
Memory Leaks	Memory usage increases over time without release.	Monitor with `mod_status` and `top` or `htop`. Restart server if necessary.
Slow Response Times	Requests take longer than expected to process.	Enable `LogLevel debug` and review logs for bottlenecks.

Example: Enabling Performance Debugging

LogLevel debug

Example: Using mod_status for Monitoring

<Location /server-status>
    SetHandler server-status
    Require ip 127.0.0.1
</Location>

Note: Use mod_status to monitor real-time server performance and identify problematic modules or configurations.

Warning: Excessive LogLevel debug can severely impact performance and fill logs quickly.

9. Resolving Memory Leaks

Issue	Description	Solution
Memory usage increasing over time	Indicates potential memory leak in modules or configurations	Use `mod_status` to monitor memory usage and identify problematic modules
High RAM consumption in long-running processes	Can lead to system instability or crashes	Implement `MaxRequestWorkers` and `ServerLimit` to limit worker processes

Example: Monitoring Memory Usage

curl -s http://localhost/server-status?auto | grep -E "Total kBytes|IdleWorkers"

Example: Configuring Memory Limits

<IfModule mpm_prefork_module>
    StartServers 5
    MinSpareServers 5
    MaxSpareServers 10
    ServerLimit 100
    MaxRequestWorkers 100
</IfModule>

Note: Use mod_status to monitor memory usage and identify modules causing leaks.

Warning: Excessive memory usage can lead to system instability or crashes.

10. Fixing Process Spawn Issues

Issue	Symptoms	Possible Causes	Solutions
Child process not starting	Server logs show no child processes, high CPU usage on parent	Permission denied, missing binaries, resource limits	Check file permissions, verify binary paths, increase ulimit
Child process crashing immediately	Repeated child process exits, error logs show segmentation fault or abort	Memory corruption, incompatible module versions, bad config	Run with gdb to debug, update modules, validate config files
High child process count	Too many child processes, server becomes unresponsive	Improperly configured MaxRequestWorkers or KeepAlive settings	Adjust MaxRequestWorkers, enable KeepAlive, tune timeouts

Example: Checking Process Limits

ulimit -u

Example: Verifying Binary Path

which httpd

Note: Ensure that the user running Apache has sufficient permissions to spawn child processes and access required binaries.

Warning: Increasing MaxRequestWorkers too high can lead to system instability or out-of-memory errors.

11. Debugging CGI Script Failures

Issue	Description	Solution
Script not executing	CGI script returns no output or errors	Ensure script has proper shebang line and executable permissions
500 Internal Server Error	Server fails to execute CGI script	Check error logs for detailed messages; verify script path and permissions
Incorrect output format	CGI script does not send proper headers	Always send `Content-Type: text/html` before any output

Example: Basic CGI Script

#!/bin/bash
echo "Content-Type: text/html"
echo ""
echo "<html><body><h1>Hello World</h1></body></html>"

Note: Ensure the script is executable (chmod +x script.cgi) and located in a directory configured for CGI execution.

12. Resolving Proxy Backend Connection Issues

Issue	Description	Solution
ProxyPass connection timeout	Backend server does not respond within the configured timeout.	Adjust `ProxySet` with `timeout` parameter or increase `ProxyTimeout`.
Connection refused	Backend server is not running or listening on the specified port.	Verify backend service status and ensure it's binding to the correct IP/port.
502 Bad Gateway	Proxy received an invalid response from backend server.	Check backend server logs for errors and ensure it returns valid HTTP responses.
503 Service Unavailable	Backend server is temporarily unable to handle the request.	Implement retry logic or configure `ProxySet` with `retry` and `retries`.

Example: Configuring Proxy Timeout

<Proxy *>
    ProxySet timeout=30
</Proxy>

ProxyPass /backend/ http://192.168.1.100:8080/
ProxyPassReverse /backend/ http://192.168.1.100:8080/

Note: Use ProxyTimeout directive to set a global timeout for all proxy connections.

Warning: Setting timeouts too low can cause premature connection drops; setting them too high may delay error detection.