Managing Traditional System Logs
1. Understanding Syslog Architecture
| Component | Description | Role |
|---|---|---|
| syslogd | /usr/sbin/syslogd |
Daemon responsible for receiving, filtering, and logging messages |
| rsyslogd | /usr/sbin/rsyslogd |
Enhanced version of syslogd with support for advanced features like TCP, TLS, and database output |
| syslog.conf | /etc/rsyslog.conf or /etc/syslog.conf |
Configuration file that defines log message routing rules and facilities |
| Log Facilities | auth, daemon, local0 to local7 |
Classes of messages that can be logged (e.g., authentication, system daemons) |
| Log Priorities | emerg, alert, crit, err, warn, notice, info, debug |
Severity levels indicating importance of log messages |
Example: Basic rsyslog Configuration
# /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none /var/log/messages
Note: Modern Linux systems typically use
rsyslogd as the default syslog daemon, offering more features than the basic syslogd.
Key Concepts in Syslog Architecture
- Facilities categorize the source of log messages (e.g.,
auth,daemon) - Priorities define the severity of messages (e.g.,
info,error) - Rules in configuration files determine where messages are logged (e.g., to files, remote servers)
2. Reading Log Files
| Log File | Description | Common Use Case |
|---|---|---|
| /var/log/syslog | System log containing messages from the kernel and system services | General system diagnostics and error tracking |
| /var/log/messages | General system messages and warnings (used in some distributions) | System-wide event monitoring |
| /var/log/auth.log | Authentication logs including login attempts and privilege changes | Security audit and user access tracking |
| /var/log/kern.log | Kernel messages and system boot logs | Boot issues and kernel-level errors |
| /var/log/boot.log | Boot process logs and startup messages | System boot diagnostics |
Note: Ensure you have appropriate permissions to read log files. Some logs may require
sudo access.
3. Viewing Authentication Logs
| Log File | Description | Common Commands |
|---|---|---|
| /var/log/auth.log | Records authentication-related events such as login attempts, user sessions, and privilege changes. | tail -f /var/log/auth.log |
| /var/log/syslog | General system log containing a wide range of system messages including authentication info. | grep "auth" /var/log/syslog |
| /var/log/secure | Used on some systems (e.g., RHEL/CentOS) to store authentication logs. | journalctl -u ssh.service |
Example: Monitor Real-Time Authentication Events
tail -f /var/log/auth.log | grep "Accepted\|Failed"
Note: On systems using systemd,
journalctl can be used to view logs from the system journal.
Warning: Ensure you have appropriate permissions to read authentication logs, as they may contain sensitive information.
4. Viewing System Messages
| Command | Description | Notes |
|---|---|---|
| dmesg | Displays kernel ring buffer messages | Useful for hardware and driver issues |
| journalctl | Query the systemd journal for logs | Most modern Linux systems use systemd |
| tail -f /var/log/syslog | Live view of system log file | Classic syslog format |
| tail -f /var/log/messages | Live view of system messages log | Used in some distributions like CentOS/RHEL |
Note: On systems using systemd,
journalctl is the preferred method for viewing logs.
Warning: Logs can contain sensitive information. Ensure appropriate permissions and access controls are in place.
5. Following Logs with tail
| Option | Values | Description |
|---|---|---|
| -f | --follow |
Output appended data as the file grows |
| -n | --lines |
Output the last N lines of the file |
| -s | --sleep-interval |
Number of seconds to sleep between polls (default: 1) |
| -v | --verbose |
Output a message when a file is opened or reopened |
Note: The
-f option is particularly useful for monitoring logs during system events or application debugging.
6. Configuring rsyslog
| Directive | Description | Example |
|---|---|---|
| action(type="omfile" file="/var/log/myapp.log") | Specifies the output file for log messages | action(type="omfile" file="/var/log/myapp.log") |
| template(name="myTemplate" type="string" string="%timestamp% %syslogtag% %msg%") | Defines a custom log message format template | template(name="myTemplate" type="string" string="%timestamp% %syslogtag% %msg%") |
| ruleset(name="myRuleset") | Creates a named ruleset for organizing log processing | ruleset(name="myRuleset") { ... } |
Example: Basic rsyslog configuration
# /etc/rsyslog.conf
*.* /var/log/all_messages.log
# Custom template
template(name="CustomFormat" type="string" string="%timestamp% %syslogtag% %msg%\n")
# Use custom template
*.info /var/log/info.log;CustomFormat
Example: Filtering logs by facility and priority
# /etc/rsyslog.conf
# Log only authpriv messages to a separate file
authpriv.* /var/log/auth.log
# Log all mail messages with priority info or higher to a file
mail.info /var/log/mail.log
# Log all messages from local0 facility to a file
local0.* /var/log/local0.log
Note: The rsyslog configuration file is typically located at
/etc/rsyslog.conf or /etc/rsyslog.d/. Changes require restarting the rsyslog service to take effect.
7. Rotating Logs
| Option | Values | Description |
|---|---|---|
| /etc/logrotate.conf | /etc/logrotate.conf |
Primary configuration file for log rotation |
| logrotate | /usr/sbin/logrotate |
Command to rotate logs based on configuration |
| maxsize | 10M |
Maximum size before rotation occurs |
| rotate | 5 |
Number of rotated log files to keep |
| compress | true |
Compress rotated log files with gzip |
Example: Basic logrotate configuration
/var/log/myapp.log {
daily
size 10M
rotate 5
compress
missingok
}
Note: The
logrotate utility uses a configuration file to define how logs should be rotated. It supports various rotation strategies like daily, weekly, monthly, or size-based.
Warning: Ensure that the log files are not being actively written to during rotation, or it may cause data loss or corruption.
8. Configuring Log Rotation
| Configuration File | Path | Description |
|---|---|---|
| /etc/logrotate.conf | /etc/logrotate.conf |
Primary configuration file for log rotation settings |
| /etc/logrotate.d/ | /etc/logrotate.d/ |
Directory containing individual log rotation configuration files |
Example: Basic logrotate configuration for a web server
/var/log/nginx/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 644 nginx nginx
}| Directive | Description | Values |
|---|---|---|
| daily | Schedule log rotation daily | daily, weekly, monthly |
| rotate | Number of rotated logs to keep | Integer (e.g., 52) |
| compress | Compress rotated logs | true, false |
| delaycompress | Delay compression of the previous log file | true, false |
| create | Create new log file with specified permissions | 644 nginx nginx |
Note: The
logrotate utility is typically run by cron daily, usually via /etc/cron.daily/logrotate.
9. Sending Logs to Remote Server
| Configuration Option | Description | Example Value |
|---|---|---|
| syslog-ng | Centralized logging system that supports remote log forwarding | destination(d_remote { tcp("192.168.1.100" port(514)); }); |
| rsyslog | Advanced logging daemon supporting remote logging via modules | $ActionRemoteForwardMode TCP |
| systemd-journald | Journal logging service that can forward logs to remote servers | ForwardToSyslog=yes |
Note: Ensure that the remote server is configured to accept incoming logs and that network access is allowed.
10. Using logger Command
| Option | Values | Description |
|---|---|---|
| -p priority | auth|authpriv|cron|daemon|ftp|kern|lpr|mail|mark|news|syslog|user|uucp|local0|local1|local2|local3|local4|local5|local6|local7 |
Set the priority level of the log message |
| -t tag | string |
Set the tag for the log message |
| -i | flag |
Log the process ID (PID) of the logger command |
| -n | flag |
Log to the local system log (default) |
| -f | flag |
Log to the console (stdout) instead of system log |
Example: Logging a message with logger
logger -p local0.info "Application started successfully"
logger -t "myapp" -p user.notice "User login detected"
Note: The
logger command is part of the NEW systemd logging stack and is preferred over direct writes to /var/log/messages.