Linux Log Server and Log File Management: rsyslog Complete Guide

Log management is the foundation of troubleshooting, security auditing, and compliance. Linux systems generate extensive logs about every aspect of system operation. Understanding where logs come from, how to configure centralized logging, and how to use journalctl for real-time analysis is essential for RHCA administrators.

Linux Logging Architecture

Modern Linux uses two logging systems in parallel:

• rsyslog: The traditional text-file based logging daemon. Receives messages via the /dev/log socket, filters by facility and severity, and writes to files in /var/log/.
• systemd-journald: The modern logging daemon for systemd. Captures all stdout/stderr from systemd services, kernel messages, and syslog messages. Stores in binary format with fast querying.

In RHEL 7, both run simultaneously — journald collects everything, and rsyslog reads from journald and writes to traditional log files.

rsyslog — Traditional Logging

Facility and Severity

rsyslog messages have two dimensions: the facility (source/category) and the severity (importance level):

# /etc/rsyslog.conf rule format:
# facility.severity  destination

# Examples:
kern.crit           /var/log/kernel.log      # kernel critical errors to file
auth.*              /var/log/secure          # all auth messages
mail.info           /var/log/maillog         # mail info and above
*.emerg             :omusrmsg:*              # emergency messages to all users
cron.*              /var/log/cron            # all cron messages
*.info;mail.none;authpriv.none  /var/log/messages  # all info except mail/auth

# Send to remote log server:
*.* @192.168.1.11:514             # UDP (less reliable)
*.* @@192.168.1.11:514            # TCP (more reliable)

# Severity modifiers:
# facility.severity    = this severity AND ALL HIGHER (more critical)
# facility.=severity   = ONLY this severity
# facility.!severity   = NOT this severity (exclude)
# *.*                  = all facilities, all severities

Centralized Log Server Configuration

# On the LOG SERVER:
# Edit /etc/rsyslog.conf
# Uncomment UDP/TCP reception (lines 13-19):

# UDP:
$ModLoad imudp
$UDPServerRun 514

# TCP (more reliable):
$ModLoad imtcp
$InputTCPServerRun 514

# Store client logs in per-host files:
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

# systemctl restart rsyslog
# Verify: ss -tulnp | grep 514

# On each CLIENT:
# vim /etc/rsyslog.conf
# Add at end:
*.* @@192.168.1.11:514             # send all logs to server via TCP

# systemctl restart rsyslog

systemd Journal (journalctl)

# journalctl reads the binary journal maintained by systemd-journald

# Basic usage:
# journalctl                         # all logs, oldest first
# journalctl -r                      # reverse order (newest first)
# journalctl -f                      # follow (like tail -f)
# journalctl -n 50                   # last 50 lines

# ── FILTERING ─────────────────────────────────────────────────
# By service/unit:
# journalctl -u httpd                # logs for httpd service
# journalctl -u httpd -u sshd        # multiple services

# By priority:
# journalctl -p err                  # errors and above (0-3)
# journalctl -p warning              # warnings and above (0-4)
# journalctl -p 4                    # same as warning (numeric)
# journalctl -p 3..7                 # range of priorities

# By time:
# journalctl --since "today"
# journalctl --since "yesterday"
# journalctl --since "2026-06-01"
# journalctl --since "2026-06-01 00:00:00" --until "2026-06-07 23:59:59"
# journalctl --since "-1h"           # last hour
# journalctl --since "-30min"        # last 30 minutes

# By boot:
# journalctl -b                      # current boot
# journalctl -b -1                   # previous boot
# journalctl --list-boots            # all recorded boots

# ── KERNEL MESSAGES ────────────────────────────────────────────
# journalctl -k                      # kernel messages only (dmesg replacement)
# journalctl -k -p err               # kernel errors

# ── SPECIFIC PROCESS/PID ───────────────────────────────────────
# journalctl _PID=1234               # by PID
# journalctl _UID=1001               # by user ID
# journalctl _COMM=sshd              # by command name

# ── OUTPUT FORMATS ─────────────────────────────────────────────
# journalctl -o json                 # JSON format
# journalctl -o json-pretty          # pretty JSON
# journalctl -o short-iso            # ISO timestamps
# journalctl -o cat                  # message only (no metadata)

# ── SIZE AND DISK USAGE ────────────────────────────────────────
# journalctl --disk-usage            # journal disk usage
# journalctl --vacuum-size=500M      # reduce to 500 MB
# journalctl --vacuum-time=30d       # delete logs older than 30 days

Making Journal Persistent Across Reboots

# By default, journal is in /run/log/journal/ — lost on reboot
# To persist:
# mkdir -p /var/log/journal
# chown root:systemd-journal /var/log/journal
# chmod g+s /var/log/journal
# systemctl restart systemd-journald

# Or set in /etc/systemd/journald.conf:
[Journal]
Storage=persistent              # auto, persistent, volatile, or none

Important Log Files Reference

/var/log/messages      # General system log (first place to check any issue)
/var/log/secure        # Authentication: login attempts, su, sudo, sshd
/var/log/audit/audit.log # SELinux denials, auditd events
/var/log/cron          # All cron job executions
/var/log/maillog       # Postfix/sendmail/dovecot messages
/var/log/httpd/        # Apache access and error logs
/var/log/mariadb/      # MariaDB server log
/var/log/samba/        # Samba server logs
/var/log/boot.log      # Boot messages (init/systemd startup)
/var/log/dmesg         # Kernel ring buffer (hardware events, driver loading)
/var/log/lastlog       # Last login time for each user
/var/log/wtmp          # Login/logout history (read with 'last' command)
/var/log/btmp          # Failed login attempts (read with 'lastb' command)
/var/log/yum.log       # Package install/remove history

Log Rotation with logrotate
# logrotate prevents log files from consuming all disk space
# runs daily via /etc/cron.daily/logrotate

# Global defaults: /etc/logrotate.conf
# Per-application configs: /etc/logrotate.d/

# Example config for application logs:
/var/log/myapp/*.log {
    daily                       # rotate daily
    rotate 14                   # keep 14 rotated logs
    compress                    # gzip old logs
    delaycompress               # compress previous, not current (service still writing)
    missingok                   # no error if log file missing
    notifempty                  # don't rotate empty files
    create 0640 myapp myapp    # create new log with these permissions
    sharedscripts               # run postrotate once, not per-file
    postrotate
        /bin/systemctl reload myapp 2>/dev/null || true
    endscript
}

# Test logrotate (dry run):
# logrotate -d /etc/logrotate.d/myapp

# Force rotation now:
# logrotate -f /etc/logrotate.conf