Linux Log Management: journalctl and /var/log Explained

Logs are the sysadmin's black box recorder. When something breaks, logs tell you what happened and when. Linux stores logs in two places: the traditional /var/log directory (text files) and the systemd journal (binary, structured). This guide covers both, plus practical techniques for searching and managing log data.

The /var/log Directory

Traditional logs are plain text files in /var/log. Key files:

/var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL) — general system log
/var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL) — authentication events
/var/log/kern.log — kernel messages
/var/log/dpkg.log — package install/remove history (Debian/Ubuntu)
/var/log/nginx/access.log — web server requests
/var/log/nginx/error.log — web server errors
/var/log/mysql/error.log — database errors

Reading Log Files

tail -f /var/log/syslog                          # Follow live
tail -100 /var/log/auth.log                      # Last 100 lines
less /var/log/syslog                             # Paginated reading
grep "sshd" /var/log/auth.log                    # Filter for SSH events
grep "Failed password" /var/log/auth.log         # Failed logins
grep -i "error" /var/log/nginx/error.log         # Nginx errors
zcat /var/log/syslog.2.gz                        # Read compressed log

journalctl — Reading the systemd Journal

The systemd journal captures all output from systemd-managed services, the kernel, and the boot process.

journalctl                              # All logs (use arrow keys or PgUp/PgDn)
journalctl -b                           # Current boot only
journalctl -b -1                        # Previous boot
journalctl -b -2                        # Boot before that
journalctl -b --list-boots              # Show all recorded boots

Filtering by Service

journalctl -u nginx                     # All nginx logs
journalctl -u nginx -b                  # nginx logs from current boot
journalctl -u nginx -f                  # Follow nginx logs live
journalctl -u ssh -n 50                 # Last 50 SSH log lines
journalctl -u mysql -u nginx            # Multiple services at once

Filtering by Time

journalctl --since "2026-04-10 08:00:00"
journalctl --since "2026-04-10" --until "2026-04-11"
journalctl --since "1 hour ago"
journalctl --since "yesterday"

Filtering by Priority

journalctl -p err                       # Error and above
journalctl -p warning                   # Warning and above
journalctl -p debug                     # All messages including debug
journalctl -u nginx -p err              # Nginx errors only

Priority levels: emerg, alert, crit, err, warning, notice, info, debug.

Useful Output Formats

journalctl -u nginx -o json             # JSON output
journalctl -u nginx -o short-iso        # ISO timestamp format
journalctl -u nginx --no-pager          # No pagination (pipe to grep)

logrotate — Rotating and Compressing Logs

Without rotation, log files grow forever. logrotate handles compression and pruning automatically.

cat /etc/logrotate.conf               # Global settings
ls /etc/logrotate.d/                  # Per-app configurations
cat /etc/logrotate.d/nginx

Example logrotate config for a custom app:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl reload myapp
    endscript
}

sudo logrotate -d /etc/logrotate.d/myapp    # Dry run (debug)
sudo logrotate -f /etc/logrotate.d/myapp    # Force immediate rotation

Managing Journal Size

journalctl --disk-usage                 # Journal disk usage
sudo journalctl --vacuum-size=500M      # Keep only 500MB of journal
sudo journalctl --vacuum-time=30d       # Keep only last 30 days

Persistent journal storage config: set Storage=persistent in /etc/systemd/journald.conf. By default, the journal may be volatile (lost on reboot).

Practical Log Investigation Workflow

# 1. Check recent system errors
journalctl -p err -b

# 2. Check specific service
journalctl -u nginx -n 100

# 3. Check for failed login attempts
grep "Failed password" /var/log/auth.log | awk "{print $11}" | sort | uniq -c | sort -rn

# 4. Watch live while reproducing an issue
journalctl -u myapp -f

# 5. Check kernel for disk or hardware errors
journalctl -k | grep -i "error|fail|warn"

Summary

Effective log management separates reactive sysadmins from proactive ones. Know where logs live, how to filter them efficiently, and how to keep them from eating your disk. journalctl is your primary tool for systemd services; /var/log files cover applications and the kernel. Set up logrotate for any app that writes its own log files.