Configuring Persistent journald Log Storage on Linux

systemd-journald is the default logging system on modern Linux, but by default it may not persist logs across reboots. Configuring persistent storage ensures you retain logs for debugging and auditing. I set this up on my production servers in Lisbon after losing critical logs during a reboot.

Environment

• OS: Ubuntu 22.04 LTS
• journald version: systemd 249
• Goal: Persistent journal storage with 1GB size limit

$ systemctl --version
systemd 249 (249.11-0ubuntu3.12)

$ journalctl --disk-usage
Archived and active journals take up 156.0M in the file system.

The Problem

# Check current storage mode
$ journalctl --header | grep "File path"
File path:             /run/log/journal/...
# /run/log/journal is tmpfs - cleared on reboot!

# Check persistent directory
$ ls -la /var/log/journal/
ls: cannot access '/var/log/journal/': No such file or directory

Analysis

Step 1: Check Current journald Configuration

$ cat /etc/systemd/journald.conf
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#Mode=0640
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1day
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes

Step 2: Check Storage Mode

# Current storage mode
$ systemd-analyze cat-config systemd/journald.conf | grep Storage
#Storage=auto

# Check where logs are stored
$ journalctl --header | grep -E "(File path|File mmap)"
File path:             /run/log/journal/abcdef1234567890/system.journal
# Running in volatile mode (tmpfs)

Solution

Fix 1: Enable Persistent Storage

# Create persistent directory
$ sudo mkdir -p /var/log/journal

# Restart journald
$ sudo systemctl restart systemd-journald

# Verify
$ journalctl --header | grep "File path"
File path:             /var/log/journal/abcdef1234567890/system.journal

# Check storage is now persistent
$ ls -la /var/log/journal/
total 0
drwxr-xr-x 3 root root 4096 May  8 10:00 .
drwxr-xr-x 1 root root 4096 May  8 10:00 ..
drwxr-xr-x 3 root root 4096 May  8 10:00 abcdef1234567890

Fix 2: Configure Size Limits

$ sudo nano /etc/systemd/journald.conf

[Journal]
Storage=persistent
Compress=yes
Seal=yes
SplitMode=uid
Mode=0640

# Size limits
SystemMaxUse=1G          # Maximum disk usage
SystemKeepFree=2G        # Minimum free space to keep
SystemMaxFileSize=128M   # Maximum size per journal file
SystemMaxFiles=10        # Maximum number of journal files

# Retention
MaxRetentionSec=30day    # Keep logs for 30 days
MaxFileSec=1day          # Rotate daily

# Forwarding
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes

Fix 3: Restart and Verify

# Restart journald
$ sudo systemctl restart systemd-journald

# Verify configuration
$ systemd-analyze cat-config systemd/journald.conf | grep -E "(Storage|SystemMaxUse)"
Storage=persistent
SystemMaxUse=1G

# Check disk usage
$ journalctl --disk-usage
Archived and active journals take up 156.0M in the file system.

# Test log persistence across reboot
$ sudo reboot
# After reboot
$ journalctl --disk-usage
Archived and active journals take up 156.0M in the file system  # Logs preserved!

Fix 4: Manual Cleanup

# Vacuum by size
$ sudo journalctl --vacuum-size=500M

# Vacuum by time
$ sudo journalctl --vacuum-time=30d

# Vacuum by files
$ sudo journalctl --vacuum-files=5

# Check after cleanup
$ journalctl --disk-usage
Archived and active journals take up 487.2M in the file system.

Useful journald Commands

# View logs
$ journalctl -u nginx          # Logs for nginx service
$ journalctl -f                # Follow (tail) logs
$ journalctl --since today     # Today's logs
$ journalctl -p err            # Error priority and above

# Disk usage
$ journalctl --disk-usage

# Verify persistent storage
$ journalctl --header | grep "File path"
File path:             /var/log/journal/.../system.journal

# List booted journals
$ journalctl --list-boots
-3 12345678-1234-1234-1234-123456789012 Sun 2025-05-01 10:00:00 WET—Thu 2025-05-08 09:59:59 WET
-2 23456789-2345-2345-2345-234567890123 Thu 2025-05-08 10:00:00 WET—Fri 2025-05-09 10:00:00 WET
-1 34567890-3456-3456-3456-345678901234 Fri 2025-05-09 10:00:00 WET—Sat 2025-05-10 10:00:00 WET
 0 45678901-4567-4567-8901-456789012345 Sat 2025-05-10 10:00:00 WET—now

Monitoring Log Usage

#!/bin/bash
# /usr/local/bin/journald-monitor.sh

USAGE=$(journalctl --disk-usage | grep -o '[0-9.]*[MG]' | head -1)
THRESHOLD="1G"

echo "Current journald usage: $USAGE"

# Check if exceeds threshold
if [[ "$USAGE" > "$THRESHOLD" ]]; then
    echo "WARNING: Journald usage exceeds $THRESHOLD"
    journalctl --vacuum-size=800M
fi

Troubleshooting

Logs Not Persisting

# Check if directory exists
$ ls -la /var/log/journal/
# If not: sudo mkdir -p /var/log/journal

# Check permissions
$ sudo chmod 755 /var/log/journal
$ sudo systemctl restart systemd-journald

# Verify
$ journalctl --header | grep "File path"
File path:             /var/log/journal/.../system.journal

Disk Space Issues

# Check disk usage
$ df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       20G   15G    5G  75% /var/log

# Vacuum old logs
$ sudo journalctl --vacuum-size=500M
$ sudo journalctl --vacuum-time=30d

Lessons Learned

1. Always enable persistent storage for production servers - volatile logs are lost on reboot.
2. Set size limits to prevent journal from consuming all disk space.
3. Use journalctl --vacuum-size regularly to manage disk usage.
4. Monitor journal usage as part of your disk monitoring strategy.
5. Keep at least 7 days of logs for debugging production issues.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.