System logs in Debian/Raspbian

Notes on how to minimize writes to the persistant storage by system logging and keep system logs within a safe size limit. This is tested on Debian/Raspbian 10 Buster but should be generally applicable on all Debian-based distributions with systemD.

Check current size of syslog dir: du -sh /var/log and/or df -h

Check current status (with sizes) of systemd journal logs: systemctl status systemd-journald

log2ram

| github page |

Simple, tested and true solution to keep current logs in RAM (tmpfs filesystem) with a daily copy to persistent storage (/var/hdd.log) + on a clean reboot/poweroff.

Check current usage of /var/log to be able to configure SIZE= properly: sudo du -sh /var/log/

Enable rsync option if rsync is available on the system.

Check status: systemctl status log2ram

Check associated files: sudo find / -iname '*log2ram*'

systemd unit file: /etc/systemd/system/log2ram.service

If /var/log become full, there are two main solutions:

• Increase available size; edit /etc/log2ram.conf; item SIZE=, and reboot.
• Tighten rules for logrotate.

 

Logrotate

| debian man page |

Works primarily on syslog files in /var/log dir.

Main config file: /etc/logrotate.conf

Dir for included config files: /etc/logrotate.d/

Change rotation to be kicked off by size rather than time.

Logrotate timing

Timing is set in the service unit file: /lib/systemd/system/logrotate.timer This is the minimum effective time interval even if a smaller interval are configured for individual log files.

To (re)read the unit files: systemctl daemon-reload

Check service status: systemctl status logrotate.timer

There is also a file in /etc/cron.daily/logrotate but it does nothing if systemd timers are used (default).

 

systemd journal logs

| debian journald.conf man page | loggly.com guide | manjoro forum post |

Check status: systemctl status systemd-journald

Main configuration file: /etc/systemd/journald.conf

Check if journald is in /run/log (and total size): sudo du -sh /run/log/journal

Items if journald logs reside in /var/log (default in Ubuntu 18.04):

SystemMaxUse=14M
SystemKeepFree=22M
SystemMaxFileSize=10M

Items if journald logs reside in /run/log (default in Raspbian 10 and in Ubuntu 16.04):

RuntimeMaxUse=12M
RuntimeKeepFree=20M
RuntimeMaxFileSize=8M

Items to reduce log noice:

MaxLevelStore=warning
MaxLevelSyslog=warning
MaxLevelKMsg=warning
MaxLevelConsole=notice
MaxLevelWall=crit

Reload configuration in unit files: systemctl force-reload systemd-journald or reboot.

 

Moving systemd journal

If /var/log is moved to tmpfs filesystem (with log2ram), then it might be useful to activate /var/log/journal over /run/log/journal. Guide on freedesktop.org.

Move /run/log/journal to /var/log/journal:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

Check result with: ls -l /var/log | grep journal

drwxr-sr-x+ 3 root   systemd-journal      60 jan  9 07:06 journal

Reboot.

 

Kommentarer till sidan System logs

---