Optimizing Log Storage on Devices with SD Cards

On devices that utilise an SD card for their file system such as the Raspberry Pi, card wear emerges as a significant concern. Linux tends to generate an extensive log of information in /var/log, which, over time, leads to a high volume of disk writes and ultimately shortens the device's lifespan. While this is a well-known issue, the proper and permanent solution is less commonly understood.

Instead of suggesting the use of 'tmpfs' or 'ramfs' to mount at '/var/log', which presents two primary drawbacks:

  1. It consumes system RAM, which can be severely limited on small embedded devices.
  2. It can fill up and impede the proper functioning of services.

A more effective solution can be found within the kernel itself. During the boot process, when no file system is available for logging, the kernel already possesses a ring buffer, a memory-based message buffer. Additionally, if a storage failure occurs, the kernel still requires a location to output informational messages for diagnostic purposes. As the name implies, this buffer operates in a circular manner, overwriting itself from the start when it becomes full.

We can leverage this existing mechanism by configuring 'rsyslog' to write directly to the kernel's ring buffer instead of a file, and the same can be done for the 'systemd' journal.

To redirect all logging to the kernel's ring buffer, replace '/etc/rsyslog.conf' with the following:


# provides support for local system logging
module(load="imuxsock") 

# keep the output short and simple
$template precise,"%timegenerated%|%syslogtag%%msg%\n"
$ActionFileDefaultTemplate precise

# send everything to the kernel's kmsg
*.*    /dev/kmsg

# Configure imuxsock module for local system logging (e.g., via logger command)
input(type="imuxsock" Socket="/dev/log")

To configure the 'systemd' journal to also utilise the kernel's ring buffer, make the following changes in '/etc/systemd/journald.conf':


[Journal]
Storage=none
ForwardToSyslog=yes

After implementing these modifications, either reboot the device or execute the following commands:


systemctl restart rsyslog
systemctl restart systemd-journald

To view the output, use the command dmesg to display the contents of the kernel's ring buffer, or employ dmesg -w to continuously monitor it.

Note that it still may be prudent to mount a `tmpfs` at `/var/log` for services that write here directly.