Systemd journal corruption on Fedora
I always keep a terminal window open to monitor system logs in real-time when I’m in front of my computer. Therefore, it immediately caught my attention when the systemd journal offered zero new lines of output. The most recent entry simply contained information about the previous shutdown.
My experience with troubleshooting enterprise Linux on the desktop has taught me to always reboot the system before addressing any underlying issue. But alas, and to my surprise, a reboot did not bring the journal service back to life. However, systemctl status systemd-journald.service did insist that the journal was running just fine.
The consistency check
After browsing man journalctl I found that the --verify command can check the journal file for internal consistency. Running journalctl --verify returned several pages detailing what appeared to be severe file corruption:
Checking the journal files for internal consistency (machine ID has since been changed).
Fixing a corrupt systemd journal file
With the problem identified, the fix should have been (in my humble opinion) something along the lines of journalctl --repair for fsck sake. However, there is no “fix” for corrupted journal files. In the end, I simply opted to purge them all.
sudo systemctl stop systemd-journald.service
sudo rm /var/log/journal/machine-id/*
sudo systemctl start systemd-journald.service
That quickly got the journal back up and running, but the following ominous messages started pouring out from the Automatic Bug Reporting Tool (ABRT):
abrt-server[7813]: Preserving oops '.' because DropNotReportableOopses is 'no'
abrt-notification[7830]: [🡕] System encountered a non-fatal error in ??()
abrt-dump-journal-oops[1021]: Reported 1 kernel oopses to Abrt
abrt-server[7832]: Can't find a meaningful backtrace for hashing in '.'
abrt-server[7832]: Preserving oops '.' because DropNotReportableOopses is 'no'
abrt-notification[7849]: [🡕] System encountered a non-fatal error in ??()
abrt-dump-journal-oops[1021]: Reported 1 kernel oopses to Abrt
abrt-server[7851]: Can't find a meaningful backtrace for hashing in '.'
abrt-server[7851]: Preserving oops '.' because DropNotReportableOopses is 'no'
hail-satan abrt-notification[7871]: [🡕] System encountered a non-fatal error in ??()
Amusingly, there was no end in sight to these messages so I had to stop the service with sudo systemctl stop abrtd.service.
Ex’s and the oops, oops, oopses they haunt me. Like gho-o-osts.
I fondly remember simpler times when log files were written in plain text /0\