Linux Logging with journalctl: Filters, Fields, and Forwarding

What and Why

On systemd Linux, journald is the kernel-side and userspace log collector. It captures stdout/stderr from every service, kernel messages, audit records, and structured fields like the unit name, PID, and exit code. journalctl is the query tool. Master it and you stop grepping plain log files for almost everything.

The reason to bother: structured queries are far faster and more accurate than text-grep. You can find every error from nginx.service in the last 10 minutes across all log streams with one command.

Mental Model

Sources                                Storage
-------                                -------
kernel  --+
syslog  --+--> journald --> in-memory + /var/log/journal/*
stdout  --+                       |
stderr  --+                       +--> forward to syslog
audit   --+                       +--> forward to remote (network)

Query: journalctl filter & format

Each entry has fields you can filter on: _SYSTEMD_UNIT, _PID, _HOSTNAME, PRIORITY, _BOOT_ID. Filters compose with logical AND (same call) or OR (multiple = of the same field).

Hands-on Example

Basic queries you will run constantly:

# Logs for a specific service since boot
journalctl -u nginx.service -b

# Tail in real time
journalctl -u nginx.service -f

# Last 15 minutes, only warnings and worse
journalctl -u nginx.service --since "-15min" -p warning

# All logs at priority error or above, across the system
journalctl -p err -b

# JSON output for machine parsing
journalctl -u nginx.service --since today -o json | jq '.MESSAGE'

# Disk usage of the journal
journalctl --disk-usage

Find the previous boot’s kernel panics:

journalctl -k -b -1 -p crit

-k is kernel only, -b -1 is the previous boot.

Enable persistent journals so logs survive reboots:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Tune retention in /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=2week
ForwardToSyslog=no

Apply with sudo systemctl restart systemd-journald.

Forward to a remote aggregator. With systemd-journal-upload:

# /etc/systemd/journal-upload.conf
[Upload]
URL=https://logs.example.com:19532
TrustedCertificateFile=/etc/ssl/certs/ca.pem

Or pipe to Vector, Fluent Bit, or Loki’s Promtail, which all read from the journal natively.

Common Pitfalls

• No persistence. Default many distros keep journals in tmpfs, so reboots wipe them. Always create /var/log/journal.
• Disk runaway. Without SystemMaxUse=, the journal can fill a disk. Set a cap.
• Mismatched timezones. Journal stores UTC and displays local. journalctl --utc for unambiguous timestamps when correlating across regions.
• PID-only filters. _PID=1234 is wrong after the process restarts. Filter by _SYSTEMD_UNIT for stable identity.
• Forgotten -b. Without -b, you query all boots, which can return surprising old entries.

Production Tips

• Pair journald with rsyslog if you must support legacy collectors. ForwardToSyslog=yes plus rsyslog rules forward to remote receivers.
• For containerized hosts, ship via Fluent Bit or Vector reading from the journal cursor. They handle backpressure and store offsets.
• Send important security events (_TRANSPORT=audit) to a separate sink with stricter retention.
• Use the PRIORITY field consistently in your apps. Logging in plain text loses severity; emit to journal via sd-journal or structured stdout that systemd can categorize.
• Add journalctl --verify to a monthly health check. Journal files can corrupt; better to know early.

Wrap-up

journalctl turns log triage into structured queries: by unit, by boot, by priority, by time. Make journals persistent, cap their disk usage, forward to a central system, and learn the four or five filter flags you reach for every day. Once it clicks, you will reach for grep on raw files only when no other option exists - and your future on-call self will thank you.