systemd Basics: Services, Units, and journalctl

systemd is the init system and service manager on most modern Linux distributions. It boots the machine, supervises long-running services, schedules timers, mounts filesystems, and aggregates logs. You do not need to love it, but you do need to drive it confidently. This guide gives you the working subset that covers about ninety percent of real tasks.

The unit concept

Everything systemd manages is a unit. A unit has a name, a type (suffix), and a configuration file. The types you will meet most often:

• .service — a long-running or one-shot process
• .socket — a listening socket that can activate a service on demand
• .timer — a scheduled trigger for another unit
• .target — a grouping unit, similar in spirit to a runlevel
• .mount — a filesystem mount, often generated from /etc/fstab

Unit files live in three locations, listed in increasing priority:

/usr/lib/systemd/system/   # packaged by distro
/etc/systemd/system/       # local admin overrides
/run/systemd/system/       # runtime, ephemeral

When a file with the same name exists in multiple locations, the higher-priority one wins. You almost always put your own units in /etc/systemd/system/.

Driving services with systemctl

systemctl is the main command. The pattern is systemctl <verb> <unit>.

# Status and the last log lines
systemctl status nginx

# Start now (does not survive reboot)
sudo systemctl start nginx

# Start now and on every boot
sudo systemctl enable --now nginx

# Stop now and remove from boot
sudo systemctl disable --now nginx

# Reload config without restarting (if supported)
sudo systemctl reload nginx

# Restart
sudo systemctl restart nginx

# Did it come up cleanly?
systemctl is-active nginx
systemctl is-enabled nginx
systemctl is-failed nginx

To list units:

systemctl list-units --type=service
systemctl list-units --state=failed
systemctl list-unit-files --type=service

After editing a unit file, always run:

sudo systemctl daemon-reload

Otherwise systemd serves you the cached version.

Reading logs with journalctl

systemd aggregates stdout and stderr from services into the journal. journalctl queries it.

# Logs for a specific unit
journalctl -u nginx

# Follow live (like tail -f)
journalctl -u nginx -f

# Last 200 lines
journalctl -u nginx -n 200

# Since boot
journalctl -u nginx -b

# Time window
journalctl -u nginx --since '1 hour ago'
journalctl --since '2026-06-18 09:00' --until '2026-06-18 10:00'

# Errors and worse
journalctl -p err -u nginx

# Kernel ring buffer
journalctl -k

A subtle but important flag is -o. The default short format hides timestamps you might want; -o short-iso gives ISO 8601 times, and -o json-pretty is useful when piping into jq.

Journals can be volatile (memory only) or persistent. To make them persistent across reboots:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Writing a simple service

Suppose you have a Python program at /opt/codeloom/server.py that should run forever and restart on failure. Create /etc/systemd/system/codeloom.service:

[Unit]
Description=Codeloom API server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=codeloom
Group=codeloom
WorkingDirectory=/opt/codeloom
ExecStart=/opt/codeloom/.venv/bin/python server.py
Restart=on-failure
RestartSec=3
Environment=PYTHONUNBUFFERED=1
EnvironmentFile=-/etc/codeloom/env

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/codeloom

[Install]
WantedBy=multi-user.target

Activate it:

sudo systemctl daemon-reload
sudo systemctl enable --now codeloom
journalctl -u codeloom -f

A few field notes:

• Type=simple is the default. The main process is ExecStart and systemd considers the service started as soon as it forks. Use Type=notify if your program calls sd_notify, or Type=oneshot for short scripts that just need to run to completion.
• Restart=on-failure restarts only on non-zero exits, not on a clean stop. Use always if you really mean it.
• EnvironmentFile=-/etc/codeloom/env reads KEY=value lines; the leading dash means “do not error if missing.” This is a good place for secrets you do not want hardcoded in the unit.
• The Protect* and Private* options harden the service cheaply. If your program does not need to write to /etc or /usr, do not let it.
• WantedBy=multi-user.target is what systemctl enable hooks into. Without [Install], enable has nothing to do.

Restart policies and rate limits

If a service crashes in a tight loop, you do not want systemd to spin forever. The defaults already protect you, but you can tune them:

[Service]
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

After 5 starts in 60 seconds, systemd will give up and mark the unit failed. Inspect with systemctl status and journalctl -u; reset with systemctl reset-failed <unit>.

Overrides without editing the original

If a unit ships with your distro and you want to tweak one field, do not edit the original. Use:

sudo systemctl edit nginx

This creates /etc/systemd/system/nginx.service.d/override.conf. Put only the sections you are changing:

[Service]
LimitNOFILE=65535

Overrides survive package upgrades; edits to the packaged file do not.

Dependencies and ordering

Two distinctions trip everyone up. Wants= and Requires= express that another unit should also start, but they say nothing about order. Before= and After= express order, but say nothing about whether the other unit starts. You usually want a pair, like:

After=postgresql.service
Requires=postgresql.service

network-online.target is the right choice when you truly need the network up before your service starts; plain network.target only means “the network stack is configured,” which is often not enough.

Pairing with shell scripts

systemd services often wrap small shell scripts. The discipline from shell scripting basics applies double here: set -euo pipefail, explicit exit codes, and writing to stdout (which the journal captures). For inspecting what the service is actually doing at runtime, mix systemctl status with the tools from process management like ps -ef --forest and pstree.

Wrap up

Most day-to-day systemd work is the same five verbs (status, start, stop, enable, restart) plus journalctl -u -f. The leap from user to author is writing a clean unit file: pick the right Type, set a sane Restart, isolate the process with Protect* options, and use EnvironmentFile for secrets. Once you have that pattern down, deploying your own services becomes a fifteen-line file and a daemon-reload away.