systemd Service Units: A Practical Tutorial

What and Why

systemd is the init system on most modern Linux distributions. It boots the machine, starts your services, restarts them when they crash, captures their logs, and gives you a unified interface to all of it. If you have ever written a flaky nohup ... & line in a Bash script and prayed it would survive a reboot, systemd is the cure.

A service unit is a small INI-style file that tells systemd how to run a program: which command, which user, when to start, when to restart, and what to depend on.

Mental Model

systemd organizes the world into units. Service units run programs. Target units group other units (similar to runlevels). Socket and timer units replace inetd and cron. Each unit has a name like nginx.service or multi-user.target.

inactive --> activating --> active (running)
                            |
                            v
                        deactivating --> inactive
                            ^
                            |
                        (crash) --> auto-restart per policy

Units live in three directories, listed in priority order: /etc/systemd/system (your overrides, highest), /run/systemd/system (runtime), and /usr/lib/systemd/system (packages, lowest). To install your own service, drop a file in /etc/systemd/system.

Hands-on Example

Suppose you have a small Node app at /opt/echo/server.js. Create the unit:

# /etc/systemd/system/echo.service
[Unit]
Description=Echo HTTP service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=echo
Group=echo
WorkingDirectory=/opt/echo
ExecStart=/usr/bin/node server.js
Restart=on-failure
RestartSec=3
Environment=NODE_ENV=production
Environment=PORT=8080
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Then activate it:

sudo useradd --system --home /opt/echo --shell /usr/sbin/nologin echo
sudo systemctl daemon-reload
sudo systemctl enable --now echo.service
sudo systemctl status echo.service

enable creates the symlink so it starts on boot; --now also starts it immediately. status shows the current state, recent log lines, and the PID.

Tail the logs:

journalctl -u echo.service -f
journalctl -u echo.service --since "10 min ago"
journalctl -u echo.service -p err

Edit safely with a drop-in instead of touching the original file:

sudo systemctl edit echo.service
# Opens /etc/systemd/system/echo.service.d/override.conf

Add overrides:

[Service]
Environment=LOG_LEVEL=debug
MemoryMax=256M

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart echo.service

Need it to run on a schedule instead of always-on? Create a timer unit:

# /etc/systemd/system/echo-cleanup.timer
[Unit]
Description=Daily cleanup for echo

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Pair it with a echo-cleanup.service of Type=oneshot that runs your cleanup script.

Common Pitfalls

Forgetting daemon-reload after editing a unit. systemd caches parsed units; changes do not apply until you tell it to re-read them.

Using Type=simple for a program that forks into the background. Your unit will appear to succeed and then systemd will think it has died. Use Type=forking with a PIDFile= or, better, run your daemon in the foreground.

Restart loops. With Restart=always and a misconfigured app, systemd will happily spin up your service hundreds of times per minute. Set StartLimitBurst= and StartLimitIntervalSec= to cap the storm.

Hard-coding network.target instead of network-online.target. The first only means the stack is up, not that DNS resolves. Most apps want the second.

Writing logs to a file when you could let the journal handle it. The journal is structured, indexed, rotated, and queryable; piping into a file inside the unit reinvents all of that.

Practical Tips

Use systemd-analyze verify /etc/systemd/system/echo.service to lint your file before you reload.

systemd-analyze blame and systemd-analyze critical-chain show what is slowing your boot. Indispensable on bare metal.

Lock down services with sandboxing directives: NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=true, PrivateTmp=true, ReadWritePaths=/var/lib/echo. Each adds a thin layer of defense in depth.

Use EnvironmentFile=/etc/echo/env to load secrets without committing them to the unit file. Set the file mode to 600 and own it by the service user.

For one-off debugging, systemd-run --unit=test --service-type=exec /path/to/cmd launches an ad-hoc transient service that you can journalctl -u test -f like any other.

Wrap-up

A systemd unit is just a config file plus three commands: daemon-reload, enable --now, and status. Add journalctl for logs and you have a complete production-grade service runner. Once you stop fighting it, systemd quietly handles the parts of operations you used to lose sleep over: crashes, restarts, boot order, and log rotation.