Writing and Managing systemd Service Files

What Is systemd?

systemd is the init system and service manager used by most modern Linux distributions, including Ubuntu, Debian, Fedora, CentOS, and Arch Linux. It is responsible for bootstrapping the system, managing services, and handling dependencies between them.

At the heart of systemd are unit files — configuration files that describe how a service (or timer, mount, socket, etc.) should behave. When you write a custom service file, you are telling systemd exactly how to start, stop, and monitor your application.

Where Service Files Live

systemd looks for unit files in several directories, in order of priority:

# System packages install their units here (do not edit directly)
/usr/lib/systemd/system/

# System administrator overrides and custom units go here
/etc/systemd/system/

# Runtime units (temporary, cleared on reboot)
/run/systemd/system/

For custom services, you should always place your files in /etc/systemd/system/. This directory takes precedence over the package-provided defaults in /usr/lib/systemd/system/.

Anatomy of a Service File

A systemd service file is an INI-style configuration file divided into sections. Here is a complete example for a Node.js web application:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js Web Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
Environment=NODE_ENV=production
Environment=PORT=3000

[Install]
WantedBy=multi-user.target

Let’s break down each section.

The [Unit] Section

This section provides metadata and defines dependencies:

• Description — A human-readable name for the service.
• Documentation — A URL or man page reference for documentation.
• After — Specifies ordering. The service starts after the listed units are active.
• Wants — Declares a weak dependency. If the wanted unit fails, this service still starts.
• Requires — Declares a hard dependency. If the required unit fails, this service also fails.

[Unit]
Description=Backend API Server
After=network.target postgresql.service
Requires=postgresql.service

In this example, the API server will only start after PostgreSQL is running, and will stop if PostgreSQL goes down.

The [Service] Section

This is where the real configuration happens.

Service Types

The Type directive controls how systemd tracks your process:

# Most common -- the process started by ExecStart is the main process
Type=simple

# The process forks and the parent exits; systemd tracks the child
Type=forking
PIDFile=/var/run/myapp.pid

# Like simple, but the service signals readiness via sd_notify
Type=notify

# The service runs once and exits (e.g., a database migration)
Type=oneshot
RemainAfterExit=yes

Execution Directives

ExecStart=/usr/bin/python3 /opt/myapp/main.py
ExecStartPre=/opt/myapp/check-config.sh
ExecStartPost=/usr/bin/curl -s http://localhost:8080/health
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID

• ExecStartPre — Runs before the main process starts (e.g., config validation).
• ExecStartPost — Runs after the main process starts (e.g., health checks).
• ExecReload — Defines how systemctl reload behaves.

Restart Policies

# Restart only on non-zero exit codes or signals
Restart=on-failure
RestartSec=5

# Always restart, no matter what
Restart=always
RestartSec=10

# Limit restart attempts (5 attempts within 60 seconds)
StartLimitBurst=5
StartLimitIntervalSec=60

User and Environment

User=deploy
Group=deploy
WorkingDirectory=/opt/myapp

# Inline environment variables
Environment=APP_ENV=production
Environment=LOG_LEVEL=info

# Or load from an environment file
EnvironmentFile=/etc/myapp/env

The [Install] Section

This section determines what happens when you run systemctl enable:

[Install]
WantedBy=multi-user.target

multi-user.target is the standard target for non-graphical multi-user systems. When you enable the service, systemd creates a symlink so that it starts automatically on boot.

Creating Your First Service

Let’s create a service for a Python Flask application step by step.

First, create the service file:

sudo nano /etc/systemd/system/flask-app.service

Add the following content:

[Unit]
Description=Flask Production Server
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/flask-app
ExecStart=/opt/flask-app/venv/bin/gunicorn \
    --workers 4 \
    --bind 0.0.0.0:8000 \
    app:create_app()
Restart=always
RestartSec=3
Environment=FLASK_ENV=production

[Install]
WantedBy=multi-user.target

Now reload systemd, enable, and start the service:

# Reload systemd to pick up the new unit file
sudo systemctl daemon-reload

# Enable the service to start on boot
sudo systemctl enable flask-app.service

# Start the service immediately
sudo systemctl start flask-app.service

# Check the status
sudo systemctl status flask-app.service

Managing Services with systemctl

Here are the essential commands for day-to-day service management:

# Start, stop, restart
sudo systemctl start myapp
sudo systemctl stop myapp
sudo systemctl restart myapp

# Reload configuration without restarting (if ExecReload is defined)
sudo systemctl reload myapp

# View detailed status
systemctl status myapp

# View logs (uses journald)
journalctl -u myapp -f          # follow logs in real time
journalctl -u myapp --since today
journalctl -u myapp -n 100      # last 100 lines

# Check if a service is enabled
systemctl is-enabled myapp

# List all running services
systemctl list-units --type=service --state=running

Hardening Your Service

systemd provides powerful sandboxing options to limit what a service can do:

[Service]
# Filesystem restrictions
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/myapp/data /var/log/myapp

# Network restrictions
PrivateNetwork=no

# Prevent privilege escalation
NoNewPrivileges=yes

# Restrict system calls
SystemCallFilter=@system-service

# Private /tmp directory
PrivateTmp=yes

# Make /dev minimal
PrivateDevices=yes

You can analyze the security posture of any service with:

systemd-analyze security myapp.service

This command gives a security score and lists potential hardening improvements.

Debugging Failed Services

When a service fails to start, use these techniques:

# Check the status for error messages
systemctl status myapp.service

# Read the full journal output
journalctl -u myapp.service -e --no-pager

# Verify the unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service

# Check for dependency issues
systemctl list-dependencies myapp.service

Common pitfalls include incorrect file paths in ExecStart, missing permissions for the specified User, and forgetting to run daemon-reload after editing a unit file.

Overriding Existing Services

If you need to modify a package-provided service without editing its original file, use drop-in overrides:

# Create an override directory and file
sudo systemctl edit myapp.service

This opens an editor for /etc/systemd/system/myapp.service.d/override.conf. You can add or override specific directives:

[Service]
Environment=LOG_LEVEL=debug
RestartSec=10

To see the merged configuration:

systemctl cat myapp.service

Summary

systemd service files give you fine-grained control over how your applications run on Linux. The key concepts to remember are the three-section structure ([Unit], [Service], [Install]), choosing the right service Type, configuring restart policies for reliability, and using hardening directives for security. Always run daemon-reload after editing unit files, and use journalctl for troubleshooting.