Linux Systemd Units: How systemd Starts, Stops, and Watches Your Services

Photo by imgix on Unsplash

Published: — | Reading Time: 12 min | Category: Linux System Administration

TL;DR

• systemd is the init system and service manager for virtually all modern Linux distributions.
• A systemd unit is any file that tells systemd how to start, stop, and watch a resource — the most common being service units (.service).
• The systemctl command is your interface: systemctl start, stop, restart, enable, disable, status, daemon-reload.
• Three unit types matter for operations: services (.service), snapshots / timers (.timer), and sockets (.socket) — socket activation lets systemd launch a service on first connection.
• Restart=on-failure + WatchdogSec= turns a fragile service into a self-healing one, but only if the watchdog is wired up in the app.

Introduction: Why systemd (and why units?)

Before systemd, Linux startup was a chain of shell scripts under /etc/init.d. It worked, but it was slow and hard to compose. systemd replaced that with a dependency-aware, parallel boot and a uniform control interface.

The key realization that makes systemd powerful is this: systemd doesn’t run your application — it runs your unit. The unit is a declarative record of what to run, as what user, under what conditions, and what to do when it dies. That single idea is why systemctl restart nginx is atomic, why systemctl status can tell you a service crashed 3 times in 2 minutes, and why systemctl --failed surfaces boot problems in seconds.

Everything systemd manages is represented as a unit object in /usr/lib/systemd/system/ (distro-provided), /etc/systemd/system/ (admin overrides — preferred for your own services), or /run/systemd/system/ (runtime, ephemeral).

1. The Unit Types You’ll Actually Use

Commands you’ll use daily: systemctl list-units, systemctl list-unit-files, systemctl cat <name>, systemctl show <name>, systemctl list-dependencies <name>.

2. Anatomy of a .service Unit

Every service unit is a plain .ini-style file with [Unit], [Service], and optionally [Install] sections.

# /etc/systemd/system/myapp.service
[Unit]
Description=My Ops Platform Web Service
Documentation=man:systemd.service(5)
After=network.target mysql.service redis.socket
Wants=mysql.service
# Ensures this service is watched and auto-restarted
Restart=on-failure
RestartSec=5

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/start.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
TimeoutStopSec=30
# Heartbeat: systemd sends SIGUSR1 to app every 60s; app must update the
# "watchdog" via sd_notify(); if systemd doesn't get a reply it kills the app.
WatchdogSec=60
# Security hardening: run as least privilege
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target

What each section does:

• [Unit] — metadata and ordering. After= defers start until the listed units are active. Wants= starts them but only weakly (no hard order-failure). For hard ordering, use Requires=.
• [Service] — the runtime contract. Type= tells systemd how the process signals readiness:
• simple (default): ExecStart launches immediately; systemd considers it up. Most apps.
• forking: parent spawns child then exits; systemd tracks the child via PIDFile=. For daemons that fork.
• exec: same as simple, but ExecStartPre=/ExecStartPost= run before/after; systemd does not consider the service up.
• notify: app calls sd_notify() (systemd’s IPC) to signal “ready”. Combined with NotifyAccess=main.
• on-failure → RestartPreventExitStatus= (don’t restart a clean exit), or SuccessExitStatus= (which exit codes are “clean”).
• [Install] — how it’s wired into the boot. WantedBy=multi-user.target is the equivalent of an init.d runlevel. systemctl enable creates a symlink; disable removes it.
• WatchdogSec= (covered below) — the self-healing contract.

3. Practical Example: Create, Enable, and Test a Service

Step 1 — Write the unit file

sudo tee /etc/systemd/system/webapp.service > /dev/null <<'EOF'
[Unit]
Description=MyOps Platform Web App
After=network.target
Wants=network.target

[Service]
Type=simple
User=myops
Group=myops
WorkingDirectory=/opt/myops/webapp
ExecStart=/opt/myops/webapp/bin/run
Restart=on-failure
RestartSec=5
Environment=PORT=8080
Environment=LOG_LEVEL=info
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/myops/webapp/data

[Install]
WantedBy=multi-user.target
EOF

Step 2 — Reload systemd and start

sudo systemctl daemon-reload      # Pick up the new unit file
sudo systemctl start webapp       # Launch immediately
sudo systemctl enable webapp      # Start on every boot (creates the WantedBy symlink)
sudo systemctl status webapp

What you’ll see:

● webapp.service - MyOps Platform Web App
     Loaded: loaded (/etc/systemd/system/webapp.service; enabled)
     Active: active (running) since Tue 2026-10-27 10:00:00 IST;
               2s ago
   Main PID: 1234 (run)
      Tasks: 8 (limit: 4915)
     Memory: 14.2M
     CGroup: /system.slice/webapp.service
             ├─1234 /opt/myops/webapp/bin/run
             └─1241 /opt/myops/webapp/bin/worker

Oct 27 10:00:00 server myops[1234]: listening on 0.0.0.0:8080

enabled in the Loaded: column means it will start at boot. active (running) means it’s up now. If Restart=on-failure is set, a Failed state will automatically attempt a restart.

Step 3 — Test restart policy

# Force-crash it to verify Restart=on-failure works
sudo kill -9 1234
sleep 6
sudo systemctl status webapp --no-pager

If configured correctly, systemd will respawn the process within 5 seconds and the service returns to active (running) without you touching anything.

Step 4 — Make a change effective

After editing the unit file, you must run systemctl daemon-reload — systemd caches unit definitions and won’t pick up a new file until you do. Skipping this is the #1 reason a new service won’t start.

4. Controlling the Service Lifecycle

Stopping a stuck service:

# 1. Try a normal stop (SIGTERM)
sudo systemctl stop webapp

# 2. If it's frozen, escalate
sudo systemctl kill -s KILL webapp   #STONIT

# 3. Verify
systemctl is-active webapp

Pausing a service (systemctl freeze isn’t a real systemctl command — use systemctl mask to block it or systemctl set-property):

# Temporarily disable at boot
sudo systemctl mask webapp

# Re-enable later
sudo systemctl unmask webapp

Checking the process hierarchy

# Show all services with their PIDs and cgroup tree
systemctl --tree

# Or: list units the service depends on
systemctl list-dependencies webapp --reverse

systemd --tree at the top shows a systemd instance with child “buckets” (cgroups) — each service gets its own cgroup, so you can see webapp.service and its workers in one view.

5. Restart=, WatchdogSec=, and Building Self-Healing Services

Restart policies (a non-exhaustive list)

For production services, Restart=on-failure + RestartSec=5 is the sweet spot.

Making the service watch itself with WatchdogSec=

systemd can act as a health sentinel. When you set WatchdogSec=60, systemd starts a timer and expects the service to call sd_notify(FD_DISCONNECT) or update its “health file” every 60 seconds. If the service goes silent for two watchdog cycles, systemd considers it hung and Restart=always (or on-failure) kicks in.

Because this is a contract, you should pair it with the systemd/doxygen or sd-daemon headers in your app and test with systemd-run --property=WatchdogSec=10 --unit=test-watch /bin/sleep 300 (then watch it get killed and restarted).

Testing restart behavior

# Inject a failure to verify restart policy
sudo systemctl stop webapp
sudo systemctl start webapp
# Kill -9 once, then watch
sudo kill -9 $(systemctl show -p MainPID --value webapp)
sleep 6
systemctl is-active webapp   # should still be "active"

6. Socket Activation (No Port Clash, Faster Boot)

Instead of a service binding a port before it’s ready, systemd keeps a listening socket and starts the .service only when the first connection arrives. This:

• Eliminates port-clash race conditions between multiple services
• Makes systemctl start fast — the socket is already listening
• Lets you systemctl stop the service and still accept connections (the socket stays open)

Example: ssh.socket (socket activation for sshd)

# /lib/systemd/system/ssh.socket
[Socket]
ListenStream=22
Accept=yes

[Install]
WantedBy=sockets.target

The .service corresponding to this socket is sshd.service, and it’s configured with SocketActivation=yes (or Sockets= in [Unit]). When a client connects to port 22, systemd starts sshd.service on demand.

Link a service to a socket

# /etc/systemd/system/myapp.socket
[Unit]
Description=MyOps App Socket Activation
Requires=myapp.service

[Socket]
ListenStream=127.0.0.1:8080
Accept=yes

The .socket unit is what nginx, sshd, and other daemons use for “lazy” startup. You can test it by systemctl stop myapp.service, then curling 127.0.0.1:8080 — the service will start automatically when the connection arrives.

7. Timers: systemd Built-in Cron

systemd.timer is a clean replacement for cron for scheduled tasks, with better dependency handling and better integration with systemd’s logging (journalctl).

# /etc/systemd/system/cleanup.timer
[Unit]
Description=Daily cleanup of old temp files
Requires=cleanup.service

[Timer]
# Run at 02:30 daily
OnCalendar=*-*-* 02:30:00
# Run 5 min after the next calendar hit as a fallback
RandomizedDelaySec=300
Persistent=true   # Catch up if the machine was off at 02:30

[Install]
WantedBy=timers.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now cleanup.timer
systemctl list-timers

--no-pager givess a clean table. Disable a timer: systemctl disable --now cleanup.timer. Note that a timer requires the service it triggers (cleanup.service) to exist; otherwise systemd logs a Created async job for ... warning and the timer appears to do nothing.

8. Quick Reference: Most-Used systemctl Commands

Ways to read logs

Inspecting the service’s dependency graph

# Reverse dependency tree
systemctl list-dependencies nginx --reverse

# Full dependency graph (top-down)
systemctl list-dependencies nginx

9. Troubleshooting Checklist

Deep-dive into a failing service

# 1. What's the state?
systemctl status webapp

# 2. Where's the error?
journalctl -u webapp --no-pager -n 50

# 3. What is the unit actually loaded as?
systemctl cat webapp
systemctl show webapp -p MainPID -p ActiveState -p ExecMainStatus

# 4. Trace the dependency graph
systemctl list-dependencies webapp --reverse

# 5. If it's a service that keeps dying, watch the process
systemd-cgtop

10. Real-World Scenario: Wrap an Existing Script as a systemd Service

You have a backup script at /opt/scripts/backup.sh and want systemd to run it every hour and make sure it succeeds.

1. Create the service (runs as root, but runs in a controlled environment):

[Unit]
Description=Daily Backup Service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=root
StandardOutput=journal
StandardError=journal
SyslogIdentifier=backup
# on-failure restarts; for oneshot, exit 0 is success
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

2. Create the timer (as above) to run daily at 2:30.

3. Test:

sudo systemctl daemon-reload
sudo systemctl start backup.service
sudo systemctl list-timers --no-pager
sudo journalctl -u backup -f

Type=oneshot means the service runs once and exits; systemd doesn’t keep it running. The Restart=on-failure is useful if the script fails and you want it retried; set SuccessExitStatus= to treat certain exit codes as success.

11. Summary

systemd is the orchestrator of your Linux services, and the unit file is its contract. Three things to remember:

1. Units are declarative. You describe what and when; systemd figures out how to compose and order it.
2. The systemctl interface is your control surface. Start/stop/restart/status/cat/daemon-reload are the daily primitives.
3. Restart= + WatchdogSec= + Type= give you self-healing — but the contract must be matched: WatchdogSec= requires Type=notify or an explicit NotifyAccess=main, and the app must implement the health protocol.

Master these, and you stop fighting your services and start orchestrating them.

Frequently Asked Questions (FAQ)

How do I start a service only after another one finishes?
Use After=webapp.service mysql.service + Wants= (weak) or Requires= (strong) in [Unit]. After= orders the start; Requires= also chains failure.

Why does my service keep restarting?
Check Restart=on-failure and the exit code — a non-zero exit triggers on-failure. Look at journalctl -u <name> for the crash reason. Use RestartSec=5 to avoid a rapid restart loop.

How do I check what systemd loaded for a service?
systemctl cat <name> prints the resolved file path; systemctl show <name> -p MainPID -p ActiveState -p SubState prints the live state.

How do I prevent a service from starting at boot?
systemctl disable <name> removes the enable symlink. For temporary blocking, systemctl mask <name> creates a .wants symlink to /dev/null.

How do I make systemd wait for a network interface?
Add After=network-online.target and add Wants=network-online.target to [Unit]. Also systemctl enable systemd-networkd-wait-online.service if you want a hard wait for the network to be-online.

How do I use socket activation?
Create a .socket unit with ListenStream=PORT and link it to a .service with Sockets= in [Unit] (or SocketActivation=yes). systemd listens on the port and starts the service on first connection.

Related Articles

• Linux User Management & Permissions
• Linux Commands for Admins
• Systemd Units Explained: How systemd Starts, Stops, and Watches Your Services in 2026