Introduction
If you’ve ever opened a Linux distribution that ships with systemd, you’ve probably run into the command systemctl and seen a handful of unit files scattered across /etc/systemd/system/ and /usr/lib/systemd/system/. Yet, behind those simple lines of text lies a sophisticated orchestration engine that decides when, how, and in what order services are brought up or taken down.
This guide is for system administrators, developers, and curious users who want to understand the mechanics of systemd units beyond the surface commands. We’ll cover:
- The different unit types and where they live.
- How systemd interprets dependencies to build a startup graph.
- The lifecycle of a service: start → run → stop.
- Watching services with watchdogs, timers, and logs.
- Practical examples for creating, enabling, and troubleshooting units.
By the end you’ll be able to write clean unit files, debug startup failures, and use systemd’s powerful features to keep your applications healthy.
1. Unit Types: The Building Blocks of systemd
| Unit type | Purpose | Typical file location |
|---|---|---|
service |
Runs a long‑running program (daemon) | /etc/systemd/system/*.service, /usr/lib/systemd/system/*.service |
socket |
Activates services on demand via sockets | /etc/systemd/system/*.socket, /usr/lib/systemd/system/*.socket |
target |
Groups units for ordering and boot stages | /etc/systemd/system/*.target, /usr/lib/systemd/system/*.target |
path |
Triggers actions when files or directories change | /etc/systemd/system/*.path, /usr/lib/systemd/system/*.path |
timer |
Schedules time‑based activation of units | /etc/systemd/system/*.timer, /usr/lib/systemd/system/*.timer |
slice / scope |
Resource control groups (cgroups) for resource limits | /etc/systemd/system/*.slice, /usr/lib/systemd/system/*.slice |
device |
Activates units when devices appear | /etc/systemd/system/*.device, /usr/lib/systemd/system/*.device |
Tip: Systemd automatically loads unit files from the two directories above. Files in
/etc/...override those in/usr/lib/....
1.1 Why “Units” Matter
A unit is any named component that systemd can manage. The name ends with a suffix indicating its type (*.service, *.socket, etc.). Systemd treats every unit as a node in a directed graph, where edges represent dependencies such as requires, after, or binds-to. Understanding this graph is key to predicting startup order and diagnosing failures.
2. The Startup Graph: How systemd Decides What Runs First
2.1 Dependency Directives
| Directive | Meaning |
|---|---|
Requires= |
If the listed unit fails, this unit is stopped. |
After= |
This unit starts after the listed units have started (does not imply a dependency). |
Before= |
Opposite of After=; this unit starts before the listed units. |
Wants= |
Soft dependency: if the listed unit is present, start it, but don’t fail if it’s missing. |
Conflicts= |
If both units are active, the one with the conflict is stopped. |
PartOf= |
Stopping or reloading this unit stops all units that list it as part of. |
These directives let you fine‑tune order without hardcoding start times.
2.2 Boot Targets
Systemd defines a series of targets representing boot stages:
basic.target– minimal system state (e.g., initramfs).multi-user.target– normal non‑graphical multi‑user mode.graphical.target– adds a display manager on top ofmulti-user.rescue.target,emergency.target– single‑user or emergency shells.
Your units typically specify WantedBy= or RequiredBy= in the [Install] section to attach them to one of these targets. For example:
[Unit]
Description=My Web Server
[Service]
ExecStart=/usr/bin/myweb --config /etc/myweb.conf
[Install]
WantedBy=multi-user.target
When you run systemctl enable myweb.service, systemd creates a symlink from the unit file to /etc/systemd/system/multi-user.target.wants/.
2.3 The Graph in Action
Consider three units:
database.service(requires a database daemon).backend.service(needs the database and an HTTP socket).frontend.socket(provides the HTTP endpoint).
Their unit files might contain:
# frontend.socket
[Socket]
ListenStream=8080
[Install]
WantedBy=sockets.target
# backend.service
[Unit]
Requires=database.service frontend.socket
After=database.service frontend.socket
[Service]
ExecStart=/usr/bin/backend
When the system boots:
sockets.targetpulls infrontend.socket.- The socket activation creates a listening socket; it does not start the backend yet.
backend.service’sAfter=ensures it starts after both the database and the socket are ready.- If either dependency fails,
backend.servicewill be stopped because ofRequires=.
3. The Service Lifecycle: Start → Run → Stop
3.1 Starting a Service
When you run:
systemctl start myapp.service
systemd performs the following steps:
- Resolve Dependencies – Load all units referenced by
Requires=,After=, etc. - Create Cgroup – Allocate a cgroup for the service (unless it’s a “scope” unit).
- Execute Process – Run the command specified in
ExecStart=. If multiple commands are listed, they run sequentially; if any fails, the unit enters the failed state. - Monitor Status – systemd tracks whether the process exits normally or is killed.
3.2 Running State
A service can be in several states:
| State | Description |
|---|---|
active (running) |
Process is alive and has not exited. |
activating |
Still starting; systemd hasn’t yet confirmed it’s up. |
failed |
Execution failed or the process crashed. |
inactive (dead) |
Service was stopped, no active process. |
You can query this with:
systemctl status myapp.service
3.3 Stopping a Service
Stopping is symmetrical to starting:
- systemd sends the configured stop signal (
SIGTERMby default) to the main PID. - If the service does not exit within
TimeoutStopSec=(default 90s), it receivesSIGKILL. - The cgroup is cleaned up.
If a unit has ExecStop= defined, that command runs after sending the stop signal but before killing the process outright.
4. Watching Services: Keep an Eye on Health
4.1 Watchdog Timers
A watchdog monitors whether a service periodically reports “I’m alive.” The unit file must include:
[Service]
WatchdogSec=30s
ExecStart=/usr/bin/myapp --watchdog
The program must call sd_notify("WATCHDOG=1") at least every 30 seconds. If the watchdog message is missed, systemd will consider the service failed and restart it (if Restart= is set).
4.2 Timers
Timers schedule unit activation based on time or calendar events:
# backup.timer
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
This timer starts backup.service every day at 2 AM, even if the system was down during that time (because of Persistent=true).
4.3 Logging and Journal
All unit output goes to the journal by default (stdout and stderr). You can tail it with:
journalctl -u myapp.service -f
If you need structured logs, have your application emit JSON or use sd_notify() for status messages.
5. Writing a Clean Unit File: A Step‑by‑Step Example
Let’s create a unit that runs a simple HTTP server written in Python. The service should:
- Start after the network is ready.
- Restart on crash.
- Use a watchdog to ensure it stays alive.
- Log output to the journal.
# /etc/systemd/system/myhttp.service
[Unit]
Description=My Simple HTTP Server
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/python3 -m http.server 8000
Restart=on-failure
RestartSec=5s
WatchdogSec=30s
Environment=MYAPP_ENV=production
# If the process exits cleanly, consider it a success.
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
Key points
Type=simpletells systemd that the main PID is the process started byExecStart=. For forking daemons useType=forking.Restart=on-failurerestarts only if the exit code indicates failure.SuccessExitStatus=143(signal 15 + 128) lets us stop the service withsystemctl stop myhttp.servicewithout triggering a restart.
6. Common Pitfalls and How to Fix Them
| Symptom | Likely Cause | Fix |
|---|---|---|
| Service remains in activating state forever | ExecStart= command never exits or blocks |
Verify the command runs successfully outside systemd; check for missing files or permissions. |
| Unit fails with “Unit file … has no matching installed unit” | Unit not in /etc/systemd/system/ or /usr/lib/systemd/system/ |
Place the file correctly and run systemctl daemon-reload. |
| Service restarts infinitely | Exit code is considered failure; no Restart= guard |
Add SuccessExitStatus= or adjust Restart=. |
| Watchdog timeout occurs immediately | Application never sends watchdog ping | Ensure the app calls sd_notify("WATCHDOG=1") in a loop. |
| Socket activation starts service but it never listens | Service does not bind to the socket file; missing BindTo= |
Add BindTo=frontend.socket or use ListenStream= correctly. |
6.1 Debugging with systemctl
# Show detailed state and dependencies
systemctl show myapp.service
# View recent logs (last 50 lines)
journalctl -u myapp.service -n 50
# Check why a unit failed to start
systemctl status myapp.service
7. Advanced Topics for Power Users
7.1 Resource Limits with Slices and Cgroups
You can limit CPU, memory, or I/O for a group of services:
# /etc/systemd/system/mygroup.slice
[Slice]
CPUQuota=50%
MemoryMax=200M
Then in each service:
[Service]
Slice=mygroup.slice
7.2 Runtime Configuration with RuntimeDirectory= and StateDirectory=
These directives create temporary directories under /run/systemd/ or /var/lib/systemd/, ensuring isolation and automatic cleanup.
[Service]
RuntimeDirectory=myapp
ExecStart=/usr/bin/myapp --runtime-dir /run/systemd/myapp
7.3 Using systemd-notify for Status Updates
Applications can inform systemd about their health, progress, or readiness:
sd_notify(0, "READY=1\nSTATUS=Processing request");
This is useful for services that perform long‑running initialization before becoming fully ready.
Conclusion
Systemd’s unit files are more than just service wrappers; they’re declarative specifications of how your software fits into the system’s lifecycle. By mastering unit types, dependency directives, and lifecycle hooks, you can:
- Build reliable boot sequences.
- Automate restarts and watchdogs for high‑availability services.
- Isolate resources with slices and cgroups.
- Simplify troubleshooting through precise logs and status messages.
With these concepts in hand, you’re ready to write clean, maintainable units that let systemd do what it does best: orchestrate your Linux environment efficiently and predictably. Happy unit‑file crafting!
Frequently Asked Questions (FAQ)
What is a systemd unit and how does it differ from a traditional init script?
A systemd unit is a declarative configuration file that describes a service, socket, target, or other resource. Unlike imperative init scripts, units use dependency directives (Requires=, After=) to build a graph that determines start order automatically.
Which directories should I modify when creating custom units and why?
Place custom units in /etc/systemd/system/. Files here override those shipped with the distribution under /usr/lib/systemd/system/. Keeping them separate preserves package upgrades while allowing local overrides.
How do Requires= and After= interact, and what happens if a required unit fails to start?
Requires= establishes a hard dependency: if the listed unit fails or stops, this unit is stopped as well. After= only orders startup; it does not create a failure link. If a required unit fails, systemd aborts the dependent unit’s start and cleans up.
What is the role of timers in relation to services, and how do I schedule one?
A timer unit (*.timer) triggers activation of another unit (usually a service) based on time rules (OnCalendar=, OnBootSec=). To schedule, create both a .service and a corresponding .timer, enable the timer with systemctl enable --now <unit>.timer, and systemd will fire it according to the defined schedule.


