systemd Deep Dive: Services, Timers, and Logs Like a Pro

You SSH into a server and run systemctl status nginx. It works — but do you know what's actually happening under the hood? systemd is the init system on virtually every modern Linux distribution (Ubuntu, Debian, RHEL, Fedora, Arch, openSUSE). It's the first process (PID 1) and it decides when everything else starts, stops, and restarts. Yet most DevOps engineers only ever use four commands: start, stop, restart, and enable.

This guide goes past those four commands: writing robust unit files, replacing cron with timers, querying the journal like a pro, and using systemd's built-in sandboxing to harden every service you run.

1. Anatomy of a Unit File

Every service on a systemd machine is described by a unit file. Here's a production-grade example for a Node.js API:

# /etc/systemd/system/my-api.service
[Unit]
Description=My API service
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=deploy
WorkingDirectory=/opt/my-api
ExecStart=/usr/bin/node /opt/my-api/server.js
Restart=on-failure
RestartSec=3
Environment=NODE_ENV=production
EnvironmentFile=/etc/my-api/env

[Install]
WantedBy=multi-user.target

The three sections matter:

[Unit] — metadata and ordering. After= says "start after this target/service"; Requires= says "if this fails, stop me too"; Wants= is weaker — "try to start it, but don't fail if it can't".
[Service] — how to run and supervise the process. Type=simple means the process itself is the main process. Restart=on-failure is the bare minimum for any service that should self-heal.
[Install] — how the unit gets enabled. WantedBy=multi-user.target is the standard "start at boot" hook.

After editing a unit file, always reload before restarting:

systemctl daemon-reload
systemctl restart my-api

Forgetting daemon-reload is the most common systemd mistake. systemd caches unit files — your edit silently does nothing until you reload.

2. systemctl Commands You Actually Need

Beyond the basics, these are the commands that save you during an incident:

Command	What it does
`systemctl status my-api`	State, recent log lines, and the PID — first stop when something breaks.
`systemctl list-units --failed`	Every failed unit. Run this first on any unknown server.
`systemctl list-dependencies --reverse my-api`	What depends on this unit — finds broken chains after an upgrade.
`systemctl show my-api -p Restart`	Inspect the effective (post-override) value of any property.
`systemctl mask docker.service`	Hard-disable a unit — stronger than disable; even manual starts fail.

3. Timers: The Modern cron

systemd timers are cron with three huge advantages: missed-run catching, precise scheduling, and full logging. A backup timer looks like this:

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-db.sh
# If a run takes 20 minutes, don't let the next one start on top of it
ExecStart=/usr/bin/touch /var/log/backup-ran.marker

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly at 02:30

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Then systemctl enable --now backup.timer. Key directives:

Persistent=true — if the machine was off at 02:30, the timer fires immediately on next boot. cron loses that job entirely.
RandomizedDelaySec=300 — spreads load so 50 servers don't all hammer the backup target at exactly 02:30.
OnCalendar — accepts cron-style Mon..Fri *-*-* 09:00:00 plus natural syntax like hourly, daily, weekly.

Check timers with systemctl list-timers — it shows the last run and the next fire time, so you can verify a scheduled job actually ran.

4. journalctl: Querying the Journal

Since journald captures stdout/stderr of every service, you get centralized logs without configuring anything. The queries that matter:

# Last 100 lines, follow like tail -f
journalctl -u my-api -n 100 -f

# Everything from the last hour
journalctl -u my-api --since "1 hour ago"

# Errors only, across all services
journalctl -p err -b

# A specific PID (follow a process across restarts)
journalctl _PID=12345

# Show how much disk the journal uses
journalctl --disk-usage

# Trim old journals to 200 MB
journalctl --vacuum-size=200M

To make logs survive reboots, set Storage=persistent in /etc/systemd/journald.conf (the default on most distros, but verify on minimal images). For long-term retention, ship the journal to Loki or Elasticsearch instead of growing local disk — the systemd-journal-remote service can also forward logs to a central collector over the network.

5. Sandboxing: Hardening Services for Free

This is the underrated gem. systemd can restrict what a service can do with a few lines in the [Service] section — no seccomp tooling, no container runtime required:

[Service]
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/etc /usr
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
MemoryDenyWriteExecute=true
CapabilityBoundingSet=

Directive	Protection
`NoNewPrivileges=true`	Blocks setuid binaries (sudo, su) from escalating — kills most privilege-escalation chains.
`ProtectSystem=strict`	Whole filesystem read-only except explicitly writable paths.
`PrivateTmp=true`	Isolated /tmp per service — no /tmp symlink attacks.
`MemoryDenyWriteExecute=true`	Blocks W^X memory — stops JIT-based and shellcode exploits.
`CapabilityBoundingSet=`	Drops every Linux capability the service doesn't need.

Test your sandbox with systemd-analyze security my-api. It prints a vulnerability score (0 = excellent, 10 = terrible) with a per-directive breakdown. Run it on every service you deploy — it's the cheapest security audit you'll ever do.

6. Debugging Boot and Service Failures

When a service won't start, work through these in order:

# 1. Why did it fail?
systemctl status my-api --no-pager

# 2. What did it print before dying?
journalctl -u my-api -n 50 --no-pager

# 3. Is the unit file even valid?
systemd-analyze verify /etc/systemd/system/my-api.service

# 4. Which dependencies are dragging the boot down?
systemd-analyze blame

# 5. Watch the whole boot sequence graphically
systemd-analyze plot > boot.svg

Two failures worth recognizing on sight: Unit my-api.service not found means you enabled a unit that doesn't exist (usually after daemon-reload was skipped), and Failed with result 'exit-code' means the process itself crashed — read the journal, not the status line.

Summary

systemd isn't just "the thing that starts services". It's a complete supervision, scheduling, logging, and sandboxing platform:

Unit files — declare dependencies with After=/Requires=/Wants=, and always run daemon-reload after editing.
Timers — replace cron with Persistent=true timers so missed runs don't silently vanish.
journalctl — -u, -p, and --since give you per-service log queries with zero setup.
Sandboxing — 5-10 directives harden a service against the most common exploit classes, and systemd-analyze security audits it for you.

Master these four areas and you'll stop fighting your init system — and start using it as a force multiplier for reliability and security.