systemd Services skill for Claude Code

A unit is done when it starts from a clean boot in the right order, restarts on failure with backoff instead of hammering, logs to the journal, runs as an unprivileged user with a sandbox score from systemd-analyze security in the “OK” range or better, and every change goes in a drop-in or a versioned unit file, not an ad-hoc edit.

Ground rules

• Put custom units in /etc/systemd/system/. Never edit files under /usr/lib/systemd/system/ or /lib/systemd/system/; use systemctl edit <unit> (creates a drop-in under /etc/systemd/system/<unit>.d/).
• Run systemctl daemon-reload after every unit file change.
• No sudo, su, nohup, &, or bash -c "cd x && ..." in ExecStart. Use User=, WorkingDirectory=, Environment=; the process must stay in the foreground.
• Restarting or stopping a production service drops connections; confirm with the user before systemctl restart on a live host, and prefer reload when supported.
• Secrets never go in Environment= lines of world-readable unit files; use LoadCredential=/SetCredentialEncrypted= or an EnvironmentFile= with mode 0600.

Workflow

- [ ] 1. Inspect what exists: unit, status, logs, dependencies
- [ ] 2. Choose Type= and the restart policy
- [ ] 3. Write the unit (or drop-in) with user, env, logging
- [ ] 4. Add hardening; check the exposure score
- [ ] 5. Load, enable, start; read logs
- [ ] 6. Prove restart, reboot, and failure behavior

1. Inspect

systemctl status myapp.service               # state, main PID, last log lines
systemctl cat myapp.service                  # effective unit including drop-ins
systemctl show myapp.service -p Restart,NRestarts,ExecMainStatus,Result,User
journalctl -u myapp.service -b --no-pager -n 200
journalctl -u myapp.service -p err --since "1 hour ago"
systemctl list-dependencies myapp.service
systemd-analyze verify /etc/systemd/system/myapp.service
systemctl --failed

2. Type and restart decisions

Process behavior	Type=
Runs in foreground, ready immediately	exec (reports exec failures correctly; prefer over simple)
Supports sd_notify (READY=1)	notify (accurate readiness for dependents)
Runs to completion (migrations, backups, timer jobs)	oneshot (+ RemainAfterExit=yes if it sets up state)
Classic daemon that forks	forking with PIDFile=; better: run it in foreground mode

Situation	Restart policy
Long-running server	Restart=on-failure, RestartSec=2s
Must always run, even after clean exit	Restart=always
Oneshot job	No restart; let the timer retry, alert on failure
Crash loops must not hammer dependencies	RestartSteps= + RestartMaxDelaySec= (systemd 254+) for exponential backoff

Start limits live in [Unit]: StartLimitIntervalSec=300 and StartLimitBurst=5. After that the unit enters failed state; systemctl reset-failed myapp clears it.

3. Service template

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp HTTP API
Documentation=https://git.acme.dev/myapp
Wants=network-online.target
After=network-online.target postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=exec
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
# env file: mode 0640 root:myapp
EnvironmentFile=/etc/myapp/env
Environment=PORT=8080 NODE_ENV=production
ExecStartPre=/opt/myapp/bin/check-config
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2s
RestartSteps=5
RestartMaxDelaySec=60s
TimeoutStartSec=30s
TimeoutStopSec=30s
KillSignal=SIGTERM
LimitNOFILE=65536
# creates /var/lib/myapp owned by User=
StateDirectory=myapp
LogsDirectory=myapp
RuntimeDirectory=myapp

# hardening baseline (see step 4)
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
LockPersonality=yes
# MemoryDenyWriteExecute=yes breaks JITs (Node, Java, .NET); enable for non-JIT binaries
SystemCallFilter=@system-service
SystemCallArchitectures=native
CapabilityBoundingSet=

[Install]
WantedBy=multi-user.target

Need port 80/443 without root: AmbientCapabilities=CAP_NET_BIND_SERVICE and CapabilityBoundingSet=CAP_NET_BIND_SERVICE, or bind high and proxy.

Create the user: useradd --system --no-create-home --shell /usr/sbin/nologin myapp, or use DynamicUser=yes for stateless services (combine with StateDirectory= for persistent data).

Credentials (systemd 247+): LoadCredential=dbpass:/etc/myapp/secrets/dbpass exposes the file at $CREDENTIALS_DIRECTORY/dbpass, readable only by the service. Encrypted: systemd-creds encrypt --name=dbpass plain.txt /etc/credstore.encrypted/dbpass then LoadCredentialEncrypted=dbpass.

4. Hardening

systemd-analyze security myapp.service       # exposure score 0-10 and per-directive findings

Target: score below ~4.0 (“OK”) for network services. Add directives one at a time, restart, and exercise the app; read references/hardening.md for what each directive blocks and which ones break common runtimes. If the app writes outside its State/Logs/Runtime directories under ProtectSystem=strict, add ReadWritePaths=/specific/path, not ProtectSystem=full.

5. Load and start

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
systemctl is-active myapp && systemctl is-enabled myapp
journalctl -u myapp -f

Drop-in override (changes survive package upgrades):

sudo systemctl edit myapp.service
# [Service]
# Environment=LOG_LEVEL=debug
# ExecStart=                      <- empty line clears the list before redefining
# ExecStart=/usr/bin/node /opt/myapp/server.js --inspect=127.0.0.1:9229

Timers (replacing cron)

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
[Service]
Type=oneshot
User=backup
ExecStart=/opt/backup/run.sh
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup timer
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
# run at boot if the last scheduled run was missed
Persistent=true
AccuracySec=1m
[Install]
WantedBy=timers.target

systemd-analyze calendar "*-*-* 02:30:00"     # validate expression, show next run
sudo systemctl enable --now backup.timer
systemctl list-timers --all
sudo systemctl start backup.service           # run once now to test

Alert on failures with OnFailure=notify-failure@%n.service in the service [Unit]. Timers do not overlap: a oneshot still running blocks the next activation. For cross-platform scheduling concerns (locking, missed-run monitoring) see scheduled-jobs.

journald

• Logs: write to stdout/stderr; journald captures them with metadata. Do not log to files and also stdout.
• Retention: /etc/systemd/journald.conf -> SystemMaxUse=2G, MaxRetentionSec=30day; persistent storage requires /var/log/journal (or Storage=persistent).
• Rate limiting can drop bursts: LogRateLimitIntervalSec=/LogRateLimitBurst= per unit.
• Useful: journalctl -u myapp -o json-pretty, journalctl --disk-usage, journalctl -u myapp --since today --grep "timeout".

User services

~/.config/systemd/user/x.service, managed with systemctl --user. They stop at logout unless loginctl enable-linger <user>. Use WantedBy=default.target.

Debugging

Symptom	Check
status=203/EXEC	Binary path wrong, not executable, missing shebang, or noexec mount
status=217/USER	User= does not exist
status=200/CHDIR	WorkingDirectory missing or not accessible to the user
status=226/NAMESPACE	Sandboxing path missing (ReadWritePaths to nonexistent dir)
Works by hand, fails as unit	Different env/PATH/HOME; run systemd-run --pty --uid=myapp --property=... cmd to reproduce
“start request repeated too quickly”	Start limit hit: fix cause, then systemctl reset-failed
Dependents start before app is ready	Use Type=notify or a readiness ExecStartPost= check
Stop takes 90 s	App ignores SIGTERM; fix signal handling or set KillSignal=/TimeoutStopSec=
Starts before network is usable	After= and Wants=network-online.target (both)

Gotchas

• After= orders but does not pull in; Wants=/Requires= pull in but do not order. You usually need both.
• Environment= values with spaces need quotes per assignment: Environment="JAVA_OPTS=-Xms512m -Xmx512m".
• systemd does not support trailing comments: Key=value # note makes # note part of the value. Put comments on their own line.
• ExecStart does not use a shell: no pipes, globs, ~, or $(...). Variables expand only as ${VAR}/$VAR from the unit environment.
• Restart=always with a config error loops until the start limit; keep ExecStartPre validation so failures are fast and visible.
• PrivateTmp=yes means /tmp files are not shared with other processes; sockets in /tmp move to RuntimeDirectory=.
• Persistent=true on a timer with a long downtime fires immediately on boot; make jobs idempotent.

Verification

systemd-analyze verify /etc/systemd/system/myapp.service
systemd-analyze security myapp.service | tail -1
sudo systemctl kill -s SIGKILL myapp && sleep 5 && systemctl is-active myapp   # restarts? (non-prod or confirmed)
systemctl show myapp -p NRestarts

Then reboot a staging host (or confirm with the user for production) and check systemctl is-active myapp and journalctl -u myapp -b.