systemd & journald in depth · Prasanta Kumar Behera

systemd starts and supervises everything on a modern Linux host. Read and write unit files, make services restart themselves, override vendor settings safely, and search the journal like a pro.

systemd in one picture

systemd is PID 1. It starts services in the right order at boot, restarts them when they crash, collects their logs (journald), and manages mounts, timers, sockets and more. Everything it manages is a unit:

Unit type	Example	Is…
`.service`	`nginx.service`	A process to run and supervise
`.timer`	`logrotate.timer`	A schedule (a modern cron)
`.mount`	`var-lib-data.mount`	A filesystem mount
`.target`	`multi-user.target`	A group of units, like a "run level"
`.socket`	`ssh.socket`	A socket that starts a service on demand

systemd is the school's timetable and caretaker. The timetable says who starts when and after whom (dependencies). The caretaker checks every room, and if a teacher walks out mid-lesson it sends them back in (Restart=). Everything anyone says is written in one big diary (the journal), so you can look up exactly what happened, and when.

Reading a unit file

$ systemctl cat cron
# /usr/lib/systemd/system/cron.service
[Unit]
Description=Regular background program processing daemon
Documentation=man:cron(8)
After=remote-fs.target nss-user-lookup.target

[Service]
EnvironmentFile=-/etc/default/cron
ExecStart=/usr/sbin/cron -f -P $EXTRA_OPTS
IgnoreSIGPIPE=false
KillMode=process
Restart=on-failure
SyslogFacility=cron

[Install]
WantedBy=multi-user.target

(Exact contents vary by distribution version.)

[Unit]: description, ordering (After=, Before=), dependencies (Wants=, Requires=).
[Service]: how to run it (ExecStart=, User=, Environment=, Restart=).
[Install]: what enable hooks it into (WantedBy=multi-user.target means "start at normal boot").

Write your own service

A tiny web server as a proper service, running as an unprivileged user and restarting if it crashes. Save as /etc/systemd/system/hello.service:

[Unit]
Description=Hello demo web server
After=network-online.target
Wants=network-online.target

[Service]
User=nobody
WorkingDirectory=/tmp
ExecStart=/usr/bin/python3 -m http.server 8080
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now hello
$ curl -sI http://localhost:8080 | head -1
HTTP/1.0 200 OK
$ sudo pkill -9 -f "http.server 8080"      # simulate a crash
$ sleep 4; systemctl status hello | grep Active
     Active: active (running) since … 2s ago

systemd noticed the crash and restarted it after 3 seconds, just as a Kubernetes kubelet restarts a crashed container.

Changing vendor units safely: drop-ins

Never edit files in /usr/lib/systemd/system/, because package upgrades overwrite them. Use an override:

$ sudo systemctl edit nginx

This opens an editor for /etc/systemd/system/nginx.service.d/override.conf. Add only what you want to change:

[Service]
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

Save, and systemd reloads automatically. systemctl cat nginx now shows both files, and yours wins.

The journal: every log in one place

$ journalctl -u hello --since "10 min ago"          # one unit, recent
$ journalctl -u nginx -p warning                      # priority warning and worse
$ journalctl -f                                       # everything, live
$ journalctl -b -p err                                # errors since this boot
$ journalctl -k | tail                                # kernel messages
$ journalctl _PID=1487                                # by process ID
$ journalctl -u nginx -o json-pretty -n 1             # all fields of one entry

Priorities, from most to least severe: emerg, alert, crit, err, warning, notice, info, debug.

Keep the journal across reboots

If /var/log/journal/ exists, the journal is persistent and journalctl -b -1 shows the previous boot. That's essential after a crash. Most server distributions enable this; if not, sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald. Limit its size with SystemMaxUse= in /etc/systemd/journald.conf.

Why did boot take so long?

$ systemd-analyze
Startup finished in 3.1s (kernel) + 18.4s (userspace) = 21.5s
$ systemd-analyze blame | head -3
12.012s systemd-networkd-wait-online.service
 2.101s snapd.seeded.service
 1.450s cloud-init.service

Try it: supervise, override, investigate

Create hello.service as above; enable it, crash it, and confirm it comes back.
Change Restart=on-failure to Restart=no with systemctl edit hello, crash it again, and see the difference.
Make it fail on purpose (set ExecStart=/usr/bin/python3 -m http.server 99999). Find the reason with systemctl status hello and journalctl -u hello -n 20.
Show only errors from the current boot, then the journal's disk usage.
Clean up: sudo systemctl disable --now hello && sudo rm /etc/systemd/system/hello.service && sudo systemctl daemon-reload.

Going deeper: systemd for platform engineers

Hardening in units: ProtectSystem=strict, PrivateTmp=yes, NoNewPrivileges=yes, CapabilityBoundingSet=, ReadWritePaths=. systemd-analyze security <unit> scores a service's exposure.
Resource control: MemoryMax=, CPUQuota= and TasksMax= use the same cgroups that the kubelet uses for pods (lesson 14).
Timers replace cron with logging, dependencies and Persistent=true for missed runs: systemctl list-timers.
The kubelet and containerd are systemd services on almost every Kubernetes node. journalctl -u kubelet is the first stop for NotReady nodes.

Recap

Everything is a unit; services have [Unit], [Service] and [Install] sections.
daemon-reload after edits; drop-ins (systemctl edit) instead of editing vendor files.
Restart=on-failure + RestartSec= = self-healing services.
journalctl filters by unit, time, priority, boot and PID; keep it persistent and size-limited.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.