Capstone: a real ops script · Prasanta Kumar Behera

Build a production-quality health-check script from everything in this course: options, strict mode, retries, structured logging, alerting, locking, tests and a systemd timer.

The brief

Write healthcheck.sh: a small, reliable tool that checks a list of endpoints and services, retries flaky ones, logs structured results, alerts on failure, and runs every 5 minutes from a systemd timer.

You're building the school nurse's daily round: visit every classroom on the list, check each one twice if the first look is worrying, write a neat note in the logbook, and phone the head teacher only if someone is really unwell.

Requirements

Checks file (checks.conf): one check per line, comments allowed (example below).
Options: -c FILE (required), -r RETRIES (default 3), -w WEBHOOK_URL (optional), -h.
Strict mode, quoted variables, trap clean-up, a lock so runs never overlap.
Retries with backoff for each failed check.
Structured logs to stderr: ts=… check=api type=http result=ok latency_ms=85.
Exit codes: 0 all ok, 1 some failed, 2 usage, 3 missing dependency.
Alert (optional): POST a short JSON message to the webhook when anything fails.
shellcheck-clean, with bats tests for the parsing and the check functions.
Runs from a systemd timer every 5 minutes.

Example checks.conf:

# type   name        target
http     api         https://api.example.com/healthz
tcp      postgres    db.internal:5432
service  nginx       nginx

Skeleton

Fill in the TODOs:

#!/usr/bin/env bash
set -euo pipefail

RETRIES=3 CONFIG="" WEBHOOK=""

log() { printf 'ts=%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
usage() { echo "Usage: $(basename "$0") -c FILE [-r RETRIES] [-w WEBHOOK_URL]" >&2; exit 2; }
need() { command -v "$1" >/dev/null || { log "result=error msg=\"missing $1\""; exit 3; }; }

check_http()    { curl -fsS -o /dev/null --max-time 5 "$1"; }
check_tcp()     { local host=${1%:*} port=${1##*:}; timeout 5 bash -c "exec 3<>/dev/tcp/$host/$port"; }
check_service() { systemctl is-active --quiet "$1"; }

with_retries() {                     # with_retries <command...>
  local n=1
  until "$@"; do
    (( n >= RETRIES )) && return 1
    sleep $(( n * 2 )); (( n += 1 ))
  done
}

alert() {                            # TODO: POST JSON to "$WEBHOOK" with curl, if set
  :
}

main() {
  while getopts ":c:r:w:h" opt; do
    case "$opt" in
      c) CONFIG=$OPTARG ;; r) RETRIES=$OPTARG ;; w) WEBHOOK=$OPTARG ;; *) usage ;;
    esac
  done
  [[ -r "$CONFIG" ]] || usage
  need curl; need timeout

  exec 9>/run/lock/healthcheck.lock   # TODO: choose a path your user can write
  flock -n 9 || { log "result=skip msg=\"previous run still active\""; exit 0; }

  local failed=0 type name target start ms
  while read -r type name target; do
    [[ -z "${type:-}" || "$type" == \#* ]] && continue
    start=$(date +%s%N)
    if with_retries "check_$type" "$target"; then result=ok; else result=fail; failed=$((failed + 1)); fi
    ms=$(( ($(date +%s%N) - start) / 1000000 ))
    log "check=$name type=$type result=$result latency_ms=$ms"
  done < "$CONFIG"

  (( failed == 0 )) || { alert "$failed check(s) failed on $(hostname)"; exit 1; }
}

[[ "${BASH_SOURCE[0]}" == "$0" ]] && main "$@"

Notes on the skeleton:

check_tcp uses Bash's built-in /dev/tcp/host/port to open a TCP connection, with no extra tools needed.
"check_$type" calls the function matching the check type. Add a guard so unknown types are reported instead of failing oddly (TODO).
date +%s%N (nanoseconds) is a GNU date feature; fine on Linux servers.

Deploy it

# /etc/systemd/system/healthcheck.service
[Unit]
Description=Endpoint and service health checks

[Service]
Type=oneshot
ExecStart=/opt/healthcheck/healthcheck.sh -c /opt/healthcheck/checks.conf

# /etc/systemd/system/healthcheck.timer
[Unit]
Description=Run health checks every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

$ sudo systemctl daemon-reload && sudo systemctl enable --now healthcheck.timer
$ journalctl -u healthcheck.service -o cat --since "15 min ago"
ts=2026-09-27T12:05:01Z check=api type=http result=ok latency_ms=85
ts=2026-09-27T12:05:01Z check=postgres type=tcp result=ok latency_ms=3
ts=2026-09-27T12:05:01Z check=nginx type=service result=ok latency_ms=9

Review checklist

Before you call it done

shellcheck -x healthcheck.sh is clean (or every exception is justified).
bats tests cover: comment and blank-line skipping, unknown check types, retry exhaustion, and exit codes 0/1/2/3.
Running it twice at once: the second copy logs result=skip and exits 0.
A failing check produces exactly one alert per run, not one per retry.
Logs contain no secrets (webhook URLs often embed tokens; don't log them).
The timer survives a reboot and shows in systemctl list-timers.

Going deeper: from script to product

When the checks file grows into dozens of types with thresholds and dependencies, this becomes a job for a real monitoring system (Prometheus blackbox exporter, synthetic monitoring), and your script becomes a prototype that proved what's needed.
Rewriting it in Python (see Python for Infrastructure) is a good exercise: compare readability, testing and error handling.

You've finished Bash Scripting

You can write scripts that are readable, safe, tested and scheduled, and you know when to reach for Python instead. Next: Python for Infrastructure Automation.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.