Safe scripts · Prasanta Kumar Behera

Make scripts safe to run twice, at 2 a.m., by someone else: strict mode, clean-up with trap, safe temporary files, idempotent steps, locking against overlapping runs, retries, and dry runs.

The 2 a.m. test

A safe script is one you'd let someone else run at 2 a.m., twice by accident, on the wrong server, and it would either do the right thing or stop clearly without damage.

A good recipe says: "if the oven isn't hot yet, wait", "if you already added sugar, don't add it again", and "if you burn it, turn off the oven before leaving the kitchen". Safe scripts are recipes like that: they check, they don't repeat steps twice, and they always clean up.

Strict mode

Start scripts with:

#!/usr/bin/env bash
set -euo pipefail

Option	Effect
`-e`	Exit when a command fails (with sensible exceptions, e.g. inside `if` conditions)
`-u`	Treat unset variables as errors, instead of silently empty strings
`-o pipefail`	A pipeline fails if any stage fails

set -euo pipefail
curl -fsSL https://example.com/release.tgz | tar -xz -C /opt/app   # a failed download now fails the script
echo "$UNDEFINED_VAR"                                             # -u: stops here with a clear error

Know set -e's surprises

-e doesn't trigger for commands in if/while conditions or before &&/||, and (( count++ )) when count is 0 returns status 1, which does stop the script. Strict mode is a safety net, not a substitute for checking important commands explicitly.

trap: always clean up

#!/usr/bin/env bash
set -euo pipefail
tmp=$(mktemp -d)                         # unique, private temp directory
cleanup() { rm -rf -- "$tmp"; }
trap cleanup EXIT                        # runs on success, failure or Ctrl+C
trap 'echo "error on line $LINENO" >&2' ERR

curl -fsSL -o "$tmp/pkg.tgz" https://example.com/pkg.tgz
tar -xzf "$tmp/pkg.tgz" -C "$tmp"

mktemp avoids predictable names like /tmp/myscript.tmp (which other users could pre-create).

Guard dangerous variables

: "${BACKUP_DIR:?BACKUP_DIR must be set}"     # abort with a message if empty or unset
rm -rf -- "${BACKUP_DIR:?}"/old/               # never becomes rm -rf /old/ or rm -rf /

Idempotency: safe to run twice

Instead of	Use	Why
`mkdir /srv/app`	`mkdir -p /srv/app`	No error if it exists
`useradd app`	`id app &>/dev/null \\|\\| useradd app`	Check first
`echo "line" >> file`	`grep -qxF "line" file \\|\\| echo "line" >> file`	Don't append duplicates
`ln -s new current`	`ln -sfn new current`	Replace the link atomically
`systemctl start x`	`systemctl enable --now x`	Already running is fine

When a script fails halfway, you fix the cause and run it again. Idempotent steps make that safe.

Only one at a time: flock

exec 9>/var/lock/nightly-backup.lock
if ! flock -n 9; then
  echo "another run is in progress; exiting" >&2
  exit 0
fi
# ... the job ...

The lock is released automatically when the script exits, even if it crashes.

Retries with backoff

retry() {                     # retry <attempts> <command...>
  local n=1 max=$1; shift
  until "$@"; do
    (( n >= max )) && { echo "failed after $n attempts: $*" >&2; return 1; }
    sleep $(( 2 ** n )); (( n += 1 ))
  done
}
retry 5 curl -fsS https://api.internal/health

Dry runs and confirmation

DRY_RUN=${DRY_RUN:-true}          # safe by default
run() { if [[ "$DRY_RUN" == true ]]; then echo "DRY-RUN: $*"; else "$@"; fi; }
run rm -f -- /var/cache/app/*.tmp

Run it once as-is to see what would happen, then with DRY_RUN=false.

Here's a fragile script for the lab below:

#!/bin/bash
cd $1
rm -rf $TARGET/*
curl https://example.com/data.tgz | tar xz
echo "done" >> /var/log/sync.log

Try it: harden the fragile script above

Add strict mode, and quote everything.
Make cd failure stop the script (strict mode does; also check that $1 was given).
Guard $TARGET with ${TARGET:?}, and make deletion dry-run by default.
Download to a mktemp directory with curl -fsSL, and clean up with trap.
Add a flock so two copies can't run together.
Run it twice in a row. Is the result the same?

Going deeper: operational hygiene

Log with timestamps to stderr, and let systemd or cron capture it into the journal. Don't invent your own log files unless you also rotate them.
Make scripts declare their dependencies at the top (command -v jq >/dev/null || { echo "jq required" >&2; exit 3; }).
For destructive fleet operations, add a canary step: run on one host, verify, then continue, the same idea as upgrade waves.
If you find yourself writing error-handling frameworks in Bash, it's time for Python or a configuration-management tool (Ansible).

Recap

Start with set -euo pipefail, and know its exceptions.
trap cleanup EXIT + mktemp for clean temporary files.
Guard with ${VAR:?}; use -- before file arguments.
Idempotent steps, flock against overlap, retries with backoff, dry run by default.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.