Security & troubleshooting · Prasanta Kumar Behera

Run containers with the least privilege they need, scan images before they ship, and work through the errors you'll actually meet: exit codes, OOM kills, port clashes, full disks, DNS and pull failures.

Least privilege, in five flags

By default a container runs as root (UID 0) inside its namespaces with a reduced but still generous set of Linux capabilities. If an attacker breaks the app, that's their starting point. Take away what the app doesn't need:

$ docker run -d --name api \
    --user 10001:10001 \
    --cap-drop ALL \
    --read-only --tmpfs /tmp \
    --security-opt no-new-privileges \
    --memory 512m --cpus 1 --pids-limit 200 \
    shop/api:1.4

Flag	Stops
`--user 10001` (or `USER` in the Dockerfile)	Root inside the container, which is one kernel bug away from root on the host
`--cap-drop ALL`	Raw sockets, changing file ownership, binding low ports, and so on. Add back only what's needed, e.g. `--cap-add NET_BIND_SERVICE`
`--read-only`	Attackers dropping tools or modifying the app. Mount `tmpfs` or volumes where writing is legitimate
`no-new-privileges`	`setuid` binaries raising privileges
`--memory`, `--cpus`, `--pids-limit`	One container starving the host, fork bombs

The same settings in Compose are user:, cap_drop:, read_only:, security_opt: and deploy.resources.limits.

Things to avoid

--privileged: all capabilities and all host devices. Almost never needed; grant the specific device or capability.
-v /var/run/docker.sock:/var/run/docker.sock: full control of Docker, which means root on the host. Tools that need it (CI runners, Portainer) should be treated as highly trusted.
--network host and --pid host without a clear reason.
Secrets in ENV, ARG or image layers (see lesson 2).

You hire a painter for one room. You give them the key to that room, a ladder and paint, not the keys to the whole house, the car and the safe. If they turn out to be a burglar, the damage stays in one room.

Scan before you ship

Images carry hundreds of OS packages and dependencies, and new CVEs are published every day. Scan in CI and fail the build on serious, fixable findings:

$ trivy image --severity HIGH,CRITICAL --ignore-unfixed shop/api:1.4
shop/api:1.4 (debian 12.x)
Total: 2 (HIGH: 2, CRITICAL: 0)
...
$ trivy image --severity CRITICAL --exit-code 1 shop/api:1.4   # non-zero exit fails the pipeline

Most findings disappear by rebuilding on a current base image and using smaller bases (distroless, slim). docker scout and registry scanners (Harbor with Trivy, ECR scanning) do the same job. Signing and SBOMs are covered in the "CI/CD & Software Supply Chain" track.

Troubleshooting: the errors you will meet

Start with the same three commands every time:

$ docker ps -a                                   # state and exit code
$ docker logs --tail 100 app                     # what the app said
$ docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' app

Symptom	Likely cause	What to do
Exited (0) straight away	The main process finished (no foreground process)	Run the server in the foreground; check `CMD`
Exited (1) / (2)	The app failed: config, missing env var, can't reach a dependency	`docker logs`; check `environment:` and start order
Exited (126) / (127)	Command not executable / not found	Wrong path, missing `chmod +x`, Windows line endings in a script, or the binary isn't in a minimal image
Exited (137), `OOMKilled: true`	Killed for exceeding the memory limit	Raise the limit or fix the leak
Exited (137), `OOMKilled: false`	Killed by `docker stop` after the timeout	App ignores SIGTERM: use the exec form of `CMD`
Exited (139)	Segmentation fault	Often a native library / architecture mismatch (arm64 vs amd64)
`port is already allocated`	Another container or process holds the host port	`docker ps --filter publish=8080`, `ss -ltnp`
`no space left on device`	Images, build cache, logs or volumes filled the disk	`docker system df -v`, prune, log rotation
`exec format error`	Image built for another CPU architecture	`docker buildx build --platform` for the right one
Can't resolve other containers	On the default bridge, or on different networks	Put them on one user-defined network
`toomanyrequests`, `denied`, `x509` on pull	Rate limit, login/permissions, certificate trust	Lessons 6, 7 and 9

Disk full

$ docker system df
TYPE            TOTAL   ACTIVE   SIZE      RECLAIMABLE
Images          48      6        21.3GB    18.9GB (88%)
Containers      9       6        1.2GB     310MB (25%)
Local Volumes   14      5        6.4GB     2.1GB (32%)
Build Cache     212     0        9.8GB     9.8GB
$ sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log' | sort -h | tail -3
$ docker system prune                   # stopped containers, unused networks, dangling images, build cache
$ docker image prune -a --filter until=168h

Huge *-json.log files mean log rotation isn't set: fix it in daemon.json (Level 4) and recreate the containers. Never delete files under /var/lib/docker by hand, and look before running docker volume prune, because volumes hold data.

Network debugging without tools in the image

Borrow a toolbox container that joins the same network namespace:

$ docker run --rm -it --network container:api nicolaka/netshoot
# curl -v http://db:5432        # can we reach it?
# dig db                        # does the name resolve?
# ss -ltnp                      # what is the app listening on?

A classic finding: the app listens on 127.0.0.1:8000 inside the container, so the published port can't reach it. Make it listen on 0.0.0.0.

Try it: harden and break

Run your API from lesson 2 with all five hardening flags. Does it still work? If not, the error tells you what it needs (a writable directory, a capability).
Scan it with Trivy, rebuild on a newer or smaller base, and compare the counts.
Run docker run --memory 32m python:3.12-slim python -c "x = ' ' * 100_000_000" and read the exit code and OOMKilled.
Start two containers with -p 8080:80 and read the error; find the owner with docker ps --filter publish=8080.

Going deeper: rootless mode and user namespaces

Rootless Docker runs the daemon itself as an ordinary user, so even a full container escape lands as that user, not root. It has limits (some network modes, ports below 1024 need extra setup, some storage drivers). The alternative on a normal daemon is userns-remap in daemon.json: container root is mapped to an unprivileged UID range on the host. Podman is rootless by default. In Kubernetes, the same ideas appear as Pod Security Standards and securityContext (runAsNonRoot, readOnlyRootFilesystem, capabilities.drop: [ALL]), covered in the "Kubernetes Security & Hardening" track.

Recap

Run as non-root, drop capabilities, read-only root filesystem, no-new-privileges, resource limits. Avoid --privileged and mounting the Docker socket.
Scan images in CI and rebuild on fresh bases.
Debug with ps -a → logs → inspect. Know the exit codes: 0 done, 1 app error, 126/127 command, 137 killed (OOM or stop timeout), 139 crash.
Disk full: system df, prune safely, and set log rotation.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.