Docker & Containers — Level by Level›09 · Registry certificates & the insecure bypass

Lesson 09 of 13 · Level 4 — Daemon & certificates

Registry certificates & the insecure bypass

Fix the x509 errors that stop docker pull and docker login: trust a registry per host with certs.d, trust a company CA system-wide, use client certificates, and understand exactly what the insecure-registries bypass does, when it's acceptable, and the equivalents for Podman and containerd.

Practitioner
Key wordsx509certificate signed by unknown authoritycerts.dca.crtclient.certclient.keymTLSupdate-ca-certificatesupdate-ca-trustinsecure-registriesskip_verifyhosts.tomltls-verifyTLS-inspecting proxy
docker pull reg.lab:5000/app:1 HTTPS to reg.lab:5000 does the cert chain to a CA we trust? /etc/docker/certs.d/reg.lab:5000/ ca.crt (+ client.cert / client.key) system trust store update-ca-certificates / update-ca-trust checks pull works verified TLS trusted insecure-registries daemon.json bypass: skip verification, allow HTTP not trusted x509 error certificate signed by unknown authority bypass = no protection from a fake registry
Docker checks certs.d, then the system trust store. Only the insecure bypass skips verification, and with it the protection.

Why TLS matters for images

When a host pulls an image, it's about to run whatever comes back as root-equivalent code. TLS verification is what proves the bytes came from your registry and not from something in between. That's why Docker refuses a certificate it can't verify, and why the fix should almost always be "teach Docker to trust the right CA", not "stop checking".

A courier arrives with a parcel and an ID badge. Checking the badge against the company's list of real badges is TLS verification. Adding the courier company to your list (trusting the CA) is the fix. "Accept parcels from anyone with any badge" is the insecure bypass: it works right up to the day someone brings you the wrong parcel.

Read the error first

Error Meaning Fix
x509: certificate signed by unknown authority The chain doesn't lead to a CA this host trusts Trust the CA: certs.d or the system store
x509: certificate is valid for registry.lab.local, not 192.168.56.10 You used a name or IP that isn't in the certificate's SANs Use the right name, or reissue with that SAN
x509: certificate relies on legacy Common Name field, use SANs instead The cert has only a CN Reissue with subjectAltName
x509: certificate has expired or is not yet valid Expired, or the host's clock is wrong Renew, or fix NTP (timedatectl)
http: server gave HTTP response to HTTPS client The registry serves plain HTTP Give it TLS (or, lab only, the bypass)
remote error: tls: certificate required / bad certificate The registry wants a client certificate Add client.cert / client.key

Then look at what the registry actually presents:

$ openssl s_client -connect registry.lab.local:5000 -servername registry.lab.local -showcerts </dev/null \
    | openssl x509 -noout -subject -issuer -enddate -ext subjectAltName
subject=CN = registry.lab.local
issuer=CN = Lab Internal CA
notAfter=Sep 28 10:00:00 2027 GMT
X509v3 Subject Alternative Name:
    DNS:registry.lab.local, IP Address:192.168.56.10

If the server sends only its own certificate but it was signed by an intermediate CA, clients fail too: configure the registry with the full chain (server certificate followed by the intermediate) in its certificate file.

Docker looks for per-registry trust in /etc/docker/certs.d/<host:port>/:

/etc/docker/certs.d/
└── registry.lab.local:5000/
    ├── ca.crt          # CA that signed the registry certificate (or the self-signed cert itself)
    ├── client.cert     # optional: client certificate for mutual TLS
    └── client.key      # optional: its private key
$ sudo mkdir -p /etc/docker/certs.d/registry.lab.local:5000
$ sudo cp lab-ca.crt /etc/docker/certs.d/registry.lab.local:5000/ca.crt
$ docker pull registry.lab.local:5000/tools/alpine:3.20      # no restart needed

The rules that catch people out:

  • The folder name must match the image reference exactly. registry.lab.local:5000 and registry.lab.local (port 443) are different folders, and so are a name and its IP.
  • Extensions mean something: .crt = CA to trust, .cert = client certificate, .key = client key. A client certificate saved as .crt is treated as a CA and mTLS fails.
  • It's read on each connection, so no daemon restart. It applies only to the Docker daemon on that host. curl, Podman and Kubernetes nodes have their own trust.

Fix 2: trust a CA system-wide

When all internal services use one company CA, add it to the operating system's trust store. Docker (and curl, git, most tools) then trusts every certificate that CA signs:

# Debian / Ubuntu
$ sudo cp corp-root-ca.crt /usr/local/share/ca-certificates/corp-root-ca.crt
$ sudo update-ca-certificates

# RHEL / Rocky / Fedora
$ sudo cp corp-root-ca.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

$ sudo systemctl restart docker     # the daemon loads the system pool at start

This is also the fix for the classic corporate case: every pull, even from Docker Hub, fails with unknown authority, because a TLS-inspecting proxy re-signs all HTTPS traffic with the company's CA. Install that CA; don't mark Docker Hub insecure.

Fix 3: client certificates (mutual TLS)

Some registries (often at edge sites or in regulated environments) require the client to present a certificate as well:

$ sudo cp edge-node-01.crt /etc/docker/certs.d/registry.lab.local:5000/client.cert
$ sudo cp edge-node-01.key /etc/docker/certs.d/registry.lab.local:5000/client.key
$ sudo chmod 600 /etc/docker/certs.d/registry.lab.local:5000/client.key

The registry side sets REGISTRY_HTTP_TLS_CLIENTCAS_0=/certs/client-ca.crt so it accepts only clients signed by that CA.

The bypass: insecure-registries

When you truly can't get a trusted certificate (a throwaway lab, a registry on plain HTTP during a first bootstrap), Docker has an explicit escape hatch in daemon.json:

{
  "insecure-registries": ["registry.lab.local:5000", "10.10.0.0/24"]
}
$ sudo dockerd --validate --config-file /etc/docker/daemon.json
$ sudo systemctl reload docker
$ docker info -f '{{json .RegistryConfig.IndexConfigs}}' | python3 -m json.tool | grep -B2 -A2 '"Secure": false'

What it does for those registries (by host:port, or a whole CIDR range):

  1. Docker still tries HTTPS first, but ignores certificate errors (unknown CA, wrong name, expired).
  2. If HTTPS isn't available at all, it falls back to plain HTTP.

What you give up:

  • No proof of identity. Anyone who can redirect traffic (DNS, ARP, a compromised router) can serve you any image, and you'll run it.
  • With HTTP fallback, passwords from docker login and every layer travel in clear text.
  • It's easy to forget, and it's set per host, so it spreads quietly across a fleet.

When the bypass is acceptable

  • A local lab or CI sandbox with no real data, on an isolated network.
  • A short bootstrap window where a registry comes up before its certificate, with a ticket to remove the setting.

Not acceptable: production hosts, anything reachable from the internet, and never docker.io or another public registry. localhost / 127.0.0.0/8 is treated as insecure by Docker automatically, so a local registry on localhost:5000 works without any setting.

Remove it as soon as the real certificate is in place:

$ sudoedit /etc/docker/daemon.json          # delete the insecure-registries entry
$ sudo systemctl reload docker

The same fixes for Podman and containerd

Podman (and Buildah, Skopeo) read per-registry trust from /etc/containers/certs.d/<host:port>/ with the same file names. The bypass is per command or per registry:

$ podman pull --tls-verify=false registry.lab.local:5000/tools/alpine:3.20
# /etc/containers/registries.conf.d/lab.conf
[[registry]]
location = "registry.lab.local:5000"
insecure = true

containerd (Kubernetes nodes) uses one hosts.toml per registry, as long as the CRI registry config_path points at /etc/containerd/certs.d (the key's location in config.toml differs between containerd 1.x and 2.x, so check the docs for your version):

# /etc/containerd/certs.d/registry.lab.local:5000/hosts.toml
server = "https://registry.lab.local:5000"

[host."https://registry.lab.local:5000"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/registry.lab.local:5000/ca.crt"
  # client = [["/path/client.crt", "/path/client.key"]]   # mutual TLS
  # skip_verify = true                                     # the bypass, labs only

Because every node pulls its own images, trust has to be on every node, usually delivered by the node image or configuration management rather than by hand.

Try it: break trust, then fix it three ways

Use the registry from lesson 7 (self-signed certificate) and a second VM as the client.

  1. On the client, remove any /etc/docker/certs.d/registry.lab.local:5000 folder and try docker pull. Note the exact x509 error.
  2. Fix 1: copy domain.crt to certs.d/registry.lab.local:5000/ca.crt. Pull works with no restart. Remove it again.
  3. Fix 2: install it in the system trust store, restart Docker, pull again. Undo it (update-ca-certificates --fresh on Ubuntu after deleting the file).
  4. Bypass: add the registry to insecure-registries, reload, pull. Check docker info for the insecure entry. Now pull using the IP instead of the name: it fails, because the setting is per host:port as written.
  5. Remove the bypass and put Fix 1 back. Write down which fix you would use on a fleet of 200 hosts, and why.

One good answer for step 5

Issue the registry certificate from the company (or platform) CA and put that CA into the system trust store through the host image or configuration management (Ansible, cloud-init). Every host, tool and container runtime then trusts it with no per-registry files, and certificate renewals need no client changes as long as the CA stays the same. Use certs.d only for one-off registries or for client certificates.

Going deeper: rotating certificates without an outage

Clients trust the CA, not the server certificate, so renewing the registry's certificate from the same CA needs no client change: plan renewals well before expiry and monitor notAfter (Prometheus blackbox exporter's probe_ssl_earliest_cert_expiry is a common choice). Replacing the CA is the hard case: distribute the new CA to all clients first (both CAs trusted side by side), then switch the registry to a certificate from the new CA, and only then remove the old CA. Doing it in the opposite order breaks every pull at once.

Recap

  • x509 errors mean trust, names (SANs), expiry/clock or a missing chain. Read the error, then openssl s_client.
  • Proper fixes: /etc/docker/certs.d/<host:port>/ca.crt (per registry, no restart) or the system trust store (company CA, restart Docker).
  • .crt = CA, .cert + .key = client certificate for mTLS.
  • insecure-registries skips verification and allows HTTP: labs and short bootstraps only, never public registries, remove it afterwards.
  • Podman: /etc/containers/certs.d, --tls-verify=false. containerd/Kubernetes: hosts.toml with ca or skip_verify, on every node.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.