Lesson 09 of 13 · Level 4 — Daemon & certificates
Registry certificates & the insecure bypass
Fix the x509 errors that stop docker pull and docker login: trust a registry per host with certs.d, trust a company CA system-wide, use client certificates, and understand exactly what the insecure-registries bypass does, when it's acceptable, and the equivalents for Podman and containerd.
Why TLS matters for images
When a host pulls an image, it's about to run whatever comes back as root-equivalent code. TLS verification is what proves the bytes came from your registry and not from something in between. That's why Docker refuses a certificate it can't verify, and why the fix should almost always be "teach Docker to trust the right CA", not "stop checking".
A courier arrives with a parcel and an ID badge. Checking the badge against the company's list of real badges is TLS verification. Adding the courier company to your list (trusting the CA) is the fix. "Accept parcels from anyone with any badge" is the insecure bypass: it works right up to the day someone brings you the wrong parcel.
Read the error first
| Error | Meaning | Fix |
|---|---|---|
x509: certificate signed by unknown authority |
The chain doesn't lead to a CA this host trusts | Trust the CA: certs.d or the system store |
x509: certificate is valid for registry.lab.local, not 192.168.56.10 |
You used a name or IP that isn't in the certificate's SANs | Use the right name, or reissue with that SAN |
x509: certificate relies on legacy Common Name field, use SANs instead |
The cert has only a CN | Reissue with subjectAltName |
x509: certificate has expired or is not yet valid |
Expired, or the host's clock is wrong | Renew, or fix NTP (timedatectl) |
http: server gave HTTP response to HTTPS client |
The registry serves plain HTTP | Give it TLS (or, lab only, the bypass) |
remote error: tls: certificate required / bad certificate |
The registry wants a client certificate | Add client.cert / client.key |
Then look at what the registry actually presents:
$ openssl s_client -connect registry.lab.local:5000 -servername registry.lab.local -showcerts </dev/null \
| openssl x509 -noout -subject -issuer -enddate -ext subjectAltName
subject=CN = registry.lab.local
issuer=CN = Lab Internal CA
notAfter=Sep 28 10:00:00 2027 GMT
X509v3 Subject Alternative Name:
DNS:registry.lab.local, IP Address:192.168.56.10
If the server sends only its own certificate but it was signed by an intermediate CA, clients fail too: configure the registry with the full chain (server certificate followed by the intermediate) in its certificate file.
Fix 1: trust one registry with certs.d (recommended)
Docker looks for per-registry trust in /etc/docker/certs.d/<host:port>/:
/etc/docker/certs.d/
└── registry.lab.local:5000/
├── ca.crt # CA that signed the registry certificate (or the self-signed cert itself)
├── client.cert # optional: client certificate for mutual TLS
└── client.key # optional: its private key
$ sudo mkdir -p /etc/docker/certs.d/registry.lab.local:5000
$ sudo cp lab-ca.crt /etc/docker/certs.d/registry.lab.local:5000/ca.crt
$ docker pull registry.lab.local:5000/tools/alpine:3.20 # no restart needed
The rules that catch people out:
- The folder name must match the image reference exactly.
registry.lab.local:5000andregistry.lab.local(port 443) are different folders, and so are a name and its IP. - Extensions mean something:
.crt= CA to trust,.cert= client certificate,.key= client key. A client certificate saved as.crtis treated as a CA and mTLS fails. - It's read on each connection, so no daemon restart. It applies only to the Docker daemon on that host.
curl, Podman and Kubernetes nodes have their own trust.
Fix 2: trust a CA system-wide
When all internal services use one company CA, add it to the operating system's trust store. Docker (and curl, git, most tools) then trusts every certificate that CA signs:
# Debian / Ubuntu
$ sudo cp corp-root-ca.crt /usr/local/share/ca-certificates/corp-root-ca.crt
$ sudo update-ca-certificates
# RHEL / Rocky / Fedora
$ sudo cp corp-root-ca.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust
$ sudo systemctl restart docker # the daemon loads the system pool at start
This is also the fix for the classic corporate case: every pull, even from Docker Hub, fails with unknown authority, because a TLS-inspecting proxy re-signs all HTTPS traffic with the company's CA. Install that CA; don't mark Docker Hub insecure.
Fix 3: client certificates (mutual TLS)
Some registries (often at edge sites or in regulated environments) require the client to present a certificate as well:
$ sudo cp edge-node-01.crt /etc/docker/certs.d/registry.lab.local:5000/client.cert
$ sudo cp edge-node-01.key /etc/docker/certs.d/registry.lab.local:5000/client.key
$ sudo chmod 600 /etc/docker/certs.d/registry.lab.local:5000/client.key
The registry side sets REGISTRY_HTTP_TLS_CLIENTCAS_0=/certs/client-ca.crt so it accepts only clients signed by that CA.
The bypass: insecure-registries
When you truly can't get a trusted certificate (a throwaway lab, a registry on plain HTTP during a first bootstrap), Docker has an explicit escape hatch in daemon.json:
{
"insecure-registries": ["registry.lab.local:5000", "10.10.0.0/24"]
}
$ sudo dockerd --validate --config-file /etc/docker/daemon.json
$ sudo systemctl reload docker
$ docker info -f '{{json .RegistryConfig.IndexConfigs}}' | python3 -m json.tool | grep -B2 -A2 '"Secure": false'
What it does for those registries (by host:port, or a whole CIDR range):
- Docker still tries HTTPS first, but ignores certificate errors (unknown CA, wrong name, expired).
- If HTTPS isn't available at all, it falls back to plain HTTP.
What you give up:
- No proof of identity. Anyone who can redirect traffic (DNS, ARP, a compromised router) can serve you any image, and you'll run it.
- With HTTP fallback, passwords from
docker loginand every layer travel in clear text. - It's easy to forget, and it's set per host, so it spreads quietly across a fleet.
When the bypass is acceptable
- A local lab or CI sandbox with no real data, on an isolated network.
- A short bootstrap window where a registry comes up before its certificate, with a ticket to remove the setting.
Not acceptable: production hosts, anything reachable from the internet, and never docker.io or another public registry. localhost / 127.0.0.0/8 is treated as insecure by Docker automatically, so a local registry on localhost:5000 works without any setting.
Remove it as soon as the real certificate is in place:
$ sudoedit /etc/docker/daemon.json # delete the insecure-registries entry
$ sudo systemctl reload docker
The same fixes for Podman and containerd
Podman (and Buildah, Skopeo) read per-registry trust from /etc/containers/certs.d/<host:port>/ with the same file names. The bypass is per command or per registry:
$ podman pull --tls-verify=false registry.lab.local:5000/tools/alpine:3.20
# /etc/containers/registries.conf.d/lab.conf
[[registry]]
location = "registry.lab.local:5000"
insecure = true
containerd (Kubernetes nodes) uses one hosts.toml per registry, as long as the CRI registry config_path points at /etc/containerd/certs.d (the key's location in config.toml differs between containerd 1.x and 2.x, so check the docs for your version):
# /etc/containerd/certs.d/registry.lab.local:5000/hosts.toml
server = "https://registry.lab.local:5000"
[host."https://registry.lab.local:5000"]
capabilities = ["pull", "resolve"]
ca = "/etc/containerd/certs.d/registry.lab.local:5000/ca.crt"
# client = [["/path/client.crt", "/path/client.key"]] # mutual TLS
# skip_verify = true # the bypass, labs only
Because every node pulls its own images, trust has to be on every node, usually delivered by the node image or configuration management rather than by hand.
Try it: break trust, then fix it three ways
Use the registry from lesson 7 (self-signed certificate) and a second VM as the client.
- On the client, remove any
/etc/docker/certs.d/registry.lab.local:5000folder and trydocker pull. Note the exact x509 error. - Fix 1: copy
domain.crttocerts.d/registry.lab.local:5000/ca.crt. Pull works with no restart. Remove it again. - Fix 2: install it in the system trust store, restart Docker, pull again. Undo it (
update-ca-certificates --freshon Ubuntu after deleting the file). - Bypass: add the registry to
insecure-registries, reload, pull. Checkdocker infofor the insecure entry. Now pull using the IP instead of the name: it fails, because the setting is per host:port as written. - Remove the bypass and put Fix 1 back. Write down which fix you would use on a fleet of 200 hosts, and why.
One good answer for step 5
Issue the registry certificate from the company (or platform) CA and put that CA into the system trust store through the host image or configuration management (Ansible, cloud-init). Every host, tool and container runtime then trusts it with no per-registry files, and certificate renewals need no client changes as long as the CA stays the same. Use certs.d only for one-off registries or for client certificates.
Going deeper: rotating certificates without an outage
Clients trust the CA, not the server certificate, so renewing the registry's certificate from the same CA needs no client change: plan renewals well before expiry and monitor notAfter (Prometheus blackbox exporter's probe_ssl_earliest_cert_expiry is a common choice). Replacing the CA is the hard case: distribute the new CA to all clients first (both CAs trusted side by side), then switch the registry to a certificate from the new CA, and only then remove the old CA. Doing it in the opposite order breaks every pull at once.
Recap
- x509 errors mean trust, names (SANs), expiry/clock or a missing chain. Read the error, then
openssl s_client. - Proper fixes:
/etc/docker/certs.d/<host:port>/ca.crt(per registry, no restart) or the system trust store (company CA, restart Docker). .crt= CA,.cert+.key= client certificate for mTLS.insecure-registriesskips verification and allows HTTP: labs and short bootstraps only, never public registries, remove it afterwards.- Podman:
/etc/containers/certs.d,--tls-verify=false. containerd/Kubernetes:hosts.tomlwithcaorskip_verify, on every node.
This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.