Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›09 · Certificates expired after a failed rotation
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 09 of 19 · Real-world incident scenarios

Certificates expired after a failed rotation

kubectl suddenly fails with 'x509: certificate has expired or is not yet valid'. Work out which certificates expired (control plane, kubelet, ingress), renew kubeadm certificates safely, restart the control plane, fix nodes whose kubelet rotation failed, and make sure expiry never surprises you again.

Advanced
Key wordsx509 expiredkubeadm certscheck-expirationrenew allkubelet client certificaterotateCertificatesadmin.confcert-managercertificate monitoring

The page

Monday 08:15 — CI fails to deploy to the edge staging cluster: Unable to connect to the server: x509: certificate has expired or is not yet valid. Workloads are still serving, but no one can change anything and GitOps shows errors. The cluster was built exactly a year ago and never upgraded. The scheduled renewal job had been failing quietly for weeks.

First five minutes

  • Impact: running pods usually keep serving; the control plane can't be reached or can't talk to itself, so no changes, no self-healing for long.
  • Check the clock first: "not yet valid" can mean time drift, not expiry (lesson 13).
  • Identify which certificate: API server, kubelet, front-proxy, etcd, or an application/ingress certificate.

Every staff member at school has an ID badge with an expiry date. If the head teacher's badge expires, the office stops recognising her: nothing new gets signed until new badges are printed and everyone puts the new badge on (restart). A calendar reminder a month earlier would have avoided the whole morning.

Diagnose

$ sudo kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 26, 2026 08:02 UTC   <invalid>       ca                      no
apiserver                  Sep 26, 2026 08:02 UTC   <invalid>       ca                      no
...
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Sep 24, 2035 08:02 UTC   9y              no
$ echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -dates

Renew the control plane (kubeadm)

On each control-plane node:

$ sudo cp -a /etc/kubernetes /root/kubernetes-backup-$(date +%F)     # backup first
$ sudo kubeadm certs renew all
$ sudo mkdir -p /root/manifests-tmp && sudo mv /etc/kubernetes/manifests/*.yaml /root/manifests-tmp/
$ sleep 30                                                           # kubelet stops the static pods
$ sudo mv /root/manifests-tmp/*.yaml /etc/kubernetes/manifests/      # kubelet starts them with new certs
$ sudo cp /etc/kubernetes/admin.conf ~/.kube/config
$ kubectl get nodes

Restart control-plane nodes one at a time in HA clusters. Update any copied kubeconfigs (CI, GitOps tools, automation) that embedded the old admin credentials; better, don't use admin.conf for automation at all.

Nodes whose kubelet certificate expired

Kubelets normally rotate their client certificates automatically (rotateCertificates: true in the kubelet config). A node that was powered off for a long time, or had rotation disabled, may end up with an expired client cert and go NotReady. Options:

  • Follow the kubeadm documentation for kubelet client certificate rotation failure (it regenerates the node's kubelet kubeconfig using the cluster CA from a control-plane node).
  • Or, for workers, drain, reset and re-join the node (kubeadm reset, then a fresh kubeadm join); this is often the simplest.

Application and ingress certificates

If users see browser errors, check cert-manager (kubectl get certificate,certificaterequest,order,challenge -A): failed ACME challenges, DNS changes, rate limits or a broken issuer are the usual causes. Fix the issuer, then delete the failed CertificateRequest to retry.

Prevent

  • Alert on expiry at 30 and 14 days for kubeadm certificates (a small exporter or scheduled kubeadm certs check-expiration check), API server client cert metrics, and cert-manager's certificate expiry metrics.
  • Upgrade regularly (upgrades renew kubeadm certificates) and track cluster age.
  • Monitor the renewal job itself: a silent failure is how this incident happened.
  • Write the runbook now (lesson 04), including the static-pod restart step.

Try it: expire and renew (lab only)

  1. On a kubeadm lab, run kubeadm certs check-expiration and note the dates.
  2. Renew all certificates and restart the control-plane pods using the manifest-move method; verify with openssl s_client that the API server presents the new cert.
  3. Update your kubeconfig from admin.conf.
  4. Stop a worker's kubelet, then read its client certificate expiry with openssl.
  5. Write an expiry alert (or a cron job that posts to chat) for 30 days before expiry.

Going deeper: PKI hygiene

  • Plan CA rotation long before the 10-year CA expires; it's a bigger procedure than leaf renewal.
  • Keep external PKI options in mind for regulated environments (see Kubernetes Security & Hardening, lesson 01).
  • Short-lived, automatically rotated certificates beat long-lived ones, as long as rotation is monitored.

Recap

  • Identify which certificate expired; rule out clock drift.
  • kubeadm: backup → renew all → restart control-plane static pods (one node at a time) → refresh kubeconfigs.
  • Nodes with expired kubelet certs: follow the kubeadm procedure or reset and re-join.
  • Alert on expiry, upgrade regularly, and monitor the renewal automation.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.