Cluster PKI & certificates · Prasanta Kumar Behera

Map every certificate in a Kubernetes cluster: the three CAs, who presents which certificate to whom, how kubelet certificates rotate, how to inspect expiry and SANs, and how to avoid the classic 'certificate expired' outage.

Kubernetes runs on certificates

Almost every connection inside a cluster is mutual TLS: both sides prove who they are with certificates. The API server presents a certificate to kubectl; kubectl presents one back; kubelets, the scheduler, the controller manager and etcd do the same.

A CA (certificate authority) is like the school office that prints ID cards. Every teacher, pupil and visitor carries one, and everyone trusts cards printed by their office. A card has a name and an expiry date. When the date passes, the door readers stop letting that person in, even though they're the same person as yesterday.

Three CAs in a kubeadm cluster

All under /etc/kubernetes/pki/:

CA	Files	Signs certificates for
Cluster CA	`ca.crt`, `ca.key`	API server serving cert, kubelets, admin and component clients, users
etcd CA	`etcd/ca.crt`, `etcd/ca.key`	etcd server and peer certs, the API server's etcd client cert
Front-proxy CA	`front-proxy-ca.crt`, `.key`	The API server's client cert for aggregated APIs (e.g. metrics-server)

Plus sa.key / sa.pub: a key pair (not certificates) used to sign and verify ServiceAccount tokens.

Who shows which certificate to whom

Connection	Server presents	Client presents
kubectl → API server	`apiserver.crt`	a user cert (in kubeconfig) or a token
API server → etcd	`etcd/server.crt`	`apiserver-etcd-client.crt`
API server → kubelet (logs, exec)	kubelet serving cert	`apiserver-kubelet-client.crt`
kubelet → API server	`apiserver.crt`	kubelet client cert (`/var/lib/kubelet/pki/kubelet-client-current.pem`)
scheduler / controller-manager → API server	`apiserver.crt`	certs embedded in their kubeconfigs

Inspecting certificates

$ sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -enddate
subject=CN = kube-apiserver
issuer=CN = kubernetes
notAfter=Sep 27 09:12:44 2027 GMT
$ sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:cp1, DNS:k8s-api, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc,
    DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.10.0.10

The SANs list every name and IP clients may use to reach the API server, including the in-cluster Service IP 10.96.0.1. Connect with a name not in that list and TLS fails.

To add a SAN in kubeadm: put it in apiServer.certSANs in the kubeadm config, move the old apiserver.crt/.key aside, run kubeadm init phase certs apiserver --config kubeadm.yaml, then restart the API server (move its static pod manifest out and back in).

Expiry and rotation

Certificate	Default lifetime (kubeadm)	Renewed by
CAs	10 years	You, deliberately (a planned CA rotation)
Control-plane leaf certs	1 year	`kubeadm upgrade apply` or `kubeadm certs renew`
kubelet client cert	1 year	Automatic rotation by the kubelet (enabled by kubeadm)
kubelet serving cert	Self-signed by default	Or `serverTLSBootstrap: true` → CSRs that must be approved

$ sudo kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 27, 2027 09:12 UTC   364d            ca                      no
apiserver                  Sep 27, 2027 09:12 UTC   364d            ca                      no
apiserver-etcd-client      Sep 27, 2027 09:12 UTC   364d            etcd-ca                 no
...

Serving certificates that need approval

With serverTLSBootstrap: true, each kubelet requests a serving certificate through the CSR API, and nothing approves them automatically. Approve only CSRs whose requested names match real nodes (kubectl get csr, then kubectl describe csr), or run an approver that checks this for you. Blindly approving CSRs lets anyone mint certificates.

Try it: map your cluster's PKI (kind or kubeadm)

On the control-plane node (docker exec -it lab-control-plane bash for kind), list /etc/kubernetes/pki and match each file to the table above.
Print the subject, issuer and expiry of apiserver.crt, apiserver-etcd-client.crt and front-proxy-client.crt. Which CA signed each?
Print the API server's SANs. Is 10.96.0.1 there? Why?
From your laptop, check the live certificate: echo | openssl s_client -connect 127.0.0.1:<kind-api-port> 2>/dev/null | openssl x509 -noout -dates (find the port with kubectl cluster-info).
On a worker, find the kubelet's client certificate and its expiry.

Going deeper: certificates as an operational risk

Monitor expiry everywhere: API server client certs (apiserver_client_certificate_expiration_seconds), kubelet certs, etcd, ingress and webhook certificates. Alert at 30 and 7 days.
Certificates can't be revoked in Kubernetes. A leaked admin client cert is valid until it expires, which is why humans should use OIDC (lessons 03–04), not long-lived certs.
Since kubeadm v1.29, admin.conf is bound to the kubeadm:cluster-admins group via RBAC, and a separate super-admin.conf holds the system:masters identity (which bypasses RBAC). Store super-admin.conf like a break-glass key.
Plan a CA rotation long before year 10. It's a multi-step, rehearsed procedure, not a quick renew.

Recap

Three CAs (cluster, etcd, front-proxy) plus the SA signing key pair.
Mutual TLS everywhere: know who presents which certificate.
Check expiry (kubeadm certs check-expiration) and SANs (openssl x509 -ext subjectAltName).
Leaf certs: 1 year, renewed by upgrades; kubelet client certs rotate automatically; serving-cert CSRs need careful approval.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.