Secrets management · Prasanta Kumar Behera

Protect credentials end to end: encrypt Secrets at rest in etcd (ideally with KMS), keep them out of Git with External Secrets or Sealed Secrets, use Vault or a cloud secret manager as the source of truth, and rotate them.

Three places a secret can leak

In Git: someone commits a Secret YAML (base64 isn't encryption).
In etcd and its backups: Secrets are stored readable unless encryption at rest is on.
Through the API: anyone with get/list on Secrets (see lesson 06).

A good design closes all three.

A secret is like the combination to a safe. Don't write it on the classroom whiteboard (Git). Don't leave it in the filing cabinet in plain writing (etcd); write it in code, and keep the code book in a different building (KMS). And only give it to the people who need it (RBAC), fetching a fresh copy from the head office (a secret manager) when it changes.

1. Encrypt Secrets at rest in etcd

Create an EncryptionConfiguration and point the API server at it:

# /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:                         # or kms (v2) with an external key service: preferred
          keys:
            - name: key1
              secret: <32 random bytes, base64>   # head -c 32 /dev/urandom | base64
      - identity: {}                    # lets the API server still read old, unencrypted data

Add --encryption-provider-config=/etc/kubernetes/enc/enc.yaml to kube-apiserver (and mount the directory into the static pod).
The first provider encrypts new writes; the others can decrypt. After enabling, rewrite all Secrets so existing ones get encrypted: kubectl get secrets -A -o json | kubectl replace -f -.
Verify directly in etcd: the stored value starts with k8s:enc:aescbc:v1:key1: instead of readable text.

Prefer KMS v2 (GA in recent Kubernetes versions): a KMS plugin talks to AWS KMS, Azure Key Vault, Google Cloud KMS, Vault Transit or an HSM, so the key-encryption key never sits on the control-plane disk. Managed Kubernetes services usually offer this as a setting (e.g. EKS envelope encryption with KMS).

Back up the key with the etcd backup, separately

An encrypted etcd snapshot without the key is unrecoverable. Store the encryption configuration (or KMS access) as carefully as the CA, and not in the same place as the snapshots.

2. Keep secrets out of Git

Option A: External Secrets Operator (ESO). The source of truth is a secret manager; Git holds only a reference:

apiVersion: external-secrets.io/v1               # older ESO releases: v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-platform                         # a ClusterSecretStore configured by the platform team
    kind: ClusterSecretStore
  target:
    name: db-credentials                         # the Kubernetes Secret to create
  data:
    - secretKey: password
      remoteRef:
        key: shop/db
        property: password

Option B: Sealed Secrets. Encrypt a Secret with the cluster's public key; only the controller in that cluster can decrypt it, so the sealed version can live in Git:

$ kubectl create secret generic db-credentials -n shop --from-literal=password='S3cr3t!' --dry-run=client -o yaml \
    | kubeseal --format yaml > db-credentials-sealed.yaml
$ git add db-credentials-sealed.yaml

	External Secrets	Sealed Secrets
Source of truth	External secret manager	Git (encrypted)
Rotation	Change it in the manager; synced automatically	Re-seal and commit
Best for	Organisations with Vault or a cloud secret manager	Small setups without a secret manager

3. Use secrets carefully in workloads

Mount as files rather than env vars where possible (env vars leak into child processes and crash dumps).
One ServiceAccount per app, with RBAC that allows reading only its own Secrets, and usually none at all, because the kubelet mounts them without the app calling the API.
Consider the Secrets Store CSI driver or Vault Agent to fetch secrets directly at runtime, so they never become Kubernetes Secret objects.
Rotate: short-lived dynamic credentials (Vault database secrets engine, cloud IAM roles) beat long-lived passwords.

Try it: encryption at rest on kind

Create a Secret demo with value hello. On the control-plane node, read it straight from etcd (with the etcd client certs from Kubernetes Administration, lesson 13) and find hello in plain text.
Write an EncryptionConfiguration with an aescbc key, copy it to the node, add the flag and a volume mount to the kube-apiserver manifest, and wait for the API server to restart.
Rewrite all Secrets with kubectl get secrets -A -o json | kubectl replace -f -.
Read demo from etcd again: it now starts with k8s:enc:aescbc:v1:key1:, and hello is gone.
(Optional) Install Sealed Secrets, seal a Secret, delete the original, and apply the sealed version.

Going deeper: secrets as a platform service

Offer one ClusterSecretStore per backend, managed by the platform team, with per-namespace access policies in the secret manager itself.
Use workload identity (ServiceAccount token → Vault or cloud IAM) so ESO and apps authenticate without static credentials.
Detect leaks: secret scanning in CI and in Git history (gitleaks, trufflehog), and alerts on unusual Secret reads in audit logs.
Plan key rotation for the encryption configuration: add a new first key, rewrite all Secrets, then remove the old key.

Recap

Close three leaks: Git, etcd/backups, API access.
Encryption at rest with an EncryptionConfiguration, preferably KMS v2; rewrite existing Secrets; protect the key separately.
External Secrets Operator (secret manager as source of truth) or Sealed Secrets (encrypted in Git).
Mount as files, least-privilege RBAC, and rotate, ideally with short-lived dynamic credentials.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.