GitOps Principles & Practice›04 · Secrets in GitOps

Lesson 04 of 7 · Level 2 — Running it

Secrets in GitOps

GitOps puts everything in Git, except secret values. Compare the three standard approaches (Sealed Secrets, SOPS-encrypted files, and the External Secrets Operator referencing a secret manager), what each stores in Git, how rotation works, and which to choose.

Practitioner → Advanced
Key wordssecretsGitOpsSealed SecretskubesealSOPSageKMSExternal Secrets OperatorExternalSecretSecretStoreVaultAWS Secrets Managerrotation

The rule

Secret values never go into Git in plain text, not even in private repositories. Git history is forever, and repositories are cloned, mirrored and backed up far beyond your control. But GitOps wants everything declared in Git. Three patterns resolve this:

Approach In Git Value lives Decrypted/fetched by
Sealed Secrets A SealedSecret: the value encrypted with the cluster's public key Git (encrypted) Controller in the cluster (private key)
SOPS The Secret with values encrypted (age, PGP or cloud KMS keys) Git (encrypted) The GitOps agent (Flux natively; Argo CD via a plugin)
External Secrets Operator An ExternalSecret: a reference to a key A secret manager (Vault, AWS Secrets Manager, GCP, Azure) The operator, using its own identity

Three ways to handle the office safe code. Write it in the shared notebook in a cipher only the safe can read (Sealed Secrets). Write it in a cipher that anyone holding the right key can read (SOPS). Or write "the code is in the manager's vault, drawer 3" and let a trusted runner fetch it each morning (External Secrets).

Sealed Secrets

$ kubectl create secret generic db -n shop --from-literal=password='S3cret!' --dry-run=client -o yaml > db.yaml
$ kubeseal --format yaml < db.yaml > db-sealed.yaml && rm db.yaml
$ git add db-sealed.yaml && git commit -m "shop: db credentials (sealed)"

Simple and fully in-cluster. Downsides: each value is sealed for one cluster's key; back up the controller's keys, or you can't decrypt after rebuilding the cluster; rotation means re-sealing and committing.

SOPS

SOPS encrypts only the values, so diffs stay readable:

apiVersion: v1
kind: Secret
metadata: { name: db, namespace: shop }
stringData:
  password: ENC[AES256_GCM,data:9x8f...,iv:...,tag:...,type:str]
sops:
  age:
    - recipient: age1qz...

Keys can be age keys or cloud KMS keys (so access to decrypt is controlled by IAM). Flux decrypts SOPS natively; Argo CD needs a plugin (for example helm-secrets or a config management plugin). Good when teams want everything, including encrypted values, reviewed in Git.

The value stays in a secret manager; Git holds only a reference:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: db, namespace: shop }
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: aws-secrets-manager }
  target: { name: db }                     # the Kubernetes Secret to create
  data:
    - secretKey: password
      remoteRef: { key: prod/shop/db, property: password }
  • The operator authenticates with its own identity (for example EKS Pod Identity or IRSA), scoped to the paths each team may read.
  • Rotation happens in the secret manager; the Kubernetes Secret refreshes on the interval, and nothing changes in Git.
  • Check the API version your installed release serves (older releases use v1beta1).

Choosing

Situation Good fit
Small setup, single cluster, no external secret manager Sealed Secrets
Everything must be reviewable in Git, Flux, cloud KMS available SOPS
Multiple clusters, rotation, central audit, cloud or Vault already used External Secrets Operator

Whichever you choose, apps should still read secrets from mounted files or environment injected by Kubernetes, and access to the Secret objects themselves should be limited with RBAC.

Try it: two approaches side by side

  1. On a kind cluster, install Sealed Secrets; seal a secret, commit it to a test config repo and let the GitOps agent apply it.
  2. Install the External Secrets Operator with a fake or local backend (for example its Kubernetes provider, or LocalStack Secrets Manager) and sync one value.
  3. Change the value in the backend and watch the Kubernetes Secret update after the refresh interval, with no Git change.

Recap

  • Plain-text secrets never in Git.
  • Sealed Secrets: encrypted per cluster, simple; back up keys.
  • SOPS: encrypted values in Git with age/KMS keys; native in Flux.
  • External Secrets: references in Git, values in a secret manager, easy rotation; the usual choice at scale.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.