Blog / Kubernetes

Default to a policy engine, not custom webhooks

Kubernetes admission control: when to mutate, validate, or use a policy engine.

Default to a policy engine, not custom webhooks
Topicsadmission controlmutating webhookvalidating webhookKyvernoOPA Gatekeeperpolicy engineimage signatureverifyImagesfailurePolicy

Stop writing custom webhooks. Write policy instead.

The admission pipeline

Every object you kubectl apply passes through a pipeline before it's persisted:

Authentication → Authorization (RBAC) → Mutating admission → Schema validation → Validating admission → etcd

Mutating webhooks always run before validating webhooks: defaults get injected first, then the final object gets checked.

Three tools in that pipeline

  • Mutating webhook: changes the object before it's saved. Example: a pod applied without resource limits gets defaults patched in automatically, no incident required.
  • Validating webhook: allows or rejects, with no changes, and runs on the final object after mutation. Examples: "no privileged containers", "images must come from our internal registry".
  • Policy engine (Kyverno / OPA Gatekeeper): both of the above, minus writing and operating a custom webhook server for every rule. Declarative CRDs instead of Go microservices, plus audit/dry-run mode so you see what a policy would break before it blocks anything.

One example that ties both stages together: signature verification

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "registry.internal.com/*"
          failureAction: Enforce
          attestors:
            - count: 1
              entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----
          mutateDigest: true
          verifyDigest: true

No custom webhook code: just a CRD naming the registry and the key to trust. Signed and trusted → the pod runs. Unsigned or tampered → rejected at admission, before it ever touches a node.

Where this matters most

Environments where trust can't be assumed by default: air-gapped clusters, regulated infrastructure, and anywhere bare-metal provisioning means an image goes straight from registry to hardware with no manual gate in between. In a zero-touch-provisioning pipeline, this is the one check standing between "verified software" and whatever happens to be in the registry.

My rule of thumb

Default to a policy engine. Only drop to a hand-rolled webhook when the logic needs something a policy language genuinely can't express.

One gotcha: failurePolicy: Fail on an unreachable webhook blocks admissions cluster-wide.

Where do you draw this line: custom webhook or policy engine?