Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›15 · A broken admission webhook blocks every deploy
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 15 of 19 · Real-world incident scenarios

A broken admission webhook blocks every deploy

Suddenly nothing can be created or updated: 'failed calling webhook … connection refused'. A policy or mutation webhook is down and set to fail closed. Find the webhook, restore it or temporarily relax it safely, and design webhooks that can't take the whole cluster hostage.

Advanced
Key wordsadmission webhooksValidatingWebhookConfigurationMutatingWebhookConfigurationfailurePolicytimeoutSecondsnamespaceSelectorkube-system exclusionpolicy enginesescape hatch

The page

11:05 — every deploy fails: Internal error occurred: failed calling webhook "validate.policy.example.com": … connect: connection refused. Autoscaling can't create pods either. The policy engine's pods were all scheduled on one node, which just failed.

First five minutes

  • Impact: no creates/updates for resources the webhook covers: deploys, scaling, sometimes even node or pod replacements. Running workloads are untouched, but self-healing is impaired: this can become SEV1 fast.
  • Identify the webhook from the error message.

The school has a security guard who checks every visitor. That's good, until the guard falls ill and the rule says "nobody enters without the guard". Now even the nurse can't get in to help. Good rules say "if the guard is away, staff with badges may still enter" and there's always more than one guard.

Diagnose

$ kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
$ kubectl get validatingwebhookconfiguration policy-validate -o yaml | grep -E 'failurePolicy|timeoutSeconds|namespaceSelector' -A3
    failurePolicy: Fail
    timeoutSeconds: 10
$ kubectl -n policy-system get pods -o wide
$ kubectl -n policy-system get endpoints policy-webhook
NAME             ENDPOINTS   AGE
policy-webhook   <none>      90d

No endpoints: the webhook backend is down. Other causes: an expired webhook certificate (TLS errors), network policies blocking the API server from reaching it, or timeouts under load.

Recover

  1. Best: restore the webhook backend (reschedule its pods, fix the node, renew its serving certificate).
  2. If its pods can't be created because the webhook blocks its own namespace (a design bug), or restoration will take long: temporarily fail open for the affected webhook, with a record in the timeline:
$ kubectl get validatingwebhookconfiguration policy-validate -o yaml > /tmp/policy-validate.backup.yaml
$ kubectl patch validatingwebhookconfiguration policy-validate --type=json \
    -p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'
  1. Once the backend is healthy, revert to the original setting (or let GitOps restore it) and confirm policies apply again.
  2. Last resort: delete the configuration (after backing it up) and restore it after recovery.

Fail-open means policies aren't enforced for a while: involve security if these are security controls, and review what was created during that window.

Verify

  • A test deployment succeeds; the webhook sees requests again (its logs/metrics).
  • Configuration restored to the intended failurePolicy.

Prevent

  • Run webhooks highly available: 2+ replicas spread across nodes, PodDisruptionBudget, priority class.
  • Scope them: exclude kube-system, the webhook's own namespace and other critical namespaces (namespaceSelector), and only match the resources and operations needed.
  • Choose failurePolicy deliberately per webhook (Fail for true security gates, Ignore for convenience mutations), with short timeouts.
  • Monitor webhook latency and rejections (API server metrics apiserver_admission_webhook_*) and the webhook's certificate expiry.

Try it: a webhook that blocks everything (lab)

  1. Install a policy engine (Kyverno or Gatekeeper) in a lab cluster with its default settings and note its webhook configurations.
  2. Scale its deployment to 0 and try to create a pod in a covered namespace; read the error.
  3. Check whether its own namespace is excluded; could it recover itself?
  4. Practise the temporary fail-open patch and the revert.
  5. Configure 2 replicas with anti-affinity and a PDB.

Going deeper: admission as a dependency

  • Treat admission webhooks as tier-0 components with SLOs.
  • Prefer in-process admission where possible (e.g. ValidatingAdmissionPolicy with CEL) for simple rules, avoiding a network hop.
  • Keep a documented break-glass procedure, including who may fail open security webhooks.

Recap

  • "failed calling webhook" on every write → a fail-closed webhook is unreachable.
  • Restore the backend first; if needed, temporarily fail open (backed up, recorded, reverted).
  • Prevent with HA webhooks, scoped selectors (exclude system and self), deliberate failurePolicy/timeouts, and monitoring.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.