Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›19 · AWS: ALB 502s during every deployment
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 19 of 19 · Real-world incident scenarios

AWS: ALB 502s during every deployment

Every rolling deployment on EKS causes a burst of 502/504 errors from the Application Load Balancer. Pods stop before the ALB stops sending them traffic, or receive traffic before they're ready. Line up pod termination with target deregistration using preStop delays, readiness gates and sensible timeouts.

Advanced
Key wordsALBAWS Load Balancer Controller502 Bad Gateway504target deregistrationderegistration delaypreStopterminationGracePeriodSecondspod readiness gatesIP targets

The page

Every afternoon deploy of the orders API shows a spike of 502s and a few 504s on the ALB for about a minute. Customers occasionally see failed checkouts during releases. The service uses the AWS Load Balancer Controller with IP targets; pods have no preStop hook and the default 30-second grace period.

First five minutes

  • Impact: short error bursts at every deploy; possibly failed writes if clients don't retry.
  • Correlate: overlay deploy times on ALB 5xx metrics.
  • Short term: deploy in quieter hours, slow the rollout (maxSurge/maxUnavailable), until fixed.

A shop is closing a till. If the cashier walks away the moment the manager decides to close it, customers already in that queue are left standing (502). The right way: put up the "this till is closing" sign first (deregister), let the queue finish (preStop wait), then leave. And a new till only opens when the cashier is actually sitting there (readiness gate).

Why it happens

When a pod is deleted, Kubernetes in parallel:

  1. Sends SIGTERM to the container (after any preStop hook), and
  2. Removes the pod from endpoints; the controller then deregisters the target from the ALB, which takes time to take effect.

If the app exits immediately on SIGTERM, the ALB may still send it requests for a few seconds → 502. If new pods are marked Ready before the ALB considers them healthy, rollouts remove old pods too early → capacity dips and 504/502.

Diagnose

  • ALB CloudWatch metrics (HTTPCode_ELB_502_Count, HTTPCode_ELB_504_Count) vs deploy timestamps.
  • ALB access logs: elb_status_code=502 with target_status_code=- means the ALB couldn't get a response from the target (connection reset/closed).
  • Check pods for readiness gates and preStop hooks.

Fix

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: orders-api
      lifecycle:
        preStop:
          exec:
            command: [ "sh", "-c", "sleep 25" ]   # or the built-in sleep action on recent Kubernetes versions
      readinessProbe:
        httpGet: { path: /ready, port: http }
  • preStop delay longer than the time the ALB needs to stop routing (deregistration + propagation), and the app should also drain gracefully on SIGTERM.
  • Grace period > preStop + drain time.
  • Readiness gates: label the namespace elbv2.k8s.aws/pod-readiness-gate-inject=enabled so new pods only count as Ready once healthy in the target group.
  • Deregistration delay: set via Ingress/Service annotations (deregistration_delay.timeout_seconds) to match your drain time; long values slow deploys.
  • Make sure ALB idle timeout and app keep-alive timeouts are compatible (the app's keep-alive should be longer than the ALB's idle timeout).

Verify

  • Deploy repeatedly under synthetic load (k6) and watch ALB 5xx stay at zero.
  • Check ALB access logs during a rollout.

Prevent

  • Bake preStop, grace period and readiness gates into the platform's app template/Helm chart (see Kubernetes Administration, lesson 17).
  • Alert on ALB 5xx rate and include deploy markers on dashboards.
  • The same pattern applies to other load balancers and ingress controllers: stop receiving traffic before stopping the process.

Try it: reproduce deploy-time 502s (sandbox EKS)

  1. Deploy a simple app behind an ALB (AWS Load Balancer Controller, IP targets) without preStop.
  2. Run k6 at a steady rate and trigger kubectl rollout restart; count 5xx in k6 and ALB metrics.
  3. Add the preStop sleep and grace period; repeat.
  4. Enable readiness gate injection for the namespace; repeat and compare.
  5. Enable ALB access logs and inspect elb vs target status codes.

Going deeper: zero-downtime releases

  • Combine with PodDisruptionBudgets for node drains, which have the same issue.
  • Long-lived connections (websockets, gRPC streams) need application-level draining, not just delays.
  • Canary and blue-green releases (see Kubernetes Administration, lesson 21) reduce blast radius further.

Recap

  • Deploy-time 502/504s: termination races deregistration, or pods become Ready before the ALB sees them healthy.
  • Fix with preStop delay + graceful drain, grace period, readiness gates, and aligned deregistration/idle timeouts.
  • Verify under load during rollouts; bake the pattern into app templates.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.