Traffic management · Prasanta Kumar Behera

Control traffic between services: weighted canaries with Gateway API or Istio resources, timeouts on every call, retries that help without amplifying outages, retry budgets, circuit breaking and outlier detection, and fault injection to test it all.

Canary releases

Send a small share of traffic to a new version, watch, then increase. With Gateway API (supported by Istio and Linkerd):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cart
  namespace: shop
spec:
  parentRefs:
    - group: ""
      kind: Service
      name: cart                   # mesh (east-west) route attached to the Service
      port: 8080
  rules:
    - backendRefs:
        - name: cart-v1
          port: 8080
          weight: 90
        - name: cart-v2
          port: 8080
          weight: 10

(Attaching routes to a Service for east-west traffic is the Gateway API "GAMMA" pattern; check your mesh version's support.) Automate the steps and the rollback with Argo Rollouts or Flagger, which adjust weights based on metrics (success rate, latency).

A new recipe at the school canteen: first one table in ten gets it (canary). If nobody gets a tummy ache, more tables get it. Timeouts mean "if the lunch isn't ready in 5 minutes, serve sandwiches". Retries mean "ask the kitchen again", but if every child asks again and again when the kitchen is struggling, the kitchen collapses (retry storm). So each table may only ask again a few times (retry budget).

Timeouts everywhere

Every call needs a timeout shorter than its caller's timeout:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
  namespace: shop
spec:
  hosts: [ payments ]
  http:
    - route:
        - destination: { host: payments }
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 600ms
        retryOn: connect-failure,refused-stream,unavailable

(Istio applies a small default retry policy for some connection-level failures unless you override it; check your version's defaults.)

Retries without storms

Retry idempotent requests on transient failures only.
Retry at one layer (usually the closest caller), not at every hop.
Use retry budgets where available (e.g. Linkerd ServiceProfiles: retries limited to a percentage of normal traffic) or cap outstanding retries (Istio DestinationRule connectionPool.http.maxRetries).
Make sure attempts × perTryTimeout fits inside the route timeout.

Scenario: retries turned a blip into an outage

The inventory service had a 30-second database hiccup. inventory itself retried DB calls 3 times; cart retried inventory 3 times; frontend retried cart 3 times. Load on the database rose ~27× during the hiccup, and the "blip" became a 20-minute outage.

What changes prevent it?

One retry layer: keep retries at the edge-most sensible point (e.g. frontend → cart), remove them elsewhere, or use budgets (retries ≤ ~20% of traffic).
Timeouts that shrink down the call chain, so callers give up before piling on.
Circuit breaking / outlier detection and load shedding so the struggling dependency gets breathing room.
Backoff with jitter in application retries.
Alerts on retry rate and upstream overflow metrics, not just errors.

Circuit breaking and outlier detection

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
  namespace: shop
spec:
  host: payments
  trafficPolicy:
    connectionPool:
      tcp: { maxConnections: 200 }
      http: { http1MaxPendingRequests: 100, maxRequestsPerConnection: 0, maxRetries: 10 }
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Fault injection

Test your timeouts and retries by injecting delays or errors for a subset of traffic:

    - fault:
        delay: { percentage: { value: 10 }, fixedDelay: 3s }
      route:
        - destination: { host: payments }

(Part of a VirtualService http list; run in staging or behind a header match.)

Try it: canary and resilience

Deploy cart-v1 and cart-v2 with separate Services and a 90/10 HTTPRoute; generate load and count responses per version.
Add a 2 s timeout and inject a 3 s delay for 50% of requests; observe 504s and latency.
Build the three-layer retry chain from the scenario in a lab (three small services), make the bottom one fail briefly, and measure request amplification with mesh metrics.
Remove retries from two layers and add outlier detection; repeat the test.
(Stretch) Automate the canary with Flagger or Argo Rollouts using mesh success-rate metrics.

Going deeper: traffic policy as a product

Provide defaults (timeouts, retry policy) per platform, so teams don't start from zero or copy bad examples.
Keep routing config next to the app (GitOps), reviewed like code.
Watch for conflicting configuration (multiple routes for the same host); istioctl analyze helps.

Recap

Canaries via weighted routing (Gateway API HTTPRoute or Istio resources), automated with Argo Rollouts/Flagger.
Timeouts on every call, shrinking down the chain.
Retries only for idempotent, transient failures, at one layer, with budgets.
Circuit breaking/outlier detection and fault injection to prove resilience.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.