Prasanta Kumar Behera
Networking Deep Dive›12 · Capstone: one request, every layer
Learning Hub / Kubernetes & Platform / Networking Deep Dive

Lesson 12 of 12 · Application Layer

Capstone: one request, every layer

Trace one HTTPS request from a laptop to a pod and back through every layer in this course, then diagnose five injected faults (DNS, MTU, policy, TLS, load balancing) purely from symptoms and evidence.

Advanced
Key wordscapstonerequest pathlayer-by-layerfault injectionevidenceRCA

The request

laptop ──DNS──► shop.example.com = 10.10.0.200 (MetalLB VIP)
       ──TCP/TLS (SNI, ALPN h2)──► ingress/gateway pod (TLS termination, HTTP routing)
       ──HTTP──► Service web (ClusterIP, kube-proxy/eBPF DNAT) ──► web pod (veth, CNI, maybe overlay)
       web pod ──DNS (CoreDNS)──► api.shop ──HTTP──► Service api ──► api pod (NetworkPolicy allows)

You're following a letter's journey through the whole postal system: the address book (DNS), the roads (routing and MTU), the sorting office (load balancer), the locked mailbox (TLS), the building receptionist (ingress), and finally the right desk (the pod). When a letter goes missing, you check each stop in order until you find where it was dropped.

Part 1: trace a healthy request

For each hop, collect one piece of evidence that it works:

  1. DNS: dig +short shop.example.com (or --resolve in the lab).
  2. L2/L3: which node owns the VIP (MetalLB speaker logs), and the route from your laptop.
  3. TCP: nc -vz 10.10.0.200 443.
  4. TLS: openssl s_client … -servername shop.example.com: the chain and the negotiated ALPN.
  5. L7 routing: curl -v, and the ingress/gateway access log line for your request.
  6. Service: kubectl get endpointslices for web, and the kube-proxy rules or cilium service list entry.
  7. Pod network: the web pod's IP, node, veth, and route (nsenter).
  8. Internal hop: web → api through CoreDNS and the NetworkPolicy that allows it.

Write it down as a one-page diagram with your evidence next to each arrow. That page is gold during the next incident.

Part 2: diagnose injected faults

Ask a colleague to inject these one at a time (or script them and run them in random order). For each, record symptom → first failing layer → proof → fix.

# Injected fault How to inject (lab)
1 DNS Change a CoreDNS stub domain or the app's upstream name to something wrong
2 MTU Set a pod interface's MTU above the overlay's capacity (e.g. force 1500 on a VXLAN cluster)
3 Policy Remove the NetworkPolicy allowing web → api
4 TLS Point the Ingress/Gateway at a Secret with a certificate for a different name
5 Load balancing Scale a gRPC backend and observe (or simulate) imbalance behind an L4 Service
What good diagnoses look like
  1. DNS: curl errors Could not resolve host, or web logs show lookup api… no such host. Proof: nslookup from the pod fails while pod-IP calls work. Fix: correct the name or Corefile.
  2. MTU: TCP connects, small responses work, large ones hang. Proof: ping -M do -s <size> fails above a threshold between pods on different nodes. Fix: set the CNI MTU correctly for the encapsulation.
  3. Policy: web → api times out (not refused). Proof: wget -T 3 from web to the api pod IP times out; flow logs (Hubble/Calico) show DROPPED. Fix: restore the allow rule, via Git.
  4. TLS: clients report a name mismatch. Proof: openssl s_client -servername shop.example.com shows a certificate with other SANs. Fix: reference the correct Secret, and let cert-manager manage it.
  5. Load balancing: new pods idle, one pod hot. Proof: per-pod request metrics or logs; one long-lived connection. Fix: L7/mesh balancing, headless + round_robin, or max connection age.

Part 3: write the runbook entry

Turn your notes into a short "network triage" runbook for your team:

  • The ladder in order, with one command per rung (this lesson's cheat sheet).
  • Refused vs timeout vs reset, and what each means.
  • Where to find flow logs, ingress access logs, CoreDNS logs and kube-proxy/CNI state.
  • Known cluster-specific facts: MTU, CIDRs, CNI mode, where TLS terminates, VIP ranges.

You've finished Networking Deep Dive

From Ethernet frames and ARP to HTTP/3 and gRPC, you can follow a request through every layer and find where it breaks, with evidence. Next: Kubernetes Storage & Data Protection, or Service Mesh to go further with L7 traffic.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.