Linux — Level by Level›09 · Networking from the host

Lesson 09 of 19 · Level 2 — Intermediate: troubleshooting

Networking from the host

Debug network problems from the host: interfaces and routes with ip, listening sockets with ss, name resolution with dig, HTTP with curl, packets with tcpdump, and firewalls, walking the layers in order.

Practitioner
Key wordsip addrip routessDNSdigcurltcpdumpfirewallnftablesMTU
Application curl api.example.com 1 Resolver /etc/resolv.conf → DNS 2 Routing table ip route: which interface? 3 Firewall nftables / iptables 4 Interface ip link, MTU, IP 5 Network gateway → internet 6 troubleshoot in the same order: name → route → firewall → link → path
What happens when a program opens a connection, and the order to troubleshoot it.

Walk the layers, bottom up

Network problems feel mysterious until you check them in order. Stop at the first layer that fails:

  1. Link: is the interface up and has an address?
  2. Route: does the host know where to send packets?
  3. Name: does the name resolve, to the right address?
  4. Port: is something listening, and does a firewall allow it?
  5. Application: does the service answer correctly (HTTP, TLS)?

Sending a letter: is your postbox working (link)? Does the postman know which road to take (route)? Is the address in the phone book right (DNS)? Is someone at home to open the door (port)? And do they understand your language (application)? If the letter comes back, you check these in order, instead of blaming the post office.

1–2. Interfaces and routes

$ ip -br addr
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             10.10.0.21/24 fe80::5054:ff:fe12:3456/64
$ ip route
default via 10.10.0.1 dev eth0 proto dhcp src 10.10.0.21 metric 100
10.10.0.0/24 dev eth0 proto kernel scope link src 10.10.0.21
$ ip route get 10.20.5.9
10.20.5.9 via 10.10.0.1 dev eth0 src 10.10.0.21 uid 1000

No default via line means the host can only reach its local subnet.

3. DNS

$ dig +short api.example.com
203.0.113.20
$ dig api.example.com @1.1.1.1 +short      # ask a specific resolver, to compare
$ resolvectl status | grep -A2 'DNS Servers'
$ cat /etc/resolv.conf

A name that resolves differently from different resolvers, or to an old IP, explains many "works for me, not for you" problems.

4. Ports: listening, reachable, allowed

On the server: is anything listening, and on which address?

$ sudo ss -ltnp
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       511           0.0.0.0:80          0.0.0.0:*      users:(("nginx",pid=1487,fd=6))
LISTEN  0       244         127.0.0.1:5432        0.0.0.0:*      users:(("postgres",pid=990,fd=7))

PostgreSQL listens on 127.0.0.1 only, so it's unreachable from other hosts no matter what the firewall says.

From the client: can I connect?

$ nc -vz 10.10.0.30 5432
nc: connect to 10.10.0.30 port 5432 (tcp) failed: Connection refused
Result Usually means
Connected Network path and port are fine; look at the app or TLS
Connection refused Host reached; nothing listening there, or a firewall REJECT
Timeout Packets dropped: firewall DROP, security group, routing, host down
No route to host Routing problem, or an ICMP rejection from a firewall

Firewalls: modern distributions use nftables (sudo nft list ruleset); ufw status (Ubuntu) and firewall-cmd --list-all (RHEL family) are friendlier front-ends. In clouds, also check security groups: they're firewalls outside the host.

5. Application: curl tells you everything

$ curl -v https://api.example.com/health
*   Trying 203.0.113.20:443...
* Connected to api.example.com (203.0.113.20) port 443
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* Server certificate:
*  subject: CN=api.example.com
*  expire date: Dec 20 23:59:59 2026 GMT
> GET /health HTTP/1.1
< HTTP/1.1 200 OK

-v shows DNS, the TCP connect, TLS details (certificate subject and expiry) and headers. -k skips certificate checks (testing only); --resolve api.example.com:443:10.0.5.20 tests a specific backend with the right hostname.

When in doubt: look at the packets

$ sudo tcpdump -ni any host 10.10.0.30 and port 5432
10:21:05.114 eth0 Out IP 10.10.0.21.51234 > 10.10.0.30.5432: Flags [S], seq 1234, length 0
10:21:05.114 eth0 In  IP 10.10.0.30.5432 > 10.10.0.21.51234: Flags [R.], seq 0, ack 1235, length 0

[S] is our SYN; [R.] is a reset (refused). Only SYNs repeating with no answer = packets dropped somewhere. Save captures with -w file.pcap to open in Wireshark.

Try it: one fault per layer

On a VM with nginx installed:

  1. Port: stop nginx, then curl -v http://localhost. Read "Connection refused". Start it again.
  2. Listen address: change nginx to listen 127.0.0.1:80;, reload, and try from another machine (or the host's real IP). What does ss -ltnp show?
  3. Firewall: sudo ufw enable (allow ssh first: sudo ufw allow OpenSSH!), then sudo ufw deny 80 and test from another machine: refused or timeout? Remove the rule.
  4. DNS: add a wrong entry to /etc/hosts (10.9.9.9 myapp.local) and curl http://myapp.local. Which layer failed?
  5. Capture the three-way handshake of a working curl http://localhost with sudo tcpdump -ni lo port 80.

Going deeper: host networking for Kubernetes

  • Every pod has its own network namespace. sudo ip netns (or nsenter -t <pid> -n ip addr) enters one, so you can run these same tools from inside a pod's view.
  • MTU mismatches (overlays, VPNs) cause "small requests work, large ones hang". Test with ping -M do -s 1472 <ip> (1472 + 28 bytes of headers = 1500).
  • conntrack -L (conntrack-tools) shows NAT and connection-tracking state; a full conntrack table drops new connections under load.
  • All of this, layer by layer, is the Networking Deep Dive course.

Recap

  • Debug bottom up: link → route → DNS → port and firewall → application.
  • ip -br addr, ip route, dig, ss -ltnp, nc -vz, curl -v.
  • Refused = reached but nothing listening; timeout = dropped somewhere.
  • Check the listen address, host firewall and cloud security groups; use tcpdump -n when the tools disagree.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.