HTTP & REST APIs · Prasanta Kumar Behera

Talk to any REST API reliably with requests: sessions, timeouts, status codes and errors, token authentication, automatic retries with backoff, and walking through paginated results.

APIs are how infrastructure talks

Clouds, Git hosting, monitoring, ticketing, DNS providers, internal platform services: almost everything has a REST API that speaks JSON over HTTP. The requests library makes calling them pleasant; a few habits make it reliable.

An API is a restaurant's order window. You pass a slip (the request), you get a tray back (the response) with a number on it: 200 means "here's your food", 404 "we don't have that dish", 429 "slow down, the kitchen is busy", 500 "the kitchen had an accident, try again later". Good customers read the number before eating.

A first call

import requests

r = requests.get(
    "https://api.github.com/repos/kubernetes/kubernetes/releases",
    params={"per_page": 3},
    timeout=10,                     # ALWAYS set a timeout
)
r.raise_for_status()                # turn 4xx/5xx into an exception
for rel in r.json():
    print(rel["tag_name"], rel["published_at"])

Sessions, authentication and JSON bodies

A Session reuses connections (faster) and carries headers for every call:

import os, requests

s = requests.Session()
s.headers.update({
    "Authorization": f"Bearer {os.environ['API_TOKEN']}",   # from the environment, never hard-coded
    "Accept": "application/json",
})

r = s.post("https://netbox.example.com/api/dcim/devices/",
           json={"name": "edge-042-n1", "site": 12, "role": 3, "device_type": 7},
           timeout=15)
if r.status_code == 400:
    raise SystemExit(f"rejected: {r.json()}")     # validation errors explain what's wrong
r.raise_for_status()
print("created id", r.json()["id"])

(The NetBox call is illustrative. Every API documents its own fields.)

Retries with backoff

Networks blip and servers restart. Let a transport adapter retry idempotent requests on 429 and 5xx, with exponential backoff:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1,                                # waits grow exponentially between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD", "PUT", "DELETE"],
    respect_retry_after_header=True,
)
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry))

Don't blindly retry POST

A POST that timed out may have succeeded on the server. Retrying could create duplicates. Only retry non-idempotent calls if the API supports an idempotency key.

Pagination

APIs return large collections in pages. Two common styles: a next link, or a cursor/token:

def get_all(session: requests.Session, url: str) -> list[dict]:
    """Follow 'next' links (NetBox / Django REST style) until done."""
    items = []
    while url:
        r = session.get(url, timeout=15)
        r.raise_for_status()
        page = r.json()
        items.extend(page["results"])
        url = page.get("next")
    return items

GitHub uses the HTTP Link header instead: requests parses it into r.links.get("next", {}).get("url").

Status codes you'll handle

Code	Meaning	What to do
200/201/204	OK / created / no content	Continue
400	Bad request	Show the error body; fix the input
401 / 403	Not authenticated / not allowed	Check the token and its permissions
404	Not found	Maybe fine ("ensure absent"), maybe a wrong URL
409	Conflict	Already exists or changed concurrently: fetch and reconcile
429	Rate limited	Back off; honour `Retry-After`
5xx	Server error	Retry idempotent calls with backoff; alert if persistent

Try it: GitHub release watcher

Using a Session with retries, fetch the latest 5 releases of kubernetes/kubernetes and print tag and date.
Add a --repo argument so it works for any repository (e.g. argoproj/argo-cd).
Follow pagination via r.links to count all releases of a repository.
Handle a missing repository (404) with a friendly message and exit code 1.
Unauthenticated GitHub API calls have a low rate limit: print r.headers.get("X-RateLimit-Remaining") after each call.

Going deeper: well-behaved API clients

Put API access behind a small client class (class NetboxClient) with methods like get_devices(). The rest of your code never builds URLs, and tests can mock one class.
Log request IDs returned by APIs (X-Request-Id). Support teams can trace your exact call.
For heavy concurrency, httpx offers async support; for most ops tools, requests plus a thread pool is enough.
Store tokens in a secret manager or the environment, scope them minimally, and rotate them.

Recap

requests + timeout= + raise_for_status() + .json().
Sessions for connection reuse and auth headers; tokens from the environment.
Retries with backoff for 429/5xx on idempotent calls; be careful with POST.
Follow pagination until the API says you're done.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.