Files, JSON & YAML · Prasanta Kumar Behera

Read, transform and write the data formats infrastructure runs on: text files with pathlib, JSON, YAML (safely) and CSV, and write config files atomically so a crash never leaves them half-written.

Everything is structured text

Configs are YAML or INI, APIs speak JSON, reports are CSV. Python turns each into lists and dictionaries you can work with, then back again.

JSON, YAML and CSV are different languages for writing down a shopping list: one uses curly brackets, one uses indentation, one uses commas. Python can read all three and turn them into the same thing in its head: a list of items with names and amounts. Then it can write them back in whichever language you like.

Files with pathlib

from pathlib import Path

log_dir = Path("/var/log")
for f in sorted(log_dir.glob("*.log")):
    size_mb = f.stat().st_size / 1_000_000
    print(f"{f.name:<30} {size_mb:8.1f} MB")

hosts = [line.strip() for line in Path("hosts.txt").read_text().splitlines()
         if line.strip() and not line.startswith("#")]

For very large files, iterate instead of reading everything:

with open("/var/log/nginx/access.log", encoding="utf-8", errors="replace") as fh:
    errors = sum(1 for line in fh if '" 5' in line)    # rough count of 5xx lines

JSON

import json, subprocess

raw = subprocess.run(["kubectl", "get", "pods", "-A", "-o", "json"],
                     capture_output=True, text=True, check=True).stdout
data = json.loads(raw)
not_running = [
    f"{p['metadata']['namespace']}/{p['metadata']['name']}"
    for p in data["items"]
    if p["status"].get("phase") != "Running"
]
print(json.dumps({"not_running": not_running}, indent=2))

(Running commands safely is lesson 03; the Kubernetes Python client is lesson 06.)

YAML: always safe_load

import yaml
from pathlib import Path

cfg = yaml.safe_load(Path("inventory.yaml").read_text())
for site in cfg["sites"]:
    print(site["name"], site.get("nodes", 3))

cfg["sites"].append({"name": "edge-043", "nodes": 3})
Path("inventory.new.yaml").write_text(yaml.safe_dump(cfg, sort_keys=False))

# inventory.yaml
sites:
  - name: edge-041
    nodes: 3
  - name: edge-042
    nodes: 1

Never yaml.load untrusted input

yaml.load() with the full loader can construct arbitrary Python objects. yaml.safe_load limits YAML to plain data. Use it everywhere. Note that PyYAML doesn't preserve comments when you rewrite a file; ruamel.yaml does, if you need round-trips that keep comments.

CSV reports

import csv

rows = [{"host": "web-01", "disk_pct": 91}, {"host": "db-01", "disk_pct": 83}]
with open("report.csv", "w", newline="") as fh:
    w = csv.DictWriter(fh, fieldnames=["host", "disk_pct"])
    w.writeheader()
    w.writerows(rows)

Writing config files atomically

A crash halfway through writing a config file can leave a service unable to start. Write to a temporary file in the same directory, then rename. The rename is atomic:

import os, tempfile
from pathlib import Path

def atomic_write(path: Path, content: str) -> None:
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}.")
    try:
        with os.fdopen(fd, "w") as fh:
            fh.write(content)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp, path)          # atomic on the same filesystem
    except BaseException:
        os.unlink(tmp)
        raise

Try it: inventory transformer

Write inventory.yaml with 3 sites (name, region, nodes).
Load it with yaml.safe_load and print sites per region.
Add a site programmatically and write the result with atomic_write as YAML.
Export the same data as sites.csv and as sites.json (with indent=2).
Run kubectl get nodes -o json on your kind cluster and print each node's name and kubelet version from the JSON.

Going deeper: data you can trust

Validate input structures instead of assuming keys exist: pydantic or jsonschema turn "KeyError at line 83" into "site 'edge-042' is missing 'region'".
Mind encodings (encoding="utf-8") and line endings when files come from Windows machines.
For huge JSON (large kubectl dumps), stream with ijson or filter at the source (-o jsonpath, label selectors) instead of loading gigabytes into memory.

Recap

pathlib.Path for files; iterate line by line for big files.
json.loads/dumps, yaml.safe_load/safe_dump, csv.DictReader/DictWriter.
Infrastructure data = nested lists and dicts (items, metadata.name…).
Write important files atomically (temp file + os.replace).

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.