Text processing · Prasanta Kumar Behera

Turn raw text and JSON into answers: extended grep, sed for search-and-replace, awk for columns and totals, and jq for JSON from APIs and kubectl, with one-liners you'll reuse for years.

The right tool for each shape of text

Shape	Tool
Lines matching a pattern	`grep` (`-E` for extended regex)
Search and replace, line ranges	`sed`
Columns, filters, totals	`awk`
Fixed-delimiter fields	`cut`
Counting and ranking	`sort`, `uniq -c`
JSON (APIs, kubectl, cloud CLIs)	`jq`

Think of text as a big pile of LEGO. grep picks out only the red bricks. sed repaints some bricks. awk sorts bricks into columns and counts them. jq is for the special LEGO sets that come with an instruction booklet (JSON): it reads the booklet and hands you exactly the pieces you asked for.

grep with extended regex

$ grep -E "error|fail|timeout" app.log                # any of three words
$ grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" app.log | sort -u   # every IPv4-looking string
$ grep -E "^[^#]" /etc/ssh/sshd_config                 # lines not starting with #

sed: stream editor

$ sed -n '100,120p' big.log                            # print a range of lines
$ sed 's/http:/https:/g' links.txt                     # replace (output only)
$ sed -i.bak 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
$ sed '/^\s*#/d; /^\s*$/d' nginx.conf                   # strip comments and blanks

s/pattern/replacement/g: the g replaces every match on the line, not just the first. Always test without -i first, and use -i.bak so you can undo.

awk: columns, filters and sums

awk splits each line into fields $1, $2, … ($0 is the whole line; NF is the number of fields).

$ df -h | awk 'NR > 1 && $5+0 > 80 {print $6 " is " $5 " full"}'
/var is 91% full
$ awk -F: '$3 >= 1000 && $7 !~ /nologin/ {print $1}' /etc/passwd
asha
$ awk '{bytes += $10} END {printf "%.1f MB\n", bytes/1024/1024}' access.log

NR > 1 skips the header line.
$5+0 turns "91%" into the number 91.
END { … } runs after the last line: perfect for totals.

Status codes per minute from an nginx access log (combined format: $4 = [27/Sep/2026:10:15:02, $9 = status):

$ awk '{split($4, t, ":"); print t[2]":"t[3], $9}' access.log | sort | uniq -c | tail -3
    412 10:15 200
      9 10:15 502
    398 10:16 200

jq: JSON on the command line

$ kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase != "Running") | "\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"'
shop/migrate-7x2kq Succeeded
default/api-6d9f-2mzq Pending

$ curl -s https://api.github.com/repos/kubernetes/kubernetes | jq '{stars: .stargazers_count, forks: .forks_count}'
$ jq -r '.[] | [.name, .size] | @csv' files.json > files.csv

jq piece	Meaning
`.field`, `.a.b`	Select a field
`.items[]`	Iterate an array
`select(cond)`	Keep matching items
`\(.x)` inside a string	Interpolate
`-r`	Raw output (no quotes)
`length`, `keys`, `sort_by(.x)`, `group_by(.x)`	Built-in functions

kubectl has its own JSON tools too

kubectl get pods -o jsonpath='{.items[*].metadata.name}' and -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName often avoid needing jq at all.

Try it: answer questions from real text

From /etc/passwd, list users whose shell is /bin/bash (awk with -F:).
From df -h, print filesystems more than 50% full.
Copy /etc/ssh/sshd_config to /tmp, and with one sed -i.bak command set PasswordAuthentication no whether the line is commented out or not. Diff it against the .bak file.
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name) \(.status.nodeInfo.kubeletVersion)"' on your kind cluster.
Download any JSON API (e.g. curl -s https://api.github.com/repos/kubernetes/kubernetes/releases?per_page=5) and print each release's tag_name and published_at.

Going deeper: when to stop one-lining

A pipeline you'll run once: one-liner. A pipeline you'll run weekly: a script with comments. Logic with state, retries or error handling: probably Python (see Python for Infrastructure).
GNU vs BSD tools differ (sed -i on macOS needs -i ''). Scripts meant for Linux servers should assume GNU; note it in the script.
Regex flavours differ too: grep -E/awk use extended regex; grep -P (Perl-compatible) adds \d, look-arounds and more.
For big logs, filter as early as possible (grep first, then awk), and use LC_ALL=C for faster byte-wise sorting.

Recap

grep -E for patterns, sed for replace/ranges (-i.bak in place), awk for columns, filters and totals.
sort | uniq -c | sort -rn to count and rank.
jq for JSON: .field, .[], select(), -r, @csv.
Test edits without -i first; keep backups.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.