Prasanta Kumar Behera
AI-Assisted Infrastructure Engineering›12 · Building an ops agent
Learning Hub / AI-Assisted Engineering / AI-Assisted Infrastructure Engineering

Lesson 12 of 14 · Modules

Building an ops agent

Design an operations agent that automates runbooks safely: an LLM that gathers evidence with read-only tools, proposes a plan, waits for human approval before any change, executes through narrow audited actions, verifies the outcome, and is evaluated on realistic scenarios before anyone relies on it.

Advanced
Key wordsops agentrunbook automationtool callingplan-approve-execute-verifyhuman in the loopapproval gatesdry runevaluationaudit logconfident and wrong
Gather read-only tools 1 Plan diagnosis + evidence 2 Approve a human decides 3 Act narrow tool, dry run 4 Verify success criteria 5 not met → roll back, escalate, learn the agent does the legwork; humans own every change
Gather → plan → approve → act → verify.

What we're building

An ops agent handles routine operational tasks: triage a failing deployment, collect diagnostics for an alert, execute a well-known runbook step. The design goal: useful legwork, zero unsupervised risk.

The ops agent is a very fast junior assistant on the night shift. It can look at every screen and read every manual, then says: "I think the kitchen oven is overheating because of X; I'd like to turn it down to 180°. Shall I?" It waits for a yes, does exactly that one thing, and then checks the temperature really dropped.

Tools: read vs act

Read-only (always available) Actions (after approval, with limits)
Pod/deployment status, events Restart one deployment (rollout restart)
Logs (templated, redacted) Scale a deployment within min/max bounds
Metrics queries (fixed PromQL templates) Cordon a node (never drain more than one)
Runbook lookup Toggle a feature flag in a named set

Action tools validate inputs (namespace allow-list, replica bounds), support dry run, and log every call. No generic shell or kubectl passthrough.

A runbook as a tool plan

runbook: deployment-crashloop
steps:
  - gather: [ rollout_status, recent_events, container_last_state, logs_templates(last=15m) ]
  - diagnose: "Match evidence to known causes: bad config/env, failing dependency, OOM, bad image."
  - propose:
      options:
        - action: rollout_undo            # if a new revision started the crashloop
          requires_approval: true
        - action: restart_deployment      # if a dependency recovered
          requires_approval: true
  - verify: "All replicas Ready for 5 minutes and error rate back under SLO threshold."
  - on_failure: "Escalate to on-call with collected evidence; take no further action."

Scenario: the agent was confident and wrong

Scenario: the agent was confident and wrong

During a game day, the agent diagnosed a crashlooping service as "OOMKilled, increase memory limit" and proposed raising the limit. The container was actually exiting with code 1 because of a missing environment variable after a config change; the agent had pattern-matched on a stale OOM event from the previous day.

What design changes prevent harm?
  • Evidence requirements: every diagnosis must cite current evidence (container lastState with timestamp, exit code). "OOMKilled" requires reason: OOMKilled in the latest termination.
  • Time-bounded tools: read tools return only recent data by default (e.g. last 30 minutes), with timestamps.
  • Approval with evidence visible: the approver sees the cited evidence and can spot the mismatch.
  • Success criteria: had the change been approved, verification would fail (still crashlooping) and trigger automatic rollback and escalation.
  • Evaluation set: add this case to the agent's test scenarios.

Evaluate before trusting

  • Build a set of realistic scenarios (from past incidents, sanitised, and simulated ones from lesson 07).
  • Score: correct diagnosis, safe proposal, asked for help when uncertain, no unsafe actions attempted.
  • Start in shadow mode: the agent proposes during real incidents; humans act and compare.
  • Expand scope only for runbooks where it's consistently right.

Try it: a minimal approved-action agent

  1. Reuse your read-only MCP server (lesson 05) and add one action tool, restart_deployment(namespace, name), that only works for namespaces in an allow-list and supports dry run.
  2. Configure your client so the action requires explicit approval every time (tool permission prompts, or a custom approval step).
  3. Break a deployment in several ways (bad image, missing env var, OOM) and ask the agent to diagnose and propose.
  4. Approve only when the evidence supports the proposal; record right/wrong diagnoses.
  5. Write a verification check for each scenario and add failing cases to an evaluation list.

Going deeper: production-grade agents

  • Keep write credentials in a separate service that only executes approved, signed action requests.
  • Log prompts, tool calls, approvals and outcomes immutably for audit (lesson 14).
  • Watch for prompt injection through logs, tickets and alerts: the agent's inputs are untrusted.
  • Integrate with incident tooling (chat, tickets) so approvals and evidence live where responders already work.

Recap

  • Loop: gather (read-only) → diagnose/propose → human approval → narrow action (dry run) → verify.
  • Separate read and write tools and credentials; actions are allow-listed and bounded.
  • Prevent "confident and wrong" with evidence requirements, time-bounded data, visible evidence at approval, and success criteria with rollback.
  • Evaluate on realistic scenarios and start in shadow mode.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.