SRE in the Age of AI Agents
Also in Chinese: 中文版
Strip SRE down to first principles and it is one job: operating the loop between what a system should be doing and what it is actually doing. Detect the gap, explain it, close it, keep it closed. AI agents have not changed that definition at all. What they have changed is who performs which part of the loop. I say this not as a forecast but as a description of my working week: agents run my production forensics, an agent harness runs my oncall triage, and the evidence behind this resume was itself distilled by an agent pipeline. This page is what the shift looks like from inside it.
What has changed
The execution layer of oncall has moved to agents. When a compaction score on a distributed storage system pinned at ~4,500 and refused to move, I did not investigate serially. I fanned out three read-only agents in parallel: one enumerating logs, one walking the tablet-metadata endpoints, one inspecting the catalog recycle bin. All three converged on the same orphaned tablet from an already-dropped table, structurally excluded from scheduling. One chain of reasoning can be led astray; three independent evidence chains pointing at the same object is a verdict. That fan-out-and-converge pattern is now my default forensics posture, not an experiment.
The same shift is institutionalized in my oncall triage harness. An agent picks up an alert and investigates autonomously, but inside a structure I designed: stage-locked phases it cannot skip, and conclusions that must cite evidence, a query result, a log line, a file and line number. Mutations are never auto-executed; every state change is proposed, and a human approves each one. The agent does the legwork that used to consume the first thirty minutes of every page.
Knowledge changed shape. A runbook written for humans is narrative. What compounds for an agent is the discriminator: a cheap check that splits the root-cause candidate set in half. Recording the same dead end ten times as a story yields nothing; recording it once as “if A is normal, prune this entire branch” turns a past investigation into future routing. My triage knowledge base is organized around this unit, decision trees ordered by which checks are cheap and which are expensive, because answers rot as infrastructure changes but probe order barely does.
The human’s position moved, and so did the leverage. I spend my time designing harnesses and adjudicating their output, not executing steps. The result is reach: the evidence base of this resume was produced by a re-runnable mining pipeline that distilled 82 working sessions into structured highlights with provenance, then into a dossier. One person plus a harness now owns a surface that used to need several.
What still needs an engineer
Start from one distinction. A Kubernetes controller is deterministic, so it needs no guardrail. An agent is a probabilistic controller, so a policy layer is a production prerequisite, not an option. Most of the engineering that remains human falls out of that fact.
Intent and constraint definition. Someone has to write the Spec, the desired end state with machine-checkable acceptance criteria, and place the Hooks: audit hooks that record evidence, deny hooks that hard-block irreversible actions, human-in-the-loop hooks that pause for approval. This is admission control for decisions, and it cannot be delegated to the thing being admitted. Constraints are the source of safety; the model is the source of capability. Confusing the two is how agents end up helpful and dangerous at once.
Mutation sovereignty. My harness enforces a strict read/write asymmetry: agents are excellent at reading and diagnosing, and structurally dangerous at writing. All the compounding value lives on the diagnostic side, so that side runs wide open. The write side is proposal-only, because blast radius is a business judgment and because agents fail silently: a hallucinated conclusion returns the equivalent of HTTP 200. You cannot budget what you cannot detect, and automating mutation on top of undetected error is not speed, it is laundering error with speed.
Eval as the measurement system. An agent without eval is not operable; you are shipping vibes. Per-step reliability compounds badly, a 95% step success rate is roughly 36% over twenty steps, so the SRE toolkit transfers with modification: task completion rate and convergence time as the first SLIs, an eval dataset with expected outcomes run on every change, and human review treated as a way to spend error budget on the irreversible tier rather than a default everywhere. A reviewer who approves two hundred times a day approves nothing. When the budget burns, autonomy should contract automatically.
Platform primitives. Agent systems need exactly what production systems need: a reconcile loop that compares state against Spec, observability over decisions and not just outputs, and explicit convergence criteria. My working summary is three imperatives: constrain it, see it, make it converge.
Final judgment. The rule in my harness, agent emits evidence and the human decides policy, is the same principle I argued in database-engine work, where the engine emits estimates and the caller decides routing policy. Mechanism can be shared, delegated, even upstreamed. Policy encodes your risk appetite and your accountability, and it stays with the engineer whose name is on the pager. Two lines of my career, storage engines and agent operations, converge on exactly this sentence.
That is the internal logic of the path from SRE to AI infrastructure. Not a career change: the same loop, desired state against actual state, closed safely, in its next form.