An SRE Oncall Triage Harness — Autonomous Investigation, Human-Owned Mutation
Also in Chinese: 中文版
Why
In an oncall investigation the main agent’s real job is judgment: is this a false alarm, where is the root cause, what to query next, do we escalate. Judgment needs a clear context window. But a single raw range query can come back at 30K tokens, and a slice of logs is worse. If the main agent pulls that data itself, then by the time it reaches the phase where the hard calls happen, its context is drowned in raw numbers and its judgment is spent.
That is the first-principles constraint the whole system is built on: context is scarce RAM, not an infinite disk. Whoever occupies it should pay for the occupancy. One iron rule follows directly — the main agent keeps only navigation, judgment, and command generation; all raw-data fetching is pushed down to a subagent that returns a structured summary of no more than 500 tokens. This is not an optimization. It is an architectural constraint.
I built this harness to hold that line under real oncall load, and the design below is a chain of consequences from that one constraint.
The design: four control primitives, not role-play
Most “multi-agent” frameworks do role-play: a PM agent, an engineer agent, a QA agent. That is skeuomorphism — copying a human org chart onto agents, intuitive but without engineering meaning. What actually carries weight is four control primitives:
Three design decisions do most of the work.
Subagent isolation. The main agent inherits the strongest model and only judges; raw VictoriaMetrics/Loki queries and Slack reads are dispatched to a subagent that returns at most 500 tokens — a max value with its timestamp, a step-jump flag, a baseline ratio. This is the capacity-bound and attention-bound case for Fork made concrete: a 30K-token query is isolated into a fresh window so the main context never sees it. The anti-pattern is the main agent running the query itself, pulling back 200 data points, and arriving at the decisive phase already token-exhausted.
Phase lock. Karpathy described a classic agent failure: see a deploy record, hallucinate a root cause — swept off by “what changed recently” before the symptom is even understood. I designed that failure mode out with a state machine. A phase: field at the top of the plan file physically restricts what the main agent can read: in Phase A it cannot read deploy history at all, so it cannot confabulate one. The gate to the next phase is an explicit precondition, not a suggestion.
The red line. Models hallucinate. That is a premise, not a defect, so system safety can never rest on “the model will comply.” Judgment — false alarm or not, where the root cause is — goes to the model. Every irreversible operation (delete, scale, drain, IAM change) goes to a deterministic shell hook that hard-blocks. No amount of hallucination gets past an exit 2. This turns safety from “hope the model behaves” into “it’s fine if the model misbehaves.”
One transferable intuition: this is a Kubernetes control plane
For an SRE none of this is new — it is the Kubernetes control-plane pattern moved inside the agent. Spec is the desired-state manifest; Loop is the controller’s reconcile loop; Hook is the admission webhook; Fork is pod-level isolation plus an independent auditor. The proposition underneath: we are not replacing SREs with AI. We are using decades of reliability engineering to constrain and operate a non-deterministic reasoning core.
Safety that actually lands
A single mutating command has to clear four gates before it reaches production; any one of them stops it:
Gate 1 is the skill layer: absent approval, the command just sits in the report. Gate 2 is a static allow/deny list — fastest, coarsest. Gate 3 is the deterministic executor: six shell hooks that understand cluster-alias-to-environment grading and fail closed — an unclassified alias is treated as production. This is the layer the model cannot route around, and it is where the red line is actually enforced. Gate 4 is mandatory post-verification. Two of the hooks even nest a second model call (claude -p) to review the current model’s plan at the moment of approval — AI reviewing AI, the red-team hook mounted exactly at Spec-freeze time.
Why it gets better with use
The compounding is not “save the report.” It is a delta judgment: against 130+ existing knowledge files, what did this investigation actually learn that is new? Matches update an existing file; a new root-cause path creates a new case; a new signal updates the routing table. Three anti-rot rules keep it clean: record only the delta, promote AI-written knowledge from draft to stable only after human review, and keep every entry traceable via derived_from back to the triage report. The knowledge base today spans 34+ cases, 21+ runbooks, and 15+ cards, and the next alert’s fast path retrieves straight into it.
And every real oncall is an eval run. verify.py gates single-investigation quality with an exit code — required sections present, evidence chain per conclusion, no over-assertion in the Slack response — while slo.py tracks pass-rate trends across investigations and exposes decay in the skill itself. The system is therefore not a static tool but a closed loop that improves as it is used.
Takeaways
- The binding constraint of an agent system is the context window, not the model. Treat context as RAM and the isolation architecture designs itself: judgment on the main agent, raw data on subagents behind a 500-token contract.
- Safety belongs in deterministic code, not in the prompt. Let the model judge; let a shell hook that returns
exit 2own every irreversible action. Then hallucination stops being a safety risk. - This is the concrete evidence for a larger thesis I hold elsewhere on this site: in the agent era, the SRE’s job is to be the reliable outer shell around a non-deterministic core.