A Project-Level Harness for AI Agents: Constraints in Hooks and Tests, Not Prompts
Also in Chinese: 中文版
The premise
An AI agent does not usually fail by crashing or refusing. It fails by saying “done” when it is not. The report is fluent and specific, and sometimes wrong in exactly the place you did not check. If that is the failure mode, then the controls cannot live in the instruction text. A prompt is a request to a system that has just demonstrated it will tell you what you want to hear. Controls have to sit where the agent’s confidence cannot reach: in hooks that run in the harness, and in tests that either pass or do not.
I built the work around that one sentence. The result was that over four months, agents did most of the typing in rebuilding a data-platform stack (a database fork, a compute-platform repo, a Helm chart repo) and it went live in production. I should be precise about what “most of the typing” means: 566+ commits across three repositories, and more than 80% of them carry a Claude co-author trailer. That is a measure of how much passed through an agent, not a measure of how much of the thinking it did. The design, the choices of what to build, and the judgment of whether a result was believable stayed with me.
Three mechanisms, and why each is where it is
An admission gate before edits (scope gate). For medium and large changes, an agent cannot write to protected paths until a confirmed scope document exists: goal, what is out of scope, acceptance criteria, and which paths may be touched. It is enforced by a pre-tool hook that blocks the write, not by asking nicely. The point is not bureaucracy. Writing “what is not in scope” before starting is the cheapest way to catch an agent about to solve an adjacent, more interesting problem.
I also have to report where this went wrong, because it is the useful part. The gate was originally on by default everywhere. In practice it was mostly being cleared to unlock writes, not to clarify anything: in one repository, two days produced 15 confirmed scopes, and 11 were never reviewed to a conclusion. A control that people route around by clicking through is worse than none, because it manufactures the look of review. I made the gate opt-in, with extra friction only on paths where a mistake is costly. Tighter was not better.
An audit trail that records intent. Every mutating command carries a one-line # INTENT: comment, what was observed, what was concluded, what is expected, which a hook captures into an append-only log alongside the git ref at that moment. There is also a rule that a mutating command is followed by a verification command, and a failed verification stops the work. The log answers “what changed” and the intent line answers “why did it think that was a good idea”, which is the question nobody can reconstruct afterward from the diff alone.
Deterministic end-to-end acceptance. The platform has 250+ end-to-end test cases. They are deterministic: same input, same verdict, no model in the loop deciding whether it passed. This is the mechanism that actually removes the failure mode. An agent can write “all green” in a summary all day; it cannot make a failing test pass by wording.
An example of the failure, caught
I asked an audit sub-agent to compare two files. It reported them identical. A plain diff showed otherwise: 96 lines on one side, 115 on the other. The sub-agent had not lied in any usual sense; it produced a confident, well-formed answer without doing the comparison the claim needed. If I had acted on the summary, one file would have been archived as a duplicate while carrying logic the other did not.
What made this useful rather than just alarming is what it changed. I stopped accepting a sub-agent’s “same”, “complete” or “verified” as a result, and started requiring the evidence artifact: the diff itself, the exit code, the test output. Where a conclusion matters, there is now a check I can run without asking the agent anything. That is the general form of the whole project: replace “the agent says” with “the artifact shows”.
What is public, and what is not
The harness itself is public: hooks for boundaries and logging, skills for scope and review, and a dashboard that shows which sessions are waiting and what each has finished. It is on GitHub at https://github.com/shaorui0/agent-harness. The repository shows the control mechanism. It does not show the platform work it was applied to, and the end-to-end suite and the commit history live in private company repositories. I would rather say that than let the public link imply the numbers can be checked from it.
What I would still not claim
- The commit counts and the co-author percentage are counts of commits, not of effective work. A trailer on a one-line fix and a trailer on a large change each count once.
- The harness reduces the confident-wrong failure; it does not remove it. Tests catch what tests cover. The sub-agent example was caught by a diff I happened to run, which is the uncomfortable part: some fraction of such claims were not checked.
- “Four months, live in production” describes the platform. The tests and hooks are why I trust the path to it, but I cannot isolate how much faster it was than doing the work without agents, and I do not make that claim.