Anatomy of an agent harness: hooks, loops, memory, RAG

Every component of an agentic system except one can be run twice and give the same answer. Hooks, loops, status files, the model ladder, retrieval, memory: ordinary code that fails loudly and can be tested on a Tuesday afternoon. In the middle sits a model that is non-deterministic by design, that will be retired on somebody else's schedule, and that praises its own work when you ask it to grade. An agent harness is everything I have built around that one fact.

What follows is the anatomy of mine, six parts in the order they arrived rather than the order that would look tidy. Each has its own post underneath this one, because each earned more explanation than a section can carry.

What is an agent harness?

It is the middleware between a person and a model, and it decides more of the outcome than most people assume. The clearest definition I have read comes from a July 2026 paper that set out to measure exactly that, Don't Blame the Large Language Model by Ben Sghaier, Li, Adams and Hassan:

agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops

That sentence is useful because it names the parts. System prompts, tools, context, and the loop. Everything I run sits in one of those four buckets or underneath them.

How much variation hides inside those four words is worth knowing before you start building. Benjamin Rombaut's April 2026 taxonomy, Inside the Scaffold, read the source of 13 open-source coding agent scaffolds at pinned commits and reported that "scaffold architectures resist discrete classification": tool counts ranged from 0 to 37, context compaction spanned seven distinct strategies, and 11 of the 13 composed several control primitives rather than settling on one. There is no reference implementation here. There is a design space, and you are standing in it whether or not you have noticed.

Why build a harness instead of renting a bot?

Because the harness is the part that stays yours while everything else moves under you. I worked through that choice in August when xAI shipped bots that get their own cloud machine and sign into your accounts the way you do, and the honest answer was that it depends on which pile the work falls in. Errands across four portals while your laptop is shut: rent. Anything that has to sound like you, follow your standards, or touch your data: own, because all the value is in files that a rented runtime cannot hold.

The same paper that gave me the definition also gave me the argument. Its authors surveyed five major open-source harnesses and found "extreme release velocities exceeding two releases per day and thousands of issues within months," alongside a habit they name plainly: practitioners report quality regressions after a harness update and "consistently attribute them to the underlying model rather than the harness itself." So they fixed the model and varied only the harness, across 35 sequential releases of one CLI measured against 50 stratified SWE-bench Verified tasks, and watched quality move anyway.

Read that next to what surveyed teams actually ship. Measuring Agents in Production (Pan et al., December 2025), 20 interviews and 86 surveyed practitioners across 26 domains, found production agents built with "simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation." None of that describes an autonomous swarm. It describes a short leash and somebody checking the work.

What can a hook do that an instruction cannot?

Stop something. That is the whole difference, and it took me a while to feel it properly. A rule written into a markdown file is read by a model that then has to choose to follow it, on turn forty, behind a hundred thousand tokens of other material. A hook is a subprocess with an exit code. Anthropic's hooks guide puts the value in one line: Claude Code runs hooks "at specific points in its lifecycle, which gives you deterministic control: certain actions always happen rather than relying on the LLM to choose to run them."

Two run on every session here. One reads my prompt before the model does, because my keyboard sometimes repeats keys and sometimes fires the space bar in bursts, and the dangerous version of that is the prompt that still reads fine. The other opens each session by counting the links I have texted myself in the last fortnight, which is the same mechanism used for the opposite purpose: putting a fact in front of an agent at the moment it matters.

The trap worth knowing before you install one is that the gate fails open in more ways than it fails closed. Exit 2 is the blocking exit code on the events that can block; any other nonzero code is a non-blocking error, and a hook whose path is mistyped exits 127 and the session runs exactly as though the guard were not there. The full lifecycle, the JSON contract, and the two guards I actually run are in the post on hooks as guardrails.

What turns a loop into something you can leave running?

A verify step performed by something other than the thing that did the work. Everything else in loop design is scheduling. Anthropic's harness design writeup from March 2026 named the reason: "when asked to evaluate work they've produced, agents tend to respond by confidently praising the work." Their fix was structural rather than motivational, a separate evaluator agent that clicked through the running application the way a user would. Their earlier long-running harness report found the same thing from the other side: verification happened well, but only "once explicitly prompted" to drive a browser and test as a human user would.

My own version of that lesson arrived in a duller form. A weekly job that reads Search Console tested its own success with [ -s "$OUT" ], which asks whether a file has bytes in it. The session's login had expired, so for two Mondays the file contained the words "Not logged in" and a desktop notification announced that the report was ready. The check now asserts the report's content and fails loudly, which is a two-line repair and a lesson I would rather have bought cheaper. A verifier that checks for output rather than for the right output is a smoke alarm wired to the light switch. Both loop shapes, the build loop on a five-minute clock and the business loop on a weekly one, are in agentic loops in production.

Where does a harness keep what it knows?

Outside the context window, in files that outlive the session. This is the component people skip, and it is the one that decides whether a long build compounds or restarts. Anthropic's context engineering post from September 2025 states the constraint that forces it: context "must be treated as a finite resource with diminishing marginal returns," because "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases." Their three answers for long-horizon work are compaction, structured note-taking, and sub-agents with clean context windows. I run all three, and the notes are the half that survives a crash.

In practice that is a progress file per long task, an append-only decision log where an old line is superseded rather than edited, and 122 memory notes for this project alone. On the operational side it is one small JSON per worker, rewritten by that worker and committed with the work it describes, so that a dashboard can tell alive from quiet without a message bus anywhere: a status file per bot covers the classifier, the staleness tiers, and the route I closed ten minutes after opening it.

One rule runs through all of it, learned the expensive way. Any claim a script can verify should be produced by that script. A hand-written index in one of my repos asserted, across three consecutive rewrites, that a function was dropping three fields it had been mapping correctly for weeks. Two agents took orders from that sentence. Nobody caught it, because prose does not fail. Hand-written notes are for judgment and intent; anything structural gets generated from the source and exits non-zero when it stops being true.

How does the harness decide which model runs a task?

Before generation, by what a wrong answer costs rather than by how hard the task feels. Difficulty is a seductive signal and a bad one: renaming a field across forty files feels enormous and reverts with one git command, while changing how a session cookie is set feels tiny and is a security incident if it goes wrong.

The ladder is seven standing agents rather than a decision made fresh each time, because a rule you have to remember is a rule you skip at eleven at night. Each is a short markdown file with its tier pinned in front matter: a cheap locator that returns paths and line numbers, three mid-tier workers for reading bulk, applying decided changes and running live round-trips, two top-tier agents for a hard problem on a clean context and for adversarial review of a diff, and one reserved tier above those for the cases where being wrong is expensive.

Underneath the ladder sits a different mechanism with a similar diagram, and conflating them cost me an outage. A cascade earns its keep on availability rather than on price, and it should fire on any per-model failure, including a 200 whose content came back empty. The ladder, the cascade, and the settings row that lets a retired model be pulled without a deploy are in the post on model routing. The bigger lever is not tier at all: when I tallied seven days of my own transcripts, 75% of my cache-read tokens came from requests carrying more than 250k of context and 53% of those tokens from above 400k, and the ten costliest sessions had spawned no subagents whatsoever. Context length is the bill. Model choice is the second lever.

What keeps retrieval from going stale?

Writing the knowledge in the same branch as the feature it describes. Retrieval quality is a real problem and it is not usually the problem. A stale chunk embeds as confidently as a fresh one, scores as high, and reads to the model with the same authority, so a missing document produces "I don't know" while an outdated one produces a wrong answer in the tone of a right one.

The standing order in this project's deploy checklist is that an admin feature is not finished until the migration that ships it has also written the chunks describing it and the embed step has run. It is one branch, not a feature plus documentation later. The knowledge base is 283 hand-curated chunks, and the wall between what a visitor's assistant knows and what a signed-in staff session knows is a SQL predicate rather than a line of prompt asking a model to be discreet. RAG that stays current has the write path, the incremental-update trick of leaving the embedding column NULL, and what a full rebuild actually costs.

When does a graph beat a similarity search?

When the answer has to be assembled from facts that were stored apart. Embeddings put a passage in a space where nearby means similar in meaning, which is a very good trick with one structural gap: two facts that need laying end to end sit wherever their wording puts them, and that is usually nowhere near each other.

My version is deliberately at the cheap end. Alongside the 283 chunks there are 227 edges, built by two rules that call no model at all: a chunk whose text contains another chunk's title, and two chunks sharing a tag that few other chunks use. Retrieval blends keyword and vector scores, keeps the top five, then walks one hop and appends at most two neighbours. It buys one behaviour, which is that a chunk worded nothing like the question can still arrive because something already retrieved points at it. The benchmarks say the gains are large on multi-hop questions and close to nothing on lookups, which makes this an accounting decision. Graph RAG against vector RAG has the numbers and the extraction bill.

What does a harness look like when it ships a client product?

Mostly like guards. The IDX site I built this summer for a Portland-area brokerage client runs on a regional MLS feed, under the client's own participant licence, into the client's own database, and the interesting engineering is almost entirely defensive.

A pass that wants to delete more than a fifth of the catalogue refuses and raises an alarm, because from inside the code a fat-fingered filter and a real mass delisting look identical. The photo queue guarantees one cover for every listing before any listing gets forty, because a backlog that takes a fortnight to drain is the site for a fortnight. A photograph is never cropped, because watermarks are burned into the pixels and a crop can amputate a legal notice. And listing rows never reach a language model and never enter a retrieval index: the site's assistant parses a visitor's question into a structured search and receives server-rendered cards, so the model sees the question and never the listings. The site was built to the RMLS Rules and Regulations and reviewed by RMLS before go-live.

Three posts came out of that build, and they are the closest thing I have to proof that the components above are not a hobby. Building an IDX site on a RESO Web API feed covers what the standard guarantees and what your board decides. The image pipeline that failed silently is a postmortem on a disk quota that made five layers report success while a visitor read "48 photos" over blank tiles. 28,000 listings on a map is the rendering side, and the honest note about which half of the basemap is live.

What I would build first

The verify step, before any of the rest of it. Not because it is the most interesting piece, but because every other component is worth less without it. A hook you never triggered on purpose might be failing open. A loop that grades itself is a machine for producing confident nonsense on a schedule. A knowledge base nobody has interrogated in staff mode is a set of claims, not facts. The verify step is what converts each of those from an intention into an observation, and it is cheap: a real round-trip, a status code, a screenshot somebody actually reads.

Second would be the memory files, because they are what makes the third session cheaper than the first. Hooks third, one rule at a time, each one triggered deliberately the day it is installed. Retrieval last, and smaller than you think.

The one I have not built is sitting in a checklist. A PostToolUse match on Write and Edit that greps whatever was just written for an em dash and hands it straight back. Every argument in this post says that rule belongs in code, and it has been living in a markdown file since July, which is either the last gap in the anatomy or a decent illustration of why the anatomy exists.