Agentic loops in production: generate, deploy, verify, repeat

In mid-July I wrote that the interesting use of an AI loop is not building the software but running the business. Seven weeks on I can describe what agentic loops actually look like in production, because I have been running two kinds all summer: one that generates and deploys on a clock measured in minutes, and one that wakes up every Monday morning, reads a number off Search Console, and goes back to sleep. The first kind is well documented. The second is where I learned something, including that one of my own guardrails had been passing a failed cycle for two weeks.

What is an agentic loop in production?

A loop is three things: a goal, a number the agent can fetch without a human interpreting it, and permission to take one reversible step per cycle until the number says stop. That definition has not changed since the post I wrote about running a business on loops. What running them taught me is that there are two shapes, and treating them as one is how people get hurt.

The build loop generates, deploys, verifies against the running thing, fixes, and repeats. Its scoreboard is pass or fail, its clock is minutes, and its characteristic failure is an agent that reports success without ever loading the page. The business loop measures, proposes one step, waits at a gate, applies, verifies on the next cycle, and sleeps. Its scoreboard moves slowly, its clock is a week or a month, and its characteristic failure is more expensive: a change that quietly spends money or gets published in somebody else's name.

Same eight stations. Only the sleep changes.

What is the build loop allowed to call done?

Done means observed working in production, and nothing softer. The standing order in this project's own instructions file has said so for months, in these words:

Agentic loops, not one-shots, generate → deploy → verify live (curl + real round-trips + screenshots) → fix → repeat until masterful. No change is "done" until observed working in production.

Anthropic's engineering team arrived at the same rule from a very different direction. In Effective harnesses for long-running agents (26 November 2025), describing an agent building a whole application across many context windows, the report was that Claude "mostly did well at verifying features end-to-end once explicitly prompted to use browser automation tools and do all testing as a human user would." The harness told it to "only mark features as 'passing' after careful testing" and to "end the session by writing a git commit and progress update."

Every clause there is load-bearing. Explicitly prompted means verification does not happen on its own. As a human user would means a passing unit test is not the round-trip. And the commit is not bookkeeping, it is the revert path: a loop that cannot roll back one step cannot safely take the next one.

On my machine that means Playwright driving WebKit, because WebKit is the engine behind iPhone Safari. A verification pass opens a real 390 by 844 viewport, clicks the actual element rather than a selector that looks close enough, asserts the URL afterwards, and screenshots at twice device scale. Then the image gets read back before anyone claims the visual fix worked. A screenshot nobody looks at is not evidence.

Why can't the agent that wrote the code judge it?

Because it grades itself generously, and that is now measured rather than suspected. Anthropic's follow-up, Harness design for long-running application development (24 March 2026), puts it plainly: "when asked to evaluate work they've produced, agents tend to respond by confidently praising the work." Their answer was structural rather than motivational.

Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue.

In their three-agent harness the evaluator is a different agent from the generator, and it "used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states."

My version is smaller: an adversarial pass over the actual diff, with access to the real data, before anything ships. In one week in August that pass caught a count rendering as zero across an entire section of a client site, a bare comma being parsed as a number, and a rewrite that ate the year out of the phrase "As of April 2026". None of the three were visible from the change description. All three would have shipped if the author had been the reviewer.

How do you pick the number a business loop steps on?

Put the candidate metric through three questions before building anything. Can an agent fetch it unattended? Is it the same question in the same units every cycle? Is "better" unambiguous? Three yeses and you can build a loop that runs for a year. One no and you do not have a loop, you have a report that needs a person every week.

Mine is Search Console, on one cron line, Monday morning:

37 8 * * 1  /Users/eric/Sites/frida/tools/seo-weekly.sh

The script is two stages on purpose, and the order is the whole design. Stage one is deterministic: a PHP command pulls a weekly snapshot for each property and appends one JSON line per property to an append-only log. No model sits in that path. Stage two hands the last two snapshots per property to a headless Claude session, which writes playbook/seo-weekly/<date>.md: week-over-week deltas, named movers, two or three specific suggestions per site. The comment at the top of the file states the rule the whole thing runs under: monitoring only, this loop never changes a site.

That is a deliberate decision about the gate, not a limitation. Search rank is exactly the sort of scoreboard an agent could chase unattended, and the steps it would take (retitling pages, adding FAQ blocks, rewriting service copy) are outward-facing and land in a client's name. So the loop reads and proposes, and a person applies. The step menu is not automated, and I am in no hurry to automate it.

Four weekly cycles in, the log already argues better than my memory of the same period would. On my own site, the week of 30 July to 6 August drew 19 impressions and zero clicks at an average position of 36.3. The week of 22 to 29 August: 195 impressions, three clicks, average position 24.4. Small numbers, and I am not going to dress them up. They are the same question asked the same way four times, which is worth more than a bigger number I cannot compare to anything.

Watching two numbers instead of one also heads off a familiar mistake. A site can post a worse average position in a week when nothing went wrong, because it started appearing for many more queries at deeper positions. The mean slid, the reach grew. Position alone reads that as a regression, and a loop with auto-revert wired to position alone would undo the change that was working.

The guardrail that missed, and what it cost to find

Here is the part I did not know until I sat down to write this post. On 24 August, and again on 31 August, stage two wrote a file whose entire contents were:

Not logged in · Please run /login

The credential for the headless session had expired. Stage one never noticed and never needed to, which is exactly why the deterministic half goes first: the snapshots kept accruing on schedule, so the two lost weeks cost me analysis, not data. The success check, though, was this:

if [ -s "$OUT" ]; then
  # ...notify: "Weekly SEO report ready"
else
  # ...notify: analysis failed
fi

-s asks one question: does the file have bytes in it? A login prompt has bytes. So both weeks fired a desktop notification announcing a report was ready, using the login message as its headline, and the failure branch never ran once. The repair, already in the file: a report has to be at least 150 words, it has to contain the word clicks, and it must not contain the login stub. Anything else gets renamed to .failed.md and exits non-zero, so a failed cycle cannot sit in the reports folder looking like a report. The lesson is not small. A verifier that checks for output rather than for the right output is not a verifier. It is a smoke alarm wired to the light switch.

The research is starting to make this point in a more organized way. In Towards a Science of AI Agent Reliability (arXiv, 18 February 2026, revised June), Rabanser, Kapoor, Narayanan and colleagues argue that "compressing agent behavior into a single success metric obscures critical operational flaws," and propose twelve metrics that ask instead whether agents "behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity." Having evaluated 15 models on two benchmarks, they report that "recent capability gains have only yielded small improvements in reliability." A better model would not have caught my check. A better check would have.

Which decisions stay human?

Anything irreversible or outward-facing. Spending money, publishing in someone's name, emailing a real person, deleting, touching production data: the loop proposes, a person approves. The gate that holds is the one written in code rather than in policy. On my machine a development environment physically cannot send mail to a real address, because the reroute is decided by the environment rather than by a setting somebody could flip in a hurry. That mechanism gets its own treatment in the post on hooks as guardrails; the short version is that a rule the model can talk itself out of was never a rule.

This is also what teams shipping agents actually do, which I found reassuring. Measuring Agents in Production (arXiv, 2 December 2025) interviewed 20 teams in depth and surveyed 86 practitioners across 26 domains, and reports that production agents are built with "simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation." Their headline challenge is the one my Monday morning found: "Reliability (consistent correct behavior over time) remains the top development challenge." LangChain's State of Agent Engineering, surveying 1,340 practitioners in late 2025 and published 12 June 2026, has 57.3% of respondents running agents in production, with quality rather than cost named as the biggest barrier.

How do you schedule a loop that outlives the session?

Four options, and they are not interchangeable. Claude Code's /loop is session-scoped, and the documentation is direct about its ceiling: "Recurring tasks automatically expire 7 days after creation. The task fires one final time, then deletes itself. This bounds how long a forgotten loop can run." Cloud routines run without your machine, but from a fresh clone with no access to local files. Desktop scheduled tasks run locally and survive restarts. Plain cron is still plain cron, and still the right answer more often than people expect.

The first three columns follow Claude Code's own comparison table; the expiry row is drawn from the seven-day-expiry section. The plain cron column is mine. One documented detail is worth knowing before you build on timing: the scheduler adds deterministic jitter, so recurring tasks "fire up to 30 minutes after the scheduled time," so leave 30 minutes of slack before anything downstream. Picking an odd minute only helps one-shot tasks.

My weekly loop needs a local checkout, local credentials, and a local database, so it lives in crontab. Inside a build session I use the self-paced form: give /loop a prompt and no interval and it picks its own delay each iteration, between one minute and one hour, based on what it just observed, and it can end itself when the work is provably finished.

What every cycle has to leave behind

Three things: what it changed, what the number did, what it will try next. In practice that is two files. The append-only JSONL log is the machine's memory, and the reader keys each entry by property and week, so re-running on the same day replaces that week's snapshot instead of double-counting it. The markdown report is the human copy, and it is the one that gets read in ten seconds over coffee. Neither is a nice-to-have. If you cannot reconstruct what a loop did last month, you do not have a loop, you have a liability.

Anthropic's long-running harness work reaches the same conclusion from the build side: end the session with a commit and a progress update so the next session starts from artifacts rather than from a model's recollection. Same instinct, different clock. It is the reason the SEO skill has to show its receipts, and the reason the loop is only one component in a larger agent harness rather than the whole of it.

I changed the check the day I found it; Monday at 8:37 is the first run under it.