We watched an AI agent burn $65 overnight. Here’s the execution control system we built.
A reproducible benchmark, a 13-class failure taxonomy, and why “Ralph loops” need brakes.
Published May 8, 2026 · Last updated May 10, 2026 · 6 min read
The estimate said $2.40. The morning bill said $65.
The agent had retried dozens of times. Nothing merged. No rollback. No audit trail. Just a quiet loop, burning credits while we slept, retrying the same broken approach with a slightly different wrapper each time.
That’s the moment we stopped trying to make agents smarter and started giving them brakes.
This post is about that — the failure mode, the benchmark, and the open-source runtime we wrote to govern it. The whole thing is reproducible in one command and Apache 2.0 licensed: github.com/Keesan12/martin-loop.
The failure has a name now
“Everybody has gotten infatuated with what we call these Ralph Wiggum loops, just like send the thing off and it’ll just go figure something out. A, it never figures anything out. And B, you just get this ginormous bill.”
That’s the loop. Attempt → check → retry → repeat. The pattern works. The pattern is also unbounded. A typical autonomous coding loop has no answer to five questions: what changed, what did it cost, why was it allowed, why did it stop, and can we inspect or resume it later?
Without those answers, your agent isn’t a teammate. It’s a credit-card-shaped hole in the wall.
The benchmark: same task, two runtimes
We built a deterministic benchmark called flaky-CI-gate. It’s a real-world failure mode: a test suite that passes locally but fails in CI because of an environment mismatch the agent has to diagnose and fix. Same starting state, same agent, same model, two configurations:
| Configuration | Result |
|---|---|
| Ungoverned loop | $5.20 · no merge, no audit trail |
| MartinLoop-governed | $2.30 · verified merge with full JSONL record |
2.3× cost reduction. Reproducible:
git clone https://github.com/Keesan12/martin-loop cd martin-loop && pnpm install pnpm --filter @martin/benchmarks eval
What the governed run actually does
When you wrap a loop with MartinLoop, four things change on every attempt: failed attempts are distilled into structured signals so the context window doesn’t fill with noise; failures get classified across a 13-class taxonomy and routed to the right response (a syntax error gets a constraint fix, a hallucination gets a grounding check); a hard --budget cap rejects the next attempt before the API call goes out if it would breach; and the loop can ship a partial, inspectable result instead of stalling.
See the full 13-class failure taxonomy for the breakdown.
Try it in 60 seconds
npx -y martin-loop@latest start # Alternative: install globally npm install -g martin-loop@latest martin run "fix the auth regression" --budget-usd 3.00 --verify "pnpm test" martin dossier --latest
That’s it. Apache 2.0 licensed, no account required. The runtime is and will stay open source — we charge for the hosted dashboard and finance-ready cost reports.
Prefer to drive it from your editor? MartinLoop also ships an MCP server for Claude Desktop, Gemini & Cline.