Your AGENTS.md Is a Stale Spec With a Loudspeaker: why your agent obeys the file over the code

Your agent trusts a file you last touched in March more than the code you merged this morning.

Short version: Stale instruction files beat your code. In 320 controlled runs across 40 Python projects, agents followed an out-of-date CLAUDE.md or AGENTS.md instead of the code 90.1% of the time. Keep that file owned and reviewed like code, and add one sentence telling the agent the code wins.

In Issue #1 I argued your AGENTS.md is too long. Here's the other half of the charge: it's also out of date, and your agent doesn't know.

That file is now the most-read spec in your company and the least-maintained one. Every agent reads it on every run. Nobody reads it on purpose.

I've watched this happen. On a client project we had to prove our agentic development capabilities, so we ran a spike before we'd even won the work. We called it Project Mad Hatter, the obvious Alice in Wonderland nod, and every package and class started with mad_hatter.

We won the work. The project got a new name. The agent didn't get the memo.

It kept writing mad_hatter, silently, and the developers kept renaming it by hand, because nobody asked why it kept happening. A tea party where nobody was invited and everyone was tidying up after it. Then one day an Architect opened the CLAUDE.md. The old name was sitting right there, giving orders. The fix was one instruction away the whole time.

The agent wasn't wrong. It was loyal.

It's the Mumbai local timetable taped to the station wall in 2019. The trains changed. The paper didn't. And your agent is the one passenger who still believes it.

Now the number. Xu and Wu ran 320 controlled runs across 40 Python projects, each with an instruction file that contradicted the code. Agents followed the stale file 90.1% of the time: 86.4% for Claude Code, 93.9% for Codex CLI. (Xu and Wu, arXiv:2610.30571, 22 Oct)

The detail that stings: Claude Code named the conflict in 79 of 80 runs and followed the file anyway. It saw the problem. It obeyed. Codex went further and rewrote working code to match the file in 20 of 80 runs, removing a median 198 lines across 9 files, and the original tests then failed in 17 of those. That isn't following orders. That's following an outdated map off a cliff.

The fix is one sentence in your user-level instructions: when the documentation and the existing code disagree, follow the code, and point out each inconsistency in the final reply. I'm paraphrasing the paper's wording. With it, rewrites dropped to zero and the test pass rate went from 89.4% to 100%.

The authors' explanation is a hypothesis they didn't test: the file arrives as an instruction, the code arrives as tool output, and models are trained to rank the first above the second.

Three more papers from the same fortnight:

• CONTRIBUTING files drift too. A conflict verifier found the gaps between contribution guides and what repositories actually do at 97.3% precision and an estimated 91.5% recall. (arXiv:2610.21316, 15 Oct)
• More instructions isn't better instructions. Across all 31 combinations of five AGENTS.md instruction categories, only 12.99% of the GPT-5.4 variants passed every test. (arXiv:2610.21421, 15 Oct)
• Written intent steers the build, for better or worse. Recovered design documents lifted module-structure F1 from 0.54 to 0.63, while held-out pass rates stayed at 33.7% and 35.8%. (arXiv:2610.20068, 14 Oct)

The fair pushback: "That's the agent working as designed. If I write a rule, I want it obeyed." The fix doesn't ignore your file. It ranks it below the code and makes the agent report every disagreement, so each run becomes a free drift audit.

And yes, these are small synthetic Python projects, two agent and model pairs, an LLM judge, and a paper that isn't peer reviewed yet. But 90% against 4% is lopsided enough that real repos would have to behave very differently to flip it.

What would change my mind: a study on large real repositories showing agents weigh the code above instruction files by default, or that the one-sentence rule makes agents ignore instructions that are correct.

Open the oldest instruction file in a repo you work in and find the first line that's no longer true. Then give the file an owner.

Your AI budget is a number per head, and nobody in the room can say what one head's worth of it produced.

Short version: Budget AI by cost per delivered outcome, not tokens per developer. Gartner found only 13% of 2,002 CIOs report significant value from AI, and only 31% set kill criteria for pilots. Answer five questions each quarter, starting with what one escaped bug costs you.

You know the finance partner who asks "so what did we get for the $1,200 a seat?" and the engineering lead who answers with a PR chart. Both leave the meeting unhappy. Both are right.

The problem is that spend is set per developer, value shows up per outcome, and the cheapest-seat instinct is wrong. Herbold's simulation of 432 scenarios found escaped bugs dominate cost, and the authors conclude there is "neither a universal token budget per developer nor a universal agentic team structure" (arXiv:2610.20394, 14 Oct). Orosz's interviews with 27 workplaces found one company capping spend at $150 per developer per month and another budgeting about $1,200, with merged velocity roughly unchanged at the second (Pragmatic Engineer, 20 Oct). And Gartner found only 13% of 2,002 CIOs report significant or transformational value from AI, while organisations with pilot kill criteria were 2.3x more likely to be among the winners, yet only 31% have them (Gartner, 19 Oct).

So here's the one-page review I'd run once a quarter.

The one metric: cost per delivered outcome. Tool spend, plus supervision hours times the loaded rate, plus escaped-bug cost, divided by outcomes shipped. Use whatever your team already counts as an outcome: features, resolved issues. Track it before and after. Not PRs.

• The team: 10 developers at $1,200 each per month, so $12,000. PRs go up 40%.
• The review cost: 25 extra hours a month at a $60 loaded rate, so $1,500.
• The escapes: two extra escaped bugs a month at $4,000 each, so $8,000.
• The outcome: resolved issues go from 60 to 63. That's $21,500 of AI-related cost for 3 extra issues, about $7,200 each.

Now the conversation is about whether to buy better bug detection, not more seats.

The honest caveat: Herbold's work is a simulation, not field data. Use it for the shape of the argument, bugs dominate and capability beats price, not as a benchmark for your own 61%.

What I'd do: write the kill criteria before I signed the purchase order. What I wouldn't: set a per-developer token cap and call it a strategy. That's a diet, not a meal plan.

A good surgeon walks the patient, reads the scans and orders the tests. Then, before the first incision, someone stops the room and says "tell me the plan."

Short version: Make a stronger model write one plan at the agent's first source edit, then let the cheaper agent carry on. The PAUSE paper reports resolution gains of 16.8% and 14.7% on two benchmarks, at roughly 11 to 15% of a periodic overseer's planner cost. Try it on unattended bug-fix runs.

That's the surgical time-out. PAUSE is the time-out for coding agents: let the agent explore freely, but nobody cuts until a senior has heard the plan. The authors found agents reach their first source-code edit at a median of 60% of the way through a run, so by then they know a lot. (Patel et al., arXiv:2610.29870, 21 Oct)

• Points or percent? The abstract reports gains of 16.8% on SWE-bench Verified (OpenHands) and 14.7% on SWE-bench Multilingual (SWE-agent) without saying whether those are percentage points or relative gains. Don't quote them as points.
• A confident wrong plan anchors the Executor. One plan, no second opinion. If the Planner misreads the bug, the Executor now has authority to be wrong.
• "First source edit" is a heuristic. Agents that edit a config or test file first, or give up without editing, slip past the trigger. Tune what counts as source.
• The Planner sees only the transcript. If exploration was shallow, the plan inherits the gaps.
• Latency. One extra call to a slower model mid-run. Fine for batch, noticeable in an interactive IDE.
• The setup tested was cheap and strong. A Qwen3-coder Executor with a GPT-5.6 Sol Planner. My guess, not the paper's: the gain shrinks when the Executor is already frontier-grade.

The published numbers: against LivePlan, PAUSE scored 3.2 higher on Verified and 3.7 higher on Multilingual, and it matched SICA's periodic overseer at about 11 to 15% of that overseer's Planner cost.

When I'd use it: unattended bug-fix runs where a cheap model does the legwork and I'm paying for a strong model anyway. When I wouldn't: when a human is pairing live with the agent. You're already the time-out.

86.6% is a wonderful number. I just can't check it, and neither can you, and neither can the bank that's about to put it on a slide.

Short version: How good is AI at COBOL migration? OpenAI says GPT-6.1 Sol scores 86.6% at low effort, up from 5.4% for the previous model, and 97.9% at higher effort. It's self-reported, and the post doesn't say how many programs were tested or whether copybooks and job control were in scope. Ask for the estate, not the file.

OpenAI's post, covered by BleepingComputer on 20 October, says GPT-6.1 Sol migrates COBOL to C++, Java, Kotlin, Python or Rust "without changing their behaviour". Success is judged by hidden tests that need byte-for-byte parity on observable outputs, and a migration counts only if every test passes. The previous model scored 5.4% at low effort. GPT-6.1 Sol scored 86.6% at 61% of the API cost, and 97.9% at higher effort, which OpenAI calls close to saturation. It's all self-reported. (OpenAI; BleepingComputer)

That's a strict bar and a real jump, and I'll say so plainly. OpenAI also says production migrations still need expert review and testing. Credit for that.

Here's what I read the post for and didn't find. It doesn't say how many programs were tested or what they were, and it doesn't mention copybooks, JCL or CICS. Seven questions before this number goes on any slide:

This isn't a dig at OpenAI. Publishing a hard, behaviour-level test is better than most vendors manage. Translating one COBOL program is the vada pav of modernisation: everyone can make one. The question is whether you can feed the whole of Dadar station at 6 pm.