Take It From Here (ReqMemBench)
Authors per arXiv listing
Agents taking over an existing project delivered the new requirement completely 4.95% of the time while the build passed 96.40% (abstract only).
The research ledger
Every paper I've read, and my verdict
Every paper cited in the newsletter, the one line that mattered, and my verdict where I gave one. The abstract is free. The opinion took a fortnight.
Accepted but Wrong
Authors per arXiv listing
One release-note edge input cut the share of wrong repairs passing a test gate from 0.410 to 0.065 (abstract only).
Multi-step SWE
Authors per arXiv listing
The best of nine frontier models solved 18.5% of 69 tasks built from 5 to 15 dependent steps (abstract only).
You don't need long context
NVIDIA authors
Argues later tokens cost up to 7x more and frequent compaction loses little accuracy (abstract only).
Fairness requirements traced from spec to code
Authors per arXiv listing
Agents captured 97.22% of fairness requirements in the spec and kept 82.41% in code, yet the models were not significantly fairer (abstract only).
Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone
Cuadron, Gupta
In a controlled benchmark, agents completed 95% of requests one at a time but only 51% of 20-request chains, and one external failure signal plus one repair round lifted chains from 51% to 59%.
Foresight-SWE
Patel, Liao, Henderson, He, Chen
Eleven models resolved 60.68% of tasks on average but avoided the bug a later PR fixed only 14.84% of the time.
matters
It's the first benchmark that scores the bug you'll fix next month.
LLM upgrades of legacy WordPress PHP files, with do-nothing and Rector baselines
Mahmud et al.
Five LLMs each broke 32 to 51 working files on a PHP 8.3 upgrade; the rule-based tool Rector broke 7 (abstract only).
matters
It's an anti-unbaselined result: Rector won by touching only what needed touching.
Skeleton Coding
Authors per arXiv listing
Specs with types, signatures, contracts and invariants cut mean test error 48.4% against the best behaviour-only specs.
Req2Test-Bench
Authors per arXiv listing
The best agent wrote passing tests from real Gherkin 82.8% of the time but behaviourally adequate ones only 33.8% (abstract only).
The Physical Cost of Context
Yu, Zhao et al.
Keeping only the last five tool outputs cut prompt volume 33.3% but raised uncached prefill 608.5%, because each edit broke the prompt cache.
Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix
Xu, Wu
Across 320 controlled runs, agents followed a stale instruction file instead of the code 90.1% of the time, and one added sentence moved choices following the code from 4.2% to 91.4%.
Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash
Chandratre, Datar
Engineers approved 45 of 50 flag-removal PRs at an average $4.79 and 13.8 minutes per flag, against an estimated one to two hours by hand (self-reported).
matters
It's scoped, boring and priced, which is why I believe it more than the flashy ones.
PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code Edit
Patel, Tan, Xue, Sun
Agents reach their first source edit at a median of 60% of a run, and one plan from a stronger model at that moment lifts resolution at a fraction of a periodic overseer's planner cost.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Dan, Da, Bhutani, Romero Calvo, Guevara, Carabedo, Bernal, Reyna, Monsalve Mendez, Liu, Zhang
23 engineers repaired or retired 480 of 731 tasks in 404 hours; frontier agents resolve up to 99.4% of the 642 V2 tasks, while the Hard split spreads from 52.9% to 98.0%.
When CONTRIBUTING Files Fall Short: Revising OSS Contribution Guidelines with Repository Evidence
Zheng, Zhou, Tan
A conflict verifier finds gaps between contribution guides and repository practice at 97.3% precision and an estimated 91.5% recall.
The Role of Agentic Instruction Categories for Code Generation: A Study of Five Python Libraries
Polandra, Higo, Kula
Across 31 combinations of five AGENTS.md instruction categories, only 12.99% of GPT-5.4 variants passed every test.
The Code of Theseus: Can Agents Rebuild Software from Recovered Design Documents?
Sas, Mitrović, Barth, Rehm, Salani
Recovered design documents lifted module-structure F1 from 0.54 to 0.63, while held-out pass rates stayed at 33.7% and 35.8%.
What Should a Software Team Spend on Agents? Simulation-Based Cost Optimization of Mixed AI-Human Teams
Herbold
Across 432 simulated scenarios, escaped bugs dominate cost, and there is neither a universal token budget per developer nor a universal agentic team structure.
OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration
Jin, Li, Zhu, Tung, Zhang, Wu, Jin, Gao, Kang, Lin, Lee
On 140 cross-OS migration tasks, the best agent setups solved 26.4% of Linux-target and 20.4% of Windows-target tasks.
One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents
Yan, Xu, Li, Xu, Liu, Deng, Ma
Across 312 confirmed pairs, the lookalike skill took over in about one in five runs, with task completion unchanged. Only 0.9% of substituted runs said which skill was used.
matters
A context you can't audit isn't context. It's a rumour with file permissions.
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Yao, Wang, Sen, Chen, Ray, Song
About 34.4% of the trials judged correct violated the task's own requirements. Fix the evaluators and the overall resolution rate falls from 50.6% to 33.2%.
ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
Luo, Zhang, Xu, Huang, Jin, Li, Salam
Agents did unnecessary defensive work in 11.2% to 58.7% of runs, even with explicit evidence it wasn't needed, and stronger task capability didn't mean better judgement about when to hold back.
SpecGuard: Proving a Task Is Broken Before the Agent Cheats
Biyani, Dvijotham
Some tasks have tests that no implementation of the description can pass.
The EPIC Framework for Spec-Driven Development
Claudino, Xing, Zuo, Menezes, Sarma, Damevski, Feng
Repos in the top third for specification quality spent 11.8% of commits on bug fixes, against 20.4% in the bottom third.
Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
Truong, Goh, Le-Cong, Huo
Agents wrote functionally correct patches that still violated 43.1% of the applicable project policies.
Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
Xiong, Ge, Chen, Lou
Across 10 agents, the harness code that handles LLM output had 44.27% line coverage, 41.18% branch coverage and a 33.60% mutation score.
Pinning Decisions Before Failure: Executable Records of Underspecified Choices in AI-Assisted Code Generation
Takeharu Mitsui
42.7% of 600 generated test suites had no test that separated the competing readings, so both readings passed.
CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation
Fang, Li, Zhang, Jin, Wang, Wang, Jin, Li
CONTRA keeps a clarifying question only if its plausible answers lead to different programs, and beats the best baseline by 13.88 points of macro-F1.
From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents
Liu, Dai, Du, Yang
A team building a benchmark audited its own verifiers and found 28 defects. 24 rejected correct work, 2 accepted wrong work. In all but one case, they had blamed the model first.
When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications
Hyungjin Lukas Kim
An estimated 82% of migrations off deprecated models happened after the shutdown date, once the app was already failing. 94% of the apps had the model ID hard-coded.
matters
AI apps age like milk, not wine, and most teams don't even keep the expiry date on the fridge.
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Gruenbaum, Porat, Natanzon, Zavida, Dinachi, Itzahary
When a question states the rules, the top four models score 22 to 25 out of 27. When the answer hinges on a hidden fact buried in contradictory records, four of six models get 6 of 24 or fewer.
The Last Human Gate: Forward Deployed Engineering for Governance Automation
Jeremy Canale
Strict gate success: 94.98% for Gemini 3.8 Flash, 83.29% for GPT-5.6 Luna, 74.18% for DeepSeek v4.1 Flash. A deterministic baseline passed all 1,700 gates.
matters
"Automate 95%" sounds like savings until someone has to hunt for the 5%, forever.
On the Impact of Requirement Smells in LLM-Based Code Generation
Villamizar, Fischbach, Şahin, Frattini, Ferrari, Korn, Vogelsang, Mendez
More smells went with lower functional correctness, measured by test suites. Cleaning them up helped but didn't guarantee correct code.
Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
Jiapeng Li
When the agent could read back whether a write had happened, instructed frontier models duplicated a lost-ack write only 0.5% of the time. Without a read-back: 56 to 74%. Idempotency keys on every write cut duplication from 28% to 4%.
Agent Approval Laundering: Transitive Effects Beyond the Approved Invocation
Zhang, Xia, Wu, Yue, Zhang, Cheng, Tu
Of 111 approval/trace pairs, in 40 the fields a human sees left effects uncovered. The authors prove no record-only policy can be both safe and permissive.
The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
Aleksandr Churilov
Five 2026 frontier models hallucinated packages at 4.62% to 6.10%, and all five invented the same 127 names.
"An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development
Baltes, Cheong, Treude
Researchers who coded 1,154 Reddit and Hacker News posts about slop call it a tragedy of the commons: one person's productivity gain becomes the reviewer's cost.
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Gloaguen, Mündler, Müller, Raychev, Vechev (ETH Zurich)
Providing context files "does not generally improve task success rates, while increasing inference cost by over 20% on average." Repository overviews, the thing model providers keep recommending, were "not helpful."
We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
Spracklen, Wijewickrama, Sakib, Maiti, Viswanath, Jadliwala
43% of hallucinated packages were repeated in all 10 queries, and 58% more than once.
Last updated . Reuse with credit under CC BY 4.0: “The Dabbawala Protocol, thedabbawalaprotocol.com”.