AI Coding Agents in a Chain: 95% Alone, 51% Together

The Take

Agents that build

Your agent closes 95 tickets out of 100. Hand it twenty in a row and it finishes the job about half the time.

Short version: Yes, once the work becomes a chain. In the BOBA benchmark by Cuadron and Gupta, agents completed 95% of requests one at a time but only 51% of 20-request chains. Building on silently broken code doubled the odds the next request failed. Put an outside check between every request before you run agents unattended.

One of our infra tickets reported a pod failure. The agent fixed it from the one set of steps the ticket gave. We deployed and closed it. Two weeks later, a team testing a different feature hit the same failure, with different replication steps. The root cause had never been touched.

The ticket was not wrong. It showed one path into a bigger crack, and the agent only ever saw that one path. It never tried the other scenarios.

The developer who closed the first ticket got the blame first. The real fault was the floor: one view of the problem, handed down as if it were the whole problem.

Here's what the benchmark did. BOBA-Bench (1 Nov) took 280 chains of 20 consecutive requests from 140 Python projects and ran ten models through them blind. Every request started in a fresh conversation, on whatever the previous one left behind. No pass or fail signal. The agent said "done" when its own tests went green, and a hidden grader found the crack three storeys later.

It's the dabbawala who gets every tiffin to the right station, then hands it on with the wrong code chalked on the lid. Each leg was perfect. Lunch still went to Andheri instead of Churchgate.

Three numbers should worry anyone selling overnight agent runs:

A request built on silently broken code failed about twice as often.
Asked whether an implementation was correct, agents were right 53% of the time. That's a coin with a slight lean.
One outside "verification failed" message plus a single repair round lifted chain completion from 51% to 59%.

95% vs 51%

Requests completed one at a time, against 20-request chains completed, across ten models and 280 chains. A request built on silently broken code failed about twice as often.

✓ Cuadron and Gupta, arXiv:2611.02207 · 1 Nov 2026 · authors' headline, controlled benchmark

And it isn't one paper. The same week, Multi-step SWE found the best of nine frontier models solved 18.5% of tasks built from 5 to 15 dependent steps, across nine languages. In Take It From Here, agents taking over an existing project delivered the new requirement completely 4.95% of the time, while the build stayed green 96.40% of the time.

"Nobody runs twenty requests blind," you'll say. Good. Then this is the price tag on the review you were about to automate away. Look at this fortnight's launches: overnight builds, bots that own triage, agents that fix and merge. The human in between is the bit being sold off.

Fair caveats: Python only, model-written specs, and the paper's own per-request completion inside chains (66 to 84%) doesn't fully square with its headline, so I'm quoting the headline as the authors state it, from a controlled benchmark. What would change my mind? A study showing an agent's own end-of-step checks catch most silent defects on long chains. Nobody's published that yet.

Check the floor before you build the next storey.

From the Research Desk

Verification & trust

Foresight-SWE: the agent passes your tests and plants next month's bug

The agent passed the test you wrote. The bug lived in the test you didn't think of.

Short version: Mostly no. In Foresight-SWE, a Princeton and NYU benchmark of 212 real pull-request pairs, 11 models resolved 60.68% of tasks on average but avoided the bug someone later fixed only 14.84% of the time. Told about the defect, five models fixed it 91.23% of the time. Ask every review what someone will fix next.

Foresight-SWE (31 Oct) pairs a real PR with the later PR that fixed a bug it introduced, across 35 repos and four languages. The agent only sees the first task. Best foresight score: 24.53%. Worst: 6.13%. Describe the defect and models fix it 91.23% of the time; leave it unstated and that drops to 19.91%. So fixing isn't the hard part. Noticing is. Built-in /review commands helped models spot more, but cut how many tasks they finished, every single time.

My verdict: it's the first benchmark that scores the bug you'll fix next month. Steal its question for your PR template: what will someone have to fix after this?

Patel et al., arXiv:2611.01181 · 31 Oct · read at sourceMatters

Understanding & modernisation

Legacy PHP: the best modernisation tool was the one that did nothing

Asked to modernise a hundred PHP files, the best model broke more working files than it fixed.

Short version: Not by default. When five LLMs upgraded 100 WordPress files to PHP 8.3, 83 files already passed every behavioural check untouched and the best model left only 59 passing. Each model broke 32 to 51 working files, against 7 for the rule-based tool Rector. Demand a do-nothing baseline beside any migration score.

Mahmud et al. (29 Oct) judged five LLMs with 630 hidden behavioural checks, against two baselines: leave the file alone, or run Rector. Each LLM repaired at most 10 files. Rector broke 7 and repaired 6. Only 37 of 511 findings (7%) actually needed a change under PHP 8.3. On files that needed nothing at all, the LLMs broke 36 to 60% of them; Rector broke 8%. Caveats: single files, no project context, no execution feedback, and I've read the abstract, not the full paper.

My verdict: this isn't an anti-LLM result. It's an anti-unbaselined result. Rector won by touching only what needed touching, and "files broken" is the number your CFO actually pays for.

Mahmud et al., arXiv:2610.40269 · 29 Oct · abstract readMatters

Deep PM

Intent & specs

Your User Story Is Too Polite for an Agent: write the skeleton, not the paragraph

We spent twenty years teaching PMs to write stories humans could argue with. Agents don't argue. They guess.

Short version: Give a coding agent structure, not just behaviour. In the Skeleton Coding study, specs naming types, signatures, contracts and invariants cut mean test error 48.4% against the best behaviour-only specs. Add four lines to every story an agent builds unattended: the shape, the contracts, one invariant and one real-world edge case.

You know the PM who writes "As a user, I want to export my data", and the developer who asks fifteen questions in refinement. The agent asks none of them. It picks an answer and ships it.

Three papers from the last fortnight show the cost:

Behaviour alone leaves the shape open. Skeleton Coding (28 Oct) gave agents a code-like skeleton (types, signatures, local contracts, global invariants) alongside the tests. Mean test error fell from 40.7% to 21.0%, a 48.4% cut, across four models and three benchmarks. Prose design docs did worse than code-like skeletons. Overly detailed contracts hurt small models.
Passing isn't adequate. In Req2Test-Bench (27 Oct), the best agent wrote passing tests from real Gherkin 82.8% of the time, but behaviourally adequate ones only 33.8% of the time. Vague setup and implied outcomes were a main cause.
Traceable isn't enforced. In a fairness-by-design study (2 Nov), agents captured 97.22% of fairness requirements in the spec and kept 82.41% in code, yet the models weren't significantly fairer. The authors' phrase: "traceable but not enforceable".

The agent-ready story. Add these layers in order, one screen maximum:

Intent. One line in human words: who, what, why. It's for people. Keep it.
Shape. The types and function or endpoint signatures the change introduces or touches. No bodies. If you can't name them, pair with an engineer for fifteen minutes. That's the cheapest refinement you'll ever do.
Contracts. For each named piece: preconditions, postconditions, and what it must not change.
One or two invariants. Rules that hold everywhere after the change. Put them where they block delivery, not where they get audited.
One real-world edge case. From a release note, an incident or a support ticket. In Accepted but Wrong (5 Nov), one edge input written from a release note cut the share of wrong repairs slipping through a test gate from 0.410 to 0.065.

Worked example (illustrative). "Users can export their transaction history as CSV."

Shape: export_transactions(user_id, from_date, to_date) -> CsvFile, touching TransactionRepo.query.
Contracts: from_date must be on or before to_date, else 400; 24 months maximum; read-only on the repo.
Invariant: only the requesting user's own data, amounts in the account's currency.
Edge case: a transaction posted at 23:59 IST on the boundary date.

Four extra lines. The agent now has a fence, not a meadow.

The metric to watch: adequacy, not pass rate. Seed a few bugs and count how many the agent's new tests catch. Req2Test's gap between 82.8 and 33.8 is the gap you're closing.

Caveat: the paper's skeletons came from reference code, so they're probably better than what you'll write on a Tuesday, and nobody measured what writing them costs.

What I'd do: add shape and one invariant to every story an agent builds unattended. What I wouldn't: turn PMs into part-time type designers for a two-line copy change. That's a spec for its own sake.

The agent will fill whatever space you leave it. Leave less space.

Deep Tech

Context & knowledge

Stop Trimming Your Agent's Prompt: why fewer tokens can mean more compute

You cut your agent's prompt by a third and your bill went up. Congratulations, you've discovered prompt caching the expensive way.

Short version: Yes, trimming agent history can cost more. In The Physical Cost of Context, keeping only the last five tool outputs cut prompt volume 33.3% but raised uncached prefill 608.5%, because every edit broke the prompt cache. Keep history append-only, prune once at task boundaries, and measure cache misses, not prompt tokens.

Prompt caching is the Irani café that keeps your usual on the counter. Order the same thing and it's on the table in a minute. Change one word at the start of your order every visit ("bun maska, but today less butter") and the cook starts from scratch every single time.

How it works, step by step (from Yu, Zhao et al., 26 Oct):

The accounting. A prefix-caching server (vLLM, SGLang, or a provider's prompt caching) only computes the part of each prompt after the longest prefix it has seen. Uncached prefill = prompt tokens minus cache-hit tokens. That's the real work.
Why pruning backfires. "Keep only the last N tool outputs" edits history at every step, and every edit invalidates the cache after it. In the paper, Mask-5 removed about 75k tokens per task and threw away about 136k reusable ones.
Quality gate first. A policy only gets priced if its quality isn't meaningfully worse than the control. Failed tasks stay in the denominator, so "cheap because it gave up" can't win.
Four policies. Grep (search on demand, full history). Eager (preload 10 retrieved code spans before turn one). Compact (replace exploration with a short anchor, once). Mask-5 (placeholders for tool outputs older than the last five).
Results. Eager cut model calls 35.7%, wall time 29.4% and energy 33.2% with no quality loss, and caching cut its uncached prefill 67.7%. Mask-5 cut prompt volume 33.3%, then raised uncached prefill 608.5% and energy 125.6%, and failed the quality gate (26 against 30 of 100 resolved).
Rules that follow. Put context up front instead of discovering it call by call. Keep history append-only. If you must prune, prune once, at a task boundary. Measure cache misses, not prompt tokens.

Mask-5 against full history

✓ Yu, Zhao et al., arXiv:2610.34763 · 26 Oct · one GPU, one request at a time

Code sketch (Python, illustrative):

cache_sketch.py · illustrative

from dataclasses import dataclass

@dataclass
class Turn:
    prompt_tokens: int   # logical tokens sent
    cached_tokens: int   # cache-hit tokens reported by the provider or server

def uncached(turns):
    return sum(t.prompt_tokens - t.cached_tokens for t in turns)

def compare(policy, baseline, solved, solved_base, max_drop=0.05):
    # Quality gate first: never price a policy that solves less.
    if solved_base and (solved_base - solved) / solved_base > max_drop:
        return "fails quality gate"
    return {
        "prompt_change": sum(t.prompt_tokens for t in policy) / sum(t.prompt_tokens for t in baseline) - 1,
        "uncached_change": uncached(policy) / uncached(baseline) - 1,
        "uncached_per_solved": uncached(policy) / max(solved, 1),
    }

def next_messages(history, new_msgs, at_task_boundary, compact):
    if at_task_boundary:
        return compact(history) + new_msgs  # prune once, then append-only again
    return history + new_msgs                # never edit earlier turns mid-task

Trade-offs and failure modes:

One machine, one request at a time. No batching, no cache pressure, no cloud bill. The percentages won't transfer. The mechanism will.
Eager retrieval can preload the wrong code. When retrieval misses, you carry irrelevant context for the whole run.
Caches expire. Provider caches last minutes unless you pay for longer, so a long human pause resets the advantage.
Windows still fill. Append-only needs a planned compaction point. That's why "prune at task boundaries" matters.
The other side. NVIDIA's You don't need long context (3 Nov) argues later tokens cost up to 7x more and frequent compaction loses little accuracy. Both can hold: compact per task boundary, not per step. Thoughtworks' Technology Radar Vol 35 landed in the same place, putting naive context compression in Caution (Radar, 4 Nov).

When I'd use it: audit cache hits before touching any context-trimming tool on an agent that runs more than ten turns. When I wouldn't: one-shot calls. There's no history to protect.

Stop editing the past. Your bill remembers it.

Myth vs Reality

People & operating model

A Merge Is a Signature, Not a Test Result

Short version: No. GitHub's Octoverse 2026 reports 90% of agent-started pull requests merged within two weeks, a figure GitHub reports about its own platform. Merging measures acceptance, not correctness: in Foresight-SWE, models avoided the bug someone later fixed only 14.84% of the time. Track escaped defects per merged agent PR, not merge rate.

What people say

"Ninety percent of agent PRs get merged. Agent code is production-ready now."

Evidence for

GitHub's Octoverse 2026 (28 Oct) says agent-started PRs merged within two weeks rose from 83% to 90% between February and September, above roughly 86% for the rest. It also says more than 40% of September's PRs were agent-started, up from 17% in March. All self-reported.

The evidence againstMyth

A merge is a reviewer saying yes. Foresight-SWE shows passing the task and avoiding the next bug are different skills (60.68% against 14.84%). In BOBA, agents judging correctness were right 53% of the time. In a TestMu survey (vendor survey, sample size not stated), 81% of teams said they'd merged AI-written code or tests they didn't fully understand.

Myth. Ninety percent of agent PRs got approved. Nobody has measured how many were right.

Teardown

Agents that build

Half a Claim Survives: Copilot Fusion's cost holds, its speed is unchecked

Half the cost is checked and holds. Forty percent faster is a sentence in a keynote.

Short version: Partly. GitHub says Copilot Fusion came within 2 points of Opus 5.5 Max at about half the cost and 40% less time. Artificial Analysis independently measured $5.59 per task against $11.42 for Claude Code on Opus 5.5 (max), but published no timing. Put the cost on your business case and leave the speed off.

At GitHub Universe (28 Oct), GitHub said Copilot Fusion, which routes work between Claude and GPT models at runtime, came within 2 points of Opus 5.5 Max on the Artificial Analysis Coding Agent Index at about half the cost and 40% less time (self-reported). Artificial Analysis checked the cost: $5.59 per task against $11.42, top score on Terminal-Bench 4.0 (71). It published no time figures.

Questions before the speed claim goes on a slide: wall clock or agent steps? On whose hardware? Does routing hold on private code, and what happens to cost when the router escalates hard tasks?

Credit where due: GitHub pointed at an independent index, and the cost number survived. Publish the timing method and the other half survives too.

One Question

How many tickets in a row would you let an agent run before a human checks its work? Give me a number, not "it depends".

Short version: How many requests in a row would you let a coding agent run before a human checks its work? Reply with one number, not a shrug. The best answers, and the reasoning behind them, will feed a future issue of The Dabbawala Protocol.

Measured what breaks when agents chain work, or caught a "done" that wasn't? Send me the idea. The good ones run here, with your name on them.

Ved

Views are my own, not my employer's. Issue text is CC BY 4.0: borrow the tiffin, just leave my name on the lid.