AB Agents that build
Coding agents, multi-agent delivery, harnesses and scaffolds, agent-to-agent handoffs.
From the issues
Your Agent Is a Great Sprinter and a Terrible Relay Runner
Yes, once the work becomes a chain. In the BOBA benchmark by Cuadron and Gupta, agents completed 95% of requests one at a time but only 51% of 20-request chains. Building on silently broken code doubled the odds the next request failed. Put an outside check between every request before you run agents unattended.
Half a Claim Survives: Copilot Fusion's cost holds, its speed is unchecked
Partly. GitHub says Copilot Fusion came within 2 points of Opus 5.5 Max at about half the cost and 40% less time. Artificial Analysis independently measured $5.59 per task against $11.42 for Claude Code on Opus 5.5 (max), but published no timing. Put the cost on your business case and leave the speed off.
Fifty Stale Flags at $4.79 Each: the agent job that finally makes sense
Where do coding agents earn their keep? On scoped, boring chores. DoorDash reports a multi-agent system removing stale feature flags at $4.79 and 13.8 minutes each, with engineers approving 45 of 50 pull requests, against one to two hours by hand. Self-reported. Start with the chore you've postponed since 2022.
Stop the Agent Before It Picks Up the Knife: one plan, injected at the first edit
Make a stronger model write one plan at the agent's first source edit, then let the cheaper agent carry on. The PAUSE paper reports resolution gains of 16.8% and 14.7% on two benchmarks, at roughly 11 to 15% of a periodic overseer's planner cost. Try it on unattended bug-fix runs.
Your Agent's Weakest Code Is the Code You Wrote: testing the harness with a fake LLM
Test the code around your agent's model, not just the model. Across 10 well-known agents, the code that handles LLM output had 44.27% line coverage and a 33.60% mutation score, so two in three injected faults went unnoticed. A fake LLM that returns badly shaped replies is the cheap place to start.
Don't Hit Pay Twice: exactly-once writes for agents that retry
Make agent writes safe to retry in the tool contract, not the prompt. Mint an idempotency key in the harness, send it on every non-idempotent write, and read back before any retry. In the LIMBO study, read-back cut duplicate writes from 56 to 74% down to 0.5%, and keys cut them from 28% to 4%.
1,000 PRs in a week: five questions the headline didn't answer
A PR count on its own proves nothing about AI productivity. Kiro reports 1,000 merged PRs in 7 days with 3 engineers and 50+ agent sessions, self-reported, with no rollback rate, review time, PR size or escaped defects. Ask for the denominator before you clap for the number.
Numbers on this track
| Number | What it measures | Source | Checked | Track · issue |
|---|---|---|---|---|
| 0.5% | Lost-ack writes duplicated by instructed frontier-model agents when a read-back was available | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 56–74% | Lost-ack writes duplicated by frontier-model agents with no read-back | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 28% → 4% | Agent duplicate writes without vs with an idempotency key on every write | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 1,000 PRs in 7 days | PRs merged by Kiro's team with 3 engineers and 50+ concurrent agent sessions (no quality metrics disclosed) | Kiro blog | self-reported | ABAgents that build Issue #01 |
| 44.27% / 41.18% | Average line and branch coverage of the code that handles LLM output, across 10 agents | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 33.60% | Average mutation score of the same code | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 122 / 88 / 69 | Harness bugs found / previously unknown / confirmed by developers | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 45 of 50 | Engineer-approved flag-removal PRs at DoorDash, per the company's own report | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| $4.79 / 13.8 min | Average cost and time per stale feature flag cleaned, per DoorDash | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| 1,659 | Stale feature flags at DoorDash as of May 2026 (over 56,000 flags across 623 repositories) | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| 60% | Median point in a run at which a coding agent reaches its first source-code edit | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| +16.8% / +14.7% | Resolution gains on SWE-bench Verified and Multilingual with a plan at the first edit (relative or points not stated) | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 11 to 15% | PAUSE's planner cost as a share of a periodic overseer's (SICA) at comparable performance | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 35.4% | Eligible failed SWE-bench Verified tasks recovered by replaying to the first edit and injecting a plan | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 33 to 41% | Fewer Executor tokens when resuming from the recorded prefix than restarting | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 95% vs 51% | Share of requests completed one at a time against share of 20-request chains completed (280 chains, 140 Python projects, 10 models; authors' headline, controlled benchmark) | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| about 2x | How much more often a request failed when built on silently broken code | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 53% | How often agents were right when asked whether an implementation was correct | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 51% to 59% | Chain completion with one external "verification failed" signal and one repair round | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 18.5% | Share of multi-step tasks (5 to 15 dependent steps, 69 tasks) solved by the best of nine frontier models (abstract) | Multi-step SWE, arXiv:2611.07166 | ✓verified at source | ABAgents that build Issue #04 |
| 4.95% | How often agents taking over an existing project delivered the new requirement completely; build pass 96.40% (abstract) | Take It From Here, arXiv:2611.07307 | ✓verified at source | ABAgents that build Issue #04 |
| 2 pts, half the cost, 40% less time | Copilot Fusion against Opus 5.5 Max on the Artificial Analysis Coding Agent Index (GitHub's claim; the time figure is not independently checked) | GitHub Universe 2026 | self-reported | ABAgents that build Issue #04 |
| $5.59 vs $11.42 | Cost per task for Copilot Fusion against Claude Code on Opus 5.5 (max), independently measured | Artificial Analysis | ✓verified at source | ABAgents that build Issue #04 |
Papers on this track
- Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents arXiv:2609.29095
- Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild arXiv:2610.04921
- Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash arXiv:2610.31351
- PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code Edit arXiv:2610.29870
- Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone arXiv:2611.02207
- Multi-step SWE arXiv:2611.07166
- Take It From Here (ReqMemBench) arXiv:2611.07307
Other tracks
ISIntent & specs CKContext & knowledge UMUnderstanding & modernisation VTVerification & trust POPeople & operating model
Last updated . Reuse with credit under CC BY 4.0: “The Dabbawala Protocol, thedabbawalaprotocol.com”.