The numbers ledger
Every number, with its receipt
Every number I've cited, with the primary source, the date and how I checked it. ✓ means I read it at the source. Self-reported means a vendor said so and nobody audited it. Reuse any of it under CC BY 4.0. Just say where you got it.
Download: numbers.json · numbers.csv
| Number | What it measures | Source | Checked | Track · issue |
|---|---|---|---|---|
| >20% | Extra inference cost from repository context files (AGENTS.md), with no general gain in task success | Gloaguen et al., ETH Zurich, arXiv:2602.11988 | ✓verified at source | CKContext & knowledge Issue #01 |
| ~66% | Share of Cursor's system prompt trimmed | Cursor blog | self-reported | CKContext & knowledge Issue #01 |
| 7% | Cursor's total token reduction, "without degrading quality" (method not shown) | Cursor blog | self-reported | CKContext & knowledge Issue #01 |
| 22–25 of 27 | Top-four model scores on enterprise questions where the rules are stated | Era by Eon, arXiv:2609.30055 | ✓verified at source | CKContext & knowledge Issue #01 |
| ≤6 of 24 | Scores of four of six models when the answer hinges on a hidden fact in contradictory records | Era by Eon, arXiv:2609.30055 | ✓verified at source | CKContext & knowledge Issue #01 |
| 82% | Estimated share of LLM migrations made after the model's shutdown date (22,555 commits, 17,703 GitHub repos) | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 94% | LLM apps with the model ID hard-coded | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 89% vs 13% | Post-shutdown migrations under Anthropic's 60 to 114-day notices vs OpenAI's one-year notice | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 8% | LLM migrations that switched model provider | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 94.98% | Best strict governance-gate success by an LLM (Gemini 3.8 Flash; GPT-5.6 Luna 83.29%, DeepSeek v4.1 Flash 74.18%; 300 synthetic projects, 899 runs) | Canale, arXiv:2609.29345 | ✓verified at source | POPeople & operating model Issue #01 |
| 1,700 of 1,700 | Governance gates passed by a deterministic rules baseline | Canale, arXiv:2609.29345 | ✓verified at source | POPeople & operating model Issue #01 |
| 0.5% | Lost-ack writes duplicated by instructed frontier-model agents when a read-back was available | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 56–74% | Lost-ack writes duplicated by frontier-model agents with no read-back | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 28% → 4% | Agent duplicate writes without vs with an idempotency key on every write | Li, LIMBO, arXiv:2609.29095 | ✓verified at source | ABAgents that build Issue #01 |
| 40 of 111 | Agent approval records where the fields a human sees left effects uncovered | Agent Approval Laundering, arXiv:2609.28586 | ✓verified at source | VTVerification & trust Issue #01 |
| 1,000 PRs in 7 days | PRs merged by Kiro's team with 3 engineers and 50+ concurrent agent sessions (no quality metrics disclosed) | Kiro blog | self-reported | ABAgents that build Issue #01 |
| 34.4% | Share of agent runs judged correct that violated the task's own requirements (6 frontier models, 5 agentic coding benchmarks) | Yao et al., TestJack, arXiv:2610.10619 | ✓verified at source | VTVerification & trust Issue #02 |
| 50.6% → 33.2% | Overall resolution rate before and after fixing the evaluators | Yao et al., TestJack, arXiv:2610.10619 | ✓verified at source | VTVerification & trust Issue #02 |
| 1,154 | Reddit and Hacker News posts coded in a study of AI slop in software development | Baltes, Cheong and Treude, arXiv:2603.27249 | ✓verified at source | VTVerification & trust Issue #02 |
| 28 (24 / 2) | Verifier defects found in ReviveBench: 24 false negatives, 2 false positives | Liu et al., ReviveBench, arXiv:2609.36161 | ✓verified at source | VTVerification & trust Issue #02 |
| 43.1% | Applicable project policies violated by agents even when patches were functionally correct | Truong et al., SWE-CC, arXiv:2610.06193 | ✓verified at source | VTVerification & trust Issue #02 |
| 11.2% to 58.7% | Runs with unnecessary defensive work despite explicit evidence it wasn't needed | Luo et al., ParanoiaEval, arXiv:2610.08662 | ✓verified at source | VTVerification & trust Issue #02 |
| ~21% / ~91% | More tasks completed / more review time, per Bain's survey of senior technology leaders | Bain Technology Report 2026 | self-reported | POPeople & operating model Issue #02 |
| 47% | More concurrent workstreams, per the same Bain survey | Bain Technology Report 2026 | self-reported | POPeople & operating model Issue #02 |
| ~1 in 4 | Installed agent skills that sit beside another skill doing the same job (20,947 repositories) | Yan et al., One Skill Too Many, arXiv:2610.11647 | ✓verified at source | CKContext & knowledge Issue #02 |
| ~1 in 5 | Runs in which the lookalike skill took over, task completion unchanged (312 pairs, 6,368 runs) | Yan et al., One Skill Too Many, arXiv:2610.11647 | ✓verified at source | CKContext & knowledge Issue #02 |
| 0.9% | Substituted runs whose final reply named the skill actually used | Yan et al., One Skill Too Many, arXiv:2610.11647 | ✓verified at source | CKContext & knowledge Issue #02 |
| 42.7% | Of 600 generated test suites, share with no test separating two competing readings of the requirement | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 92.5% | Decision points where a separating input was found (separate 40-task benchmark) | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 85% to 90% | Cases where a pin identified the intended reading (separate 40-task benchmark) | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 13.88 points | CONTRA's macro-F1 gain over the best baseline on behaviour-changing clarifying questions | Fang et al., CONTRA, arXiv:2610.01769 | ✓verified at source | ISIntent & specs Issue #02 |
| 11.8% vs 20.4% | Share of commits spent on bug fixes, top vs bottom third of repositories by specification quality | Claudino et al., EPIC, arXiv:2610.07534 | ✓verified at source | ISIntent & specs Issue #02 |
| 44.27% / 41.18% | Average line and branch coverage of the code that handles LLM output, across 10 agents | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 33.60% | Average mutation score of the same code | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 122 / 88 / 69 | Harness bugs found / previously unknown / confirmed by developers | Xiong et al., Complex Agents, Shallow Tests, arXiv:2610.04921 | ✓verified at source | ABAgents that build Issue #02 |
| 19.7% | Generated packages that were hallucinated, across 16 models on 2024-era models (at least 5.2% commercial, 21.7% open-source) | Spracklen et al., arXiv:2406.10279 (v3) | ✓verified at source | VTVerification & trust Issue #02 |
| 43% / 58% / 39% | Hallucinated packages repeated in all 10 runs / more than once / never repeated | Spracklen et al., arXiv:2406.10279 (v3) | ✓verified at source | VTVerification & trust Issue #02 |
| 4.62% to 6.10% | Package hallucination rates on five 2026 frontier models | Churilov, arXiv:2605.17062 | ✓verified at source | VTVerification & trust Issue #02 |
| 127 | Package names all five 2026 models invented identically (109 on PyPI, 18 on npm) | Churilov, arXiv:2605.17062 | ✓verified at source | VTVerification & trust Issue #02 |
| up to 45x | GitLab Orbit's claimed reduction in agent retries (no method given) | GitLab press release | self-reported | CKContext & knowledge Issue #02 |
| up to 4.5x | GitLab Orbit's claimed reduction in tokens (no method given) | GitLab press release | self-reported | CKContext & knowledge Issue #02 |
| 3,500+ | Organisations in the Orbit beta, per GitLab | GitLab press release | self-reported | CKContext & knowledge Issue #02 |
| 90.1% | Share of agent choices that followed a stale instruction file instead of the code (320 controlled runs, 40 Python projects) | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 86.4% / 93.9% | Same share for Claude Code and for Codex CLI | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 79 of 80 | Runs in which Claude Code named the conflict and still followed the stale file | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 20 of 80 | Codex CLI runs that rewrote working code to match the stale file (17 of those then failed the original tests) | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 4.2% → 91.4% | Share of choices following the code, before and after the one-sentence fix | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 89.4% → 100% | Test pass rate before and after the one-sentence fix | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 97.3% / 91.5% | Precision and estimated recall of a verifier that finds conflicts between CONTRIBUTING files and repository practice | Zheng, Zhou and Tan, arXiv:2610.21316 | ✓verified at source | ISIntent & specs Issue #03 |
| 12.99% | Share of GPT-5.4 variants that passed all tests across 31 combinations of five AGENTS.md instruction categories | Polandra, Higo and Kula, arXiv:2610.21421 | ✓verified at source | ISIntent & specs Issue #03 |
| 0.54 → 0.63 | Module-structure F1 of rebuilt libraries with recovered design documents | Sas et al., arXiv:2610.20068 | ✓verified at source | ISIntent & specs Issue #03 |
| 718 | Firms in the Harvard study of coding-agent adoption | Chen and Stratton (Harvard), PDF | ✓verified at source | POPeople & operating model Issue #03 |
| +23% | Pull requests after adopting agents on top of assistants (resolved Jira issues: not statistically significant). Data from vendor Jellyfish's platform | Chen and Stratton (Harvard), PDF | ✓verified at source | POPeople & operating model Issue #03 |
| +3.45 days (+49%) | Increase in time from PR to merge after adopting agents | Chen and Stratton (Harvard), PDF | ✓verified at source | POPeople & operating model Issue #03 |
| 45 of 50 | Engineer-approved flag-removal PRs at DoorDash, per the company's own report | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| $4.79 / 13.8 min | Average cost and time per stale feature flag cleaned, per DoorDash | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| 1,659 | Stale feature flags at DoorDash as of May 2026 (over 56,000 flags across 623 repositories) | Chandratre and Datar, arXiv:2610.31351 | self-reported | ABAgents that build Issue #03 |
| 13% | Share of 2,002 CIOs reporting significant or transformational value from AI | Gartner CIO survey | ✓verified at source | POPeople & operating model Issue #03 |
| 31% | Share of organisations that have kill criteria for AI pilots (those that do are 2.3x more likely to be among the winners) | Gartner CIO survey | ✓verified at source | POPeople & operating model Issue #03 |
| 432 / 61% | Scenarios simulated, and the share of agent budget spent on finding bugs in the optimal teams | Herbold, arXiv:2610.20394 | ✓verified at source | POPeople & operating model Issue #03 |
| $150 to $1,200 | Per-developer monthly AI spend at two of 27 workplaces interviewed (anonymised) | Pragmatic Engineer (Orosz) | ✓verified at source | POPeople & operating model Issue #03 |
| 60% | Median point in a run at which a coding agent reaches its first source-code edit | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| +16.8% / +14.7% | Resolution gains on SWE-bench Verified and Multilingual with a plan at the first edit (relative or points not stated) | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 11 to 15% | PAUSE's planner cost as a share of a periodic overseer's (SICA) at comparable performance | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 35.4% | Eligible failed SWE-bench Verified tasks recovered by replaying to the first edit and injecting a plan | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 33 to 41% | Fewer Executor tokens when resuming from the recorded prefix than restarting | Patel et al., PAUSE, arXiv:2610.29870 | ✓verified at source | ABAgents that build Issue #03 |
| 480 of 731 | SWE-Bench Pro tasks repaired or retired by 23 engineers in 404 hours | Dan et al., SWE-Bench Pro V2, arXiv:2610.24206 | ✓verified at source | VTVerification & trust Issue #03 |
| 99.4% | Best resolution rate by frontier agents on the 642 SWE-Bench Pro V2 tasks | Dan et al., SWE-Bench Pro V2, arXiv:2610.24206 | ✓verified at source | VTVerification & trust Issue #03 |
| 52.9% to 98.0% | Spread of systems on the 51-task Hard split of SWE-Bench Pro V2 | Dan et al., SWE-Bench Pro V2, arXiv:2610.24206 | ✓verified at source | VTVerification & trust Issue #03 |
| 26.4% / 20.4% | Best success rate on Linux-target and Windows-target cross-OS migration tasks (140 tasks) | Jin et al., OS-Bench, arXiv:2610.19368 | ✓verified at source | UMUnderstanding & modernisation Issue #03 |
| 86.6% | GPT-6.1 Sol's accuracy at low reasoning effort on OpenAI's COBOL migration evaluation (previous model 5.4%; 97.9% at higher effort; 61% of the API cost) | OpenAI, low-resource programming languages post | self-reported | UMUnderstanding & modernisation Issue #03 |
| 95% vs 51% | Share of requests completed one at a time against share of 20-request chains completed (280 chains, 140 Python projects, 10 models; authors' headline, controlled benchmark) | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| about 2x | How much more often a request failed when built on silently broken code | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 53% | How often agents were right when asked whether an implementation was correct | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 51% to 59% | Chain completion with one external "verification failed" signal and one repair round | Cuadron and Gupta, arXiv:2611.02207 | ✓verified at source | ABAgents that build Issue #04 |
| 18.5% | Share of multi-step tasks (5 to 15 dependent steps, 69 tasks) solved by the best of nine frontier models (abstract) | Multi-step SWE, arXiv:2611.07166 | ✓verified at source | ABAgents that build Issue #04 |
| 4.95% | How often agents taking over an existing project delivered the new requirement completely; build pass 96.40% (abstract) | Take It From Here, arXiv:2611.07307 | ✓verified at source | ABAgents that build Issue #04 |
| 60.68% vs 14.84% | Average task resolve rate against average foresight rate (avoiding the bug a later PR fixed), 11 models, 212 PR pairs | Patel et al., arXiv:2611.01181 | ✓verified at source | VTVerification & trust Issue #04 |
| 91.23% vs 19.91% | How often five models fixed the later bug when it was described, against when it was unstated | Patel et al., arXiv:2611.01181 | ✓verified at source | VTVerification & trust Issue #04 |
| 83 vs 59 | WordPress files passing every hidden check untouched, against the best LLM after upgrading to PHP 8.3 (100 files, 630 checks; abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
| 32 to 51 vs 7 | Working files broken by each LLM, against the rule-based tool Rector (abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
| 7% | Share of 511 findings (37) that needed a change under PHP 8.3 (abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
| 48.4% | Cut in mean test error from skeleton specs against the best behaviour-only specs (40.7% to 21.0%) | Skeleton Coding, arXiv:2610.38214 | ✓verified at source | ISIntent & specs Issue #04 |
| 82.8% vs 33.8% | Passing tests against behaviourally adequate tests written from real Gherkin by the best agent (abstract) | Req2Test-Bench, arXiv:2610.36584 | ✓verified at source | ISIntent & specs Issue #04 |
| 97.22% / 82.41% | Fairness requirements captured in the spec and kept in code (abstract) | Fairness-by-design study, arXiv:2611.03231 | ✓verified at source | ISIntent & specs Issue #04 |
| 0.410 to 0.065 | Share of wrong repairs passing a test gate, before and after adding one release-note edge input (abstract) | Accepted but Wrong, arXiv:2611.08269 | ✓verified at source | ISIntent & specs Issue #04 |
| 33.3% | Cut in prompt volume from keeping only the last five tool observations (Mask-5) | Yu, Zhao et al., arXiv:2610.34763 | ✓verified at source | CKContext & knowledge Issue #04 |
| +608.5% | Rise in uncached prefill under Mask-5 (energy +125.6%; failed the quality gate, 26 against 30 of 100 resolved). One GPU, one request at a time | Yu, Zhao et al., arXiv:2610.34763 | ✓verified at source | CKContext & knowledge Issue #04 |
| 35.7% / 33.2% | Cut in model calls and in energy from preloading 10 retrieved code blocks (Eager), with no quality loss | Yu, Zhao et al., arXiv:2610.34763 | ✓verified at source | CKContext & knowledge Issue #04 |
| up to 7x | Claimed extra cost of later tokens in long contexts (abstract) | NVIDIA, arXiv:2611.05306 | ✓verified at source | CKContext & knowledge Issue #04 |
| 83% to 90% | Share of agent-started PRs merged within two weeks, February to September (above about 86% for others) | GitHub Octoverse 2026 | self-reported | POPeople & operating model Issue #04 |
| more than 40% | Share of September 2026 PRs that were agent-started, up from 17% in March ("agent-started" is a detection floor) | GitHub Octoverse 2026 | self-reported | POPeople & operating model Issue #04 |
| 81% | Share of teams that merged AI-written code or tests they did not fully understand (sample size not stated) | TestMu AI survey | self-reported | POPeople & operating model Issue #04 |
| 2 pts, half the cost, 40% less time | Copilot Fusion against Opus 5.5 Max on the Artificial Analysis Coding Agent Index (GitHub's claim; the time figure is not independently checked) | GitHub Universe 2026 | self-reported | ABAgents that build Issue #04 |
| $5.59 vs $11.42 | Cost per task for Copilot Fusion against Claude Code on Opus 5.5 (max), independently measured | Artificial Analysis | ✓verified at source | ABAgents that build Issue #04 |
Last updated . Reuse with credit under CC BY 4.0: “The Dabbawala Protocol, thedabbawalaprotocol.com”.