IS Intent & specs
Spec-driven development, requirements for agents, acceptance criteria, spec drift.
From the issues
Your User Story Is Too Polite for an Agent: write the skeleton, not the paragraph
Give a coding agent structure, not just behaviour. In the Skeleton Coding study, specs naming types, signatures, contracts and invariants cut mean test error 48.4% against the best behaviour-only specs. Add four lines to every story an agent builds unattended: the shape, the contracts, one invariant and one real-world edge case.
Your AGENTS.md Is a Stale Spec With a Loudspeaker: why your agent obeys the file over the code
Stale instruction files beat your code. In 320 controlled runs across 40 Python projects, agents followed an out-of-date CLAUDE.md or AGENTS.md instead of the code 90.1% of the time. Keep that file owned and reviewed like code, and add one sentence telling the agent the code wins.
Pin the Decision: the sentence in your ticket that two smart people read two ways
Before an agent builds from a ticket, find every sentence with two readings and pin the answer with one example input. In a study of 600 generated test suites, 42.7% had no test that told two competing readings apart, so both readings passed. Fifteen minutes in refinement closes the gap.
Your Ticket Is the Prompt: smell it before the agent eats it
Lint every ticket an agent will build from. Vague words, hidden actors and missing edge cases went with lower functional correctness in LLM-written code. Ten minutes fixes most of it: three passes for smells, two or three Given/When/Then examples, explicit non-goals, one "must never happen" line and one failing test first.
Numbers on this track
| Number | What it measures | Source | Checked | Track · issue |
|---|---|---|---|---|
| 42.7% | Of 600 generated test suites, share with no test separating two competing readings of the requirement | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 92.5% | Decision points where a separating input was found (separate 40-task benchmark) | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 85% to 90% | Cases where a pin identified the intended reading (separate 40-task benchmark) | Mitsui, Pinning Decisions Before Failure, arXiv:2610.03237 | ✓verified at source | ISIntent & specs Issue #02 |
| 13.88 points | CONTRA's macro-F1 gain over the best baseline on behaviour-changing clarifying questions | Fang et al., CONTRA, arXiv:2610.01769 | ✓verified at source | ISIntent & specs Issue #02 |
| 11.8% vs 20.4% | Share of commits spent on bug fixes, top vs bottom third of repositories by specification quality | Claudino et al., EPIC, arXiv:2610.07534 | ✓verified at source | ISIntent & specs Issue #02 |
| 90.1% | Share of agent choices that followed a stale instruction file instead of the code (320 controlled runs, 40 Python projects) | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 86.4% / 93.9% | Same share for Claude Code and for Codex CLI | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 79 of 80 | Runs in which Claude Code named the conflict and still followed the stale file | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 20 of 80 | Codex CLI runs that rewrote working code to match the stale file (17 of those then failed the original tests) | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 4.2% → 91.4% | Share of choices following the code, before and after the one-sentence fix | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 89.4% → 100% | Test pass rate before and after the one-sentence fix | Xu and Wu, arXiv:2610.30571 | ✓verified at source | ISIntent & specs Issue #03 |
| 97.3% / 91.5% | Precision and estimated recall of a verifier that finds conflicts between CONTRIBUTING files and repository practice | Zheng, Zhou and Tan, arXiv:2610.21316 | ✓verified at source | ISIntent & specs Issue #03 |
| 12.99% | Share of GPT-5.4 variants that passed all tests across 31 combinations of five AGENTS.md instruction categories | Polandra, Higo and Kula, arXiv:2610.21421 | ✓verified at source | ISIntent & specs Issue #03 |
| 0.54 → 0.63 | Module-structure F1 of rebuilt libraries with recovered design documents | Sas et al., arXiv:2610.20068 | ✓verified at source | ISIntent & specs Issue #03 |
| 48.4% | Cut in mean test error from skeleton specs against the best behaviour-only specs (40.7% to 21.0%) | Skeleton Coding, arXiv:2610.38214 | ✓verified at source | ISIntent & specs Issue #04 |
| 82.8% vs 33.8% | Passing tests against behaviourally adequate tests written from real Gherkin by the best agent (abstract) | Req2Test-Bench, arXiv:2610.36584 | ✓verified at source | ISIntent & specs Issue #04 |
| 97.22% / 82.41% | Fairness requirements captured in the spec and kept in code (abstract) | Fairness-by-design study, arXiv:2611.03231 | ✓verified at source | ISIntent & specs Issue #04 |
| 0.410 to 0.065 | Share of wrong repairs passing a test gate, before and after adding one release-note edge input (abstract) | Accepted but Wrong, arXiv:2611.08269 | ✓verified at source | ISIntent & specs Issue #04 |
Papers on this track
- On the Impact of Requirement Smells in LLM-Based Code Generation arXiv:2609.29208
- Pinning Decisions Before Failure: Executable Records of Underspecified Choices in AI-Assisted Code Generation arXiv:2610.03237
- CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation arXiv:2610.01769
- SpecGuard: Proving a Task Is Broken Before the Agent Cheats arXiv:2610.09159
- The EPIC Framework for Spec-Driven Development arXiv:2610.07534
- Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix arXiv:2610.30571
- When CONTRIBUTING Files Fall Short: Revising OSS Contribution Guidelines with Repository Evidence arXiv:2610.21316
- The Role of Agentic Instruction Categories for Code Generation: A Study of Five Python Libraries arXiv:2610.21421
- The Code of Theseus: Can Agents Rebuild Software from Recovered Design Documents? arXiv:2610.20068
- Skeleton Coding arXiv:2610.38214
- Req2Test-Bench arXiv:2610.36584
- Fairness requirements traced from spec to code arXiv:2611.03231
- Accepted but Wrong arXiv:2611.08269
Other tracks
CKContext & knowledge ABAgents that build UMUnderstanding & modernisation VTVerification & trust POPeople & operating model
Last updated . Reuse with credit under CC BY 4.0: “The Dabbawala Protocol, thedabbawalaprotocol.com”.