TheDabbawala Protocol
#02by 14 min read

Slop Is Whatever Passes: the AI junk worth worrying about isn't the kind you can spot

Slop in code arrives with a green tick. Plus: the 91% review bill, the lookalike skill that took the job, a fifteen-minute decision pin, and why a made-up package is only harmless until someone registers it.

TK

The Take

Verification & trust

Slop in a LinkedIn post is easy. You smell it in two lines and keep scrolling. Slop in code is worse, because it turns up with a green tick.

Short version: AI slop in code is output that never met a check that could fail it. In the TestJack audit, 34.4% of agent runs that passed evaluation broke the task's own requirements. Audit your tests and judges the way you audit the agent, and report a checker error rate next to every pass rate.

We've all learnt to spot the writing kind. Too many adjectives, the same point thrice, a sentence that says it's not this, it's that. Nobody's taught us to spot the kind that compiles.

So here's my definition, and I'll defend it. Slop is output that cost nothing to make and never met a check that could fail it. That makes it a checking problem, not a taste problem. Researchers who coded 1,154 Reddit and Hacker News posts about slop call it a tragedy of the commons: one person's productivity gain becomes the reviewer's cost. (Baltes, Cheong and Treude, arXiv:2603.27249, March)

I nearly approved some myself. I asked AI to list improvements and efficiencies for a product and its codebase. It had access to the code. It gave me a confident list anyway, and it never checked that code or ran an independent review. The list was headed for a recommendation to fund a small team for focused improvements, and I'd have signed off on it. Then I got talking to a colleague and found that one item, a control and benchmark, was already live. It didn't need doing at all. That chat saved me from looking foolish in front of client leadership. It wasn't a check. It was luck.

Now the number that should bother you. TestJack audited five agentic coding benchmarks across six frontier models. About 34.4% of the trials judged correct violated the task's own requirements. Fix the evaluators and the overall resolution rate falls from 50.6% to 33.2%. (Yao et al., arXiv:2610.10619, 7 Oct)

34.4%

Share of agent runs that passed evaluation but broke the task's own requirements. Fix the evaluators and the resolution rate falls from 50.6% to 33.2%.

✓ Yao et al., TestJack, arXiv:2610.10619 · 7 Oct 2026

It's the board exam where the answer key has mistakes. The topper isn't the one who knew the subject. It's the one who matched the key.

And it isn't one paper. Three more from the same fortnight:

  • ReviveBench. A team building a benchmark audited its own verifiers and found 28 defects. 24 rejected correct work, 2 accepted wrong work. In all but one case, they had blamed the model first. (arXiv:2609.36161, 28 Sep)
  • SWE-CC. Agents wrote functionally correct patches that still violated 43.1% of the applicable project policies. (arXiv:2610.06193, 5 Oct)
  • ParanoiaEval. Agents did unnecessary defensive work in 11.2% to 58.7% of runs across agent configurations, even with explicit evidence it wasn't needed, and stronger task capability didn't mean better judgement about when to hold back. (arXiv:2610.08662, 6 Oct)

The fair pushback: "That's a benchmark problem. My CI runs real tests my team wrote." How many of those did the agent write this quarter? And a bad checker cuts both ways. 24 of ReviveBench's 28 defects rejected good work, which means rework nobody needed.

What would change my mind: a field audit of production CI on agent-written PRs showing checker error under about 5% in both directions. Send it to me.

Don't ask whether it's slop. Ask what could have failed it. Before you ask how good the agent is, ask who's marking the examiner.

RD

From the Research Desk

People & operating model

The Review Bill: 21% more work, 91% more checking

Twenty-one percent more work done, ninety-one percent more time checking it. Somebody call that a productivity gain with a straight face.

Short version: Yes, agents raise output, and the review bill rises faster. Bain's survey of 293 senior technology leaders found developers complete roughly 21% more tasks while review time rises about 91%. Plan reviewer hours the way you plan sprint capacity, before you buy another agent seat.

Bain's Technology Report 2026 (29 Sep) says its survey found developers "complete roughly 21% more tasks but review time rises approximately 91%", while managing 47% more concurrent workstreams. These are leader-reported figures from Bain's own April survey, not a time-and-motion study. (Bain)

Why you should care: it's the first big-consultancy number that puts the review bill next to the output gain. Your team's own ratio is the one that matters, and most teams haven't measured it.

Verdict: The gain is real and so is the bill, and only one of them made it to the slide.

Bain Technology Report 2026 · 29 Sep · leader-reportedMatters
Context & knowledge

One Skill Too Many: the lookalike that quietly took the job

You hired a plumber. His cousin turned up, fixed the tap, and signed the plumber's name on the bill.

Short version: Check which agent skill actually ran, not which one you installed. A study of 20,947 repositories found nearly one in four installed skills duplicates another. In 312 confirmed pairs, the lookalike took over in one in five runs, and only 0.9% of those runs named the skill used.

The study mined 20,947 repositories for installed agent skills. Across 312 confirmed pairs (6,368 runs, 542 agent-hours), the lookalike skill took over in about one in five runs, with task completion unchanged. Only 0.9% of substituted runs said which skill was used. (Yan et al., arXiv:2610.11647, 8 Oct)

Why you should care: your audit trail says skill A ran. Skill B ran. Nothing failed, so nobody looked.

Verdict: A context you can't audit isn't context. It's a rumour with file permissions.

✓ Yan et al., arXiv:2610.11647 · 8 OctMatters
PM

Deep PM

Intent & specs

Pin the Decision: the sentence in your ticket that two smart people read two ways

Your requirement has a sentence with two meanings, the agent has already picked one, and it didn't call to ask.

Short version: Before an agent builds from a ticket, find every sentence with two readings and pin the answer with one example input. In a study of 600 generated test suites, 42.7% had no test that told two competing readings apart, so both readings passed. Fifteen minutes in refinement closes the gap.

You know the stakeholder who says "obviously" right before the one sentence nobody in the room agreed on. The agent was in that room too. It nodded.

The problem is that agents resolve ambiguity silently, and the tests they write don't catch it. In Pinning Decisions, 42.7% of 600 generated test suites, from three models, had no test that separated the competing readings, so both readings passed. On a separate 40-task benchmark, the author's method found a separating input for 92.5% of decision points, and a pin identified the intended reading in 85 to 90% of cases. (arXiv:2610.03237, 2 Oct, single author)

So here's the decision pin I'd run in refinement. It takes fifteen minutes.

The decision pin
  1. Name the decision points. Read the ticket and mark every phrase where two reasonable people could differ: boundaries (above, within, up to), order of operations (before or after the discount), scope words (all users, active orders), defaults, and what happens on failure.
  2. Write both readings, one line each. If you can't write a second reading, it isn't a decision point.
  3. Find the separating input. One concrete case where the two readings give different results. If no input separates them, the difference doesn't matter. Delete the point.
  4. Pin it. Record the point, the input and the chosen outcome. That's the acceptance example the agent must satisfy.
  5. Ask only what changes behaviour. CONTRA keeps a clarifying question only if its plausible answers lead to different programs, and beats the best baseline by 13.88 points of macro-F1. (arXiv:2610.01769) Same rule for humans: a question whose answers lead to the same build is a meeting, not a requirement.
  6. Check the task and the tests don't contradict. SpecGuard shows some tasks have tests that no implementation of the description can pass. (arXiv:2610.09159) A pin that fights an existing test is a conversation to have before the agent starts.
  7. Measure. Count decision points per ticket, pins added, and agent PRs rejected for "not what I meant", split by pinned and unpinned tickets.
Worked example (illustrative)
  • Ticket: "Free delivery on orders above ₹499."
  • Point 1, "above": more than ₹499, or ₹499 and up? Separating input: a cart of exactly ₹499. Pin: delivery is charged.
  • Point 2, order of operations: before or after a coupon? Separating input: a ₹520 cart with a ₹50 coupon. Pin: the threshold applies after the coupon, so delivery is charged.
  • Point 3, scope: does the total include tax? Separating input: ₹480 of items plus ₹24 tax. Pin: items only, so delivery is charged.

Three pins, three tests, zero "but I assumed".

The tracker, one row per decision point: ticket, phrase, reading A, reading B, separating input, chosen outcome, who decided, date.

The honest caveat: in the paper, agents honoured the pins, but a pin was no better than prose stating the same scope. The value is in finding the decision and catching drift later, not in some magic prompt format. There's also a correlation worth knowing: repos with the best spec quality spent 11.8% of commits on bug fixes, against 20.4% for the worst third. That's a pattern across repositories, not proof that better specs cause fewer fixes. (EPIC, arXiv:2610.07534)

What I'd do: pin every boundary and every "obviously" before an agent touches the ticket. What I wouldn't: write forty pins for a two-line change. If no input separates the readings, it's not a decision, it's a nervous habit.

The agent will pick a meaning. Pick it first.

DT

Deep Tech

Agents that build

Your Agent's Weakest Code Is the Code You Wrote: testing the harness with a fake LLM

You ran three interview rounds for the cook and never checked whether the gas regulator leaks.

Short version: Test the code around your agent's model, not just the model. Across 10 well-known agents, the code that handles LLM output had 44.27% line coverage and a 33.60% mutation score, so two in three injected faults went unnoticed. A fake LLM that returns badly shaped replies is the cheap place to start.

Everyone benchmarks the model. Almost nobody tests the code around it: the parser that reads its replies, the router that picks the tool, the retry loop, the state carried to the next turn. That's the harness. It's dull, it's cheap, and it's the reason the kitchen is still standing.

A new study measured it. Across 10 agents, including OpenHands, Aider and Browser Use, the harness code that handles LLM output had 44.27% line coverage, 41.18% branch coverage and a 33.60% mutation score. Flip a condition or delete a line, and two times out of three no test notices. The authors' tool found 122 harness bugs, 88 of them previously unknown, and developers confirmed 69. (Xiong et al., arXiv:2610.04921, 4 Oct)

How it works
  1. Find the LLM-dependent code. Every place the harness consumes model output: response parsing, tool-call extraction, argument validation, retry and fallback logic, history and state updates, stop conditions.
  2. Measure it separately. Run coverage on just those modules. The published baseline is 44.27% line and 41.18% branch.
  3. Build a fake LLM at the real injection point. Replace the provider client, not the internals, with a test double that returns scripted responses in the true runtime shape: the provider's actual response objects, tool-call blocks and stop reasons included.
  4. Script the unhappy shapes. Empty content, text where a tool call was expected, two tool calls where one was expected, malformed JSON arguments, an unknown tool name, output truncated at the token limit, a refusal, a tool observation that is an error string.
  5. Assert on harness behaviour, not model quality. Does it retry, stop, surface an error, keep the history valid, avoid running a half-parsed command?
  6. Mutation-test the harness. Flip conditions and delete lines in the parsing and control code. A surviving mutant is a branch your tests don't check. The published mutation score is 33.60%.
  7. Keep the fixtures. Every production incident becomes a new scripted response.
harness_test.py · illustrative
# Illustrative sketch: Agent and the sandbox fixture are yours to supply
import pytest

class FakeLLM:
    class="s">""class="s">"Stands in for the provider client; returns scripted replies in runtime shape."class="s">""
    def __init__(self, script): self.script = iter(script)
    def create(self, **_): return next(self.script)

def reply(text=None, tool=None, args=None, stop=class="s">"end_turn"):
    content = []
    if text is not None: content.append({class="s">"type": class="s">"text", class="s">"text": text})
    if tool: content.append({class="s">"type": class="s">"tool_use", class="s">"name": tool, class="s">"input": args})
    return {class="s">"content": content, class="s">"stop_reason": stop}

CASES = [
    (class="s">"empty reply",         [reply()]),
    (class="s">"malformed args",      [reply(tool=class="s">"run_shell", args=class="s">"{not json")]),
    (class="s">"unknown tool",        [reply(tool=class="s">"rm_everything", args={})]),
    (class="s">"truncated tool call", [reply(tool=class="s">"edit_file", args={class="s">"path": class="s">"a.py"}, stop=class="s">"max_tokens")]),
    (class="s">"two tools at once",   [{class="s">"content": [
        {class="s">"type": class="s">"tool_use", class="s">"name": class="s">"run_shell", class="s">"input": {class="s">"cmd": class="s">"ls"}},
        {class="s">"type": class="s">"tool_use", class="s">"name": class="s">"run_shell", class="s">"input": {class="s">"cmd": class="s">"pwd"}}],
        class="s">"stop_reason": class="s">"tool_use"}]),
]

@pytest.mark.parametrize(class="s">"name,script", CASES)
def test_harness_survives(name, script, sandbox):
    agent = Agent(llm=FakeLLM(script + [reply(text=class="s">"done")]), tools=sandbox.tools)
    result = agent.run(class="s">"any task", max_steps=3)
    assert result.status in {class="s">"done", class="s">"error_surfaced"}   # never a crash or a hang
    assert sandbox.commands_run_with_bad_args == 0       # nothing half-parsed executed
    assert agent.history_is_valid()                      # the next turn can still be sent
Where it breaks
  • Fixture rot. Providers change response shapes. Record real responses on a schedule and diff them against your fixtures.
  • Mocking too deep. A double placed inside the parser tests nothing. Put it where the SDK client is injected.
  • False comfort. Harness tests prove the plumbing holds. They say nothing about whether the model's plan was sensible.
  • Mutation testing is slow. Scope it to the LLM-dependent modules and run it nightly.
  • Shared blind spots. If an agent writes these tests, review the case list by hand.

When I'd use it: the week a home-grown harness goes anywhere near a real repository. When I wouldn't: if you run a vendor's agent unmodified. Then your job is testing your own tools and hooks, not their loop.

You benchmarked the brain. Now test the nerves.

MR

Myth vs Reality

Verification & trust

The invented package is only harmless until someone registers it

Short version: No. A made-up package fails to install only until someone registers the name. Models repeat their inventions: 43% of hallucinated names came back in all 10 runs, and five 2026 frontier models invented the same 127. Install from a lockfile, and put an age check in front of any package the repo has never seen.

What people say

"If the AI makes up a package, the install just fails. No harm done."

Evidence for
  • Most invented names do not exist, so the install does fail. In the original study, 39% of hallucinated names never repeated across 10 runs of the same prompt. (Spracklen et al., USENIX Security 2025)
  • Rates have fallen. That study found 19.7% overall across 16 models, with at least 5.2% for commercial models and 21.7% for open-source ones, on 2024-era models. A 2026 re-run on five frontier models found 4.62% to 6.10%. (Churilov, arXiv:2605.17062) So "1 in 5" is stale. Today it's closer to 1 in 20.
The evidence againstMyth
  • The names are predictable. 43% of hallucinated packages came back in all 10 runs, and 58% came back more than once. An attacker doesn't guess. They ask.
  • The models now agree with each other. All five 2026 models invented the same 127 names, 109 on PyPI and 18 on npm. One registration covers every tool your team uses.

A made-up package fails to install only until somebody registers it, and the AI will tell anyone who asks which name to register. It's a wrong number that a stranger can go and buy.

TD

Teardown

Context & knowledge

"Up to 45x fewer retries"

"Up to 45x" is my favourite kind of number. It tells you the best day the product ever had and nothing about a Tuesday.

Short version: A launch number with no method is a ceiling, not a result. GitLab says its Orbit code graph gives agents up to 45x fewer retries and 4.5x fewer tokens, self-reported, with 3,500+ organisations in beta. The release names no task, baseline or median. Ask for the floor before you clap for the ceiling.

GitLab's 6 October release says Orbit, its graph of code and delivery data, gives agents "up to 45x fewer retries" and "up to 4.5x fewer tokens", with 3,500+ organisations and 280,000+ agent queries since the June beta. General availability is "next month". It's self-reported, with no method or footnote (GitLab). Two days later Harness bought select Augment Code assets, including its Code Context Engine, which tells you the knowledge layer is turning into the product (Harness). That part is right. The numbers are the problem.

Five questions the release didn't answer:

  1. Up to 45x on which task, and what's the median?
  2. Fewer retries than what baseline: no context at all, or plain grep?
  3. Did the answers get more correct, or only cheaper?
  4. How complete is the graph on a repo with generated code and reflection?
  5. What happens to it when the code changes?

This isn't a dig at GitLab. A graph of code and delivery data is a sensible thing to build, and every vendor in the category quotes "up to". That's the problem. We clap at the ceiling and never ask for the floor.

OQ

One Question

What's the worst AI slop that got past you: a post, a PR or a package? And what check should have caught it?

Audited your own tests or evals and found something embarrassing? Reply with the idea. The best ones run here, with your name on them.

Ved

Views are my own, not my employer's. Issue text is CC BY 4.0: borrow the tiffin, just leave my name on the lid.