REPORT · JULY 2026 · WAVES 1–3

Four code-review agents, three Claude models, seven runs per case.
Smaller models keep recall — and lose precision.

A reproducible flakiness benchmark built with 7runs. The agents still catch real vulnerabilities on the cheapest model. What collapses is the other half of the job: on clean code that merely looks dangerous, they start inventing issues.

336 AGENT EXECUTIONS · EVERY CHECK JUDGED TWICE (672 JUDGE CALLS) · 0 INFRA ERRORS · JUDGE PINNED TO SONNET-5 ACROSS ALL WAVES · WRONG ANSWERS NEVER RETRIED
THREE WAVES, THREE FINDINGS
WAVE 1 — EASY CASES

The only failure was environmental

84 runs. One failure — and it wasn't a missed vulnerability. One agent got distracted by the machine's local config (plugins, connectors) and never reviewed the code. Isolate the environment, and the failure disappears.

WAVE 2 — HARD CASES

At Opus, we couldn't break them

Obfuscated injection, false-positive bait, a vulnerability split across two files. All four agents: 21/21 each. Difficulty with a clear answer is not the discriminating axis at the top model tier.

WAVE 3 — MODEL AXIS

The curve finally bends

Same hard cases, smaller models. Sonnet holds ≈ Opus reliability. Haiku drops to 67–76% at ~3× lower cost — and the collapse is almost entirely on one case type.

THE LEADERBOARD — PASS RATE PER 21 RUNS (7 REPS × 3 CASES)
AGENTW1 · EASY
(NATIVE)		W2 · HARD
(OPUS)		W3 · HARD
(SONNET-5)		W3 · HARD
(HAIKU-4-5)		COST/RUN
(OPUS · SONNET · HAIKU)
security-auditor wshobson/agents100%100%95%76%$0.034 · $0.020‡ · $0.010
code-reviewer wshobson/agents100%100%100%67%$0.037 · $0.020‡ · $0.010
security-auditor VoltAgent100%100%100%67%$0.039 · $0.024‡ · $0.012
Claude Code claude -p, Read/Grep/Glob only95%*100%100%76%n/a†
Every 100% carries a Wilson 95% interval of [85%, 100%] at N=21 — "no failure observed in 21 tries," never "perfect." Haiku CIs overlap heavily; the two same-name auditors swap ranks between Sonnet and Haiku, so we do not claim an ordering between them.
* Wave 1 ran Claude Code with the user's full local config; the one failure was config distraction. From Wave 2 on, the row is Claude Code restricted to Read/Grep/Glob with no skills/MCP.
† The shell adapter cannot capture token usage, so this row gets a reliability curve but no cost point — we don't fabricate one. Costs are modeled from measured tokens × list prices;
‡ Sonnet-5 pricing is promotional ($2/$10 per MTok) through Aug 31, 2026; at the standard $3/$15 rate from Sep 1 the Sonnet points are $0.030–$0.035 — still at or below Opus. execution actually ran on a Claude Code subscription.
WHERE IT BREAKS — THE FALSE-POSITIVE COLLAPSE
clean-code bait — runs that correctly said "no issue" (of 7)
7/7ws-sec7/7ws-rev7/7volt7/7ccOPUS-4-8
2/7ws-sec0/7ws-rev0/7volt2/7ccHAIKU-4-5

On the two real vulnerabilities, every agent stayed at or near 7/7 even on Haiku — recall survives the cheap model. On the clean-code bait, Haiku collapsed: two agents became stable-fail (0/7 — reliably inventing an issue in safe code, sometimes in code the diff never touched) and two became flaky (2/7). A consistently-wrong agent is arguably worse than a flaky one: a single test run makes it look dependable. This distinction — stable_fail vs flaky — only exists if you run the case more than once.

What this report does not claim

  • No model-vs-model verdicts: every number is a whole system (prompt + scaffolding + model). Two agents on the same model still differ.
  • No ranking of the two same-name security auditors: the winner flips between tiers and the intervals overlap. Ranking them needs more data, and we'd rather say that than force one.
  • Part of the Haiku penalty is rubric strictness: it often correctly calls the new code safe, then over-claims about pre-existing code and is failed for it. Logged, not hidden.
  • Three cases per suite is small. That's what makes it cheap to reproduce — and wide intervals are printed instead of pretended away.
REPRODUCE IT

Everything is public at github.com/7runs/leaderboard — suites, fixtures, vendored agent definitions (pinned to upstream commit SHAs), and the raw run records and judge transcripts of all 336 executions, for replay. Each maintainer whose agent appears here was notified with their results before publication and given a week to respond; responses are reflected in the repo. Built with the 7runs measurement engine — to see it run on your own agent, book a demo.