7runs runs your AI agent N times against a golden test suite and reports what one run can never tell you: pass rate with an honest confidence interval, flakiness, and whether your last change made things worse. We publish the methodology and the data — and measure your agent for you.
Pass rate against a golden set — measured, not vibes. Deterministic matchers plus an LLM judge for the fuzzy parts.
Same input, N runs. Agents are non-deterministic — one run proves nothing. Flakiness is a first-class metric here, not a footnote.
Compare any two runs. Regressions beyond statistical noise turn your CI red. Noise doesn't.
name: pr-review-inspector
adapter:
type: claude-agent
agent: .claude/agents/security-inspector.md
defaults:
reps: 7
timeout_s: 180
cases:
- id: sql-injection-obvious
label: positive
input:
prompt: "Review this diff for security issues:"
files: [fixtures/diff-sqli.patch]
checks:
- regex: "VERDICT:\\s*ISSUE_FOUND"
- llm_rubric: "Finding names the vulnerable parameter."
- id: clean-diff-no-false-positive
label: negative
input:
prompt: "Review this diff for security issues:"
files: [fixtures/diff-clean.patch]
checks:
- regex: "VERDICT:\\s*CLEAN"
| MODEL AXIS | PASS RATE | WHAT HAPPENED |
|---|---|---|
| Claude Opus | 100% | every agent, every case — ceiling |
| Claude Sonnet 5 | 95–100% | first cracks appear |
| Claude Haiku 4.5 | 67–76% | collapse — concentrated on the false-positive bait |
Four agents (wshobson security-auditor & code-reviewer, VoltAgent security-auditor, Claude Code) measured across three waves. The headline: the model axis discriminates where case difficulty did not — smaller models keep recall, lose precision. Every suite, fixture, raw run record and both judge transcripts are published for replay: github.com/7runs/leaderboard. Maintainers were notified before publication and given a response window.
We turn one critical workflow of your agent into a measurable baseline:
Test your own agents, on your own schedule:
Every case runs N times (default 7). Stable-pass, stable-fail, or flaky — the flaky ones are where agents burn you in production.
Every llm_rubric check is judged twice. If the judges disagree, the result is disputed — counted against you and listed separately. Raw judge output is stored for audit.
The unit under test is the whole agent — multi-turn, tool-using, black box. First-class adapter for Claude Code / Claude Agent SDK, plus a shell adapter for anything you can invoke from a command line.
Your suite, your outputs, your report — delivered to you, retained only under the data policy we agree on. No telemetry, no training on your data, and deletion that actually deletes.
book a 30-min demoWe walk through a live measurement on a reference agent — real runs, real statistics, nothing staged. Bring your hardest workflow.
we measure your agent15–25 golden test cases built from your real scenarios, run 7×. You get the full report: trust score, confidence intervals, flakiness, failure modes.
you keep the suiteThe golden suite and a regression gate stay with your team — every agent change gets measured before it ships. Self-serve dashboard in development.
Is the engine open source? No. The methodology, the benchmark suites, and every raw run record are public and replayable — the engine that produces them is the product. Pricing? Shared at the demo — it depends on what you're measuring; no seat math, no surprises. Your data? Your suite and your results belong to you. No telemetry, no training on your data, real deletion on request.
A 30-minute live demo: we run a real measurement in front of you and show exactly what a trust report would say about your agent. No slides, no staged output.