Run your agent seven times.
See how often it changes its answer.

7runs is an open-source CLI that runs your AI agent N times against a golden test suite and reports what one run can never tell you: pass rate with an honest confidence interval, flakiness, and whether your last change made things worse.

$ npm i -g @7runs/7runs
MIT · TypeScript · Node ≥ 20 · no telemetry
free & open source — the CLI stays free, forever
EVERY AGENT TEAM ASKS THE SAME THREE QUESTIONS
01

Does it actually work?

Pass rate against a golden set — measured, not vibes. Deterministic matchers plus an LLM judge for the fuzzy parts.

02

How unstable is it?

Same input, N runs. Agents are non-deterministic — one run proves nothing. Flakiness is a first-class metric here, not a footnote.

03

Did I just make it worse?

Compare any two runs. Regressions beyond statistical noise turn your CI red. Noise doesn't.

A SUITE IS A YAML FILE — A REPORT IS AN ANSWER
suite.yaml 
name: pr-review-inspector
adapter:
  type: claude-agent
  agent: .claude/agents/security-inspector.md
defaults:
  reps: 7
  timeout_s: 180
cases:
  - id: sql-injection-obvious
    label: positive
    input:
      prompt: "Review this diff for security issues:"
      files: [fixtures/diff-sqli.patch]
    checks:
      - regex: "VERDICT:\\s*ISSUE_FOUND"
      - llm_rubric: "Finding names the vulnerable parameter."
  - id: clean-diff-no-false-positive
    label: negative
    input:
      prompt: "Review this diff for security issues:"
      files: [fixtures/diff-clean.patch]
    checks:
      - regex: "VERDICT:\\s*CLEAN"
7runs report — what you get
pass rate 78.6%CI95 62.8–88.9
flaky cases 2 / 8same input, different verdicts
precision / recall 0.91 / 0.83over labeled cases
judge disputes 1two judges disagreed → flagged
p95 latency · cost 41s · $0.84per full run
Numbers above are an illustration of the report format. Every percentage ships with a Wilson 95% interval — a bare number would be lying.
BUILT ON FOUR OPINIONS

Variance is the product

Every case runs N times (default 7). Stable-pass, stable-fail, or flaky — the flaky ones are where agents burn you in production.

Never trust one judge

Every llm_rubric check is judged twice. If the judges disagree, the result is disputed — counted against you and listed separately. Raw judge output is stored for audit.

Agents, not prompts

The unit under test is the whole agent — multi-turn, tool-using, black box. First-class adapter for Claude Code / Claude Agent SDK, plus a shell adapter for anything you can invoke from a command line.

Your data stays yours

No telemetry. No database. No network calls except to the model APIs you configured. Runs live in .7runs/ on your filesystem; API keys live in env vars, never on disk.

ZERO TO FIRST REPORT IN FIVE MINUTES
STEP 1
7runs init

Scaffolds config and an example suite you can edit.

STEP 2
7runs run suite.yaml

Executes every case × N reps with a concurrency pool. Wrong answers are never retried — that would corrupt the statistics.

STEP 3
7runs report latest

Self-contained HTML + JSON. Add --ci --baseline in CI to fail builds on real regressions.

WHAT IT DOESN'T DO

No hosted dashboard (yet). No prompt auto-tuning. No step-level assertions on intermediate tool calls — transcripts are recorded when available, but checks target final output only. If you ship agents to production and want to shape what gets built next, we're looking for a handful of design partners. Say hello on X.