Benchmarks

Build SWE-bench-style benchmarks from your own repositories and compare models and harnesses on tasks that reflect your actual codebase.

How It Works

Mine — Tuneloop scans your merged PR history and identifies PRs suitable as benchmark tasks. For each task, Tuneloop creates a prompt based on the linked ticket, existing session transcript, or the PR description.
Validate — Each candidate is validated: can the change be cleanly reverted, and does the test suite pass? Tasks that fail validation are filtered out.
Benchmark — Run any combination of model and harness against the validated tasks. Tuneloop handles the infra to run each task in an isolated container and report metrics.

Tuneloop benchmark creation — pipeline showing Mine PRs, Task prompts, Review and validate, Freeze, and Run stages with funnel from 121 PRs mined to 16 validated tasks

Leaderboard

Results show per-model/harness performance:

Resolve rate — what percentage of tasks the agent solved
Cost per task — average spend per attempt
Token usage — input/output token counts

Tuneloop benchmark leaderboard — ranked model/harness configurations showing resolve rate, solved count, cost per task, and tokens per task

Drill into any configuration to see per-task results, including difficulty, verdict, file and patch accuracy, cost, and time.

Tuneloop benchmark detail view — per-task attempts showing difficulty, verdict, file-to-pass and patch-to-pass accuracy, cost, and time

Use Cases

Model selection: before committing to a model upgrade, compare resolve rates on your own code
Harness evaluation: test whether switching from one agent framework to another improves outcomes
Cost optimization: find the cheapest model that meets your quality bar

Requirements

Benchmarks require the runner service, which needs Docker socket access to spin up isolated containers. See Docker Compose or Helm Charts for setup.