Building a coding benchmark from your own repository

Public leaderboards tell you how a model does on Django and sympy. They say nothing about your repo. If you want that number you have to build the benchmark yourself, and this post is what that took on Apache Airflow.

There are two halves. Mining turns merged pull requests into tasks, each backed by a real change and the tests that shipped with it. That part is mechanical and well documented. Then comes the refinement: deciding which of those tests are fair to someone who has not seen the fix, which tasks can be answered as written, and what to add to the ones that cannot. Every serious benchmark since SWE-bench does some version of this, mostly by hand, and it is where most of our time went. We touched on why in an earlier post on under- and over-specified tasks. This is the procedure we ended up with, with the numbers.

We also wrote about frontier models cheating on benchmarks. That was the answer leaking into the environment. The problems here are different: the model does honest work and the test still fails it.

One term, used throughout: the problem statement is the text the agent is given. It is the GitHub issue when the PR has one, and a description written from the PR otherwise.

Mining: turn merged pull requests into tasks

The construction is called a commit pair. Every merged PR that changed both source and tests gives you one candidate.

1. Filter the PRs. Keep a PR only if it changes at least one source file and at least one test file, by path pattern. Drop bot authors, dependency bumps, and release chores by title. Bound the size: we kept 3 to 1,500 changed lines, counting source and test lines only, so a small fix that also regenerates a lockfile is not thrown out.
2. Check out the commit before the merge, overlay only the PR's test files on top of it, and run exactly those tests.
3. Check out the merge commit and run them again.
4. Tests that failed before and pass after are the fail-to-pass set; tests that pass on both sides are the pass-to-pass set, which guards against regressions. A candidate becomes a task when it has at least one fail-to-pass test and everything is green after the fix.
5. Write the problem statement. Use the linked issue when there is one. Otherwise generate a symptom-side description from the PR title and body, never from the diff, and reject any draft that contains an identifier that only appears in the fix.
6. The agent gets the statement and the repository at the parent commit. The grader gets the agent's patch and the hidden tests.

Running the tests on both sides is what we mean by execution validation: the candidate is kept only if the PR's own tests fail before the fix and pass after it. Applying a patch cleanly is not the same thing. In our earliest pipeline 84% of transplanted diffs applied and 35% survived being run.

What mining produced on Airflow

The eval set came from four months of history, March through June 2026, processed newest first. We stopped validating once we had more than enough, so the third row is not a yield.

The first batch of 66 candidates validated at 94%, with a median of 139 seconds per candidate. The environment image built in five minutes. Only 26% of candidates had a linked issue, which is why half the final statements are generated.

A few things cost more time than the mining. We hit them running the same pipeline over five years of Airflow history for training data, and they apply to a four-month window too.

One environment cannot serve a long history

An editable install freezes the package-to-path map and the console entry points at the commit it was built on. If a later commit moves a package, the tests import nothing. Over five years Airflow needed 14 environment pins, one per release line. One of them, 2.1.0, we gave up on: its flask-openid dependency does not compile on any toolchain that still exists.

Environment bugs look like a low-yield month

Our 2.10.0 pin was returning 20% valid against a 70% norm everywhere else. Broken down by month:

It was a typo. A sync script matched the directory task-sdk; Airflow's main branch spelled it task_sdk for exactly those two months. Every test failed at collection, the harness scored that as "no fail-to-pass", and nothing raised an error. Fixing the spelling took the pin from 20% to 60%. Two days later the same kind of thing hit a second pin when Airflow split its providers into sixty packages. We now break every environment's yield down by month before trusting it, because a layout change is a sharp drop on one date and noise is not.

Grade the human fix before you grade any model

Run the gold patch through the packaged verifier on every task. It is cheap and it kept catching things. One Airflow task failed seven regression tests on the human fix itself, because reproducible image layers had stamped fixture files with dates before 1980, which Python's ZIP writer rejects. Until the gold patch passes everywhere you cannot trust any model failure.

Why a validated task can still be unfair

We ran three models on the 54 tasks through the current OpenHands agent SDK, with each model's reasoning replayed across tool turns, a single attempt per task, 200 iterations, a one-hour cap, and no network access. Opus 5 solved 21 of 54. That is low for a frontier model with a full development environment and an hour per task, so we read the failures.

Localization was not the problem. There was no task that every model failed to locate. What we kept seeing instead was all three models editing the right file and failing the same hidden test.

So we read the tests. One task asked for a new state-store backend, and the issue described its behaviour in detail. The hidden test file began:

from airflow.state_store.backend import StateStoreObjectStorageBackend

The issue named neither the class nor the module. Opus implemented all six behaviours the issue described, called its class ObjectStorageStateBackend, put it in state.backend, and scored 0 of 11. The test file did not even collect.

The tests were written by someone reviewing the PR, with the implementation open in the next tab. From that seat, asserting the exact class name is a perfectly reasonable check. The agent does not have the implementation. It has the issue and the repository, and it has to guess the name. We call these hidden choices: a behaviourally correct fix fails because it guessed a different name, keyword argument, or error string.

Hidden-choice tests fail every model equally, so they act as a shared noise floor, and a noise floor compresses exactly the gap between models that the benchmark exists to measure. We had spent a while asking how to make the tasks harder. What actually helped was making the tests fairer.

Refinement 1: review every hidden test

A model reads every fail-to-pass test individually, given the problem statement, the human fix, and the source of the test function. It never sees any model's output. Each test gets one of four verdicts:

On the 54 tasks it excluded 183 of 377 fail-to-pass tests, touching 29 of the 54. This is not saying half of Airflow's tests are bad. They were written by people who had seen the fix.

There are two ways to deal with a test that fails this review. You can change the test so it accepts other correct answers, or you can leave the test alone and stop counting it. Other benchmarks do the first, or rewrite the statement to match the test. We chose the second because it is cheaper: nobody has to decide how loose an assertion should be, the excluded tests stay in the file so the task is still validated by the real upstream suite, the exclusion list is a small file you can publish and hash, and because nothing in the environment changed, every earlier run can be re-scored from its saved test report without running a model again.

Two things we settled before scoring anything of our own. The fairness rule and its lock hash were frozen before we evaluated a model we had trained, because a curated benchmark declared after the fact reads as cherry-picking. And every test must have an explicit verdict. Our first run of the judge sent it too many tests at once and it silently skipped 74 of them on two tasks, which then counted as excluded. A missing verdict now blocks the freeze.

The re-score:

All three went up and the order stayed the same. The gap between Opus and DeepSeek went from four tasks to six.

Refinement 2: review each task as a whole

The per-test review has a blind spot, and one task shows it well. The problem statement, generated from the PR, asked for a way to copy every object under an S3 prefix to another bucket:

The hidden tests import S3CopyPrefixOperator from the provider's operators.s3 module, construct it with source_bucket_name, source_bucket_prefix, dest_bucket_name and dest_bucket_prefix, call execute, and check what was copied. Each test checks behaviour the statement asks for.

The per-test review accepts a class name or an import path as public contract, because nearly every test for a new feature has to import something, and one unstated name on its own is a small thing to guess. So each test passed. But the class name, its import path, and four constructor arguments are all absent from the statement, all have to be guessed at once, and any miss fails every test. The per-test review does not add these up. Six of the thirteen tasks nobody solved had this shape.

So a second review reads the statement and all the surviving tests together and answers one question: could a very competent engineer, given only this statement and this repository, pass all of these tests together? It returns:

• a joint solvability verdict: yes, maybe, no
• a label: well-specified, underspecified, mega-feature, or mispaired
• two difficulty ratings: as written, and as if the statement were complete

The difference between the two ratings is how much is missing. If a task is hard as written and easy once complete, the statement is missing a sentence.

We checked the review against which tasks any model had actually solved:

Refinement 3: add what is missing, without leaking the fix

The whole-task review hands you a list of underspecified tasks. Editing the tests would not help, because every surviving test presupposes the same unstated names. So the missing facts go into the problem statement, and then the clarified statement is reviewed the same way the tests were.

The clarification is drafted by a model that sees the issue, the fair tests, and what the whole-task review reported missing, but never the human fix. Every draft goes through a mechanical leak check: an identifier that appears only in the fix's changed lines, and not in the original statement or the fair tests, rejects the draft. What survives, we review by hand against one rule.

A clarification fact is fair if it is a name or public contract the tests import or assert. It is a leak if it says where the bug is or how to fix it.

What the operator task got

Four facts, appended under an "acceptance criteria" heading. Nothing about how to implement it. We had hand-written a six-bullet version of this first; the model's four-bullet draft dropped two that were derivable from the repo and did the same job.

What the review rejected

• One task's only candidate clarification named the buggy function and the remedy. That is a leak. We rejected it and left the statement as it was.
• Another needed seven bullets of API surface, at which point the clarification is a spec and the task is "implement this spec". We dropped the task. Our working cutoff is about six facts.
• Four tasks got a clarification of one to four facts, all of them names or contracts the tests import. Those stayed in.

When the models tell you

On a pagination task, Opus, DeepSeek, and Qwen each produced a different design. Each passed the three API-level tests. Each failed the same two unit tests, which called an internal helper with the exact four-argument signature the upstream author had chosen. Three different designs failing the same two tests points at the tests, not the models. The two tests stopped counting, and the human patch and all three saved model patches passed the remaining three.

After the three refinements, four clarified statements and one dropped task, the 54 became Airflow-53: Opus 36, DeepSeek 29, Qwen 28. Two of the four clarified tasks are now solved by all three models, so the clarifications did not favour one model.

What the unsolved tasks turned out to be

Six of 53 remain unsolved by any model. All are judged well-specified, and three are rated easy. Reading the trajectories, they fall into four groups:

• The model fixes the bug correctly and one existing test in the same area regresses. Every model did this on an easy task.
• The happy path is done and a case the statement implies but does not spell out is missed. A bare postgresql:// URL also had to be rewritten.
• The base class establishes a convention and no model imitates it. One rename was derivable only from the parent class.
• A long mechanical build where models stop about halfway, at 4 or 5 of 11 tests.

These are not capability limits. They are sloppiness: not running the neighbouring tests, not reading the parent class, not finishing. That is why the tasks stay in even though they fail everyone.

If you are sizing your own set, the best predictor of difficulty we found was the number of fail-to-pass tests.

We do not know how much of this transfers to a repo with a different test culture. Airflow's tests often assert exact names and signatures; a repo that tests mostly through a public API would probably lose fewer tests to the fairness review.

Freeze it

The same model, run through three different agent harnesses, gave us 29%, 40%, and 55% on the same tasks. The number alone means nothing.

Every number we publish cites the dataset lock hash and task count, the scoring policy and verifier version, the agent harness with its iteration cap and time limit, and how retries, timeouts, and errors are counted. Airflow-53's human fixes pass 53 of 53 on two separate grading machines under that frozen identity.

• A frozen dataset is never modified in place. A repair is a new dated build with a diff showing exactly which tasks changed.
• Publish both numbers: raw fail-to-pass for comparability with anyone else, curated for the claim you care about.
• Timeouts count as unresolved and are reported next to the score. One lane hit the one-hour cap on 23 of 54 tasks. Its 50% among graded tasks was not usable.
• Partial results are biased upward, because easy tasks finish first. One lane read 50% at 30 tasks and finished at 35%.
• On 53 tasks, one solve is 1.9 points. Two runs that differ by two tasks are within noise.

The short version

1. Filter PRs to source-plus-tests within size bounds, then replay each one's tests before and after the fix. Keep only what fails before and passes after.
2. Build one environment per release line, and check every environment's yield by month.
3. Grade the human fix first.
4. Review every fail-to-pass test for fairness. Stop counting the unfair ones and leave them in the file.
5. Review each task as a whole. Clarify the underspecified ones, check the clarification for leaks, and drop the ones that cannot be clarified without leaking.
6. Require a verdict for every test.
7. Declare the curation policy before you run the model you care about.
8. Report the full tuple, both numbers, and the failure counts.

We are building this at Tuneloop so teams can run it on their own code. If you are evaluating models on your repository, we would like to compare notes.