CI failures are constant. Every active repository produces them, most are small, and diagnosing one still takes a person time: read a long log, form a hypothesis, change something, run the failing command again. It is the shape of work an agent should take over, which is why it needs measuring rather than assuming.
This is the result of that measurement. We built CI-bench from 80 real GitHub Actions failures, ran each agent three times, and scored all three attempts. The headline is simple: first-attempt scores hide reliability. The rest of this post is what the runs showed.
Why CI is a useful benchmark
A CI failure has an executable problem statement. The job, the step and the log are given. There is no ambiguous issue description to interpret. That makes it more objective than an issue-to-patch benchmark and closer to production.
It also tests skills that code benchmarks do not isolate:
- Reading long, noisy logs under context pressure.
- Reasoning about toolchains, lockfiles, caches and OS matrices.
- Making a change and observing whether it worked.
- Recognizing a flaky test or a transient dependency failure and rerunning only the affected step.
The last one is triage, where the interesting failures were.
The corpus
The corpus is 80 failures drawn from real GitHub Actions runs, across Python, JavaScript, Go and Rust.
Each failure produced a base task and a variation that share the underlying run.
Scoring, and why reward hacking is the first problem
Agents are good at making CI pass, and at making CI pass without fixing anything: delete the failing test, add continue-on-error, pin an old dependency, append || true. A benchmark that grades on the exit code alone rewards the cheapest of those moves.
Each submission therefore runs hidden tests and task-specific tamper checks: protected paths, deleted tests, weakened assertions and suspicious configuration changes are rejected. The score reflects whether the problem was fixed, not whether the job turned green.
Three metrics: resolved@1 for the first attempt, pass@k for at least one success in k attempts, and pass^k for success on every attempt. For an agent that runs unattended, pass^k is the one that matters.
Results: the ranking changes with repetition
Muse Spark 1.2 had the best first-attempt result at 73 of 80 (91.2%). Antigravity Gemini 3.7 Flash had the best all-three result at 66 of 80 (82.5%). Muse Spark lost nine tasks when all three attempts had to pass, dropping to 64 of 80 (80%).
DeepSeek V4 Pro tied for second on the first attempt but fell to 55% when every attempt had to pass. Codex Sol had a lower first-attempt score and barely moved across three runs. Those are the cases resolved@1 hides: a model can solve a task once without solving it reliably.
Rust had the highest resolution rate in this set. The compiler catches many issues early, which probably makes the failures more legible from the logs. It is a 10-task slice, so treat it as a pattern to investigate rather than a conclusion.
Failure type matters more than model
Configuration, lint, static-analysis and compile failures were usually resolved. Those tasks tend to have local fixes and direct evidence in the logs.
Flaky tests and environment drift were harder, because the log can point away from the underlying issue, or because the repository does not need a change at all. Tasks where the right answer was no fix were resolved at about half the rate of tasks that needed a patch: agents changed the repository when the correct response was to retry.
One Python task shows it clearly. A wheel download was interrupted, leaving the package half-unpacked and its metadata unwritten. The lockfile, toolchain and checks were correct. The right response was to classify the failure as infrastructure and rerun the step. None of the models did that on the first attempt.
The results point to a distinction between repair and triage. Models produce a working patch when the cause is clear. They are less dependable at deciding whether the problem is in the code, the environment or a transient failure. That is the part of CI work that targeted retries make cheaper.
Cost and speed
Cost did not track reliability. At list prices, configurations spanned almost two orders of magnitude, and the highest all-three score came from a model that cost a few dollars to run across the corpus.
Retrying the failed step
Preloop keeps the microVM and the agent session after a failure, so the failed step can be retried without restarting the run. The benchmark asked whether that saves work.
On a matched DeepSeek V4 Flash set, a warm retry cut model calls from 21 to 7 and lowered median wall-clock time from 2,458 seconds to 195 seconds.
A warm retry is not an independent attempt: it reuses the VM and the failure context, which helps with a flaky test, a temporary dependency failure or a runner issue. It does not fix a bad patch or a wrong diagnosis. In the matched set, most repair cases still needed a fresh attempt.
Limits
The corpus is small and its tasks are not independent: 80 tasks from 40 real GitHub Actions runs, with each run producing a base task and a variation. The tasks are reconstructed rather than replayed byte for byte.
Difficulty is narrow. The median fix changes one line in one file, and the largest changes twelve. That leaves out the multi-file work engineers face. Two tasks defeated every model.
The scope is limited: Linux, GitHub Actions, four languages and an offline environment, with a five-minute limit on most tasks. No Windows or macOS runners, no monorepo builds, no secrets, GPUs or high memory use.
Grading is strict by design. An agent that correctly decides not to patch but assigns the wrong root-cause category is still wrong: diagnosis is part of the task, but one score then combines two skills.
Corpus-v2 will add harder, multi-file tasks and non-Linux runners.
Summary
Three conclusions. Resolved@1 is not enough; a model can solve a task once and still fail to solve it consistently. Lower-cost models were competitive with more expensive ones. And CI success alone is not a sufficient grader; hidden tests and tamper checks catch a change that makes a job pass without fixing anything.
Preloop builds automated, systematic code verification, starting with CI.