CI repair is a high-frequency task. A repository with a few active branches produces failures every day, and each one is a small, bounded investigation: read the log, find the cause, decide whether to patch or retry. If an agent handles that loop, the model behind it runs dozens of times a day. Model choice becomes an operating cost, not a one-off decision.
What the benchmark asks
CI-bench is 80 real GitHub Actions failures, run three times per agent. The headline metric is pass^3: all three attempts have to succeed. First-attempt success (resolved@1) is easier to read and less useful, because a model that solves a task once and fails it twice is not a default you can leave running.
About 15% of the task failures come from environmental causes: a flaky test, a transient dependency failure, a runner problem. The correct action on those tasks is to recognize that no code change is needed and retry. That is a different skill from writing a patch, and it shows up in the results.
The cost frontier
On the cost chart, three names sit at the interesting corner: Muse Spark 1.3 Contributor, GLM 5.3 Flash and DeepSeek V4.1 Flash. All three are cheap, and all three are near the top on pass^3.
The most expensive configurations do not win. At list prices, the cost range spans more than an order of magnitude, and the models at the high end score at or below models that cost a fraction as much. Paying more buys capability on hard, long-horizon tasks; a CI failure is usually not one.
Speed tells the same story
The second chart plots pass^3 against median time to a passing attempt. The useful configurations finish in under three minutes, and the fastest ones are not trading quality for speed: Muse Spark 1.3 and Antigravity 3.7 Flash sit near the top at around a minute per attempt. The slow corner is a small model that scores 27% with a base configuration and 12% with more thinking time, at four to five minutes per attempt. On this task, longer reasoning did not buy reliability.
Speed and reliability tend to arrive together here. The tasks are short and log-driven: read a large log, form a hypothesis, make a small change, run the failing command. A model that is good at that pattern is also fast at it. A model that needs a long reasoning pass has less to gain from the extra time.
What I actually run
I have been daily-driving one of the three: Muse Spark 1.3 Contributor, GLM 5.3 Flash or DeepSeek V4.1 Flash. The choice among them comes down to availability and the task at hand, not to a meaningful quality gap on CI work. That is the practical outcome of a flat frontier: when several models are within a few points on pass^3 and all of them are cheap, pick the one that is fast and available.
I would like to see the same comparison on the long-horizon version of the corpus. Repairing a CI failure is a short task; the open question is whether the ranking survives when the task becomes multi-file and the diagnosis takes longer. My guess is that the cheap models lose ground there, but that is a guess, and it is the next run.
Caveats
Eighty tasks, three attempts. Pass^3 on a corpus this size is a coarse estimate, and closely matched models are within noise of each other. Model versions move under the same name, and API prices change.
The right way to use this is to run the comparison on your own failures. The recipe is simple: pick the failures your repository actually produces, run each three times, and score all three. A model that is two points better on a public set and five seconds slower per attempt may still be the wrong default for a loop that runs every push.
The one conclusion I would defend without a re-run: for CI repair, the largest model is not the obvious choice. The task rewards reliability and speed more than depth, and the cheapest models are now good enough to be the default.