# How coding models perform on real CI failures.

> Not a synthetic benchmark. Every row is a model working the Factory lane on production repositories Hive runs CI for — red checks and tickets — with the fix verified by a fresh CI gate on the exact commit. Measured by the same instrument the operators use.

Source: https://hiveci.io/bench
Language: en
Alternate: https://hiveci.io/de/bench

A dated snapshot, refreshed from the live measurement. Rates carry their denominators; rows below the sample bar are shown, not ranked.
## What a row means, and what it cannot mean.
Three measurements are joined once, on the control plane, and published unchanged. The rules below are the ones the instrument enforces, not editorial choices.
- **Real work, verified by a gate** — A run is a real red check or ticket on a production repository. A fix counts as held only when a fresh CI gate on the exact commit passed. Nothing here is a model grading its own answer.
- **Judged fixes are the denominator** — The fix-held rate is over fixes a gate actually judged — green or red. A fix that was pushed and never judged is counted beside the rate as coverage, never inside it as a failure.
- **Intervals, not point estimates** — Every rate is drawn with a 95% Wilson interval and ranked on its lower bound. A model with three perfect runs ranks below one with sixty good ones.
- **A sample bar before ranking** — A row is ranked only with at least thirty judged fixes and at least half of its pushes judged. Rows below the bar are listed beneath the table with the bar they missed.
- **Screened before traffic, ranked by delivery** — The offline screen replays a frozen set of historical failures and decides only whether a model is fit to run unattended: it never edited when the right answer was to change nothing, and it completed the protocol. Live delivery ranks what passes.
- **Ladders, and where a model sat on one** — The fleet runs each level as a ladder: a primary and fallbacks. A drained primary hands its traffic to the next rung, so a fallback can carry most of a level. The primary share says how much of a model’s work came from the top rung — a low share means it was measured on failures a rung above had already lost.
## The ladder each level runs, as configured.
Read from the fleet’s own configuration when the snapshot was taken. Quota state is not published: a drained rung is a week-old fact by the time you read it.
The current table is at https://hiveci.io/bench — dated, with its window and denominators.
## Fix held, on judged fixes.
Ranked on the lower bound of the fix-held rate. Coverage is judged fixes over pushes. Time to green is the median from the run’s start to the first green gate on its commit.
The current table is at https://hiveci.io/bench — dated, with its window and denominators.
## Fit to run unattended — a gate, not a score.
Each model replays the same frozen set of real historical failures through the production harness. “Eligible” means it never violated scope, completed the protocol on every case, and never edited on a case whose right answer was to change nothing.
The current table is at https://hiveci.io/bench — dated, with its window and denominators.
## Left out on purpose.
Each omission is a rule, not an oversight.
| | |
|---|---|
| Cost | No currency, token or cost index of any kind. Model billing is between you and your provider. |
| Provider | Models are named by family. One model reached by several routes keeps the route in the lineup; routes are never summed. |
| Merge rate | Hive holds no pull-request merge state. “Fix held” is the strongest landed-ness fact it can verify. |
| Cross-fleet pooling | Factory runs only. Interactive sessions are a different fleet with different denominators and are not mixed in. |
| Vendor benchmarks | No published index from any vendor enters a row. Delivery on real work is the only ranking signal. |
| Reasoning effort | Each provider’s default effort runs; it is neither controlled nor recorded, so every cross-model comparison carries that confound. |
## See the lane these numbers come from.
- **From failed check to verified fix** — How an eligible red check becomes a bounded coding task and a candidate your branch gate verifies.
- **Workstation, cloud and Factory lanes** — One operator seat for coding agents wherever they run, with the branch, PR and CI beside every session.
## Bring one repository. See the whole system work.
Start on your own runner, keep your existing CI beside it, and watch Hive turn the next eligible red check into a reviewable fix.