Hive Bench
How coding models perform on real CI failures.
Not a synthetic benchmark. Every row is a model working the Factory lane on production repositories Hive runs CI for — red checks and tickets — with the fix verified by a fresh CI gate on the exact commit. Measured by the same instrument the operators use.
A dated snapshot, refreshed from the live measurement. Rates carry their denominators; rows below the sample bar are shown, not ranked.
Method
Real work, verified by a gate
A run is a real red check or ticket on a production repository. A fix counts as held only when a fresh CI gate on the exact commit passed. Nothing here is a model grading its own answer.
Method
Judged fixes are the denominator
The fix-held rate is over fixes a gate actually judged — green or red. A fix that was pushed and never judged is counted beside the rate as coverage, never inside it as a failure.
Method
Intervals, not point estimates
Every rate is drawn with a 95% Wilson interval and ranked on its lower bound. A model with three perfect runs ranks below one with sixty good ones.
- Source
- Production CI failures
- Verdict
- A fresh gate on the exact commit
- Ranking
- Lower bound of the fix-held rate
Method
What a row means, and what it cannot mean.
Three measurements are joined once, on the control plane, and published unchanged. The rules below are the ones the instrument enforces, not editorial choices.
Real work, verified by a gate
A run is a real red check or ticket on a production repository. A fix counts as held only when a fresh CI gate on the exact commit passed. Nothing here is a model grading its own answer.
Judged fixes are the denominator
The fix-held rate is over fixes a gate actually judged — green or red. A fix that was pushed and never judged is counted beside the rate as coverage, never inside it as a failure.
Intervals, not point estimates
Every rate is drawn with a 95% Wilson interval and ranked on its lower bound. A model with three perfect runs ranks below one with sixty good ones.
A sample bar before ranking
A row is ranked only with at least thirty judged fixes and at least half of its pushes judged. Rows below the bar are listed beneath the table with the bar they missed.
Screened before traffic, ranked by delivery
The offline screen replays a frozen set of historical failures and decides only whether a model is fit to run unattended: it never edited when the right answer was to change nothing, and it completed the protocol. Live delivery ranks what passes.
Ladders, and where a model sat on one
The fleet runs each level as a ladder: a primary and fallbacks. A drained primary hands its traffic to the next rung, so a fallback can carry most of a level. The primary share says how much of a model’s work came from the top rung — a low share means it was measured on failures a rung above had already lost.
Current lineup
The ladder each level runs, as configured.
Read from the fleet’s own configuration when the snapshot was taken. Quota state is not published: a drained rung is a week-old fact by the time you read it.
The first snapshot has not been exported yet.
Delivery
Fix held, on judged fixes.
Ranked on the lower bound of the fix-held rate. Coverage is judged fixes over pushes. Time to green is the median from the run’s start to the first green gate on its commit.
No snapshot yet. The table appears here once the first export has run; the method above does not change.
Offline screen
Fit to run unattended — a gate, not a score.
Each model replays the same frozen set of real historical failures through the production harness. “Eligible” means it never violated scope, completed the protocol on every case, and never edited on a case whose right answer was to change nothing.
No snapshot yet. The table appears here once the first export has run; the method above does not change.
What this page does not show
Left out on purpose.
Each omission is a rule, not an oversight.
- Cost
- No currency, token or cost index of any kind. Model billing is between you and your provider.
- Provider
- Models are named by family. One model reached by several routes keeps the route in the lineup; routes are never summed.
- Merge rate
- Hive holds no pull-request merge state. “Fix held” is the strongest landed-ness fact it can verify.
- Cross-fleet pooling
- Factory runs only. Interactive sessions are a different fleet with different denominators and are not mixed in.
- Vendor benchmarks
- No published index from any vendor enters a row. Delivery on real work is the only ranking signal.
- Reasoning effort
- Each provider’s default effort runs; it is neither controlled nor recorded, so every cross-model comparison carries that confound.
Next
See the lane these numbers come from.
Self-healing CI
From failed check to verified fix
How an eligible red check becomes a bounded coding task and a candidate your branch gate verifies.
Explore the repair loopAgent platform
Workstation, cloud and Factory lanes
One operator seat for coding agents wherever they run, with the branch, PR and CI beside every session.
Explore agent lanesPrivate beta
Bring one repository. See the whole system work.
Start on your own runner, keep your existing CI beside it, and watch Hive turn the next eligible red check into a reviewable fix.