How HumanEval.org ratings work

HumanEval.org answers one question: when real humans compare two AI models blind, side by side, which one do they actually prefer? This article explains how that preference becomes the number on our leaderboards — and how you can recompute that number yourself, byte for byte. It is the readable companion to the normative methodology shipped with our open rating engine.

Blind battles, honest votes

Every data point starts as a battle: two models answer the same prompt under identical settings, and a human judges the pair without knowing which model is which. Positions are randomized per battle, so "the left one" carries no information. Only after voting does the site reveal the identities.

Not every vote counts toward ratings, and that is deliberate:

• Only battles on curated, approved prompts are eligible — prompts submitted ad hoc by users are great for exploring, but they would let people farm specific matchups.
• New accounts vote inside a provisional window; those votes never enter the ratings.
• A hidden vote-quality pipeline (consensus agreement, gold-standard attention checks, behavioral signals) excludes or down-weights low-quality votes before export. The weights it assigns are public in every data dump; how they are derived is deliberately unpublished, because a published anti-gaming filter is an instruction manual for gaming it.

From votes to ratings: Bradley-Terry, not Elo

We fit a Bradley-Terry model per category: each model gets a latent strength, and the probability that model i beats model j is a logistic function of the strength difference. Crucially, this is a maximum-likelihood fit over all votes at once — unlike sequential Elo, the order votes arrive in doesn't matter, and one early lucky streak can't haunt a model forever.

A few properties worth knowing:

• Ties count as half a win for each side, at the vote's weight.
• Vote weights are applied exactly as published in the dump — a vote with weight 0.25 moves ratings a quarter as much.
• Ratings are anchored so the mean of established models sits at 1000, on an Elo-equivalent scale: a 400-point gap means 10:1 expected odds.
• A tiny ridge penalty keeps the fit well-defined even when a model has only wins on record.

Uncertainty is part of the result

Point estimates without uncertainty are marketing, so every rating ships with a 95% bootstrap confidence interval: we resample the votes ~100 times, refit, and report the spread. Two consequences:

1. Rank bands. When two models' intervals overlap, the data cannot order them — so they share a rank. You will see two models both ranked #1 and the next at #3. That is the honest reading.
2. Provisional models. Below a minimum vote count a model keeps its estimate and interval but gets no rank at all — it is listed unranked until enough humans have judged it.

Style control: substance vs. presentation

Humans reward presentation — longer answers, confident formatting, tidy bullet lists — as well as substance. Alongside the raw rating we publish a style-controlled rating from an extended fit with shared covariates for response length, markdown density, list count and header count. Roughly: "how strong is this model with presentation differences held constant?" Both numbers are always published; neither is hidden.

Reproduce it yourself

This is the part we care most about. Every night we publish an anonymized dump of all eligible battles and votes on the downloads page, with SHA-256 digests in a manifest. The official leaderboard is produced by running our open humaneval-ratings package — pinned dependencies, fixed seed, deterministic output — on exactly that dump:

humaneval-ratings compute \
  --battles battles.jsonl --votes votes.jsonl \
  --out leaderboard.json --seed 42

Same inputs, same command, same pinned environment → byte-identical leaderboard.json, digests and all. No hidden inputs, no editorial step between votes and numbers. There is also a free JSON API if you'd rather query the current snapshot than recompute it.

What we don't publish, and why

Two things stay closed, both stated openly: the derivation of vote weights (anti-gaming — the weights themselves are in every dump) and voter identities (dumps carry stable pseudonymous tokens that cannot be linked back to accounts). Everything else — the math, the code, the data, the seeds — is on the table. If you find a discrepancy, that's a bug report we want.