Methodology

Benchmarking methodology

HumanEval.org measures which AI models people actually prefer when they compare two outputs blind, side by side. This page is the normative reference for how that preference becomes a number: the protocol under which votes are collected, the exact eligibility rules applied before export, the rating mathematics shipped with the open humaneval-ratings engine, the definition of every performance metric on the site, and the procedure by which anyone can reproduce the official leaderboard byte for byte.

Rating engine: humaneval-ratings 1.1.0
Dump schema: v2
Uncertainty: 100× bootstrap · 95% CI
Provisional below: 30 votes
Official seed: 42
Last methodology change: 2026-09-08

Section 1

Scope

What is measured, what is not, and the categories in which it is measured.

The unit of measurement is a blind pairwise comparison: two models complete the same task under identical conditions and a human judge who cannot see which model is which picks the better result, or calls a tie. Every number on this site — ratings, confidence intervals, rank bands, head-to-head matrices — is computed from those votes, per category, by a published and deterministic procedure.

Preference, not correctness. A vote records which of two outputs a person preferred for the task at hand. We do not grade answers against a key; where task success matters (computer use), the judge sees what each agent achieved and decides.
Per category. Each category is an independent competition with its own task set, harness and display mode. Ratings are anchored per category and are not comparable across categories.
Real-world runs. Performance and cost figures describe the runs that produced the judged outputs — through our inference gateway, at the moment of the battle. They are not maximum achievable figures for any hardware.
Proprietary and open weights alike. Every contestant runs through the same harness with identical settings; the roster and every addition or retirement is public in the changelog.

Category	Modality	How a battle runs	Shown to judges as	Who can start battles
Human-Like Chat human-like-chat	Text	Live: both responses stream side by side; the judge votes immediately.	Markdown rendered	Any registered account
Stealth Text stealth-text	Text	Queued: responses are generated in the background and judged later, by many independent judges, from the judging queue.	Formatting stripped	Any registered account
Browser Use browser-use	Web (computer use)	Computer use: both agents drive a real browser from an identical scene on a dedicated runner; judges watch synchronized recordings.	Verbatim text	Vetted accounts

Section 2

How the data is collected

The battle protocol, from roster to reveal. Blinding is structural, not a policy.

Roster and routing. Active models are listed on /models. Inside a battle a contestant is addressed only by its slot and an internal route name through a single inference gateway; the harness never receives roster identity, so it cannot leak it.
Prompts. Each category draws from a curated, versioned task set with exactly one active version. The arena also accepts a judge's own prompt for exploration, but battles on such community prompts are flagged at creation and never enter the ratings or the public dump. For computer use the “prompt” is a curated scene: a snapshotted browser state plus a task, of which both agents receive an identical copy.
Pairing. Two distinct active models are drawn uniformly at random from the category's pool. There is no seeding, matchmaking or rating-aware pairing in the current methodology version.
Generation. Both contestants receive the same prompt with the same generation settings and run concurrently. A random position seed persisted with the battle decides which output is shown as A and which as B. Telemetry (timing, tokens, estimated cost) is recorded for both.
Presentation. The two outputs are shown side by side under the category's normalization mode (rendered, verbatim, or with formatting stripped). Identities are hidden until after the vote; positions carry no information.
Judging. The judge chooses A, B or tie. One vote per user per battle. Live battles are judged by their creator; queued battles are judged by many independent users, never by their own creator. Skipping a battle reveals the models and excludes that battle from ratings. Judges earn credits for voting; the amount never depends on how they voted.
Anti-abuse, public parts. New accounts vote inside a provisional window whose votes never enter the ratings; provisional-window votes additionally pass a bot challenge. Battle creation and voting are rate-limited per account and per network address. Blinded attention-check battles with a known answer are interleaved into the judging queue. A further hidden layer is disclosed in section 3.

Section 3

Vote quality and eligibility

The exact conditions under which a vote counts. The public dump contains precisely the votes that pass them.

A vote enters the ratings only if all of the following hold. These are the conditions the nightly exporter applies, in the order the code lists them; the site's own win-rate and head-to-head counts apply the same conditions through a shared query fragment.

The vote was cast outside the voter's provisional window.
The vote is not flagged as excluded.
The vote's quality weight clears the export floor (currently 0.2). Votes below the floor are excluded entirely, not down-weighted.
The battle is neither excluded nor skipped.
The battle's prompt is curated and approved. Community and own-prompt battles never count.
The battle is not an attention check — regardless of whether that check is currently active.
Both outputs resolve: response text for text battles, or a recording and an action trace for computer-use battles. This rule is checked when the dump is built; the site's counts do not need response bodies and skip it (at current data shapes the difference is empty).
The battle is not assisted: no human intervened in either computer-use run.

In addition, the export guarantees at most one vote per (battle, voter) pair, and a battle enters the dump only through at least one eligible vote.

Disclosure: hidden vote-quality pipeline

Before export, votes pass a hidden quality pipeline (consensus agreement with other judges, gold-standard attention checks, behavioral signals). Votes it excludes never appear in the public dump; votes it down-weights appear with their reduced effective weight. That weights are applied — and their values — is fully public in every dump; how they are derived is deliberately unpublished, because a published anti-gaming filter is an instruction manual for gaming it. Reproducing the leaderboard from a published dump therefore needs no hidden inputs.

Two further things stay closed and are stated openly: voter identities (dumps carry stable pseudonymous tokens that cannot be linked back to accounts) and the raw network and device signals used for sockpuppet detection, which are stored only as keyed hashes and never exported.

Section 4

Rating mathematics

humaneval-ratings 1.1.0

The normative text shipped inside the open rating engine, rendered from the repository as-is. It is quoted, not paraphrased; the site never forks it.

At a glance

Per category, a Bradley-Terry model is fitted by maximum likelihood on all eligible votes at once (not sequential Elo): each model i has a latent strength θ_i, and the probability that it beats model j is

P (i beats j) = σ (θ_{i} - θ_{j}), σ (x) = \frac{1}{1 + e^{- x}}

Votes enter the likelihood at their published weight; a tie is half a win for each side. Strengths are shown on an Elo-equivalent scale, anchored so the mean of the category's established models is 1000:

{rating}_{i} = 1000 + \frac{400}{\ln 10} (θ_{i} - \bar{θ})

A 400-point gap means 10:1 expected odds, exactly as in classical Elo.

Uncertainty comes from a case bootstrap on votes (100 resamples): the 95% interval is the 2.5th–97.5th percentile of the resampled rating. Models whose intervals overlap share a rank band; models with fewer than 30 votes are rated but unranked. A second, style-controlled fit adds shared covariates for presentation (length, Markdown density, lists, headers) and is always published alongside the raw rating. The full normative text follows.

Source: services/ratings/METHODOLOGY.md, shipped with humaneval-ratings 1.1.0. Section numbers below are the document's own.

Rating methodology — humaneval-ratings v1

Normative description of the mathematics implemented by this package. The public methodology page is derived from this file. Input/output formats are specified in DUMP_FORMAT.md. Every choice below is deliberate and fixed; changing any of it is a methodology version change and will be announced in the ranking changelog.

1. Model: Bradley-Terry maximum likelihood, per category

Ratings are computed independently per category from that category's eligible votes. We fit a Bradley-Terry model: each model i has a latent strength theta_i, and the probability that model i beats model j is

P(i beats j) = sigma(theta_i - theta_j),   sigma(x) = 1/(1+e^{-x})

This is a maximum-likelihood fit on the whole vote set at once — NOT sequential Elo. Order of votes does not matter; there is no recency weighting in v1. Slot randomization is handled upstream by the platform; the model contains no slot/position term in v1 (the raw fit), while the style-controlled fit (§7) absorbs presentation differences.

Only models that appear in at least one eligible vote's battle in the category are fitted there. A model can appear (and be rated) in several categories independently.

2. Ties

A tie vote of weight w is treated as half a win and half a loss for both sides (LMArena-style): it contributes an event "A beats B" with weight w/2 and an event "B beats A" with weight w/2 to the likelihood. Ties therefore pull the two models' ratings together and still count toward vote totals at full weight w.

3. Vote weights

Every vote carries an effective weight weight > 0 from the dump (DUMP_FORMAT.md §4.2). The log-likelihood is weight-weighted:

L(theta) = sum over events  w_e * log sigma(delta_e)

where delta_e is the winner-minus-loser strength difference of event e. A vote with weight 0.25 moves the ratings a quarter as much as a full vote. vote_count in the output is the unweighted record count (transparency: how many humans voted), while the fit uses weights.

4. Anchor and scale

The natural-parameter fit is converted to displayed ratings as

rating_i = 1000 + (400 / ln 10) * (theta_i - mean(theta))

Center: the unweighted arithmetic mean of the fitted strengths of the category's non-provisional models (vote_count >= --min-votes, determined once from the full data) is anchored at 1000; when no model qualifies, the mean of all models is used. Anchoring on the non-provisional set matters for the bootstrap: a provisional model with a handful of votes can hit perfect separation in a resample and blow up, and with an all-model anchor that explosion would shift the center and contaminate every model's confidence interval. The anchor set is fixed across bootstrap rounds. The anchor is per category and per snapshot; ratings are not comparable across categories.
Scale: 400 / ln 10 ≈ 173.7178 rating points per natural-log unit. This makes the scale Elo-equivalent: a 400-point gap means 10:1 expected odds, exactly as in classical Elo (P = 1/(1+10^{-(R_a-R_b)/400})).

5. Regularization, existence, uniqueness

A small L2 (ridge) penalty is applied to make the optimum exist and be unique even under perfect separation (a model with only wins) or a disconnected comparison graph:

objective = L(theta) - (lambda/2) * ||theta||^2,
lambda = 1e-6 * (total vote weight entering the fit)

(for the full fit that is the category's total vote weight; for a bootstrap round it is the resample's total weight).

Scaling lambda with total weight makes the fit invariant to multiplying all weights by a constant, and keeps the relative bias at the 1e-6 level — far below bootstrap CI widths. In the style-controlled fit the same lambda also penalizes the style coefficients.

If the comparison graph is disconnected, the ridge yields finite numbers but strengths across components are only weakly identified; the platform's pairing keeps the graph connected in practice, and the huge bootstrap CIs such models get are the honest signal.

Optimization: Newton-Raphson on the penalized objective (strictly concave, so the optimum is unique and start-point independent), from theta = 0, until the max absolute Newton step is < 1e-10 (cap 100 iterations). The Hessian solve is a small dense numpy.linalg.solve (dimension = number of models + 4 covariates at most); no scipy needed.

6. Uncertainty: bootstrap CIs and rank bands

Point estimates are the fit on the full vote set. Uncertainty comes from a case bootstrap on votes: each of --bootstrap-rounds (default 100) rounds draws N votes uniformly with replacement from the category's N vote records (each keeping its weight) and refits both the raw and the style-controlled model on the resample. A model can lose all its votes in a resample; its strength then falls back toward the ridge prior for that round — widening its CI, which is the honest outcome.

ci_low/ci_high are the 2.5th and 97.5th percentiles of the bootstrapped rating (numpy percentile, method="linear"), i.e. a 95% percentile interval. Same for the _sc fields from the same resamples.

Provisional: a model with vote_count < --min-votes (default 30) is flagged provisional: true. It keeps its rating estimate, CI and order, but gets rank: null — no rank number is displayed.

Rank bands: computed over non-provisional models only, in leaderboard order (published rating descending, ties by slug), using the published (4-decimal-rounded) CI bounds so bands are verifiable from leaderboard.json alone:

The first rated model gets rank = 1.
Each next rated model at position p (1-based among rated models) gets the same rank as the model immediately above it if their CIs overlap (closed intervals: ci_high >= above.ci_low and ci_low <= above.ci_high); otherwise it starts a new band with rank = p (competition numbering — after two models share rank 1, the next distinct band is rank 3).

The chain rule means a band extends while each member overlaps its immediate predecessor. order (the plain sort position, all models including provisional) is emitted alongside rank.

Canonical rank/order come from the raw ratings; the analytics site may re-sort by style-controlled values client-side, but the published band structure is raw-based.

7. Style control

Votes reward presentation as well as substance. To separate the two we also fit an extended Bradley-Terry model with shared style covariates (Arena-style contestant-difference features):

P(A beats B) = sigma(theta_A - theta_B + beta . z)

where z is the battle's style-difference feature vector and beta is shared across all models in the category. The style-controlled rating is theta from this joint fit (anchored and scaled as in §4): it is the estimate of model strength with presentation differences held constant. Both raw and style-controlled ratings are always published.

7.1 Features (computed per response text)

Deterministic heuristics on the exact response_* strings from the dump (normative definitions — they matter for reproducibility):

Feature	Definition
`length_chars`	Number of Unicode code points in the text
`markdown_density`	(count of characters in the set ` `*` `_` `#` `~` `>` `\|`) / max(1, `length_chars`) — a simple, deterministic proxy for formatting-character density, not a Markdown parser
`list_count`	Number of lines matching `^\s(?:[-+]
`header_count`	Number of lines matching `^ {0,3}#{1,6}\s` (ATX headings)

7.2 Per-battle covariates

For each battle, each feature f becomes a normalized difference

d = (f_A - f_B) / (f_A + f_B)    (0 when the denominator is 0)

so z is bounded in [-1, 1] and scale-free. Each of the four d components is then standardized to zero mean / unit variance across the category's votes (each vote contributes its battle's d once; population std, ddof = 0). A component with zero variance is dropped (kept at 0). Standardization parameters are computed once on the full vote set and reused inside bootstrap resamples.

For an event with the roles reversed (B beats A, e.g. from a win_b vote or a tie half), the sign of z flips with the roles.

7.3 Reported coefficients

style_coefficients in the output are beta converted to rating points (beta * 400/ln 10) per +1 standard deviation of the normalized feature difference — e.g. length_chars: 24.18 means "being one SD longer than your opponent is worth ~24 rating points in raw votes".

7.4 Fallback

Style fitting needs enough data. If a category has fewer than --style-min-votes (default 50) vote records, or all four covariates are degenerate (zero variance), the category falls back to raw ratings: style_control: false in the output, the _sc fields mirror the raw values, and style_coefficients is null.

7.5 Battles without response text (recordings)

Computer-use battles (DUMP_FORMAT.md §3.1, modality: "recording") carry no response text — the "response" is a recorded browser session. Their four style covariates are defined as exactly zero. A category consisting only of recording battles is therefore degenerate under §7.4 and always reports style_control: false (raw = style-controlled); inside a text category a recording battle cannot move the style fit. Whether presentation effects exist for recordings (e.g. speed) is an open question; v1 of the methodology deliberately does not model them.

8. Determinism

RNG: numpy.random.default_rng seeded per category with SeedSequence([seed, h]) where h is the first 8 bytes (big-endian) of sha256(category_slug). Adding or removing a category therefore never changes another category's numbers under the same seed.
Categories are processed in sorted slug order; models are indexed in sorted slug order; all reductions run in fixed order.
Output serialization is fixed-format (DUMP_FORMAT.md §6.4): floats rounded half-even to 4 decimals, printed with exactly 4 decimals, sorted keys, ASCII-only, LF.
Byte-identical reproduction is guaranteed under the pinned environment in requirements-lock.txt. Other numpy builds should agree to far below CI widths, but byte-identity is only promised under the pins.

9. Defaults summary

Parameter	CLI flag	Default
Bootstrap rounds	`--bootstrap-rounds`	100
Provisional threshold (votes)	`--min-votes`	30
Style-control minimum votes	`--style-min-votes`	50
Ridge strength	(fixed)	1e-6 x total category vote weight
CI level	(fixed)	95% (2.5/97.5 percentile)
Anchor / scale	(fixed)	mean 1000 / 400 per log10-odds

10. Computer-use battles ("Browser Use")

The ratings mathematics above is modality-agnostic: a vote is a vote. What differs for computer-use battles is how the battle itself is produced and shown, and those rules are part of the methodology because they decide what a vote means.

Live sites are the game. Both contestants drive a real browser through the same task on the live web — not a replay, not a sandbox. Consequently the two runs are not deterministic replicas of each other: site latency, rate limits, A/B tests and what a page showed at that moment are real life. A slower model may legitimately see a different world. We accept this nondeterminism and state it here rather than hide it behind a sandbox.
Identical starting scene. A curated scene (a saved browser profile: cookies, logged-in throwaway accounts, open tabs) is snapshotted once by a curator and both contestants start from an identical copy. Scenes are versioned; a battle records which version it used. Public users never log into their own accounts inside our infrastructure; scenes use throwaway accounts only. Tasks that mutate account state give each contestant its own throwaway account cloned to the same scene, so contestants compete against the live world, not against each other's writes.
Concurrent, simultaneous start. The two runs are always executed concurrently, and a barrier releases control to both agents at the same tick. That tick is t=0 of both recordings and of every latency number below.
Equal resources, one scaffold. Each contestant gets pinned, equal CPU/memory; every model runs through ONE uniform scaffold — screenshot + accessibility tree in, one structured action (click/type/scroll/key/navigate/done/needs_human) out — with no vendor-native computer-use tools, or the leaderboard would measure scaffolds instead of models. Per-step harness overhead (harness_ms) is logged for every contestant and published in the dump's telemetry so that equal treatment is checkable, alongside the model's own latency (model_ms).
Wall-clock-faithful recordings. Judges see continuous video whose duration equals the real run: a model that thinks for nine seconds is seen waiting for nine seconds. The two recordings share t=0 and are scrubbed together; a run that finishes early holds its last frame. The pointer is not in the browser's frames, so cursor travel, clicks and typing are drawn from the harness's own timestamped event trace — the pointer animation is identical for every model (a constant, not a bias). Keystrokes into password/OTP fields are never recorded in clear (traces say [redacted, N chars]; frames are dropped while such a field has focus).
Hard budgets. Every run has the same wall-clock and step budget; exceeding either ends the run (budget), and the vote is on what was achieved within it.
The assisted rule. If a run required a human — the agent asked for help (needs_human) or an operator took over — the battle is marked assisted. This is disclosed to the judge only at reveal (after the vote), and assisted battles never enter the public dump or the ratings (DUMP_FORMAT.md exporter rule 8). Symmetric assistance does not rescue a battle in v1.
Latency on the analytics site. Rating-vs-latency charts use time_to_first_action_ms for computer-use categories (the closest analogue of time-to-first-token), where text categories use response duration.

Section 5

Performance and cost metrics

Definitions of every telemetry figure on the site, stated as the code computes them.

Every battle records telemetry for both contestants, whether or not its votes end up ratings-eligible: generation performance is a property of the run, not of the vote. Figures on model pages, charts and the compare view are aggregates of these per-response values: the median for latencies (response time for text, time to first action for computer use) and the mean for estimated cost per response.

Text categories

Time to first token

ttft_ms

Milliseconds from the moment the request is sent through our inference gateway to the arrival of the first streamed content delta from the model. It includes network time to the provider and any provider-side queueing. For categories whose battles are generated without streaming, no partial deltas exist and this value equals the full response time.

TTFT = t_{first delta} - t_{request}

Response time (duration)

duration_ms

Milliseconds from request to stream completion — the whole response, time to first token included. Both contestants in a battle are generated concurrently, so their durations are measured under the same conditions at the same moment.

Duration = t_{stream end} - t_{request}

Output speed

tokens_per_sec

Output tokens divided by the response time in seconds. Note the denominator is the full duration, time to first token included — a deliberately conservative choice. Benchmarks that measure decode speed only after the first token report higher numbers for the same run; the two are not directly comparable.

Output speed = \frac{{tokens}_{out} \times 1000}{{duration}_{ms}}

Stored with two decimals; undefined (null) when the token count is missing or the duration is zero.

Tokens

tokens_in · tokens_out

Native tokens as reported by the provider in the request's usage data (prompt and completion tokens). We do not re-tokenize outputs with a common tokenizer, so token figures are comparable across runs of the same model, and only approximately across models with different tokenizers. When a provider stream carries no usage data, counts are estimated at roughly four characters per token; such estimates are rare and are not distinguished in aggregates.

Estimated cost

est_cost_usd

What the two responses of a battle would have cost at the list prices recorded on our roster for the model at the time of the battle — input and output prices per million tokens, as shown on each model page. Cached-input or batch discounts are not modeled. Because cost reflects actual token usage, a model that writes longer answers costs more per response even at identical per-token prices.

Cost = \frac{{tokens}_{in} \cdot P_{in} + {tokens}_{out} \cdot P_{out}}{10^{6}}

P_in and P_out are the roster's USD prices per million tokens; stored with six decimals; null when the roster has no price for the model.

Computer-use categories

Time to first action

time_to_first_action_ms

Milliseconds from the shared start tick of the battle — the barrier that releases both agents at the same instant — to the contestant's first dispatched browser action. It is the computer-use analogue of time to first token, and it is what the rating-vs-latency chart uses for computer-use categories.

Time to first action = t_{first action} - t_{0}

Run duration

duration_ms

Wall-clock length of the run from the shared start tick to its end — the agent declaring completion, asking for a human, or exhausting its budget. The recording a judge watches has exactly this duration; nothing is sped up or cut.

Steps

steps

Number of observation → action cycles the agent performed. Each cycle presents a screenshot and an accessibility tree and receives exactly one structured action.

Model time and harness time

model_ms_total · harness_ms_total · median_step_ms

Every step is split into the time spent waiting for the model's answer and the time the harness spent observing the page and executing the action. The harness is identical for every contestant; publishing its share per run is how equal treatment can be checked from the data. The median step latency is the median over steps of the two added together.

{step}_{k} = {model}_{k} + {harness}_{k}

Budgets

budget

Every run in a battle has the same step budget and the same wall-clock budget (at the time of writing, 40 steps and 10 minutes). Exceeding either ends the run, and the vote is on what was achieved within the budget. Both budgets are recorded in the battle's job record.

Section 6

Reproducibility and versioning

Same inputs, same command, same pinned environment: a byte-identical leaderboard.

The official numbers are produced by running the open humaneval-ratings package on the nightly public dump with seed 42 — no database, no network, no hidden inputs. Every leaderboard.json records the seed, bootstrap round count, thresholds, package version and the SHA-256 digests of its exact input files, and contains no compute-time timestamps, so the output is a pure function of (inputs, parameters, pinned environment).

curl -O https://humaneval.org/downloads/files/latest/battles.jsonl
curl -O https://humaneval.org/downloads/files/latest/votes.jsonl
curl -O https://humaneval.org/downloads/files/latest/manifest.json
sha256sum battles.jsonl votes.jsonl        # compare with manifest.json

humaneval-ratings compute \
    --battles battles.jsonl --votes votes.jsonl \
    --out leaderboard.json --seed 42
sha256sum leaderboard.json                 # compare with the published snapshot

The complete walkthrough, including the pinned environment, is on the downloads page together with the canonical dump format specification. The public API serves the same snapshot values without recomputation.

Methodology version history

Every choice in the rating mathematics is deliberate and fixed. Changing any of it is a methodology version change, announced in the changelog and recorded here.

Date	Versions	Change
2026-08-23	engine 1.0.0 · dump v1	Initial methodology: weighted Bradley-Terry maximum likelihood, ridge regularization, case-bootstrap confidence intervals, style control, rank bands, byte-deterministic output.
2026-09-08	engine 1.1.0 · dump v2	Computer-use battles: recording modality in the dump, zero style covariates for recordings (§7.5), the computer-use protocol and assisted rule (§10). Numbers for v1 dumps are byte-identical to 1.0.0.

Discrepancies

A difference between a published number and an independent run is treated as a bug in the engine or the export, never as a matter of interpretation. A useful report states the dump date and the two input digests from manifest.json, the exact command, the digest of the resulting leaderboard.json, and whether the pinned environment was used. Fixes are announced in the changelog.

Citing this methodology

@misc{humaneval_methodology,
  title        = {HumanEval.org Benchmarking Methodology},
  author       = {{HumanEval.org}},
  year         = {2026},
  howpublished = {\url{https://humaneval.org/methodology}},
  note         = {humaneval-ratings 1.1.0; dump schema v2}
}

Section 7

Definitions

Terms as used on this site and in the public data.

Model: A specific AI model version listed on /models under a stable public slug. Proprietary and open-weights models are rated alike. During a battle a model's identity is visible neither to the harness that runs it nor to the judge.
Model creator: The organization that trained the model, shown as the provider on model pages. All models are reached through one inference gateway under identical settings, so hosting differences are not part of what we rate.
Open weights: A model whose weights have been published by its creator (flagged on model pages). We say “open weights” rather than “open source” because many releases carry licenses that do not meet the open-source definition.
Category: An independent competition with its own task set, harness, display mode and leaderboard. Ratings are anchored per category and are not comparable across categories.
Task set / prompt: The curated prompts (or, for computer use, scenes) a category's battles are drawn from. Task sets are versioned and exactly one version is active at a time. Prompts typed by users in the arena are allowed for exploration but never count toward ratings.
Scene: Computer use only: a versioned, snapshotted browser starting state (profile, cookies, throwaway logins, open tabs) together with a task. Both contestants start from an identical copy; the battle records which version it used.
Battle: One pairwise comparison: two distinct models, one prompt, identical generation settings, blind side-by-side presentation, and one or more votes.
Slot (A / B): The two display positions. A random seed persisted with the battle decides which model is A, so position carries no information. Public dumps list models by slot after that randomization.
Normalization mode: How outputs are shown to judges: rendered (Markdown rendered to HTML), raw (verbatim text), or noformat (Markdown formatting stripped). Each category has a default; the mode actually used is recorded per battle and exported.
Judge (voter): An authenticated human casting a vote. In the arena the judge is the battle's creator; in the judging queue any other signed-in user may judge, never their own battles.
Vote: win A, win B or tie — one per user per battle. A vote copies its battle's eligibility flags at the moment it is cast. Judges earn credits for voting; the amount never depends on which way they voted.
Provisional window: New accounts vote inside a provisional window (a minimum account age and vote count). Those votes are collected but never enter ratings or public dumps.
Effective weight: The per-vote weight published in the dump (0 < w ≤ 1; 1.0 is a full vote). It is set by the hidden vote-quality pipeline; votes below the export floor (currently 0.2) are excluded entirely rather than down-weighted.
Assisted: A computer-use run in which a human intervened (the agent asked for help, or an operator took over). Disclosed to the judge only after the vote; assisted battles never enter the dump or the ratings.
Rating: A model's Bradley-Terry strength on an Elo-equivalent scale: the mean of a category's established models is anchored at 1000, and a 400-point gap means 10:1 expected odds. Raw and style-controlled ratings are both always published.
Confidence interval (CI): The 95% percentile interval of a rating under a case bootstrap on votes (100 resamples). Every rating on the site is displayed with its interval.
Rank band: When adjacent models' intervals overlap, the data cannot order them and they share a rank (competition numbering: after two models at #1 the next distinct band is #3). Bands are computed from the published, rounded values so they are verifiable from leaderboard.json alone.
Provisional model: A model with fewer than 30 eligible votes in a category. It is rated, has an interval and a sort position, but is shown unranked below the board.
Style control: An extended fit with shared covariates for presentation (response length, Markdown density, list count, header count). The style-controlled rating estimates strength with presentation differences held constant.
Snapshot: One official run of the rating engine on one dump — the numbers the site and the public API serve. Rating changes between consecutive snapshots are the “movers” on the homepage and model pages.
Dump: The nightly public export (battles.jsonl + votes.jsonl with a SHA-256 manifest) on the downloads page — exactly the inputs of the official snapshot, nothing more and nothing less.
Tokens: Native tokens as counted by the provider for a request. We do not normalize to a common tokenizer; token-based figures compare exactly within a model and only approximately across models.

Formulas on this page are set in MathML; the canonical text keeps its plain-text notation so that it is identical in the shipped package and here. Where the two could ever disagree, the canonical text governs. Conflict-of-interest and funding disclosure: /about/funding.