HumanEval.org public data dump format — v2

This document is the canonical, normative specification of the HumanEval.org public data dump and of the leaderboard.json file the humaneval-ratings package produces from it. The platform's nightly export is built to match this spec; the official leaderboard is produced by running this package on the published dump, so anyone can reproduce the official numbers byte-for-byte (see README.md).

Version note. Dumps published from 2026-09-08 on carry schema_version: 2 (section 3.1 adds recording battles for the computer-use category). v2 is a superset of v1: every v1 record is a valid v2 text record. The engine (humaneval-ratings ≥ 1.1.0) reads both versions; v1-only readers must be updated before consuming new dumps (the header version is exactly what they check).

The rating mathematics applied to a dump is specified in METHODOLOGY.md.

1. General conventions

• A dump consists of exactly two files: battles.jsonl and votes.jsonl.
• Encoding: UTF-8, Unix line endings (LF), one JSON object per line, no blank lines, file ends with a trailing LF.
• Header line: the first line of each file is a metadata object (section 2). All subsequent lines are data records.
• IDs are JSON strings (battle_id, vote_id, voter_id). The platform's internal keys are 64-bit integers which can exceed JavaScript's 2^53 safe-integer range; strings keep IDs safe and opaque. IDs are unique within their file and stable across dumps.
• Models and categories are identified by their public slugs (models.slug, categories.slug on the platform side), never by internal numeric ids.
• Timestamps are ISO 8601 UTC with a trailing Z, second precision: 2026-08-22T19:31:04Z.
• Forward compatibility: readers MUST ignore unknown object fields. Additive changes (new optional fields) do not bump schema_version; breaking changes do.
• Privacy: battles and responses are public content. No user data is ever exported; voter_id is a pseudonymous opaque token (see 4.1).

2. Header lines

First line of battles.jsonl:

{"kind": "battles", "schema_version": 2, "generated_at": "2026-09-08T04:00:00Z", "categories": ["browser-use", "chat-writing", "chat-coding"]}

First line of votes.jsonl:

{"kind": "votes", "schema_version": 2, "generated_at": "2026-09-08T04:00:00Z"}

Headers may carry extra fields (e.g. a source URL); readers ignore them.

3. battles.jsonl — data records

One record per voted battle. Battles without eligible votes MAY be included (the engine ignores them) but the export SHOULD omit them.

{"battle_id": "1024", "category_slug": "chat-writing", "model_a": "gpt-alpha", "model_b": "claude-beta", "created_at": "2026-08-22T19:31:04Z", "normalization_mode": "rendered", "modality": "text", "response_a": "# Draft\n\nDear team, ...", "response_b": "Dear team, ..."}

Notes for the platform exporter:

• Response bodies live in object storage as text artifacts (artifacts.storage_ref); the exporter MUST resolve and inline the raw text (the stored artifact content, not the rendered HTML) into response_a/response_b. Style covariates are computed from these texts (METHODOLOGY.md §7), so their exact bytes matter.
• v1 exported text-modality battles only. v2 adds the recording modality below; any further artifact modality is another bump.

3.1 recording battles (v2 — computer-use category)

Computer-use ("Browser Use") battles have no response text: each contestant drove a real browser through the same task on live websites, concurrently from an identical starting scene, and voters judged wall-clock-faithful recordings side by side (METHODOLOGY.md §10). A recording battle is exported as:

{"battle_id": "2048", "category_slug": "browser-use", "model_a": "gpt-alpha", "model_b": "claude-beta", "created_at": "2026-09-08T10:00:00Z", "normalization_mode": "raw", "modality": "recording", "recording_a": {"video_key": "battles/2048/a.mp4", "trace_key": "battles/2048/a.trace.jsonl", "screenshot_key": "battles/2048/a.png", "assisted": false}, "recording_b": {"video_key": "battles/2048/b.mp4", "trace_key": "battles/2048/b.trace.jsonl", "screenshot_key": null, "assisted": false}, "telemetry_a": {"duration_ms": 184000, "time_to_first_action_ms": 6200, "steps": 14, "median_step_ms": 9100, "model_ms_total": 121000, "harness_ms_total": 4900}, "telemetry_b": {"duration_ms": 152000, "time_to_first_action_ms": 4100, "steps": 11, "median_step_ms": 8700, "model_ms_total": 88000, "harness_ms_total": 3900}}

Object keys are relative to the platform's artifacts bucket; the media themselves are large binary objects and are not part of the dump (they are served through the platform, not published in bulk). The rating engine does not read them: it uses model_a/model_b/ category_slug and the votes exactly as for text battles, and treats the style covariates of a recording battle as zero (METHODOLOGY.md §7.5).

4. votes.jsonl — data records

One record per ratings-eligible vote.

{"vote_id": "5001", "battle_id": "1024", "voter_id": "v_9f2c7a1e", "choice": "win_a", "created_at": "2026-08-22T19:32:40Z", "weight": 1.0}

4.1 Voter pseudonyms

voter_id is an opaque token derived by the platform (e.g. a keyed hash of the internal user id). It is stable within a dump (and across dumps, so long-term voter behavior is analyzable) but cannot be linked back to an account. The derivation is internal and never published.

4.2 Vote weights and the quality pipeline

The platform runs a hidden vote-quality pipeline (consensus agreement, gold-standard checks, behavioral signals, provisional-account windows). That pipeline runs before export:

• Votes it excludes never appear in the dump.
• Votes it down-weights appear with their reduced effective weight.
• Full-quality votes carry weight: 1.0.

The dump therefore contains exactly the votes that count, each with the weight it counts at. Ratings computed from the dump MUST use these weights (METHODOLOGY.md §3). How weights are derived is deliberately unpublished (anti-gaming); that they are applied, and their values, are fully public here. The exporter also guarantees at most one vote per (battle, voter) pair; the engine does not re-check this.

5. Validation rules (normative for the engine)

The engine rejects a dump (non-zero exit, no output) when:

1. A header line is missing, has the wrong kind, or an unsupported schema_version.
2. Any required field is missing or has the wrong JSON type (for v2 recording battles: recording_a/recording_b missing or not objects, or a response_* text present).
3. choice, normalization_mode or (v2) modality has an unknown value.
4. weight is not a finite number > 0.
5. A battle_id (in battles) or vote_id (in votes) is duplicated.
6. A vote references a battle_id not present in battles.jsonl.
7. model_a == model_b in any battle.
8. A battle's category_slug is absent from the header categories list.

Unknown fields are ignored everywhere. Battles with zero votes are ignored. Categories with zero votes produce no leaderboard section. Models enter a category's leaderboard only via voted battles there.

6. Output: leaderboard.json

Produced by humaneval-ratings compute. Top-level shape:

{
  "metadata": {
    "package": "humaneval-ratings",
    "package_version": "1.0.0",
    "leaderboard_schema_version": 1,
    "dump_schema_version": 1,
    "dump_generated_at": "2026-08-23T04:00:00Z",
    "seed": 42,
    "bootstrap_rounds": 100,
    "min_votes": 30,
    "style_min_votes": 50,
    "input_digests": {
      "battles_sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
      "votes_sha256": "60303ae22b998861bce3b28f33eec1be758a213c86c93c076dbe9f558c11c752"
    }
  },
  "categories": [
    {
      "category": "chat-writing",
      "vote_count": 8231,
      "style_control": true,
      "style_coefficients": {
        "length_chars": 24.1830,
        "markdown_density": 6.0021,
        "list_count": 3.1187,
        "header_count": -0.4402
      },
      "entries": [
        {
          "model": "claude-beta",
          "rating": 1041.2211,
          "ci_low": 1027.0110,
          "ci_high": 1055.9024,
          "rating_style_controlled": 1030.5470,
          "ci_low_sc": 1016.2001,
          "ci_high_sc": 1046.0193,
          "vote_count": 4110,
          "provisional": false,
          "rank": 1,
          "order": 1
        }
      ]
    }
  ]
}

6.1 Metadata

Every knob that affects the numbers is recorded: the seed, bootstrap round count, thresholds, package version, and the SHA-256 digests of the exact input files. There are no timestamps generated at compute time — dump_generated_at is echoed from the battles-file header (dump_schema_version likewise) — so the output is a pure function of (inputs, CLI parameters, pinned environment). Verifiers check both digests, run the same command, and compare sha256(leaderboard.json).

6.2 Category objects

Sorted by category slug (ascending, bytewise). Fields:

6.3 Entry fields

Sorting and rank bands are computed from the rounded, published values (see 6.4), so the ordering and bands are verifiable from leaderboard.json alone: entries sort by rating descending, ties broken by model slug ascending; order is that position. Rank bands use the published ci_low/ci_high (METHODOLOGY.md §6).

6.4 Determinism and precision policy (normative)

• Every floating-point value in leaderboard.json is rounded to 4 decimal places (round-half-even, Python round(x, 4); matches the platform's numeric(10,4) snapshot columns) and serialized with exactly four decimal digits (-?d+.dddd).
• Object keys are sorted (bytewise ascending); output is ASCII-only (non-ASCII escaped as in JSON \uXXXX); separators are ", " / ": " with 2-space indent; LF newlines; single trailing LF.
• The RNG and every iteration order in the engine are deterministic (METHODOLOGY.md §8). Same input files + same CLI parameters + the pinned environment (requirements-lock.txt) → byte-identical output.

7. Worked example

A minimal, valid dump (v1 headers — still accepted; a v2 dump would say "schema_version": 2 and may add "modality": "text"). battles.jsonl:

{"kind": "battles", "schema_version": 1, "generated_at": "2026-08-23T04:00:00Z", "categories": ["chat-writing"]}
{"battle_id": "1", "category_slug": "chat-writing", "model_a": "gpt-alpha", "model_b": "claude-beta", "created_at": "2026-08-22T10:00:00Z", "normalization_mode": "rendered", "response_a": "## Plan\n\n- step one\n- step two", "response_b": "Do step one, then step two."}
{"battle_id": "2", "category_slug": "chat-writing", "model_a": "claude-beta", "model_b": "gpt-alpha", "created_at": "2026-08-22T11:00:00Z", "normalization_mode": "raw", "response_a": "Short answer.", "response_b": "A considerably longer answer with more detail."}

votes.jsonl:

{"kind": "votes", "schema_version": 1, "generated_at": "2026-08-23T04:00:00Z"}
{"vote_id": "10", "battle_id": "1", "voter_id": "v_a1", "choice": "win_a", "created_at": "2026-08-22T10:05:00Z", "weight": 1.0}
{"vote_id": "11", "battle_id": "1", "voter_id": "v_b2", "choice": "tie", "created_at": "2026-08-22T10:06:00Z", "weight": 0.5}
{"vote_id": "12", "battle_id": "2", "voter_id": "v_a1", "choice": "win_b", "created_at": "2026-08-22T11:09:00Z", "weight": 1.0}

Reading it: battle 1 was won by its slot-A model (gpt-alpha) at full weight, plus a half-weight tie (which counts as half a win for each side, METHODOLOGY.md §2). Battle 2 was won by its slot-B model (gpt-alpha again — note the slots swapped). Running

humaneval-ratings compute --battles battles.jsonl --votes votes.jsonl \
    --out leaderboard.json --seed 42

yields a chat-writing leaderboard where both models are provisional (3 votes < the default --min-votes 30) with rank: null, gpt-alpha at order 1, and style_control: false (3 votes < the default --style-min-votes 50), so the SC fields mirror the raw ones.

8. Version history

• v1 (2026-08-23): initial spec — battles + votes JSONL with header lines; leaderboard schema v1.
• v2 (2026-09-08, humaneval-ratings 1.1.0 / humaneval-exporter 1.1.0): battle records gain modality; new recording modality (section 3.1) with per-slot recording_* object references, assisted flags and telemetry_* for computer-use battles, no response_* texts. Text records are unchanged apart from modality: "text". Exporter rule 8: assisted battles are excluded. leaderboard.json schema stays 1 (only metadata.package_version and dump_schema_version change); the engine's numbers for v1 dumps are byte-identical to 1.0.0.