# Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources.
#
# Canonical fields (see aiatlas.ontology.benchmarks):
#   family            benchmark family key (swe-bench, gpqa, livebench…); `variant` names the member, `version` the dataset revision when one exists
#   family_head       true on the representative entry of a family — every other member gets a `variant_of` relation to it (aia seed)
#   metric            canonical metric key from ontology.benchmarks.METRICS (accuracy, pass@1, resolved, elo, index…); `metric_label` = human label
#   higher_is_better  score direction · metric_min / metric_max = bounds of the reported metric (null = unbounded)
#   harness           who runs the evaluation / with what scaffold; comparability_note = what makes two scores on this board (not) comparable
#   aliases           labels used by Artificial Analysis, LiveBench, model cards, papers… (the API resolves benchmarks by alias)
benchmarks:
  - key: mmlu
    name: MMLU
    aliases: [Massive Multitask Language Understanding, MMLU 5-shot, MMLU (5-shot)]
    family: mmlu
    variant: original
    family_head: true
    category: knowledge
    task: 57-subject multiple-choice questions
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs (shot count and prompt format vary)
    comparability_note: Widespread training-data contamination; 0-shot vs 5-shot and chain-of-thought prompting change scores by several points.
    creator: Hendrycks et al.
    website: https://github.com/hendrycks/test
    paper: https://arxiv.org/abs/2009.03300
    known_limitations: Widespread training-data contamination; saturated at the frontier.
    source_url: https://arxiv.org/abs/2009.03300
  - key: mmlu-pro
    name: MMLU-Pro
    aliases: [MMLU Pro, MMLU-Pro (0-shot CoT)]
    family: mmlu
    variant: Pro
    category: knowledge
    task: harder, 10-option MMLU variant
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs or independent evaluators
    comparability_note: 10-option questions; chain-of-thought is the standard regime, direct-answer scores are lower and not comparable.
    website: https://github.com/TIGER-AI-Lab/MMLU-Pro
    paper: https://arxiv.org/abs/2406.01574
    source_url: https://arxiv.org/abs/2406.01574
  - key: gpqa
    name: GPQA
    aliases: [GPQA main, Graduate-Level Google-Proof Q&A]
    family: gpqa
    variant: main
    family_head: true
    category: reasoning
    task: graduate-level science questions (main set, 448 questions)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs or independent evaluators
    comparability_note: Main and Diamond subsets are different question sets; reasoning effort and sampling (pass@1 vs majority vote) change scores.
    website: https://github.com/idavidrein/gpqa
    paper: https://arxiv.org/abs/2311.12022
    source_url: https://arxiv.org/abs/2311.12022
  - key: gpqa-diamond
    name: GPQA Diamond
    aliases: [GPQA-Diamond, gpqa_diamond, GPQA (Diamond), GPQA Diamond (0-shot)]
    family: gpqa
    variant: Diamond
    category: reasoning
    task: graduate-level science questions — the 198-question Diamond subset (expert-validated, non-expert-failed)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling
    comparability_note: 198 questions → ±3 pt noise; reasoning effort, thinking budget and repeated sampling strongly affect scores.
    website: https://github.com/idavidrein/gpqa
    paper: https://arxiv.org/abs/2311.12022
    source_url: https://arxiv.org/abs/2311.12022
  - key: humaneval
    name: HumanEval
    aliases: [HumanEval pass@1, OpenAI HumanEval]
    family: humaneval
    variant: original
    family_head: true
    category: coding
    task: Python function synthesis from docstrings
    metric: pass@1
    metric_label: pass@1
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs
    comparability_note: Saturated; pass@1 vs pass@k and prompt wrapping differ across reporters.
    website: https://github.com/openai/human-eval
    paper: https://arxiv.org/abs/2107.03374
    known_limitations: Saturated; small (164 problems).
    source_url: https://arxiv.org/abs/2107.03374
  - key: swe-bench-verified
    name: SWE-bench Verified
    aliases: [SWE-bench, SWE-Bench Verified, SWE-bench-Verified, swebench verified, SWE Bench Verified]
    family: swe-bench
    variant: Verified
    category: coding
    task: resolve real GitHub issues (500 human-validated instances)
    metric: resolved
    metric_label: "% resolved"
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness, submissions optionally checked by the SWE-bench team
    comparability_note: Scores depend on the agent scaffold as much as on the model; compare only within one scaffold (config.system) and one attempt regime.
    website: https://www.swebench.com
    paper: https://arxiv.org/abs/2310.06770
    known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.
    source_url: https://www.swebench.com/
  - key: aider-polyglot
    name: Aider polyglot
    aliases: [Aider polyglot coding leaderboard, Aider Polyglot, aider polyglot benchmark, Aider]
    family: aider-polyglot
    variant: polyglot
    family_head: true
    category: coding
    task: 225 Exercism exercises in 6 languages, edit-format aware
    metric: pass_rate_2
    metric_label: pass rate (2 attempts)
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: aider (config.edit_format = whole | diff | diff-fenced | architect), second attempt after test feedback
    comparability_note: Edit format and aider version change results; the well-formed rate is a separate benchmark entry.
    website: https://aider.chat/docs/leaderboards/
    known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing.
    source_url: https://aider.chat/docs/leaderboards/
  - key: aider-polyglot-well-formed
    name: Aider polyglot — well-formed responses
    aliases: [Aider well-formed rate, percent_cases_well_formed, Aider polyglot well-formed]
    family: aider-polyglot
    variant: well-formed
    category: coding
    task: share of aider polyglot cases where every edit was syntactically well formed (parsable edit blocks)
    metric: percent_cases_well_formed
    metric_label: "% cases well formed"
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: aider (config.edit_format), same runs as aider-polyglot
    comparability_note: Measures edit-format compliance, not correctness; only comparable within one edit format.
    website: https://aider.chat/docs/leaderboards/
    source_url: https://aider.chat/docs/leaderboards/
  - key: livebench
    name: LiveBench
    aliases: [LiveBench global average, LiveBench Global, LiveBench overall]
    family: livebench
    variant: global
    family_head: true
    category: general
    task: contamination-limited, monthly refreshed questions across 7 categories — overall mean of the category averages
    metric: global_average
    metric_label: global average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs; config.release = question-set release date
    comparability_note: Each release is a new question set — compare scores within one release (run group) only; effort variants are configurations.
    website: https://livebench.ai
    paper: https://arxiv.org/abs/2406.19314
    source_url: https://livebench.ai/
  - key: livebench-reasoning
    name: LiveBench Reasoning
    aliases: [LiveBench — Reasoning, LiveBench reasoning average]
    family: livebench
    variant: Reasoning
    category: reasoning
    task: LiveBench reasoning category — mean of its subtasks (web of lies, zebra puzzles, spatial…)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-coding
    name: LiveBench Coding
    aliases: [LiveBench — Coding, LiveBench coding average]
    family: livebench
    variant: Coding
    category: coding
    task: LiveBench coding category — mean of its subtasks (LCB generation, code completion)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-agentic-coding
    name: LiveBench Agentic Coding
    aliases: [LiveBench — Agentic Coding, LiveBench agentic coding average]
    family: livebench
    variant: Agentic Coding
    category: coding
    task: LiveBench agentic coding category — mean of its subtasks
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-mathematics
    name: LiveBench Mathematics
    aliases: [LiveBench — Mathematics, LiveBench math average, LiveBench Math]
    family: livebench
    variant: Mathematics
    category: math
    task: LiveBench mathematics category — mean of its subtasks (competition math, AMPS hard, olympiad)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-data-analysis
    name: LiveBench Data Analysis
    aliases: [LiveBench — Data Analysis, LiveBench data analysis average]
    family: livebench
    variant: Data Analysis
    category: general
    task: LiveBench data analysis category — mean of its subtasks (table reformatting, join, column type)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-language
    name: LiveBench Language
    aliases: [LiveBench — Language, LiveBench language average]
    family: livebench
    variant: Language
    category: general
    task: LiveBench language category — mean of its subtasks (typos, connections, plot unscrambling)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: livebench-if
    name: LiveBench Instruction Following
    aliases: [LiveBench — IF, LiveBench IF, LiveBench instruction following average]
    family: livebench
    variant: IF
    category: instruction-following
    task: LiveBench instruction-following category — mean of its subtasks (paraphrase, simplify, story generation, summarize)
    metric: average score
    metric_label: category average
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: LiveBench official runs
    comparability_note: Compare within one release only; subtask mix changes between releases.
    website: https://livebench.ai
    source_url: https://livebench.ai/
  - key: math-500
    name: MATH-500
    aliases: [MATH, MATH 500, MATH500]
    family: math
    variant: "500"
    family_head: true
    category: math
    task: competition mathematics (500-problem subset of MATH used since the PRM800K paper)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs or independent evaluators
    comparability_note: Saturated at the frontier; sampling regime (pass@1 vs majority vote) must match.
    paper: https://arxiv.org/abs/2103.03874
    source_url: https://arxiv.org/abs/2103.03874
  - key: aime-2024
    name: AIME 2024
    aliases: [AIME24, AIME 24, AIME I/II 2024]
    family: aime
    variant: "2024"
    version: "2024"
    category: math
    task: American Invitational Mathematics Examination 2024 problems (30 problems, I + II)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs or independent evaluators
    comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting (cons@64) are not comparable; 2024 problems appear in training data of later models.
    source_url: https://maa.org/maa-invitational-competitions/
  - key: aime-2025
    name: AIME 2025
    aliases: [AIME, AIME25, AIME 25, AIME I/II 2025]
    family: aime
    variant: "2025"
    version: "2025"
    family_head: true
    category: math
    task: American Invitational Mathematics Examination 2025 problems (30 problems, I + II)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling
    comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting are not comparable; reasoning effort matters.
    known_limitations: 30 problems per year; high variance; often reported with majority voting.
    source_url: https://maa.org/maa-invitational-competitions/
  - key: arc-agi
    name: ARC-AGI
    aliases: [ARC-AGI-1, ARC AGI 1, Abstraction and Reasoning Corpus, ARC-AGI v1]
    family: arc-agi
    variant: "1"
    version: "1"
    family_head: true
    category: reasoning
    task: novel visual abstraction puzzles (ARC-AGI-1 semi-private / public evaluation sets)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs
    comparability_note: Public, semi-private and private evaluation sets differ; the ARC Prize reports cost per task alongside the score.
    website: https://arcprize.org
    source_url: https://arcprize.org/
  - key: arc-agi-2
    name: ARC-AGI-2
    aliases: [ARC AGI 2, ARC-AGI v2, ARC-AGI 2]
    family: arc-agi
    variant: "2"
    version: "2"
    category: reasoning
    task: harder abstraction puzzles released in 2025 (ARC-AGI-2), designed to resist brute-force program search
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs
    comparability_note: Not comparable with ARC-AGI-1 scores; evaluation set (public / semi-private) and cost budget must match.
    website: https://arcprize.org/arc-agi/2/
    source_url: https://arcprize.org/arc-agi/2/
  - key: humanitys-last-exam
    name: Humanity's Last Exam
    aliases: [HLE, Humanity’s Last Exam, HLE (no tools)]
    family: humanitys-last-exam
    variant: full
    family_head: true
    category: knowledge
    task: expert-written frontier questions
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Artificial Analysis and labs; text-only subset vs full multimodal set, with or without tools
    comparability_note: Tool use (search) and the text-only subset change scores substantially; check config before comparing.
    website: https://lastexam.ai
    paper: https://arxiv.org/abs/2501.14249
    source_url: https://lastexam.ai/
  - key: lmarena-text
    name: LMArena text leaderboard
    aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo, LMArena, Arena Score]
    family: lmarena
    variant: text
    family_head: true
    category: preference
    task: crowdsourced pairwise human preference
    metric: elo
    metric_label: Elo / Bradley–Terry score
    unit: ""
    metric_min: 0
    metric_max: null
    higher_is_better: true
    harness: LMArena live human votes (Bradley–Terry model)
    comparability_note: Relative score that drifts as the model pool changes; only a snapshot of the same date is comparable; style control changes rankings.
    website: https://lmarena.ai
    known_limitations: Style bias; sampling of prompts by users.
    source_url: https://lmarena.ai/
  - key: mmmu
    name: MMMU
    aliases: [MMMU (val), Massive Multi-discipline Multimodal Understanding]
    family: mmmu
    variant: original
    family_head: true
    category: multimodal
    task: college-level multimodal understanding
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs (validation split)
    comparability_note: Validation vs test split and chain-of-thought prompting differ across reporters.
    website: https://mmmu-benchmark.github.io
    paper: https://arxiv.org/abs/2311.16502
    source_url: https://arxiv.org/abs/2311.16502
  - key: tau-bench
    name: τ-bench
    aliases: [tau-bench, TAU-bench, tau bench, τ-Bench]
    family: tau-bench
    variant: v1
    family_head: true
    category: agentic
    task: tool-agent-user interaction in retail/airline domains
    metric: pass^1
    metric_label: pass^1
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: simulated user (LLM) + tool environment; pass^k over k trials
    comparability_note: Domain (retail / airline), pass^k regime and the user-simulator model must match.
    paper: https://arxiv.org/abs/2406.12045
    source_url: https://arxiv.org/abs/2406.12045
  - key: terminal-bench
    name: Terminal-Bench
    aliases: [TerminalBench, Terminal Bench, Terminal-Bench 1.0]
    family: terminal-bench
    variant: "1.0"
    version: "1.0"
    family_head: true
    category: agentic
    task: terminal tasks solved by agents
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Terminus agent (official) or the submitter's agent; Artificial Analysis reports its own harness versions (config.variant)
    comparability_note: Agent scaffold and harness version change results; config.variant (v2.1 / v4.0 / hard) must match.
    website: https://www.tbench.ai
    source_url: https://www.tbench.ai/
  - key: terminal-bench-2
    name: Terminal-Bench 2.0
    aliases: [Terminal Bench 2, TerminalBench 2.0, Terminal-Bench 2, TB2]
    family: terminal-bench
    variant: "2.0"
    version: "2.0"
    category: agentic
    task: revised, harder set of terminal tasks (Terminal-Bench 2.0)
    metric: accuracy
    metric_label: accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Terminus 2 agent (official) or the submitter's agent
    comparability_note: Not comparable with Terminal-Bench 1.0; agent scaffold must match.
    website: https://www.tbench.ai
    source_url: https://www.tbench.ai/
  - key: ifeval
    name: IFEval
    aliases: [IF-Eval, Instruction-Following Eval, IFEval strict]
    family: ifeval
    variant: original
    family_head: true
    category: instruction-following
    task: verifiable instruction following
    metric: accuracy
    metric_label: prompt-level strict accuracy
    unit: "%"
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: self-reported by labs
    comparability_note: Prompt-level vs instruction-level and strict vs loose accuracy are four different numbers.
    paper: https://arxiv.org/abs/2311.07911
    source_url: https://arxiv.org/abs/2311.07911
  - key: mteb
    name: MTEB
    aliases: [Massive Text Embedding Benchmark, MTEB (English), MTEB v1]
    family: mteb
    variant: v1
    family_head: true
    category: embeddings
    task: embedding tasks across retrieval, classification, clustering…
    metric: mean score
    metric_label: mean score
    unit: ""
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: MTEB library, official leaderboard
    comparability_note: Task set (English v1, v2, multilingual) and aggregation (mean of tasks vs mean of task types) must match.
    website: https://huggingface.co/spaces/mteb/leaderboard
    paper: https://arxiv.org/abs/2210.07316
    source_url: https://arxiv.org/abs/2210.07316
  - key: mmteb
    name: MMTEB
    aliases: [Massive Multilingual Text Embedding Benchmark, MTEB multilingual, MTEB (Multilingual)]
    family: mteb
    variant: MMTEB
    category: embeddings
    task: multilingual extension of MTEB (500+ tasks across 250+ languages)
    metric: mean score
    metric_label: mean score
    unit: ""
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: MTEB library, official leaderboard
    comparability_note: Not comparable with English-only MTEB; the leaderboard's Borda rank and task-type mean are different aggregations.
    website: https://huggingface.co/spaces/mteb/leaderboard
    paper: https://arxiv.org/abs/2502.13595
    source_url: https://arxiv.org/abs/2502.13595
  - key: artificial-analysis-intelligence-index
    name: Artificial Analysis Intelligence Index
    aliases: [AA Intelligence Index, Intelligence Index, Artificial Analysis Index]
    family: artificial-analysis-intelligence-index
    variant: index
    family_head: true
    category: composite
    task: composite of several evaluations run by Artificial Analysis
    metric: index
    metric_label: index
    unit: ""
    metric_min: 0
    metric_max: 100
    higher_is_better: true
    harness: Artificial Analysis internal runs; config.version = index methodology version
    comparability_note: Methodology versions change the component set — compare within one version (run group) only; reasoning-effort variants are configurations.
    website: https://artificialanalysis.ai
    known_limitations: Proprietary composite; methodology versions change.
    source_url: https://artificialanalysis.ai/methodology