HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources.2#3# Canonical fields (see aiatlas.ontology.benchmarks):4# family benchmark family key (swe-bench, gpqa, livebench…); `variant` names the member, `version` the dataset revision when one exists5# family_head true on the representative entry of a family — every other member gets a `variant_of` relation to it (aia seed)6# metric canonical metric key from ontology.benchmarks.METRICS (accuracy, pass@1, resolved, elo, index…); `metric_label` = human label7# higher_is_better score direction · metric_min / metric_max = bounds of the reported metric (null = unbounded)8# harness who runs the evaluation / with what scaffold; comparability_note = what makes two scores on this board (not) comparable9# aliases labels used by Artificial Analysis, LiveBench, model cards, papers… (the API resolves benchmarks by alias)10benchmarks:11 - key: mmlu12 name: MMLU13 aliases: [Massive Multitask Language Understanding, MMLU 5-shot, MMLU (5-shot)]14 family: mmlu15 variant: original16 family_head: true17 category: knowledge18 task: 57-subject multiple-choice questions19 metric: accuracy20 metric_label: accuracy21 unit: "%"22 metric_min: 023 metric_max: 10024 higher_is_better: true25 harness: self-reported by labs (shot count and prompt format vary)26 comparability_note: Widespread training-data contamination; 0-shot vs 5-shot and chain-of-thought prompting change scores by several points.27 creator: Hendrycks et al.28 website: https://github.com/hendrycks/test29 paper: https://arxiv.org/abs/2009.0330030 known_limitations: Widespread training-data contamination; saturated at the frontier.31 source_url: https://arxiv.org/abs/2009.0330032 - key: mmlu-pro33 name: MMLU-Pro34 aliases: [MMLU Pro, MMLU-Pro (0-shot CoT)]35 family: mmlu36 variant: Pro37 category: knowledge38 task: harder, 10-option MMLU variant39 metric: accuracy40 metric_label: accuracy41 unit: "%"42 metric_min: 043 metric_max: 10044 higher_is_better: true45 harness: self-reported by labs or independent evaluators46 comparability_note: 10-option questions; chain-of-thought is the standard regime, direct-answer scores are lower and not comparable.47 website: https://github.com/TIGER-AI-Lab/MMLU-Pro48 paper: https://arxiv.org/abs/2406.0157449 source_url: https://arxiv.org/abs/2406.0157450 - key: gpqa51 name: GPQA52 aliases: [GPQA main, Graduate-Level Google-Proof Q&A]53 family: gpqa54 variant: main55 family_head: true56 category: reasoning57 task: graduate-level science questions (main set, 448 questions)58 metric: accuracy59 metric_label: accuracy60 unit: "%"61 metric_min: 062 metric_max: 10063 higher_is_better: true64 harness: self-reported by labs or independent evaluators65 comparability_note: Main and Diamond subsets are different question sets; reasoning effort and sampling (pass@1 vs majority vote) change scores.66 website: https://github.com/idavidrein/gpqa67 paper: https://arxiv.org/abs/2311.1202268 source_url: https://arxiv.org/abs/2311.1202269 - key: gpqa-diamond70 name: GPQA Diamond71 aliases: [GPQA-Diamond, gpqa_diamond, GPQA (Diamond), GPQA Diamond (0-shot)]72 family: gpqa73 variant: Diamond74 category: reasoning75 task: graduate-level science questions — the 198-question Diamond subset (expert-validated, non-expert-failed)76 metric: accuracy77 metric_label: accuracy78 unit: "%"79 metric_min: 080 metric_max: 10081 higher_is_better: true82 harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling83 comparability_note: 198 questions → ±3 pt noise; reasoning effort, thinking budget and repeated sampling strongly affect scores.84 website: https://github.com/idavidrein/gpqa85 paper: https://arxiv.org/abs/2311.1202286 source_url: https://arxiv.org/abs/2311.1202287 - key: humaneval88 name: HumanEval89 aliases: [HumanEval pass@1, OpenAI HumanEval]90 family: humaneval91 variant: original92 family_head: true93 category: coding94 task: Python function synthesis from docstrings95 metric: pass@196 metric_label: pass@197 unit: "%"98 metric_min: 099 metric_max: 100100 higher_is_better: true101 harness: self-reported by labs102 comparability_note: Saturated; pass@1 vs pass@k and prompt wrapping differ across reporters.103 website: https://github.com/openai/human-eval104 paper: https://arxiv.org/abs/2107.03374105 known_limitations: Saturated; small (164 problems).106 source_url: https://arxiv.org/abs/2107.03374107 - key: swe-bench-verified108 name: SWE-bench Verified109 aliases: [SWE-bench, SWE-Bench Verified, SWE-bench-Verified, swebench verified, SWE Bench Verified]110 family: swe-bench111 variant: Verified112 category: coding113 task: resolve real GitHub issues (500 human-validated instances)114 metric: resolved115 metric_label: "% resolved"116 unit: "%"117 metric_min: 0118 metric_max: 100119 higher_is_better: true120 harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness, submissions optionally checked by the SWE-bench team121 comparability_note: Scores depend on the agent scaffold as much as on the model; compare only within one scaffold (config.system) and one attempt regime.122 website: https://www.swebench.com123 paper: https://arxiv.org/abs/2310.06770124 known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.125 source_url: https://www.swebench.com/126 - key: aider-polyglot127 name: Aider polyglot128 aliases: [Aider polyglot coding leaderboard, Aider Polyglot, aider polyglot benchmark, Aider]129 family: aider-polyglot130 variant: polyglot131 family_head: true132 category: coding133 task: 225 Exercism exercises in 6 languages, edit-format aware134 metric: pass_rate_2135 metric_label: pass rate (2 attempts)136 unit: "%"137 metric_min: 0138 metric_max: 100139 higher_is_better: true140 harness: aider (config.edit_format = whole | diff | diff-fenced | architect), second attempt after test feedback141 comparability_note: Edit format and aider version change results; the well-formed rate is a separate benchmark entry.142 website: https://aider.chat/docs/leaderboards/143 known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing.144 source_url: https://aider.chat/docs/leaderboards/145 - key: aider-polyglot-well-formed146 name: Aider polyglot — well-formed responses147 aliases: [Aider well-formed rate, percent_cases_well_formed, Aider polyglot well-formed]148 family: aider-polyglot149 variant: well-formed150 category: coding151 task: share of aider polyglot cases where every edit was syntactically well formed (parsable edit blocks)152 metric: percent_cases_well_formed153 metric_label: "% cases well formed"154 unit: "%"155 metric_min: 0156 metric_max: 100157 higher_is_better: true158 harness: aider (config.edit_format), same runs as aider-polyglot159 comparability_note: Measures edit-format compliance, not correctness; only comparable within one edit format.160 website: https://aider.chat/docs/leaderboards/161 source_url: https://aider.chat/docs/leaderboards/162 - key: livebench163 name: LiveBench164 aliases: [LiveBench global average, LiveBench Global, LiveBench overall]165 family: livebench166 variant: global167 family_head: true168 category: general169 task: contamination-limited, monthly refreshed questions across 7 categories — overall mean of the category averages170 metric: global_average171 metric_label: global average172 unit: "%"173 metric_min: 0174 metric_max: 100175 higher_is_better: true176 harness: LiveBench official runs; config.release = question-set release date177 comparability_note: Each release is a new question set — compare scores within one release (run group) only; effort variants are configurations.178 website: https://livebench.ai179 paper: https://arxiv.org/abs/2406.19314180 source_url: https://livebench.ai/181 - key: livebench-reasoning182 name: LiveBench Reasoning183 aliases: [LiveBench — Reasoning, LiveBench reasoning average]184 family: livebench185 variant: Reasoning186 category: reasoning187 task: LiveBench reasoning category — mean of its subtasks (web of lies, zebra puzzles, spatial…)188 metric: average score189 metric_label: category average190 unit: "%"191 metric_min: 0192 metric_max: 100193 higher_is_better: true194 harness: LiveBench official runs195 comparability_note: Compare within one release only; subtask mix changes between releases.196 website: https://livebench.ai197 source_url: https://livebench.ai/198 - key: livebench-coding199 name: LiveBench Coding200 aliases: [LiveBench — Coding, LiveBench coding average]201 family: livebench202 variant: Coding203 category: coding204 task: LiveBench coding category — mean of its subtasks (LCB generation, code completion)205 metric: average score206 metric_label: category average207 unit: "%"208 metric_min: 0209 metric_max: 100210 higher_is_better: true211 harness: LiveBench official runs212 comparability_note: Compare within one release only; subtask mix changes between releases.213 website: https://livebench.ai214 source_url: https://livebench.ai/215 - key: livebench-agentic-coding216 name: LiveBench Agentic Coding217 aliases: [LiveBench — Agentic Coding, LiveBench agentic coding average]218 family: livebench219 variant: Agentic Coding220 category: coding221 task: LiveBench agentic coding category — mean of its subtasks222 metric: average score223 metric_label: category average224 unit: "%"225 metric_min: 0226 metric_max: 100227 higher_is_better: true228 harness: LiveBench official runs229 comparability_note: Compare within one release only; subtask mix changes between releases.230 website: https://livebench.ai231 source_url: https://livebench.ai/232 - key: livebench-mathematics233 name: LiveBench Mathematics234 aliases: [LiveBench — Mathematics, LiveBench math average, LiveBench Math]235 family: livebench236 variant: Mathematics237 category: math238 task: LiveBench mathematics category — mean of its subtasks (competition math, AMPS hard, olympiad)239 metric: average score240 metric_label: category average241 unit: "%"242 metric_min: 0243 metric_max: 100244 higher_is_better: true245 harness: LiveBench official runs246 comparability_note: Compare within one release only; subtask mix changes between releases.247 website: https://livebench.ai248 source_url: https://livebench.ai/249 - key: livebench-data-analysis250 name: LiveBench Data Analysis251 aliases: [LiveBench — Data Analysis, LiveBench data analysis average]252 family: livebench253 variant: Data Analysis254 category: general255 task: LiveBench data analysis category — mean of its subtasks (table reformatting, join, column type)256 metric: average score257 metric_label: category average258 unit: "%"259 metric_min: 0260 metric_max: 100261 higher_is_better: true262 harness: LiveBench official runs263 comparability_note: Compare within one release only; subtask mix changes between releases.264 website: https://livebench.ai265 source_url: https://livebench.ai/266 - key: livebench-language267 name: LiveBench Language268 aliases: [LiveBench — Language, LiveBench language average]269 family: livebench270 variant: Language271 category: general272 task: LiveBench language category — mean of its subtasks (typos, connections, plot unscrambling)273 metric: average score274 metric_label: category average275 unit: "%"276 metric_min: 0277 metric_max: 100278 higher_is_better: true279 harness: LiveBench official runs280 comparability_note: Compare within one release only; subtask mix changes between releases.281 website: https://livebench.ai282 source_url: https://livebench.ai/283 - key: livebench-if284 name: LiveBench Instruction Following285 aliases: [LiveBench — IF, LiveBench IF, LiveBench instruction following average]286 family: livebench287 variant: IF288 category: instruction-following289 task: LiveBench instruction-following category — mean of its subtasks (paraphrase, simplify, story generation, summarize)290 metric: average score291 metric_label: category average292 unit: "%"293 metric_min: 0294 metric_max: 100295 higher_is_better: true296 harness: LiveBench official runs297 comparability_note: Compare within one release only; subtask mix changes between releases.298 website: https://livebench.ai299 source_url: https://livebench.ai/300 - key: math-500301 name: MATH-500302 aliases: [MATH, MATH 500, MATH500]303 family: math304 variant: "500"305 family_head: true306 category: math307 task: competition mathematics (500-problem subset of MATH used since the PRM800K paper)308 metric: accuracy309 metric_label: accuracy310 unit: "%"311 metric_min: 0312 metric_max: 100313 higher_is_better: true314 harness: self-reported by labs or independent evaluators315 comparability_note: Saturated at the frontier; sampling regime (pass@1 vs majority vote) must match.316 paper: https://arxiv.org/abs/2103.03874317 source_url: https://arxiv.org/abs/2103.03874318 - key: aime-2024319 name: AIME 2024320 aliases: [AIME24, AIME 24, AIME I/II 2024]321 family: aime322 variant: "2024"323 version: "2024"324 category: math325 task: American Invitational Mathematics Examination 2024 problems (30 problems, I + II)326 metric: accuracy327 metric_label: accuracy328 unit: "%"329 metric_min: 0330 metric_max: 100331 higher_is_better: true332 harness: self-reported by labs or independent evaluators333 comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting (cons@64) are not comparable; 2024 problems appear in training data of later models.334 source_url: https://maa.org/maa-invitational-competitions/335 - key: aime-2025336 name: AIME 2025337 aliases: [AIME, AIME25, AIME 25, AIME I/II 2025]338 family: aime339 variant: "2025"340 version: "2025"341 family_head: true342 category: math343 task: American Invitational Mathematics Examination 2025 problems (30 problems, I + II)344 metric: accuracy345 metric_label: accuracy346 unit: "%"347 metric_min: 0348 metric_max: 100349 higher_is_better: true350 harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling351 comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting are not comparable; reasoning effort matters.352 known_limitations: 30 problems per year; high variance; often reported with majority voting.353 source_url: https://maa.org/maa-invitational-competitions/354 - key: arc-agi355 name: ARC-AGI356 aliases: [ARC-AGI-1, ARC AGI 1, Abstraction and Reasoning Corpus, ARC-AGI v1]357 family: arc-agi358 variant: "1"359 version: "1"360 family_head: true361 category: reasoning362 task: novel visual abstraction puzzles (ARC-AGI-1 semi-private / public evaluation sets)363 metric: accuracy364 metric_label: accuracy365 unit: "%"366 metric_min: 0367 metric_max: 100368 higher_is_better: true369 harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs370 comparability_note: Public, semi-private and private evaluation sets differ; the ARC Prize reports cost per task alongside the score.371 website: https://arcprize.org372 source_url: https://arcprize.org/373 - key: arc-agi-2374 name: ARC-AGI-2375 aliases: [ARC AGI 2, ARC-AGI v2, ARC-AGI 2]376 family: arc-agi377 variant: "2"378 version: "2"379 category: reasoning380 task: harder abstraction puzzles released in 2025 (ARC-AGI-2), designed to resist brute-force program search381 metric: accuracy382 metric_label: accuracy383 unit: "%"384 metric_min: 0385 metric_max: 100386 higher_is_better: true387 harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs388 comparability_note: Not comparable with ARC-AGI-1 scores; evaluation set (public / semi-private) and cost budget must match.389 website: https://arcprize.org/arc-agi/2/390 source_url: https://arcprize.org/arc-agi/2/391 - key: humanitys-last-exam392 name: Humanity's Last Exam393 aliases: [HLE, Humanity’s Last Exam, HLE (no tools)]394 family: humanitys-last-exam395 variant: full396 family_head: true397 category: knowledge398 task: expert-written frontier questions399 metric: accuracy400 metric_label: accuracy401 unit: "%"402 metric_min: 0403 metric_max: 100404 higher_is_better: true405 harness: Artificial Analysis and labs; text-only subset vs full multimodal set, with or without tools406 comparability_note: Tool use (search) and the text-only subset change scores substantially; check config before comparing.407 website: https://lastexam.ai408 paper: https://arxiv.org/abs/2501.14249409 source_url: https://lastexam.ai/410 - key: lmarena-text411 name: LMArena text leaderboard412 aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo, LMArena, Arena Score]413 family: lmarena414 variant: text415 family_head: true416 category: preference417 task: crowdsourced pairwise human preference418 metric: elo419 metric_label: Elo / Bradley–Terry score420 unit: ""421 metric_min: 0422 metric_max: null423 higher_is_better: true424 harness: LMArena live human votes (Bradley–Terry model)425 comparability_note: Relative score that drifts as the model pool changes; only a snapshot of the same date is comparable; style control changes rankings.426 website: https://lmarena.ai427 known_limitations: Style bias; sampling of prompts by users.428 source_url: https://lmarena.ai/429 - key: mmmu430 name: MMMU431 aliases: [MMMU (val), Massive Multi-discipline Multimodal Understanding]432 family: mmmu433 variant: original434 family_head: true435 category: multimodal436 task: college-level multimodal understanding437 metric: accuracy438 metric_label: accuracy439 unit: "%"440 metric_min: 0441 metric_max: 100442 higher_is_better: true443 harness: self-reported by labs (validation split)444 comparability_note: Validation vs test split and chain-of-thought prompting differ across reporters.445 website: https://mmmu-benchmark.github.io446 paper: https://arxiv.org/abs/2311.16502447 source_url: https://arxiv.org/abs/2311.16502448 - key: tau-bench449 name: τ-bench450 aliases: [tau-bench, TAU-bench, tau bench, τ-Bench]451 family: tau-bench452 variant: v1453 family_head: true454 category: agentic455 task: tool-agent-user interaction in retail/airline domains456 metric: pass^1457 metric_label: pass^1458 unit: "%"459 metric_min: 0460 metric_max: 100461 higher_is_better: true462 harness: simulated user (LLM) + tool environment; pass^k over k trials463 comparability_note: Domain (retail / airline), pass^k regime and the user-simulator model must match.464 paper: https://arxiv.org/abs/2406.12045465 source_url: https://arxiv.org/abs/2406.12045466 - key: terminal-bench467 name: Terminal-Bench468 aliases: [TerminalBench, Terminal Bench, Terminal-Bench 1.0]469 family: terminal-bench470 variant: "1.0"471 version: "1.0"472 family_head: true473 category: agentic474 task: terminal tasks solved by agents475 metric: accuracy476 metric_label: accuracy477 unit: "%"478 metric_min: 0479 metric_max: 100480 higher_is_better: true481 harness: Terminus agent (official) or the submitter's agent; Artificial Analysis reports its own harness versions (config.variant)482 comparability_note: Agent scaffold and harness version change results; config.variant (v2.1 / v4.0 / hard) must match.483 website: https://www.tbench.ai484 source_url: https://www.tbench.ai/485 - key: terminal-bench-2486 name: Terminal-Bench 2.0487 aliases: [Terminal Bench 2, TerminalBench 2.0, Terminal-Bench 2, TB2]488 family: terminal-bench489 variant: "2.0"490 version: "2.0"491 category: agentic492 task: revised, harder set of terminal tasks (Terminal-Bench 2.0)493 metric: accuracy494 metric_label: accuracy495 unit: "%"496 metric_min: 0497 metric_max: 100498 higher_is_better: true499 harness: Terminus 2 agent (official) or the submitter's agent500 comparability_note: Not comparable with Terminal-Bench 1.0; agent scaffold must match.501 website: https://www.tbench.ai502 source_url: https://www.tbench.ai/503 - key: ifeval504 name: IFEval505 aliases: [IF-Eval, Instruction-Following Eval, IFEval strict]506 family: ifeval507 variant: original508 family_head: true509 category: instruction-following510 task: verifiable instruction following511 metric: accuracy512 metric_label: prompt-level strict accuracy513 unit: "%"514 metric_min: 0515 metric_max: 100516 higher_is_better: true517 harness: self-reported by labs518 comparability_note: Prompt-level vs instruction-level and strict vs loose accuracy are four different numbers.519 paper: https://arxiv.org/abs/2311.07911520 source_url: https://arxiv.org/abs/2311.07911521 - key: mteb522 name: MTEB523 aliases: [Massive Text Embedding Benchmark, MTEB (English), MTEB v1]524 family: mteb525 variant: v1526 family_head: true527 category: embeddings528 task: embedding tasks across retrieval, classification, clustering…529 metric: mean score530 metric_label: mean score531 unit: ""532 metric_min: 0533 metric_max: 100534 higher_is_better: true535 harness: MTEB library, official leaderboard536 comparability_note: Task set (English v1, v2, multilingual) and aggregation (mean of tasks vs mean of task types) must match.537 website: https://huggingface.co/spaces/mteb/leaderboard538 paper: https://arxiv.org/abs/2210.07316539 source_url: https://arxiv.org/abs/2210.07316540 - key: mmteb541 name: MMTEB542 aliases: [Massive Multilingual Text Embedding Benchmark, MTEB multilingual, MTEB (Multilingual)]543 family: mteb544 variant: MMTEB545 category: embeddings546 task: multilingual extension of MTEB (500+ tasks across 250+ languages)547 metric: mean score548 metric_label: mean score549 unit: ""550 metric_min: 0551 metric_max: 100552 higher_is_better: true553 harness: MTEB library, official leaderboard554 comparability_note: Not comparable with English-only MTEB; the leaderboard's Borda rank and task-type mean are different aggregations.555 website: https://huggingface.co/spaces/mteb/leaderboard556 paper: https://arxiv.org/abs/2502.13595557 source_url: https://arxiv.org/abs/2502.13595558 - key: artificial-analysis-intelligence-index559 name: Artificial Analysis Intelligence Index560 aliases: [AA Intelligence Index, Intelligence Index, Artificial Analysis Index]561 family: artificial-analysis-intelligence-index562 variant: index563 family_head: true564 category: composite565 task: composite of several evaluations run by Artificial Analysis566 metric: index567 metric_label: index568 unit: ""569 metric_min: 0570 metric_max: 100571 higher_is_better: true572 harness: Artificial Analysis internal runs; config.version = index methodology version573 comparability_note: Methodology versions change the component set — compare within one version (run group) only; reasoning-effort variants are configurations.574 website: https://artificialanalysis.ai575 known_limitations: Proprietary composite; methodology versions change.576 source_url: https://artificialanalysis.ai/methodology577