1# Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources.2benchmarks:3 - key: mmlu4 name: MMLU5 aliases: [Massive Multitask Language Understanding]6 category: knowledge7 task: 57-subject multiple-choice questions8 metric: accuracy9 unit: "%"10 creator: Hendrycks et al.11 website: https://github.com/hendrycks/test12 paper: https://arxiv.org/abs/2009.0330013 known_limitations: Widespread training-data contamination; saturated at the frontier.14 source_url: https://arxiv.org/abs/2009.0330015 - key: mmlu-pro16 name: MMLU-Pro17 category: knowledge18 task: harder, 10-option MMLU variant19 metric: accuracy20 unit: "%"21 website: https://github.com/TIGER-AI-Lab/MMLU-Pro22 paper: https://arxiv.org/abs/2406.0157423 source_url: https://arxiv.org/abs/2406.0157424 - key: gpqa25 name: GPQA26 aliases: [GPQA Diamond]27 category: reasoning28 task: graduate-level science questions29 metric: accuracy30 unit: "%"31 website: https://github.com/idavidrein/gpqa32 paper: https://arxiv.org/abs/2311.1202233 source_url: https://arxiv.org/abs/2311.1202234 - key: humaneval35 name: HumanEval36 category: coding37 task: Python function synthesis from docstrings38 metric: pass@139 unit: "%"40 website: https://github.com/openai/human-eval41 paper: https://arxiv.org/abs/2107.0337442 known_limitations: Saturated; small (164 problems).43 source_url: https://arxiv.org/abs/2107.0337444 - key: swe-bench-verified45 name: SWE-bench Verified46 aliases: [SWE-bench]47 category: coding48 task: resolve real GitHub issues (500 human-validated instances)49 metric: resolved50 unit: "%"51 website: https://www.swebench.com52 paper: https://arxiv.org/abs/2310.0677053 known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.54 source_url: https://www.swebench.com/55 - key: aider-polyglot56 name: Aider polyglot57 aliases: [Aider polyglot coding leaderboard]58 category: coding59 task: 225 Exercism exercises in 6 languages, edit-format aware60 metric: pass rate (2 attempts)61 unit: "%"62 website: https://aider.chat/docs/leaderboards/63 known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing.64 source_url: https://aider.chat/docs/leaderboards/65 - key: livebench66 name: LiveBench67 category: general68 task: contamination-limited, monthly refreshed questions across 6 categories69 metric: average score70 unit: "%"71 website: https://livebench.ai72 paper: https://arxiv.org/abs/2406.1931473 source_url: https://livebench.ai/74 - key: math-50075 name: MATH-50076 aliases: [MATH]77 category: math78 task: competition mathematics79 metric: accuracy80 unit: "%"81 paper: https://arxiv.org/abs/2103.0387482 source_url: https://arxiv.org/abs/2103.0387483 - key: aime-202584 name: AIME 202585 aliases: [AIME]86 category: math87 task: American Invitational Mathematics Examination problems88 metric: accuracy89 unit: "%"90 known_limitations: 30 problems per year; high variance; often reported with majority voting.91 source_url: https://maa.org/maa-invitational-competitions/92 - key: arc-agi93 name: ARC-AGI94 aliases: [ARC-AGI-1, ARC-AGI-2, Abstraction and Reasoning Corpus]95 category: reasoning96 task: novel visual abstraction puzzles97 metric: accuracy98 unit: "%"99 website: https://arcprize.org100 source_url: https://arcprize.org/101 - key: humanitys-last-exam102 name: Humanity's Last Exam103 aliases: [HLE]104 category: knowledge105 task: expert-written frontier questions106 metric: accuracy107 unit: "%"108 website: https://lastexam.ai109 paper: https://arxiv.org/abs/2501.14249110 source_url: https://lastexam.ai/111 - key: lmarena-text112 name: LMArena text leaderboard113 aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo]114 category: preference115 task: crowdsourced pairwise human preference116 metric: Elo / Bradley–Terry score117 unit: ""118 website: https://lmarena.ai119 known_limitations: Style bias; sampling of prompts by users.120 source_url: https://lmarena.ai/121 - key: mmmu122 name: MMMU123 category: multimodal124 task: college-level multimodal understanding125 metric: accuracy126 unit: "%"127 website: https://mmmu-benchmark.github.io128 paper: https://arxiv.org/abs/2311.16502129 source_url: https://arxiv.org/abs/2311.16502130 - key: tau-bench131 name: τ-bench132 aliases: [tau-bench, TAU-bench]133 category: agentic134 task: tool-agent-user interaction in retail/airline domains135 metric: pass^1136 unit: "%"137 paper: https://arxiv.org/abs/2406.12045138 source_url: https://arxiv.org/abs/2406.12045139 - key: terminal-bench140 name: Terminal-Bench141 category: agentic142 task: terminal tasks solved by agents143 metric: accuracy144 unit: "%"145 website: https://www.tbench.ai146 source_url: https://www.tbench.ai/147 - key: ifeval148 name: IFEval149 category: instruction-following150 task: verifiable instruction following151 metric: prompt-level strict accuracy152 unit: "%"153 paper: https://arxiv.org/abs/2311.07911154 source_url: https://arxiv.org/abs/2311.07911155 - key: mteb156 name: MTEB157 aliases: [Massive Text Embedding Benchmark]158 category: embeddings159 task: embedding tasks across retrieval, classification, clustering…160 metric: mean score161 unit: ""162 website: https://huggingface.co/spaces/mteb/leaderboard163 paper: https://arxiv.org/abs/2210.07316164 source_url: https://arxiv.org/abs/2210.07316165 - key: artificial-analysis-intelligence-index166 name: Artificial Analysis Intelligence Index167 category: composite168 task: composite of several evaluations run by Artificial Analysis169 metric: index170 unit: ""171 website: https://artificialanalysis.ai172 known_limitations: Proprietary composite; methodology versions change.173 source_url: https://artificialanalysis.ai/methodology174