1# Benchmarks reported by the leaderboard connectors (SWE-bench boards, Artificial Analysis component evaluations). Fragment of registry/benchmarks.yaml.2benchmarks:3 - key: swe-bench-lite4 name: SWE-bench Lite5 category: coding6 task: resolve real GitHub issues (300-instance subset)7 metric: resolved8 unit: "%"9 website: https://www.swebench.com10 paper: https://arxiv.org/abs/2310.0677011 known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.12 source_url: https://www.swebench.com/13 - key: swe-bench-full14 name: SWE-bench (full test split)15 aliases: [SWE-bench Test, SWE-bench Full]16 category: coding17 task: resolve real GitHub issues (2,294 instances)18 metric: resolved19 unit: "%"20 website: https://www.swebench.com21 paper: https://arxiv.org/abs/2310.0677022 source_url: https://www.swebench.com/23 - key: swe-bench-multimodal24 name: SWE-bench Multimodal25 category: coding26 task: resolve visual JavaScript issues (screenshots + code)27 metric: resolved28 unit: "%"29 website: https://www.swebench.com/multimodal30 source_url: https://www.swebench.com/31 - key: swe-bench-multilingual32 name: SWE-bench Multilingual33 category: coding34 task: resolve GitHub issues across 9 programming languages35 metric: resolved36 unit: "%"37 website: https://www.swebench.com/multilingual38 source_url: https://www.swebench.com/39 - key: mmmu-pro40 name: MMMU-Pro41 category: multimodal42 task: robust multimodal understanding (10-option, vision-only variants)43 metric: accuracy44 unit: "%"45 paper: https://arxiv.org/abs/2409.0281346 source_url: https://arxiv.org/abs/2409.0281347 - key: livecodebench48 name: LiveCodeBench49 category: coding50 task: contamination-free competitive programming problems51 metric: pass@152 unit: "%"53 website: https://livecodebench.github.io54 paper: https://arxiv.org/abs/2403.0797455 source_url: https://arxiv.org/abs/2403.0797456 - key: scicode57 name: SciCode58 category: coding59 task: research-level scientific coding problems60 metric: accuracy61 unit: "%"62 website: https://scicode-bench.github.io63 paper: https://arxiv.org/abs/2407.1316864 source_url: https://arxiv.org/abs/2407.1316865 - key: ifbench66 name: IFBench67 category: instruction-following68 task: precise instruction following with novel constraints69 metric: accuracy70 unit: "%"71 paper: https://arxiv.org/abs/2507.0283372 source_url: https://arxiv.org/abs/2507.0283373 - key: tau2-bench74 name: τ²-bench75 aliases: [tau2-bench, TAU2-bench, tau-squared bench]76 category: agentic77 task: dual-control tool-agent-user interaction (telecom, retail, airline)78 metric: pass^179 unit: "%"80 paper: https://arxiv.org/abs/2506.0798281 source_url: https://arxiv.org/abs/2506.0798282