HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# Benchmarks reported by the leaderboard connectors (SWE-bench boards, Artificial Analysis component evaluations). Fragment of registry/benchmarks.yaml2# (same canonical fields: family / variant / version / family_head / metric / metric_min / metric_max / higher_is_better / harness / comparability_note).3benchmarks:4 - key: swe-bench-full5 name: SWE-bench (full test split)6 aliases: [SWE-bench Test, SWE-bench Full, SWE-bench (full), swebench full]7 family: swe-bench8 variant: full9 family_head: true10 category: coding11 task: resolve real GitHub issues (2,294 instances)12 metric: resolved13 metric_label: "% resolved"14 unit: "%"15 metric_min: 016 metric_max: 10017 higher_is_better: true18 harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness19 comparability_note: Scaffold-dependent; compare within one scaffold and attempt regime only.20 website: https://www.swebench.com21 paper: https://arxiv.org/abs/2310.0677022 source_url: https://www.swebench.com/23 - key: swe-bench-lite24 name: SWE-bench Lite25 aliases: [SWE-Bench Lite, swebench lite]26 family: swe-bench27 variant: Lite28 category: coding29 task: resolve real GitHub issues (300-instance subset)30 metric: resolved31 metric_label: "% resolved"32 unit: "%"33 metric_min: 034 metric_max: 10035 higher_is_better: true36 harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness37 comparability_note: Scaffold-dependent; Lite is an easier subset — never compare with Verified or full scores.38 website: https://www.swebench.com39 paper: https://arxiv.org/abs/2310.0677040 known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.41 source_url: https://www.swebench.com/42 - key: swe-bench-multimodal43 name: SWE-bench Multimodal44 aliases: [SWE-Bench Multimodal, SWE-bench MM]45 family: swe-bench46 variant: Multimodal47 category: coding48 task: resolve visual JavaScript issues (screenshots + code)49 metric: resolved50 metric_label: "% resolved"51 unit: "%"52 metric_min: 053 metric_max: 10054 higher_is_better: true55 harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness56 comparability_note: Requires image input; scaffold-dependent.57 website: https://www.swebench.com/multimodal58 source_url: https://www.swebench.com/59 - key: swe-bench-multilingual60 name: SWE-bench Multilingual61 aliases: [SWE-Bench Multilingual]62 family: swe-bench63 variant: Multilingual64 category: coding65 task: resolve GitHub issues across 9 programming languages66 metric: resolved67 metric_label: "% resolved"68 unit: "%"69 metric_min: 070 metric_max: 10071 higher_is_better: true72 harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness73 comparability_note: Scaffold-dependent; different instance set from Verified.74 website: https://www.swebench.com/multilingual75 source_url: https://www.swebench.com/76 - key: swe-bench-pro77 name: SWE-Bench Pro78 aliases: [SWE-bench Pro, SWE Bench Pro, swebench pro]79 family: swe-bench80 variant: Pro81 category: coding82 task: long-horizon, enterprise-grade software engineering tasks (public set) curated by Scale AI83 metric: resolved84 metric_label: "% resolved"85 unit: "%"86 metric_min: 087 metric_max: 10088 higher_is_better: true89 harness: Scale AI evaluation with a fixed scaffold per leaderboard column90 comparability_note: Public vs commercial (held-out) sets are different leaderboards; not comparable with SWE-bench Verified.91 website: https://scale.com/leaderboard/swe_bench_pro_public92 source_url: https://scale.com/leaderboard/swe_bench_pro_public93 - key: mmmu-pro94 name: MMMU-Pro95 aliases: [MMMU Pro]96 family: mmmu97 variant: Pro98 category: multimodal99 task: robust multimodal understanding (10-option, vision-only variants)100 metric: accuracy101 metric_label: accuracy102 unit: "%"103 metric_min: 0104 metric_max: 100105 higher_is_better: true106 harness: Artificial Analysis runs its own evaluation; labs self-report107 comparability_note: Standard (10-option) vs vision-only settings are different numbers.108 paper: https://arxiv.org/abs/2409.02813109 source_url: https://arxiv.org/abs/2409.02813110 - key: livecodebench111 name: LiveCodeBench112 aliases: [LCB, Live Code Bench]113 family: livecodebench114 variant: rolling115 family_head: true116 category: coding117 task: contamination-free competitive programming problems118 metric: pass@1119 metric_label: pass@1120 unit: "%"121 metric_min: 0122 metric_max: 100123 higher_is_better: true124 harness: Artificial Analysis runs its own evaluation; labs self-report on a chosen date window125 comparability_note: The problem window (release dates) differs between reporters — scores from different windows are not comparable.126 website: https://livecodebench.github.io127 paper: https://arxiv.org/abs/2403.07974128 source_url: https://arxiv.org/abs/2403.07974129 - key: scicode130 name: SciCode131 aliases: [SciCode benchmark]132 family: scicode133 variant: main134 family_head: true135 category: coding136 task: research-level scientific coding problems137 metric: accuracy138 metric_label: accuracy139 unit: "%"140 metric_min: 0141 metric_max: 100142 higher_is_better: true143 harness: Artificial Analysis runs its own evaluation144 comparability_note: Sub-problem vs main-problem accuracy are different numbers; with/without background prompts.145 website: https://scicode-bench.github.io146 paper: https://arxiv.org/abs/2407.13168147 source_url: https://arxiv.org/abs/2407.13168148 - key: ifbench149 name: IFBench150 aliases: [IF-Bench]151 family: ifeval152 variant: IFBench153 category: instruction-following154 task: precise instruction following with novel constraints155 metric: accuracy156 metric_label: accuracy157 unit: "%"158 metric_min: 0159 metric_max: 100160 higher_is_better: true161 harness: Artificial Analysis runs its own evaluation162 comparability_note: Different constraint set from IFEval; not comparable with IFEval scores.163 paper: https://arxiv.org/abs/2507.02833164 source_url: https://arxiv.org/abs/2507.02833165 - key: tau2-bench166 name: τ²-bench167 aliases: [tau2-bench, TAU2-bench, tau-squared bench, τ²-Bench, τ2-bench, τ²-Bench Telecom, tau2 bench telecom]168 family: tau-bench169 variant: τ²170 category: agentic171 task: dual-control tool-agent-user interaction (telecom, retail, airline)172 metric: pass^1173 metric_label: pass^1174 unit: "%"175 metric_min: 0176 metric_max: 100177 higher_is_better: true178 harness: simulated user (LLM) + dual-control tool environment; Artificial Analysis reports the Telecom domain179 comparability_note: Domain (config.variant = Telecom / Retail / Airline) and the user-simulator model must match.180 paper: https://arxiv.org/abs/2506.07982181 source_url: https://arxiv.org/abs/2506.07982182