spb/llmindex Public
The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.
TypeScript 77.9%
TeX 15.2%
Python 3.7%
SQL 1.4%
JavaScript 1.1%
Shell 0.5%
1<div align="center">23<img src="docs/assets/logo.svg" width="110" alt="LLM Index logo" />45# LLM Index67**The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.**89[**www.llmindex.io**](https://www.llmindex.io) · [Methodology](https://www.llmindex.io/methodology) · [White Paper (PDF)](docs/whitepaper/llmindex-whitepaper.pdf) · [Public API](https://www.llmindex.io/api/v1/leaderboard)1011<!-- status pastilles -->12[](https://www.llmindex.io)131415161718[](docs/whitepaper/llmindex-whitepaper.pdf)1920<!-- metric pastilles -->21222324252627282930<!-- stack pastilles -->3132333435363738394041</div>4243---4445## Why another index? Because leaderboards are broken.4647Classic leaderboards fail twice: **top models cluster at 95%+ on saturated benchmarks** (zero48discrimination), and **fixed test sets leak into training data** (contamination). LLM Index is49engineered against both, from the psychometrics up:5051| | |52|---|---|53| 🎯 **IRT 2PL scoring** | Every item has a fitted difficulty *b* and discrimination *a*; ability θ is a MAP estimate with Fisher-information standard errors. Items that don't separate models are auto-retired. Raw accuracy is never the score. |54| 🎲 **Dynamic item generation** | Every scored batch is freshly generated from seeded template generators (values, paraphrases, structures). There is no fixed test set to memorize — the fixed-vs-fresh gap is *published* per model as `contamination_delta`. |55| 🤖 **Home-made agentic bench** | Simulated tool-calling environments (triage-under-policy, treasury ledger, deployment DAGs) with distractor tools; a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges. |56| 🧠 **Agentic under context load** | 120–300-row generated ledgers packed with near-miss decoys: the model must find, order, and act on the few matching records. Prompt length is the difficulty knob. |57| 💻 **Home-made terminal bench** | No shell executes: a closed, unambiguous POSIX subset is simulated in TypeScript. Models predict exact pipeline stdout, file trees after `mv/cp/rm/cd`, and `&&`/`\|\|` exit-code traces. |58| 🎨 **SVG logo duels** | Models reproduce real-world logos in raw SVG from memory; a 3-judge cross-provider panel (position-swapped, never self-judging) feeds a Bradley-Terry fit. |59| 👁 **Vision OCR under clutter** | Generated scenes (rotated codes, noise, low-contrast, decoys) rasterized to PNG; ambiguous glyphs excluded by design. Text-only models skip; weights renormalize. |60| 🛡 **Extraction that never cheats models** | Lenient cascade (`ANSWER:` in any markdown, `FINAL ANSWER`, `\boxed{}`, fenced blocks), number normalization, 16k completion budgets, truncated ≠ wrong. Formatting is measured in its own domain — never silently everywhere. |61| 📡 **Live, one model at a time** | Parallel evaluation lanes stream results; after every completed model the IRT refit re-runs and the public leaderboard re-ranks in real time. |62| 🔍 **Total transparency** | Every model has a page showing **every answer on every test**, judge verdicts, confidence, latency and cost. Every number traces to an immutable score run. Answer keys never leave the server. |6364## Screenshots6566<div align="center">6768**Live leaderboard — desktop**6970<img src="docs/assets/home-desktop.png" width="860" alt="LLM Index home, live leaderboard" />7172<table>73<tr>74<td align="center" width="300">7576**Mobile**7778<img src="docs/assets/home-mobile.png" width="240" alt="Mobile view" />7980</td>81<td align="center">8283**Model transparency page**8485<img src="docs/assets/model-page.png" width="560" alt="Model profile with every answer" />8687</td>88</tr>89</table>9091**Methodology — public, detailed, with live sample items and full references**9293<img src="docs/assets/methodology.png" width="860" alt="Methodology page" />9495</div>9697## The 12 domains9899100101102103104105106107108109110111112Domain weights are **equal by design** — the maximum-entropy prior; any other weighting is an113editorial value judgment. Per-domain scores are always published so you can re-weight. Cost and114latency are **never** blended into quality: they live on a separate Pareto frontier.115116## Architecture117118```119llmindex/120├── apps/121│ ├── web/ # Next.js 14 — live leaderboard, transparency pages, /api/v1/*122│ ├── worker/ # eval runner · duel runner · parallel benchmark orchestrator · refit123│ └── psychometrics/ # Python: 2PL IRT (MAP + Fisher SE), Bradley-Terry (MM + SE), calibration124├── packages/125│ ├── scoring/ # domains, weights, INDEX_VERSION, pure score aggregation (no I/O)126│ ├── items/ # 25 template generators + simulators + robust grading cascade127│ ├── openrouter/ # typed client: retries, timeouts, cost tracking, multimodal128│ └── db/ # Prisma: models, eval_items, model_responses, pairwise_duels, score_runs129├── docs/methodology/ # METHODOLOGY, CHANGELOG (semver), IRT_SPEC, JUDGE_PROTOCOL130└── data/item-bank/ # public template manifest (instantiated items stay server-side)131```132133**Pipeline:** OpenRouter catalog sync → seeded batch generation → parallel evaluation lanes134(temperature 0 scored + k consistency samples, full audit trail: raw responses, tokens, latency,135cost) → lenient extraction → 2PL fit per domain + Bradley-Terry for duels → scores with 95% CIs →136leaderboard updates live → discrimination hygiene auto-flags dead items.137138## Public API139140```bash141curl https://www.llmindex.io/api/v1/leaderboard # global ranking + CIs142curl https://www.llmindex.io/api/v1/leaderboard/agentic # per-domain + sub-metrics143curl https://www.llmindex.io/api/v1/models/anthropic/claude-sonnet-5144curl https://www.llmindex.io/api/v1/methodology # machine-readable weights & hyperparams145curl https://www.llmindex.io/api/v1/benchmark/progress # live run status146```147148Rate limits: 60 req/min anonymous, 600 with an API key. Breaking changes ship as `/api/v2` — v1149shapes are frozen.150151## Run it152153```bash154pnpm install155pnpm db:migrate && pnpm db:seed # sync models + pricing from OpenRouter156pnpm dev # web on :3000157158pnpm eval:run --model <slug> --domain math --n 30 --k 2159pnpm duel:run --domain svg_design --pairs 100160pnpm benchmark:run --parallel 6 # full live benchmark, refit after every model161pnpm index:refit # IRT + BT refit → new immutable score run162163pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test # all gates164```165166## Integrity rules (non-negotiable)1671681. Answer keys and rubrics never ship to the client or public API.1692. Scored batches use freshly perturbed items; the frozen anchor subset is ≤20% of any run.1703. Every displayed number traces to an immutable `score_runs` row (item-set hash, model set,171 index version, fit diagnostics). Batches with >2% failed calls are excluded until re-run.1724. Weights and IRT hyperparameters live in exactly one file and are served machine-readable.1735. Every methodology change bumps the semver `INDEX_VERSION` with a public changelog entry.174175The design draws on 60+ published sources (metabench, GSM-Symbolic, ZebraLogic, R-Horizon,176τ-bench, BFCL, Terminal-Bench audits, Math-Verify, ReasonIF, …) — full list with links on the177[methodology page](https://www.llmindex.io/methodology). Everything here is original: our own178environments, items, simulators and graders.179180---181182<div align="center">183184**Author & maintainer:** Simon-Pierre Boucher · [contact@spboucher.ai](mailto:contact@spboucher.ai)185186License: Proprietary — © Simon-Pierre Boucher, all rights reserved. Source visible for187transparency and auditability.188189<img src="docs/assets/logo.svg" width="28" alt="" />190191</div>192