LLM Index
The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.
www.llmindex.io · Methodology · White Paper (PDF) · Public API
Why another index? Because leaderboards are broken.
Classic leaderboards fail twice: top models cluster at 95%+ on saturated benchmarks (zero discrimination), and fixed test sets leak into training data (contamination). LLM Index is engineered against both, from the psychometrics up:
| 🎯 IRT 2PL scoring | Every item has a fitted difficulty b and discrimination a; ability θ is a MAP estimate with Fisher-information standard errors. Items that don't separate models are auto-retired. Raw accuracy is never the score. |
| 🎲 Dynamic item generation | Every scored batch is freshly generated from seeded template generators (values, paraphrases, structures). There is no fixed test set to memorize — the fixed-vs-fresh gap is published per model as contamination_delta. |
| 🤖 Home-made agentic bench | Simulated tool-calling environments (triage-under-policy, treasury ledger, deployment DAGs) with distractor tools; a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges. |
| 🧠 Agentic under context load | 120–300-row generated ledgers packed with near-miss decoys: the model must find, order, and act on the few matching records. Prompt length is the difficulty knob. |
| 💻 Home-made terminal bench | No shell executes: a closed, unambiguous POSIX subset is simulated in TypeScript. Models predict exact pipeline stdout, file trees after mv/cp/rm/cd, and &&/|| exit-code traces. |
| 🎨 SVG logo duels | Models reproduce real-world logos in raw SVG from memory; a 3-judge cross-provider panel (position-swapped, never self-judging) feeds a Bradley-Terry fit. |
| 👁 Vision OCR under clutter | Generated scenes (rotated codes, noise, low-contrast, decoys) rasterized to PNG; ambiguous glyphs excluded by design. Text-only models skip; weights renormalize. |
| 🛡 Extraction that never cheats models | Lenient cascade (ANSWER: in any markdown, FINAL ANSWER, \boxed{}, fenced blocks), number normalization, 16k completion budgets, truncated ≠ wrong. Formatting is measured in its own domain — never silently everywhere. |
| 📡 Live, one model at a time | Parallel evaluation lanes stream results; after every completed model the IRT refit re-runs and the public leaderboard re-ranks in real time. |
| 🔍 Total transparency | Every model has a page showing every answer on every test, judge verdicts, confidence, latency and cost. Every number traces to an immutable score run. Answer keys never leave the server. |
Screenshots
Live leaderboard — desktop
|
Mobile 
|
Model transparency page 
|
Methodology — public, detailed, with live sample items and full references
The 12 domains
Domain weights are equal by design — the maximum-entropy prior; any other weighting is an editorial value judgment. Per-domain scores are always published so you can re-weight. Cost and latency are never blended into quality: they live on a separate Pareto frontier.
Architecture
llmindex/
├── apps/
│ ├── web/ # Next.js 14 — live leaderboard, transparency pages, /api/v1/*
│ ├── worker/ # eval runner · duel runner · parallel benchmark orchestrator · refit
│ └── psychometrics/ # Python: 2PL IRT (MAP + Fisher SE), Bradley-Terry (MM + SE), calibration
├── packages/
│ ├── scoring/ # domains, weights, INDEX_VERSION, pure score aggregation (no I/O)
│ ├── items/ # 25 template generators + simulators + robust grading cascade
│ ├── openrouter/ # typed client: retries, timeouts, cost tracking, multimodal
│ └── db/ # Prisma: models, eval_items, model_responses, pairwise_duels, score_runs
├── docs/methodology/ # METHODOLOGY, CHANGELOG (semver), IRT_SPEC, JUDGE_PROTOCOL
└── data/item-bank/ # public template manifest (instantiated items stay server-side)Pipeline: OpenRouter catalog sync → seeded batch generation → parallel evaluation lanes (temperature 0 scored + k consistency samples, full audit trail: raw responses, tokens, latency, cost) → lenient extraction → 2PL fit per domain + Bradley-Terry for duels → scores with 95% CIs → leaderboard updates live → discrimination hygiene auto-flags dead items.
Public API
curl https://www.llmindex.io/api/v1/leaderboard # global ranking + CIs
curl https://www.llmindex.io/api/v1/leaderboard/agentic # per-domain + sub-metrics
curl https://www.llmindex.io/api/v1/models/anthropic/claude-sonnet-5
curl https://www.llmindex.io/api/v1/methodology # machine-readable weights & hyperparams
curl https://www.llmindex.io/api/v1/benchmark/progress # live run statusRate limits: 60 req/min anonymous, 600 with an API key. Breaking changes ship as /api/v2 — v1
shapes are frozen.
Run it
pnpm install
pnpm db:migrate && pnpm db:seed # sync models + pricing from OpenRouter
pnpm dev # web on :3000
pnpm eval:run --model <slug> --domain math --n 30 --k 2
pnpm duel:run --domain svg_design --pairs 100
pnpm benchmark:run --parallel 6 # full live benchmark, refit after every model
pnpm index:refit # IRT + BT refit → new immutable score run
pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test # all gatesIntegrity rules (non-negotiable)
- Answer keys and rubrics never ship to the client or public API.
- Scored batches use freshly perturbed items; the frozen anchor subset is ≤20% of any run.
- Every displayed number traces to an immutable
score_runsrow (item-set hash, model set, index version, fit diagnostics). Batches with >2% failed calls are excluded until re-run. - Weights and IRT hyperparameters live in exactly one file and are served machine-readable.
- Every methodology change bumps the semver
INDEX_VERSIONwith a public changelog entry.
The design draws on 60+ published sources (metabench, GSM-Symbolic, ZebraLogic, R-Horizon, τ-bench, BFCL, Terminal-Bench audits, Math-Verify, ReasonIF, …) — full list with links on the methodology page. Everything here is original: our own environments, items, simulators and graders.
Author & maintainer: Simon-Pierre Boucher · contact@spboucher.ai
License: Proprietary — © Simon-Pierre Boucher, all rights reserved. Source visible for transparency and auditability.