# LLM Index
**The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.**
[**www.llmindex.io**](https://www.llmindex.io) · [Methodology](https://www.llmindex.io/methodology) · [White Paper (PDF)](docs/whitepaper/llmindex-whitepaper.pdf) · [Public API](https://www.llmindex.io/api/v1/leaderboard)
[](https://www.llmindex.io)
[](docs/whitepaper/llmindex-whitepaper.pdf)
---
## Why another index? Because leaderboards are broken.
Classic leaderboards fail twice: **top models cluster at 95%+ on saturated benchmarks** (zero
discrimination), and **fixed test sets leak into training data** (contamination). LLM Index is
engineered against both, from the psychometrics up:
| | |
|---|---|
| 🎯 **IRT 2PL scoring** | Every item has a fitted difficulty *b* and discrimination *a*; ability θ is a MAP estimate with Fisher-information standard errors. Items that don't separate models are auto-retired. Raw accuracy is never the score. |
| 🎲 **Dynamic item generation** | Every scored batch is freshly generated from seeded template generators (values, paraphrases, structures). There is no fixed test set to memorize — the fixed-vs-fresh gap is *published* per model as `contamination_delta`. |
| 🤖 **Home-made agentic bench** | Simulated tool-calling environments (triage-under-policy, treasury ledger, deployment DAGs) with distractor tools; a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges. |
| 🧠 **Agentic under context load** | 120–300-row generated ledgers packed with near-miss decoys: the model must find, order, and act on the few matching records. Prompt length is the difficulty knob. |
| 💻 **Home-made terminal bench** | No shell executes: a closed, unambiguous POSIX subset is simulated in TypeScript. Models predict exact pipeline stdout, file trees after `mv/cp/rm/cd`, and `&&`/`\|\|` exit-code traces. |
| 🎨 **SVG logo duels** | Models reproduce real-world logos in raw SVG from memory; a 3-judge cross-provider panel (position-swapped, never self-judging) feeds a Bradley-Terry fit. |
| 👁 **Vision OCR under clutter** | Generated scenes (rotated codes, noise, low-contrast, decoys) rasterized to PNG; ambiguous glyphs excluded by design. Text-only models skip; weights renormalize. |
| 🛡 **Extraction that never cheats models** | Lenient cascade (`ANSWER:` in any markdown, `FINAL ANSWER`, `\boxed{}`, fenced blocks), number normalization, 16k completion budgets, truncated ≠ wrong. Formatting is measured in its own domain — never silently everywhere. |
| 📡 **Live, one model at a time** | Parallel evaluation lanes stream results; after every completed model the IRT refit re-runs and the public leaderboard re-ranks in real time. |
| 🔍 **Total transparency** | Every model has a page showing **every answer on every test**, judge verdicts, confidence, latency and cost. Every number traces to an immutable score run. Answer keys never leave the server. |
## Screenshots
## The 12 domains
Domain weights are **equal by design** — the maximum-entropy prior; any other weighting is an
editorial value judgment. Per-domain scores are always published so you can re-weight. Cost and
latency are **never** blended into quality: they live on a separate Pareto frontier.
## Architecture
```
llmindex/
├── apps/
│ ├── web/ # Next.js 14 — live leaderboard, transparency pages, /api/v1/*
│ ├── worker/ # eval runner · duel runner · parallel benchmark orchestrator · refit
│ └── psychometrics/ # Python: 2PL IRT (MAP + Fisher SE), Bradley-Terry (MM + SE), calibration
├── packages/
│ ├── scoring/ # domains, weights, INDEX_VERSION, pure score aggregation (no I/O)
│ ├── items/ # 25 template generators + simulators + robust grading cascade
│ ├── openrouter/ # typed client: retries, timeouts, cost tracking, multimodal
│ └── db/ # Prisma: models, eval_items, model_responses, pairwise_duels, score_runs
├── docs/methodology/ # METHODOLOGY, CHANGELOG (semver), IRT_SPEC, JUDGE_PROTOCOL
└── data/item-bank/ # public template manifest (instantiated items stay server-side)
```
**Pipeline:** OpenRouter catalog sync → seeded batch generation → parallel evaluation lanes
(temperature 0 scored + k consistency samples, full audit trail: raw responses, tokens, latency,
cost) → lenient extraction → 2PL fit per domain + Bradley-Terry for duels → scores with 95% CIs →
leaderboard updates live → discrimination hygiene auto-flags dead items.
## Public API
```bash
curl https://www.llmindex.io/api/v1/leaderboard # global ranking + CIs
curl https://www.llmindex.io/api/v1/leaderboard/agentic # per-domain + sub-metrics
curl https://www.llmindex.io/api/v1/models/anthropic/claude-sonnet-5
curl https://www.llmindex.io/api/v1/methodology # machine-readable weights & hyperparams
curl https://www.llmindex.io/api/v1/benchmark/progress # live run status
```
Rate limits: 60 req/min anonymous, 600 with an API key. Breaking changes ship as `/api/v2` — v1
shapes are frozen.
## Run it
```bash
pnpm install
pnpm db:migrate && pnpm db:seed # sync models + pricing from OpenRouter
pnpm dev # web on :3000
pnpm eval:run --model 
