SPB Git

spb/llmindex Public

The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.

webappaillmbenchmark
TypeScript 77.9% TeX 15.2% Python 3.7% SQL 1.4% JavaScript 1.1% Shell 0.5%
10.6 KB
View source Raw Blame History
LLM Index logo

# LLM Index

The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.

www.llmindex.io · Methodology · White Paper (PDF) · Public API

Website Index Live Models Domains Judges White paper

IRT BT CI accuracy consistency contamination calibration pareto

Next.js TypeScript Tailwind Prisma PostgreSQL Redis Python pnpm OpenRouter

# Why another index? Because leaderboards are broken.

Classic leaderboards fail twice: top models cluster at 95%+ on saturated benchmarks (zero discrimination), and fixed test sets leak into training data (contamination). LLM Index is engineered against both, from the psychometrics up:
🎯 IRT 2PL scoring Every item has a fitted difficulty b and discrimination a; ability θ is a MAP estimate with Fisher-information standard errors. Items that don't separate models are auto-retired. Raw accuracy is never the score.
🎲 Dynamic item generation Every scored batch is freshly generated from seeded template generators (values, paraphrases, structures). There is no fixed test set to memorize — the fixed-vs-fresh gap is published per model as contamination_delta.
🤖 Home-made agentic bench Simulated tool-calling environments (triage-under-policy, treasury ledger, deployment DAGs) with distractor tools; a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges.
🧠 Agentic under context load 120–300-row generated ledgers packed with near-miss decoys: the model must find, order, and act on the few matching records. Prompt length is the difficulty knob.
💻 Home-made terminal bench No shell executes: a closed, unambiguous POSIX subset is simulated in TypeScript. Models predict exact pipeline stdout, file trees after mv/cp/rm/cd, and &&/|| exit-code traces.
🎨 SVG logo duels Models reproduce real-world logos in raw SVG from memory; a 3-judge cross-provider panel (position-swapped, never self-judging) feeds a Bradley-Terry fit.
👁 Vision OCR under clutter Generated scenes (rotated codes, noise, low-contrast, decoys) rasterized to PNG; ambiguous glyphs excluded by design. Text-only models skip; weights renormalize.
🛡 Extraction that never cheats models Lenient cascade (ANSWER: in any markdown, FINAL ANSWER, \boxed{}, fenced blocks), number normalization, 16k completion budgets, truncated ≠ wrong. Formatting is measured in its own domain — never silently everywhere.
📡 Live, one model at a time Parallel evaluation lanes stream results; after every completed model the IRT refit re-runs and the public leaderboard re-ranks in real time.
🔍 Total transparency Every model has a page showing every answer on every test, judge verdicts, confidence, latency and cost. Every number traces to an immutable score run. Answer keys never leave the server.

# Screenshots

Live leaderboard — desktop

LLM Index home, live leaderboard

Mobile

Mobile view

Model transparency page

Model profile with every answer

Methodology — public, detailed, with live sample items and full references

Methodology page

# The 12 domains

code math reasoning agentic terminal knowledge multilingual instruction writing safety svg vision

Domain weights are equal by design — the maximum-entropy prior; any other weighting is an editorial value judgment. Per-domain scores are always published so you can re-weight. Cost and latency are never blended into quality: they live on a separate Pareto frontier.

# Architecture

text
llmindex/
├── apps/
│   ├── web/             # Next.js 14 — live leaderboard, transparency pages, /api/v1/*
│   ├── worker/          # eval runner · duel runner · parallel benchmark orchestrator · refit
│   └── psychometrics/   # Python: 2PL IRT (MAP + Fisher SE), Bradley-Terry (MM + SE), calibration
├── packages/
│   ├── scoring/         # domains, weights, INDEX_VERSION, pure score aggregation (no I/O)
│   ├── items/           # 25 template generators + simulators + robust grading cascade
│   ├── openrouter/      # typed client: retries, timeouts, cost tracking, multimodal
│   └── db/              # Prisma: models, eval_items, model_responses, pairwise_duels, score_runs
├── docs/methodology/    # METHODOLOGY, CHANGELOG (semver), IRT_SPEC, JUDGE_PROTOCOL
└── data/item-bank/      # public template manifest (instantiated items stay server-side)

Pipeline: OpenRouter catalog sync → seeded batch generation → parallel evaluation lanes (temperature 0 scored + k consistency samples, full audit trail: raw responses, tokens, latency, cost) → lenient extraction → 2PL fit per domain + Bradley-Terry for duels → scores with 95% CIs → leaderboard updates live → discrimination hygiene auto-flags dead items.

# Public API

bash
curl https://www.llmindex.io/api/v1/leaderboard            # global ranking + CIs
curl https://www.llmindex.io/api/v1/leaderboard/agentic    # per-domain + sub-metrics
curl https://www.llmindex.io/api/v1/models/anthropic/claude-sonnet-5
curl https://www.llmindex.io/api/v1/methodology            # machine-readable weights & hyperparams
curl https://www.llmindex.io/api/v1/benchmark/progress     # live run status

Rate limits: 60 req/min anonymous, 600 with an API key. Breaking changes ship as /api/v2 — v1 shapes are frozen.

# Run it

bash
pnpm install
pnpm db:migrate && pnpm db:seed        # sync models + pricing from OpenRouter
pnpm dev                               # web on :3000

pnpm eval:run --model <slug> --domain math --n 30 --k 2
pnpm duel:run --domain svg_design --pairs 100
pnpm benchmark:run --parallel 6        # full live benchmark, refit after every model
pnpm index:refit                       # IRT + BT refit → new immutable score run

pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test   # all gates

# Integrity rules (non-negotiable)

  1. Answer keys and rubrics never ship to the client or public API.
  2. Scored batches use freshly perturbed items; the frozen anchor subset is ≤20% of any run.
  3. Every displayed number traces to an immutable score_runs row (item-set hash, model set, index version, fit diagnostics). Batches with >2% failed calls are excluded until re-run.
  4. Weights and IRT hyperparameters live in exactly one file and are served machine-readable.
  5. Every methodology change bumps the semver INDEX_VERSION with a public changelog entry.

The design draws on 60+ published sources (metabench, GSM-Symbolic, ZebraLogic, R-Horizon, τ-bench, BFCL, Terminal-Bench audits, Math-Verify, ReasonIF, …) — full list with links on the methodology page. Everything here is original: our own environments, items, simulators and graders.

Author & maintainer: Simon-Pierre Boucher · contact@spboucher.ai

License: Proprietary — © Simon-Pierre Boucher, all rights reserved. Source visible for transparency and auditability.