spb/llmindex Public

1# LLM Index Methodology (v0.1.0)23**Author:** Simon-Pierre Boucher — contact@spboucher.ai45The LLM Index ranks large language models globally and per domain with a methodology built to fix6the two failure modes of classic leaderboards: **no discrimination** (top models cluster at 95%+ on7saturated benchmarks) and **contamination** (test sets leak into training data).89## 1. IRT-based scoring1011Every graded item carries an estimated **difficulty** (b) and **discrimination** (a) under a122-parameter logistic model:1314```15P(correct | θ, a, b) = σ(a · (θ − b))16```1718Model ability **θ** is a MAP estimate with priors θ ~ N(0,1), b ~ N(0,1.5), log a ~ N(0,0.5),19fitted in `apps/psychometrics` (custom numpy 2PL; missing-aware; SE from Fisher information).20Items with `a < 0.3` or `|b| > 3` logits are auto-flagged for retirement review after every refit.2122## 2. Dynamic item generation2324Items are instantiated from **versioned templates** (`packages/items`) with seeded value25substitution and paraphrase perturbation — fresh for every scored batch. A fixed **anchor** stream26(≤20% of any run) is kept identical across runs for longitudinal comparability. The accuracy gap27`anchor − fresh` is published per model/domain as **contamination_delta**; it maps to a28`contamination_resistance` sub-metric via `clamp01(1 − delta / 0.2)`.2930Template shapes are public; instantiated items and answer keys never leave the server.3132## 3. Pairwise Bradley-Terry layer3334Open-ended domains (`writing`, `safety_refusal_quality`) are scored by LLM-judged pairwise duels35feeding a Bradley-Terry model (MM algorithm, ties as half-wins). Judge protocol: see36`JUDGE_PROTOCOL.md` — ≥2 judges from different providers, position swap, agreement rate published,37no self-judging.3839## 4. Consistency4041The same item is asked k times (default temperature); the fraction of samples agreeing with the42modal answer is the **consistency** sub-metric. A model that flips answers is less trustworthy at43equal accuracy.4445## 5. Calibration4647Every graded item requires a reported confidence (0–100). Calibration = 1 − ECE (10 equal-width48bins) per domain. Brier score is also computed in `apps/psychometrics`.4950## 6. Scores5152- Sub-metrics blend into a **domain composite** in [0,1] with weights53  (accuracy_irt 0.55, consistency 0.15, calibration 0.15, contamination_resistance 0.15),54  renormalized over the metrics actually measured. Accuracy enters as σ(θ).55- **Domain score** = 1000 × composite, with a 95% CI propagated from the θ standard error56  (delta method through the accuracy term).57- **Global Index (0–1000)** = weighted mean of domain composites (equal domain weights in v1,58  renormalized over covered domains), CI combined in quadrature.59- **Latency and cost are never blended in.** They are published as an efficiency (Pareto)60  frontier: Global Index vs. cost per 1k items (OpenRouter pricing × measured usage) and measured61  latency p50.6263## 7. Integrity rules64651. Answer keys and rubrics never ship to the client or public API.662. Scored batches use freshly perturbed items; anchors ≤20%.673. Every displayed number traces to an immutable `score_runs` row (model set, item-set hash,68   index version, fit diagnostics). Batches with >2% failed calls are flagged `degraded` and69   excluded from fits until re-run.704. Weights and IRT hyperparameters live only in `packages/scoring/src/weights.ts` and are served71   machine-readable at `/api/v1/methodology`.725. Every methodology change bumps `INDEX_VERSION` (semver) with a changelog entry.7374## Demo seed runs7576Before the first real fit, the platform may display a run of kind `demo_seed`: deterministic,77clearly-banner-labelled illustrative scores used to exercise the UI/API. Demo runs are never mixed78with real fits and disappear from the leaderboard as soon as an `index_fit` run completes.79