# LLM Index Methodology (v0.1.0)

Author: Simon-Pierre Boucher — contact@spboucher.ai

The LLM Index ranks large language models globally and per domain with a methodology built to fix the two failure modes of classic leaderboards: no discrimination (top models cluster at 95%+ on saturated benchmarks) and contamination (test sets leak into training data).

# 1. IRT-based scoring

Every graded item carries an estimated difficulty (b) and discrimination (a) under a 2-parameter logistic model:

text

P(correct | θ, a, b) = σ(a · (θ − b))

Model ability θ is a MAP estimate with priors θ ~ N(0,1), b ~ N(0,1.5), log a ~ N(0,0.5), fitted in apps/psychometrics (custom numpy 2PL; missing-aware; SE from Fisher information). Items with a < 0.3 or |b| > 3 logits are auto-flagged for retirement review after every refit.

# 2. Dynamic item generation

Items are instantiated from versioned templates (packages/items) with seeded value substitution and paraphrase perturbation — fresh for every scored batch. A fixed anchor stream (≤20% of any run) is kept identical across runs for longitudinal comparability. The accuracy gap anchor − fresh is published per model/domain as contamination_delta; it maps to a contamination_resistance sub-metric via clamp01(1 − delta / 0.2).

Template shapes are public; instantiated items and answer keys never leave the server.

# 3. Pairwise Bradley-Terry layer

Open-ended domains (writing, safety_refusal_quality) are scored by LLM-judged pairwise duels feeding a Bradley-Terry model (MM algorithm, ties as half-wins). Judge protocol: see JUDGE_PROTOCOL.md — ≥2 judges from different providers, position swap, agreement rate published, no self-judging.

# 4. Consistency

The same item is asked k times (default temperature); the fraction of samples agreeing with the modal answer is the consistency sub-metric. A model that flips answers is less trustworthy at equal accuracy.

# 5. Calibration

Every graded item requires a reported confidence (0–100). Calibration = 1 − ECE (10 equal-width bins) per domain. Brier score is also computed in apps/psychometrics.

# 6. Scores

Sub-metrics blend into a domain composite in [0,1] with weights (accuracy_irt 0.55, consistency 0.15, calibration 0.15, contamination_resistance 0.15), renormalized over the metrics actually measured. Accuracy enters as σ(θ).
Domain score = 1000 × composite, with a 95% CI propagated from the θ standard error (delta method through the accuracy term).
Global Index (0–1000) = weighted mean of domain composites (equal domain weights in v1, renormalized over covered domains), CI combined in quadrature.
Latency and cost are never blended in. They are published as an efficiency (Pareto) frontier: Global Index vs. cost per 1k items (OpenRouter pricing × measured usage) and measured latency p50.

# 7. Integrity rules

Answer keys and rubrics never ship to the client or public API.
Scored batches use freshly perturbed items; anchors ≤20%.
Every displayed number traces to an immutable score_runs row (model set, item-set hash, index version, fit diagnostics). Batches with >2% failed calls are flagged degraded and excluded from fits until re-run.
Weights and IRT hyperparameters live only in packages/scoring/src/weights.ts and are served machine-readable at /api/v1/methodology.
Every methodology change bumps INDEX_VERSION (semver) with a changelog entry.

# Demo seed runs

Before the first real fit, the platform may display a run of kind demo_seed: deterministic, clearly-banner-labelled illustrative scores used to exercise the UI/API. Demo runs are never mixed with real fits and disappear from the leaderboard as soon as an index_fit run completes.