# LLM Index Methodology (v0.1.0)

**Author:** Simon-Pierre Boucher — contact@spboucher.ai

The LLM Index ranks large language models globally and per domain with a methodology built to fix
the two failure modes of classic leaderboards: **no discrimination** (top models cluster at 95%+ on
saturated benchmarks) and **contamination** (test sets leak into training data).

## 1. IRT-based scoring

Every graded item carries an estimated **difficulty** (b) and **discrimination** (a) under a
2-parameter logistic model:

```
P(correct | θ, a, b) = σ(a · (θ − b))
```

Model ability **θ** is a MAP estimate with priors θ ~ N(0,1), b ~ N(0,1.5), log a ~ N(0,0.5),
fitted in `apps/psychometrics` (custom numpy 2PL; missing-aware; SE from Fisher information).
Items with `a < 0.3` or `|b| > 3` logits are auto-flagged for retirement review after every refit.

## 2. Dynamic item generation

Items are instantiated from **versioned templates** (`packages/items`) with seeded value
substitution and paraphrase perturbation — fresh for every scored batch. A fixed **anchor** stream
(≤20% of any run) is kept identical across runs for longitudinal comparability. The accuracy gap
`anchor − fresh` is published per model/domain as **contamination_delta**; it maps to a
`contamination_resistance` sub-metric via `clamp01(1 − delta / 0.2)`.

Template shapes are public; instantiated items and answer keys never leave the server.

## 3. Pairwise Bradley-Terry layer

Open-ended domains (`writing`, `safety_refusal_quality`) are scored by LLM-judged pairwise duels
feeding a Bradley-Terry model (MM algorithm, ties as half-wins). Judge protocol: see
`JUDGE_PROTOCOL.md` — ≥2 judges from different providers, position swap, agreement rate published,
no self-judging.

## 4. Consistency

The same item is asked k times (default temperature); the fraction of samples agreeing with the
modal answer is the **consistency** sub-metric. A model that flips answers is less trustworthy at
equal accuracy.

## 5. Calibration

Every graded item requires a reported confidence (0–100). Calibration = 1 − ECE (10 equal-width
bins) per domain. Brier score is also computed in `apps/psychometrics`.

## 6. Scores

- Sub-metrics blend into a **domain composite** in [0,1] with weights
  (accuracy_irt 0.55, consistency 0.15, calibration 0.15, contamination_resistance 0.15),
  renormalized over the metrics actually measured. Accuracy enters as σ(θ).
- **Domain score** = 1000 × composite, with a 95% CI propagated from the θ standard error
  (delta method through the accuracy term).
- **Global Index (0–1000)** = weighted mean of domain composites (equal domain weights in v1,
  renormalized over covered domains), CI combined in quadrature.
- **Latency and cost are never blended in.** They are published as an efficiency (Pareto)
  frontier: Global Index vs. cost per 1k items (OpenRouter pricing × measured usage) and measured
  latency p50.

## 7. Integrity rules

1. Answer keys and rubrics never ship to the client or public API.
2. Scored batches use freshly perturbed items; anchors ≤20%.
3. Every displayed number traces to an immutable `score_runs` row (model set, item-set hash,
   index version, fit diagnostics). Batches with >2% failed calls are flagged `degraded` and
   excluded from fits until re-run.
4. Weights and IRT hyperparameters live only in `packages/scoring/src/weights.ts` and are served
   machine-readable at `/api/v1/methodology`.
5. Every methodology change bumps `INDEX_VERSION` (semver) with a changelog entry.

## Demo seed runs

Before the first real fit, the platform may display a run of kind `demo_seed`: deterministic,
clearly-banner-labelled illustrative scores used to exercise the UI/API. Demo runs are never mixed
with real fits and disappear from the leaderboard as soon as an `index_fit` run completes.