SPB Git

spb/llmindex Public

The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.

TypeScript 77.9% TeX 15.2% Python 3.7% SQL 1.4% JavaScript 1.1% Shell 0.5%
1.4 KB

# IRT Specification (2PL)

Author: Simon-Pierre Boucher — contact@spboucher.ai

# Model

For model m and item i: P(x_mi = 1) = σ(a_i (θ_m − b_i)).

# Estimation

MAP via Adam gradient ascent on the penalized log-likelihood (apps/psychometrics/llmindex_psycho/irt.py):

  • Priors: θ ~ N(0, 1); b ~ N(0, 1.5); log a ~ N(0, 0.5) (a > 0 by construction).
  • Warm start: θ from row-accuracy logits (centered), b from inverted column-accuracy logits.
  • Identification: θ recentered to mean 0 each step (shift absorbed into b); scale pinned by priors.
  • Missing cells masked (models need not share identical item sets).
  • Convergence: relative objective change < tolerance (default 1e-6), max 500 iterations.

# Uncertainty

SE(θ_m) = 1/√(Σ_i a_i² P(1−P) + 1/σ_θ²) over observed items (Fisher information + prior precision). Published as 95% CIs on every score.

# Hygiene

After each refit: items with a < 0.3 or |b| > 3 are flagged flagged_for_retirement in the item bank and reviewed before the next run. Fit diagnostics (iterations, convergence, final log-likelihood, matrix sizes) are stored on the score_runs row.

# Deviation note

The spec allows py-irt / PyTorch; v1 ships a dependency-light custom numpy implementation with the same 2PL likelihood and priors — verified by parameter-recovery tests (apps/psychometrics/tests/test_psycho.py).