spb/llmindex Public

1% llmindex.io — methodology white paper2% Author:  Simon-Pierre Boucher3% Contact: contact@spboucher.ai4% License: Proprietary — © Simon-Pierre Boucher, all rights reserved5% Build:   latexmk -pdf llmindex-whitepaper.tex6\documentclass[11pt,a4paper]{article}78\usepackage[utf8]{inputenc}9\usepackage[T1]{fontenc}10\usepackage{lmodern}11\usepackage[margin=2.6cm]{geometry}12\usepackage{amsmath,amssymb,amsthm}13\usepackage{booktabs}14\usepackage{array}15\usepackage{longtable}16\usepackage{microtype}17\usepackage{xcolor}18\usepackage{enumitem}19\usepackage[numbers,sort&compress]{natbib}20\usepackage{tikz}21\usetikzlibrary{arrows.meta,positioning,shapes.misc,calc,fit,backgrounds}22\usepackage{pgfplots}23\pgfplotsset{compat=1.18}24\usepackage{pdflscape}25\usepackage{fancyhdr}26\usepackage{titlesec}27\usepackage[colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black]{hyperref}28\usepackage{setspace}2930\definecolor{llmgreen}{HTML}{10B981}31\definecolor{llmcyan}{HTML}{22D3EE}32\definecolor{llmdark}{HTML}{09090B}33\definecolor{llmgray}{HTML}{71717A}34\definecolor{llmamber}{HTML}{F59E0B}35\definecolor{llmviolet}{HTML}{8B5CF6}3637% Colored, sharp section titles38\titleformat{\section}{\Large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesection}}{0.8em}{}[{\color{llmgreen!60}\titlerule[0.8pt]}]39\titleformat{\subsection}{\large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesubsection}}{0.7em}{}4041% Running headers42\pagestyle{fancy}43\fancyhf{}44\fancyhead[L]{\small\textsc{LLMindex} --- Methodology White Paper}45\fancyhead[R]{\small v\version}46\fancyfoot[C]{\small\thepage}47\renewcommand{\headrulewidth}{0.4pt}4849% The LLMindex mark, redrawn in TikZ (ascending bars + index needle)50\newcommand{\llmlogo}[1][1.4cm]{%51\begin{tikzpicture}[x=#1/6.4, y=#1/6.4, baseline]52  \fill[llmdark, rounded corners=0.22*#1/1.4cm] (0,0) rectangle (6.4,6.4);53  \fill[llmgray!45, rounded corners=1pt] (1.10,1.10) rectangle (2.00,2.70);54  \fill[llmgray!75, rounded corners=1pt] (2.40,1.10) rectangle (3.30,3.50);55  \fill[llmgreen,  rounded corners=1pt] (3.70,1.10) rectangle (4.60,4.50);56  \draw[line width=0.09*#1/1.4cm, llmcyan, line cap=round] (1.30,3.40) -- (4.70,5.20);57  \fill[llmcyan] (4.90,5.30) circle (0.32);58\end{tikzpicture}}5960% Common TikZ styles for diagrams61\tikzset{62  stage/.style={draw=llmdark!60, fill=llmgreen!8, rounded corners=2pt, align=center,63                font=\small, inner sep=5pt, minimum height=9mm},64  stageB/.style={draw=llmdark!60, fill=llmcyan!10, rounded corners=2pt, align=center,65                 font=\small, inner sep=5pt, minimum height=9mm},66  flow/.style={-{Stealth[length=2.2mm]}, thick, llmdark!70},67  note/.style={font=\scriptsize\itshape, text=llmgray}68}6970\newcommand{\llmindex}{\textsc{LLMindex}}71\newcommand{\version}{0.2.0}72\newcommand{\clampzo}{\operatorname{clamp}_{[0,1]}}7374\setlength{\parskip}{0.35em}7576\title{\vspace{-2.2em}\llmlogo[2.2cm]\\[0.9em]\Huge \llmindex\\[0.35em]77\Large A Psychometric, Contamination-Resistant, and Fully Auditable\\ Methodology for Ranking Large Language Models\\[0.6em]78\normalsize Methodology White Paper --- Index Version \version}79\author{Simon-Pierre Boucher\\[0.2em]80\normalsize \href{mailto:contact@spboucher.ai}{\texttt{contact@spboucher.ai}}\\81\normalsize \href{https://www.llmindex.io}{\texttt{www.llmindex.io}}}82\date{August 2026}8384\begin{document}8586\maketitle8788\begin{abstract}89\noindent90Public leaderboards have become the de facto interface between frontier language-model development and its consumers, yet the dominant evaluation paradigm is failing in two structural ways: \emph{saturation}, in which top models compress into statistically indistinguishable score bands on static benchmarks, and \emph{contamination}, in which fixed test sets leak into training corpora and convert measurement into memorization. \llmindex{} is an operating evaluation platform built to address both failures at the level of measurement design rather than test-set curation. Every scored item is instantiated at evaluation time from seeded, versioned template generators, so no fixed test set exists to memorize; ability is estimated with a two-parameter logistic (2PL) item response model rather than raw accuracy, so uninformative items are automatically identified and retired; open-ended capabilities are ranked through pairwise duels judged by a cross-provider panel with position-swap bias controls and aggregated with a Bradley--Terry model; and answer stability, confidence calibration, and a directly measured memorization gap enter the score as explicit, separately reported sub-metrics. Cost and latency are published as a Pareto frontier and are never blended into quality. All scores carry 95\% confidence intervals, every displayed number traces to an immutable, fully audited evaluation run, and the methodology itself is versioned under semantic versioning with a public changelog. This paper motivates the design, develops its mathematical foundations, explains how scores should be interpreted, situates the platform among existing evaluation efforts, and states its limitations and roadmap candidly.91\end{abstract}9293\vspace{0.5em}94\begin{center}95\small\textit{Keywords: large language models, evaluation, item response theory, Bradley--Terry, benchmark contamination, calibration, LLM-as-judge, agentic evaluation}96\end{center}9798\newpage99\tableofcontents100\newpage101102% =====================================================================103\section{Introduction: Why Current Benchmarks Fall Short}104\label{sec:intro}105106Language-model benchmarks began as research instruments and became market infrastructure. Procurement decisions, deployment choices, and public narratives about model progress now route through a handful of numbers. Those numbers are increasingly unreliable, for reasons that are structural rather than incidental.107108\subsection{Saturation and the collapse of discrimination}109110A benchmark discriminates only where models can fail. On the static suites that dominate public discourse --- MMLU~\citep{hendrycks2021mmlu}, GSM8K, HumanEval and their descendants --- frontier models now cluster above 90\% accuracy. From a measurement-theoretic standpoint the consequence is precise: the Fisher information an item contributes to an ability estimate is proportional to $P(1-P)$, where $P$ is the probability the examinee answers correctly (Section~\ref{sec:irt}). An item every strong model solves contributes essentially \emph{zero} information about how strong models differ. Empirically, sparse-benchmark analyses find that fewer than 3\% of items on major benchmarks carry nearly all of their evaluative information~\citep{kipnis2025metabench}, and adaptive-testing studies show that replacing raw accuracy with latent-ability estimation reorders a substantial fraction of model rankings~\citep{polo2024tinybenchmarks}. Raw accuracy on a saturated suite is not a noisy version of the right answer; it is dominated by items that no longer measure anything.111112\subsection{Contamination converts measurement into memorization}113114Static test sets published on the public internet are, with high probability, present in the pretraining corpora of the models they are meant to evaluate~\citep{oren2024proving}. The clearest evidence is perturbational: when the numeric values of GSM8K-style templates are re-randomized while the logical structure is preserved, measured accuracy drops materially and variance across instantiations inflates~\citep{mirzadeh2024gsmsymbolic}. A model that solves the published instance but not its structural twin has memorized, not reasoned; a benchmark that cannot distinguish these two has lost its object of measurement.115116\subsection{Format compliance masquerades as capability}117118Evaluation harnesses grade what they can parse. Rigid answer-extraction rules --- a single accepted tag, strict multiple-choice letter matching --- systematically penalize model families whose reasoning-tuned outputs wrap answers in markdown, \LaTeX{} \verb|\boxed{}| expressions, or free-form prose. When the Open LLM Leaderboard replaced strict extraction with a lenient mathematical-equivalence grader, some model families' scores tripled and the top of the ranking reshuffled --- with no change to any model. Regex-based extraction has been measured at only $\sim$74\% accuracy~\citep{yu2024xfinder}; formatting alone can move measured accuracy by tens of points~\citep{sclar2024formatspread}; and reasoning-tuned models comply with embedded format instructions less than 25\% of the time~\citep{tian2023just,zhou2023ifeval}. A harness that conflates parsing with ability measures its own regexes.119120\subsection{Single numbers hide the trade-offs that matter}121122A model that is slightly less accurate but five times cheaper, or equally accurate but incapable of saying \emph{how sure it is}, is a materially different engineering proposition. Blending accuracy, price, speed, and reliability into one scalar destroys exactly the information a practitioner needs. Similarly, a point estimate without an interval invites over-reading: a three-point gap between two models is meaningless if the measurement uncertainty is ten points, yet leaderboards routinely present such gaps as rank orderings.123124\subsection{Opacity prevents correction}125126Most public rankings cannot answer elementary audit questions: Which exact prompts were used? What did the model actually reply? Under which grader version? What changed between last month's score and today's? Without an immutable audit trail, errors persist silently and disputes cannot be adjudicated on evidence.127128\subsection{The \llmindex{} response}129130\llmindex{} responds to each failure at the level of design rather than patching:131132\begin{enumerate}[leftmargin=1.6em,itemsep=0.15em]133  \item \textbf{Against saturation}: items are \emph{generated} with difficulty knobs calibrated so that frontier models fail a substantial fraction of them, and ability is estimated with a 2PL item response model in which uninformative items are automatically flagged and retired (Section~\ref{sec:irt}).134  \item \textbf{Against contamination}: every scored batch is freshly instantiated from seeded template generators; a small frozen ``anchor'' stream measures the memorization gap directly, and that gap is a published, score-relevant quantity (Section~\ref{sec:items}).135  \item \textbf{Against format confounds}: a lenient, ordered extraction cascade with truncation-aware grading separates parsing from ability; format compliance is measured in its own domain, never silently everywhere (Section~\ref{sec:extraction}).136  \item \textbf{Against scalar blending}: quality, cost, and latency are never merged; the efficiency frontier is published as a Pareto set, every score carries a 95\% confidence interval, and per-domain scores are always available for user-side re-weighting (Section~\ref{sec:scores}).137  \item \textbf{Against opacity}: every displayed number traces to an immutable run record with stored raw responses; every model has a public page exposing every graded answer and every judge verdict; the methodology is versioned with a public changelog (Section~\ref{sec:governance}).138\end{enumerate}139140% =====================================================================141\section{Design Principles}142\label{sec:principles}143144Seven principles govern every design decision in the platform.145146\paragraph{P1 --- Measure ability, not accuracy.} Accuracy conflates the difficulty of the item set with the capability of the model. Latent-ability estimation under an item response model separates the two and yields principled uncertainty (Section~\ref{sec:irt}).147148\paragraph{P2 --- Generate, never fix.} A test item that exists before evaluation time can be memorized. Items are therefore functions (seeded generators), not data. The template \emph{code} is public --- transparency lives at the level of the generating process --- while instantiated items and answer keys remain server-side.149150\paragraph{P3 --- Grade mechanically wherever possible.} Every closed-ended domain is graded by deterministic computation: normalized string equality, numeric tolerance, canonical-JSON equality of tool-call sequences, exact multi-line output matching, or constraint-checker stacks. LLM judges are confined to genuinely open-ended domains, and there they are treated as noisy instruments to be bias-controlled and audited, not as oracles (Section~\ref{sec:duels}).151152\paragraph{P4 --- Target the information optimum.} Item information peaks where solve probability is near one half. Generators expose difficulty knobs (composition depth, distractor density, counterfactual rules, context load) whose settings are chosen so strong models operate near that optimum.153154\paragraph{P5 --- Report uncertainty or report nothing.} Every published score carries a 95\% interval derived from the Fisher information of the underlying fit. Interfaces are required to render the interval wherever the score appears.155156\paragraph{P6 --- Never blend incommensurables.} Quality sub-metrics with a shared construct (ability, stability, honesty, memorization-resistance) may be combined under published weights; cost and latency may not. The efficiency frontier is a set, not a number.157158\paragraph{P7 --- History is immutable; method is versioned.} Runs are append-only. Corrections happen by new runs under a bumped semantic version with a changelog entry, never by editing the past.159160% =====================================================================161\section{System Overview}162\label{sec:system}163164\subsection{Architecture}165166\llmindex{} is a monorepo comprising a public web application and API; an evaluation worker executing batches, duels, and refits; a psychometrics engine (2PL and Bradley--Terry fitting with uncertainty quantification); a pure scoring library holding domain definitions, weights, and the index version as the single source of truth; and an item bank of template generators with their embedded simulators.167168\subsection{Unified model access}169170All evaluated models are reached through a single OpenAI-compatible aggregation layer (OpenRouter), which provides a uniform chat interface, per-token pricing metadata synchronized daily, and multimodal input for vision-capable models. Every call is issued with explicit generation parameters --- temperature $0$ for scored samples, the model's default temperature for consistency samples --- and a completion budget of up to 16{,}384 tokens, capped per model by its context window so that reasoning-tuned models are never silently truncated into failure. Transient errors (HTTP 429 and 5xx) are retried up to five times with exponential backoff; a batch in which more than 2\% of calls ultimately fail is flagged \emph{degraded} and excluded from every fit until re-run.171172\subsection{Audit trail}173174Each model call persists an audit row containing the exact request parameters, the raw provider response, extracted answer, grading verdict, self-reported confidence, token usage, measured wall-clock latency, and measured cost. A ranking whose raw responses are not stored is, by the platform's own rules, invalid. This audit surface is public: every model's page exposes every graded answer of the current index version, and every judged duel with its verdict, judge identity, and position-swap flag.175176\subsection{Evaluation pipeline}177178Figure~\ref{fig:pipeline} shows the end-to-end flow from seeded generation to the live leaderboard.179180\begin{figure}[t]181\centering182\resizebox{\textwidth}{!}{%183\begin{tikzpicture}[node distance=4mm and 7mm]184  \node[stage] (gen) {Seeded template\\generators};185  \node[stage, right=of gen] (adm) {Administration\\{\scriptsize $T{=}0$ scored $+\,k$ samples}};186  \node[stage, right=of adm] (ext) {Extraction\\cascade};187  \node[stage, right=of ext] (grd) {Mechanical\\grading};188  \node[stage, right=of grd] (irt) {2PL IRT fit\\{\scriptsize $\hat\theta,\ \mathrm{SE},\ a_i, b_i$}};189  \node[stageB, below=9mm of adm] (duel) {Pairwise duels\\{\scriptsize generated open tasks}};190  \node[stageB, right=of duel] (judge) {3-judge panel\\{\scriptsize swap, no self-judging}};191  \node[stageB, right=of judge] (bt) {Bradley--Terry\\{\scriptsize $\log p_i,\ \mathrm{SE}$}};192  \node[stage, fill=llmamber!15, right=9mm of irt, yshift=-8.5mm] (score) {Scores + 95\% CI\\{\scriptsize composite, global}};193  \node[stage, fill=llmviolet!12, below=4mm of score] (live) {Live leaderboard\\{\scriptsize refit per model}};194  \draw[flow] (gen) -- (adm);195  \draw[flow] (adm) -- (ext);196  \draw[flow] (ext) -- (grd);197  \draw[flow] (grd) -- (irt);198  \draw[flow] (gen.south) |- (duel.west);199  \draw[flow] (duel) -- (judge);200  \draw[flow] (judge) -- (bt);201  \draw[flow] (irt.east) -- (score.north west);202  \draw[flow] (bt.east) -| ($(score.south west)+(2mm,0)$);203  \draw[flow] (score) -- (live);204  \node[note, below=1.2mm of duel.south west, anchor=north west, xshift=-2mm]205    {every call audited: raw response, tokens, latency, cost};206\end{tikzpicture}}207\caption{The \llmindex{} evaluation pipeline. Closed-ended domains flow through mechanical grading into the 2PL layer; open-ended domains flow through bias-controlled judged duels into the Bradley--Terry layer; both meet in a uniform score construction with published uncertainty.}208\label{fig:pipeline}209\end{figure}210211\subsection{Evaluation domains}212213Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed-ended domains graded mechanically under the IRT layer, and three open-ended domains ranked by judged duels under the Bradley--Terry layer.214215\begin{table}[t]216\centering217\small218\begin{tabular}{@{}p{3.7cm}p{8.0cm}p{2.6cm}@{}}219\toprule220\textbf{Domain} & \textbf{Item family (generated)} & \textbf{Grading} \\221\midrule222code & program-trace prediction incl.\ nested control flow & numeric \\223math & deep chains, linear systems, counterfactual base-$k$, chained subproblems & numeric/exact \\224reasoning & 7-entity order deduction, positional puzzles, decoy entities & exact \\225agentic & simulated tool-calling under policy; context-load ledgers & canonical JSON \\226terminal & simulated POSIX subset: pipelines, file trees, exit-code chains & exact lines \\227knowledge & free-response curated facts (no options, no guessing floor) & exact \\228multilingual & number words 0--999, French/Spanish, both directions & exact/numeric \\229instruction\_\-following & repetition/acronym tasks; 5-constraint stacks & exact/checkers \\230vision\_ocr & generated cluttered scenes, rasterized; table reading & exact/numeric \\231\midrule232writing & constrained creative briefs & duels + BT \\233safety\_\-refusal\_\-quality & delicate gray-zone assistance scenarios & duels + BT \\234svg\_design & reproduce real logos as raw SVG from memory & duels + BT \\235\bottomrule236\end{tabular}237\caption{The twelve scored domains of index version \version. The upper block is graded mechanically and scored by 2PL IRT; the lower block is ranked by cross-provider judged duels aggregated with Bradley--Terry.}238\label{tab:domains}239\end{table}240241Two domain families deserve elaboration because they are fully home-made simulations. The \textbf{agentic} family presents a mock tool catalog salted with distractor tools, a world state, and deterministic business rules (escalation policies, overdraft pre-funding, dependency-ordered deployment); a built-in simulator computes the unique correct call sequence, and grading is exact canonical-JSON equality --- a design informed by the state-comparison graders of $\tau$-bench~\citep{yao2024taubench} and BFCL~\citep{patil2025bfcl} but requiring no live infrastructure and admitting no reward hacking. A dedicated \emph{context-load} template buries the handful of policy-relevant records inside a generated ledger of 120--300 near-miss decoys, making prompt length itself the difficulty knob. The \textbf{terminal} family never executes a shell: a closed, deliberately unambiguous POSIX subset (fixed-string \texttt{grep}, \texttt{cut}, byte-order C-locale \texttt{sort}, integer-only \texttt{awk} aggregation, \texttt{head}/\texttt{tail}) is simulated in TypeScript, following the output-prediction paradigm of CRUXEval~\citep{gu2024cruxeval}; constructs with locale-dependent or implementation-divergent semantics are excluded by construction~\citep{greenberg2020smoosh}, so every answer is unambiguous, and the container-escape and answer-key-reading exploits documented against agent-executed terminal benchmarks~\citep{tbench2025} are impossible by design.242243% =====================================================================244\section{Item Generation and Contamination Resistance}245\label{sec:items}246247\subsection{Seeded template generators}248249An item template is a pure function $g_T(\sigma) \mapsto (\text{prompt}, \text{key}, \text{grading mode})$ from a seed string $\sigma$ to an instantiated item. Seeds are expanded through a deterministic PRNG (a 32-bit mixing generator keyed by an FNV-1a hash of the seed), so any batch is exactly reproducible from its recorded seed, yet unpredictable before the run. Templates randomize surface values, entity names, paraphrase frames, and structural parameters; the space of instantiations per template ranges from $10^4$ to beyond $10^{12}$.250251\subsection{Difficulty engineering}252253Because generation is programmatic, difficulty is a controllable parameter rather than an accident of curation. The generators implement difficulty mechanisms whose discriminative power is documented in the literature: multi-step dependent computation in which errors compound multiplicatively~\citep{dziri2023faith}; chained sub-problems whose intermediate answers feed forward, converting small per-step differences into large end-to-end separations~\citep{yu2025rhorizon}; provably inert distractor clauses whose quantities never enter the computation~\citep{shi2023distracted,mirzadeh2024gsmsymbolic}; counterfactual rule perturbations such as base-$k$ arithmetic that separate rule-following computation from memorized decimal facts~\citep{wu2024reasoning}; large constraint-satisfaction and deduction spaces~\citep{lin2025zebralogic}; and stacked, individually verifiable output constraints~\citep{zhou2023ifeval}. Free-response formats are preferred over multiple choice wherever an unambiguous key exists, eliminating the guessing floor that both inflates scores and depresses item information.254255\subsection{Anchors and the measured memorization gap}256257A fixed fraction of each batch --- at most 20\%, enforced in code --- derives from a \emph{frozen} seed stream that is byte-identical in every run, forever. These anchor items serve longitudinal comparability and, more importantly, turn contamination from a suspicion into a measurement:258\begin{equation}259\Delta_{\text{contam}} \;=\; \operatorname{acc}(\text{anchor items}) - \operatorname{acc}(\text{fresh items}),260\qquad261R \;=\; \clampzo\!\left(1 - \frac{\Delta_{\text{contam}}}{0.2}\right),262\label{eq:contam}263\end{equation}264where $R$ is the contamination-resistance sub-metric entering the domain composite (Section~\ref{sec:scores}). A model that performs 20 accuracy points better on the fixed stream than on structurally identical fresh instantiations receives zero resistance credit. Because anchors are never published and fresh items never repeat, the only path to a high score is capability on unseen instances.265266% =====================================================================267\section{Psychometric Scoring: the 2PL Layer}268\label{sec:irt}269270\subsection{Model}271272Let $x_{mi} \in \{0,1\}$ indicate whether model $m$ answered item $i$ correctly. \llmindex{} adopts the two-parameter logistic model~\citep{birnbaum1968some,embretson2000item}:273\begin{equation}274P(x_{mi}=1 \mid \theta_m, a_i, b_i) \;=\; \sigma\!\big(a_i(\theta_m - b_i)\big)275\;=\; \frac{1}{1+e^{-a_i(\theta_m-b_i)}},276\label{eq:2pl}277\end{equation}278where $\theta_m$ is the latent ability of model $m$, $b_i$ the item difficulty (in logits), and $a_i>0$ the item discrimination --- the steepness with which the item separates abilities around $b_i$. Figure~\ref{fig:icc} illustrates both the item characteristic curves and the induced information functions.279280\begin{figure}[t]281\centering282\begin{tikzpicture}283\begin{axis}[284  width=0.48\textwidth, height=5.2cm,285  xlabel={ability $\theta$}, ylabel={$P(\text{correct})$},286  xmin=-4, xmax=4, ymin=0, ymax=1,287  legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none},288  grid=major, grid style={llmgray!20},289  tick label style={font=\scriptsize}, label style={font=\small},290  domain=-4:4, samples=120]291\addplot[llmgray!80, thick] {1/(1+exp(-0.4*x))};292\addlegendentry{$a{=}0.4,\ b{=}0$ (flagged)}293\addplot[llmgreen, very thick] {1/(1+exp(-1.2*x))};294\addlegendentry{$a{=}1.2,\ b{=}0$}295\addplot[llmcyan!80!black, very thick, dashed] {1/(1+exp(-2.2*(x-1.2)))};296\addlegendentry{$a{=}2.2,\ b{=}1.2$}297\end{axis}298\end{tikzpicture}\hfill299\begin{tikzpicture}300\begin{axis}[301  width=0.48\textwidth, height=5.2cm,302  xlabel={ability $\theta$}, ylabel={information $I(\theta)$},303  xmin=-4, xmax=4, ymin=0, ymax=1.3,304  legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none},305  grid=major, grid style={llmgray!20},306  tick label style={font=\scriptsize}, label style={font=\small},307  domain=-4:4, samples=120]308\addplot[llmgray!80, thick] {0.16*exp(-0.4*x)/(1+exp(-0.4*x))^2};309\addlegendentry{$a{=}0.4$: near-zero everywhere}310\addplot[llmgreen, very thick] {1.44*exp(-1.2*x)/(1+exp(-1.2*x))^2};311\addlegendentry{$a{=}1.2$}312\addplot[llmcyan!80!black, very thick, dashed] {4.84*exp(-2.2*(x-1.2))/(1+exp(-2.2*(x-1.2)))^2};313\addlegendentry{$a{=}2.2$: sharp, at frontier $\theta$}314\draw[llmamber, thick, dotted] (axis cs:1.2,0) -- (axis cs:1.2,1.21);315\node[note, anchor=west] at (axis cs:1.35,1.05) {peak at $P{=}\tfrac12$};316\end{axis}317\end{tikzpicture}318\caption{Left: 2PL item characteristic curves. Right: the corresponding item information $I(\theta)=a_i^2P(1-P)$. A flat item ($a{=}0.4$) contributes almost nothing anywhere and is auto-flagged; a discriminative item placed at frontier ability ($a{=}2.2$, $b{=}1.2$) measures precisely where saturated benchmarks are blind.}319\label{fig:icc}320\end{figure}321322\subsection{Estimation}323324Parameters are estimated per domain by maximum a posteriori over the observed (possibly incomplete) response matrix:325\begin{equation}326\mathcal{L} = \sum_{(m,i)\,\in\,\text{obs}} \Big[ x_{mi}\log P_{mi} + (1-x_{mi})\log(1-P_{mi}) \Big]327- \sum_m \frac{\theta_m^2}{2\sigma_\theta^2}328- \sum_i \frac{b_i^2}{2\sigma_b^2}329- \sum_i \frac{(\log a_i)^2}{2\sigma_{\log a}^2},330\label{eq:map}331\end{equation}332with priors $\theta \sim \mathcal{N}(0,1)$, $b \sim \mathcal{N}(0,1.5^2)$, and $\log a \sim \mathcal{N}(0,0.5^2)$; the log-normal prior on $a$ enforces positivity and regularizes the small-sample regime. Optimization uses Adam on the penalized likelihood with warm starts from row- and column-accuracy logits; abilities are re-centered to mean zero after each step (the shift being absorbed into $b$), with scale pinned by the priors. Convergence is declared at a relative objective change below $10^{-6}$, capped at 500 iterations, and convergence diagnostics are stored with every fit.333334\subsection{Uncertainty}335336The standard error of each ability follows from the Fisher information of the 2PL likelihood at the estimate, plus the prior precision:337\begin{equation}338\operatorname{SE}(\theta_m) = \Big( \textstyle\sum_{i \in \text{obs}(m)} a_i^2\, P_{mi}(1-P_{mi}) \;+\; \sigma_\theta^{-2} \Big)^{-1/2}.339\label{eq:se}340\end{equation}341Equation~\eqref{eq:se} is also the design compass of the platform: the summand $a_i^2 P(1-P)$ is the \emph{item information function}, maximal at $P=\tfrac12$ and vanishing at the extremes. Generating items whose difficulty places strong models near $P \approx 0.5$ is not sadism but statistics --- it is where measurement is most efficient.342343\subsection{Item hygiene}344345After every refit, items with fitted discrimination $a_i < 0.3$ or difficulty $|b_i| > 3$ logits are automatically flagged for retirement review; flagged template-parameter regions are examined before subsequent runs. Dead items are thereby prevented from silently diluting the index --- the failure mode that afflicts every static benchmark as models improve.346347% =====================================================================348\section{Pairwise Duels and the Bradley--Terry Layer}349\label{sec:duels}350351Open-ended capabilities --- constrained creative writing, the quality of refusal behavior in delicate gray-zone situations, and the reproduction of real-world logos as raw SVG code --- admit no mechanical key. For these domains \llmindex{} uses pairwise comparison, the measurement design with the longest pedigree in preference scaling~\citep{bradley1952rank,elo1978rating} and the design underlying Chatbot Arena~\citep{chiang2024chatbot}, but with LLM judges under explicit bias controls in place of crowdsourced votes.352353\subsection{Judge protocol}354355Each duel presents the same generated task to two models; their responses are then rated by a panel of three judge models drawn from at least two different providers. The protocol enforces: (i) \emph{no self-judging} --- a judge that is a duel participant is excluded from that duel, and duels with fewer than two eligible judges are discarded; (ii) \emph{position swapping} --- judges alternate which response appears first, the swap is recorded on every verdict row, and verdicts are un-swapped before aggregation, rendering position bias both mitigated and measurable; (iii) \emph{conservative parsing} --- judges answer with a single verdict token at temperature 0, unparseable verdicts count as ties, and empty or degenerate responses can never win; (iv) \emph{anti-verbosity instruction} --- judge prompts explicitly forbid rewarding length or style over substance. Panel agreement rates are computed per run and published in the run's immutable diagnostics.356357\subsection{Aggregation}358359Verdicts feed a Bradley--Terry model: model $i$ beats model $j$ with probability $p_i/(p_i+p_j)$, ties contributing half a win to each side. Strengths are fitted with the minorization--maximization iteration~\citep{hunter2004mm}360\begin{equation}361p_i \;\leftarrow\; W_i \Big/ \sum_{j \ne i} \frac{N_{ij}}{p_i + p_j},362\label{eq:mm}363\end{equation}364where $W_i$ is the (fractional) win count of model $i$ and $N_{ij}$ the number of comparisons between $i$ and $j$; a small damping term ($0.1$ phantom wins per pair) keeps strengths finite for sparsely compared models. Uncertainty follows from the Fisher information of the log-strength parametrization,365\begin{equation}366\mathcal{I}_{ii} = \sum_{j\neq i} N_{ij}\, p_{ij}(1-p_{ij}), \qquad p_{ij} = \frac{p_i}{p_i+p_j}, \qquad367\operatorname{SE}(\log p_i) = \mathcal{I}_{ii}^{-1/2}.368\end{equation}369Log-strengths are standardized to zero mean and unit variance, placing duel domains on the same $\theta$-scale as the IRT domains before rescaling; downstream score construction is thereby uniform across both layers.370371% =====================================================================372\section{Robustness Metrics}373\label{sec:robustness}374375Three additional measurements enter each domain's composite, each capturing a failure mode invisible to accuracy.376377\subsection{Consistency}378379Every scored item is also administered $k$ additional times at the model's default sampling temperature. With $A_{mi}$ the multiset of extracted answers of model $m$ on item $i$,380\begin{equation}381\operatorname{consistency}(m) = \frac{1}{|I_m|}\sum_{i \in I_m}382\frac{\max_{v} \#\{a \in A_{mi} : a = v\}}{|A_{mi}|},383\end{equation}384the mean share of samples agreeing with the modal answer. Large accuracy variance across re-instantiations of identical templates is a documented phenomenon~\citep{mirzadeh2024gsmsymbolic}; a model that flips answers under resampling is less trustworthy than its accuracy suggests, and this metric prices that in.385386\subsection{Calibration}387388Every graded item requires the model to append a confidence self-report on a 0--100 scale --- verbalized confidence being better calibrated than token likelihoods for RLHF-trained models~\citep{tian2023just}. With confidences binned into ten equal-width bins $B_1,\dots,B_{10}$,389\begin{equation}390\operatorname{ECE} = \sum_{b=1}^{10} \frac{|B_b|}{N} \Big| \operatorname{acc}(B_b) - \overline{\operatorname{conf}}(B_b) \Big|,391\qquad392\operatorname{calibration} = 1 - \operatorname{ECE},393\end{equation}394following~\citet{guo2017calibration}; the Brier score~\citep{brier1950verification} is computed alongside as a diagnostic. Missing confidences are treated as missing, not imputed at 50, so calibration is never polluted by non-compliance --- which is measured elsewhere.395396\subsection{Contamination resistance}397398As defined in Equation~\eqref{eq:contam}, the anchor-versus-fresh accuracy gap maps to a resistance score with a hard floor: a gap of 20 accuracy points or more earns zero credit. Unlike post-hoc contamination detection, this quantity is measured \emph{inside} the evaluation, per model, per domain, on every run.399400% =====================================================================401\section{Answer Extraction and Grading}402\label{sec:extraction}403404Grading robustness is treated as a first-class methodological concern. Extraction proceeds through an ordered, lenient cascade: the last markdown-stripped line matching an answer tag (\texttt{ANSWER:}, \texttt{FINAL ANSWER:}, with any bold/backtick wrapping absorbed); failing that, the last \LaTeX{} \verb|\boxed{}| expression; for structured answers (tool-call sequences, terminal output), the last fenced code block with a balanced-bracket JSON scan and repair pass. Numeric grading normalizes thousands separators, currency symbols, unit suffixes, and Unicode minus signs, and compares under relative tolerance $10^{-6}$; string grading normalizes case, punctuation, and hyphen/space orthographic variants; tool-call grading canonicalizes JSON (sorted keys, whitespace-free) before exact comparison; constraint-stack items are validated by deterministic checker functions, so any of the astronomically many conforming outputs grades correct.405406Two policies deserve emphasis. First, \emph{truncation is not failure}: a completion cut off at the token budget with no extractable answer is recorded as unscored, never as incorrect, because grading a severed chain of thought measures the budget rather than the model. Second, \emph{format compliance is measured where it belongs}: the instruction-following domain explicitly tests constraint adherence, so leniency elsewhere does not erase the signal --- it relocates it to the construct actually being claimed.407408% =====================================================================409\section{Score Structure and Interpretation}410\label{sec:scores}411412\subsection{From sub-metrics to domain scores}413414Each closed-ended domain yields an ability estimate $\hat\theta$ with standard error, plus the three robustness metrics. The domain composite in $[0,1]$ is415\begin{equation}416C \;=\; \frac{w_{\text{acc}}\,\sigma(\hat\theta) + w_{\text{con}}\cdot\text{consistency} + w_{\text{cal}}\cdot\text{calibration} + w_{\text{res}}\cdot R}417{\sum_{\text{measured}} w},418\label{eq:composite}419\end{equation}420with weights $(w_{\text{acc}}, w_{\text{con}}, w_{\text{res}}, w_{\text{cal}}) = (0.60,\, 0.15,\, 0.15,\, 0.10)$ and renormalization over whichever sub-metrics were actually measured, so a missing measurement neither rewards nor punishes. The weight rationale is explicit: accuracy-as-ability is the primary construct and dominates; consistency and contamination resistance are robustness corrections grounded in documented failure modes; calibration, the noisiest of the four at current sample sizes, receives the smallest weight. The displayed domain score is $1000\,C$, with a 95\% interval propagated from $\operatorname{SE}(\hat\theta)$ by the delta method through the accuracy term:421\begin{equation}422\text{half-width} \;=\; 1000 \cdot \frac{w_{\text{acc}}}{\sum w} \cdot \sigma(\hat\theta)\big(1-\sigma(\hat\theta)\big) \cdot 1.96 \cdot \operatorname{SE}(\hat\theta).423\end{equation}424Duel domains enter the same pipeline with the standardized Bradley--Terry log-strength in place of $\hat\theta$. Figure~\ref{fig:composition} summarizes the construction and the deliberate wall between quality and economics.425426\begin{figure}[t]427\centering428\resizebox{0.98\textwidth}{!}{%429\begin{tikzpicture}[node distance=3.5mm and 8mm]430  \node[stage, minimum width=32mm] (acc) {accuracy$_{\text{IRT}}$ $\sigma(\hat\theta)$ \\ {\scriptsize $w=0.60$}};431  \node[stage, minimum width=32mm, below=of acc] (con) {consistency \\ {\scriptsize $w=0.15$}};432  \node[stage, minimum width=32mm, below=of con] (res) {contamination resist. \\ {\scriptsize $w=0.15$}};433  \node[stage, minimum width=32mm, below=of res] (cal) {calibration $1{-}\mathrm{ECE}$ \\ {\scriptsize $w=0.10$}};434  \node[stage, fill=llmamber!15, right=14mm of con, yshift=-5.5mm, minimum width=30mm] (comp)435    {domain composite $C$\\{\scriptsize weights renormalize over}\\{\scriptsize measured terms}};436  \node[stage, fill=llmamber!25, right=of comp, minimum width=30mm] (dom)437    {domain score\\$1000\,C$ $\pm$ 95\% CI};438  \node[stage, fill=llmviolet!15, right=of dom, minimum width=30mm] (glob)439    {\textbf{Global Index}\\{\scriptsize equal domain weights,}\\{\scriptsize quadrature CI}};440  \draw[flow] (acc.east) -- (comp);441  \draw[flow] (con.east) -- (comp);442  \draw[flow] (res.east) -- (comp);443  \draw[flow] (cal.east) -- (comp);444  \draw[flow] (comp) -- (dom);445  \draw[flow] (dom) -- (glob);446  \node[stageB, below=13mm of comp, minimum width=30mm] (cost) {latency $p_{50}$ \& cost/1k items};447  \node[stageB, right=of cost, minimum width=30mm] (pareto) {Pareto frontier\\{\scriptsize published as a \emph{set}}};448  \draw[flow] (cost) -- (pareto);449  \draw[very thick, red!60, dash pattern=on 3pt off 2.5pt]450    ($(cost.north west)+(-3mm,3.5mm)$) -- ($(pareto.north east)+(3mm,3.5mm)$);451  \node[note, text=red!60!black] at ($(pareto.north east)+(-12mm,6mm)$) {never blended into quality};452\end{tikzpicture}}453\caption{Score construction. Four commensurable quality sub-metrics blend under published weights into the domain composite; economics stays below the red line, summarized only as a Pareto set.}454\label{fig:composition}455\end{figure}456457\subsection{The Global Index}458459The Global Index is the weighted mean of a model's domain composites over the domains it was actually evaluated on, rescaled to $[0,1000]$:460\begin{equation}461G \;=\; 1000 \cdot \frac{\sum_{d \in D_m} w_d\, C_d}{\sum_{d \in D_m} w_d},462\qquad463\text{half-width}(G) = \Big( \textstyle\sum_{d} \big(w_d'\,h_d\big)^2 \Big)^{1/2},464\end{equation}465where $h_d$ are domain half-widths, $w_d'$ normalized weights, and independence across domain fits justifies combination in quadrature. Domain weights are \emph{equal} --- deliberately. Absent a task-utility function, any unequal weighting is an editorial value judgment; the maximum-entropy prior is the only non-arbitrary default, and because per-domain scores are always published, any reader with a genuine utility function can re-weight in seconds.466467\subsection{The efficiency frontier}468469For each model the platform publishes measured median latency and measured cost per thousand items (from metered token usage and live per-token pricing). The quality--cost plane is summarized by its Pareto frontier: the set of models not dominated on both axes (Figure~\ref{fig:pareto}). No blended ``value score'' exists anywhere on the platform, by principle P6.470471\begin{figure}[t]472\centering473\begin{tikzpicture}474\begin{axis}[475  width=0.72\textwidth, height=6cm,476  xlabel={cost per 1{,}000 items (USD, log scale)}, ylabel={Global Index},477  xmode=log, xmin=0.02, xmax=60, ymin=350, ymax=900,478  grid=major, grid style={llmgray!20},479  tick label style={font=\scriptsize}, label style={font=\small},480  legend style={font=\scriptsize, at={(0.98,0.05)}, anchor=south east, draw=none, fill=none}]481\addplot[only marks, mark=*, mark size=1.8pt, llmgray!70] coordinates {482  (0.4,455) (0.9,520) (2.2,585) (3.5,610) (6,640) (9,700) (14,690) (28,745) (45,760)};483\addlegendentry{dominated}484\addplot[only marks, mark=*, mark size=2.6pt, llmgreen] coordinates {485  (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820)};486\addlegendentry{frontier}487\addplot[llmgreen!70, thick, const plot mark right] coordinates {488  (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820) (60,820)};489\end{axis}490\end{tikzpicture}491\caption{Illustrative efficiency frontier. Frontier models (green) are undominated on quality and cost simultaneously; every other model is strictly worse on both axes than some frontier point. The frontier is published as a set --- deliberately never collapsed into a single ``value'' number.}492\label{fig:pareto}493\end{figure}494495\subsection{How to read the numbers}496497\begin{itemize}[leftmargin=1.6em,itemsep=0.15em]498  \item \textbf{Overlapping intervals mean no established order.} A 15-point gap under 60-point intervals is noise, and the interface will show exactly that.499  \item \textbf{Scores are version-scoped.} A score under index version \version{} is not comparable to one under 0.1.0; the version is printed beside every ranking, and fits never mix versions.500  \item \textbf{Domain profiles beat the Global Index for decisions.} The Global Index summarizes; procurement should read the domain radar and the sub-metrics (a high-accuracy, poorly calibrated model is a specific risk profile).501  \item \textbf{Duel scores are comparative.} Bradley--Terry strengths locate models relative to the evaluated pool, not on an absolute craft scale.502  \item \textbf{Anchor-gap tells you about training data.} A large published $\Delta_{\text{contam}}$ for a model is direct evidence that its static-benchmark results elsewhere should be discounted.503\end{itemize}504505% =====================================================================506\section{Governance, Versioning, and Transparency}507\label{sec:governance}508509\paragraph{Semantic versioning of method.} Domains, weights, hyperparameters, and grading rules live in a single source-of-truth module whose version follows semver. Any change bumps the version and appends a public changelog entry; the machine-readable configuration is served by the public API.510511\paragraph{Immutable runs.} Every evaluation batch, duel batch, and index fit is a database row carrying its item-set hash, model set, index version, status, and fit diagnostics. Historical runs are never edited. Corrections --- including successful score disputes --- are executed as \emph{new} runs and acknowledged in the changelog.512513\paragraph{Total response transparency.} Each model's public page lists every graded answer of the current version (verdict, confidence, latency, cost, and the full prompt for non-anchor items) and every judged duel (judge identity, position-swap flag, verdict). Two things are withheld, on stated principle: answer keys, and anchor-item prompts --- publishing either would destroy the measurement.514515\paragraph{Independence.} The platform evaluates models from all providers under identical protocols, takes no compensation from evaluated vendors, and publishes the judge configuration. Disputes are received at \href{mailto:contact@spboucher.ai}{contact@spboucher.ai} and adjudicated on the stored evidence.516517% =====================================================================518\section{Known Limitations}519\label{sec:limitations}520521Intellectual honesty requires stating what the platform does \emph{not} establish.522523\textbf{Small-examinee-population IRT.} Item-parameter recovery in IRT degrades when the examinee population is small; with on the order of $10^2$ models, item parameters are regularized by priors rather than pinned by data. The roster deliberately spans very weak to frontier models to anchor the scale, and score intervals are published precisely so users do not over-read; nevertheless, item-level parameters should be treated as noisier than the ability estimates built on them.524525\textbf{Judge subjectivity survives bias control.} Position swapping, cross-provider panels, self-judging exclusion, and agreement reporting mitigate --- but cannot eliminate --- shared aesthetic preferences among LLM judges. Duel-domain scores should be read comparatively and alongside the published agreement rates.526527\textbf{Provider routing variance.} Models are reached through an aggregation layer whose upstream routing can affect latency and, occasionally, behavior. Latency is reported as measured, not normalized to reference hardware.528529\textbf{Construct coverage.} Generated items measure deep, narrow, mechanically-gradeable slices of each capability. This is a deliberate trade --- breadth is the enemy of grading validity --- but it means the index does not measure, e.g., long-horizon interactive tool use with error recovery (single-shot planned sequences are graded today), open-domain factuality at web scale, or multi-session memory. The template inventory is public; readers can judge coverage directly.530531\textbf{Anchor stream aging.} Anchors resist contamination only until models train on traffic that includes them. Anchor rotation with overlap (retiring streams gradually while preserving longitudinal linkage) is on the roadmap.532533\textbf{Consistency confound.} The consistency metric samples at each model's default temperature, which providers set differently; part of the observed stability difference is configuration rather than capability. It is reported as measured, with this caveat documented.534535% =====================================================================536\section{Comparison with Existing Efforts}537\label{sec:comparison}538539\begin{landscape}540\begin{table}[p]541\centering542\small543\setlength{\tabcolsep}{6pt}544\renewcommand{\arraystretch}{1.25}545\begin{tabular}{@{}p{4.0cm}p{5.2cm}p{5.6cm}p{8.0cm}@{}}546\toprule547\textbf{Effort} & \textbf{Measurement} & \textbf{Contamination stance} & \textbf{Relation to \llmindex} \\548\midrule549MMLU / static suites~\citep{hendrycks2021mmlu,wang2024mmlupro} & raw accuracy, fixed items & fixed public sets; saturated at the frontier & \llmindex{} replaces raw accuracy with 2PL ability and fixed items with generators \\550\addlinespace[2pt]551HELM~\citep{liang2023helm} & multi-metric, fixed scenarios & fixed sets & shares multi-metric philosophy; \llmindex{} adds IRT, generation, and per-score CIs \\552\addlinespace[2pt]553Chatbot Arena~\citep{chiang2024chatbot} & human pairwise votes, BT/Elo & organic prompts; style confounds documented & same aggregation family for duels; \llmindex{} judges are bias-controlled panels on generated tasks, and duels are one layer, not the whole index \\554\addlinespace[2pt]555LiveBench~\citep{white2024livebench} & accuracy on refreshed items & periodic refresh & \llmindex{} regenerates \emph{every batch} and measures the memorization gap explicitly \\556\addlinespace[2pt]557tinyBenchmarks / metabench~\citep{polo2024tinybenchmarks,kipnis2025metabench} & IRT on existing benchmark items & inherits source-set exposure & validates the IRT layer; \llmindex{} applies it to unexposed generated items \\558\addlinespace[2pt]559BFCL / $\tau$-bench~\citep{patil2025bfcl,yao2024taubench} & AST/state-graded tool use & fixed task sets & inspiration for simulator-graded agentic items, re-implemented as seeded generators \\560\addlinespace[2pt]561Terminal-Bench~\citep{tbench2025} & containerized tasks, test scripts & fixed tasks; grader-exploit incidents documented & \llmindex{} predicts simulated-shell outcomes: no container, no readable answer key, no exploit surface \\562\addlinespace[2pt]563GAIA / HLE~\citep{mialon2023gaia,phan2025hle} & exact-match hard questions & held-out but fixed & shares frontier-difficulty targeting; \llmindex{} achieves it with regenerable items \\564\bottomrule565\end{tabular}566\caption{Positioning relative to representative evaluation efforts.}567\label{tab:compare}568\end{table}569\end{landscape}570571Table~\ref{tab:compare} summarizes. The synthesis is the contribution: psychometric scoring \emph{on} contamination-proof generated items, duels \emph{as one bias-controlled layer} rather than the whole story, robustness and honesty metrics \emph{inside} the score, efficiency \emph{outside} it, and an audit trail underneath everything.572573% =====================================================================574\section{Roadmap}575\label{sec:roadmap}576577\begin{enumerate}[leftmargin=1.6em,itemsep=0.2em]578  \item \textbf{Adaptive administration.} Selecting the next item family and difficulty bin to maximize expected Fisher information at the model's running $\hat\theta$, manufacturing items at requested difficulty --- the capability fixed benchmarks cannot have.579  \item \textbf{Multi-turn agentic episodes.} Interactive tool loops against the deterministic simulators with injected typed faults, grading final state, required reads, and recovery under a strict failed-call budget.580  \item \textbf{Anchor rotation with overlap.} Gradual retirement of anchor streams with linking items to preserve longitudinal comparability while bounding anchor exposure.581  \item \textbf{Knob-to-difficulty calibration.} Learning the mapping from generator parameters to fitted $b$ per template family, enabling item manufacture at prescribed difficulty.582  \item \textbf{Judge audit corpus.} A fixed, hand-adjudicated duel set for measuring judge-panel drift across judge model upgrades.583  \item \textbf{Public data releases.} Periodic dumps of expired (non-anchor) items with full response matrices, enabling third-party psychometric reanalysis.584  \item \textbf{Third-party methodology audits} and a standing correction bounty on grading errors.585\end{enumerate}586587% =====================================================================588\section{Conclusion}589590Benchmarking is measurement, and measurement has a discipline. \llmindex{} applies that discipline end to end: items that cannot be memorized because they do not exist until asked; ability estimated where information lives rather than where accuracy saturates; judges treated as instruments with quantified bias, not oracles; honesty about uncertainty in every number printed; a hard wall between quality and economics; and an audit trail that makes every claim checkable and every error correctable in public. The platform is live, its methodology is versioned, and its mistakes --- when found --- will be fixed in the open.591592\vspace{1em}593\noindent\textbf{Correspondence:} Simon-Pierre Boucher, \href{mailto:contact@spboucher.ai}{contact@spboucher.ai}.594595\bibliographystyle{plainnat}596\bibliography{references}597598\end{document}599