SPB Git

spb/llmindex Public

The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.

TypeScript 77.9% TeX 15.2% Python 3.7% SQL 1.4% JavaScript 1.1% Shell 0.5%

docs(whitepaper): TikZ figures, landscape comparison table, sharp visual design

- TikZ-redrawn LLMindex logo on the title page
- Figures: evaluation pipeline flow, 2PL item characteristic curves +
  item information functions (pgfplots), score-composition diagram with
  the quality/economics wall, illustrative Pareto frontier
- Comparison table rotated to landscape (fixes right-edge overflow)
- Colored section titles, running headers, overfull boxes eliminated
- 19 pages, clean latexmk build

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 days ago (Aug 5, 2026) parent b2a07ba

Showing 3 changed files with +191 and −11

modified README.md +1 −1
@@ -15,7 +15,7 @@
15 15 ![Models](https://img.shields.io/badge/models-135-8b5cf6)
16 16 ![Domains](https://img.shields.io/badge/domains-12-f59e0b)
17 17 ![Judges](https://img.shields.io/badge/judge%20panel-3%20cross--provider-ec4899)
18 [![White paper](https://img.shields.io/badge/white%20paper-17%20pages%20·%20LaTeX-b91c1c?logo=latex&logoColor=white)](docs/whitepaper/llmindex-whitepaper.pdf)
18 +[![White paper](https://img.shields.io/badge/white%20paper-19%20pages%20·%20LaTeX-b91c1c?logo=latex&logoColor=white)](docs/whitepaper/llmindex-whitepaper.pdf)
19 19
20 20 <!-- metric pastilles -->
21 21 ![IRT](https://img.shields.io/badge/scoring-IRT%202PL%20%CE%B8-10b981)
modified docs/whitepaper/llmindex-whitepaper.pdf +0 −0

Binary file not shown.

modified docs/whitepaper/llmindex-whitepaper.tex +190 −10
@@ -17,10 +17,55 @@
17 17 \usepackage{xcolor}
18 18 \usepackage{enumitem}
19 19 \usepackage[numbers,sort&compress]{natbib}
20 +\usepackage{tikz}
21 +\usetikzlibrary{arrows.meta,positioning,shapes.misc,calc,fit,backgrounds}
22 +\usepackage{pgfplots}
23 +\pgfplotsset{compat=1.18}
24 +\usepackage{pdflscape}
25 +\usepackage{fancyhdr}
26 +\usepackage{titlesec}
20 27 \usepackage[colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black]{hyperref}
21 28 \usepackage{setspace}
22 29
23 30 \definecolor{llmgreen}{HTML}{10B981}
31 +\definecolor{llmcyan}{HTML}{22D3EE}
32 +\definecolor{llmdark}{HTML}{09090B}
33 +\definecolor{llmgray}{HTML}{71717A}
34 +\definecolor{llmamber}{HTML}{F59E0B}
35 +\definecolor{llmviolet}{HTML}{8B5CF6}
36 +
37 +% Colored, sharp section titles
38 +\titleformat{\section}{\Large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesection}}{0.8em}{}[{\color{llmgreen!60}\titlerule[0.8pt]}]
39 +\titleformat{\subsection}{\large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesubsection}}{0.7em}{}
40 +
41 +% Running headers
42 +\pagestyle{fancy}
43 +\fancyhf{}
44 +\fancyhead[L]{\small\textsc{LLMindex} --- Methodology White Paper}
45 +\fancyhead[R]{\small v\version}
46 +\fancyfoot[C]{\small\thepage}
47 +\renewcommand{\headrulewidth}{0.4pt}
48 +
49 +% The LLMindex mark, redrawn in TikZ (ascending bars + index needle)
50 +\newcommand{\llmlogo}[1][1.4cm]{%
51 +\begin{tikzpicture}[x=#1/6.4, y=#1/6.4, baseline]
52 + \fill[llmdark, rounded corners=0.22*#1/1.4cm] (0,0) rectangle (6.4,6.4);
53 + \fill[llmgray!45, rounded corners=1pt] (1.10,1.10) rectangle (2.00,2.70);
54 + \fill[llmgray!75, rounded corners=1pt] (2.40,1.10) rectangle (3.30,3.50);
55 + \fill[llmgreen, rounded corners=1pt] (3.70,1.10) rectangle (4.60,4.50);
56 + \draw[line width=0.09*#1/1.4cm, llmcyan, line cap=round] (1.30,3.40) -- (4.70,5.20);
57 + \fill[llmcyan] (4.90,5.30) circle (0.32);
58 +\end{tikzpicture}}
59 +
60 +% Common TikZ styles for diagrams
61 +\tikzset{
62 + stage/.style={draw=llmdark!60, fill=llmgreen!8, rounded corners=2pt, align=center,
63 + font=\small, inner sep=5pt, minimum height=9mm},
64 + stageB/.style={draw=llmdark!60, fill=llmcyan!10, rounded corners=2pt, align=center,
65 + font=\small, inner sep=5pt, minimum height=9mm},
66 + flow/.style={-{Stealth[length=2.2mm]}, thick, llmdark!70},
67 + note/.style={font=\scriptsize\itshape, text=llmgray}
68 +}
24 69
25 70 \newcommand{\llmindex}{\textsc{LLMindex}}
26 71 \newcommand{\version}{0.2.0}
@@ -28,7 +73,7 @@
28 73
29 74 \setlength{\parskip}{0.35em}
30 75
31 \title{\vspace{-1.2em}\Huge \llmindex\\[0.35em]
76 +\title{\vspace{-2.2em}\llmlogo[2.2cm]\\[0.9em]\Huge \llmindex\\[0.35em]
32 77 \Large A Psychometric, Contamination-Resistant, and Fully Auditable\\ Methodology for Ranking Large Language Models\\[0.6em]
33 78 \normalsize Methodology White Paper --- Index Version \version}
34 79 \author{Simon-Pierre Boucher\\[0.2em]
@@ -128,6 +173,41 @@ All evaluated models are reached through a single OpenAI-compatible aggregation
128 173
129 174 Each model call persists an audit row containing the exact request parameters, the raw provider response, extracted answer, grading verdict, self-reported confidence, token usage, measured wall-clock latency, and measured cost. A ranking whose raw responses are not stored is, by the platform's own rules, invalid. This audit surface is public: every model's page exposes every graded answer of the current index version, and every judged duel with its verdict, judge identity, and position-swap flag.
130 175
176 +\subsection{Evaluation pipeline}
177 +
178 +Figure~\ref{fig:pipeline} shows the end-to-end flow from seeded generation to the live leaderboard.
179 +
180 +\begin{figure}[t]
181 +\centering
182 +\resizebox{\textwidth}{!}{%
183 +\begin{tikzpicture}[node distance=4mm and 7mm]
184 + \node[stage] (gen) {Seeded template\\generators};
185 + \node[stage, right=of gen] (adm) {Administration\\{\scriptsize $T{=}0$ scored $+\,k$ samples}};
186 + \node[stage, right=of adm] (ext) {Extraction\\cascade};
187 + \node[stage, right=of ext] (grd) {Mechanical\\grading};
188 + \node[stage, right=of grd] (irt) {2PL IRT fit\\{\scriptsize $\hat\theta,\ \mathrm{SE},\ a_i, b_i$}};
189 + \node[stageB, below=9mm of adm] (duel) {Pairwise duels\\{\scriptsize generated open tasks}};
190 + \node[stageB, right=of duel] (judge) {3-judge panel\\{\scriptsize swap, no self-judging}};
191 + \node[stageB, right=of judge] (bt) {Bradley--Terry\\{\scriptsize $\log p_i,\ \mathrm{SE}$}};
192 + \node[stage, fill=llmamber!15, right=9mm of irt, yshift=-8.5mm] (score) {Scores + 95\% CI\\{\scriptsize composite, global}};
193 + \node[stage, fill=llmviolet!12, below=4mm of score] (live) {Live leaderboard\\{\scriptsize refit per model}};
194 + \draw[flow] (gen) -- (adm);
195 + \draw[flow] (adm) -- (ext);
196 + \draw[flow] (ext) -- (grd);
197 + \draw[flow] (grd) -- (irt);
198 + \draw[flow] (gen.south) |- (duel.west);
199 + \draw[flow] (duel) -- (judge);
200 + \draw[flow] (judge) -- (bt);
201 + \draw[flow] (irt.east) -- (score.north west);
202 + \draw[flow] (bt.east) -| ($(score.south west)+(2mm,0)$);
203 + \draw[flow] (score) -- (live);
204 + \node[note, below=1.2mm of duel.south west, anchor=north west, xshift=-2mm]
205 + {every call audited: raw response, tokens, latency, cost};
206 +\end{tikzpicture}}
207 +\caption{The \llmindex{} evaluation pipeline. Closed-ended domains flow through mechanical grading into the 2PL layer; open-ended domains flow through bias-controlled judged duels into the Bradley--Terry layer; both meet in a uniform score construction with published uncertainty.}
208 +\label{fig:pipeline}
209 +\end{figure}
210 +
131 211 \subsection{Evaluation domains}
132 212
133 213 Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed-ended domains graded mechanically under the IRT layer, and three open-ended domains ranked by judged duels under the Bradley--Terry layer.
@@ -135,7 +215,7 @@ Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed-
135 215 \begin{table}[t]
136 216 \centering
137 217 \small
138 \begin{tabular}{@{}lll@{}}
218 +\begin{tabular}{@{}p{3.7cm}p{8.0cm}p{2.6cm}@{}}
139 219 \toprule
140 220 \textbf{Domain} & \textbf{Item family (generated)} & \textbf{Grading} \\
141 221 \midrule
@@ -146,11 +226,11 @@ agentic & simulated tool-calling under policy; context-load ledgers & canonical
146 226 terminal & simulated POSIX subset: pipelines, file trees, exit-code chains & exact lines \\
147 227 knowledge & free-response curated facts (no options, no guessing floor) & exact \\
148 228 multilingual & number words 0--999, French/Spanish, both directions & exact/numeric \\
149 instruction\_following & repetition/acronym tasks; 5-constraint stacks & exact/checkers \\
229 +instruction\_\-following & repetition/acronym tasks; 5-constraint stacks & exact/checkers \\
150 230 vision\_ocr & generated cluttered scenes, rasterized; table reading & exact/numeric \\
151 231 \midrule
152 232 writing & constrained creative briefs & duels + BT \\
153 safety\_refusal\_quality & delicate gray-zone assistance scenarios & duels + BT \\
233 +safety\_\-refusal\_\-quality & delicate gray-zone assistance scenarios & duels + BT \\
154 234 svg\_design & reproduce real logos as raw SVG from memory & duels + BT \\
155 235 \bottomrule
156 236 \end{tabular}
@@ -195,7 +275,49 @@ P(x_{mi}=1 \mid \theta_m, a_i, b_i) \;=\; \sigma\!\big(a_i(\theta_m - b_i)\big)
195 275 \;=\; \frac{1}{1+e^{-a_i(\theta_m-b_i)}},
196 276 \label{eq:2pl}
197 277 \end{equation}
198 where $\theta_m$ is the latent ability of model $m$, $b_i$ the item difficulty (in logits), and $a_i>0$ the item discrimination --- the steepness with which the item separates abilities around $b_i$.
278 +where $\theta_m$ is the latent ability of model $m$, $b_i$ the item difficulty (in logits), and $a_i>0$ the item discrimination --- the steepness with which the item separates abilities around $b_i$. Figure~\ref{fig:icc} illustrates both the item characteristic curves and the induced information functions.
279 +
280 +\begin{figure}[t]
281 +\centering
282 +\begin{tikzpicture}
283 +\begin{axis}[
284 + width=0.48\textwidth, height=5.2cm,
285 + xlabel={ability $\theta$}, ylabel={$P(\text{correct})$},
286 + xmin=-4, xmax=4, ymin=0, ymax=1,
287 + legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none},
288 + grid=major, grid style={llmgray!20},
289 + tick label style={font=\scriptsize}, label style={font=\small},
290 + domain=-4:4, samples=120]
291 +\addplot[llmgray!80, thick] {1/(1+exp(-0.4*x))};
292 +\addlegendentry{$a{=}0.4,\ b{=}0$ (flagged)}
293 +\addplot[llmgreen, very thick] {1/(1+exp(-1.2*x))};
294 +\addlegendentry{$a{=}1.2,\ b{=}0$}
295 +\addplot[llmcyan!80!black, very thick, dashed] {1/(1+exp(-2.2*(x-1.2)))};
296 +\addlegendentry{$a{=}2.2,\ b{=}1.2$}
297 +\end{axis}
298 +\end{tikzpicture}\hfill
299 +\begin{tikzpicture}
300 +\begin{axis}[
301 + width=0.48\textwidth, height=5.2cm,
302 + xlabel={ability $\theta$}, ylabel={information $I(\theta)$},
303 + xmin=-4, xmax=4, ymin=0, ymax=1.3,
304 + legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none},
305 + grid=major, grid style={llmgray!20},
306 + tick label style={font=\scriptsize}, label style={font=\small},
307 + domain=-4:4, samples=120]
308 +\addplot[llmgray!80, thick] {0.16*exp(-0.4*x)/(1+exp(-0.4*x))^2};
309 +\addlegendentry{$a{=}0.4$: near-zero everywhere}
310 +\addplot[llmgreen, very thick] {1.44*exp(-1.2*x)/(1+exp(-1.2*x))^2};
311 +\addlegendentry{$a{=}1.2$}
312 +\addplot[llmcyan!80!black, very thick, dashed] {4.84*exp(-2.2*(x-1.2))/(1+exp(-2.2*(x-1.2)))^2};
313 +\addlegendentry{$a{=}2.2$: sharp, at frontier $\theta$}
314 +\draw[llmamber, thick, dotted] (axis cs:1.2,0) -- (axis cs:1.2,1.21);
315 +\node[note, anchor=west] at (axis cs:1.35,1.05) {peak at $P{=}\tfrac12$};
316 +\end{axis}
317 +\end{tikzpicture}
318 +\caption{Left: 2PL item characteristic curves. Right: the corresponding item information $I(\theta)=a_i^2P(1-P)$. A flat item ($a{=}0.4$) contributes almost nothing anywhere and is auto-flagged; a discriminative item placed at frontier ability ($a{=}2.2$, $b{=}1.2$) measures precisely where saturated benchmarks are blind.}
319 +\label{fig:icc}
320 +\end{figure}
199 321
200 322 \subsection{Estimation}
201 323
@@ -299,7 +421,38 @@ with weights $(w_{\text{acc}}, w_{\text{con}}, w_{\text{res}}, w_{\text{cal}}) =
299 421 \begin{equation}
300 422 \text{half-width} \;=\; 1000 \cdot \frac{w_{\text{acc}}}{\sum w} \cdot \sigma(\hat\theta)\big(1-\sigma(\hat\theta)\big) \cdot 1.96 \cdot \operatorname{SE}(\hat\theta).
301 423 \end{equation}
302 Duel domains enter the same pipeline with the standardized Bradley--Terry log-strength in place of $\hat\theta$.
424 +Duel domains enter the same pipeline with the standardized Bradley--Terry log-strength in place of $\hat\theta$. Figure~\ref{fig:composition} summarizes the construction and the deliberate wall between quality and economics.
425 +
426 +\begin{figure}[t]
427 +\centering
428 +\resizebox{0.98\textwidth}{!}{%
429 +\begin{tikzpicture}[node distance=3.5mm and 8mm]
430 + \node[stage, minimum width=32mm] (acc) {accuracy$_{\text{IRT}}$ $\sigma(\hat\theta)$ \\ {\scriptsize $w=0.60$}};
431 + \node[stage, minimum width=32mm, below=of acc] (con) {consistency \\ {\scriptsize $w=0.15$}};
432 + \node[stage, minimum width=32mm, below=of con] (res) {contamination resist. \\ {\scriptsize $w=0.15$}};
433 + \node[stage, minimum width=32mm, below=of res] (cal) {calibration $1{-}\mathrm{ECE}$ \\ {\scriptsize $w=0.10$}};
434 + \node[stage, fill=llmamber!15, right=14mm of con, yshift=-5.5mm, minimum width=30mm] (comp)
435 + {domain composite $C$\\{\scriptsize weights renormalize over}\\{\scriptsize measured terms}};
436 + \node[stage, fill=llmamber!25, right=of comp, minimum width=30mm] (dom)
437 + {domain score\\$1000\,C$ $\pm$ 95\% CI};
438 + \node[stage, fill=llmviolet!15, right=of dom, minimum width=30mm] (glob)
439 + {\textbf{Global Index}\\{\scriptsize equal domain weights,}\\{\scriptsize quadrature CI}};
440 + \draw[flow] (acc.east) -- (comp);
441 + \draw[flow] (con.east) -- (comp);
442 + \draw[flow] (res.east) -- (comp);
443 + \draw[flow] (cal.east) -- (comp);
444 + \draw[flow] (comp) -- (dom);
445 + \draw[flow] (dom) -- (glob);
446 + \node[stageB, below=13mm of comp, minimum width=30mm] (cost) {latency $p_{50}$ \& cost/1k items};
447 + \node[stageB, right=of cost, minimum width=30mm] (pareto) {Pareto frontier\\{\scriptsize published as a \emph{set}}};
448 + \draw[flow] (cost) -- (pareto);
449 + \draw[very thick, red!60, dash pattern=on 3pt off 2.5pt]
450 + ($(cost.north west)+(-3mm,3.5mm)$) -- ($(pareto.north east)+(3mm,3.5mm)$);
451 + \node[note, text=red!60!black] at ($(pareto.north east)+(-12mm,6mm)$) {never blended into quality};
452 +\end{tikzpicture}}
453 +\caption{Score construction. Four commensurable quality sub-metrics blend under published weights into the domain composite; economics stays below the red line, summarized only as a Pareto set.}
454 +\label{fig:composition}
455 +\end{figure}
303 456
304 457 \subsection{The Global Index}
305 458
@@ -313,7 +466,31 @@ where $h_d$ are domain half-widths, $w_d'$ normalized weights, and independence
313 466
314 467 \subsection{The efficiency frontier}
315 468
316 For each model the platform publishes measured median latency and measured cost per thousand items (from metered token usage and live per-token pricing). The quality--cost plane is summarized by its Pareto frontier: the set of models not dominated on both axes. No blended ``value score'' exists anywhere on the platform, by principle P6.
469 +For each model the platform publishes measured median latency and measured cost per thousand items (from metered token usage and live per-token pricing). The quality--cost plane is summarized by its Pareto frontier: the set of models not dominated on both axes (Figure~\ref{fig:pareto}). No blended ``value score'' exists anywhere on the platform, by principle P6.
470 +
471 +\begin{figure}[t]
472 +\centering
473 +\begin{tikzpicture}
474 +\begin{axis}[
475 + width=0.72\textwidth, height=6cm,
476 + xlabel={cost per 1{,}000 items (USD, log scale)}, ylabel={Global Index},
477 + xmode=log, xmin=0.02, xmax=60, ymin=350, ymax=900,
478 + grid=major, grid style={llmgray!20},
479 + tick label style={font=\scriptsize}, label style={font=\small},
480 + legend style={font=\scriptsize, at={(0.98,0.05)}, anchor=south east, draw=none, fill=none}]
481 +\addplot[only marks, mark=*, mark size=1.8pt, llmgray!70] coordinates {
482 + (0.4,455) (0.9,520) (2.2,585) (3.5,610) (6,640) (9,700) (14,690) (28,745) (45,760)};
483 +\addlegendentry{dominated}
484 +\addplot[only marks, mark=*, mark size=2.6pt, llmgreen] coordinates {
485 + (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820)};
486 +\addlegendentry{frontier}
487 +\addplot[llmgreen!70, thick, const plot mark right] coordinates {
488 + (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820) (60,820)};
489 +\end{axis}
490 +\end{tikzpicture}
491 +\caption{Illustrative efficiency frontier. Frontier models (green) are undominated on quality and cost simultaneously; every other model is strictly worse on both axes than some frontier point. The frontier is published as a set --- deliberately never collapsed into a single ``value'' number.}
492 +\label{fig:pareto}
493 +\end{figure}
317 494
318 495 \subsection{How to read the numbers}
319 496
@@ -359,11 +536,13 @@ Intellectual honesty requires stating what the platform does \emph{not} establis
359 536 \section{Comparison with Existing Efforts}
360 537 \label{sec:comparison}
361 538
362 \begin{table}[t]
539 +\begin{landscape}
540 +\begin{table}[p]
363 541 \centering
364 542 \small
365 \setlength{\tabcolsep}{4pt}
366 \begin{tabular}{@{}p{2.9cm}p{3.4cm}p{3.6cm}p{4.6cm}@{}}
543 +\setlength{\tabcolsep}{6pt}
544 +\renewcommand{\arraystretch}{1.25}
545 +\begin{tabular}{@{}p{4.0cm}p{5.2cm}p{5.6cm}p{8.0cm}@{}}
367 546 \toprule
368 547 \textbf{Effort} & \textbf{Measurement} & \textbf{Contamination stance} & \textbf{Relation to \llmindex} \\
369 548 \midrule
@@ -387,6 +566,7 @@ GAIA / HLE~\citep{mialon2023gaia,phan2025hle} & exact-match hard questions & hel
387 566 \caption{Positioning relative to representative evaluation efforts.}
388 567 \label{tab:compare}
389 568 \end{table}
569 +\end{landscape}
390 570
391 571 Table~\ref{tab:compare} summarizes. The synthesis is the contribution: psychometric scoring \emph{on} contamination-proof generated items, duels \emph{as one bias-controlled layer} rather than the whole story, robustness and honesty metrics \emph{inside} the score, efficiency \emph{outside} it, and an audit trail underneath everything.
392 572
393 573