docs(whitepaper): TikZ figures, landscape comparison table, sharp visual design
- TikZ-redrawn LLMindex logo on the title page - Figures: evaluation pipeline flow, 2PL item characteristic curves + item information functions (pgfplots), score-composition diagram with the quality/economics wall, illustrative Pareto frontier - Comparison table rotated to landscape (fixes right-edge overflow) - Colored section titles, running headers, overfull boxes eliminated - 19 pages, clean latexmk build Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 changed files +191 −11
modified
README.md
+1 −1
@@ -15,7 +15,7 @@ | |||||
| 15 |  | 15 |  | ||
| 16 |  | 16 |  | ||
| 17 |  | 17 |  | ||
| 18 | −[](docs/whitepaper/llmindex-whitepaper.pdf) | 18 | +[](docs/whitepaper/llmindex-whitepaper.pdf) | ||
| 19 | 19 | ||||
| 20 | <!-- metric pastilles --> | 20 | <!-- metric pastilles --> | ||
| 21 |  | 21 |  | ||
modified
docs/whitepaper/llmindex-whitepaper.pdf
+0 −0
Binary file not shown.
modified
docs/whitepaper/llmindex-whitepaper.tex
+190 −10
@@ -17,10 +17,55 @@ | |||||
| 17 | \usepackage{xcolor} | 17 | \usepackage{xcolor} | ||
| 18 | \usepackage{enumitem} | 18 | \usepackage{enumitem} | ||
| 19 | \usepackage[numbers,sort&compress]{natbib} | 19 | \usepackage[numbers,sort&compress]{natbib} | ||
| 20 | +\usepackage{tikz} | ||||
| 21 | +\usetikzlibrary{arrows.meta,positioning,shapes.misc,calc,fit,backgrounds} | ||||
| 22 | +\usepackage{pgfplots} | ||||
| 23 | +\pgfplotsset{compat=1.18} | ||||
| 24 | +\usepackage{pdflscape} | ||||
| 25 | +\usepackage{fancyhdr} | ||||
| 26 | +\usepackage{titlesec} | ||||
| 20 | \usepackage[colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black]{hyperref} | 27 | \usepackage[colorlinks=true,linkcolor=blue!50!black,citecolor=blue!50!black,urlcolor=blue!50!black]{hyperref} | ||
| 21 | \usepackage{setspace} | 28 | \usepackage{setspace} | ||
| 22 | 29 | ||||
| 23 | \definecolor{llmgreen}{HTML}{10B981} | 30 | \definecolor{llmgreen}{HTML}{10B981} | ||
| 31 | +\definecolor{llmcyan}{HTML}{22D3EE} | ||||
| 32 | +\definecolor{llmdark}{HTML}{09090B} | ||||
| 33 | +\definecolor{llmgray}{HTML}{71717A} | ||||
| 34 | +\definecolor{llmamber}{HTML}{F59E0B} | ||||
| 35 | +\definecolor{llmviolet}{HTML}{8B5CF6} | ||||
| 36 | + | ||||
| 37 | +% Colored, sharp section titles | ||||
| 38 | +\titleformat{\section}{\Large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesection}}{0.8em}{}[{\color{llmgreen!60}\titlerule[0.8pt]}] | ||||
| 39 | +\titleformat{\subsection}{\large\bfseries\color{llmdark}}{\textcolor{llmgreen}{\thesubsection}}{0.7em}{} | ||||
| 40 | + | ||||
| 41 | +% Running headers | ||||
| 42 | +\pagestyle{fancy} | ||||
| 43 | +\fancyhf{} | ||||
| 44 | +\fancyhead[L]{\small\textsc{LLMindex} --- Methodology White Paper} | ||||
| 45 | +\fancyhead[R]{\small v\version} | ||||
| 46 | +\fancyfoot[C]{\small\thepage} | ||||
| 47 | +\renewcommand{\headrulewidth}{0.4pt} | ||||
| 48 | + | ||||
| 49 | +% The LLMindex mark, redrawn in TikZ (ascending bars + index needle) | ||||
| 50 | +\newcommand{\llmlogo}[1][1.4cm]{% | ||||
| 51 | +\begin{tikzpicture}[x=#1/6.4, y=#1/6.4, baseline] | ||||
| 52 | + \fill[llmdark, rounded corners=0.22*#1/1.4cm] (0,0) rectangle (6.4,6.4); | ||||
| 53 | + \fill[llmgray!45, rounded corners=1pt] (1.10,1.10) rectangle (2.00,2.70); | ||||
| 54 | + \fill[llmgray!75, rounded corners=1pt] (2.40,1.10) rectangle (3.30,3.50); | ||||
| 55 | + \fill[llmgreen, rounded corners=1pt] (3.70,1.10) rectangle (4.60,4.50); | ||||
| 56 | + \draw[line width=0.09*#1/1.4cm, llmcyan, line cap=round] (1.30,3.40) -- (4.70,5.20); | ||||
| 57 | + \fill[llmcyan] (4.90,5.30) circle (0.32); | ||||
| 58 | +\end{tikzpicture}} | ||||
| 59 | + | ||||
| 60 | +% Common TikZ styles for diagrams | ||||
| 61 | +\tikzset{ | ||||
| 62 | + stage/.style={draw=llmdark!60, fill=llmgreen!8, rounded corners=2pt, align=center, | ||||
| 63 | + font=\small, inner sep=5pt, minimum height=9mm}, | ||||
| 64 | + stageB/.style={draw=llmdark!60, fill=llmcyan!10, rounded corners=2pt, align=center, | ||||
| 65 | + font=\small, inner sep=5pt, minimum height=9mm}, | ||||
| 66 | + flow/.style={-{Stealth[length=2.2mm]}, thick, llmdark!70}, | ||||
| 67 | + note/.style={font=\scriptsize\itshape, text=llmgray} | ||||
| 68 | +} | ||||
| 24 | 69 | ||||
| 25 | \newcommand{\llmindex}{\textsc{LLMindex}} | 70 | \newcommand{\llmindex}{\textsc{LLMindex}} | ||
| 26 | \newcommand{\version}{0.2.0} | 71 | \newcommand{\version}{0.2.0} | ||
@@ -28,7 +73,7 @@ | |||||
| 28 | 73 | ||||
| 29 | \setlength{\parskip}{0.35em} | 74 | \setlength{\parskip}{0.35em} | ||
| 30 | 75 | ||||
| 31 | −\title{\vspace{-1.2em}\Huge \llmindex\\[0.35em] | 76 | +\title{\vspace{-2.2em}\llmlogo[2.2cm]\\[0.9em]\Huge \llmindex\\[0.35em] | ||
| 32 | \Large A Psychometric, Contamination-Resistant, and Fully Auditable\\ Methodology for Ranking Large Language Models\\[0.6em] | 77 | \Large A Psychometric, Contamination-Resistant, and Fully Auditable\\ Methodology for Ranking Large Language Models\\[0.6em] | ||
| 33 | \normalsize Methodology White Paper --- Index Version \version} | 78 | \normalsize Methodology White Paper --- Index Version \version} | ||
| 34 | \author{Simon-Pierre Boucher\\[0.2em] | 79 | \author{Simon-Pierre Boucher\\[0.2em] | ||
@@ -128,6 +173,41 @@ All evaluated models are reached through a single OpenAI-compatible aggregation | |||||
| 128 | 173 | ||||
| 129 | Each model call persists an audit row containing the exact request parameters, the raw provider response, extracted answer, grading verdict, self-reported confidence, token usage, measured wall-clock latency, and measured cost. A ranking whose raw responses are not stored is, by the platform's own rules, invalid. This audit surface is public: every model's page exposes every graded answer of the current index version, and every judged duel with its verdict, judge identity, and position-swap flag. | 174 | Each model call persists an audit row containing the exact request parameters, the raw provider response, extracted answer, grading verdict, self-reported confidence, token usage, measured wall-clock latency, and measured cost. A ranking whose raw responses are not stored is, by the platform's own rules, invalid. This audit surface is public: every model's page exposes every graded answer of the current index version, and every judged duel with its verdict, judge identity, and position-swap flag. | ||
| 130 | 175 | ||||
| 176 | +\subsection{Evaluation pipeline} | ||||
| 177 | + | ||||
| 178 | +Figure~\ref{fig:pipeline} shows the end-to-end flow from seeded generation to the live leaderboard. | ||||
| 179 | + | ||||
| 180 | +\begin{figure}[t] | ||||
| 181 | +\centering | ||||
| 182 | +\resizebox{\textwidth}{!}{% | ||||
| 183 | +\begin{tikzpicture}[node distance=4mm and 7mm] | ||||
| 184 | + \node[stage] (gen) {Seeded template\\generators}; | ||||
| 185 | + \node[stage, right=of gen] (adm) {Administration\\{\scriptsize $T{=}0$ scored $+\,k$ samples}}; | ||||
| 186 | + \node[stage, right=of adm] (ext) {Extraction\\cascade}; | ||||
| 187 | + \node[stage, right=of ext] (grd) {Mechanical\\grading}; | ||||
| 188 | + \node[stage, right=of grd] (irt) {2PL IRT fit\\{\scriptsize $\hat\theta,\ \mathrm{SE},\ a_i, b_i$}}; | ||||
| 189 | + \node[stageB, below=9mm of adm] (duel) {Pairwise duels\\{\scriptsize generated open tasks}}; | ||||
| 190 | + \node[stageB, right=of duel] (judge) {3-judge panel\\{\scriptsize swap, no self-judging}}; | ||||
| 191 | + \node[stageB, right=of judge] (bt) {Bradley--Terry\\{\scriptsize $\log p_i,\ \mathrm{SE}$}}; | ||||
| 192 | + \node[stage, fill=llmamber!15, right=9mm of irt, yshift=-8.5mm] (score) {Scores + 95\% CI\\{\scriptsize composite, global}}; | ||||
| 193 | + \node[stage, fill=llmviolet!12, below=4mm of score] (live) {Live leaderboard\\{\scriptsize refit per model}}; | ||||
| 194 | + \draw[flow] (gen) -- (adm); | ||||
| 195 | + \draw[flow] (adm) -- (ext); | ||||
| 196 | + \draw[flow] (ext) -- (grd); | ||||
| 197 | + \draw[flow] (grd) -- (irt); | ||||
| 198 | + \draw[flow] (gen.south) |- (duel.west); | ||||
| 199 | + \draw[flow] (duel) -- (judge); | ||||
| 200 | + \draw[flow] (judge) -- (bt); | ||||
| 201 | + \draw[flow] (irt.east) -- (score.north west); | ||||
| 202 | + \draw[flow] (bt.east) -| ($(score.south west)+(2mm,0)$); | ||||
| 203 | + \draw[flow] (score) -- (live); | ||||
| 204 | + \node[note, below=1.2mm of duel.south west, anchor=north west, xshift=-2mm] | ||||
| 205 | + {every call audited: raw response, tokens, latency, cost}; | ||||
| 206 | +\end{tikzpicture}} | ||||
| 207 | +\caption{The \llmindex{} evaluation pipeline. Closed-ended domains flow through mechanical grading into the 2PL layer; open-ended domains flow through bias-controlled judged duels into the Bradley--Terry layer; both meet in a uniform score construction with published uncertainty.} | ||||
| 208 | +\label{fig:pipeline} | ||||
| 209 | +\end{figure} | ||||
| 210 | + | ||||
| 131 | \subsection{Evaluation domains} | 211 | \subsection{Evaluation domains} | ||
| 132 | 212 | ||||
| 133 | Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed-ended domains graded mechanically under the IRT layer, and three open-ended domains ranked by judged duels under the Bradley--Terry layer. | 213 | Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed-ended domains graded mechanically under the IRT layer, and three open-ended domains ranked by judged duels under the Bradley--Terry layer. | ||
@@ -135,7 +215,7 @@ Version \version{} scores twelve domains (Table~\ref{tab:domains}): nine closed- | |||||
| 135 | \begin{table}[t] | 215 | \begin{table}[t] | ||
| 136 | \centering | 216 | \centering | ||
| 137 | \small | 217 | \small | ||
| 138 | −\begin{tabular}{@{}lll@{}} | 218 | +\begin{tabular}{@{}p{3.7cm}p{8.0cm}p{2.6cm}@{}} | ||
| 139 | \toprule | 219 | \toprule | ||
| 140 | \textbf{Domain} & \textbf{Item family (generated)} & \textbf{Grading} \\ | 220 | \textbf{Domain} & \textbf{Item family (generated)} & \textbf{Grading} \\ | ||
| 141 | \midrule | 221 | \midrule | ||
@@ -146,11 +226,11 @@ agentic & simulated tool-calling under policy; context-load ledgers & canonical | |||||
| 146 | terminal & simulated POSIX subset: pipelines, file trees, exit-code chains & exact lines \\ | 226 | terminal & simulated POSIX subset: pipelines, file trees, exit-code chains & exact lines \\ | ||
| 147 | knowledge & free-response curated facts (no options, no guessing floor) & exact \\ | 227 | knowledge & free-response curated facts (no options, no guessing floor) & exact \\ | ||
| 148 | multilingual & number words 0--999, French/Spanish, both directions & exact/numeric \\ | 228 | multilingual & number words 0--999, French/Spanish, both directions & exact/numeric \\ | ||
| 149 | −instruction\_following & repetition/acronym tasks; 5-constraint stacks & exact/checkers \\ | 229 | +instruction\_\-following & repetition/acronym tasks; 5-constraint stacks & exact/checkers \\ | ||
| 150 | vision\_ocr & generated cluttered scenes, rasterized; table reading & exact/numeric \\ | 230 | vision\_ocr & generated cluttered scenes, rasterized; table reading & exact/numeric \\ | ||
| 151 | \midrule | 231 | \midrule | ||
| 152 | writing & constrained creative briefs & duels + BT \\ | 232 | writing & constrained creative briefs & duels + BT \\ | ||
| 153 | −safety\_refusal\_quality & delicate gray-zone assistance scenarios & duels + BT \\ | 233 | +safety\_\-refusal\_\-quality & delicate gray-zone assistance scenarios & duels + BT \\ | ||
| 154 | svg\_design & reproduce real logos as raw SVG from memory & duels + BT \\ | 234 | svg\_design & reproduce real logos as raw SVG from memory & duels + BT \\ | ||
| 155 | \bottomrule | 235 | \bottomrule | ||
| 156 | \end{tabular} | 236 | \end{tabular} | ||
@@ -195,7 +275,49 @@ P(x_{mi}=1 \mid \theta_m, a_i, b_i) \;=\; \sigma\!\big(a_i(\theta_m - b_i)\big) | |||||
| 195 | \;=\; \frac{1}{1+e^{-a_i(\theta_m-b_i)}}, | 275 | \;=\; \frac{1}{1+e^{-a_i(\theta_m-b_i)}}, | ||
| 196 | \label{eq:2pl} | 276 | \label{eq:2pl} | ||
| 197 | \end{equation} | 277 | \end{equation} | ||
| 198 | −where $\theta_m$ is the latent ability of model $m$, $b_i$ the item difficulty (in logits), and $a_i>0$ the item discrimination --- the steepness with which the item separates abilities around $b_i$. | 278 | +where $\theta_m$ is the latent ability of model $m$, $b_i$ the item difficulty (in logits), and $a_i>0$ the item discrimination --- the steepness with which the item separates abilities around $b_i$. Figure~\ref{fig:icc} illustrates both the item characteristic curves and the induced information functions. | ||
| 279 | + | ||||
| 280 | +\begin{figure}[t] | ||||
| 281 | +\centering | ||||
| 282 | +\begin{tikzpicture} | ||||
| 283 | +\begin{axis}[ | ||||
| 284 | + width=0.48\textwidth, height=5.2cm, | ||||
| 285 | + xlabel={ability $\theta$}, ylabel={$P(\text{correct})$}, | ||||
| 286 | + xmin=-4, xmax=4, ymin=0, ymax=1, | ||||
| 287 | + legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none}, | ||||
| 288 | + grid=major, grid style={llmgray!20}, | ||||
| 289 | + tick label style={font=\scriptsize}, label style={font=\small}, | ||||
| 290 | + domain=-4:4, samples=120] | ||||
| 291 | +\addplot[llmgray!80, thick] {1/(1+exp(-0.4*x))}; | ||||
| 292 | +\addlegendentry{$a{=}0.4,\ b{=}0$ (flagged)} | ||||
| 293 | +\addplot[llmgreen, very thick] {1/(1+exp(-1.2*x))}; | ||||
| 294 | +\addlegendentry{$a{=}1.2,\ b{=}0$} | ||||
| 295 | +\addplot[llmcyan!80!black, very thick, dashed] {1/(1+exp(-2.2*(x-1.2)))}; | ||||
| 296 | +\addlegendentry{$a{=}2.2,\ b{=}1.2$} | ||||
| 297 | +\end{axis} | ||||
| 298 | +\end{tikzpicture}\hfill | ||||
| 299 | +\begin{tikzpicture} | ||||
| 300 | +\begin{axis}[ | ||||
| 301 | + width=0.48\textwidth, height=5.2cm, | ||||
| 302 | + xlabel={ability $\theta$}, ylabel={information $I(\theta)$}, | ||||
| 303 | + xmin=-4, xmax=4, ymin=0, ymax=1.3, | ||||
| 304 | + legend style={font=\scriptsize, at={(0.02,0.98)}, anchor=north west, draw=none, fill=none}, | ||||
| 305 | + grid=major, grid style={llmgray!20}, | ||||
| 306 | + tick label style={font=\scriptsize}, label style={font=\small}, | ||||
| 307 | + domain=-4:4, samples=120] | ||||
| 308 | +\addplot[llmgray!80, thick] {0.16*exp(-0.4*x)/(1+exp(-0.4*x))^2}; | ||||
| 309 | +\addlegendentry{$a{=}0.4$: near-zero everywhere} | ||||
| 310 | +\addplot[llmgreen, very thick] {1.44*exp(-1.2*x)/(1+exp(-1.2*x))^2}; | ||||
| 311 | +\addlegendentry{$a{=}1.2$} | ||||
| 312 | +\addplot[llmcyan!80!black, very thick, dashed] {4.84*exp(-2.2*(x-1.2))/(1+exp(-2.2*(x-1.2)))^2}; | ||||
| 313 | +\addlegendentry{$a{=}2.2$: sharp, at frontier $\theta$} | ||||
| 314 | +\draw[llmamber, thick, dotted] (axis cs:1.2,0) -- (axis cs:1.2,1.21); | ||||
| 315 | +\node[note, anchor=west] at (axis cs:1.35,1.05) {peak at $P{=}\tfrac12$}; | ||||
| 316 | +\end{axis} | ||||
| 317 | +\end{tikzpicture} | ||||
| 318 | +\caption{Left: 2PL item characteristic curves. Right: the corresponding item information $I(\theta)=a_i^2P(1-P)$. A flat item ($a{=}0.4$) contributes almost nothing anywhere and is auto-flagged; a discriminative item placed at frontier ability ($a{=}2.2$, $b{=}1.2$) measures precisely where saturated benchmarks are blind.} | ||||
| 319 | +\label{fig:icc} | ||||
| 320 | +\end{figure} | ||||
| 199 | 321 | ||||
| 200 | \subsection{Estimation} | 322 | \subsection{Estimation} | ||
| 201 | 323 | ||||
@@ -299,7 +421,38 @@ with weights $(w_{\text{acc}}, w_{\text{con}}, w_{\text{res}}, w_{\text{cal}}) = | |||||
| 299 | \begin{equation} | 421 | \begin{equation} | ||
| 300 | \text{half-width} \;=\; 1000 \cdot \frac{w_{\text{acc}}}{\sum w} \cdot \sigma(\hat\theta)\big(1-\sigma(\hat\theta)\big) \cdot 1.96 \cdot \operatorname{SE}(\hat\theta). | 422 | \text{half-width} \;=\; 1000 \cdot \frac{w_{\text{acc}}}{\sum w} \cdot \sigma(\hat\theta)\big(1-\sigma(\hat\theta)\big) \cdot 1.96 \cdot \operatorname{SE}(\hat\theta). | ||
| 301 | \end{equation} | 423 | \end{equation} | ||
| 302 | −Duel domains enter the same pipeline with the standardized Bradley--Terry log-strength in place of $\hat\theta$. | 424 | +Duel domains enter the same pipeline with the standardized Bradley--Terry log-strength in place of $\hat\theta$. Figure~\ref{fig:composition} summarizes the construction and the deliberate wall between quality and economics. | ||
| 425 | + | ||||
| 426 | +\begin{figure}[t] | ||||
| 427 | +\centering | ||||
| 428 | +\resizebox{0.98\textwidth}{!}{% | ||||
| 429 | +\begin{tikzpicture}[node distance=3.5mm and 8mm] | ||||
| 430 | + \node[stage, minimum width=32mm] (acc) {accuracy$_{\text{IRT}}$ $\sigma(\hat\theta)$ \\ {\scriptsize $w=0.60$}}; | ||||
| 431 | + \node[stage, minimum width=32mm, below=of acc] (con) {consistency \\ {\scriptsize $w=0.15$}}; | ||||
| 432 | + \node[stage, minimum width=32mm, below=of con] (res) {contamination resist. \\ {\scriptsize $w=0.15$}}; | ||||
| 433 | + \node[stage, minimum width=32mm, below=of res] (cal) {calibration $1{-}\mathrm{ECE}$ \\ {\scriptsize $w=0.10$}}; | ||||
| 434 | + \node[stage, fill=llmamber!15, right=14mm of con, yshift=-5.5mm, minimum width=30mm] (comp) | ||||
| 435 | + {domain composite $C$\\{\scriptsize weights renormalize over}\\{\scriptsize measured terms}}; | ||||
| 436 | + \node[stage, fill=llmamber!25, right=of comp, minimum width=30mm] (dom) | ||||
| 437 | + {domain score\\$1000\,C$ $\pm$ 95\% CI}; | ||||
| 438 | + \node[stage, fill=llmviolet!15, right=of dom, minimum width=30mm] (glob) | ||||
| 439 | + {\textbf{Global Index}\\{\scriptsize equal domain weights,}\\{\scriptsize quadrature CI}}; | ||||
| 440 | + \draw[flow] (acc.east) -- (comp); | ||||
| 441 | + \draw[flow] (con.east) -- (comp); | ||||
| 442 | + \draw[flow] (res.east) -- (comp); | ||||
| 443 | + \draw[flow] (cal.east) -- (comp); | ||||
| 444 | + \draw[flow] (comp) -- (dom); | ||||
| 445 | + \draw[flow] (dom) -- (glob); | ||||
| 446 | + \node[stageB, below=13mm of comp, minimum width=30mm] (cost) {latency $p_{50}$ \& cost/1k items}; | ||||
| 447 | + \node[stageB, right=of cost, minimum width=30mm] (pareto) {Pareto frontier\\{\scriptsize published as a \emph{set}}}; | ||||
| 448 | + \draw[flow] (cost) -- (pareto); | ||||
| 449 | + \draw[very thick, red!60, dash pattern=on 3pt off 2.5pt] | ||||
| 450 | + ($(cost.north west)+(-3mm,3.5mm)$) -- ($(pareto.north east)+(3mm,3.5mm)$); | ||||
| 451 | + \node[note, text=red!60!black] at ($(pareto.north east)+(-12mm,6mm)$) {never blended into quality}; | ||||
| 452 | +\end{tikzpicture}} | ||||
| 453 | +\caption{Score construction. Four commensurable quality sub-metrics blend under published weights into the domain composite; economics stays below the red line, summarized only as a Pareto set.} | ||||
| 454 | +\label{fig:composition} | ||||
| 455 | +\end{figure} | ||||
| 303 | 456 | ||||
| 304 | \subsection{The Global Index} | 457 | \subsection{The Global Index} | ||
| 305 | 458 | ||||
@@ -313,7 +466,31 @@ where $h_d$ are domain half-widths, $w_d'$ normalized weights, and independence | |||||
| 313 | 466 | ||||
| 314 | \subsection{The efficiency frontier} | 467 | \subsection{The efficiency frontier} | ||
| 315 | 468 | ||||
| 316 | −For each model the platform publishes measured median latency and measured cost per thousand items (from metered token usage and live per-token pricing). The quality--cost plane is summarized by its Pareto frontier: the set of models not dominated on both axes. No blended ``value score'' exists anywhere on the platform, by principle P6. | 469 | +For each model the platform publishes measured median latency and measured cost per thousand items (from metered token usage and live per-token pricing). The quality--cost plane is summarized by its Pareto frontier: the set of models not dominated on both axes (Figure~\ref{fig:pareto}). No blended ``value score'' exists anywhere on the platform, by principle P6. | ||
| 470 | + | ||||
| 471 | +\begin{figure}[t] | ||||
| 472 | +\centering | ||||
| 473 | +\begin{tikzpicture} | ||||
| 474 | +\begin{axis}[ | ||||
| 475 | + width=0.72\textwidth, height=6cm, | ||||
| 476 | + xlabel={cost per 1{,}000 items (USD, log scale)}, ylabel={Global Index}, | ||||
| 477 | + xmode=log, xmin=0.02, xmax=60, ymin=350, ymax=900, | ||||
| 478 | + grid=major, grid style={llmgray!20}, | ||||
| 479 | + tick label style={font=\scriptsize}, label style={font=\small}, | ||||
| 480 | + legend style={font=\scriptsize, at={(0.98,0.05)}, anchor=south east, draw=none, fill=none}] | ||||
| 481 | +\addplot[only marks, mark=*, mark size=1.8pt, llmgray!70] coordinates { | ||||
| 482 | + (0.4,455) (0.9,520) (2.2,585) (3.5,610) (6,640) (9,700) (14,690) (28,745) (45,760)}; | ||||
| 483 | +\addlegendentry{dominated} | ||||
| 484 | +\addplot[only marks, mark=*, mark size=2.6pt, llmgreen] coordinates { | ||||
| 485 | + (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820)}; | ||||
| 486 | +\addlegendentry{frontier} | ||||
| 487 | +\addplot[llmgreen!70, thick, const plot mark right] coordinates { | ||||
| 488 | + (0.05,430) (0.12,560) (0.8,660) (2.8,720) (11,780) (33,820) (60,820)}; | ||||
| 489 | +\end{axis} | ||||
| 490 | +\end{tikzpicture} | ||||
| 491 | +\caption{Illustrative efficiency frontier. Frontier models (green) are undominated on quality and cost simultaneously; every other model is strictly worse on both axes than some frontier point. The frontier is published as a set --- deliberately never collapsed into a single ``value'' number.} | ||||
| 492 | +\label{fig:pareto} | ||||
| 493 | +\end{figure} | ||||
| 317 | 494 | ||||
| 318 | \subsection{How to read the numbers} | 495 | \subsection{How to read the numbers} | ||
| 319 | 496 | ||||
@@ -359,11 +536,13 @@ Intellectual honesty requires stating what the platform does \emph{not} establis | |||||
| 359 | \section{Comparison with Existing Efforts} | 536 | \section{Comparison with Existing Efforts} | ||
| 360 | \label{sec:comparison} | 537 | \label{sec:comparison} | ||
| 361 | 538 | ||||
| 362 | −\begin{table}[t] | 539 | +\begin{landscape} | ||
| 540 | +\begin{table}[p] | ||||
| 363 | \centering | 541 | \centering | ||
| 364 | \small | 542 | \small | ||
| 365 | −\setlength{\tabcolsep}{4pt} | 543 | +\setlength{\tabcolsep}{6pt} | ||
| 366 | −\begin{tabular}{@{}p{2.9cm}p{3.4cm}p{3.6cm}p{4.6cm}@{}} | 544 | +\renewcommand{\arraystretch}{1.25} | ||
| 545 | +\begin{tabular}{@{}p{4.0cm}p{5.2cm}p{5.6cm}p{8.0cm}@{}} | ||||
| 367 | \toprule | 546 | \toprule | ||
| 368 | \textbf{Effort} & \textbf{Measurement} & \textbf{Contamination stance} & \textbf{Relation to \llmindex} \\ | 547 | \textbf{Effort} & \textbf{Measurement} & \textbf{Contamination stance} & \textbf{Relation to \llmindex} \\ | ||
| 369 | \midrule | 548 | \midrule | ||
@@ -387,6 +566,7 @@ GAIA / HLE~\citep{mialon2023gaia,phan2025hle} & exact-match hard questions & hel | |||||
| 387 | \caption{Positioning relative to representative evaluation efforts.} | 566 | \caption{Positioning relative to representative evaluation efforts.} | ||
| 388 | \label{tab:compare} | 567 | \label{tab:compare} | ||
| 389 | \end{table} | 568 | \end{table} | ||
| 569 | +\end{landscape} | ||||
| 390 | 570 | ||||
| 391 | Table~\ref{tab:compare} summarizes. The synthesis is the contribution: psychometric scoring \emph{on} contamination-proof generated items, duels \emph{as one bias-controlled layer} rather than the whole story, robustness and honesty metrics \emph{inside} the score, efficiency \emph{outside} it, and an audit trail underneath everything. | 571 | Table~\ref{tab:compare} summarizes. The synthesis is the contribution: psychometric scoring \emph{on} contamination-proof generated items, duels \emph{as one bias-controlled layer} rather than the whole story, robustness and honesty metrics \emph{inside} the score, efficiency \emph{outside} it, and an audit trail underneath everything. | ||
| 392 | 572 | ||||
| 393 | 573 | ||||