spb/llmindex Public
The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.
TypeScript 77.9%
TeX 15.2%
Python 3.7%
SQL 1.4%
JavaScript 1.1%
Shell 0.5%
1/**2 * llmindex.io — methodology page: full pipeline detail, live sample items, references3 * Author: Simon-Pierre Boucher4 * Contact: contact@spboucher.ai5 * License: Proprietary — © Simon-Pierre Boucher, all rights reserved6 *7 * Sample items are rendered from documentation-only seeds ("docs:<template>")8 * that are never used in scored runs — templates are public by design, scored9 * instantiations are not.10 */11import {12 CONTAMINATION_DELTA_FLOOR,13 DOMAINS,14 DOMAIN_WEIGHTS,15 DUEL_DOMAINS,16 INDEX_VERSION,17 IRT_HYPERPARAMS,18 SUBMETRIC_WEIGHTS,19 THETA_SCALE,20} from '@llmindex/scoring';21import { TEMPLATES, createRng } from '@llmindex/items';2223export const revalidate = 3600;2425const PILLARS: Array<{ title: string; body: string }> = [26 {27 title: '1 · IRT-based scoring (2PL)',28 body: 'Every item carries a fitted difficulty (b) and discrimination (a): P(correct) = σ(a·(θ−b)). Ability θ is a MAP estimate with priors θ~N(0,1), b~N(0,1.5), log a~N(0,0.5), fitted by a missing-aware numpy optimizer; θ standard errors come from the Fisher information. Items solved by everyone (or no one) carry zero ranking information — items with a<0.3 or |b|>3 are auto-flagged for retirement after every refit. Raw accuracy is never the score.',29 },30 {31 title: '2 · Dynamic item generation',32 body: 'Items are instantiated from versioned template generators with seeded value substitution, paraphrase rotation and structural perturbation — fresh for every scored batch, so no fixed test set exists to memorize. A frozen anchor stream (≤20% of any run) is kept identical across runs; the accuracy gap anchor−fresh is published per model as contamination_delta and maps to a resistance sub-metric via clamp01(1 − Δ/0.2).',33 },34 {35 title: '3 · Difficulty engineered for discrimination',36 body: 'Item information peaks where a model has ~50% success probability. Generators therefore stack proven difficulty knobs: 6-8-step dependent arithmetic chains (errors compound multiplicatively), chained sub-problems whose answers feed forward, provably-inert distractor clauses, counterfactual rules (base-7/8/9/11/13 arithmetic), interior-rank deduction with decoy entities, nested control-flow traces, and constraint stacking — each knob re-randomized per run.',37 },38 {39 title: '4 · Agentic: simulated tool-calling',40 body: 'Home-made mock environments (support-desk triage under policy, treasury ledger with overdraft pre-funding, dependency-ordered deployments) present a tool catalog salted with distractor tools. A deterministic simulator computes the unique correct call sequence; grading is canonical-JSON equality of the emitted sequence — binary, no judges, no partial credit. A dedicated context-load family buries the relevant records among hundreds of near-miss decoys (same customer/wrong region, right region/wrong status), making prompt length itself the difficulty knob.',41 },42 {43 title: '5 · Terminal: simulated shell, exact prediction',44 body: 'No shell ever executes. A closed, unambiguous POSIX subset (fixed-string grep, cut, byte-order C-locale sort, integer-only awk aggregation, head/tail) is simulated in TypeScript over generated CSV data; models predict exact stdout, final file trees after mv/cp/rm/cd sequences with relative paths, and && / || short-circuit execution traces with exit codes. Locale-dependent, GNU/BSD-divergent and float-formatting constructs are excluded by design, so every answer is unambiguous.',45 },46 {47 title: '6 · Vision OCR under clutter',48 body: 'Generated SVG scenes — rasterized server-side — embed target codes among rotated decoy codes, noise strokes and low-contrast patches, plus rendered mini-tables requiring grounded arithmetic. The glyph set excludes visually ambiguous characters (0/O, 1/l/I, 5/S, 8/B), so difficulty comes from clutter and selection, never unfair ambiguity. Text-only models skip the domain; their Global Index renormalizes.',49 },50 {51 title: '7 · Judged duels + Bradley-Terry (writing, safety, SVG design)',52 body: 'Open-ended domains use pairwise duels: both models answer the same generated task (constrained creative writing; delicate gray-zone situations; reproducing a real-world logo in raw SVG from memory). A 3-judge cross-provider panel rates each duel with recorded position swaps; a model never judges its own duel; empty responses never count as wins. Verdicts feed a Bradley-Terry fit (MM algorithm, ties as half-wins, Fisher-information SEs), standardized to the same θ scale as IRT domains. Judge panel agreement is published with every run.',53 },54 {55 title: '8 · Robust answer extraction (measured, not guessed)',56 body: 'Extraction never confounds formatting with ability: an ordered cascade accepts "ANSWER: x" in any markdown wrapping, FINAL ANSWER variants, LaTeX \\boxed{}, and fenced code blocks for JSON/multi-line answers; numbers are normalized across thousands separators, currency signs and units; hyphen/space orthography variants are unified. Truncated completions (finish_reason=length) are unscored — never counted as wrong — and every model gets a 16k-token completion budget so reasoning models can finish thinking.',57 },58 {59 title: '9 · Consistency, calibration, cost',60 body: 'Each scored item is also sampled k times at the model\'s default temperature; the share of samples agreeing with the modal answer is the consistency sub-metric. Every item demands a 0-100 confidence line; calibration = 1 − ECE over 10 bins. Latency p50 and measured cost per 1k items are published but NEVER blended into quality — they live on a separate Pareto frontier.',61 },62];6364const REFERENCES: Array<{ label: string; url: string }> = [65 { label: 'metabench — sparse IRT benchmark distillation (ICLR 2025)', url: 'https://openreview.net/forum?id=4T33izzFpK' },66 { label: 'tinyBenchmarks — 100-item IRT evaluation (ICML 2024)', url: 'https://arxiv.org/abs/2402.14992' },67 { label: 'ATLAS — Fisher-information adaptive testing for LLMs', url: 'https://arxiv.org/abs/2511.04689' },68 { label: 'GSM-Symbolic — template perturbation & memorization gaps (Apple)', url: 'https://arxiv.org/abs/2410.05229' },69 { label: 'GSM-IC — irrelevant-context distractors (ICML 2023)', url: 'https://arxiv.org/abs/2302.00093' },70 { label: 'Reasoning or Reciting? — counterfactual rule perturbation (NAACL 2024)', url: 'https://arxiv.org/abs/2307.02477' },71 { label: 'Faith and Fate — compositional depth limits (NeurIPS 2023)', url: 'https://arxiv.org/abs/2305.18654' },72 { label: 'ZebraLogic — constraint-satisfaction scaling curse (ICML 2025)', url: 'https://arxiv.org/abs/2502.01100' },73 { label: 'GSM-Infinite — computational-graph difficulty scaling (ICML 2025)', url: 'https://arxiv.org/abs/2502.05252' },74 { label: 'R-Horizon — chained-problem discrimination amplification', url: 'https://arxiv.org/abs/2510.08189' },75 { label: 'IFEval — verifiable instruction constraints', url: 'https://arxiv.org/abs/2311.07911' },76 { label: 'MMLU-Pro — distractor design & discrimination (NeurIPS 2024)', url: 'https://arxiv.org/abs/2406.01574' },77 { label: 'Humanity\'s Last Exam — frontier-failure item filtering (Nature 2025)', url: 'https://www.nature.com/articles/s41586-025-09962-4' },78 { label: 'NPHardEval — monthly regenerated complexity-class items', url: 'https://arxiv.org/abs/2312.14890' },79 { label: 'Can We Trust IRT for AI Evaluation? — small-N caveats', url: 'https://arxiv.org/html/2607.15190v1' },80 { label: 'τ-bench / τ²-bench — policy-constrained agents, pass^k (Sierra)', url: 'https://arxiv.org/pdf/2406.12045' },81 { label: 'BFCL v3 — AST/state-based tool-call grading (Berkeley)', url: 'https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html' },82 { label: 'GAIA — normalized exact-match agent grading', url: 'https://huggingface.co/papers/2311.12983' },83 { label: 'Establishing Best Practices for Rigorous Agentic Benchmarks (ABC)', url: 'https://arxiv.org/abs/2507.02825' },84 { label: 'SWE-Bench+ — leakage & weak-test inflation audit', url: 'https://arxiv.org/pdf/2410.06992' },85 { label: 'Terminal-Bench 2.0 — containerized terminal tasks', url: 'https://www.tbench.ai/' },86 { label: 'The Pilot/ForgeCode cheating audit (UPenn) — why we grade by simulation', url: 'https://debugml.github.io/cheating-agents/' },87 { label: 'CRUXEval — output-prediction evaluation paradigm', url: 'https://arxiv.org/abs/2401.03065' },88 { label: 'InterCode — execution-graded interactive shell tasks', url: 'https://arxiv.org/abs/2306.14898' },89 { label: 'Smoosh — mechanized POSIX shell semantics (why we whitelist)', url: 'https://arxiv.org/abs/1907.05308' },90 { label: 'LiveBench — procedural regeneration against contamination', url: 'https://arxiv.org/abs/2406.19314' },91 { label: 'Math-Verify — extraction leniency reshuffles leaderboards (HF)', url: 'https://huggingface.co/blog/math_verify_leaderboard' },92 { label: 'xFinder — regex extraction is only ~74% accurate', url: 'https://arxiv.org/abs/2405.11874' },93 { label: 'ReasonIF — reasoning models ignore format instructions', url: 'https://arxiv.org/abs/2510.15211' },94 { label: 'FormatSpread — 76-point spreads from formatting alone', url: 'https://arxiv.org/abs/2310.11324' },95 { label: 'Just Ask for Calibration — verbalized confidence (EMNLP 2023)', url: 'https://aclanthology.org/2023.emnlp-main.330/' },96 { label: 'WebArena-Verified — checker misalignment fixes', url: 'https://openreview.net/forum?id=94tlGxmqkN' },97];9899export default function MethodologyPage() {100 const samples = TEMPLATES.map((t) => {101 const seed = `docs:${t.id}`;102 const item = t.render(createRng(seed), seed);103 return { template: t, item };104 });105 const byDomain = new Map<string, typeof samples>();106 for (const s of samples) {107 byDomain.set(s.item.domain, [...(byDomain.get(s.item.domain) ?? []), s]);108 }109110 return (111 <div className="space-y-10">112 <div className="space-y-2">113 <h1 className="text-2xl font-bold text-zinc-900 sm:text-3xl">Methodology</h1>114 <p className="max-w-3xl text-sm leading-relaxed text-zinc-600">115 Index version {INDEX_VERSION}. Everything below is reproducible: template generators are116 public, every displayed number traces to an immutable score run with stored raw117 responses, and the machine-readable configuration is served at{' '}118 <a href="/api/v1/methodology" className="underline hover:text-zinc-900">119 /api/v1/methodology120 </a>121 . Scored item instantiations and answer keys never leave the server.122 </p>123 </div>124125 <section className="grid gap-4 md:grid-cols-2">126 {PILLARS.map((p) => (127 <div key={p.title} className="rounded-xl border border-zinc-200 p-5">128 <h2 className="mb-2 font-semibold text-zinc-900">{p.title}</h2>129 <p className="text-sm leading-relaxed text-zinc-600">{p.body}</p>130 </div>131 ))}132 </section>133134 <section className="space-y-4">135 <h2 className="text-xl font-semibold text-zinc-900">Scoring mathematics</h2>136 <div className="grid gap-4 md:grid-cols-2">137 <div className="rounded-xl border border-zinc-200 bg-white p-5">138 <h3 className="mb-2 text-sm font-semibold text-zinc-900">Item response model (2PL)</h3>139 <pre className="overflow-x-auto rounded-lg bg-zinc-100 p-3 text-xs text-zinc-700">{`P(correct | θ, a, b) = 1 / (1 + exp(−a·(θ − b)))140141MAP objective (per domain, missing-aware):142 L = Σ observed [ x·log P + (1−x)·log(1−P) ]143 − θ²/2σθ² − b²/2σb² − (log a)²/2σloga²144 priors: θ ~ N(0,1) · b ~ N(0,1.5) · log a ~ N(0,0.5)145146Optimizer: Adam, warm-started from accuracy logits,147θ recentered each step (scale pinned by priors),148max 500 iterations, tolerance 1e-6.`}</pre>149 </div>150 <div className="rounded-xl border border-zinc-200 bg-white p-5">151 <h3 className="mb-2 text-sm font-semibold text-zinc-900">Uncertainty & display scale</h3>152 <pre className="overflow-x-auto rounded-lg bg-zinc-100 p-3 text-xs text-zinc-700">{`SE(θ) = 1 / √( Σᵢ aᵢ²·P·(1−P) + 1/σθ² )153 (Fisher information + prior precision)154155Domain composite (all terms in [0,1]):156 C = w_acc·σ(θ) + w_con·consistency157 + w_cal·calibration + w_res·resistance158 weights renormalize over measured terms159160Domain score = 1000·C, CI via delta method161 through the accuracy term162Global Index = 1000·Σ_d w_d·C_d (equal w_d,163 renormalized over covered domains)164Global CI = per-domain half-widths combined165 in quadrature (independent fits)`}</pre>166 </div>167 <div className="rounded-xl border border-zinc-200 bg-white p-5">168 <h3 className="mb-2 text-sm font-semibold text-zinc-900">Bradley-Terry (duel domains)</h3>169 <pre className="overflow-x-auto rounded-lg bg-zinc-100 p-3 text-xs text-zinc-700">{`P(i beats j) = pᵢ / (pᵢ + pⱼ)170ties → half-win to each side171MM update: pᵢ ← Wᵢ / Σⱼ Nᵢⱼ/(pᵢ+pⱼ)172damping: +0.1 phantom win per pair (finite173strengths for isolated models)174SE from Fisher info: Iᵢᵢ = Σⱼ Nᵢⱼ·pᵢⱼ·(1−pᵢⱼ)175log-strengths standardized → same θ scale176as IRT domains before rescaling.`}</pre>177 </div>178 <div className="rounded-xl border border-zinc-200 bg-white p-5">179 <h3 className="mb-2 text-sm font-semibold text-zinc-900">Sub-metric definitions</h3>180 <pre className="overflow-x-auto rounded-lg bg-zinc-100 p-3 text-xs text-zinc-700">{`consistency = mean over items of181 (# samples agreeing with modal answer / k)182 k samples at the model's default temperature183184calibration = 1 − ECE (10 equal-width bins over185 the model's self-reported 0-100 confidence)186187contamination_delta = acc(anchors) − acc(fresh)188resistance = clamp01(1 − Δ/0.2)189 (a 20-point memorization gap ⇒ zero credit)190191latency_p50, cost_per_1k_items: measured per192call, published, NEVER blended into quality.`}</pre>193 </div>194 </div>195 </section>196197 <section className="space-y-3">198 <h2 className="text-xl font-semibold text-zinc-900">Item lifecycle & bank hygiene</h2>199 <ol className="list-decimal space-y-2 pl-5 text-sm leading-relaxed text-zinc-600">200 <li><span className="font-medium text-zinc-800">Generation</span> — every scored batch draws a fresh batch seed; each item derives from <code>seed:domain:index</code> through a deterministic PRNG (mulberry32 over an FNV-1a hash), so a batch is fully reproducible from its recorded seed yet unpredictable in advance.</li>201 <li><span className="font-medium text-zinc-800">Anchors</span> — positions below the anchor fraction (≤20%, capped in code) derive from a <em>fixed</em> seed stream (<code>anchor:domain:index</code>) — byte-identical across every run, forever. They exist only to measure longitudinal drift and the memorization gap.</li>202 <li><span className="font-medium text-zinc-800">Administration</span> — scored sample at temperature 0 with a 16,384-token completion budget; k additional samples at the model's default temperature for consistency. Full audit row per call: exact request params, raw response, tokens, measured latency, measured cost.</li>203 <li><span className="font-medium text-zinc-800">Grading</span> — mechanical only: normalized exact match, numeric tolerance (1e-6 relative), canonical-JSON equality for call sequences, exact multi-line match for terminal output, constraint-checker stacks for instruction following. No LLM ever grades a scored item.</li>204 <li><span className="font-medium text-zinc-800">Retirement</span> — after every refit, items with discrimination a < {IRT_HYPERPARAMS.minDiscrimination} or |b| > {IRT_HYPERPARAMS.maxAbsDifficultyLogits} logits are flagged <code>flagged_for_retirement</code>; flagged parameter regions are reviewed before the next run. Dead items never silently dilute the index.</li>205 <li><span className="font-medium text-zinc-800">Degradation</span> — a batch with >2% failed calls is marked <code>degraded</code>, shown as such, and excluded from every fit until re-run. Truncated completions are unscored, never counted wrong.</li>206 </ol>207 </section>208209 <section className="space-y-3">210 <h2 className="text-xl font-semibold text-zinc-900">Judge protocol (duel domains) — full detail</h2>211 <ul className="list-disc space-y-2 pl-5 text-sm leading-relaxed text-zinc-600">212 <li><span className="font-medium text-zinc-800">Panel</span>: three judge models from at least two different providers, configured explicitly (never hardcoded); a model never judges a duel it participates in — ineligible judges are excluded per-duel, and a duel with fewer than two eligible judges is skipped entirely.</li>213 <li><span className="font-medium text-zinc-800">Position swap</span>: judges alternate which response they see first; the swap is recorded on every verdict row and verdicts are un-swapped before aggregation, so position bias is both mitigated and measurable.</li>214 <li><span className="font-medium text-zinc-800">Verdicts</span>: judges output a single-token verdict (1 / 2 / TIE) at temperature 0; unparseable verdicts count conservatively as ties; empty or degenerate model responses can never win.</li>215 <li><span className="font-medium text-zinc-800">Anti-verbosity</span>: the judge prompt explicitly forbids rewarding length or style over substance; for SVG duels judges evaluate geometric fidelity, brand-color accuracy and vector cleanliness of the code.</li>216 <li><span className="font-medium text-zinc-800">Published diagnostics</span>: per run — judged-verdict count, pair count, and full-panel agreement rate, stored in the run's immutable <code>fit_diagnostics</code>.</li>217 </ul>218 </section>219220 <section className="space-y-3">221 <h2 className="text-xl font-semibold text-zinc-900">Known limitations (read this too)</h2>222 <ul className="list-disc space-y-2 pl-5 text-sm leading-relaxed text-zinc-600">223 <li><span className="font-medium text-zinc-800">IRT with few examinees</span>: item-parameter recovery degrades below ~100 models. We deliberately rank a wide roster (135 models incl. small anchors) to stabilize the scale, and CIs are published precisely so you never over-read a 10-point gap.</li>224 <li><span className="font-medium text-zinc-800">Judge subjectivity</span>: duel domains inherit judge preferences even with a cross-provider panel, swaps and agreement reporting. Duel scores are best read comparatively, not absolutely.</li>225 <li><span className="font-medium text-zinc-800">Provider variance</span>: models are reached through OpenRouter routing; latency (and occasionally behavior) can vary by upstream provider. Latency is reported as measured p50 per run, not a hardware-normalized figure.</li>226 <li><span className="font-medium text-zinc-800">Template scope</span>: generators cover targeted, mechanically-gradeable slices of each capability — deliberately narrow and deep rather than broad and judge-dependent. The template list is public; judge for yourself what is and isn't measured.</li>227 <li><span className="font-medium text-zinc-800">Single-turn agentic</span>: v0.2 grades planned call sequences in one shot; multi-turn interactive episodes with injected tool errors are on the roadmap.</li>228 </ul>229 </section>230231 <section className="space-y-3">232 <h2 className="text-xl font-semibold text-zinc-900">The pipeline, end to end</h2>233 <ol className="list-decimal space-y-2 pl-5 text-sm leading-relaxed text-zinc-600">234 <li>Model catalog and per-token pricing sync daily from the OpenRouter /models endpoint; the ranked roster is explicit configuration.</li>235 <li>Per model, per graded domain: a fresh seeded batch (30 items, ≤20% anchors) is generated; every item is asked at temperature 0 (scored) plus k consistency samples at default temperature; every call stores exact request params, raw response, tokens, measured latency and cost.</li>236 <li>Answers are extracted by the lenient cascade; truncated completions are unscored; batches with >2% failed calls are flagged degraded and excluded from fits until re-run.</li>237 <li>After every model completes, the 2PL fit re-runs over all complete same-version batches; Bradley-Terry re-fits the duel domains; scores publish with 95% CIs and the leaderboard updates live.</li>238 <li>Items with a<{IRT_HYPERPARAMS.minDiscrimination} or |b|>{IRT_HYPERPARAMS.maxAbsDifficultyLogits} logits are auto-flagged for retirement; every methodology change bumps the semver index version with a public changelog.</li>239 </ol>240 </section>241242 <section className="space-y-4">243 <h2 className="text-xl font-semibold text-zinc-900">Real sample questions, per domain</h2>244 <p className="text-sm text-zinc-500">245 Rendered live from the public template generators with documentation-only seeds (never246 used in scoring — scored runs draw fresh seeds every time). Reference answers shown where247 the item is mechanically graded.248 </p>249 {[...byDomain.entries()].map(([domain, list]) => (250 <div key={domain} className="space-y-2">251 <h3 className="text-sm font-semibold uppercase tracking-wide text-emerald-600">252 {domain.replaceAll('_', ' ')}253 </h3>254 {list.map(({ template, item }) => (255 <details key={template.id} className="rounded-xl border border-zinc-200">256 <summary className="cursor-pointer select-none px-4 py-3 text-sm text-zinc-800 hover:bg-zinc-50">257 <code className="text-emerald-700">{template.id}</code>258 <span className="ml-2 text-xs text-zinc-500">{template.description}</span>259 </summary>260 <div className="space-y-3 border-t border-zinc-200 p-4">261 {item.svg && (262 <div263 className="max-w-md overflow-hidden rounded-lg border border-zinc-300"264 dangerouslySetInnerHTML={{ __html: item.svg }}265 />266 )}267 <pre className="max-h-96 overflow-auto whitespace-pre-wrap rounded-lg bg-zinc-100 p-3 text-xs leading-relaxed text-zinc-700">268 {item.prompt}269 </pre>270 <p className="text-xs text-zinc-500">271 grading: <code>{item.grading}</code>272 {item.answerKey && item.grading !== 'constraints' && (273 <>274 {' · '}reference answer (docs seed only):{' '}275 <code className="break-all text-zinc-700">276 {item.answerKey.slice(0, 300)}277 {item.answerKey.length > 300 ? '…' : ''}278 </code>279 </>280 )}281 {item.grading === 'constraints' && (282 <>283 {' · '}graded by mechanical constraint checkers:{' '}284 <code className="break-all text-zinc-700">{item.answerKey}</code>285 </>286 )}287 </p>288 </div>289 </details>290 ))}291 </div>292 ))}293 <div className="rounded-xl border border-zinc-200 p-4 text-sm text-zinc-600">294 <span className="font-semibold text-zinc-800">Duel domains</span> —{' '}295 {DUEL_DOMAINS.map((d) => d.replaceAll('_', ' ')).join(', ')}: open-ended generated tasks296 (constrained writing briefs; gray-zone assistance scenarios; “reproduce the297 <brand> logo in raw SVG from memory”) judged pairwise by a 3-judge298 cross-provider panel and aggregated with Bradley-Terry. Judge prompts are public in the299 repository; they contain no secret rubrics.300 </div>301 </section>302303 <section className="space-y-3">304 <h2 className="text-xl font-semibold text-zinc-900">Weights (v{INDEX_VERSION}) — and why</h2>305 <p className="max-w-3xl text-sm leading-relaxed text-zinc-600">306 Domain weights are <span className="text-zinc-800">equal by design</span>: absent a307 task-utility function, any unequal weighting is an editorial value judgment; the308 maximum-entropy prior is the only non-arbitrary default, and per-domain scores are always309 published so you can re-weight for your own use case. Sub-metric weights favor the latent310 ability estimate (accuracy_irt {SUBMETRIC_WEIGHTS.accuracy_irt}) with robustness311 corrections for answer stability ({SUBMETRIC_WEIGHTS.consistency}) and template312 memorization ({SUBMETRIC_WEIGHTS.contamination_resistance}), and a smaller calibration313 term ({SUBMETRIC_WEIGHTS.calibration}) reflecting its higher measurement noise at current314 sample sizes. Latency and cost are excluded from quality entirely — Pareto frontier only.315 </p>316 <div className="grid gap-6 md:grid-cols-2">317 <div>318 <h3 className="mb-2 text-sm uppercase tracking-wide text-zinc-500">Domain weights (Global Index)</h3>319 <ul className="space-y-1 text-sm text-zinc-700">320 {DOMAINS.map((d) => (321 <li key={d} className="flex justify-between border-b border-zinc-100 py-1">322 <span className="capitalize">{d.replaceAll('_', ' ')}</span>323 <span className="tabular-nums">{DOMAIN_WEIGHTS[d].toFixed(4)}</span>324 </li>325 ))}326 </ul>327 </div>328 <div>329 <h3 className="mb-2 text-sm uppercase tracking-wide text-zinc-500">Sub-metric weights (domain composite)</h3>330 <ul className="space-y-1 text-sm text-zinc-700">331 {Object.entries(SUBMETRIC_WEIGHTS).map(([k, w]) => (332 <li key={k} className="flex justify-between border-b border-zinc-100 py-1">333 <span>{k}</span>334 <span className="tabular-nums">{w.toFixed(2)}</span>335 </li>336 ))}337 </ul>338 <p className="mt-3 text-xs text-zinc-500">339 Missing sub-metrics renormalize (a domain measured on accuracy alone neither gains340 nor loses). θ rescale: {THETA_SCALE.center} + {THETA_SCALE.slope}·θ; contamination341 floor Δ={CONTAMINATION_DELTA_FLOOR}.342 </p>343 </div>344 </div>345 </section>346347 <section className="space-y-2">348 <h2 className="text-xl font-semibold text-zinc-900">IRT hyperparameters</h2>349 <pre className="overflow-x-auto rounded-xl border border-zinc-200 bg-zinc-50 p-4 text-xs text-zinc-700">350 {JSON.stringify(IRT_HYPERPARAMS, null, 2)}351 </pre>352 </section>353354 <section className="space-y-3">355 <h2 className="text-xl font-semibold text-zinc-900">Versioning, reproducibility & audit trail</h2>356 <ul className="list-disc space-y-2 pl-5 text-sm leading-relaxed text-zinc-600">357 <li>Every methodology change bumps the semver <code>INDEX_VERSION</code> with an entry in the public changelog; scores from different versions are never mixed in one fit.</li>358 <li>Every displayed number traces to an immutable <code>score_runs</code> row: item-set hash, model set, index version, fit diagnostics, timestamps. Historical runs are never edited.</li>359 <li>Every model call stores its exact request parameters, raw response, token usage, measured latency and measured cost — a ranking without stored raw responses would be invalid by our own rules.</li>360 <li>Weights, hyperparameters and domain definitions live in exactly one source file and are served machine-readable at <a className="underline hover:text-zinc-900" href="/api/v1/methodology">/api/v1/methodology</a>.</li>361 <li>Batches are reproducible from their recorded seeds; anchors are byte-stable across all runs.</li>362 </ul>363 </section>364365 <section className="space-y-3">366 <h2 className="text-xl font-semibold text-zinc-900">References & influences</h2>367 <p className="max-w-3xl text-sm text-zinc-600">368 The design draws on the following research — as inspiration and evidence, not as source369 material: every environment, template, item and grader here is original and home-made.370 </p>371 <ul className="grid gap-1.5 text-sm sm:grid-cols-2">372 {REFERENCES.map((r) => (373 <li key={r.url}>374 <a375 href={r.url}376 target="_blank"377 rel="noreferrer"378 className="text-zinc-600 underline decoration-zinc-300 hover:text-emerald-600"379 >380 {r.label}381 </a>382 </li>383 ))}384 </ul>385 </section>386387 <section className="rounded-xl border border-zinc-200 bg-white p-5">388 <h2 className="mb-2 text-xl font-semibold text-zinc-900">Contact & corrections</h2>389 <p className="text-sm leading-relaxed text-zinc-600">390 Methodology questions, disputes about a score, retirement appeals for an item family, or391 requests to add a model:{' '}392 <span className="font-medium text-zinc-800">Simon-Pierre Boucher</span> ·{' '}393 <a className="underline hover:text-emerald-600" href="mailto:contact@spboucher.ai">394 contact@spboucher.ai395 </a>396 . Substantiated corrections are applied in a new score run — never by editing history —397 and acknowledged in the changelog.398 </p>399 </section>400 </div>401 );402}403