spb/llmindex Public

1# Methodology Changelog23All notable methodology changes. Every entry corresponds to an `INDEX_VERSION` bump in4`packages/scoring/src/weights.ts`.56## [0.2.0] — 2026-08-0578Major discrimination & robustness overhaul (research-driven; see the References section of the9methodology page).1011### New domains (12 total, equal weights = maximum-entropy prior)12- **agentic** — simulated tool-calling with distractor tools and deterministic simulators13  (support-desk triage under policy, treasury ledger with overdraft pre-funding, DAG-ordered14  deployments) plus a dedicated **context-load** family (120-300-row ledgers dense with near-miss15  decoys). Graded by canonical-JSON call-sequence equality — binary, judge-free.16- **terminal** — home-made shell simulation on a closed unambiguous POSIX subset: pipeline stdout17  prediction, file-tree state tracking through mv/cp/rm/cd, && / || execution-trace + exit-code18  prediction. Locale/GNU-vs-BSD/float-format constructs excluded by design.19- **svg_design** — reproduce a real-world logo in raw SVG from memory; 3-judge cross-provider20  panel, position swaps, Bradley-Terry aggregation (now wired end-to-end into the refit).21- **vision_ocr** — generated SVG scenes rasterized to PNG: code transcription under clutter and22  table reading + grounded arithmetic; ambiguous glyphs excluded; text-only models skip.2324### Hardened existing domains25- math: 6-8-step dependent chains, 2-variable systems with derived queries, 3-stage percentage26  chains with provably-inert distractor clauses, counterfactual base-k arithmetic, R-Horizon-style27  chained sub-problems.28- reasoning: 7-entity orderings, non-adjacent transitive clues, decoy entity, interior-rank query.29- code: nested loops with break/continue tracing.30- knowledge: switched to free-response (removes the multiple-choice guessing floor; c≈0).31- multilingual: full 0-999 number-word space (incl. French 70-99 irregular zone) both directions.32- instruction_following: constraint stacking (5 simultaneous mechanically-verified constraints).3334### Grading & harness robustness35- Extraction cascade: markdown-wrapped ANSWER tags, FINAL ANSWER variants, LaTeX \boxed{},36  fenced-block extraction for JSON/multi-line answers; thousands separators/currency/unit37  normalization; hyphen/space orthography unification.38- Truncated completions (finish_reason=length) are unscored, never wrong; completion budget39  raised to 16384 tokens.40- New grading modes: `json` (canonical AST equality), `lines` (exact multi-line), `constraints`41  (mechanical checker stack).42- Refits only consume batches of the current INDEX_VERSION.4344### Weights45- Sub-metrics: accuracy_irt 0.60, consistency 0.15, contamination_resistance 0.15,46  calibration 0.10 (rationale documented on the methodology page). Domain weights remain equal.4748## [0.1.0] — 2026-08-054950Initial public methodology.5152- 2PL IRT scoring (custom numpy MAP fit; priors θ~N(0,1), b~N(0,1.5), log a~N(0,0.5)).53- Dynamic item generation with seeded perturbation; anchor stream capped at 20%;54  `contamination_delta` published with floor 0.2 for the resistance mapping.55- Sub-metric weights: accuracy_irt 0.55, consistency 0.15, calibration 0.15,56  contamination_resistance 0.15. Latency/cost excluded by design (Pareto frontier).57- Equal domain weights across the 8 v1 domains.58- θ → index rescale: 500 + 150·θ (display); domain scores as 1000 × composite; 95% CIs everywhere.59- Bradley-Terry layer for writing / safety_refusal_quality with 2-judge cross-provider protocol.60- Degraded-batch rule: >2% failed calls excludes a batch from fits until re-run.61