# Methodology Changelog

All notable methodology changes. Every entry corresponds to an INDEX_VERSION bump in packages/scoring/src/weights.ts.

# [0.2.0] — 2026-08-05

Major discrimination & robustness overhaul (research-driven; see the References section of the methodology page).

agentic — simulated tool-calling with distractor tools and deterministic simulators (support-desk triage under policy, treasury ledger with overdraft pre-funding, DAG-ordered deployments) plus a dedicated context-load family (120-300-row ledgers dense with near-miss decoys). Graded by canonical-JSON call-sequence equality — binary, judge-free.
terminal — home-made shell simulation on a closed unambiguous POSIX subset: pipeline stdout prediction, file-tree state tracking through mv/cp/rm/cd, && / || execution-trace + exit-code prediction. Locale/GNU-vs-BSD/float-format constructs excluded by design.
svg_design — reproduce a real-world logo in raw SVG from memory; 3-judge cross-provider panel, position swaps, Bradley-Terry aggregation (now wired end-to-end into the refit).
vision_ocr — generated SVG scenes rasterized to PNG: code transcription under clutter and table reading + grounded arithmetic; ambiguous glyphs excluded; text-only models skip.

math: 6-8-step dependent chains, 2-variable systems with derived queries, 3-stage percentage chains with provably-inert distractor clauses, counterfactual base-k arithmetic, R-Horizon-style chained sub-problems.
reasoning: 7-entity orderings, non-adjacent transitive clues, decoy entity, interior-rank query.
code: nested loops with break/continue tracing.
knowledge: switched to free-response (removes the multiple-choice guessing floor; c≈0).
multilingual: full 0-999 number-word space (incl. French 70-99 irregular zone) both directions.
instruction_following: constraint stacking (5 simultaneous mechanically-verified constraints).

Extraction cascade: markdown-wrapped ANSWER tags, FINAL ANSWER variants, LaTeX \boxed{}, fenced-block extraction for JSON/multi-line answers; thousands separators/currency/unit normalization; hyphen/space orthography unification.
Truncated completions (finish_reason=length) are unscored, never wrong; completion budget raised to 16384 tokens.
New grading modes: json (canonical AST equality), lines (exact multi-line), constraints (mechanical checker stack).
Refits only consume batches of the current INDEX_VERSION.

Sub-metrics: accuracy_irt 0.60, consistency 0.15, contamination_resistance 0.15, calibration 0.10 (rationale documented on the methodology page). Domain weights remain equal.

Initial public methodology.

2PL IRT scoring (custom numpy MAP fit; priors θ~N(0,1), b~N(0,1.5), log a~N(0,0.5)).
Dynamic item generation with seeded perturbation; anchor stream capped at 20%; contamination_delta published with floor 0.2 for the resistance mapping.
Sub-metric weights: accuracy_irt 0.55, consistency 0.15, calibration 0.15, contamination_resistance 0.15. Latency/cost excluded by design (Pareto frontier).
Equal domain weights across the 8 v1 domains.
θ → index rescale: 500 + 150·θ (display); domain scores as 1000 × composite; 95% CIs everywhere.
Bradley-Terry layer for writing / safety_refusal_quality with 2-judge cross-provider protocol.
Degraded-batch rule: >2% failed calls excludes a batch from fits until re-run.