# Methodology Changelog

All notable methodology changes. Every entry corresponds to an `INDEX_VERSION` bump in
`packages/scoring/src/weights.ts`.

## [0.2.0] — 2026-08-05

Major discrimination & robustness overhaul (research-driven; see the References section of the
methodology page).

### New domains (12 total, equal weights = maximum-entropy prior)
- **agentic** — simulated tool-calling with distractor tools and deterministic simulators
  (support-desk triage under policy, treasury ledger with overdraft pre-funding, DAG-ordered
  deployments) plus a dedicated **context-load** family (120-300-row ledgers dense with near-miss
  decoys). Graded by canonical-JSON call-sequence equality — binary, judge-free.
- **terminal** — home-made shell simulation on a closed unambiguous POSIX subset: pipeline stdout
  prediction, file-tree state tracking through mv/cp/rm/cd, && / || execution-trace + exit-code
  prediction. Locale/GNU-vs-BSD/float-format constructs excluded by design.
- **svg_design** — reproduce a real-world logo in raw SVG from memory; 3-judge cross-provider
  panel, position swaps, Bradley-Terry aggregation (now wired end-to-end into the refit).
- **vision_ocr** — generated SVG scenes rasterized to PNG: code transcription under clutter and
  table reading + grounded arithmetic; ambiguous glyphs excluded; text-only models skip.

### Hardened existing domains
- math: 6-8-step dependent chains, 2-variable systems with derived queries, 3-stage percentage
  chains with provably-inert distractor clauses, counterfactual base-k arithmetic, R-Horizon-style
  chained sub-problems.
- reasoning: 7-entity orderings, non-adjacent transitive clues, decoy entity, interior-rank query.
- code: nested loops with break/continue tracing.
- knowledge: switched to free-response (removes the multiple-choice guessing floor; c≈0).
- multilingual: full 0-999 number-word space (incl. French 70-99 irregular zone) both directions.
- instruction_following: constraint stacking (5 simultaneous mechanically-verified constraints).

### Grading & harness robustness
- Extraction cascade: markdown-wrapped ANSWER tags, FINAL ANSWER variants, LaTeX \boxed{},
  fenced-block extraction for JSON/multi-line answers; thousands separators/currency/unit
  normalization; hyphen/space orthography unification.
- Truncated completions (finish_reason=length) are unscored, never wrong; completion budget
  raised to 16384 tokens.
- New grading modes: `json` (canonical AST equality), `lines` (exact multi-line), `constraints`
  (mechanical checker stack).
- Refits only consume batches of the current INDEX_VERSION.

### Weights
- Sub-metrics: accuracy_irt 0.60, consistency 0.15, contamination_resistance 0.15,
  calibration 0.10 (rationale documented on the methodology page). Domain weights remain equal.

## [0.1.0] — 2026-08-05

Initial public methodology.

- 2PL IRT scoring (custom numpy MAP fit; priors θ~N(0,1), b~N(0,1.5), log a~N(0,0.5)).
- Dynamic item generation with seeded perturbation; anchor stream capped at 20%;
  `contamination_delta` published with floor 0.2 for the resistance mapping.
- Sub-metric weights: accuracy_irt 0.55, consistency 0.15, calibration 0.15,
  contamination_resistance 0.15. Latency/cost excluded by design (Pareto frontier).
- Equal domain weights across the 8 v1 domains.
- θ → index rescale: 500 + 150·θ (display); domain scores as 1000 × composite; 95% CIs everywhere.
- Bradley-Terry layer for writing / safety_refusal_quality with 2-judge cross-provider protocol.
- Degraded-batch rule: >2% failed calls excludes a batch from fits until re-run.