# Taxonomy validation design (Phase 2, checkpoint H-2)

Sample: 2,000 eligible works, stratified = 50 per rule-assigned primary category (40 × 50), drawn with seed 44 from works
with an abstract (month-precision or year-precision alike), plus 200 works with no category assigned (coverage check).
Raters: two independent LLM raters that never see the rule-based assignment — rater A `claude-haiku-5-5`,
rater B `claude-sonnet-5-5` (different model families within Claude; the PI is the third rater on a 200-work
subsample, `pi_review = TRUE`). Each rater receives the 40 category definitions (from `ontology.yaml`) and, per work,
title + abstract + venue + year, and returns, under a JSON schema: `categories` (all applicable ids, 0–5),
`primary` (one id or null), `confidence` (0–1), `rationale` (≤ 30 words).
Metrics (taxonomy/validation/metrics.json): per-category precision/recall/F1 of the rules against each rater and against
the rater consensus (both raters agree); micro/macro F1; Cohen's κ (primary) and Krippendorff's α (multi-label, MASI
distance) between raters; agreement on "no category". Disagreements are exported for the PI (validation/disagreements.csv).
Decision rule fixed in advance: a category with consensus F1 < 0.60 triggers a mapping revision (logged as an amendment)
and a re-run of the classification; the validation sample is then re-rated with the revised rules held out of the rater
prompt (the raters never see rules, so no leakage).
Budget: ≤ 25 USD (A-003).