# Taxonomy validation design (Phase 2, checkpoint H-2)

Sample: 2,000 eligible works, stratified = 50 per rule-assigned primary category (40 × 50), drawn with seed 44 from works with an abstract (month-precision or year-precision alike), plus 200 works with no category assigned (coverage check). Raters: two independent LLM raters that never see the rule-based assignment — rater A claude-haiku-5-5, rater B claude-sonnet-5-5 (different model families within Claude; the PI is the third rater on a 200-work subsample, pi_review = TRUE). Each rater receives the 40 category definitions (from ontology.yaml) and, per work, title + abstract + venue + year, and returns, under a JSON schema: categories (all applicable ids, 0–5), primary (one id or null), confidence (0–1), rationale (≤ 30 words). Metrics (taxonomy/validation/metrics.json): per-category precision/recall/F1 of the rules against each rater and against the rater consensus (both raters agree); micro/macro F1; Cohen's κ (primary) and Krippendorff's α (multi-label, MASI distance) between raters; agreement on "no category". Disagreements are exported for the PI (validation/disagreements.csv). Decision rule fixed in advance: a category with consensus F1 < 0.60 triggers a mapping revision (logged as an amendment) and a re-run of the classification; the validation sample is then re-rated with the revised rules held out of the rater prompt (the raters never see rules, so no leakage). Budget: ≤ 25 USD (A-003).