1# Taxonomy validation design (Phase 2, checkpoint H-2)23Sample: 2,000 eligible works, stratified = 50 per rule-assigned primary category (40 × 50), drawn with seed 44 from works4with an abstract (month-precision or year-precision alike), plus 200 works with no category assigned (coverage check).5Raters: two independent LLM raters that never see the rule-based assignment — rater A `claude-haiku-5-5`,6rater B `claude-sonnet-5-5` (different model families within Claude; the PI is the third rater on a 200-work7subsample, `pi_review = TRUE`). Each rater receives the 40 category definitions (from `ontology.yaml`) and, per work,8title + abstract + venue + year, and returns, under a JSON schema: `categories` (all applicable ids, 0–5),9`primary` (one id or null), `confidence` (0–1), `rationale` (≤ 30 words).10Metrics (taxonomy/validation/metrics.json): per-category precision/recall/F1 of the rules against each rater and against11the rater consensus (both raters agree); micro/macro F1; Cohen's κ (primary) and Krippendorff's α (multi-label, MASI12distance) between raters; agreement on "no category". Disagreements are exported for the PI (validation/disagreements.csv).13Decision rule fixed in advance: a category with consensus F1 < 0.60 triggers a mapping revision (logged as an amendment)14and a re-run of the classification; the validation sample is then re-rated with the revised rules held out of the rater15prompt (the raters never see rules, so no leakage).16Budget: ≤ 25 USD (A-003).17