# Human rating protocol (prepared, not executed)

Raters: ≥ 2 PhD-level finance researchers, blind to condition; each rates all outputs of a (task, document) set presented in random order under random letters. Scale: the six primary dimensions of prompts/evaluation/judge_rubric_v1.md (1–7) + n_invented_facts + free comment. Calibration: raters first score two anchor sets together and discuss. Reliability: Krippendorff's alpha (interval) per dimension; disagreement > 2 points resolved by discussion, original scores kept for analysis. Outputs to rate are in experiments/outputs/heldout_v1/outputs/, blinded sets are generated by scripts/06_make_rating_packets.py (to be written when raters are available). Store ratings as experiments/outputs/heldout_v1/human_ratings.csv (rater, task, doc, rep, letter, dimension, score).