# Proposed expert annotation protocol (NOT executed in this study)

WP20 reports no human labels. This protocol specifies how the AI–record agreement results should be complemented by expert annotations in a follow-up; it is reproduced in the manuscript appendix.

# Raters and sampling

• Three independent raters: two chartered appraisers (OEAQ members) and one building technologist; none involved in the schema design. Training on a 40-listing pilot with adjudication; the pilot is excluded from reliability statistics.
• 300 listings drawn from the extraction sample with stratification by CMA, age band and price tier; each rater sees the same first 10 images and the masked text, in two separate sessions (images-only first, then images + text) to measure the text effect on human judgement.

# Coding manual (summary; full manual = configs/ontology.yaml allowed values)

• One form per listing with the property-level attributes of the ontology; for every attribute the rater selects a value or not_visible and records a confidence (low / medium / high) and the image indices used as evidence.
• Condition scales are anchored with written descriptors and reference photographs (to be assembled from licensed sources).
• Latent attributes (foundation, hidden moisture, structure, permits, tenure) are not asked.

# Reliability and adjudication

• Nominal attributes: Krippendorff's α (nominal) across the three raters and Cohen's κ per pair; ordinal grades and conditions: weighted κ (quadratic) and ICC(2,1) for the composite condition; minimum acceptable α = 0.67 for reporting, 0.80 for use as reference labels (Krippendorff's conventional thresholds, cited in the paper).
• Disagreements adjudicated by majority; three-way disagreements by a fourth senior appraiser; adjudication log kept.
• AI–human comparison: the same metrics with each AI tier treated as a fourth rater; disagreement cases catalogued by error type (occlusion, staging, lighting, material look-alikes, scene misclassification, hallucinated attribute).

# Audit trail

• Rater id, timestamps, time per listing (for the review-cost analysis), form versions, image hashes (not the images), and the adjudication decisions are stored in a versioned CSV; personal data of raters are kept outside the repository.