# Proposed expert annotation protocol (NOT executed in this study)

WP20 reports no human labels. This protocol specifies how the AI–record agreement results should be complemented by expert
annotations in a follow-up; it is reproduced in the manuscript appendix.

## Raters and sampling
- Three independent raters: two chartered appraisers (OEAQ members) and one building technologist; none involved in the
  schema design. Training on a 40-listing pilot with adjudication; the pilot is excluded from reliability statistics.
- 300 listings drawn from the extraction sample with stratification by CMA, age band and price tier; each rater sees the same
  first 10 images and the masked text, in two separate sessions (images-only first, then images + text) to measure the text
  effect on human judgement.

## Coding manual (summary; full manual = `configs/ontology.yaml` allowed values)
- One form per listing with the property-level attributes of the ontology; for every attribute the rater selects a value or
  `not_visible` and records a confidence (low / medium / high) and the image indices used as evidence.
- Condition scales are anchored with written descriptors and reference photographs (to be assembled from licensed sources).
- Latent attributes (foundation, hidden moisture, structure, permits, tenure) are not asked.

## Reliability and adjudication
- Nominal attributes: Krippendorff's α (nominal) across the three raters and Cohen's κ per pair; ordinal grades and
  conditions: weighted κ (quadratic) and ICC(2,1) for the composite condition; minimum acceptable α = 0.67 for reporting,
  0.80 for use as reference labels (Krippendorff's conventional thresholds, cited in the paper).
- Disagreements adjudicated by majority; three-way disagreements by a fourth senior appraiser; adjudication log kept.
- AI–human comparison: the same metrics with each AI tier treated as a fourth rater; disagreement cases catalogued by error
  type (occlusion, staging, lighting, material look-alikes, scene misclassification, hallucinated attribute).

## Audit trail
- Rater id, timestamps, time per listing (for the review-cost analysis), form versions, image hashes (not the images),
  and the adjudication decisions are stored in a versioned CSV; personal data of raters are kept outside the repository.