# Experiment plan — WP20 (written before any result was examined; 2026-10-20)

# Units, samples, splits

Unit of analysis: a listing matched to one assessment-roll unit (1,200 sampled from a frame of 7,745; design weights for frame-level statements). Images: first ≤ 12 (Haiku) / ≤ 10 (Sonnet, Opus) in seller order.
Reference data by evidence class: ADMIN (roll), TEXT (broker fields and free text), VISUAL/PROXY (per image, aggregated).
Splits for anything fitted (grade→cost multipliers, effective-age mapping): calibration on the Montréal + Gatineau + Sherbrooke + Trois-Rivières + Saguenay + other strata, geographic holdout = Québec CMA (configs/data.yaml). No model is fitted on asking prices: cost parameters come from public schedules; the only fitted objects are the agreement/weight statistics.
Bootstrap: 999 cluster resamples by municipality for every headline metric (listings within a municipality share the assessor and the local market); percentile 95 % intervals.

# Experiments (numbering follows the CLAUDE.md list)

#	Experiment	Inputs	Output artefacts
1	Attribute-level agreement with reference fields: cladding, roof covering, garage, basement finish, storeys, building type, kitchen counters/cabinets vs broker fields; storeys vs roll; confusion matrices, accuracy/κ/F1, abstention (`not_visible`) rates; error taxonomy from `evidence` strings	img_* + text_* + frame	T1 agreement table, F1 confusion heatmaps, T2 error taxonomy
2	Visual vs text vs combined: information content of each channel for each attribute (coverage × agreement) and for cost inputs (grade, condition)	same	T3, F2
3	Image count (K = 1, 2, 4, 8, 10, 12 by truncation) and photo quality: stability of property-level attributes and of the cost indication	img_haiku	F3, T4
4	Expert agreement: not available (no human raters). Substitute: cross-model agreement (Haiku/Sonnet/Opus) per attribute, κ; disagreement cases catalogued; expert protocol proposed (docs/annotation_protocol.md)	img_*	T5, F4
5	Cost-component decomposition L / RCN / D / S by price tier, age band, CMA	cost model	F5, T6
6	Effect of visually inferred finish grade and condition on RCN and on the cost indication; B0 vs B2/B3 deltas	cost model	F6
7	Sensitivity to regional cost level (Altus band ±, StatCan CMA relative growth)	cost model	F7
8	Sensitivity to depreciation assumptions: economic life 50/60/75 years, condition→effective-age mapping strength, functional/external obsolescence scenario	cost model	F8, T7
9	MNAR photo selection: simulate scene-dependent missingness (drop basement/kitchen images with probability rising in poor condition) and measure bias of grades and cost; empirical MNAR check: listings without a basement/kitchen photo vs broker "Sous-sol" field	img_* + text	F9, T8
10	Geographic holdout (Québec CMA) and roll-vintage (2022/2023/2024 market dates) out-of-sample checks of the cost→assessor comparison	cost model	T9
11	Subgroup diagnostics: age band, price tier, building type, source, image count	all	T10, F10
12	Calibration of the extractor's stated confidence vs agreement with reference; abstention curves; selective prediction	img_*	F11, T11
13	Counterfactual renovation/condition scenarios (grade +1, condition +1, extensive renovation) on the cost indication — labelled scenarios	cost model	F12
14	Monte Carlo propagation: attribute posterior from confidence + cross-model disagreement; unit-cost band; economic life; land index → interval for the cost indication; coverage vs assessor building value	cost model	F13, T12
15	Market-value alignment (labelled as such): cost indication vs asking price; vs Vrai Prix `est`/`cost_est`; vs recorded sale price for units sold after listing (if n ≥ 30)	frame + transactions	F14, T13
16	Human-in-the-loop gains: not feasible (no raters); replaced by a cost-of-review analysis: how many listings would need review under confidence thresholds	img_*	T14

# Pre-registered reading rules

Agreement with broker fields is agreement, not accuracy: broker fields are claims. Where roll fields exist (storeys, building type via CUBF/physical link not in our extract), they are the reference.
The assessor's building value is a cost-based mass-appraisal indication at the roll date, not truth; closeness to it is reported as "assessor alignment" after indexing with the StatCan BCPI from the roll market date to 2026-Q2.
Asking price comparisons are "market alignment"; never called accuracy.
Any metric computed on Sonnet uses the common first 10 images; Haiku-only analyses (K up to 12) are flagged.