spb/wp20_uqo

1# Experiment plan — WP20 (written before any result was examined; 2026-10-20)23## Units, samples, splits45- Unit of analysis: a listing matched to one assessment-roll unit (1,200 sampled from a frame of 7,745; design weights for6  frame-level statements). Images: first ≤ 12 (Haiku) / ≤ 10 (Sonnet, Opus) in seller order.7- Reference data by evidence class: ADMIN (roll), TEXT (broker fields and free text), VISUAL/PROXY (per image, aggregated).8- Splits for anything *fitted* (grade→cost multipliers, effective-age mapping): calibration on the Montréal + Gatineau +9  Sherbrooke + Trois-Rivières + Saguenay + other strata, geographic holdout = Québec CMA (configs/data.yaml). No model is fitted10  on asking prices: cost parameters come from public schedules; the only fitted objects are the agreement/weight statistics.11- Bootstrap: 999 cluster resamples by municipality for every headline metric (listings within a municipality share the12  assessor and the local market); percentile 95 % intervals.1314## Experiments (numbering follows the CLAUDE.md list)1516| # | Experiment | Inputs | Output artefacts |17|---|---|---|---|18| 1 | Attribute-level agreement with reference fields: cladding, roof covering, garage, basement finish, storeys, building type, kitchen counters/cabinets vs broker fields; storeys vs roll; confusion matrices, accuracy/κ/F1, abstention (`not_visible`) rates; error taxonomy from `evidence` strings | img_* + text_* + frame | T1 agreement table, F1 confusion heatmaps, T2 error taxonomy |19| 2 | Visual vs text vs combined: information content of each channel for each attribute (coverage × agreement) and for cost inputs (grade, condition) | same | T3, F2 |20| 3 | Image count (K = 1, 2, 4, 8, 10, 12 by truncation) and photo quality: stability of property-level attributes and of the cost indication | img_haiku | F3, T4 |21| 4 | Expert agreement: **not available** (no human raters). Substitute: cross-model agreement (Haiku/Sonnet/Opus) per attribute, κ; disagreement cases catalogued; expert protocol *proposed* (docs/annotation_protocol.md) | img_* | T5, F4 |22| 5 | Cost-component decomposition L / RCN / D / S by price tier, age band, CMA | cost model | F5, T6 |23| 6 | Effect of visually inferred finish grade and condition on RCN and on the cost indication; B0 vs B2/B3 deltas | cost model | F6 |24| 7 | Sensitivity to regional cost level (Altus band ±, StatCan CMA relative growth) | cost model | F7 |25| 8 | Sensitivity to depreciation assumptions: economic life 50/60/75 years, condition→effective-age mapping strength, functional/external obsolescence scenario | cost model | F8, T7 |26| 9 | MNAR photo selection: simulate scene-dependent missingness (drop basement/kitchen images with probability rising in poor condition) and measure bias of grades and cost; empirical MNAR check: listings without a basement/kitchen photo vs broker "Sous-sol" field | img_* + text | F9, T8 |27| 10 | Geographic holdout (Québec CMA) and roll-vintage (2022/2023/2024 market dates) out-of-sample checks of the cost→assessor comparison | cost model | T9 |28| 11 | Subgroup diagnostics: age band, price tier, building type, source, image count | all | T10, F10 |29| 12 | Calibration of the extractor's stated confidence vs agreement with reference; abstention curves; selective prediction | img_* | F11, T11 |30| 13 | Counterfactual renovation/condition scenarios (grade +1, condition +1, extensive renovation) on the cost indication — labelled scenarios | cost model | F12 |31| 14 | Monte Carlo propagation: attribute posterior from confidence + cross-model disagreement; unit-cost band; economic life; land index → interval for the cost indication; coverage vs assessor building value | cost model | F13, T12 |32| 15 | Market-value alignment (labelled as such): cost indication vs asking price; vs Vrai Prix `est`/`cost_est`; vs recorded sale price for units sold after listing (if n ≥ 30) | frame + transactions | F14, T13 |33| 16 | Human-in-the-loop gains: **not feasible** (no raters); replaced by a cost-of-review analysis: how many listings would need review under confidence thresholds | img_* | T14 |3435## Pre-registered reading rules3637- Agreement with broker fields is **agreement**, not accuracy: broker fields are claims. Where roll fields exist (storeys,38  building type via CUBF/physical link not in our extract), they are the reference.39- The assessor's building value is a *cost-based mass-appraisal indication at the roll date*, not truth; closeness to it is40  reported as "assessor alignment" after indexing with the StatCan BCPI from the roll market date to 2026-Q2.41- Asking price comparisons are "market alignment"; never called accuracy.42- Any metric computed on Sonnet uses the common first 10 images; Haiku-only analyses (K up to 12) are flagged.43