1# Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca2# Prospective evaluation set (WP 26)34## 1. What it is56The preregistration (CLAUDE.md §Experimental design; docs/research_plan.md) calls for *a separate prospective7evaluation set gathered after model/prompt freeze*. The protocol was frozen on 2026-10-28 01:05 UTC8(`results/freeze.json`, digest `f0e9d6a9…`). The prospective set is the set of Vrai Prix sales that are present in the9**live** database on node M3U96a but **absent from the 2026-10-21 snapshot** on which every partition10(train / validation / test / holdout), every prompt iteration and every aggregation weight was built. These sales11were registered at the Registre foncier after the snapshot: no model, prompt, hyper-parameter or weight could have12seen them. They are the cleanest out-of-sample test the design allows, at the cost of a small, time-concentrated13sample (one week of registrations).1415Two properties distinguish it from `holdout`: (i) holdout rows were in the snapshot during development (never read16by the LLM arms, but visible to the data pipeline, e.g. the tercile cut-points and the municipality lookup);17(ii) the prospective rows arrived after the freeze, so their very existence post-dates every design decision.1819## 2. When and how it was fetched2021```22python3 scripts/60_fetch_prospective.py # 2026-10-27, read-only SSH to M3U96a:~/apps/vrai-prix/data/vraiprix.db23```24Manifest `data/raw/prospective_2026-10-27.manifest.json`: 36,826 live rows dated ≥ 2026-07-01, of which **1,444 new25rows** (ids absent from the snapshot), sale dates **2026-10-06 … 2026-10-13**; **1,434 roll units** fetched for their26parcels (every new row matched a unit; some parcels sold twice). The raw files `data/raw/prospective_2026-10-27.parquet`27and `prospective_units_2026-10-27.parquet` carry the platform's columns, including addresses; they are gitignored and28never printed, and no address reaches any downstream artefact.2930Registration lag is 2–8 weeks, so the live DB on 2026-10-27 only contained sales up to 2026-10-13; the sales of312026-10-06/07 are the ones registered after the snapshot although dated inside the holdout window — they are32*prospective* by construction (absent from the snapshot), not holdout rows.3334## 3. From raw rows to the analytic contract — `scripts/61_build_prospective.py`3536The script re-implements, minimally and on these rows only, the WP 22 cleaning (`sharp-hpi-quebec/src/sharp_hpi/37cleaning.py`) and then the WP 26 funnel of `data/build.py`. The frozen code (prompts, configs, roles, protocols,38aggregation) is not touched; `analytic_schema.py` helpers (`project_xy`, `grid_id_expr`, `region_group_expr`,39`coerce_analytic`, `validate_analytic`) and `build.roll_reference_at_sale_expr` are imported, not copied.4041| Contract column | Source / rule |42|---|---|43| `tx_id`, `parcel_id`, `date`, `price`, `lat`, `lng` | transactions `id`, `id_provinc`, `date`, `amount`, `lat`, `lng` |44| `property_class` | WP 22 precedence from the roll: CUBF 1000 × `nb_logements` × `lien_physique` (1 dwelling + *Intégré* → condo; 1 dwelling + Détaché/Jumelé/En rangée → house; 2–5 dwellings → plex; ≥ 6 → multi; 1100 cottage; 1211 mobile); provider label (`property_type` ∈ unifamilial/condo/plex) only when the roll is incomplete. 1,175 of 1,182 kept rows classified from the roll |45| `subtype` | house: Détaché → detached, Jumelé → semi, En rangée → row; condo → integrated; else unknown (plex, provider-classified) |46| `floor_area_m2`, `land_area_m2`, `year_built`, `storeys`, `n_units` | units `aire_etages_m2`, `superficie_terrain_m2`, `annee_construction`, `nb_etages`, `nb_logements`; provider `floor_area` / `land_area` / `year_built` as fallback when the roll field is null (0.2 % of rows for floor area) |47| `roll_total_value` | assessment in force at the sale = transactions `valeur_role` (provider), falling back to units `valeur_role` — same as `build.py` |48| `roll_land_value`, `roll_building_value` | units `valeur_terrain` / `valeur_batiment` rescaled by assessment-at-sale ÷ current roll total (same as `build.py`) |49| `roll_reference_date` | units `dat_cond_marche` stepped back by whole triennial cycles until reference + 18 months ≤ sale date (`build.roll_reference_at_sale_expr`) |50| `mun_code`, `mun_name`, `region_name`, `urban_class` | lookup from `data/processed/analytic.parquet` keyed by the roll's `code_mun` (the mapping is unique per code in the snapshot; rows whose municipality is absent from the lookup are dropped and counted — 0 on 2026-10-27). The snapshot derived geography by a spatial join; the roll code agrees in > 99 % of snapshot rows |51| `borough_name` | mode of the snapshot `borough_name` in the same municipality × 1 km cell (null elsewhere; 24 % filled, as in the snapshot's 29 %) |52| `x_m`, `y_m`, `grid1k`, `grid5k`, `region_group`, `log_price` | `analytic_schema` helpers |53| `prior_sale_price/date` | strictly earlier sale of the same `parcel_id` in the **full** snapshot (`wp22_clean_transactions.parquet`, all 769,995 rows) ∪ the prospective rows |54| `price_tercile` | the snapshot's class-specific cut points (linear 1/3 and 2/3 quantiles of `analytic.parquet`: condo 300k/410k, house 305k/494.5k, plex 350k/625k) |55| `new_construction` | `year_built ≥ sale year` (0 rows; rows built after the sale are excluded) |56| `exclude_reason` | null for every kept row |5758Exclusion flags (WP 22 precedence, computed on the prospective rows ∪ the snapshot rows of the same dates so that a59multi-parcel deed or a same-day price conflict straddling the snapshot date is caught): exact duplicate inside the set;60**duplicate of a snapshot sale** (same parcel, date and price — 0 rows); same parcel, same date, different price;61built after sale; land (`type_prop = terrain`, CUBF 4xxx/9xxx); non-residential CUBF; shared parcel (> 10 sales622021–2026 in snapshot ∪ prospective); multi-parcel candidate (same date + price + street name, or same date + price +630.01° cell); price > 10 M; assessment-ratio outlier (|log(price/assessment) − median| > log 4, median of the snapshot's642026 sales by class × region, class-level fallback).6566### Funnel (2026-10-27)6768| step | rows | dropped |69|---|---|---|70| raw new rows | 1,444 | — |71| roll matched | 1,444 | 0 |72| no WP 22 exclusion flag | 1,251 | 193 (land 96, non-residential 56, assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2) |73| class ∈ {house, condo, plex} | 1,182 | 69 (cottage, mobile, multi, unclassifiable) |74| coordinates / parcel id / price in [50k, 5M] / municipality in lookup | 1,182 | 0 |7576Result: **1,182 sales** (house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99, Outaouais 57, other 567;77dates 2026-10-06 … 2026-10-13; 321 with a prior sale in the snapshot). `validate_analytic` passes.78Summary: `results/data/prospective_summary.json`. Table: `data/processed/prospective_analytic.parquet`.7980## 4. The runner bundle — `data/processed/prospective_bundle/`8182The LLM runner loads three parquet files from one directory and retrieves comparables and market context from the83analytic table it is given. A prospective subject needs the **snapshot history** for its comparables, so the bundle84holds the union:8586| file | content |87|---|---|88| `analytic.parquet` | snapshot analytic (631,866 rows) ∪ prospective rows (1,182) = 633,048 rows, same contract, validated |89| `splits.parquet` | snapshot splits ∪ prospective rows with `split = "prospective"` (`spatial_holdout_region` flag as in `splits.py`) |90| `llm_subsamples.parquet` | snapshot subsamples (validation 240 / test 600 / holdout 300) ∪ a stratified draw of the prospective rows: same strata (region_group × property_class × price_tercile), same seed (20261027), largest-remainder allocation via `splits.stratified_subsample`; n = min(40, available) = **40 rows over 18 strata** |91| `geo_features.parquet` | snapshot geo cache ∪ `features.geo.geo_features_batch` on the prospective rows (flood mapped 38 %, dissemination area 96 %) |9293`RunSpec.data_dir` / `scripts/30_run_arm.py --data-dir <dir>` redirects the three loads and the geo cache to that94directory; without it the behaviour is byte-for-byte the previous one (`tests/test_experiments_datadir.py`). The runner95takes the first *n* tx_ids (sorted) of the `prospective` subsample, builds the packet with cutoff = sale date − 196(comparables and the market index therefore come from snapshot sales and from earlier prospective sales only), and97runs the frozen protocol unchanged; `split = "prospective"` loads the frozen aggregation weights like test/holdout.9899## 5. Baselines — `scripts/62_prospective_baselines.py`100101Arms A–D (+ CatBoost/XGBoost robustness) are refit with the protocol of `scripts/10_train_baselines.py` (train on102`train`, tune on `validation`, 10 random-search trials, seed 20261027) and predict the 1,182 prospective rows. The feature103matrix is built by `features.matrix.build_feature_matrix` on the bundle (the as-of neighbourhood and municipality104features of a prospective sale need the history) and restricted to train / validation / prospective. The manifest105records an **as-of invariance check**: train and validation feature rows rebuilt on the union must equal those of106`data/processed/feature_matrix.parquet` (only later-dated sales were added). Output:107`results/runs/prospective_baselines/predictions.parquet` (`split = "prospective"`).108109## 6. Live LLM runs (frozen protocol)110111```112B=data/processed/prospective_bundle113AV_MAX_SPEND_USD=10 python3 scripts/30_run_arm.py --arm E_single_llm --split prospective --n 40 --live --committee mixed_random --run-id prospective_E_single_llm_mixed_random_mid --data-dir $B --concurrency 3114AV_MAX_SPEND_USD=30 python3 scripts/30_run_arm.py --arm F_independent --split prospective --n 40 --live --committee mixed_random --run-id prospective_F_independent_mixed_random_mid --data-dir $B --concurrency 3115AV_MAX_SPEND_USD=90 python3 scripts/30_run_arm.py --arm G_debate --rounds 2 --split prospective --n 40 --live --committee mixed_random --run-id prospective_G_debate_r2_mixed_random_mid --data-dir $B --concurrency 3116```117(`logs/prospective_runs.sh`; every run is resumable.) Outcomes are reported in `reports/phase_06_prospective.md`.118119## 7. Evaluation120121```122python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet123```124(the truth table is the bundle's `analytic.parquet`; runs named `prospective_*` and `prospective_baselines` carry125`split = "prospective"`). With 40 LLM subjects the arm comparison is under-powered; it is a consistency check of the126test/holdout results under a genuine post-freeze sample, not a primary endpoint.127128## 8. How to extend1291301. Wait for more registrations, then re-run `python3 scripts/60_fetch_prospective.py` (new stamp; the manifest keeps the131 freeze digest). Rows already present in an earlier prospective file are *not* deduplicated across stamps by the132 fetch script — pass `--date <stamp>` to `61` to build from one stamp, or concatenate stamps before building.1332. `python3 scripts/61_build_prospective.py --date <stamp>` rebuilds the table and the bundle; the subsample is a new134 seeded draw over the new population (nested with the previous one only within unchanged strata).1353. Re-run the three live commands (existing transcripts are skipped, new tx_ids are valued) and136 `scripts/62_prospective_baselines.py`.1374. Once the prospective sales resell, `scripts/51_eval_vs_resale.py` can use them for the resale-based value proxy138 (docs/fundamental_value.md).139140Limitations: one week of registrations (seasonal and regional composition not representative of a year); the141registration lag selects faster-registered deeds; 40 LLM subjects; the municipality lookup drops municipalities with142no snapshot sales (none this time); boroughs are imputed from the nearest snapshot sales rather than from a spatial143join.144