Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
Prospective evaluation set (WP 26)
1. What it is
The preregistration (CLAUDE.md §Experimental design; docs/research_plan.md) calls for a separate prospective
evaluation set gathered after model/prompt freeze. The protocol was frozen on 2026-10-28 01:05 UTC
(results/freeze.json, digest f0e9d6a9…). The prospective set is the set of Vrai Prix sales that are present in the
live database on node M3U96a but absent from the 2026-10-21 snapshot on which every partition
(train / validation / test / holdout), every prompt iteration and every aggregation weight was built. These sales
were registered at the Registre foncier after the snapshot: no model, prompt, hyper-parameter or weight could have
seen them. They are the cleanest out-of-sample test the design allows, at the cost of a small, time-concentrated
sample (one week of registrations).
Two properties distinguish it from holdout: (i) holdout rows were in the snapshot during development (never read
by the LLM arms, but visible to the data pipeline, e.g. the tercile cut-points and the municipality lookup);
(ii) the prospective rows arrived after the freeze, so their very existence post-dates every design decision.
2. When and how it was fetched
python3 scripts/60_fetch_prospective.py # 2026-10-27, read-only SSH to M3U96a:~/apps/vrai-prix/data/vraiprix.dbManifest data/raw/prospective_2026-10-27.manifest.json: 36,826 live rows dated ≥ 2026-07-01, of which 1,444 new
rows (ids absent from the snapshot), sale dates 2026-10-06 … 2026-10-13; 1,434 roll units fetched for their
parcels (every new row matched a unit; some parcels sold twice). The raw files data/raw/prospective_2026-10-27.parquet
and prospective_units_2026-10-27.parquet carry the platform's columns, including addresses; they are gitignored and
never printed, and no address reaches any downstream artefact.
Registration lag is 2–8 weeks, so the live DB on 2026-10-27 only contained sales up to 2026-10-13; the sales of 2026-10-06/07 are the ones registered after the snapshot although dated inside the holdout window — they are prospective by construction (absent from the snapshot), not holdout rows.
3. From raw rows to the analytic contract — scripts/61_build_prospective.py
The script re-implements, minimally and on these rows only, the WP 22 cleaning (sharp-hpi-quebec/src/sharp_hpi/ cleaning.py) and then the WP 26 funnel of data/build.py. The frozen code (prompts, configs, roles, protocols,
aggregation) is not touched; analytic_schema.py helpers (project_xy, grid_id_expr, region_group_expr,
coerce_analytic, validate_analytic) and build.roll_reference_at_sale_expr are imported, not copied.
| Contract column | Source / rule |
|---|---|
tx_id, parcel_id, date, price, lat, lng |
transactions id, id_provinc, date, amount, lat, lng |
property_class |
WP 22 precedence from the roll: CUBF 1000 × nb_logements × lien_physique (1 dwelling + Intégré → condo; 1 dwelling + Détaché/Jumelé/En rangée → house; 2–5 dwellings → plex; ≥ 6 → multi; 1100 cottage; 1211 mobile); provider label (property_type ∈ unifamilial/condo/plex) only when the roll is incomplete. 1,175 of 1,182 kept rows classified from the roll |
subtype |
house: Détaché → detached, Jumelé → semi, En rangée → row; condo → integrated; else unknown (plex, provider-classified) |
floor_area_m2, land_area_m2, year_built, storeys, n_units |
units aire_etages_m2, superficie_terrain_m2, annee_construction, nb_etages, nb_logements; provider floor_area / land_area / year_built as fallback when the roll field is null (0.2 % of rows for floor area) |
roll_total_value |
assessment in force at the sale = transactions valeur_role (provider), falling back to units valeur_role — same as build.py |
roll_land_value, roll_building_value |
units valeur_terrain / valeur_batiment rescaled by assessment-at-sale ÷ current roll total (same as build.py) |
roll_reference_date |
units dat_cond_marche stepped back by whole triennial cycles until reference + 18 months ≤ sale date (build.roll_reference_at_sale_expr) |
mun_code, mun_name, region_name, urban_class |
lookup from data/processed/analytic.parquet keyed by the roll's code_mun (the mapping is unique per code in the snapshot; rows whose municipality is absent from the lookup are dropped and counted — 0 on 2026-10-27). The snapshot derived geography by a spatial join; the roll code agrees in > 99 % of snapshot rows |
borough_name |
mode of the snapshot borough_name in the same municipality × 1 km cell (null elsewhere; 24 % filled, as in the snapshot's 29 %) |
x_m, y_m, grid1k, grid5k, region_group, log_price |
analytic_schema helpers |
prior_sale_price/date |
strictly earlier sale of the same parcel_id in the full snapshot (wp22_clean_transactions.parquet, all 769,995 rows) ∪ the prospective rows |
price_tercile |
the snapshot's class-specific cut points (linear 1/3 and 2/3 quantiles of analytic.parquet: condo 300k/410k, house 305k/494.5k, plex 350k/625k) |
new_construction |
year_built ≥ sale year (0 rows; rows built after the sale are excluded) |
exclude_reason |
null for every kept row |
Exclusion flags (WP 22 precedence, computed on the prospective rows ∪ the snapshot rows of the same dates so that a
multi-parcel deed or a same-day price conflict straddling the snapshot date is caught): exact duplicate inside the set;
duplicate of a snapshot sale (same parcel, date and price — 0 rows); same parcel, same date, different price;
built after sale; land (type_prop = terrain, CUBF 4xxx/9xxx); non-residential CUBF; shared parcel (> 10 sales
2021–2026 in snapshot ∪ prospective); multi-parcel candidate (same date + price + street name, or same date + price +
0.01° cell); price > 10 M; assessment-ratio outlier (|log(price/assessment) − median| > log 4, median of the snapshot's
2026 sales by class × region, class-level fallback).
Funnel (2026-10-27)
| step | rows | dropped |
|---|---|---|
| raw new rows | 1,444 | — |
| roll matched | 1,444 | 0 |
| no WP 22 exclusion flag | 1,251 | 193 (land 96, non-residential 56, assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2) |
| class ∈ {house, condo, plex} | 1,182 | 69 (cottage, mobile, multi, unclassifiable) |
| coordinates / parcel id / price in [50k, 5M] / municipality in lookup | 1,182 | 0 |
Result: 1,182 sales (house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99, Outaouais 57, other 567;
dates 2026-10-06 … 2026-10-13; 321 with a prior sale in the snapshot). validate_analytic passes.
Summary: results/data/prospective_summary.json. Table: data/processed/prospective_analytic.parquet.
4. The runner bundle — data/processed/prospective_bundle/
The LLM runner loads three parquet files from one directory and retrieves comparables and market context from the analytic table it is given. A prospective subject needs the snapshot history for its comparables, so the bundle holds the union:
| file | content |
|---|---|
analytic.parquet |
snapshot analytic (631,866 rows) ∪ prospective rows (1,182) = 633,048 rows, same contract, validated |
splits.parquet |
snapshot splits ∪ prospective rows with split = "prospective" (spatial_holdout_region flag as in splits.py) |
llm_subsamples.parquet |
snapshot subsamples (validation 240 / test 600 / holdout 300) ∪ a stratified draw of the prospective rows: same strata (region_group × property_class × price_tercile), same seed (20261027), largest-remainder allocation via splits.stratified_subsample; n = min(40, available) = 40 rows over 18 strata |
geo_features.parquet |
snapshot geo cache ∪ features.geo.geo_features_batch on the prospective rows (flood mapped 38 %, dissemination area 96 %) |
RunSpec.data_dir / scripts/30_run_arm.py --data-dir <dir> redirects the three loads and the geo cache to that
directory; without it the behaviour is byte-for-byte the previous one (tests/test_experiments_datadir.py). The runner
takes the first n tx_ids (sorted) of the prospective subsample, builds the packet with cutoff = sale date − 1
(comparables and the market index therefore come from snapshot sales and from earlier prospective sales only), and
runs the frozen protocol unchanged; split = "prospective" loads the frozen aggregation weights like test/holdout.
5. Baselines — scripts/62_prospective_baselines.py
Arms A–D (+ CatBoost/XGBoost robustness) are refit with the protocol of scripts/10_train_baselines.py (train on
train, tune on validation, 10 random-search trials, seed 20261027) and predict the 1,182 prospective rows. The feature
matrix is built by features.matrix.build_feature_matrix on the bundle (the as-of neighbourhood and municipality
features of a prospective sale need the history) and restricted to train / validation / prospective. The manifest
records an as-of invariance check: train and validation feature rows rebuilt on the union must equal those of
data/processed/feature_matrix.parquet (only later-dated sales were added). Output:
results/runs/prospective_baselines/predictions.parquet (split = "prospective").
6. Live LLM runs (frozen protocol)
B=data/processed/prospective_bundle
AV_MAX_SPEND_USD=10 python3 scripts/30_run_arm.py --arm E_single_llm --split prospective --n 40 --live --committee mixed_random --run-id prospective_E_single_llm_mixed_random_mid --data-dir $B --concurrency 3
AV_MAX_SPEND_USD=30 python3 scripts/30_run_arm.py --arm F_independent --split prospective --n 40 --live --committee mixed_random --run-id prospective_F_independent_mixed_random_mid --data-dir $B --concurrency 3
AV_MAX_SPEND_USD=90 python3 scripts/30_run_arm.py --arm G_debate --rounds 2 --split prospective --n 40 --live --committee mixed_random --run-id prospective_G_debate_r2_mixed_random_mid --data-dir $B --concurrency 3(logs/prospective_runs.sh; every run is resumable.) Outcomes are reported in reports/phase_06_prospective.md.
7. Evaluation
python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet(the truth table is the bundle's analytic.parquet; runs named prospective_* and prospective_baselines carry
split = "prospective"). With 40 LLM subjects the arm comparison is under-powered; it is a consistency check of the
test/holdout results under a genuine post-freeze sample, not a primary endpoint.
8. How to extend
- Wait for more registrations, then re-run
python3 scripts/60_fetch_prospective.py(new stamp; the manifest keeps the freeze digest). Rows already present in an earlier prospective file are not deduplicated across stamps by the fetch script — pass--date <stamp>to61to build from one stamp, or concatenate stamps before building. python3 scripts/61_build_prospective.py --date <stamp>rebuilds the table and the bundle; the subsample is a new seeded draw over the new population (nested with the previous one only within unchanged strata).- Re-run the three live commands (existing transcripts are skipped, new tx_ids are valued) and
scripts/62_prospective_baselines.py. - Once the prospective sales resell,
scripts/51_eval_vs_resale.pycan use them for the resale-based value proxy (docs/fundamental_value.md).
Limitations: one week of registrations (seasonal and regional composition not representative of a year); the registration lag selects faster-registered deeds; 40 LLM subjects; the municipality lookup drops municipalities with no snapshot sales (none this time); boroughs are imputed from the nearest snapshot sales rather than from a spatial join.