# Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
# Prospective evaluation set (WP 26)

## 1. What it is

The preregistration (CLAUDE.md §Experimental design; docs/research_plan.md) calls for *a separate prospective
evaluation set gathered after model/prompt freeze*. The protocol was frozen on 2026-10-28 01:05 UTC
(`results/freeze.json`, digest `f0e9d6a9…`). The prospective set is the set of Vrai Prix sales that are present in the
**live** database on node M3U96a but **absent from the 2026-10-21 snapshot** on which every partition
(train / validation / test / holdout), every prompt iteration and every aggregation weight was built. These sales
were registered at the Registre foncier after the snapshot: no model, prompt, hyper-parameter or weight could have
seen them. They are the cleanest out-of-sample test the design allows, at the cost of a small, time-concentrated
sample (one week of registrations).

Two properties distinguish it from `holdout`: (i) holdout rows were in the snapshot during development (never read
by the LLM arms, but visible to the data pipeline, e.g. the tercile cut-points and the municipality lookup);
(ii) the prospective rows arrived after the freeze, so their very existence post-dates every design decision.

## 2. When and how it was fetched

```
python3 scripts/60_fetch_prospective.py          # 2026-10-27, read-only SSH to M3U96a:~/apps/vrai-prix/data/vraiprix.db
```
Manifest `data/raw/prospective_2026-10-27.manifest.json`: 36,826 live rows dated ≥ 2026-07-01, of which **1,444 new
rows** (ids absent from the snapshot), sale dates **2026-10-06 … 2026-10-13**; **1,434 roll units** fetched for their
parcels (every new row matched a unit; some parcels sold twice). The raw files `data/raw/prospective_2026-10-27.parquet`
and `prospective_units_2026-10-27.parquet` carry the platform's columns, including addresses; they are gitignored and
never printed, and no address reaches any downstream artefact.

Registration lag is 2–8 weeks, so the live DB on 2026-10-27 only contained sales up to 2026-10-13; the sales of
2026-10-06/07 are the ones registered after the snapshot although dated inside the holdout window — they are
*prospective* by construction (absent from the snapshot), not holdout rows.

## 3. From raw rows to the analytic contract — `scripts/61_build_prospective.py`

The script re-implements, minimally and on these rows only, the WP 22 cleaning (`sharp-hpi-quebec/src/sharp_hpi/
cleaning.py`) and then the WP 26 funnel of `data/build.py`. The frozen code (prompts, configs, roles, protocols,
aggregation) is not touched; `analytic_schema.py` helpers (`project_xy`, `grid_id_expr`, `region_group_expr`,
`coerce_analytic`, `validate_analytic`) and `build.roll_reference_at_sale_expr` are imported, not copied.

| Contract column | Source / rule |
|---|---|
| `tx_id`, `parcel_id`, `date`, `price`, `lat`, `lng` | transactions `id`, `id_provinc`, `date`, `amount`, `lat`, `lng` |
| `property_class` | WP 22 precedence from the roll: CUBF 1000 × `nb_logements` × `lien_physique` (1 dwelling + *Intégré* → condo; 1 dwelling + Détaché/Jumelé/En rangée → house; 2–5 dwellings → plex; ≥ 6 → multi; 1100 cottage; 1211 mobile); provider label (`property_type` ∈ unifamilial/condo/plex) only when the roll is incomplete. 1,175 of 1,182 kept rows classified from the roll |
| `subtype` | house: Détaché → detached, Jumelé → semi, En rangée → row; condo → integrated; else unknown (plex, provider-classified) |
| `floor_area_m2`, `land_area_m2`, `year_built`, `storeys`, `n_units` | units `aire_etages_m2`, `superficie_terrain_m2`, `annee_construction`, `nb_etages`, `nb_logements`; provider `floor_area` / `land_area` / `year_built` as fallback when the roll field is null (0.2 % of rows for floor area) |
| `roll_total_value` | assessment in force at the sale = transactions `valeur_role` (provider), falling back to units `valeur_role` — same as `build.py` |
| `roll_land_value`, `roll_building_value` | units `valeur_terrain` / `valeur_batiment` rescaled by assessment-at-sale ÷ current roll total (same as `build.py`) |
| `roll_reference_date` | units `dat_cond_marche` stepped back by whole triennial cycles until reference + 18 months ≤ sale date (`build.roll_reference_at_sale_expr`) |
| `mun_code`, `mun_name`, `region_name`, `urban_class` | lookup from `data/processed/analytic.parquet` keyed by the roll's `code_mun` (the mapping is unique per code in the snapshot; rows whose municipality is absent from the lookup are dropped and counted — 0 on 2026-10-27). The snapshot derived geography by a spatial join; the roll code agrees in > 99 % of snapshot rows |
| `borough_name` | mode of the snapshot `borough_name` in the same municipality × 1 km cell (null elsewhere; 24 % filled, as in the snapshot's 29 %) |
| `x_m`, `y_m`, `grid1k`, `grid5k`, `region_group`, `log_price` | `analytic_schema` helpers |
| `prior_sale_price/date` | strictly earlier sale of the same `parcel_id` in the **full** snapshot (`wp22_clean_transactions.parquet`, all 769,995 rows) ∪ the prospective rows |
| `price_tercile` | the snapshot's class-specific cut points (linear 1/3 and 2/3 quantiles of `analytic.parquet`: condo 300k/410k, house 305k/494.5k, plex 350k/625k) |
| `new_construction` | `year_built ≥ sale year` (0 rows; rows built after the sale are excluded) |
| `exclude_reason` | null for every kept row |

Exclusion flags (WP 22 precedence, computed on the prospective rows ∪ the snapshot rows of the same dates so that a
multi-parcel deed or a same-day price conflict straddling the snapshot date is caught): exact duplicate inside the set;
**duplicate of a snapshot sale** (same parcel, date and price — 0 rows); same parcel, same date, different price;
built after sale; land (`type_prop = terrain`, CUBF 4xxx/9xxx); non-residential CUBF; shared parcel (> 10 sales
2021–2026 in snapshot ∪ prospective); multi-parcel candidate (same date + price + street name, or same date + price +
0.01° cell); price > 10 M; assessment-ratio outlier (|log(price/assessment) − median| > log 4, median of the snapshot's
2026 sales by class × region, class-level fallback).

### Funnel (2026-10-27)

| step | rows | dropped |
|---|---|---|
| raw new rows | 1,444 | — |
| roll matched | 1,444 | 0 |
| no WP 22 exclusion flag | 1,251 | 193 (land 96, non-residential 56, assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2) |
| class ∈ {house, condo, plex} | 1,182 | 69 (cottage, mobile, multi, unclassifiable) |
| coordinates / parcel id / price in [50k, 5M] / municipality in lookup | 1,182 | 0 |

Result: **1,182 sales** (house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99, Outaouais 57, other 567;
dates 2026-10-06 … 2026-10-13; 321 with a prior sale in the snapshot). `validate_analytic` passes.
Summary: `results/data/prospective_summary.json`. Table: `data/processed/prospective_analytic.parquet`.

## 4. The runner bundle — `data/processed/prospective_bundle/`

The LLM runner loads three parquet files from one directory and retrieves comparables and market context from the
analytic table it is given. A prospective subject needs the **snapshot history** for its comparables, so the bundle
holds the union:

| file | content |
|---|---|
| `analytic.parquet` | snapshot analytic (631,866 rows) ∪ prospective rows (1,182) = 633,048 rows, same contract, validated |
| `splits.parquet` | snapshot splits ∪ prospective rows with `split = "prospective"` (`spatial_holdout_region` flag as in `splits.py`) |
| `llm_subsamples.parquet` | snapshot subsamples (validation 240 / test 600 / holdout 300) ∪ a stratified draw of the prospective rows: same strata (region_group × property_class × price_tercile), same seed (20261027), largest-remainder allocation via `splits.stratified_subsample`; n = min(40, available) = **40 rows over 18 strata** |
| `geo_features.parquet` | snapshot geo cache ∪ `features.geo.geo_features_batch` on the prospective rows (flood mapped 38 %, dissemination area 96 %) |

`RunSpec.data_dir` / `scripts/30_run_arm.py --data-dir <dir>` redirects the three loads and the geo cache to that
directory; without it the behaviour is byte-for-byte the previous one (`tests/test_experiments_datadir.py`). The runner
takes the first *n* tx_ids (sorted) of the `prospective` subsample, builds the packet with cutoff = sale date − 1
(comparables and the market index therefore come from snapshot sales and from earlier prospective sales only), and
runs the frozen protocol unchanged; `split = "prospective"` loads the frozen aggregation weights like test/holdout.

## 5. Baselines — `scripts/62_prospective_baselines.py`

Arms A–D (+ CatBoost/XGBoost robustness) are refit with the protocol of `scripts/10_train_baselines.py` (train on
`train`, tune on `validation`, 10 random-search trials, seed 20261027) and predict the 1,182 prospective rows. The feature
matrix is built by `features.matrix.build_feature_matrix` on the bundle (the as-of neighbourhood and municipality
features of a prospective sale need the history) and restricted to train / validation / prospective. The manifest
records an **as-of invariance check**: train and validation feature rows rebuilt on the union must equal those of
`data/processed/feature_matrix.parquet` (only later-dated sales were added). Output:
`results/runs/prospective_baselines/predictions.parquet` (`split = "prospective"`).

## 6. Live LLM runs (frozen protocol)

```
B=data/processed/prospective_bundle
AV_MAX_SPEND_USD=10 python3 scripts/30_run_arm.py --arm E_single_llm  --split prospective --n 40 --live --committee mixed_random --run-id prospective_E_single_llm_mixed_random_mid  --data-dir $B --concurrency 3
AV_MAX_SPEND_USD=30 python3 scripts/30_run_arm.py --arm F_independent --split prospective --n 40 --live --committee mixed_random --run-id prospective_F_independent_mixed_random_mid --data-dir $B --concurrency 3
AV_MAX_SPEND_USD=90 python3 scripts/30_run_arm.py --arm G_debate --rounds 2 --split prospective --n 40 --live --committee mixed_random --run-id prospective_G_debate_r2_mixed_random_mid --data-dir $B --concurrency 3
```
(`logs/prospective_runs.sh`; every run is resumable.) Outcomes are reported in `reports/phase_06_prospective.md`.

## 7. Evaluation

```
python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet
```
(the truth table is the bundle's `analytic.parquet`; runs named `prospective_*` and `prospective_baselines` carry
`split = "prospective"`). With 40 LLM subjects the arm comparison is under-powered; it is a consistency check of the
test/holdout results under a genuine post-freeze sample, not a primary endpoint.

## 8. How to extend

1. Wait for more registrations, then re-run `python3 scripts/60_fetch_prospective.py` (new stamp; the manifest keeps the
   freeze digest). Rows already present in an earlier prospective file are *not* deduplicated across stamps by the
   fetch script — pass `--date <stamp>` to `61` to build from one stamp, or concatenate stamps before building.
2. `python3 scripts/61_build_prospective.py --date <stamp>` rebuilds the table and the bundle; the subsample is a new
   seeded draw over the new population (nested with the previous one only within unchanged strata).
3. Re-run the three live commands (existing transcripts are skipped, new tx_ids are valued) and
   `scripts/62_prospective_baselines.py`.
4. Once the prospective sales resell, `scripts/51_eval_vs_resale.py` can use them for the resale-based value proxy
   (docs/fundamental_value.md).

Limitations: one week of registrations (seasonal and regional composition not representative of a year); the
registration lag selects faster-registered deeds; 40 LLM subjects; the municipality lookup drops municipalities with
no snapshot sales (none this time); boroughs are imputed from the nearest snapshot sales rather than from a spatial
join.