1<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->2# Data dictionary (WP 26)34Generated from `results/data/*.json` after running `scripts/01_build_analytic.py` → `05_leakage_audit.py` on the Vrai Prix snapshot of 2026-10-21 (WP 22 clean layer, `data/raw/wp22_clean_transactions.parquet`). Missingness is measured on the real analytic sample. The same contracts hold for the synthetic tables under `data/synthetic/` (clearly labelled fake data).56## 1. Analytic table — `data/processed/analytic.parquet`78* Rows: **631,866** sales, 541,244 parcels, 2021-01-04 → 2026-10-07; schema = `ANALYTIC_COLUMNS` (`src/agentic_valuation/data/analytic_schema.py`, 33 columns: the 24 synthetic-generator columns + 9 derived).9* Selection funnel (raw WP 22 rows → analytic): raw 769,995 → in_headline_sample 632,006 → property_class_in_scope 632,006 → coordinates_present 632,006 → parcel_id_present 632,006 → price_in_range 631,866 → roll_matched 631,866. Dropped: 137,989 outside the WP 22 headline sample (non-residential, land, unmatched roll, duplicates, assessment-ratio outliers, built after sale, shared/multi-parcel, class outside house/condo/plex) and 140 outside the 50 k–5 M price band.10* By class: condo 122,976, house 422,542, plex 86,348. By region group: Montréal CMA 274,294, Outaouais 36,280, Québec CMA 56,163, other 265,129. By year: 2021 131,800, 2022 109,090, 2023 89,138, 2024 102,363, 2025 115,986, 2026 83,489.11* Repeat sales: 90,622 rows (14.3 %) have a strictly earlier sale of the same parcel in the table.1213**As-of status** says *when* the value is observable relative to the valuation date (sale date; information cutoff = sale date − 1 day). "roll" = assessment-roll attribute, public before the sale; "static" = time-invariant geography; "target"/"post-sale" = never shown to any arm.1415| Column | Source column(s) | Definition | As-of status | Missing |16|---|---|---|---|---|17| `tx_id` | `tx_id` | Vrai Prix transaction id (22-char) | key | 0.00 % |18| `parcel_id` | `parcel_id` (= roll unit `id_provinc`) | Assessment-roll unit identifier; groups repeat sales | key | 0.00 % |19| `date` | `date` | Registration date of the deed (valuation date; information cutoff = date − 1 day) | as-of anchor | 0.00 % |20| `price` | `price` | Transaction price, CAD (filter 50 000–5 000 000). **Target; never in a packet** | target | 0.00 % |21| `lat / lng` | `lat`, `lng` | WGS84 coordinates of the roll unit (full precision here; rounded to 3 decimals in packets) | static | 0.00 % |22| `region_name` | `region_name` | Administrative region (MAMH) | static | 0.00 % |23| `mun_code / mun_name` | `mun_code`, `mun_name` | MAMH municipality code / name | static | 0.00 % |24| `borough_name` | `borough_name` | Borough (Montréal, Québec, Lévis, Longueuil, Sherbrooke, Saguenay); null elsewhere | static | 70.59 % |25| `urban_class` | `mamh_population` | metro ≥ 100 000 inh.; suburban ≥ 10 000; rural otherwise (4 rows with null population → rural) | static (population of the MAMH repertory, one snapshot) | 0.00 % |26| `property_class` | `property_class` | house / condo / plex (WP 22 classification from roll CUBF + provider type) | static | 0.00 % |27| `subtype` | `subtype` | detached / semi / row / integrated / unknown (roll link type; null → unknown) | static | 0.00 % |28| `floor_area_m2` | `roll_floor_area_m2` | Living floor area from the roll | roll (public before the sale) | 1.20 % |29| `land_area_m2` | `roll_land_area_m2` | Land area from the roll | roll | 0.07 % |30| `year_built` | `roll_year_built` | Construction year from the roll | roll | 0.96 % |31| `storeys` | `roll_storeys` | Number of storeys (roll) | roll | 0.47 % |32| `n_units` | `roll_n_units` | Number of dwellings (roll) | roll | 0.00 % |33| `roll_land_value` | `roll_land_value` × (`assessment_at_sale` / `roll_total_value`) | Land value of the roll **in force at the sale**: the snapshot roll's land share applied to the assessment recorded at the sale (the earlier roll's split is not in the snapshot) | as-of approximation (share from the snapshot roll) | 1.75 % |34| `roll_building_value` | `roll_building_value` × (same scale) | Building value, same rescaling | as-of approximation | 0.00 % |35| `roll_total_value` | `assessment_at_sale` (else `roll_total_value`) | Total assessment of the roll in force at the sale, as recorded by the provider at the time of the sale (never null here) | as-of (public at sale) | 0.00 % |36| `roll_reference_date` | `roll_reference_date` stepped back by whole triennial cycles until reference + 18 months ≤ sale date | Market reference date of the roll in force at the sale (e.g. 2020-07-01 for a 2022–2024 roll); agrees with the recorded assessment in 98.6 % of rows (`roll_as_of_diagnostic` in `analytic_summary.json`) | as-of inference | 0.00 % |37| `new_construction` | `new_construction` | WP 22 flag: built in the sale year (or the year before) — `built_after_sale` rows are already excluded | roll | 0.96 % |38| `exclude_reason` | `exclude_reason` | WP 22 exclusion reason; always null in the analytic sample (only `in_headline_sample` rows are kept) | — | 100.00 % |39| `region_group` | derived (`region_name`, `lat`, `lng`) | Montréal CMA = region ∈ {Montréal, Laval, Montérégie, Laurentides, Lanaudière} and ≤ 45 km from (45.5017, −73.5673); Québec CMA = Capitale-Nationale and ≤ 35 km from (46.8139, −71.2080); Outaouais = region Outaouais; other | static | 0.00 % |40| `x_m / y_m` | `x_m`, `y_m` (WP 22; synthetic: pyproj) | EPSG:32198 (NAD83 / Quebec Lambert) metres; all distances are Euclidean in this plane | static | 0.00 % |41| `grid1k / grid5k` | derived (`x_m`, `y_m`) | `floor(x/1000)_floor(y/1000)` and the 5 km analogue (same convention as WP 22) | static | 0.00 % |42| `prior_sale_price / prior_sale_date` | derived from the table (`parcel_id`, `date`, `price`) | Most recent sale of the same parcel with date **strictly** earlier (same-day transfers invisible); null when none in 2021-01-04 → sale | as-of (strictly earlier) | 85.66 % |43| `price_tercile` | derived (`price`, `property_class`) | 1/2/3 by price within class over the whole table (cuts: house 305 000 / 494 500; condo 300 000 / 410 000; plex 350 000 / 625 000). **Uses the price → stratification only, never a feature** | post-sale (stratification only) | 0.00 % |44| `log_price` | derived | ln(price) — target | target | 0.00 % |4546Contract enforcement: `validate_analytic` (columns, dtypes, non-null keys, domains, tx_id uniqueness, `prior_sale_date < date`, `log_price = ln(price)`); `coerce_analytic` casts, converts NaN → null, adds missing optional columns and reorders.4748## 2. Splits — `data/processed/splits.parquet`, `llm_subsamples.parquet`4950| File | Columns | Rule |51|---|---|---|52| `splits.parquet` | `tx_id`, `split`, `spatial_holdout_region` | Closed date windows of `configs/data.yaml`: train 2021-01-01 → 2024-06-30 (379,310), validation 2024-07-01 → 2025-03-31 (72,132), test 2025-04-01 → 2025-12-31 (96,935), holdout 2026-01-01 → 2026-10-07 (83,489); 0 unused. `spatial_holdout_region` = region ∈ {Outaouais, Abitibi-Témiscamingue, Bas-Saint-Laurent} (59,770 rows). |53| `llm_subsamples.parquet` | `tx_id`, `split`, `stratum`, `draw_seed` | Stratified random subsample per evaluation partition (validation 240, test 600, holdout 300); strata = `region_group|property_class|price_tercile` (35 non-empty strata), largest-remainder proportional allocation capped at stratum size, draws with `numpy.default_rng(sha256(seed|split|stratum))`, seed 20261027; `draw_seed` = that derived integer. Re-drawing reproduces the file exactly (checked by the audit). |5455## 3. Geo features — `data/processed/geo_features.parquet` (all rows; static external layers)5657Layers copied from the Immo-Ka node (`M4M64b:~/apps/immo-ka/data/`) into `data/raw/external/`: `inondation.db` (24,997 flood-zone polygons in EPSG:3857 + 1,597 mapped-coverage polygons; Quebec government flood maps), `transit.db` (37,475 stops: bus, metro, REM, train), `quartier.db` (13,805 Statistics Canada 2021 dissemination areas with census-profile statistics). One snapshot each → **time-invariant** and identical for every arm; they carry no transaction information. Computed vectorised (shapely STRtree, cKDTree) in 15.2 s for all rows.5859| Column | Source | Definition | Missing |60|---|---|---|---|61| `flood_mapped` | `couverture` ∪ `zi` (inondation.db) | Point lies inside an area covered by a provincial flood map (else exposure is unknown) | 0.00 % |62| `in_flood_zone` | `zi` polygons | Point inside a mapped flood zone; **null outside the mapped coverage** (60 % of sales — mostly municipalities without a provincial map) | 60.11 % |63| `flood_zone_kind` | `zi.description` | grand_courant_0_20 (0–20 yr), faible_courant_20_100 (20–100 yr), crue_0_100, autre; most severe when overlapping | 99.53 % |64| `transit_stops_within_1km` | `arrets` (transit.db) | Number of public-transit stops within 1 km (Euclidean, EPSG:32198) | 0.00 % |65| `dauid` | `da_poly` (quartier.db) | 2021 dissemination area containing the point (point-in-polygon after bbox prefilter) | 2.96 % |66| `da_population` | `da_stats.population` | DA population, census 2021 | 2.96 % |67| `da_population_density` | `da_stats.densite` | Inhabitants per km², census 2021 | 2.96 % |68| `da_median_age` | `da_stats.age_median` | Median age, census 2021 | 2.97 % |69| `da_median_income` | `da_stats.revenu_median` | Median household income (CAD), census 2021 (suppressed in small DAs) | 3.34 % |70| `da_pct_renters` | `da_stats.pct_locataires` | Share of renter households (%), census 2021 | 2.97 % |71| `cbd_name / distance_to_cbd_km` | derived (`lat`, `lng`) | Nearest of six CBDs (Montréal, Québec, Gatineau, Sherbrooke, Trois-Rivières, Saguenay) and its great-circle distance | 0.00 % |7273Shares: 39.9 % of sales are inside a mapped coverage; 1.18 % of those lie in a flood zone; 97.0 % matched a dissemination area. The `GeoContext` evidence (id `G1`) exposes `in_flood_zone`, `transit_stops_within_1km`, `da_population_density`, `da_median_income`, `distance_to_cbd_km` plus notes (unmapped flood exposure, zone kind, CBD reference).7475## 4. Feature matrix — `data/processed/feature_matrix.parquet` (baselines A–D)7677Rows: 631,866 (one per sale with a split); built in 67.75 s. Identifier/target columns `tx_id, split, log_price, price, date, parcel_id` are never used as inputs (`leakage.check_feature_matrix_columns`). Every feature satisfies the **as-of guarantee**: its value is a deterministic function of (i) the subject's own roll/geography attributes, (ii) sales dated **strictly before** the sale date, and (iii) static layers. The audit perturbs (×7) and deletes every sale dated ≥ 2025-04-01 and verifies that no feature of an earlier sale changes (`feature_matrix_asof`: 21,782 rows tested on the real table, 0 changes).7879| Feature | Derivation | As-of guarantee | Missing |80|---|---|---|---|81| log_floor_area, log_land_area | ln of roll areas | roll | 1.20 % |82| age_years | sale year − year_built | roll | 0.96 % |83| storeys_f, n_units_f, new_construction_f | roll attributes as floats | roll | 0.47 % |84| log_roll_total, log_roll_land, log_roll_building, roll_land_share, log_roll_per_m2 | ln of the assessment in force at the sale and its split; land share; assessment per m² | roll in force at the sale (public) | 2.02 % |85| roll_age_months | months between the roll's market reference date and the sale | roll | 0.00 % |86| lat, lng, x_m, y_m | coordinates | static | 0.00 % |87| month_index, sale_month | months since 2021-01; calendar month | calendar position only (the model learns time effects from train sales) | 0.00 % |88| prior_sale_log_price, months_since_prior_sale | ln(prior_sale_price); months since prior_sale_date | strictly earlier sale of the same parcel | 85.66 % |89| mun_n_12m, mun_med_log_price_12m, mun_med_ppm2_12m, mun_med_ratio_12m | municipality × class count / median ln price / median price per m² / median price-to-assessment ratio over the 365 days **before** the sale date (polars `rolling_*_by`, `closed="left"`: same-day sales excluded) | strictly earlier sales, exact | 0.58 % |90| nb_n_12m, nb_med_log_price, nb_med_ppm2, nb_med_ratio, nb_mean_dist_km | all same-class sales of **other parcels** within 3 km (EPSG:32198) dated in [sale − 365 d, sale − 1 d]: count, medians, mean distance (monthly KD-trees of candidate sales, vectorised grouped medians; verified against brute force in `tests/test_features.py`) | strictly earlier sales, exact; nb_n_12m quantiles 5/25/50/75/95 % = 3/38/151/332/698 | 0.00 % |91| in_flood_zone, transit_stops_within_1km, da_population_density, da_median_income, da_pct_renters, distance_to_cbd_km | from `geo_features.parquet` (`in_flood_zone` cast to 0/1, null when unmapped) | static layers | 60.11 % |92| region_name, region_group, mun_code, property_class, subtype, grid1k, grid5k, urban_class | categorical (strings) — encoders fitted on train only in the baselines | static | 0.00 % |9394## 5. Evidence given to the LLM arms (as-of rules)9596* **Comparables** (`features/comparables.py`, ids `C01..C12`): sales with `date < cutoff` and within 540 days, same class, other parcels; expanding radius 1 → 3 → 10 km (first radius with ≥ 3 candidates), extended to 25 km with a note when fewer than 3; ranked by a similarity score (distance / radius + 1.5·|Δ ln area| + 0.5·|Δ year|/50 + 0.5·recency), max 12. On the 1,140 subsample subjects: mean 11.2 comparables, minimum 3, 1 extension to 25 km.97* **Market context** (`features/market.py`, id `M1`): municipality × class 12-month median price / price per m² / count; monthly municipality index (median price per m², base 100 at the first month with ≥ 5 sales, region series when the municipality has < 20 sales in 12 months; the cutoff month is flagged `partial`); region 12-month median and year-over-year change; liquidity (all classes) and median assessment ratio within 3 km over 12 months — all from sales with `date ≤ cutoff`. `mortgage_rate_5y_fixed` is null (no external series loaded).98* **Geo context** (`features/geo.py`, id `G1`): from the cache above (identical for all arms).99* The Vrai Prix `market_index` table and the platform's AVM estimates are **not** used anywhere (computed with post-cutoff data).100101## 6. Reproduction102103```bash104python3 scripts/01_build_analytic.py # 0.5 s → data/processed/analytic.parquet, results/data/analytic_summary.json105python3 scripts/02_splits.py # 0.3 s → splits.parquet, llm_subsamples.parquet, results/data/splits_summary.json106python3 scripts/04_geo.py # 15 s → geo_features.parquet, results/data/geo_summary.json107python3 scripts/03_features.py # 68 s → feature_matrix.parquet, results/data/feature_matrix_summary.json108python3 scripts/05_leakage_audit.py # 38 s → results/data/leakage_audit.json (exit 1 on any failure)109# synthetic counterparts: add --synthetic to each (outputs under data/synthetic/, *_synthetic.json)110python3 -m pytest tests/test_data.py tests/test_features.py tests/test_leakage.py [--real]111```112