Data dictionary (WP 26)
Generated from results/data/*.json after running scripts/01_build_analytic.py → 05_leakage_audit.py on the Vrai Prix snapshot of 2026-10-21 (WP 22 clean layer, data/raw/wp22_clean_transactions.parquet). Missingness is measured on the real analytic sample. The same contracts hold for the synthetic tables under data/synthetic/ (clearly labelled fake data).
1. Analytic table — data/processed/analytic.parquet
- Rows: 631,866 sales, 541,244 parcels, 2021-01-04 → 2026-10-07; schema =
ANALYTIC_COLUMNS(src/agentic_valuation/data/analytic_schema.py, 33 columns: the 24 synthetic-generator columns + 9 derived). - Selection funnel (raw WP 22 rows → analytic): raw 769,995 → in_headline_sample 632,006 → property_class_in_scope 632,006 → coordinates_present 632,006 → parcel_id_present 632,006 → price_in_range 631,866 → roll_matched 631,866. Dropped: 137,989 outside the WP 22 headline sample (non-residential, land, unmatched roll, duplicates, assessment-ratio outliers, built after sale, shared/multi-parcel, class outside house/condo/plex) and 140 outside the 50 k–5 M price band.
- By class: condo 122,976, house 422,542, plex 86,348. By region group: Montréal CMA 274,294, Outaouais 36,280, Québec CMA 56,163, other 265,129. By year: 2021 131,800, 2022 109,090, 2023 89,138, 2024 102,363, 2025 115,986, 2026 83,489.
- Repeat sales: 90,622 rows (14.3 %) have a strictly earlier sale of the same parcel in the table.
As-of status says when the value is observable relative to the valuation date (sale date; information cutoff = sale date − 1 day). "roll" = assessment-roll attribute, public before the sale; "static" = time-invariant geography; "target"/"post-sale" = never shown to any arm.
| Column | Source column(s) | Definition | As-of status | Missing |
|---|---|---|---|---|
tx_id |
tx_id |
Vrai Prix transaction id (22-char) | key | 0.00 % |
parcel_id |
parcel_id (= roll unit id_provinc) |
Assessment-roll unit identifier; groups repeat sales | key | 0.00 % |
date |
date |
Registration date of the deed (valuation date; information cutoff = date − 1 day) | as-of anchor | 0.00 % |
price |
price |
Transaction price, CAD (filter 50 000–5 000 000). Target; never in a packet | target | 0.00 % |
lat / lng |
lat, lng |
WGS84 coordinates of the roll unit (full precision here; rounded to 3 decimals in packets) | static | 0.00 % |
region_name |
region_name |
Administrative region (MAMH) | static | 0.00 % |
mun_code / mun_name |
mun_code, mun_name |
MAMH municipality code / name | static | 0.00 % |
borough_name |
borough_name |
Borough (Montréal, Québec, Lévis, Longueuil, Sherbrooke, Saguenay); null elsewhere | static | 70.59 % |
urban_class |
mamh_population |
metro ≥ 100 000 inh.; suburban ≥ 10 000; rural otherwise (4 rows with null population → rural) | static (population of the MAMH repertory, one snapshot) | 0.00 % |
property_class |
property_class |
house / condo / plex (WP 22 classification from roll CUBF + provider type) | static | 0.00 % |
subtype |
subtype |
detached / semi / row / integrated / unknown (roll link type; null → unknown) | static | 0.00 % |
floor_area_m2 |
roll_floor_area_m2 |
Living floor area from the roll | roll (public before the sale) | 1.20 % |
land_area_m2 |
roll_land_area_m2 |
Land area from the roll | roll | 0.07 % |
year_built |
roll_year_built |
Construction year from the roll | roll | 0.96 % |
storeys |
roll_storeys |
Number of storeys (roll) | roll | 0.47 % |
n_units |
roll_n_units |
Number of dwellings (roll) | roll | 0.00 % |
roll_land_value |
roll_land_value × (assessment_at_sale / roll_total_value) |
Land value of the roll in force at the sale: the snapshot roll's land share applied to the assessment recorded at the sale (the earlier roll's split is not in the snapshot) | as-of approximation (share from the snapshot roll) | 1.75 % |
roll_building_value |
roll_building_value × (same scale) |
Building value, same rescaling | as-of approximation | 0.00 % |
roll_total_value |
assessment_at_sale (else roll_total_value) |
Total assessment of the roll in force at the sale, as recorded by the provider at the time of the sale (never null here) | as-of (public at sale) | 0.00 % |
roll_reference_date |
roll_reference_date stepped back by whole triennial cycles until reference + 18 months ≤ sale date |
Market reference date of the roll in force at the sale (e.g. 2020-07-01 for a 2022–2024 roll); agrees with the recorded assessment in 98.6 % of rows (roll_as_of_diagnostic in analytic_summary.json) |
as-of inference | 0.00 % |
new_construction |
new_construction |
WP 22 flag: built in the sale year (or the year before) — built_after_sale rows are already excluded |
roll | 0.96 % |
exclude_reason |
exclude_reason |
WP 22 exclusion reason; always null in the analytic sample (only in_headline_sample rows are kept) |
— | 100.00 % |
region_group |
derived (region_name, lat, lng) |
Montréal CMA = region ∈ {Montréal, Laval, Montérégie, Laurentides, Lanaudière} and ≤ 45 km from (45.5017, −73.5673); Québec CMA = Capitale-Nationale and ≤ 35 km from (46.8139, −71.2080); Outaouais = region Outaouais; other | static | 0.00 % |
x_m / y_m |
x_m, y_m (WP 22; synthetic: pyproj) |
EPSG:32198 (NAD83 / Quebec Lambert) metres; all distances are Euclidean in this plane | static | 0.00 % |
grid1k / grid5k |
derived (x_m, y_m) |
floor(x/1000)_floor(y/1000) and the 5 km analogue (same convention as WP 22) |
static | 0.00 % |
prior_sale_price / prior_sale_date |
derived from the table (parcel_id, date, price) |
Most recent sale of the same parcel with date strictly earlier (same-day transfers invisible); null when none in 2021-01-04 → sale | as-of (strictly earlier) | 85.66 % |
price_tercile |
derived (price, property_class) |
1/2/3 by price within class over the whole table (cuts: house 305 000 / 494 500; condo 300 000 / 410 000; plex 350 000 / 625 000). Uses the price → stratification only, never a feature | post-sale (stratification only) | 0.00 % |
log_price |
derived | ln(price) — target | target | 0.00 % |
Contract enforcement: validate_analytic (columns, dtypes, non-null keys, domains, tx_id uniqueness, prior_sale_date < date, log_price = ln(price)); coerce_analytic casts, converts NaN → null, adds missing optional columns and reorders.
2. Splits — data/processed/splits.parquet, llm_subsamples.parquet
| File | Columns | Rule |
|---|---|---|
splits.parquet |
tx_id, split, spatial_holdout_region |
Closed date windows of configs/data.yaml: train 2021-01-01 → 2024-06-30 (379,310), validation 2024-07-01 → 2025-03-31 (72,132), test 2025-04-01 → 2025-12-31 (96,935), holdout 2026-01-01 → 2026-10-07 (83,489); 0 unused. spatial_holdout_region = region ∈ {Outaouais, Abitibi-Témiscamingue, Bas-Saint-Laurent} (59,770 rows). |
llm_subsamples.parquet |
tx_id, split, stratum, draw_seed |
Stratified random subsample per evaluation partition (validation 240, test 600, holdout 300); strata = `region_group |
3. Geo features — data/processed/geo_features.parquet (all rows; static external layers)
Layers copied from the Immo-Ka node (M4M64b:~/apps/immo-ka/data/) into data/raw/external/: inondation.db (24,997 flood-zone polygons in EPSG:3857 + 1,597 mapped-coverage polygons; Quebec government flood maps), transit.db (37,475 stops: bus, metro, REM, train), quartier.db (13,805 Statistics Canada 2021 dissemination areas with census-profile statistics). One snapshot each → time-invariant and identical for every arm; they carry no transaction information. Computed vectorised (shapely STRtree, cKDTree) in 15.2 s for all rows.
| Column | Source | Definition | Missing |
|---|---|---|---|
flood_mapped |
couverture ∪ zi (inondation.db) |
Point lies inside an area covered by a provincial flood map (else exposure is unknown) | 0.00 % |
in_flood_zone |
zi polygons |
Point inside a mapped flood zone; null outside the mapped coverage (60 % of sales — mostly municipalities without a provincial map) | 60.11 % |
flood_zone_kind |
zi.description |
grand_courant_0_20 (0–20 yr), faible_courant_20_100 (20–100 yr), crue_0_100, autre; most severe when overlapping | 99.53 % |
transit_stops_within_1km |
arrets (transit.db) |
Number of public-transit stops within 1 km (Euclidean, EPSG:32198) | 0.00 % |
dauid |
da_poly (quartier.db) |
2021 dissemination area containing the point (point-in-polygon after bbox prefilter) | 2.96 % |
da_population |
da_stats.population |
DA population, census 2021 | 2.96 % |
da_population_density |
da_stats.densite |
Inhabitants per km², census 2021 | 2.96 % |
da_median_age |
da_stats.age_median |
Median age, census 2021 | 2.97 % |
da_median_income |
da_stats.revenu_median |
Median household income (CAD), census 2021 (suppressed in small DAs) | 3.34 % |
da_pct_renters |
da_stats.pct_locataires |
Share of renter households (%), census 2021 | 2.97 % |
cbd_name / distance_to_cbd_km |
derived (lat, lng) |
Nearest of six CBDs (Montréal, Québec, Gatineau, Sherbrooke, Trois-Rivières, Saguenay) and its great-circle distance | 0.00 % |
Shares: 39.9 % of sales are inside a mapped coverage; 1.18 % of those lie in a flood zone; 97.0 % matched a dissemination area. The GeoContext evidence (id G1) exposes in_flood_zone, transit_stops_within_1km, da_population_density, da_median_income, distance_to_cbd_km plus notes (unmapped flood exposure, zone kind, CBD reference).
4. Feature matrix — data/processed/feature_matrix.parquet (baselines A–D)
Rows: 631,866 (one per sale with a split); built in 67.75 s. Identifier/target columns tx_id, split, log_price, price, date, parcel_id are never used as inputs (leakage.check_feature_matrix_columns). Every feature satisfies the as-of guarantee: its value is a deterministic function of (i) the subject's own roll/geography attributes, (ii) sales dated strictly before the sale date, and (iii) static layers. The audit perturbs (×7) and deletes every sale dated ≥ 2025-04-01 and verifies that no feature of an earlier sale changes (feature_matrix_asof: 21,782 rows tested on the real table, 0 changes).
| Feature | Derivation | As-of guarantee | Missing |
|---|---|---|---|
| log_floor_area, log_land_area | ln of roll areas | roll | 1.20 % |
| age_years | sale year − year_built | roll | 0.96 % |
| storeys_f, n_units_f, new_construction_f | roll attributes as floats | roll | 0.47 % |
| log_roll_total, log_roll_land, log_roll_building, roll_land_share, log_roll_per_m2 | ln of the assessment in force at the sale and its split; land share; assessment per m² | roll in force at the sale (public) | 2.02 % |
| roll_age_months | months between the roll's market reference date and the sale | roll | 0.00 % |
| lat, lng, x_m, y_m | coordinates | static | 0.00 % |
| month_index, sale_month | months since 2021-01; calendar month | calendar position only (the model learns time effects from train sales) | 0.00 % |
| prior_sale_log_price, months_since_prior_sale | ln(prior_sale_price); months since prior_sale_date | strictly earlier sale of the same parcel | 85.66 % |
| mun_n_12m, mun_med_log_price_12m, mun_med_ppm2_12m, mun_med_ratio_12m | municipality × class count / median ln price / median price per m² / median price-to-assessment ratio over the 365 days before the sale date (polars rolling_*_by, closed="left": same-day sales excluded) |
strictly earlier sales, exact | 0.58 % |
| nb_n_12m, nb_med_log_price, nb_med_ppm2, nb_med_ratio, nb_mean_dist_km | all same-class sales of other parcels within 3 km (EPSG:32198) dated in [sale − 365 d, sale − 1 d]: count, medians, mean distance (monthly KD-trees of candidate sales, vectorised grouped medians; verified against brute force in tests/test_features.py) |
strictly earlier sales, exact; nb_n_12m quantiles 5/25/50/75/95 % = 3/38/151/332/698 | 0.00 % |
| in_flood_zone, transit_stops_within_1km, da_population_density, da_median_income, da_pct_renters, distance_to_cbd_km | from geo_features.parquet (in_flood_zone cast to 0/1, null when unmapped) |
static layers | 60.11 % |
| region_name, region_group, mun_code, property_class, subtype, grid1k, grid5k, urban_class | categorical (strings) — encoders fitted on train only in the baselines | static | 0.00 % |
5. Evidence given to the LLM arms (as-of rules)
- Comparables (
features/comparables.py, idsC01..C12): sales withdate < cutoffand within 540 days, same class, other parcels; expanding radius 1 → 3 → 10 km (first radius with ≥ 3 candidates), extended to 25 km with a note when fewer than 3; ranked by a similarity score (distance / radius + 1.5·|Δ ln area| + 0.5·|Δ year|/50 + 0.5·recency), max 12. On the 1,140 subsample subjects: mean 11.2 comparables, minimum 3, 1 extension to 25 km. - Market context (
features/market.py, idM1): municipality × class 12-month median price / price per m² / count; monthly municipality index (median price per m², base 100 at the first month with ≥ 5 sales, region series when the municipality has < 20 sales in 12 months; the cutoff month is flaggedpartial); region 12-month median and year-over-year change; liquidity (all classes) and median assessment ratio within 3 km over 12 months — all from sales withdate ≤ cutoff.mortgage_rate_5y_fixedis null (no external series loaded). - Geo context (
features/geo.py, idG1): from the cache above (identical for all arms). - The Vrai Prix
market_indextable and the platform's AVM estimates are not used anywhere (computed with post-cutoff data).
6. Reproduction
bash
python3 scripts/01_build_analytic.py # 0.5 s → data/processed/analytic.parquet, results/data/analytic_summary.json
python3 scripts/02_splits.py # 0.3 s → splits.parquet, llm_subsamples.parquet, results/data/splits_summary.json
python3 scripts/04_geo.py # 15 s → geo_features.parquet, results/data/geo_summary.json
python3 scripts/03_features.py # 68 s → feature_matrix.parquet, results/data/feature_matrix_summary.json
python3 scripts/05_leakage_audit.py # 38 s → results/data/leakage_audit.json (exit 1 on any failure)
# synthetic counterparts: add --synthetic to each (outputs under data/synthetic/, *_synthetic.json)
python3 -m pytest tests/test_data.py tests/test_features.py tests/test_leakage.py [--real]