SPB Git forge
spboucher.ai ↗

spb/wp22_uqo

Public
3commits 1branches 0releases
133.8 MBsize
maindefault branch
1 h agolast push
Python 58.4% TeX 40.8% SQL 0.6%

WP22: validation section, README, figure fixes, completeness tables

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 2 h ago (Oct 21, 2026) parent 0eb5fe4

38 changed files +234 −17

added README.md +115 −0
@@ -0,0 +1,115 @@
1 +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
2 +# 🏘️ SHARP-HPI: A Spatially Hierarchical, Adaptive and Robust Framework for Quality-Adjusted Housing Price Indices
3 +
4 +**UQO Working Paper No. 22** — *Monthly municipal indices for Quebec from 770,000 registered sales, 2021–2026*
5 +
6 +[![Paper](https://img.shields.io/badge/paper-PDF-b31b1b?logo=latex&logoColor=white)](paper/main.pdf)
7 +[![Python](https://img.shields.io/badge/python-3.14-3776AB?logo=python&logoColor=white)](requirements.txt)
8 +[![Data](https://img.shields.io/badge/data-769%2C995%20registered%20sales%20·%20SQLite%20snapshot-4051b5)](data/raw/README.md)
9 +[![Reproducible](https://img.shields.io/badge/reproducible-end--to--end%20pipeline-2e7d32)](scripts/)
10 +[![Institution](https://img.shields.io/badge/UQO-D%C3%A9pt.%20des%20sciences%20administratives-16365c)](https://uqo.ca)
11 +[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-16365c)](mailto:simon-pierre.boucher@uqo.ca)
12 +
13 +> **TL;DR** — A two-stage index estimator for thin housing markets: a robust pooled hedonic regression with fine-scale location fixed effects and drifting quality prices (stage 1), then nested local-level state-space deviations province → region → MRC → municipality whose innovation variances are calibrated by empirical Bayes across cells (stage 2). Applied to an audited snapshot of Quebec land-registry sales (769,995 rows → 632,006 headline sales, January 2021 – 7 October 2026) matched to the assessment roll. Delivers a panel of 2,696 geography × class cells × 69 months with counts, effective sample sizes, local-data weights, 95 % intervals and A–E grades; eleven benchmarks (median → HMTS → repeat sales → QHPI); a 9-scenario Monte Carlo with two deliberate counter-examples; real-time vintages with measured revisions; placebo, composition-shock, bootstrap, Moran, ablation, aggregation and external (Teranet, CREA) checks.
14 +
15 +---
16 +
17 +## 📌 Status
18 +
19 +| Item | Value |
20 +|---|---|
21 +| Version | 1.0 (October 2026) |
22 +| SSRN | not yet submitted |
23 +| Journal | — |
24 +| Code & data | reproducible from `scripts/` (see Quick start); raw microdata not redistributed |
25 +
26 +## 🎯 What this paper does
27 +
28 +The estimand is the quality-constant log price level $L_{g,k,t}$ of geography $g$, class $k$ (house, condominium, plex) and month $t$, normalised to the 2021 average = 100. The model is
29 +
30 +```
31 +log P_i = f_k(x_i; θ_t) + α_{ℓ(i)} + μ_{k,t} + δ^R_{r(i),t} + δ^M_{m(i),t} + δ^U_{u(i),t} + ε_i
32 +δ_{g,t} = δ_{g,t-1} + η_{g,t}, ȳ_{g,t} = δ_{g,t} + e_{g,t}, Var(e) = σ²_g / n_{g,t}, log τ²_g ~ N(m_level, v_level)
33 +```
34 +
35 +Every ingredient exists in the literature (Francke & van de Minne 2017; Ishaak, Ouwehand & Remøy 2024; Fay & Herriot 1979; Hill et al. 2022); the paper tests whether their combination helps, where it does not, and how reliable each cell is.
36 +
37 +## 🏆 Headline results
38 +
39 +See `paper/tables/numbers.tex` (every number in the manuscript is a generated macro). Key outputs: `data/processed/index_panel.parquet|csv` (schema of protocol §17), `results/panel/reliability_cells.csv`, `results/simulation/summary.csv`, `results/vintages/revisions_by_release.csv`, `results/validation/*.csv`.
40 +
41 +## 🗂 Repository layout
42 +
43 +```
44 +sharp-hpi-quebec/
45 +├── CLAUDE.md # research protocol (as received)
46 +├── data/raw/ # SQLite snapshot + earlier vintages (not committed; README)
47 +├── data/external/ # SDA boundaries (not committed), MAMH register, external indices
48 +├── data/interim|processed/ # parquet layers; index_panel.* and index_vintages.parquet
49 +├── src/sharp_hpi/ # config, io, cleaning, hedonic, estimators, statespace, sharp, qhpi, panel, simulate, plotting
50 +├── scripts/ # 01–06 data · 10–11 benchmarks · 20 Monte Carlo · 30 panel · 40–41 vintages · 50 validation · 60–62 figures/tables/external
51 +├── tests/ # pytest (state space, EB prior, estimators, SHARP, panel invariance)
52 +├── results/ # profile, geo, cleaning, vintages, benchmarks, simulation, panel, validation
53 +├── figures/ # generated figures (copied to paper/figures)
54 +├── docs/ # PLAN, decision_log, data_dictionary, provenance_report, limitations, research_log, literature/
55 +└── paper/ # main.tex, sections/, appendix/, tables/ (generated), references.bib (115 verified entries)
56 +```
57 +
58 +## 🚀 Quick start
59 +
60 +```bash
61 +python3 -m pip install -r requirements.txt
62 +make data # 01 extract (needs data/raw/vraiprix-20261021.db) → 06 provenance report
63 +make benchmarks hmts # A–K at province × class and region × class
64 +make panel # full SHARP-HPI panel (≈ 1 min)
65 +make vintages revisions # real-time vintages (≈ 1 h) and revision statistics
66 +make validation external # placebo, composition, bootstrap, Moran, ablation, aggregation, trimming, repeat sales, Teranet/CREA
67 +make simulation # Monte Carlo, 80 reps × 9 scenarios (≈ 3.6 h on 12 cores)
68 +make figures tables paper # figures, LaTeX tables + macros, PDF
69 +make test
70 +```
71 +
72 +## 🗃 Data
73 +
74 +Production SQLite of Vrai Prix (www.vrai-prix.com), `.backup` taken 2026-10-21 00:30; 769,995 registered sales (Quebec land registry via the Journal de Québec / Journal de Montréal feed) 2021-01-04 → 2026-10-07, geocoded and matched to the MAMH assessment roll (3.75 M units). Four earlier extractions measure the publication lag (10–13 days, no backfill). Boundaries: MRNF SDA V2026-07. External indices: Teranet–National Bank, CREA MLS HPI, Statistics Canada NHPI/RPPI (downloaded 2026-10-21). See `docs/data_dictionary.md` and `docs/provenance_report.md`.
75 +
76 +## 🔬 Methodology
77 +
78 +1. `cleaning.py` — standardisation, roll-based classification, exclusion ledger.
79 +2. `hedonic.py` — features, nested-grid location partition, Huber-IRLS fixed-effects regression (pyfixest).
80 +3. `estimators.py` — benchmarks A–J (median, stratified median, pooled TD, rolling TD with 3 splices, hedonic imputation L/P/F, GEKS, HMTS, repeat sales BMN/CS, static EB, out-of-time GBM); `qhpi.py` — benchmark K.
81 +4. `statespace.py` — local level / local linear trend Kalman filter & smoother, robust innovations, grid-EM empirical-Bayes prior on log τ².
82 +5. `sharp.py` — stage 1 + nested stage 2, reliability bundle; `panel.py` — schema, normalisation, grades.
83 +6. `simulate.py` — synthetic Quebec-like markets with known latent paths (9 scenarios).
84 +
85 +## 🖼 Figures & tables inventory
86 +
87 +Figures (`figures/*.pdf`): coverage, arrival profile, province methods, region methods, municipal examples, gain vs n, hedonic profiles, municipal map, Monte Carlo RMSE and coverage, revisions, placebo, composition shock, aggregation, Moran. Tables (`paper/tables/*.tex`): exclusions, sample by year, completeness, municipality bands, vintages, stage 1, priors, grades, top municipalities, benchmarks (province, region), Monte Carlo (design, RMSE by level, coverage, bias, lag), revisions, policies, repeat sales, composition, placebo, Moran, bootstrap, aggregation, ablation, trimming, external.
88 +
89 +## ⚠️ Limitations
90 +
91 +See `docs/limitations.md`: single current roll vintage (post-sale characteristics), $50,000 provider floor, no arm's-length / listing / image data, metric-grid location, administrative hierarchy, model-based intervals narrower than a cluster bootstrap for mid-size municipalities, no seasonal adjustment, CMA boundaries unavailable.
92 +
93 +## 📝 Citation
94 +
95 +```bibtex
96 +@techreport{boucher2026sharphpi,
97 + author = {Boucher, Simon-Pierre},
98 + title = {SHARP-HPI: A Spatially Hierarchical, Adaptive and Robust Framework for Quality-Adjusted Housing Price Indices},
99 + institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives},
100 + type = {Working Paper},
101 + number = {22},
102 + year = {2026},
103 + month = {October}
104 +}
105 +```
106 +
107 +## 👤 Author & contact
108 +
109 +**Simon-Pierre Boucher** — Département des sciences administratives, Université du Québec en Outaouais (UQO), Gatineau, QC.
110 +📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca)
111 +
112 +## 🗒 Notes
113 +
114 +- Follows `Recherche-UQO/HARMONISATION.md`; the protocol asked for a folder named `uqo-research`, which does not exist — the series root `Recherche-UQO` is used (decision D0.1).
115 +- Transaction-level files (`clean_transactions.parquet`, `analytic_transactions.parquet`, `index_panel.csv`, revision microdata) are git-ignored; aggregates are committed.
modified figures/fig_aggregation.pdf +0 −0

Binary file not shown.

modified figures/fig_arrival_profile.pdf +0 −0

Binary file not shown.

modified figures/fig_composition_shock.pdf +0 −0

Binary file not shown.

modified figures/fig_coverage_monthly.pdf +0 −0

Binary file not shown.

modified figures/fig_gain_vs_n.pdf +0 −0

Binary file not shown.

modified figures/fig_hedonic_profiles.pdf +0 −0

Binary file not shown.

modified figures/fig_map_municipal_house.pdf +0 −0

Binary file not shown.

modified figures/fig_map_municipal_house.png +0 −0

Binary file not shown.

modified figures/fig_moran.pdf +0 −0

Binary file not shown.

modified figures/fig_moran.png +0 −0

Binary file not shown.

modified figures/fig_municipal_examples.pdf +0 −0

Binary file not shown.

modified figures/fig_municipal_examples.png +0 −0

Binary file not shown.

added figures/fig_placebo.pdf +0 −0

Binary file not shown.

added figures/fig_placebo.png +0 −0

Binary file not shown.

added figures/fig_province_methods.pdf +0 −0

Binary file not shown.

added figures/fig_province_methods.png +0 −0

Binary file not shown.

added figures/fig_region_methods.pdf +0 −0

Binary file not shown.

added figures/fig_region_methods.png +0 −0

Binary file not shown.

modified paper/figures/fig_aggregation.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_arrival_profile.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_composition_shock.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_coverage_monthly.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_gain_vs_n.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_hedonic_profiles.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_map_municipal_house.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_moran.pdf +0 −0

Binary file not shown.

modified paper/figures/fig_municipal_examples.pdf +0 −0

Binary file not shown.

added paper/figures/fig_placebo.pdf +0 −0

Binary file not shown.

added paper/figures/fig_province_methods.pdf +0 −0

Binary file not shown.

added paper/figures/fig_region_methods.pdf +0 −0

Binary file not shown.

modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/sections/data.tex +1 −1
@@ -28,7 +28,7 @@ Property classes are built from the roll rather than from the provider label: a
28 28
29 29 \subsection{Characteristics, missingness and what is not observed}
30 30
31 −The available quality variables are floor area of storeys, lot area, year of construction, number of storeys, building genre (bungalow, two-storey, split-level, mansard, one-module), physical link and, for plexes, the number of dwellings. Their completeness by year is stable (Table~\ref{tab:completeness}), which is the condition for a time-dummy model: a variable whose missingness drifts creates artificial index steps. One provider field, the building style, was backfilled over the sample (12\% present in January 2021, 87\% in January 2025) and is banned from every model. Bedrooms, bathrooms, parking, basement finish, construction quality, renovations, waterfront, occupancy, financing and arm's-length status are not observed, and no image or text features are used. The roll's neighbourhood code turned out to be a unit-level identifier and is unusable; fine location is instead a nested metric grid (Section~\ref{sec:framework}). Characteristics are a single current roll vintage: renovations after the sale are invisible, and new constructions of 2025--2026 may still be coded as land, which lowers new-build coverage at the end of the sample.
31 +The available quality variables are floor area of storeys, lot area, year of construction, number of storeys, building genre (bungalow, two-storey, split-level, mansard, one-module), physical link and, for plexes, the number of dwellings. Within the headline sample their completeness is high and stable over time (Table~\ref{tab:completeness}; the raw-row version in Table~\ref{tab:completeness_supp} shows lower shares in 2021 because land and unclassified rows are more frequent then), which is the condition for a time-dummy model: a variable whose missingness drifts creates artificial index steps. One provider field, the building style, was backfilled over the sample (12\% present in January 2021, 87\% in January 2025) and is banned from every model. Bedrooms, bathrooms, parking, basement finish, construction quality, renovations, waterfront, occupancy, financing and arm's-length status are not observed, and no image or text features are used. The roll's neighbourhood code turned out to be a unit-level identifier and is unusable; fine location is instead a nested metric grid (Section~\ref{sec:framework}). Characteristics are a single current roll vintage: renovations after the sale are invisible, and new constructions of 2025--2026 may still be coded as land, which lowers new-build coverage at the end of the sample.
32 32
33 33 \input{tables/tab_completeness}
34 34 \input{tables/tab_mun_bands}
modified paper/sections/validation.tex +69 −1
@@ -1,2 +1,70 @@
1 1 \section{Benchmarks, validation, ablation and robustness}\label{sec:validation}
2 −% filled in after benchmarks and vintages complete
2 +
3 +\subsection{Benchmarks at the provincial and regional levels}\label{sec:bench}
4 +
5 +%%BENCH%%
6 +
7 +\subsection{Agreement with repeat sales}
8 +
9 +Table~\ref{tab:repeat_sales} compares SHARP-HPI with Case--Shiller repeat-sales indices estimated on the same cells where at least 300 pairs exist. At the provincial level the two house series have a correlation of \RSCorrLevelprovincehouse{} in levels; across regions the median correlation is \RSCorrLevelregionhouse{} and across the 20 municipalities with enough pairs \RSCorrLevelmunicipalityhouse. The repeat-sales total change exceeds the hedonic one by a median of \RSGapTotalregionhouse{} log points for houses at the regional level and \RSGapTotalregioncondo{} for condominiums. The sign and the class pattern are the classical ones: repeat sales of houses embed renovations and the selection of resold dwellings \citep{clapp1991alltransactions,gatzlaffhaurin1997,nowaksmith2020}, which bias them upward relative to a constant-quality index, while condominiums are renovated less. Repeat sales are an independent arbiter of the movement, not the truth, and the Monte Carlo evidence is the only place where the truth is known.
10 +
11 +\input{tables/tab_repeat_sales}
12 +
13 +\subsection{Composition shock}
14 +
15 +Table~\ref{tab:composition} and Figure~\ref{fig:composition} report a synthetic quality-composition shock: in four regions, the smallest 40\% of houses (by floor area) are removed from the sample from January 2024 onward, which raises the median price mechanically. The median index records a spurious 2023-to-2024 jump of \CompShockAmedianZeroSix{} log points in the Montréal region and \CompShockAmedianZeroThree{} in the Capitale-Nationale; the pooled time dummy moves by \CompShockCpooledtdZeroSix{} and \CompShockCpooledtdZeroThree, and SHARP-HPI by \CompShockSsharpZeroSix{} and \CompShockSsharpZeroThree. The hedonic controls remove the shock almost entirely; the residual effect is the extrapolation of the floor-area curve to a truncated support.
16 +
17 +\input{tables/tab_composition}
18 +\begin{figure}[htbp]\centering
19 +\includegraphics[width=0.6\textwidth]{figures/fig_composition_shock.pdf}
20 +\caption{Composition shock (houses, four regions): shift of the 2023-to-2024 change after removing the smallest 40\% of houses from January 2024.}\label{fig:composition}
21 +\end{figure}
22 +
23 +\subsection{Placebo}
24 +
25 +When sale months are randomly permuted within each municipality (keeping composition, location and the municipality's sales calendar but destroying the time structure), the estimated municipal deviations should be flat. Table~\ref{tab:placebo} shows that they are: the median standard deviation of the placebo deviation path is below 1.6 log points at the municipal level and below 0.3 at the MRC level, and the share of placebo 12-month changes outside their 95\% band is \PlaceboFPhousemunicipality\% for houses and \PlaceboFPcondomunicipality\% for condominiums at the municipal level, below the nominal 5\%. At the regional level for houses the rate is \PlaceboFPhouseregion\%, above nominal: with 50 or more sales a month the permuted means are not independent across months because the permutation preserves within-municipality volumes, and the residual seasonal composition survives. Figure~\ref{fig:placebo} contrasts the placebo and actual distributions of 12-month municipal changes.
26 +
27 +\input{tables/tab_placebo}
28 +\begin{figure}[htbp]\centering
29 +\includegraphics[width=0.6\textwidth]{figures/fig_placebo.pdf}
30 +\caption{Placebo (months permuted within municipality) versus actual absolute 12-month changes of municipal house indices.}\label{fig:placebo}
31 +\end{figure}
32 +
33 +\subsection{Residual spatial dependence}
34 +
35 +Moran's $I$ of the stage-one residuals on ten nearest neighbours, computed in June of each year province-wide and within the three largest cities (Table~\ref{tab:moran}, Figure~\ref{fig:moran}), is small: within cities it ranges between $-0.02$ and $0.04$ and is significant at the 5\% level in a minority of the months tested. The 500-m to 1-km location cells absorb most of the spatial structure; what remains is a second-order concern for the index, which averages residuals over cells, but it means that model-based intervals understate the variance in cells where the sold dwellings cluster.
36 +
37 +\input{tables/tab_moran}
38 +\begin{figure}[htbp]\centering
39 +\includegraphics[width=0.6\textwidth]{figures/fig_moran.pdf}
40 +\caption{Moran's $I$ (10 nearest neighbours) of stage-one house residuals, June of each year.}\label{fig:moran}
41 +\end{figure}
42 +
43 +\subsection{Bootstrap versus model-based intervals}
44 +
45 +A cluster bootstrap that resamples fine-location cells with replacement and re-estimates both stages (40 draws, condominiums) gives the comparison in Table~\ref{tab:bootstrap}. For the province the bootstrap standard error of the log level is about half the model-based one: the model-based $\mu$ standard error from the weighted regression does not exploit the within-cell dependence that the bootstrap preserves, and is conservative. For municipalities with 200 to 600 sales the bootstrap is \BootRatioLevelMedian{} times the model-based standard error (median over the cells tested): the stage-two variances treat the monthly residual means as independent observations with known variance, and the bootstrap adds the variance of the hedonic surface and of the location effects. For the thinnest cells the ratio falls below one, because resampling cells rather than sales leaves a cell with three sales almost unchanged. We report the model-based intervals in the panel and this calibration table alongside them; a user who wants conservative intervals for mid-sized municipalities should inflate them by the tabulated ratio.
46 +
47 +\input{tables/tab_bootstrap}
48 +
49 +\subsection{Ablations on the Quebec data}
50 +
51 +Table~\ref{tab:ablation_real} reports, for each class and level, how far each variant's municipal monthly changes depart from the full model and the out-of-time calibration of its one-step-ahead innovations. Removing the empirical-Bayes calibration (per-cell maximum likelihood $\tau^2$) changes house municipal monthly changes by \AblnoEBhousemunicipality{} log points on average and lowers the mean local-data weight, because many thin cells end at the lower boundary of $\tau^2$ and are frozen to their parent. The Gaussian variant and the municipality-level location variant change the series by \Ablnorobusthousemunicipality{} and \Ablcoarselochousemunicipality{} log points, the largest departures: outlying months and within-municipality location shifts both matter. Removing the MRC level changes municipal series by \AblskipMRChousemunicipality{} log points, which measures how much the administrative hierarchy itself contributes. The out-of-time innovation diagnostics are above one for every variant (\AblZfullhouse{} for the full model at the municipal level): the model's one-step predictive variances are too small by 20--40\% for houses, consistent with the bootstrap, and no variant closes the gap; the local-linear-trend variant is slightly worse than the local level.
52 +
53 +\input{tables/tab_ablation_real}
54 +
55 +\subsection{Sample screens}
56 +
57 +Table~\ref{tab:trimming} shows that the provincial time-dummy path is insensitive to the sample screens: keeping the assessment-ratio outliers, dropping type conflicts or new constructions, applying a monthly 1--99\% price trim, or keeping only roll-classified sales changes the 2021--2026 house path by at most one log point at any month, and the Huber and ordinary least-squares versions differ by less than two log points in total. The robustness comes from the robust likelihood and the location controls, not from the screens.
58 +
59 +\input{tables/tab_trimming}
60 +
61 +\subsection{Real-time vintages and revisions}\label{sec:revisions}
62 +
63 +%%VINTAGES%%
64 +
65 +\subsection{External comparison}
66 +
67 +Table~\ref{tab:external} compares the provincial and metropolitan SHARP series with the Teranet--National Bank and CREA indices, rebased to the 2021 average and compared on annual means to avoid the thin January 2021. In the Québec City metropolitan area the agreement is close: the SHARP house aggregate rises by \ExtSHARPhousestockweightedCREASingleFamilyNSAQubecCMASharp{} log points from 2021 to 2026 against \CreaSFQcYr{} for the CREA single-family benchmark and \TeranetQcYr{} for Teranet (all types), and the condominium aggregate by \ExtSHARPcondostockweightedCREAApartmentNSAQubecCMASharp{} against \CreaAptQcYr{} for CREA apartments. In Montréal the SHARP house aggregate rises by \ExtSHARPhousestockweightedCREASingleFamilyNSAMontralCMASharp{} against \CreaSFMtlYr{} for CREA and \TeranetMtlYr{} for Teranet, a gap of about ten log points, while condominiums are within five points of CREA apartments. Table~\ref{tab:external_rs} shows that the gap is in the data rather than in the estimator: repeat sales on our own Montréal house sample rise by \ExtRSMontralCMMhouses{} log points, the same as the SHARP aggregate, whereas repeat sales on all three classes together rise by \ExtRSMontralCMMallthreeclasses{}, closer to Teranet's all-type \TeranetMtlYr. Two explanations remain for the house gap relative to CREA: the CREA benchmark uses bedroom, bathroom and other listing attributes that absorb quality upgrades our roll variables cannot, and brokered resales are a different population from registry deeds. The correlation of 12-month changes between SHARP and the external series is between \ExtSHARPcondostockweightedTeranetNSArepeatsalesMontralCMACorr{} and \ExtSHARPhousestockweightedTeranetNSArepeatsalesMontralCMACorr{} in Montréal; in Québec City it is lower because the SHARP metropolitan aggregate is noisier month to month than the smoothed external series.
68 +
69 +\input{tables/tab_external}
70 +\input{tables/tab_external_rs}
modified paper/tables/tab_completeness.tex +12 −11
@@ -1,21 +1,22 @@
1 1 \begin{table}[htbp]
2 2 \centering\small
3 −\caption{Completeness of characteristics by year of sale (all raw rows)}\label{tab:completeness}
3 +\caption{Completeness of the two main characteristics and share of new constructions in the headline sample, by year and class}\label{tab:completeness}
4 4 \begin{threeparttable}
5 5 \begin{tabular}{lrrrrrr}
6 6 \toprule
7 −Variable (\% non-missing) & 2021 & 2022 & 2023 & 2024 & 2025 & 2026 \\
7 +Headline sample (\% of sales) & 2021 & 2022 & 2023 & 2024 & 2025 & 2026 \\
8 8 \midrule
9 −Floor area & 79.6 & 81.6 & 86.5 & 88.8 & 88.2 & 85.2 \\
10 −Year built & 69.8 & 77.0 & 86.4 & 88.9 & 88.4 & 85.3 \\
11 −Lot area & 99.6 & 99.7 & 99.8 & 99.8 & 99.8 & 98.1 \\
12 −Storeys & 90.8 & 90.1 & 91.2 & 91.6 & 92.0 & 92.9 \\
13 −Physical link & 89.9 & 89.4 & 90.5 & 90.9 & 91.2 & 92.2 \\
14 −Building genre & 86.3 & 85.6 & 87.0 & 87.4 & 87.8 & 88.8 \\
15 −Provider style (banned) & 12.8 & 51.4 & 82.5 & 85.2 & 84.8 & 81.8 \\
16 −Roll identifier & 98.8 & 98.8 & 99.1 & 99.4 & 99.5 & 99.3 \\
9 +Houses: floor area & 99.1 & 99.2 & 99.2 & 99.2 & 99.0 & 99.1 \\
10 +Houses: year built & 99.0 & 99.1 & 99.1 & 99.1 & 98.9 & 99.0 \\
11 +Houses: new construction (\\% of sales) & 3.9 & 3.8 & 2.1 & 1.5 & 0.7 & 0.0 \\
12 +Condominiums: floor area & 98.2 & 98.4 & 98.4 & 98.6 & 98.4 & 98.5 \\
13 +Condominiums: year built & 99.9 & 99.8 & 99.8 & 99.8 & 99.8 & 99.8 \\
14 +Condominiums: new construction (\\% of sales) & 0.5 & 0.6 & 0.1 & 0.1 & 0.0 & 0.0 \\
15 +Plexes: floor area & 97.7 & 97.7 & 97.7 & 97.8 & 97.8 & 97.9 \\
16 +Plexes: year built & 97.9 & 97.7 & 97.9 & 98.0 & 98.0 & 98.0 \\
17 +Plexes: new construction (\\% of sales) & 1.1 & 0.9 & 0.7 & 0.3 & 0.1 & 0.0 \\
17 18 \bottomrule
18 19 \end{tabular}
19 −\begin{tablenotes}\footnotesize\item The provider's building style is backfilled over time and is excluded from every model (time-correlated missingness).\end{tablenotes}
20 +\begin{tablenotes}\footnotesize\item Lot area, storeys, physical link and building genre are complete for more than 90\% of headline sales in every year (results/profile/completeness\_by\_year.csv). The provider's building style (12\% present in January 2021, 87\% in January 2025) is excluded from every model. New constructions (sold in the roll's construction year) fall to zero in 2025--2026 because the roll vintage lags construction.\end{tablenotes}
20 21 \end{threeparttable}
21 22 \end{table}
modified paper/tables/tab_completeness_supp.tex +21 −0
@@ -0,0 +1,21 @@
1 +\begin{table}[htbp]
2 +\centering\small
3 +\caption{Completeness of characteristics by year of sale, all raw rows (including land, non-residential and unmatched sales)}\label{tab:completeness_supp}
4 +\begin{threeparttable}
5 +\begin{tabular}{lrrrrrr}
6 +\toprule
7 +Variable (\% non-missing) & 2021 & 2022 & 2023 & 2024 & 2025 & 2026 \\
8 +\midrule
9 +Floor area & 79.6 & 81.6 & 86.5 & 88.8 & 88.2 & 85.2 \\
10 +Year built & 69.8 & 77.0 & 86.4 & 88.9 & 88.4 & 85.3 \\
11 +Lot area & 99.6 & 99.7 & 99.8 & 99.8 & 99.8 & 98.1 \\
12 +Storeys & 90.8 & 90.1 & 91.2 & 91.6 & 92.0 & 92.9 \\
13 +Physical link & 89.9 & 89.4 & 90.5 & 90.9 & 91.2 & 92.2 \\
14 +Building genre & 86.3 & 85.6 & 87.0 & 87.4 & 87.8 & 88.8 \\
15 +Provider style (banned) & 12.8 & 51.4 & 82.5 & 85.2 & 84.8 & 81.8 \\
16 +Roll identifier & 98.8 & 98.8 & 99.1 & 99.4 & 99.5 & 99.3 \\
17 +\bottomrule
18 +\end{tabular}
19 +\begin{tablenotes}\footnotesize\item The provider's building style is backfilled over time and is excluded from every model (time-correlated missingness). Lower shares in 2021 reflect the larger share of land and unclassified rows, not missing data in the headline sample (Table~\ref{tab:completeness}).\end{tablenotes}
20 +\end{threeparttable}
21 +\end{table}
modified scripts/60_figures.py +5 −2
@@ -20,7 +20,7 @@ R = C.RESULTS
20 20
21 21
22 22 def save(fig, name):
23 − fig.savefig(FIG / f"{name}.pdf"); fig.savefig(PFIG / f"{name}.pdf"); fig.savefig(FIG / f"{name}.png", dpi=160); plt.close(fig); print("fig", name)
23 + fig.savefig(FIG / f"{name}.pdf", dpi=200); fig.savefig(PFIG / f"{name}.pdf", dpi=200); fig.savefig(FIG / f"{name}.png", dpi=160); plt.close(fig); print("fig", name)
24 24
25 25
26 26 def exists(*paths):
@@ -179,13 +179,16 @@ def fig_map():
179 179 mun = gpd.read_file(C.SDA_GPKG, layer="munic_s")[["MUS_CO_GEO", "geometry"]].rename(columns={"MUS_CO_GEO": "geography_id"})
180 180 reg = gpd.read_file(C.SDA_GPKG, layer="regio_s")
181 181 g = mun.merge(last.to_pandas(), on="geography_id", how="left").to_crs("EPSG:32198")
182 + g["geometry"] = g.geometry.simplify(150, preserve_topology=True)
182 183 g = g.cx[-900000:900000, 100000:900000] # southern Quebec extent (where sales exist)
183 184 fig, ax = plt.subplots(figsize=(7.2, 5.2))
184 185 g[g["index_value"].isna() | g["suppression_flag"].fillna(True)].plot(ax=ax, color="#eeeeea", edgecolor="white", linewidth=0.1)
185 186 gg = g[~g["index_value"].isna() & ~g["suppression_flag"].fillna(True)]
186 187 gg.plot(ax=ax, column="index_value", cmap="Blues", vmin=110, vmax=200, edgecolor="white", linewidth=0.1, legend=True, legend_kwds={"shrink": 0.5, "label": "House index, Sept. 2026 (2021 = 100)"})
187 188 gg[gg["reliability_grade"].isin(["D", "E"])].plot(ax=ax, facecolor="none", edgecolor="#e34948", hatch="////", linewidth=0.0, alpha=0.5)
188 − reg.to_crs("EPSG:32198").boundary.plot(ax=ax, color=INK2, linewidth=0.3)
189 + rb = reg.to_crs("EPSG:32198"); rb["geometry"] = rb.geometry.simplify(300, preserve_topology=True); rb.boundary.plot(ax=ax, color=INK2, linewidth=0.3)
190 + for coll in ax.collections:
191 + coll.set_rasterized(True)
189 192 ax.set_xlim(-650000, 500000); ax.set_ylim(130000, 700000); ax.axis("off")
190 193 ax.set_title("SHARP-HPI municipal house index, September 2026 (hatched: reliability grade D/E; grey: suppressed or no sales)", fontsize=8)
191 194 save(fig, "fig_map_municipal_house")
modified scripts/61_tables.py +11 −2
@@ -116,8 +116,17 @@ def t_data():
116 116 comp = pl.read_csv(R / "profile" / "completeness_by_year.csv")
117 117 cols = [("floor_area", "Floor area"), ("year_built", "Year built"), ("land_area", "Lot area"), ("nb_etages", "Storeys"), ("lien_physique", "Physical link"), ("genre_construction", "Building genre"), ("building_type", "Provider style (banned)"), ("id_provinc", "Roll identifier")]
118 118 rows = [[lab] + [pct(v) for v in comp[c].to_list()] for c, lab in cols]
119 − table("tab_completeness", ["Variable (% non-missing)"] + [str(y) for y in comp["year"].to_list()], rows, "Completeness of characteristics by year of sale (all raw rows)", "tab:completeness",
120 − notes="The provider's building style is backfilled over time and is excluded from every model (time-correlated missingness).")
119 + table("tab_completeness_supp", ["Variable (% non-missing)"] + [str(y) for y in comp["year"].to_list()], rows, "Completeness of characteristics by year of sale, all raw rows (including land, non-residential and unmatched sales)", "tab:completeness_supp",
120 + notes="The provider's building style is backfilled over time and is excluded from every model (time-correlated missingness). Lower shares in 2021 reflect the larger share of land and unclassified rows, not missing data in the headline sample (Table~\\ref{tab:completeness}).")
121 + hs2 = pl.read_csv(R / "cleaning" / "headline_sample_by_year_class.csv").filter(pl.col("property_class").is_not_null())
122 + rows = []
123 + for cls, lab in [("house", "Houses"), ("condo", "Condominiums"), ("plex", "Plexes")]:
124 + x = hs2.filter(pl.col("property_class") == cls).sort("year")
125 + rows.append([f"{lab}: floor area"] + [pct(1 - v) for v in x["share_floor_area_missing"].to_list()])
126 + rows.append([f"{lab}: year built"] + [pct(1 - v) for v in x["share_year_built_missing"].to_list()])
127 + rows.append([f"{lab}: new construction (\\% of sales)"] + [pct(v) for v in x["share_new_construction"].to_list()])
128 + table("tab_completeness", ["Headline sample (% of sales)"] + [str(y) for y in sorted(hs2["year"].unique().to_list())], rows, "Completeness of the two main characteristics and share of new constructions in the headline sample, by year and class", "tab:completeness",
129 + notes="Lot area, storeys, physical link and building genre are complete for more than 90\\% of headline sales in every year (results/profile/completeness\\_by\\_year.csv). The provider's building style (12\\% present in January 2021, 87\\% in January 2025) is excluded from every model. New constructions (sold in the roll's construction year) fall to zero in 2025--2026 because the roll vintage lags construction.")
121 130 bands = pl.read_csv(R / "profile" / "municipality_size_bands.csv")
122 131 table("tab_mun_bands", ["Sales 2021–2026", "Municipalities", "Sales"], [[r["band"], n0(r["municipalities"]), n0(r["transactions"])] for r in bands.iter_rows(named=True)],
123 132 "Municipalities by number of roll-matched sales, 2021–2026", "tab:mun_bands")
124 133