spb/qwhpi Public
QHPI — Quebec Housing Price Index: quality-adjusted, hierarchically pooled housing price indexes.
Python 63.9%
TypeScript 25.4%
CSS 5.5%
TeX 3.5%
SQL 0.8%
Makefile 0.5%
Dockerfile 0.5%
1<!--2=============================================================================3QWHPI — Quebec Weekly Housing Price Index4Author : Simon-Pierre Boucher5Contact : contact@spboucher.ai6File : README.md7Purpose : Full platform documentation — methodology, architecture, API,8 dashboard, data policy, reproducibility.9=============================================================================10-->1112<div align="center">1314# 🏠 QHPI — Quebec Housing Price Index1516### Quality-adjusted, hierarchically pooled housing price indexes for Quebec1718*Province · 17 administrative regions · major municipalities × {all, unifamilial, condo, plex}*1920[](https://www.indexqc.house)21[](#-methodology)22[%20%2B%20weekly%20(research)-8a2be2?style=flat-square)](#-methodology)23[](#-index-families--coverage)24[-orange?style=flat-square)](#-methodology)25[](#-data-policy--no-microdata-in-this-repo)2627[](engine/pyproject.toml)28[](api/)29[](web/)30[](db/)31[](docker-compose.yml)32[](paper/)3334[](mailto:contact@spboucher.ai)35[](mailto:contact@spboucher.ai)3637**Author : Simon-Pierre Boucher — [contact@spboucher.ai](mailto:contact@spboucher.ai)**3839</div>4041---4243## Table of contents4445- [What this is](#-what-this-is)46- [Key numbers](#-key-numbers)47- [Data policy — no microdata in this repo](#-data-policy--no-microdata-in-this-repo)48- [Platform architecture](#-platform-architecture)49- [Methodology](#-methodology)50- [Validation](#-validation)51- [Index families & coverage](#-index-families--coverage)52- [Canonical dataset](#-canonical-dataset)53- [API](#-api)54- [Dashboard](#-dashboard)55- [Repository map](#-repository-map)56- [Quickstart](#-quickstart)57- [Engine pipeline](#-engine-pipeline)58- [Engineering standards](#-engineering-standards)59- [Research outputs & paper](#-research-outputs--paper)60- [Reproducibility](#-reproducibility)61- [Author & license](#-author--license)6263---6465## 🎯 What this is6667QHPI is a **production-grade economic-measurement platform**, not a median-price68tracker. It separates **price movement** from **composition movement**: a week69where only mansions sell must not register as a price increase.7071The platform has four layers:7273| Layer | What it does | Where |74|---|---|---|75| **Index Engine** | Reproducible econometric pipeline: cleaning → hedonic estimation → hierarchical pooling → validation → canonical Parquet | [`engine/`](engine/) |76| **Data Store** | Versioned Parquet lake (aggregated series only) + PostgreSQL serving layer | [`data/processed/`](data/processed/), [`db/`](db/) |77| **API** | FastAPI service exposing every published series with uncertainty, reliability and vintages | [`api/`](api/) |78| **Dashboard** | Interactive Next.js frontend — explore, compare, map, PDF reports, live at [www.indexqc.house](https://www.indexqc.house) | [`web/`](web/) |7980Every published observation looks like this:8182```83Geography: Quebec City | Type: Condo | Period: 2026-0684Index: 163.5 (2021 = 100) | 1m: +0.4% | YoY: +8.1%85Representative value: $389,200 | Transactions: 47 | Reliability: A8695% CI: [160.9, 166.1] | Vintage: 2026-08-08 | Model: 2.1.087```8889Uncertainty is **never hidden**: every row carries a 95% confidence interval,90an A–E reliability grade, effective sample size and shrinkage weight.9192---9394## 📊 Key numbers9596979899100101| Metric | Value |102|---|---|103| Raw transactions ingested | ~745,000 (2021-01 → present) |104| Estimation sample after documented exclusions | ~720,000 |105| Geographic units | Province + 17 administrative regions + major municipalities |106| Property types | `unifamilial`, `condo`, `plex` (+ `all` composite) |107| Published cells | 81 — 50 **direct** liquid cells + 31 **hierarchical** thin cells |108| Headline frequency | **Monthly** (v2.1) — weekly v1 retained as a research module |109| Base | 2021 average = 100 |110| Reliability grades | **A** ≥150 tx/mo · **B** ≥75 · **C** ≥40 · **D** ≥15 · **E** <15 (model-implied) |111| Growth horizons | 1m, 3m, 6m, YoY (weekly module: 1w, 4w, 13w, 26w, YoY) |112| Vintages | `first_release` preserved; every revision queryable |113114---115116## 🔒 Data policy — no microdata in this repo117118> **This repository contains NO individual transaction data.**119120| Category | In repo? | Detail |121|---|---|---|122| Raw transaction CSV (`data/raw/`) | ❌ **excluded** | Individual sale records — never versioned |123| Transaction-level intermediates (`data/interim/`, `transactions_clean.parquet`, `geo_join.parquet`) | ❌ **excluded** | Anything with one row per transaction |124| Cadastral boundary files (`SDA.gpkg`) | ❌ excluded (size) | Source, URL, version and CRS documented in [`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md) |125| **Aggregated published series** | ✅ included | `qhpi_monthly.parquet`, `qwhpi_weekly.parquet`, liquidity, coverage matrices, assessment gap, first releases, vintages |126| **Aggregate tables & figures** (`outputs/`) | ✅ included | Audit counts, model comparisons, validation results, publication figures |127128The [`.gitignore`](.gitignore) enforces this at the repo boundary. Aggregates129are cell-level (`geography × type × period`) with transaction **counts**, never130transaction rows. To rerun the full pipeline you must supply your own131`data/raw/province_transactions.csv` (see [Reproducibility](#-reproducibility)).132133---134135## 🏗 Platform architecture136137```138 ┌─────────────────────────────────────────────┐139 │ province_transactions.csv │140 │ (local only — NOT in this repo) │141 └──────────────────────┬──────────────────────┘142 ▼143 ┌───────────────────────────── ENGINE (Python) ─────────────────────────────┐144 │ 01 profile → 02 geography (spatial join, SDA boundaries) → 03 clean │145 │ → 04–08 weekly research module (baseline, hierarchical, validation) │146 │ → 09 monthly estimation (rolling time-dummy, Huber-IRLS) │147 │ → 10 monthly validation → 11 canonical parquet │148 └──────────────────────────────────┬────────────────────────────────────────┘149 ▼150 data/processed/qhpi_monthly.parquet ← source of truth151 (+ liquidity, coverage, vintages, assessment gap)152 ▼153 ┌────────────── PostgreSQL (docker-compose, Alembic) ──────────────┐154 │ series · observations · geographies · liquidity · vintages │155 └──────────────────────────────┬────────────────────────────────────┘156 ▼157 FastAPI /v1/* (uncertainty always returned)158 ▼159 Next.js dashboard — www.indexqc.house160 (explore · compare · map · PDF reports · methodology)161```162163Weekly refresh: `ops/scheduler/` ingests new rows → append-only clean →164re-estimation → new `data_vintage` → DB load → API cache invalidation.165166---167168## 🧪 Methodology169170The index is a **robust hedonic time-dummy index with hierarchical pooling** —171three estimation regimes arbitrated by empirical validation (repeat sales +172stratified matched-cell medians), not by convenience:173174### 1. Province × type backbone — rolling time-dummy (RTD)175176- Pooled robust regressions (`log(price)` on structural characteristics +177 fine location fixed effects + period dummies) over **13-month rolling178 windows** with **Huber-IRLS** weighting.179- Windows are combined by **mean splice**, making published history180 **revision-free** by construction.181- Coefficient drift is monitored and published182 (`outputs/tables/rtd_coefficient_drift.csv`) — e.g. the floor-area183 elasticity moves 0.556 → 0.586 across windows, which a fixed pooled model184 would have hidden.185186### 2. Liquid cells (median ≥ 40 tx/month) — direct local estimation187188Each of the **50 liquid `geography × type` cells** gets its **own robust189time-dummy regression** (own hedonic coefficients, own FSA fixed effects),190lightly smoothed by a state-space filter. This was arbitrated against191deviation-from-pooled-surface modelling, which **compressed genuine local192divergence** — Quebec City condos moved ~+55% (confirmed independently by193repeat sales *and* stratified matched-cell medians) while pooled deviations194reported only +42%.195196### 3. Thin cells — hierarchical Kalman shrinkage197198The remaining **31 thin cells** follow a latent random-walk **deviation from199their parent's published path**, estimated by Kalman filter/smoother with200observation noise σ²/nₜ. Thin months shrink toward the parent trend; the201shrinkage weight is published on every row.202203### Hedonic specification (headline = structural only)204205- `log(amount)` on `log(floorArea)` (+ time-stable missingness indicator),206 age from `yearBuilt` via **conditional-median imputation** (its missingness207 drifts 15% → 2% over the sample, so a "missing" bin would leak time),208 and fine location fixed effects (FSA).209- **`buildingType` is banned from all hedonic models.** The provider210 backfilled it in region-staggered waves (most regions 2022-01, Montreal211 2023-01); the time-correlated missingness fabricated a −38 log-point cliff212 in local Montreal indexes. The ban is documented in213 [`engine/src/qwhpi/features.py`](engine/src/qwhpi/features.py).214- Assessment values (`totalArValue`, `previousValue`) never enter the215 headline (valuation leakage); they power the separate216 **Assessment Gap Index** (`amount / totalArValue`).217218### Cleaning219220Raw data is immutable; every exclusion is economically justified and counted221in [`outputs/tables/exclusions.csv`](outputs/tables/exclusions.csv)222(assessment-ratio band, exact duplicates, non-arm's-length transfers, …).223Duplicates are **flagged, never silently deleted**. The `indéterminé`224property type never contaminates type-specific indexes.225226### Geography227228Transactions are spatially joined (`lat/lng`) against the authoritative229Quebec **SDA cadastral boundaries** (source, version and CRS documented in230[`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md)) — the free-text231`city` field alone is not trusted.232233---234235## ✅ Validation236237Every claim below has a table or figure in [`outputs/`](outputs/):238239| Check | Result |240|---|---|241| **Repeat-sales benchmark** (BMN, ~85k repeated addresses) | June 2026, 2021=100: quebec/condo **132.0** vs RS 132 (exact); quebec-city/condo **163.5** vs RS 162; montreal/condo **121.6** vs RS 122; montreal/unifamilial **126.2** between stratified 120 and RS 135 (RS renovation bias on older stock) |242| **Composition-shock simulation** | Mix shock moves the raw median **+11.7%**; the hedonic index moves **−0.7%** — the index measures prices, not composition |243| **Downsampling experiment** | Montreal condos thinned to 100/50/25/15/10/5 tx/period; RMSE, bias, volatility, turning points and CI coverage justify the A–E liquidity tiers empirically |244| **Weekly vs monthly** | Same methodology at both frequencies; signal-to-noise, turning-point detection and revision magnitude quantified in `outputs/tables/weekly_vs_monthly.csv` |245| **Seasonality** | NSA published; no stable seasonality demonstrated (`seasonality_tests.csv`) |246| **Alternative specs** | Hierarchical variants, mixed effects, ML residualization benchmarked in `estimation_methods.csv` — chosen model optimizes interpretability + stability, not transaction-level RMSE |247248Full write-ups: [`outputs/reports/`](outputs/reports/) (audit, geography,249cleaning, hierarchical, validation v1 & v2, methodology research).250251---252253## 🗺 Index families & coverage254255| Family | Scope |256|---|---|257| **QHPI-QC** | Province of Quebec |258| **QHPI-REG** | 17 administrative regions |259| **QHPI-CITY** | Montréal, Québec, Laval, Gatineau, Longueuil, Sherbrooke, Trois-Rivières, Saguenay, Lévis, Drummondville, … as liquidity permits |260| **QHPI-TYPE** | All / Unifamilial / Condo / Plex per supported geography |261| **Assessment Gap** | `amount / totalArValue` — separate concept, never mixed with the price index |262263The **coverage matrix**264([`data/processed/coverage_matrix_monthly.parquet`](data/processed/),265[`outputs/tables/coverage_monthly.csv`](outputs/tables/coverage_monthly.csv))266declares per `municipality × type`: published / conditional / not published.267268---269270## 🗃 Canonical dataset271272[`data/processed/qhpi_monthly.parquet`](data/processed/qhpi_monthly.parquet)273(7,504 rows) powers the DB, API and dashboard. Weekly research module:274[`data/processed/qwhpi_weekly.parquet`](data/processed/qwhpi_weekly.parquet)275(32,592 rows).276277```278period, geography_level, geography_id, geography_name, property_type,279index, index_smoothed, representative_value,280transactions, effective_sample_size,281monthly_pct, three_month_pct, six_month_pct, yoy_pct,282lower_95, upper_95, reliability_grade, shrinkage_weight,283is_partial_month, model_version, data_vintage284```285286- `index` = raw estimate, `index_smoothed` = **one-sided** (real-time safe)287 smoother — no look-ahead in the published real-time series.288- `representative_value` = dollar value of a documented, fixed property289 basket per segment.290- Vintage framework: first releases preserved under291 `data/processed/vintages*/`, revisions queryable via the API.292293---294295## 🔌 API296297FastAPI service (OpenAPI docs at `/docs`), CSV/JSON export toggle, ETag298caching, pagination, rate limiting, CORS. **Reliability and CIs are always299returned.**300301| Endpoint | Purpose |302|---|---|303| `GET /v1/index?geography=quebec-city&type=condo&from=2021-01&to=latest` | Full series |304| `GET /v1/index/latest?geography=…&type=…` | Latest observation |305| `GET /v1/geographies` · `GET /v1/geographies/coverage` | Hierarchy + coverage matrix |306| `GET /v1/liquidity?geography=…&type=…` | Transaction counts & effective N |307| `GET /v1/compare?series=montreal:condo,quebec-city:condo` | Multi-series comparison |308| `GET /v1/map?metric=yoy&level=region` | Choropleth payloads (+ `period` for time-lapse) |309| `GET /v1/stats` · `GET /v1/stats/overview` | Peak/drawdown/CAGR/volatility/momentum/rank, heatmap rows |310| `GET /v1/report?series=…` · `GET /v1/report/market` | Publication-grade PDF reports (matplotlib) |311| `GET /v1/vintages?geography=…&type=…` | Revision history |312| `GET /v1/meta` | `model_version`, `data_vintage`, methodology link, author credit |313| `GET /v1/health` | Liveness |314315Sample response:316317```json318{319 "geography": "Quebec City",320 "property_type": "condo",321 "frequency": "monthly",322 "latest_index": 163.5,323 "representative_value": 389200,324 "monthly_change": 0.4,325 "yoy_change": 8.1,326 "transactions": 47,327 "reliability": "A",328 "lower_95": 160.9,329 "upper_95": 166.1,330 "is_partial_month": false,331 "author": "Simon-Pierre Boucher",332 "contact": "contact@spboucher.ai"333}334```335336---337338## 💻 Dashboard339340Live at **[www.indexqc.house](https://www.indexqc.house)** — Next.js +341TypeScript, dark/light themes, fully responsive (audited at 390/360 px),342`prefers-reduced-motion` respected, zero-dependency custom hero chart,343⌘K command palette over all 112 series.344345| Page | Contents |346|---|---|347| `/` | Market pulse: heat-shaded region table with sparklines, 12 metric tiles, ticker tape, top movers, market-report PDF |348| `/explore` | Series picker, CI band, raw-median & smoothed toggles, 1Y/3Y/5Y/YTD/MAX ranges, Bank-of-Canada event annotations, brush zoom, PNG/CSV export, sortable data table, shareable URLs |349| `/compare` | Multi-series with rebasing tool, toggleable legend chips, comparison PDF |350| `/map` | Region choropleth (YoY, 3m, level, assessment gap) with monthly slider + ▶ time-lapse |351| `/methodology` | Rendered methodology + link to the paper |352| `/api-docs` | Live playground (run requests in-page) + copyable curl examples |353354Low-reliability series are **visually flagged, never hidden**. Footer credit:355*Simon-Pierre Boucher — contact@spboucher.ai*.356357---358359## 📁 Repository map360361```362├── engine/ # Index engine (Python)363│ ├── src/qwhpi/ # ingest · clean · geography · features · hedonic364│ │ # rtd · hierarchy · state_space · repeat_sales365│ │ # index · uncertainty · seasonal · nowcast366│ │ # vintages · export · plotting367│ ├── scripts/ # 01_profile … 11_monthly_canonical (ordered, idempotent)368│ └── tests/ # smoke pipeline, features/clean, state-space369├── data/370│ ├── external/ # boundary-source documentation (files excluded by size)371│ └── processed/ # ✅ aggregated series only (see Data policy)372├── db/ # PostgreSQL schema + Alembic migrations373├── api/ # FastAPI app: routers/ services/ tests/374├── web/ # Next.js dashboard: app/ components/ lib/ styles/375├── ops/ # weekly-refresh scheduler + CI pipeline376├── outputs/377│ ├── figures/ # publication figures (shared plotting config)378│ ├── tables/ # 35+ audit/validation/model tables (aggregates)379│ └── reports/ # markdown reports for every pipeline stage380├── paper/ # LaTeX methodology paper (+ compiled PDF)381├── scripts/check_headers.py # CI gate: every file carries the author header382├── Makefile # one-command targets383└── docker-compose.yml # postgres + api + web + scheduler384```385386---387388## 🚀 Quickstart389390> Requires the raw transaction CSV (not distributed — see391> [Data policy](#-data-policy--no-microdata-in-this-repo)) for `make pipeline`.392> The API and dashboard run off the **included aggregated Parquet** without it.393394```bash395# Engine (full run ~ audit → geography → clean → estimate → validate → canonical)396make pipeline397398# Incremental monthly refresh (clean → estimate → canonical, ~2 min)399make refresh400401# Services402make api # FastAPI on :8080 — docs at /docs403make web # Next.js dashboard on :3000404make up # docker compose: postgres + api + web + scheduler405make down406407# Quality408make test # engine + api test suites409make lint # ruff + mypy + tsc410make headers # author-header check (also a pre-commit hook + CI gate)411make figures # publication figures → outputs/figures/412make paper # compile the LaTeX paper413```414415---416417## ⚙️ Engine pipeline418419| Script | Stage |420|---|---|421| `01_profile.py` | Data audit: schema, descriptives, missingness, sentinel values, liquidity matrices |422| `02_geography.py` | SDA boundary spatial join, validation, persistence |423| `03_clean.py` | Research sample with fully documented exclusions |424| `04_baseline.py` | Weekly pooled hedonic time-dummy baseline *(research module)* |425| `05_hierarchical.py` | Weekly hierarchical state-space shrinkage *(research module)* |426| `06_validation.py` | Repeat sales, downsampling, composition shock *(weekly)* |427| `07_canonical.py` | Weekly canonical Parquet *(research module)* |428| `08_figures.py` | Publication figures |429| `09_monthly.py` | **Headline monthly estimation** (RTD + direct cells + Kalman) |430| `10_monthly_validation.py` | Monthly validation suite |431| `11_monthly_canonical.py` | **Canonical `qhpi_monthly.parquet`** + vintage write |432433Every run emits a structured **run manifest** (input hash, row counts, model434version, timings).435436---437438## 📐 Engineering standards439440- **Every file** starts with the author header — enforced by441 [`scripts/check_headers.py`](scripts/check_headers.py) in pre-commit **and** CI.442- Vectorized pandas/polars + numpy; no per-transaction Python loops; spatial443 joins, features and model matrices cached as Parquet.444- Diagnostics for every production model: residual distribution,445 heteroskedasticity, temporal/spatial residual patterns, coefficient446 stability, effective N.447- Model selection optimizes **interpretability + stability + calibrated448 uncertainty + reproducibility** — never transaction-level RMSE alone.449- CI (`ops/ci/ci.yml`): lint, mypy + tsc, tests, header check, small-sample450 smoke run of the pipeline.451452---453454## 📄 Research outputs & paper455456- **Figures** ([`outputs/figures/`](outputs/figures/)): province aggregate,457 region comparison, big-4 cities, condo by city, raw median vs hedonic,458 volumes, reliability matrix, downsampling stability, assessment gap,459 YoY appreciation map.460- **Tables** ([`outputs/tables/`](outputs/tables/)): audit suite, exclusions,461 liquidity, coverage, model comparison, repeat-sales comparison,462 downsampling, seasonality, coefficient drift, weekly-vs-monthly.463- **Paper** ([`paper/qwhpi.pdf`](paper/qwhpi.pdf)): *A High-Frequency Hedonic464 Housing Price Index for Quebec* — real, researched citations only (hedonic465 and repeat-sales index theory, Case-Shiller, hierarchical and state-space466 indexes, spatial hedonics, index-number theory).467468---469470## 🔁 Reproducibility4714721. Place your transaction extract at `data/raw/province_transactions.csv`473 (columns: `id, date, amount, street, zipCode, city, lat, lng,474 propertyType, yearBuilt, floorArea, buildingType, previousValue,475 totalArValue, ownerType`).4762. Download the SDA boundary file per477 [`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md).4783. `make pipeline` — deterministic given the same inputs; every run is479 stamped with `model_version` + `data_vintage` and manifested.480481Without microdata, everything downstream of the canonical Parquet (DB load,482API, dashboard, figures) is fully runnable from the aggregates included here.483484---485486## 👤 Author & license487488<div align="center">489490**Simon-Pierre Boucher**491492[](mailto:contact@spboucher.ai)493[](https://www.indexqc.house)494495</div>496497Code, methodology, figures and documentation © Simon-Pierre Boucher.498All rights reserved — contact the author for reuse, data questions or499collaboration. The repository intentionally distributes **no individual500transaction records**; published aggregates carry full uncertainty metadata501so they are never mistaken for more precision than the data supports.502