SPB Git
spboucher.ai

spb/qwhpi

Public

QHPI — Quebec Housing Price Index: quality-adjusted, hierarchically pooled housing price indexes.

www.indexqc.house
webappreal-estatequebecindexeconometrics

Initial commit — QHPI platform (engine, API, dashboard, paper)

Quebec Housing Price Index v2.1: hedonic rolling-time-dummy engine with
hierarchical pooling, FastAPI service, Next.js dashboard, validation suite
and methodology paper. Aggregated series only — no individual transaction
data is versioned (enforced by .gitignore).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 9, 2026) Browse files

164 changed files +19,433 −0

164 changed files
Load all 164 files Unified Split
added .gitignore +55 −0
@@ -0,0 +1,55 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : .gitignore
6 +# Purpose : Keep transaction-level data, build artifacts and caches out of git.
7 +# =============================================================================
8 +
9 +# ---------------------------------------------------------------------------
10 +# DATA POLICY — individual transaction data NEVER enters the repository.
11 +# Only aggregated, published series (indexes, liquidity, coverage, vintages)
12 +# are versioned. See README § Data policy.
13 +# ---------------------------------------------------------------------------
14 +data/raw/
15 +data/interim/
16 +data/processed/transactions_clean.parquet
17 +data/processed/geo_join.parquet
18 +
19 +# Large external boundary files (documented in data/external/SDA_SOURCE.md)
20 +data/external/*.gpkg
21 +data/external/*.gpkg.zip
22 +
23 +# Python
24 +__pycache__/
25 +*.py[cod]
26 +*.egg-info/
27 +.venv/
28 +venv/
29 +.pytest_cache/
30 +.mypy_cache/
31 +.ruff_cache/
32 +
33 +# Node / Next.js
34 +node_modules/
35 +web/.next/
36 +web/out/
37 +*.tsbuildinfo
38 +web/next-env.d.ts
39 +
40 +# LaTeX build artifacts (keep qwhpi.tex, references.bib, qwhpi.pdf)
41 +paper/*.aux
42 +paper/*.log
43 +paper/*.out
44 +paper/*.fls
45 +paper/*.fdb_latexmk
46 +paper/*.blg
47 +paper/*.bbl
48 +paper/*.toc
49 +
50 +# OS / editor / local
51 +.DS_Store
52 +.claude/
53 +.env
54 +.env.*
55 +*.local.json
added .pre-commit-config.yaml +22 −0
@@ -0,0 +1,22 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : .pre-commit-config.yaml
6 +# Purpose : Pre-commit hooks — mandatory header check, lint, type hygiene.
7 +# =============================================================================
8 +
9 +repos:
10 + - repo: local
11 + hooks:
12 + - id: qwhpi-headers
13 + name: QWHPI author-header check
14 + entry: python3 scripts/check_headers.py
15 + language: system
16 + pass_filenames: true
17 + types_or: [python, ts, tsx, javascript, css, sql, yaml, toml, shell]
18 + - repo: https://github.com/astral-sh/ruff-pre-commit
19 + rev: v0.6.9
20 + hooks:
21 + - id: ruff
22 + args: [--fix]
added CLAUDE.md +325 −0
@@ -0,0 +1,325 @@
1 +# CLAUDE.md — QWHPI Platform
2 +# Quebec Weekly Housing Price Index — Full Platform Build
3 +
4 +---
5 +
6 +## 0. MANDATORY FILE HEADERS — READ FIRST
7 +
8 +Every single file created in this project — Python, TypeScript, JavaScript, SQL, YAML, TOML, shell scripts, LaTeX, CSS, Dockerfiles, config files, notebooks — MUST begin with an author header.
9 +
10 +**Python / shell / YAML / TOML / Dockerfile:**
11 +
12 +```python
13 +# =============================================================================
14 +# QWHPI — Quebec Weekly Housing Price Index
15 +# Author : Simon-Pierre Boucher
16 +# Contact : contact@spboucher.ai
17 +# File : <relative/path/to/file.py>
18 +# Purpose : <one-line description>
19 +# =============================================================================
20 +```
21 +
22 +**TypeScript / JavaScript / TSX / CSS:**
23 +
24 +```ts
25 +/**
26 + * =============================================================================
27 + * QWHPI — Quebec Weekly Housing Price Index
28 + * Author : Simon-Pierre Boucher
29 + * Contact : contact@spboucher.ai
30 + * File : <relative/path/to/file.ts>
31 + * Purpose : <one-line description>
32 + * =============================================================================
33 + */
34 +```
35 +
36 +**SQL:**
37 +
38 +```sql
39 +-- =============================================================================
40 +-- QWHPI — Quebec Weekly Housing Price Index
41 +-- Author : Simon-Pierre Boucher
42 +-- Contact : contact@spboucher.ai
43 +-- File : <relative/path/to/file.sql>
44 +-- Purpose : <one-line description>
45 +-- =============================================================================
46 +```
47 +
48 +**Markdown / LaTeX:** include an equivalent comment or front-matter block at the top.
49 +
50 +Rules:
51 +- No file ships without this header. Add it at file creation time, not as a cleanup pass.
52 +- The `File` and `Purpose` lines must be accurate and kept up to date.
53 +- Add a pre-commit hook (`scripts/check_headers.py`) that fails CI if any tracked source file is missing the header.
54 +- API responses and the frontend footer must credit: **Simon-Pierre Boucher — contact@spboucher.ai**.
55 +
56 +---
57 +
58 +## 1. Mission
59 +
60 +Build a complete, production-grade **platform** around the Quebec Weekly Housing Price Index — not just a research pipeline. The platform has four layers:
61 +
62 +1. **Index Engine** — reproducible econometric pipeline producing quality-adjusted weekly indexes from `province_transactions.csv`.
63 +2. **Data Store** — versioned Parquet lake + relational database serving the API.
64 +3. **API** — FastAPI service exposing every published series with metadata, uncertainty, and reliability.
65 +4. **Dashboard** — interactive web frontend (charts, maps, comparisons, downloads).
66 +
67 +The index must NOT be a median-price tracker. It is a **hierarchical hedonic time-dummy index**: pooled hedonic estimation of characteristics, plus a latent weekly market state per `geography × property_type`, with partial pooling/shrinkage toward regional trends when weekly liquidity is thin.
68 +
69 +Target queryable observation:
70 +
71 +```
72 +Geography: Quebec City | Type: Condo | Week: 2024-05-06
73 +Index: 137.42 | 1w: +0.31% | 4w: +1.18% | YoY: +6.74%
74 +Representative value: $389,200 | Transactions: 47 | Reliability: A
75 +```
76 +
77 +---
78 +
79 +## 2. Dataset
80 +
81 +`province_transactions.csv` — ~745,119 transactions, 291 weeks, 2021-01-04 → 2026-07-27, 1,100+ cities.
82 +
83 +Columns: `id, date, amount, street, zipCode, city, lat, lng, propertyType, yearBuilt, floorArea, buildingType, previousValue, totalArValue, ownerType`
84 +
85 +Property types: `unifamilial` (~443k), `condo` (~100k), `plex` (~94k), `indéterminé` (~108k). `indéterminé` must never contaminate type-specific indexes; study what it represents and document it in an appendix.
86 +
87 +Several hedonic characteristics have missing values. Handle missingness explicitly — never `dropna()` the sample.
88 +
89 +---
90 +
91 +## 3. Repository Structure (Monorepo)
92 +
93 +```
94 +qwhpi-platform/
95 +├── CLAUDE.md
96 +├── README.md
97 +├── Makefile # one-command targets: make pipeline, make api, make web
98 +├── docker-compose.yml # db + api + web + scheduler
99 +├── .pre-commit-config.yaml
100 +│
101 +├── engine/ # Index Engine (Python)
102 +│ ├── pyproject.toml
103 +│ ├── src/qwhpi/
104 +│ │ ├── config.py ingest.py clean.py geography.py features.py
105 +│ │ ├── hedonic.py hierarchy.py state_space.py repeat_sales.py
106 +│ │ ├── index.py uncertainty.py reliability.py validation.py
107 +│ │ ├── seasonal.py nowcast.py vintages.py export.py
108 +│ ├── scripts/ # 01_profile … 10_build_report (ordered, idempotent)
109 +│ ├── tests/
110 +│ └── notebooks/exploratory_only/
111 +│
112 +├── data/
113 +│ ├── raw/ external/ interim/ processed/ # raw is immutable
114 +│
115 +├── db/
116 +│ ├── migrations/ # Alembic
117 +│ └── schema.sql
118 +│
119 +├── api/ # FastAPI service
120 +│ ├── pyproject.toml
121 +│ ├── app/
122 +│ │ ├── main.py deps.py models.py schemas.py
123 +│ │ ├── routers/ (index.py, geographies.py, liquidity.py, maps.py, meta.py)
124 +│ │ └── services/
125 +│ └── tests/
126 +│
127 +├── web/ # Frontend (Next.js + TypeScript)
128 +│ ├── package.json
129 +│ ├── app/ # routes: /, /explore, /compare, /map, /methodology, /api-docs
130 +│ ├── components/ lib/ styles/
131 +│ └── tests/
132 +│
133 +├── ops/
134 +│ ├── scheduler/ # weekly refresh job
135 +│ └── ci/
136 +│
137 +├── outputs/ # figures, tables, maps, reports
138 +├── paper/ # LaTeX methodology paper
139 +└── scripts/check_headers.py
140 +```
141 +
142 +Production logic lives in `engine/src`, `api/app`, `web/` — never scattered in notebooks.
143 +
144 +---
145 +
146 +## 4. Index Engine — Methodology (non-negotiable core)
147 +
148 +### 4.1 Weekly calendar
149 +Monday→Sunday weeks, each labeled by its Monday. Continuous grid — never drop zero-transaction weeks; flag them.
150 +
151 +### 4.2 Geography
152 +Hierarchy: Quebec → 17 administrative regions → municipality → (optional) local zone. Spatial-join `lat/lng` against an authoritative Quebec/StatCan boundary dataset (document source, URL, version, CRS). Persist the join once in `data/processed/`. Do not trust the `city` text field alone.
153 +
154 +### 4.3 Cleaning
155 +Reproducible pipeline; raw CSV untouched. Use `log(amount)`. Investigate zeros, implausible prices, non-arm's-length transfers, duplicates (flag with `duplicate_flag`/`duplicate_reason`, never silently delete). Produce a full exclusion table. Economically justified filters only — no unreported winsorizing.
156 +
157 +### 4.4 Hedonic model
158 +Pool 2021–present. Do NOT run independent weekly regressions.
159 +
160 +- **Model A (headline):** structural characteristics only — type, log(floorArea), age (spline/bins, data-driven vintage breakpoints), buildingType, fine location controls.
161 +- **Model B (robustness):** adds `totalArValue`, `previousValue` — beware valuation leakage; never the headline.
162 +
163 +Missingness: compare missing-indicator, conditional-median imputation, MICE, ML imputation. No future-price leakage. Choose stability + interpretability.
164 +
165 +### 4.5 Location control — critical
166 +Compare municipality FE, postal-code FE, H3/geohash grid, 2D splines on lat/lng, hierarchical spatial effects. Westmount ≠ rest of Montreal. Diagnose residual spatial autocorrelation; improve until it is acceptable.
167 +
168 +### 4.6 Hierarchical weekly state
169 +Latent path per `geography × property_type`:
170 +
171 +```
172 +Quebec trend + region deviation + municipality deviation + type deviation + local week deviation
173 +```
174 +
175 +Thin cells shrink toward parent trends; liquid cells (Montreal condo ~145/wk, Quebec City condo ~42/wk) are dominated by local data. Evaluate: Bayesian multilevel, empirical Bayes, mixed effects, state-space/Kalman (`μ_t = μ_{t-1} + η_t`), penalized splines. Benchmark stability and compute cost; do not choose complexity for its own sake. Also benchmark a two-stage architecture (structural hedonic residualization → hierarchical weekly time-series) against a unified time-dummy regression.
176 +
177 +### 4.7 Validation (mandatory)
178 +- Repeat-sales index (~85k repeated addresses; beware condo/plex address sharing) as directional check.
179 +- Downsampling experiment: use Montreal condos as lab; thin to 100/50/25/15/10/5 tx/week; measure RMSE, bias, volatility, turning-point accuracy, CI coverage. This empirically justifies liquidity tiers (starting hypotheses: ≥50 very strong, 20–49 strong, 10–19 shrinkage, 5–9 heavy shrinkage, <5 model-implied).
180 +- Composition-shock simulation: raw median must move, hedonic index must not.
181 +- Temporal + geographic holdouts; alternative specs (GAM, mixed effects, hierarchical Bayes, gradient-boosting residualization). ML (LightGBM/CatBoost/XGBoost) may estimate the cross-sectional component only — the time effect stays interpretable.
182 +- Weekly vs monthly comparison from the same methodology: quantify signal-to-noise, turning-point detection, revision magnitude.
183 +
184 +### 4.8 Outputs per series
185 +Base: 2021 average = 100 (keep raw latent log series). Both `index` (raw weekly) and `index_smoothed` (one-sided real-time version required alongside any two-sided smoother). Representative dollar value per segment from a documented property basket. Growth: 1w, 4w, 13w, 26w, YoY, YTD, since-2021. 95% CIs (cluster/block bootstrap or posterior). Reliability grades A–E from n, effective N, SE, shrinkage weight, missingness. NSA always; SA only if stable seasonality is demonstrated. `is_partial_week` nowcast handling for the latest week. Vintage framework: `first_release`, `current_vintage`, `revision`.
186 +
187 +### 4.9 Canonical dataset
188 +
189 +`data/processed/qwhpi_weekly.parquet`:
190 +
191 +```
192 +week, geography_level, geography_id, geography_name, property_type,
193 +index, index_smoothed, representative_value,
194 +transactions, effective_sample_size,
195 +weekly_pct, four_week_pct, thirteen_week_pct, yoy_pct,
196 +lower_95, upper_95, reliability_grade, shrinkage_weight,
197 +is_partial_week, model_version, data_vintage
198 +```
199 +
200 +This single table powers the database, API, and dashboard.
201 +
202 +### 4.10 Index families
203 +- **QWHPI-QC** (province), **QWHPI-REG** (17 regions), **QWHPI-CITY** (Montréal, Québec, Laval, Gatineau, Longueuil, Sherbrooke, Trois-Rivières, Saguenay, Lévis, Drummondville — as liquidity permits), **QWHPI-TYPE** (All / Single-family / Condo / Plex per supported geography).
204 +- Secondary module: **Assessment Gap Index** from `amount / totalArValue` — separate concept, never mixed with the price index.
205 +- Coverage matrix declaring, per municipality × type: published / conditional / not published.
206 +
207 +---
208 +
209 +## 5. Data Store
210 +
211 +- Parquet lake in `data/processed/` (partitioned by geography_level / year) is the source of truth.
212 +- PostgreSQL (via docker-compose) serves the API: tables `series`, `observations`, `geographies`, `liquidity`, `vintages`, `model_runs`. Alembic migrations in `db/migrations/`.
213 +- Loader `engine/src/qwhpi/export.py` upserts each pipeline run into the DB with `model_version` and `data_vintage`.
214 +- Never let the API read raw CSVs.
215 +
216 +---
217 +
218 +## 6. API (FastAPI)
219 +
220 +Endpoints:
221 +
222 +```
223 +GET /v1/index?geography=quebec-city&type=condo&from=2021-01-04&to=latest
224 +GET /v1/index/latest?geography=...&type=...
225 +GET /v1/geographies # hierarchy + coverage matrix
226 +GET /v1/liquidity?geography=...&type=...
227 +GET /v1/compare?series=montreal:condo,quebec-city:condo
228 +GET /v1/map?metric=yoy&level=region
229 +GET /v1/vintages?geography=...&type=...&week=...
230 +GET /v1/meta # model_version, data_vintage, methodology link, author credit
231 +GET /health
232 +```
233 +
234 +Sample response:
235 +
236 +```json
237 +{
238 + "geography": "Quebec City",
239 + "property_type": "condo",
240 + "frequency": "weekly",
241 + "latest_index": 137.42,
242 + "representative_value": 389200,
243 + "weekly_change": 0.31,
244 + "yoy_change": 6.74,
245 + "transactions": 47,
246 + "reliability": "A",
247 + "lower_95": 135.9,
248 + "upper_95": 138.9,
249 + "is_partial_week": false,
250 + "author": "Simon-Pierre Boucher",
251 + "contact": "contact@spboucher.ai"
252 +}
253 +```
254 +
255 +Requirements: Pydantic schemas, OpenAPI docs at `/docs`, pagination for full histories, CSV/JSON export toggle, ETag caching, rate limiting, CORS for the web app, tests for every router. Reliability and CIs are always returned — the API never hides uncertainty.
256 +
257 +---
258 +
259 +## 7. Dashboard (Next.js + TypeScript)
260 +
261 +Pages:
262 +- **/** — headline: Quebec aggregate, latest week, YoY, sparkline, top movers.
263 +- **/explore** — series picker (geography tree × property type), weekly chart with CI band, raw-median overlay toggle, smoothed toggle, growth-horizon selector, transaction-volume subchart, reliability badge, CSV download.
264 +- **/compare** — multi-series comparison (e.g., condo across major cities), rebasing tool.
265 +- **/map** — choropleth of the 17 regions (YoY, 13w, index level, assessment gap); H3 heat map for major urban areas where density allows.
266 +- **/methodology** — rendered methodology summary; link to the paper.
267 +- **/api-docs** — link/embed of OpenAPI docs.
268 +
269 +Requirements: charting via a solid library (e.g., ECharts/Recharts/Plotly), responsive, dark/light, consistent color config shared with engine figures, loading/empty/error states, low-reliability series visually flagged (never hidden), footer credit "Simon-Pierre Boucher — contact@spboucher.ai".
270 +
271 +---
272 +
273 +## 8. Automation & Ops
274 +
275 +- `make pipeline` → full engine run; `make refresh` → incremental update when new transactions are appended; `make api` / `make web` / `make up` (docker-compose).
276 +- Scheduler (`ops/scheduler/`) runs the weekly refresh: ingest new rows → append-only clean → re-estimate weekly states → write new `data_vintage` → load DB → invalidate API cache.
277 +- CI: lint, type-check (mypy + tsc), tests, **header check**, small-sample smoke run of the pipeline.
278 +- Structured logging + run manifest (input hash, row counts, model version, timings) for every pipeline execution.
279 +
280 +---
281 +
282 +## 9. Engineering Standards
283 +
284 +- Python: pandas/polars, numpy, statsmodels, scikit-learn, geopandas, pyarrow; PyMC/CmdStanPy or statsmodels state-space where justified. Vectorized; no per-transaction Python loops; cache spatial joins, features, model matrices; Parquet everywhere.
285 +- Diagnostics for every production model: residual distribution, heteroskedasticity, residuals-vs-fitted, temporal/spatial residual patterns, coefficient stability, weekly SEs, effective N.
286 +- Model selection optimizes interpretability + stability + low bias + responsiveness + calibrated uncertainty + reproducibility — NOT transaction-level RMSE alone. This is economic measurement, not a prediction contest.
287 +- Core principle everywhere: separate **price movement** from **composition movement**. Never present a weekly average price as a price level. Never fake precision — expose n, effective N, CI, shrinkage weight, reliability.
288 +
289 +---
290 +
291 +## 10. Research Outputs
292 +
293 +- `outputs/figures/`: Quebec aggregate; region comparison; big-4 city comparison; condo/single-family/plex by city; raw median vs hedonic; volumes; CI bands; reliability map; repeat-sales vs hedonic; downsampling stability; assessment-gap evolution; YoY appreciation map. One shared plotting config.
294 +- `outputs/tables/`: data_summary, missingness, exclusions, weekly_liquidity, coverage, model_comparison, index_latest, repeat_sales_comparison, downsampling_results, revision_statistics.
295 +- `paper/`: full LaTeX paper — *A High-Frequency Hedonic Housing Price Index for Quebec* — real, researched citations only (hedonic indexes, repeat sales, Case-Shiller, high-frequency measurement, hierarchical indexes, state-space indexes, spatial hedonics, index-number theory). No fabricated references.
296 +- Research questions to actually investigate: regional heterogeneity post-2021; synchronization of turning points; Montreal lead/lag; type-level response speeds; weekly-vs-monthly information gain; minimum transactions for reliability; value of hierarchical pooling; assessment lag dynamics.
297 +
298 +---
299 +
300 +## 11. Execution Order
301 +
302 +1. **Audit** — schema, descriptives, missingness, outliers, coverage, weekly liquidity matrices. No index yet.
303 +2. **Geography** — boundary acquisition, spatial join, validation, persistence.
304 +3. **Cleaning** — research sample with documented exclusions.
305 +4. **Baseline index** — transparent pooled hedonic time-dummy for Quebec + Montreal/Québec/Laval/Gatineau × All/Unifamilial/Condo/Plex.
306 +5. **Hierarchical weekly model** — shrinkage for thin segments.
307 +6. **Validation** — repeat sales, downsampling, monthly comparison, alternative specs.
308 +7. **Scale** — all supported region and municipality series; coverage matrix.
309 +8. **Data store + API** — DB schema, loader, FastAPI, tests.
310 +9. **Dashboard** — all pages against the live API.
311 +10. **Automation + research outputs** — scheduler, CI, figures, tables, paper.
312 +
313 +Work autonomously: inspect data → research accepted methodology → implement the most defensible option → document → benchmark alternatives → preserve reversibility. Do not settle for the first model that runs. Do not stop to ask permission on minor modeling choices.
314 +
315 +---
316 +
317 +## 12. Definition of Success
318 +
319 +One reproducible command (`make up` after `make pipeline`) yields:
320 +
321 +- a trustworthy weekly, quality-adjusted, hierarchically pooled index for every statistically defensible `geography × property_type`, 2021 → latest week, with CIs, reliability grades, representative dollar values, validation, and vintages;
322 +- a documented API serving it;
323 +- an interactive dashboard displaying it;
324 +- publication-quality figures, tables, and a methodology paper;
325 +- **every file in the repository headed with: Simon-Pierre Boucher — contact@spboucher.ai.**
added Makefile +63 −0
@@ -0,0 +1,63 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : Makefile
6 +# Purpose : One-command targets — pipeline, refresh, api, web, docker, tests.
7 +# =============================================================================
8 +
9 +PY := python3
10 +ENGINE := engine/scripts
11 +
12 +.PHONY: pipeline refresh api web up down test lint headers figures paper clean
13 +
14 +## Full engine run: audit -> geography -> clean -> baseline -> hierarchical
15 +## -> validation -> canonical dataset (+ DB load when DATABASE_URL is set)
16 +pipeline:
17 + $(PY) $(ENGINE)/01_profile.py
18 + $(PY) $(ENGINE)/02_geography.py
19 + $(PY) $(ENGINE)/03_clean.py
20 + $(PY) $(ENGINE)/04_baseline.py
21 + $(PY) $(ENGINE)/05_hierarchical.py
22 + $(PY) $(ENGINE)/06_validation.py
23 + $(PY) $(ENGINE)/07_canonical.py
24 + $(PY) $(ENGINE)/09_monthly.py
25 + $(PY) $(ENGINE)/10_monthly_validation.py
26 + $(PY) $(ENGINE)/11_monthly_canonical.py
27 + $(PY) -c "import sys; sys.path.insert(0,'engine/src'); from qwhpi.export import load_database; load_database()"
28 +
29 +## Incremental refresh (new rows appended to the raw CSV) — monthly headline
30 +refresh:
31 + $(PY) $(ENGINE)/03_clean.py
32 + $(PY) $(ENGINE)/09_monthly.py
33 + $(PY) $(ENGINE)/11_monthly_canonical.py
34 + $(PY) -c "import sys; sys.path.insert(0,'engine/src'); from qwhpi.export import load_database; load_database()"
35 +
36 +api:
37 + cd api && $(PY) -m uvicorn app.main:app --host 0.0.0.0 --port 8080
38 +
39 +web:
40 + cd web && npm run build && npm run start
41 +
42 +up:
43 + docker compose up --build -d
44 +
45 +down:
46 + docker compose down
47 +
48 +test:
49 + cd api && $(PY) -m pytest tests -q
50 + cd engine && $(PY) -m pytest tests -q
51 +
52 +headers:
53 + $(PY) scripts/check_headers.py
54 +
55 +figures:
56 + $(PY) $(ENGINE)/08_figures.py
57 +
58 +paper:
59 + cd paper && latexmk -pdf -quiet qwhpi.tex || pdflatex qwhpi.tex
60 +
61 +lint:
62 + $(PY) scripts/check_headers.py
63 + cd web && npx tsc --noEmit
added README.md +501 −0
@@ -0,0 +1,501 @@
1 +<!--
2 +=============================================================================
3 +QWHPI — Quebec Weekly Housing Price Index
4 +Author : Simon-Pierre Boucher
5 +Contact : contact@spboucher.ai
6 +File : README.md
7 +Purpose : Full platform documentation — methodology, architecture, API,
8 + dashboard, data policy, reproducibility.
9 +=============================================================================
10 +-->
11 +
12 +<div align="center">
13 +
14 +# 🏠 QHPI — Quebec Housing Price Index
15 +
16 +### Quality-adjusted, hierarchically pooled housing price indexes for Quebec
17 +
18 +*Province · 17 administrative regions · major municipalities × {all, unifamilial, condo, plex}*
19 +
20 +[![Live](https://img.shields.io/badge/live-www.indexqc.house-2ea44f?style=flat-square&logo=googlechrome&logoColor=white)](https://www.indexqc.house)
21 +[![Model](https://img.shields.io/badge/model-v2.1.0-blue?style=flat-square)](#-methodology)
22 +[![Frequency](https://img.shields.io/badge/frequency-monthly%20(headline)%20%2B%20weekly%20(research)-8a2be2?style=flat-square)](#-methodology)
23 +[![Coverage](https://img.shields.io/badge/coverage-2021--01%20%E2%86%92%20present-informational?style=flat-square)](#-index-families--coverage)
24 +[![Series](https://img.shields.io/badge/published%20cells-81%20(50%20direct%20%2B%2031%20hierarchical)-orange?style=flat-square)](#-methodology)
25 +[![Transactions](https://img.shields.io/badge/estimation%20sample-~720k%20transactions-lightgrey?style=flat-square)](#-data-policy--no-microdata-in-this-repo)
26 +
27 +[![Python](https://img.shields.io/badge/Python-3.11+-3776AB?style=flat-square&logo=python&logoColor=white)](engine/pyproject.toml)
28 +[![FastAPI](https://img.shields.io/badge/API-FastAPI-009688?style=flat-square&logo=fastapi&logoColor=white)](api/)
29 +[![Next.js](https://img.shields.io/badge/Web-Next.js%20%2B%20TypeScript-000000?style=flat-square&logo=nextdotjs&logoColor=white)](web/)
30 +[![PostgreSQL](https://img.shields.io/badge/DB-PostgreSQL-4169E1?style=flat-square&logo=postgresql&logoColor=white)](db/)
31 +[![Docker](https://img.shields.io/badge/Ops-docker--compose-2496ED?style=flat-square&logo=docker&logoColor=white)](docker-compose.yml)
32 +[![LaTeX](https://img.shields.io/badge/Paper-LaTeX-008080?style=flat-square&logo=latex&logoColor=white)](paper/)
33 +
34 +[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-111827?style=flat-square)](mailto:contact@spboucher.ai)
35 +[![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-D14836?style=flat-square&logo=maildotru&logoColor=white)](mailto:contact@spboucher.ai)
36 +
37 +**Author : Simon-Pierre Boucher — [contact@spboucher.ai](mailto:contact@spboucher.ai)**
38 +
39 +</div>
40 +
41 +---
42 +
43 +## Table of contents
44 +
45 +- [What this is](#-what-this-is)
46 +- [Key numbers](#-key-numbers)
47 +- [Data policy — no microdata in this repo](#-data-policy--no-microdata-in-this-repo)
48 +- [Platform architecture](#-platform-architecture)
49 +- [Methodology](#-methodology)
50 +- [Validation](#-validation)
51 +- [Index families & coverage](#-index-families--coverage)
52 +- [Canonical dataset](#-canonical-dataset)
53 +- [API](#-api)
54 +- [Dashboard](#-dashboard)
55 +- [Repository map](#-repository-map)
56 +- [Quickstart](#-quickstart)
57 +- [Engine pipeline](#-engine-pipeline)
58 +- [Engineering standards](#-engineering-standards)
59 +- [Research outputs & paper](#-research-outputs--paper)
60 +- [Reproducibility](#-reproducibility)
61 +- [Author & license](#-author--license)
62 +
63 +---
64 +
65 +## 🎯 What this is
66 +
67 +QHPI is a **production-grade economic-measurement platform**, not a median-price
68 +tracker. It separates **price movement** from **composition movement**: a week
69 +where only mansions sell must not register as a price increase.
70 +
71 +The platform has four layers:
72 +
73 +| Layer | What it does | Where |
74 +|---|---|---|
75 +| **Index Engine** | Reproducible econometric pipeline: cleaning → hedonic estimation → hierarchical pooling → validation → canonical Parquet | [`engine/`](engine/) |
76 +| **Data Store** | Versioned Parquet lake (aggregated series only) + PostgreSQL serving layer | [`data/processed/`](data/processed/), [`db/`](db/) |
77 +| **API** | FastAPI service exposing every published series with uncertainty, reliability and vintages | [`api/`](api/) |
78 +| **Dashboard** | Interactive Next.js frontend — explore, compare, map, PDF reports, live at [www.indexqc.house](https://www.indexqc.house) | [`web/`](web/) |
79 +
80 +Every published observation looks like this:
81 +
82 +```
83 +Geography: Quebec City | Type: Condo | Period: 2026-06
84 +Index: 163.5 (2021 = 100) | 1m: +0.4% | YoY: +8.1%
85 +Representative value: $389,200 | Transactions: 47 | Reliability: A
86 +95% CI: [160.9, 166.1] | Vintage: 2026-08-08 | Model: 2.1.0
87 +```
88 +
89 +Uncertainty is **never hidden**: every row carries a 95% confidence interval,
90 +an A–E reliability grade, effective sample size and shrinkage weight.
91 +
92 +---
93 +
94 +## 📊 Key numbers
95 +
96 +![Transactions](https://img.shields.io/badge/raw%20transactions-~745k-blue?style=for-the-badge)
97 +![Period](https://img.shields.io/badge/period-2021→2026-blueviolet?style=for-the-badge)
98 +![Cities](https://img.shields.io/badge/municipalities-1100%2B-orange?style=for-the-badge)
99 +![Regions](https://img.shields.io/badge/regions-17-green?style=for-the-badge)
100 +
101 +| Metric | Value |
102 +|---|---|
103 +| Raw transactions ingested | ~745,000 (2021-01 → present) |
104 +| Estimation sample after documented exclusions | ~720,000 |
105 +| Geographic units | Province + 17 administrative regions + major municipalities |
106 +| Property types | `unifamilial`, `condo`, `plex` (+ `all` composite) |
107 +| Published cells | 81 — 50 **direct** liquid cells + 31 **hierarchical** thin cells |
108 +| Headline frequency | **Monthly** (v2.1) — weekly v1 retained as a research module |
109 +| Base | 2021 average = 100 |
110 +| Reliability grades | **A** ≥150 tx/mo · **B** ≥75 · **C** ≥40 · **D** ≥15 · **E** <15 (model-implied) |
111 +| Growth horizons | 1m, 3m, 6m, YoY (weekly module: 1w, 4w, 13w, 26w, YoY) |
112 +| Vintages | `first_release` preserved; every revision queryable |
113 +
114 +---
115 +
116 +## 🔒 Data policy — no microdata in this repo
117 +
118 +> **This repository contains NO individual transaction data.**
119 +
120 +| Category | In repo? | Detail |
121 +|---|---|---|
122 +| Raw transaction CSV (`data/raw/`) | ❌ **excluded** | Individual sale records — never versioned |
123 +| Transaction-level intermediates (`data/interim/`, `transactions_clean.parquet`, `geo_join.parquet`) | ❌ **excluded** | Anything with one row per transaction |
124 +| Cadastral boundary files (`SDA.gpkg`) | ❌ excluded (size) | Source, URL, version and CRS documented in [`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md) |
125 +| **Aggregated published series** | ✅ included | `qhpi_monthly.parquet`, `qwhpi_weekly.parquet`, liquidity, coverage matrices, assessment gap, first releases, vintages |
126 +| **Aggregate tables & figures** (`outputs/`) | ✅ included | Audit counts, model comparisons, validation results, publication figures |
127 +
128 +The [`.gitignore`](.gitignore) enforces this at the repo boundary. Aggregates
129 +are cell-level (`geography × type × period`) with transaction **counts**, never
130 +transaction rows. To rerun the full pipeline you must supply your own
131 +`data/raw/province_transactions.csv` (see [Reproducibility](#-reproducibility)).
132 +
133 +---
134 +
135 +## 🏗 Platform architecture
136 +
137 +```
138 + ┌─────────────────────────────────────────────┐
139 + │ province_transactions.csv │
140 + │ (local only — NOT in this repo) │
141 + └──────────────────────┬──────────────────────┘
142 + ▼
143 + ┌───────────────────────────── ENGINE (Python) ─────────────────────────────┐
144 + │ 01 profile → 02 geography (spatial join, SDA boundaries) → 03 clean │
145 + │ → 04–08 weekly research module (baseline, hierarchical, validation) │
146 + │ → 09 monthly estimation (rolling time-dummy, Huber-IRLS) │
147 + │ → 10 monthly validation → 11 canonical parquet │
148 + └──────────────────────────────────┬────────────────────────────────────────┘
149 + ▼
150 + data/processed/qhpi_monthly.parquet ← source of truth
151 + (+ liquidity, coverage, vintages, assessment gap)
152 + ▼
153 + ┌────────────── PostgreSQL (docker-compose, Alembic) ──────────────┐
154 + │ series · observations · geographies · liquidity · vintages │
155 + └──────────────────────────────┬────────────────────────────────────┘
156 + ▼
157 + FastAPI /v1/* (uncertainty always returned)
158 + ▼
159 + Next.js dashboard — www.indexqc.house
160 + (explore · compare · map · PDF reports · methodology)
161 +```
162 +
163 +Weekly refresh: `ops/scheduler/` ingests new rows → append-only clean →
164 +re-estimation → new `data_vintage` → DB load → API cache invalidation.
165 +
166 +---
167 +
168 +## 🧪 Methodology
169 +
170 +The index is a **robust hedonic time-dummy index with hierarchical pooling** —
171 +three estimation regimes arbitrated by empirical validation (repeat sales +
172 +stratified matched-cell medians), not by convenience:
173 +
174 +### 1. Province × type backbone — rolling time-dummy (RTD)
175 +
176 +- Pooled robust regressions (`log(price)` on structural characteristics +
177 + fine location fixed effects + period dummies) over **13-month rolling
178 + windows** with **Huber-IRLS** weighting.
179 +- Windows are combined by **mean splice**, making published history
180 + **revision-free** by construction.
181 +- Coefficient drift is monitored and published
182 + (`outputs/tables/rtd_coefficient_drift.csv`) — e.g. the floor-area
183 + elasticity moves 0.556 → 0.586 across windows, which a fixed pooled model
184 + would have hidden.
185 +
186 +### 2. Liquid cells (median ≥ 40 tx/month) — direct local estimation
187 +
188 +Each of the **50 liquid `geography × type` cells** gets its **own robust
189 +time-dummy regression** (own hedonic coefficients, own FSA fixed effects),
190 +lightly smoothed by a state-space filter. This was arbitrated against
191 +deviation-from-pooled-surface modelling, which **compressed genuine local
192 +divergence** — Quebec City condos moved ~+55% (confirmed independently by
193 +repeat sales *and* stratified matched-cell medians) while pooled deviations
194 +reported only +42%.
195 +
196 +### 3. Thin cells — hierarchical Kalman shrinkage
197 +
198 +The remaining **31 thin cells** follow a latent random-walk **deviation from
199 +their parent's published path**, estimated by Kalman filter/smoother with
200 +observation noise σ²/nₜ. Thin months shrink toward the parent trend; the
201 +shrinkage weight is published on every row.
202 +
203 +### Hedonic specification (headline = structural only)
204 +
205 +- `log(amount)` on `log(floorArea)` (+ time-stable missingness indicator),
206 + age from `yearBuilt` via **conditional-median imputation** (its missingness
207 + drifts 15% → 2% over the sample, so a "missing" bin would leak time),
208 + and fine location fixed effects (FSA).
209 +- **`buildingType` is banned from all hedonic models.** The provider
210 + backfilled it in region-staggered waves (most regions 2022-01, Montreal
211 + 2023-01); the time-correlated missingness fabricated a −38 log-point cliff
212 + in local Montreal indexes. The ban is documented in
213 + [`engine/src/qwhpi/features.py`](engine/src/qwhpi/features.py).
214 +- Assessment values (`totalArValue`, `previousValue`) never enter the
215 + headline (valuation leakage); they power the separate
216 + **Assessment Gap Index** (`amount / totalArValue`).
217 +
218 +### Cleaning
219 +
220 +Raw data is immutable; every exclusion is economically justified and counted
221 +in [`outputs/tables/exclusions.csv`](outputs/tables/exclusions.csv)
222 +(assessment-ratio band, exact duplicates, non-arm's-length transfers, …).
223 +Duplicates are **flagged, never silently deleted**. The `indéterminé`
224 +property type never contaminates type-specific indexes.
225 +
226 +### Geography
227 +
228 +Transactions are spatially joined (`lat/lng`) against the authoritative
229 +Quebec **SDA cadastral boundaries** (source, version and CRS documented in
230 +[`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md)) — the free-text
231 +`city` field alone is not trusted.
232 +
233 +---
234 +
235 +## ✅ Validation
236 +
237 +Every claim below has a table or figure in [`outputs/`](outputs/):
238 +
239 +| Check | Result |
240 +|---|---|
241 +| **Repeat-sales benchmark** (BMN, ~85k repeated addresses) | June 2026, 2021=100: quebec/condo **132.0** vs RS 132 (exact); quebec-city/condo **163.5** vs RS 162; montreal/condo **121.6** vs RS 122; montreal/unifamilial **126.2** between stratified 120 and RS 135 (RS renovation bias on older stock) |
242 +| **Composition-shock simulation** | Mix shock moves the raw median **+11.7%**; the hedonic index moves **−0.7%** — the index measures prices, not composition |
243 +| **Downsampling experiment** | Montreal condos thinned to 100/50/25/15/10/5 tx/period; RMSE, bias, volatility, turning points and CI coverage justify the A–E liquidity tiers empirically |
244 +| **Weekly vs monthly** | Same methodology at both frequencies; signal-to-noise, turning-point detection and revision magnitude quantified in `outputs/tables/weekly_vs_monthly.csv` |
245 +| **Seasonality** | NSA published; no stable seasonality demonstrated (`seasonality_tests.csv`) |
246 +| **Alternative specs** | Hierarchical variants, mixed effects, ML residualization benchmarked in `estimation_methods.csv` — chosen model optimizes interpretability + stability, not transaction-level RMSE |
247 +
248 +Full write-ups: [`outputs/reports/`](outputs/reports/) (audit, geography,
249 +cleaning, hierarchical, validation v1 & v2, methodology research).
250 +
251 +---
252 +
253 +## 🗺 Index families & coverage
254 +
255 +| Family | Scope |
256 +|---|---|
257 +| **QHPI-QC** | Province of Quebec |
258 +| **QHPI-REG** | 17 administrative regions |
259 +| **QHPI-CITY** | Montréal, Québec, Laval, Gatineau, Longueuil, Sherbrooke, Trois-Rivières, Saguenay, Lévis, Drummondville, … as liquidity permits |
260 +| **QHPI-TYPE** | All / Unifamilial / Condo / Plex per supported geography |
261 +| **Assessment Gap** | `amount / totalArValue` — separate concept, never mixed with the price index |
262 +
263 +The **coverage matrix**
264 +([`data/processed/coverage_matrix_monthly.parquet`](data/processed/),
265 +[`outputs/tables/coverage_monthly.csv`](outputs/tables/coverage_monthly.csv))
266 +declares per `municipality × type`: published / conditional / not published.
267 +
268 +---
269 +
270 +## 🗃 Canonical dataset
271 +
272 +[`data/processed/qhpi_monthly.parquet`](data/processed/qhpi_monthly.parquet)
273 +(7,504 rows) powers the DB, API and dashboard. Weekly research module:
274 +[`data/processed/qwhpi_weekly.parquet`](data/processed/qwhpi_weekly.parquet)
275 +(32,592 rows).
276 +
277 +```
278 +period, geography_level, geography_id, geography_name, property_type,
279 +index, index_smoothed, representative_value,
280 +transactions, effective_sample_size,
281 +monthly_pct, three_month_pct, six_month_pct, yoy_pct,
282 +lower_95, upper_95, reliability_grade, shrinkage_weight,
283 +is_partial_month, model_version, data_vintage
284 +```
285 +
286 +- `index` = raw estimate, `index_smoothed` = **one-sided** (real-time safe)
287 + smoother — no look-ahead in the published real-time series.
288 +- `representative_value` = dollar value of a documented, fixed property
289 + basket per segment.
290 +- Vintage framework: first releases preserved under
291 + `data/processed/vintages*/`, revisions queryable via the API.
292 +
293 +---
294 +
295 +## 🔌 API
296 +
297 +FastAPI service (OpenAPI docs at `/docs`), CSV/JSON export toggle, ETag
298 +caching, pagination, rate limiting, CORS. **Reliability and CIs are always
299 +returned.**
300 +
301 +| Endpoint | Purpose |
302 +|---|---|
303 +| `GET /v1/index?geography=quebec-city&type=condo&from=2021-01&to=latest` | Full series |
304 +| `GET /v1/index/latest?geography=…&type=…` | Latest observation |
305 +| `GET /v1/geographies` · `GET /v1/geographies/coverage` | Hierarchy + coverage matrix |
306 +| `GET /v1/liquidity?geography=…&type=…` | Transaction counts & effective N |
307 +| `GET /v1/compare?series=montreal:condo,quebec-city:condo` | Multi-series comparison |
308 +| `GET /v1/map?metric=yoy&level=region` | Choropleth payloads (+ `period` for time-lapse) |
309 +| `GET /v1/stats` · `GET /v1/stats/overview` | Peak/drawdown/CAGR/volatility/momentum/rank, heatmap rows |
310 +| `GET /v1/report?series=…` · `GET /v1/report/market` | Publication-grade PDF reports (matplotlib) |
311 +| `GET /v1/vintages?geography=…&type=…` | Revision history |
312 +| `GET /v1/meta` | `model_version`, `data_vintage`, methodology link, author credit |
313 +| `GET /v1/health` | Liveness |
314 +
315 +Sample response:
316 +
317 +```json
318 +{
319 + "geography": "Quebec City",
320 + "property_type": "condo",
321 + "frequency": "monthly",
322 + "latest_index": 163.5,
323 + "representative_value": 389200,
324 + "monthly_change": 0.4,
325 + "yoy_change": 8.1,
326 + "transactions": 47,
327 + "reliability": "A",
328 + "lower_95": 160.9,
329 + "upper_95": 166.1,
330 + "is_partial_month": false,
331 + "author": "Simon-Pierre Boucher",
332 + "contact": "contact@spboucher.ai"
333 +}
334 +```
335 +
336 +---
337 +
338 +## 💻 Dashboard
339 +
340 +Live at **[www.indexqc.house](https://www.indexqc.house)** — Next.js +
341 +TypeScript, dark/light themes, fully responsive (audited at 390/360 px),
342 +`prefers-reduced-motion` respected, zero-dependency custom hero chart,
343 +⌘K command palette over all 112 series.
344 +
345 +| Page | Contents |
346 +|---|---|
347 +| `/` | Market pulse: heat-shaded region table with sparklines, 12 metric tiles, ticker tape, top movers, market-report PDF |
348 +| `/explore` | Series picker, CI band, raw-median & smoothed toggles, 1Y/3Y/5Y/YTD/MAX ranges, Bank-of-Canada event annotations, brush zoom, PNG/CSV export, sortable data table, shareable URLs |
349 +| `/compare` | Multi-series with rebasing tool, toggleable legend chips, comparison PDF |
350 +| `/map` | Region choropleth (YoY, 3m, level, assessment gap) with monthly slider + ▶ time-lapse |
351 +| `/methodology` | Rendered methodology + link to the paper |
352 +| `/api-docs` | Live playground (run requests in-page) + copyable curl examples |
353 +
354 +Low-reliability series are **visually flagged, never hidden**. Footer credit:
355 +*Simon-Pierre Boucher — contact@spboucher.ai*.
356 +
357 +---
358 +
359 +## 📁 Repository map
360 +
361 +```
362 +├── engine/ # Index engine (Python)
363 +│ ├── src/qwhpi/ # ingest · clean · geography · features · hedonic
364 +│ │ # rtd · hierarchy · state_space · repeat_sales
365 +│ │ # index · uncertainty · seasonal · nowcast
366 +│ │ # vintages · export · plotting
367 +│ ├── scripts/ # 01_profile … 11_monthly_canonical (ordered, idempotent)
368 +│ └── tests/ # smoke pipeline, features/clean, state-space
369 +├── data/
370 +│ ├── external/ # boundary-source documentation (files excluded by size)
371 +│ └── processed/ # ✅ aggregated series only (see Data policy)
372 +├── db/ # PostgreSQL schema + Alembic migrations
373 +├── api/ # FastAPI app: routers/ services/ tests/
374 +├── web/ # Next.js dashboard: app/ components/ lib/ styles/
375 +├── ops/ # weekly-refresh scheduler + CI pipeline
376 +├── outputs/
377 +│ ├── figures/ # publication figures (shared plotting config)
378 +│ ├── tables/ # 35+ audit/validation/model tables (aggregates)
379 +│ └── reports/ # markdown reports for every pipeline stage
380 +├── paper/ # LaTeX methodology paper (+ compiled PDF)
381 +├── scripts/check_headers.py # CI gate: every file carries the author header
382 +├── Makefile # one-command targets
383 +└── docker-compose.yml # postgres + api + web + scheduler
384 +```
385 +
386 +---
387 +
388 +## 🚀 Quickstart
389 +
390 +> Requires the raw transaction CSV (not distributed — see
391 +> [Data policy](#-data-policy--no-microdata-in-this-repo)) for `make pipeline`.
392 +> The API and dashboard run off the **included aggregated Parquet** without it.
393 +
394 +```bash
395 +# Engine (full run ~ audit → geography → clean → estimate → validate → canonical)
396 +make pipeline
397 +
398 +# Incremental monthly refresh (clean → estimate → canonical, ~2 min)
399 +make refresh
400 +
401 +# Services
402 +make api # FastAPI on :8080 — docs at /docs
403 +make web # Next.js dashboard on :3000
404 +make up # docker compose: postgres + api + web + scheduler
405 +make down
406 +
407 +# Quality
408 +make test # engine + api test suites
409 +make lint # ruff + mypy + tsc
410 +make headers # author-header check (also a pre-commit hook + CI gate)
411 +make figures # publication figures → outputs/figures/
412 +make paper # compile the LaTeX paper
413 +```
414 +
415 +---
416 +
417 +## ⚙️ Engine pipeline
418 +
419 +| Script | Stage |
420 +|---|---|
421 +| `01_profile.py` | Data audit: schema, descriptives, missingness, sentinel values, liquidity matrices |
422 +| `02_geography.py` | SDA boundary spatial join, validation, persistence |
423 +| `03_clean.py` | Research sample with fully documented exclusions |
424 +| `04_baseline.py` | Weekly pooled hedonic time-dummy baseline *(research module)* |
425 +| `05_hierarchical.py` | Weekly hierarchical state-space shrinkage *(research module)* |
426 +| `06_validation.py` | Repeat sales, downsampling, composition shock *(weekly)* |
427 +| `07_canonical.py` | Weekly canonical Parquet *(research module)* |
428 +| `08_figures.py` | Publication figures |
429 +| `09_monthly.py` | **Headline monthly estimation** (RTD + direct cells + Kalman) |
430 +| `10_monthly_validation.py` | Monthly validation suite |
431 +| `11_monthly_canonical.py` | **Canonical `qhpi_monthly.parquet`** + vintage write |
432 +
433 +Every run emits a structured **run manifest** (input hash, row counts, model
434 +version, timings).
435 +
436 +---
437 +
438 +## 📐 Engineering standards
439 +
440 +- **Every file** starts with the author header — enforced by
441 + [`scripts/check_headers.py`](scripts/check_headers.py) in pre-commit **and** CI.
442 +- Vectorized pandas/polars + numpy; no per-transaction Python loops; spatial
443 + joins, features and model matrices cached as Parquet.
444 +- Diagnostics for every production model: residual distribution,
445 + heteroskedasticity, temporal/spatial residual patterns, coefficient
446 + stability, effective N.
447 +- Model selection optimizes **interpretability + stability + calibrated
448 + uncertainty + reproducibility** — never transaction-level RMSE alone.
449 +- CI (`ops/ci/ci.yml`): lint, mypy + tsc, tests, header check, small-sample
450 + smoke run of the pipeline.
451 +
452 +---
453 +
454 +## 📄 Research outputs & paper
455 +
456 +- **Figures** ([`outputs/figures/`](outputs/figures/)): province aggregate,
457 + region comparison, big-4 cities, condo by city, raw median vs hedonic,
458 + volumes, reliability matrix, downsampling stability, assessment gap,
459 + YoY appreciation map.
460 +- **Tables** ([`outputs/tables/`](outputs/tables/)): audit suite, exclusions,
461 + liquidity, coverage, model comparison, repeat-sales comparison,
462 + downsampling, seasonality, coefficient drift, weekly-vs-monthly.
463 +- **Paper** ([`paper/qwhpi.pdf`](paper/qwhpi.pdf)): *A High-Frequency Hedonic
464 + Housing Price Index for Quebec* — real, researched citations only (hedonic
465 + and repeat-sales index theory, Case-Shiller, hierarchical and state-space
466 + indexes, spatial hedonics, index-number theory).
467 +
468 +---
469 +
470 +## 🔁 Reproducibility
471 +
472 +1. Place your transaction extract at `data/raw/province_transactions.csv`
473 + (columns: `id, date, amount, street, zipCode, city, lat, lng,
474 + propertyType, yearBuilt, floorArea, buildingType, previousValue,
475 + totalArValue, ownerType`).
476 +2. Download the SDA boundary file per
477 + [`data/external/SDA_SOURCE.md`](data/external/SDA_SOURCE.md).
478 +3. `make pipeline` — deterministic given the same inputs; every run is
479 + stamped with `model_version` + `data_vintage` and manifested.
480 +
481 +Without microdata, everything downstream of the canonical Parquet (DB load,
482 +API, dashboard, figures) is fully runnable from the aggregates included here.
483 +
484 +---
485 +
486 +## 👤 Author & license
487 +
488 +<div align="center">
489 +
490 +**Simon-Pierre Boucher**
491 +
492 +[![Email](https://img.shields.io/badge/✉️%20contact@spboucher.ai-D14836?style=for-the-badge)](mailto:contact@spboucher.ai)
493 +[![Site](https://img.shields.io/badge/🌐%20www.indexqc.house-2ea44f?style=for-the-badge)](https://www.indexqc.house)
494 +
495 +</div>
496 +
497 +Code, methodology, figures and documentation © Simon-Pierre Boucher.
498 +All rights reserved — contact the author for reuse, data questions or
499 +collaboration. The repository intentionally distributes **no individual
500 +transaction records**; published aggregates carry full uncertainty metadata
501 +so they are never mistaken for more precision than the data supports.
added api/Dockerfile +14 −0
@@ -0,0 +1,14 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/Dockerfile
6 +# Purpose : Container image for the QWHPI API service.
7 +# =============================================================================
8 +FROM python:3.12-slim
9 +WORKDIR /srv
10 +COPY pyproject.toml ./
11 +RUN pip install --no-cache-dir fastapi "uvicorn[standard]" polars pydantic "psycopg[binary]"
12 +COPY app ./app
13 +EXPOSE 8080
14 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
added api/app/__init__.py +7 −0
@@ -0,0 +1,7 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/__init__.py
6 +# Purpose : Package root for the QWHPI API application.
7 +# =============================================================================
added api/app/deps.py +44 −0
@@ -0,0 +1,44 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/deps.py
6 +# Purpose : Shared dependencies — series validation, ETag and CSV helpers.
7 +# =============================================================================
8 +"""Shared FastAPI dependencies and response helpers."""
9 +
10 +from __future__ import annotations
11 +
12 +import hashlib
13 +
14 +import polars as pl
15 +from fastapi import HTTPException, Request, Response
16 +
17 +from app.services import data
18 +
19 +VALID_TYPES = {"all", "unifamilial", "condo", "plex"}
20 +
21 +
22 +def validate_series(geography: str, property_type: str) -> None:
23 + if property_type not in VALID_TYPES:
24 + raise HTTPException(422, f"property_type must be one of {sorted(VALID_TYPES)}")
25 + if geography not in data.geography_ids():
26 + raise HTTPException(404, f"unknown geography '{geography}' — see /v1/geographies")
27 +
28 +
29 +def etag_for(request: Request, response: Response, *parts: str) -> bool:
30 + """Set the ETag; return True when the client's copy is fresh (304)."""
31 + raw = "|".join([data.data_vintage(), data.model_version(),
32 + str(request.url), *parts])
33 + tag = '"' + hashlib.sha256(raw.encode()).hexdigest()[:32] + '"'
34 + response.headers["ETag"] = tag
35 + response.headers["Cache-Control"] = "public, max-age=300"
36 + return request.headers.get("if-none-match") == tag
37 +
38 +
39 +def csv_response(df: pl.DataFrame) -> Response:
40 + return Response(
41 + content=df.write_csv(),
42 + media_type="text/csv",
43 + headers={"X-Author": "Simon-Pierre Boucher <contact@spboucher.ai>"},
44 + )
added api/app/main.py +67 −0
@@ -0,0 +1,67 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/main.py
6 +# Purpose : FastAPI application — routers, CORS, rate limiting, OpenAPI.
7 +# =============================================================================
8 +"""QWHPI API service entry point.
9 +
10 +Run locally: uvicorn app.main:app --reload --port 8080 (from api/)
11 +Docs: /docs (OpenAPI). Every response credits the author and exposes
12 +uncertainty (CIs, reliability grades) — the API never hides it.
13 +"""
14 +
15 +from __future__ import annotations
16 +
17 +import time
18 +from collections import defaultdict, deque
19 +
20 +from fastapi import FastAPI, Request
21 +from fastapi.middleware.cors import CORSMiddleware
22 +from fastapi.responses import JSONResponse
23 +
24 +from app.routers import geographies, index, liquidity, maps, meta, reports
25 +
26 +RATE_LIMIT = 120 # requests
27 +RATE_WINDOW = 60.0 # seconds
28 +_hits: dict[str, deque] = defaultdict(deque)
29 +
30 +app = FastAPI(
31 + title="QWHPI API",
32 + version="0.1.0",
33 + description=(
34 + "Quebec Weekly Housing Price Index — quality-adjusted, hierarchically "
35 + "pooled weekly indexes for Quebec geographies and property types.\n\n"
36 + "**Author: Simon-Pierre Boucher — contact@spboucher.ai**"
37 + ),
38 + contact={"name": "Simon-Pierre Boucher", "email": "contact@spboucher.ai"},
39 +)
40 +
41 +app.add_middleware(
42 + CORSMiddleware,
43 + allow_origins=["*"],
44 + allow_methods=["GET"],
45 + allow_headers=["*"],
46 +)
47 +
48 +
49 +@app.middleware("http")
50 +async def rate_limiter(request: Request, call_next):
51 + ip = request.client.host if request.client else "anon"
52 + now = time.monotonic()
53 + q = _hits[ip]
54 + while q and now - q[0] > RATE_WINDOW:
55 + q.popleft()
56 + if len(q) >= RATE_LIMIT:
57 + return JSONResponse({"detail": "rate limit exceeded"}, status_code=429)
58 + q.append(now)
59 + return await call_next(request)
60 +
61 +
62 +app.include_router(index.router, prefix="/v1", tags=["index"])
63 +app.include_router(reports.router, prefix="/v1", tags=["stats & reports"])
64 +app.include_router(geographies.router, prefix="/v1", tags=["geographies"])
65 +app.include_router(liquidity.router, prefix="/v1", tags=["liquidity"])
66 +app.include_router(maps.router, prefix="/v1", tags=["maps"])
67 +app.include_router(meta.router, tags=["meta"])
added api/app/routers/__init__.py +7 −0
@@ -0,0 +1,7 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/__init__.py
6 +# Purpose : Router package for the QWHPI API.
7 +# =============================================================================
added api/app/routers/geographies.py +63 −0
@@ -0,0 +1,63 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/geographies.py
6 +# Purpose : Geography hierarchy + coverage matrix endpoints.
7 +# =============================================================================
8 +"""Geography and coverage endpoints."""
9 +
10 +from __future__ import annotations
11 +
12 +import polars as pl
13 +from fastapi import APIRouter, Query, Request, Response
14 +
15 +from app.deps import csv_response, etag_for
16 +from app.schemas import GeographiesResponse, GeographyNode
17 +from app.services import data
18 +
19 +router = APIRouter()
20 +
21 +
22 +@router.get("/geographies", response_model=GeographiesResponse)
23 +def geographies(request: Request, response: Response):
24 + """Published series hierarchy (province → regions → municipalities)."""
25 + if etag_for(request, response):
26 + return Response(status_code=304)
27 + c = data.canonical()
28 + nodes = (
29 + c.group_by(["geography_id", "geography_name", "geography_level"])
30 + .agg(pl.col("property_type").unique().sort().alias("types"))
31 + .sort(["geography_level", "geography_id"])
32 + )
33 + return GeographiesResponse(
34 + data_vintage=data.data_vintage(),
35 + published_series=[
36 + GeographyNode(geography_id=r["geography_id"],
37 + geography_name=r["geography_name"],
38 + geography_level=r["geography_level"],
39 + property_types=r["types"])
40 + for r in nodes.iter_rows(named=True)
41 + ],
42 + )
43 +
44 +
45 +@router.get("/geographies/coverage")
46 +def coverage_matrix(
47 + request: Request,
48 + response: Response,
49 + status: str | None = Query(None, pattern="^(published|conditional|not_published)$"),
50 + format: str = Query("json", pattern="^(json|csv)$"),
51 +):
52 + """Full municipality × type coverage matrix with publication status."""
53 + df = data.coverage()
54 + if status:
55 + df = df.filter(pl.col("coverage_status") == status)
56 + if format == "csv":
57 + return csv_response(df)
58 + if etag_for(request, response, str(status)):
59 + return Response(status_code=304)
60 + return {
61 + "coverage": df.rename({"propertyType": "property_type"}).to_dicts(),
62 + "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai",
63 + }
added api/app/routers/index.py +197 −0
@@ -0,0 +1,197 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/index.py
6 +# Purpose : Index endpoints (monthly v2.1) — series, latest, compare, vintages.
7 +# =============================================================================
8 +"""Index series endpoints."""
9 +
10 +from __future__ import annotations
11 +
12 +import math
13 +
14 +import polars as pl
15 +from fastapi import APIRouter, HTTPException, Query, Request, Response
16 +
17 +from app.deps import csv_response, etag_for, validate_series
18 +from app.schemas import LatestResponse, Observation, SeriesResponse, VintageObservation
19 +from app.services import data
20 +
21 +router = APIRouter()
22 +
23 +
24 +def _clean(v):
25 + return None if (v is None or (isinstance(v, float) and math.isnan(v))) else v
26 +
27 +
28 +def _obs(row: dict) -> Observation:
29 + return Observation(
30 + period=row["period"],
31 + index=_clean(row["index"]),
32 + index_smoothed=row["index_smoothed"],
33 + representative_value=row["representative_value"],
34 + transactions=row["transactions"],
35 + effective_sample_size=row["effective_sample_size"],
36 + monthly_pct=row["monthly_pct"],
37 + three_month_pct=row["three_month_pct"],
38 + six_month_pct=row["six_month_pct"],
39 + yoy_pct=row["yoy_pct"],
40 + lower_95=row["lower_95"],
41 + upper_95=row["upper_95"],
42 + reliability_grade=row["reliability_grade"],
43 + shrinkage_weight=row["shrinkage_weight"],
44 + is_partial_month=row["is_partial_month"],
45 + )
46 +
47 +
48 +@router.get("/index", response_model=SeriesResponse)
49 +def get_index(
50 + request: Request,
51 + response: Response,
52 + geography: str = Query(..., description="geography_id, e.g. quebec, region-06, montreal"),
53 + type: str = Query("all", alias="type"),
54 + from_: str | None = Query(None, alias="from", description="first period (YYYY-MM)"),
55 + to: str | None = Query("latest"),
56 + limit: int | None = Query(None, ge=1, le=1000),
57 + offset: int = Query(0, ge=0),
58 + format: str = Query("json", pattern="^(json|csv)$"),
59 +):
60 + """Full monthly history for one series (paginated; CSV via ?format=csv)."""
61 + validate_series(geography, type)
62 + df = data.series(geography, type, from_, to)
63 + if df.height == 0:
64 + raise HTTPException(404, "no observations for this series/range")
65 + total = df.height
66 + page = df.slice(offset, limit) if limit else df.slice(offset)
67 + if format == "csv":
68 + return csv_response(page)
69 + if etag_for(request, response, geography, type, str(from_), str(to),
70 + str(limit), str(offset)):
71 + return Response(status_code=304)
72 + first = page.row(0, named=True)
73 + return SeriesResponse(
74 + geography=geography,
75 + geography_name=first["geography_name"],
76 + geography_level=first["geography_level"],
77 + property_type=type,
78 + model_version=first["model_version"],
79 + data_vintage=first["data_vintage"],
80 + total_observations=total,
81 + offset=offset,
82 + limit=limit,
83 + observations=[_obs(r) for r in page.iter_rows(named=True)],
84 + )
85 +
86 +
87 +@router.get("/index/latest", response_model=LatestResponse)
88 +def get_latest(
89 + request: Request,
90 + response: Response,
91 + geography: str,
92 + type: str = "all",
93 + include_partial: bool = Query(False, description="include the partial (nowcast) month"),
94 +):
95 + """Latest observation (complete month by default; nowcast on request)."""
96 + validate_series(geography, type)
97 + df = data.series(geography, type)
98 + if not include_partial:
99 + df = df.filter(~pl.col("is_partial_month"))
100 + if df.height == 0:
101 + raise HTTPException(404, "no observations")
102 + row = df.row(df.height - 1, named=True)
103 + if etag_for(request, response, geography, type, row["period"],
104 + str(include_partial)):
105 + return Response(status_code=304)
106 + return LatestResponse(
107 + geography=geography,
108 + geography_name=row["geography_name"],
109 + property_type=type,
110 + period=row["period"],
111 + latest_index=round(row["index_smoothed"], 2),
112 + latest_index_raw=(None if _clean(row["index"]) is None
113 + else round(row["index"], 2)),
114 + representative_value=row["representative_value"],
115 + monthly_change=row["monthly_pct"],
116 + three_month_change=row["three_month_pct"],
117 + yoy_change=row["yoy_pct"],
118 + transactions=row["transactions"],
119 + effective_sample_size=row["effective_sample_size"],
120 + reliability=row["reliability_grade"],
121 + lower_95=round(row["lower_95"], 2),
122 + upper_95=round(row["upper_95"], 2),
123 + is_partial_month=row["is_partial_month"],
124 + model_version=row["model_version"],
125 + data_vintage=row["data_vintage"],
126 + )
127 +
128 +
129 +@router.get("/compare")
130 +def compare(
131 + request: Request,
132 + response: Response,
133 + series: str = Query(..., description="comma list of geography:type, e.g. "
134 + "montreal:condo,quebec-city:condo"),
135 + rebase_period: str | None = Query(None, description="rebase all series to 100 at this period"),
136 +):
137 + """Aligned comparison of up to 8 series, optionally rebased."""
138 + pairs = [s.strip() for s in series.split(",") if s.strip()]
139 + if not 2 <= len(pairs) <= 8:
140 + raise HTTPException(422, "provide 2 to 8 series")
141 + out = {}
142 + for p in pairs:
143 + try:
144 + g, t = p.split(":")
145 + except ValueError:
146 + raise HTTPException(422, f"bad series spec '{p}' (want geography:type)")
147 + validate_series(g, t)
148 + df = data.series(g, t).select("period", "index_smoothed",
149 + "representative_value", "reliability_grade")
150 + if rebase_period:
151 + base = df.filter(pl.col("period") == rebase_period)
152 + if base.height == 0:
153 + raise HTTPException(422, f"rebase_period {rebase_period} not found for {p}")
154 + df = df.with_columns(
155 + (pl.col("index_smoothed") / base["index_smoothed"][0] * 100)
156 + .round(3).alias("index_smoothed"))
157 + out[p] = df.to_dicts()
158 + if etag_for(request, response, series, str(rebase_period)):
159 + return Response(status_code=304)
160 + return {"series": out, "rebase_period": rebase_period,
161 + "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai"}
162 +
163 +
164 +@router.get("/vintages")
165 +def vintages(
166 + geography: str,
167 + type: str = "all",
168 + period: str | None = None,
169 +):
170 + """First release vs current vintage per period (revision tracking)."""
171 + validate_series(geography, type)
172 + fr = data.first_release()
173 + if fr is None:
174 + raise HTTPException(404, "no vintage history yet")
175 + cur = data.series(geography, type).select(
176 + "period", "index_smoothed", "data_vintage")
177 + j = (
178 + fr.filter((pl.col("geography_id") == geography)
179 + & (pl.col("property_type") == type))
180 + .join(cur, on="period", how="inner")
181 + )
182 + if period:
183 + j = j.filter(pl.col("period") == period)
184 + obs = [
185 + VintageObservation(
186 + period=r["period"],
187 + first_release_index_smoothed=round(r["first_release_index_smoothed"], 3),
188 + current_index_smoothed=round(r["index_smoothed"], 3),
189 + first_release_vintage=r["first_release_vintage"],
190 + current_vintage=r["data_vintage"],
191 + revision_pct=round((r["index_smoothed"]
192 + / r["first_release_index_smoothed"] - 1) * 100, 3),
193 + ).model_dump()
194 + for r in j.sort("period").iter_rows(named=True)
195 + ]
196 + return {"geography": geography, "property_type": type, "observations": obs,
197 + "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai"}
added api/app/routers/liquidity.py +41 −0
@@ -0,0 +1,41 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/liquidity.py
6 +# Purpose : Weekly transaction-volume (liquidity) endpoint per series.
7 +# =============================================================================
8 +"""Liquidity endpoints."""
9 +
10 +from __future__ import annotations
11 +
12 +import polars as pl
13 +from fastapi import APIRouter, HTTPException, Request, Response
14 +
15 +from app.deps import etag_for, validate_series
16 +from app.schemas import LiquidityPoint, LiquidityResponse
17 +from app.services import data
18 +
19 +router = APIRouter()
20 +
21 +
22 +@router.get("/liquidity", response_model=LiquidityResponse)
23 +def liquidity(request: Request, response: Response,
24 + geography: str, type: str = "all"):
25 + """Weekly transaction counts for one series."""
26 + validate_series(geography, type)
27 + df = data.liquidity().filter(
28 + (pl.col("geography_id") == geography)
29 + & (pl.col("property_type") == type)
30 + ).sort("period")
31 + if df.height == 0:
32 + raise HTTPException(404, "no liquidity data")
33 + if etag_for(request, response, geography, type):
34 + return Response(status_code=304)
35 + return LiquidityResponse(
36 + geography=geography,
37 + property_type=type,
38 + median_monthly_transactions=float(df["transactions"].median()),
39 + observations=[LiquidityPoint(period=r["period"], transactions=r["transactions"])
40 + for r in df.iter_rows(named=True)],
41 + )
added api/app/routers/maps.py +72 −0
@@ -0,0 +1,72 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/maps.py
6 +# Purpose : Choropleth data endpoint — one metric across geographies.
7 +# =============================================================================
8 +"""Map data endpoints (region choropleth + assessment gap)."""
9 +
10 +from __future__ import annotations
11 +
12 +import polars as pl
13 +from fastapi import APIRouter, HTTPException, Query, Request, Response
14 +
15 +from app.deps import etag_for
16 +from app.schemas import MapCell, MapResponse
17 +from app.services import data
18 +
19 +router = APIRouter()
20 +
21 +METRICS = {
22 + "yoy": "yoy_pct",
23 + "six_month": "six_month_pct",
24 + "three_month": "three_month_pct",
25 + "index_level": "index_smoothed",
26 +}
27 +
28 +
29 +@router.get("/map", response_model=MapResponse)
30 +def map_data(
31 + request: Request,
32 + response: Response,
33 + metric: str = Query("yoy", description=f"one of {sorted(METRICS)} or assessment_gap"),
34 + level: str = Query("region", pattern="^(region|municipality)$"),
35 + type: str = Query("all"),
36 + period: str | None = None,
37 +):
38 + """One metric per geography for choropleth rendering."""
39 + c = data.canonical().filter(
40 + (pl.col("geography_level") == level) & (pl.col("property_type") == type))
41 + if period is None:
42 + period = c.filter(~pl.col("is_partial_month"))["period"].max()
43 + if metric == "assessment_gap":
44 + gap = data.assessment_gap().filter(
45 + (pl.col("property_type") == type) & (pl.col("period") == period)
46 + & pl.col("geography_id").str.starts_with("region-" if level == "region" else ""))
47 + c_wk = c.filter(pl.col("period") == period).select(
48 + "geography_id", "geography_name", "reliability_grade", "transactions")
49 + j = c_wk.join(gap, on="geography_id", how="left")
50 + cells = [MapCell(geography_id=r["geography_id"],
51 + geography_name=r["geography_name"],
52 + value=r["median_ratio"],
53 + reliability_grade=r["reliability_grade"],
54 + transactions=r["transactions"])
55 + for r in j.iter_rows(named=True)]
56 + else:
57 + if metric not in METRICS:
58 + raise HTTPException(422, f"metric must be one of {sorted(METRICS)} or assessment_gap")
59 + col = METRICS[metric]
60 + wk = c.filter(pl.col("period") == period)
61 + if wk.height == 0:
62 + raise HTTPException(404, f"no data for period {period}")
63 + cells = [MapCell(geography_id=r["geography_id"],
64 + geography_name=r["geography_name"],
65 + value=r[col],
66 + reliability_grade=r["reliability_grade"],
67 + transactions=r["transactions"])
68 + for r in wk.iter_rows(named=True)]
69 + if etag_for(request, response, metric, level, type, str(period)):
70 + return Response(status_code=304)
71 + return MapResponse(metric=metric, level=level, property_type=type,
72 + period=str(period), cells=cells)
added api/app/routers/meta.py +41 −0
@@ -0,0 +1,41 @@
1 +# =============================================================================
2 +# QWHPI — Quebec Weekly Housing Price Index
3 +# Author : Simon-Pierre Boucher
4 +# Contact : contact@spboucher.ai
5 +# File : api/app/routers/meta.py
6 +# Purpose : /v1/meta and /health endpoints.
7 +# =============================================================================
8 +"""Metadata and health endpoints."""
9 +
10 +from __future__ import annotations
11 +
12 +import polars as pl
13 +from fastapi import APIRouter
14 +
15 +from app.schemas import MetaResponse
16 +from app.services import data
17 +
18 +router = APIRouter()
19 +
20 +
21 +@router.get("/v1/meta", response_model=MetaResponse)
22 +def meta():
23 + c = data.canonical()
24 + return MetaResponse(
25 + model_version=data.model_version(),
26 + data_vintage=data.data_vintage(),
27 + series_count=c.select("geography_id", "property_type").unique().height,
28 + periods=c["period"].n_unique(),
29 + latest_complete_period=str(
30 + c.filter(~pl.col("is_partial_month"))["period"].max()),
31 + )
32 +
33 +
34 +@router.get("/health")
35 +def health():
36 + try:
37 + rows = data.canonical().height
38 + return {"status": "ok", "canonical_rows": rows,
39 + "data_vintage": data.data_vintage()}
40 + except Exception as exc: # pragma: no cover
41 + return {"status": "degraded", "error": str(exc)}
added api/app/routers/reports.py +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added api/app/schemas.py +148 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added api/app/services/__init__.py +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added api/app/services/data.py +96 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added api/app/services/report.py +345 −0
Large diff (345 lines) is not loaded by default to keep the page fast.Load diff
added api/app/services/stats.py +164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added api/pyproject.toml +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added api/tests/test_api.py +136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added data/external/SDA_SOURCE.md +32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added data/processed/assessment_gap.parquet +0 −0

Binary file not shown.

added data/processed/assessment_gap_monthly.parquet +0 −0

Binary file not shown.

added data/processed/coverage_matrix.parquet +0 −0

Binary file not shown.

added data/processed/coverage_matrix_monthly.parquet +0 −0

Binary file not shown.

added data/processed/first_release.parquet +0 −0

Binary file not shown.

added data/processed/first_release_monthly.parquet +0 −0

Binary file not shown.

added data/processed/monthly_liquidity.parquet +0 −0

Binary file not shown.

added data/processed/municipalities.parquet +0 −0

Binary file not shown.

added data/processed/qhpi_monthly.parquet +0 −0

Binary file not shown.

added data/processed/qwhpi_weekly.parquet +0 −0

Binary file not shown.

added data/processed/vintages/data_vintage=2026-08-06/qwhpi_weekly.parquet +0 −0

Binary file not shown.

added data/processed/vintages_monthly/data_vintage=2026-08-08/qhpi_monthly.parquet +0 −0

Binary file not shown.

added data/processed/weekly_liquidity.parquet +0 −0

Binary file not shown.

added db/alembic.ini +32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added db/migrations/env.py +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added db/migrations/versions/001_initial_schema.py +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added db/schema.sql +74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added docker-compose.yml +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added engine/pyproject.toml +43 −0
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/01_profile.py +301 −0
Large diff (301 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/02_geography.py +132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/03_clean.py +137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/04_baseline.py +98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/05_hierarchical.py +230 −0
Large diff (230 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/06_validation.py +271 −0
Large diff (271 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/07_canonical.py +228 −0
Large diff (228 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/08_figures.py +183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/09_monthly.py +275 −0
Large diff (275 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/10_monthly_validation.py +219 −0
Large diff (219 lines) is not loaded by default to keep the page fast.Load diff
added engine/scripts/11_monthly_canonical.py +220 −0
Large diff (220 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/__init__.py +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/clean.py +149 −0
Large diff (149 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/config.py +100 −0
Large diff (100 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/export.py +128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/features.py +103 −0
Large diff (103 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/geography.py +130 −0
Large diff (130 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/hedonic.py +104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/hierarchy.py +228 −0
Large diff (228 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/index.py +139 −0
Large diff (139 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/ingest.py +81 −0
Large diff (81 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/nowcast.py +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/plotting.py +63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/repeat_sales.py +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/rtd.py +289 −0
Large diff (289 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/seasonal.py +47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/state_space.py +123 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/uncertainty.py +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added engine/src/qwhpi/vintages.py +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added engine/tests/test_features_clean.py +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added engine/tests/test_smoke_pipeline.py +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added engine/tests/test_state_space.py +60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added ops/ci/ci.yml +44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added ops/scheduler/Dockerfile +13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added ops/scheduler/refresh.py +90 −0
Large diff (90 lines) is not loaded by default to keep the page fast.Load diff
added outputs/figures/assessment_gap.png +0 −0

Binary file not shown.

added outputs/figures/big4_cities.png +0 −0

Binary file not shown.

added outputs/figures/condo_by_city.png +0 −0

Binary file not shown.

added outputs/figures/downsampling_stability.png +0 −0

Binary file not shown.

added outputs/figures/quebec_aggregate.png +0 −0

Binary file not shown.

added outputs/figures/raw_median_vs_hedonic.png +0 −0

Binary file not shown.

added outputs/figures/region_comparison.png +0 −0

Binary file not shown.

added outputs/figures/reliability_matrix.png +0 −0

Binary file not shown.

added outputs/figures/volumes.png +0 −0

Binary file not shown.

added outputs/figures/yoy_map.png +0 −0

Binary file not shown.

added outputs/reports/audit_report.md +127 −0
Large diff (127 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/cleaning_report.md +42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/geography_report.md +47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/hierarchical_report.md +79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/methodology_v2_research.md +84 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/validation_report.md +43 −0
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
added outputs/reports/validation_v2_report.md +42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_amount_by_type.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_characteristics_by_type.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_city_top200.csv +201 −0
Large diff (201 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_dist_buildingType.csv +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_dist_ownerType.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_dist_propertyType.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_liquidity_matrix.csv +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_missingness.csv +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_schema.csv +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_sentinel_values.csv +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_weekly_liquidity_topcities.csv +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/audit_weekly_volume_province.csv +292 −0
Large diff (292 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/baseline_model_summary.csv +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/composition_shock.csv +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/composition_shock_monthly.csv +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/coverage.csv +2336 −0
Large diff (2,336 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/coverage_monthly.csv +2336 −0
Large diff (2,336 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/downsampling_monthly.csv +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/downsampling_results.csv +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/estimation_methods.csv +85 −0
Large diff (85 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/exclusions.csv +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/geo_city_mismatch_top300.csv +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/geo_region_distribution.csv +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/hierarchical_summary.csv +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/index_latest.csv +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/index_latest_monthly.csv +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/monthly_summary.csv +113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/repeat_sales_comparison.csv +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/repeat_sales_comparison_monthly.csv +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/research_sample_missingness.csv +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/rtd_coefficient_drift.csv +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/seasonality_monthly.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/seasonality_tests.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/weekly_liquidity.csv +112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added outputs/tables/weekly_vs_monthly.csv +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added paper/qwhpi.pdf +0 −0

Binary file not shown.

added paper/qwhpi.tex +187 −0
Large diff (187 lines) is not loaded by default to keep the page fast.Load diff
added paper/references.bib +132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added scripts/check_headers.py +91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
added web/Dockerfile +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added web/app/api-docs/page.tsx +178 −0
Large diff (178 lines) is not loaded by default to keep the page fast.Load diff
added web/app/compare/page.tsx +293 −0
Large diff (293 lines) is not loaded by default to keep the page fast.Load diff
added web/app/explore/page.tsx +310 −0
Large diff (310 lines) is not loaded by default to keep the page fast.Load diff
added web/app/icon.svg +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added web/app/layout.tsx +98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added web/app/map/page.tsx +241 −0
Large diff (241 lines) is not loaded by default to keep the page fast.Load diff
added web/app/methodology/page.tsx +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added web/app/opengraph-image.tsx +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added web/app/page.tsx +238 −0
Large diff (238 lines) is not loaded by default to keep the page fast.Load diff
added web/components/CommandPalette.tsx +146 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added web/components/CountUp.tsx +56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added web/components/DataTable.tsx +119 −0
Large diff (119 lines) is not loaded by default to keep the page fast.Load diff
added web/components/HeroChart.tsx +112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
added web/components/IndexChart.tsx +274 −0
Large diff (274 lines) is not loaded by default to keep the page fast.Load diff
added web/components/ReliabilityBadge.tsx +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added web/components/SearchButton.tsx +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added web/components/Skeleton.tsx +61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added web/components/Sparkline.tsx +43 −0
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
added web/components/ThemeToggle.tsx +42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added web/components/Ticker.tsx +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added web/lib/api.ts +166 −0
Large diff (166 lines) is not loaded by default to keep the page fast.Load diff
added web/lib/events.ts +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added web/lib/exportPng.ts +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added web/next.config.ts +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added web/package-lock.json +1413 −0
Large diff (1,413 lines) is not loaded by default to keep the page fast.Load diff
added web/package.json +25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added web/public/regions.geojson +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added web/styles/globals.css +591 −0
Large diff (591 lines) is not loaded by default to keep the page fast.Load diff
added web/tsconfig.json +22 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere