spb/qwhpi Public

1# CLAUDE.md — QWHPI Platform2# Quebec Weekly Housing Price Index — Full Platform Build34---56## 0. MANDATORY FILE HEADERS — READ FIRST78Every single file created in this project — Python, TypeScript, JavaScript, SQL, YAML, TOML, shell scripts, LaTeX, CSS, Dockerfiles, config files, notebooks — MUST begin with an author header.910**Python / shell / YAML / TOML / Dockerfile:**1112```python13# =============================================================================14# QWHPI — Quebec Weekly Housing Price Index15# Author  : Simon-Pierre Boucher16# Contact : contact@spboucher.ai17# File    : <relative/path/to/file.py>18# Purpose : <one-line description>19# =============================================================================20```2122**TypeScript / JavaScript / TSX / CSS:**2324```ts25/**26 * =============================================================================27 * QWHPI — Quebec Weekly Housing Price Index28 * Author  : Simon-Pierre Boucher29 * Contact : contact@spboucher.ai30 * File    : <relative/path/to/file.ts>31 * Purpose : <one-line description>32 * =============================================================================33 */34```3536**SQL:**3738```sql39-- =============================================================================40-- QWHPI — Quebec Weekly Housing Price Index41-- Author  : Simon-Pierre Boucher42-- Contact : contact@spboucher.ai43-- File    : <relative/path/to/file.sql>44-- Purpose : <one-line description>45-- =============================================================================46```4748**Markdown / LaTeX:** include an equivalent comment or front-matter block at the top.4950Rules:51- No file ships without this header. Add it at file creation time, not as a cleanup pass.52- The `File` and `Purpose` lines must be accurate and kept up to date.53- Add a pre-commit hook (`scripts/check_headers.py`) that fails CI if any tracked source file is missing the header.54- API responses and the frontend footer must credit: **Simon-Pierre Boucher — contact@spboucher.ai**.5556---5758## 1. Mission5960Build a complete, production-grade **platform** around the Quebec Weekly Housing Price Index — not just a research pipeline. The platform has four layers:61621. **Index Engine** — reproducible econometric pipeline producing quality-adjusted weekly indexes from `province_transactions.csv`.632. **Data Store** — versioned Parquet lake + relational database serving the API.643. **API** — FastAPI service exposing every published series with metadata, uncertainty, and reliability.654. **Dashboard** — interactive web frontend (charts, maps, comparisons, downloads).6667The index must NOT be a median-price tracker. It is a **hierarchical hedonic time-dummy index**: pooled hedonic estimation of characteristics, plus a latent weekly market state per `geography × property_type`, with partial pooling/shrinkage toward regional trends when weekly liquidity is thin.6869Target queryable observation:7071```72Geography: Quebec City | Type: Condo | Week: 2024-05-0673Index: 137.42 | 1w: +0.31% | 4w: +1.18% | YoY: +6.74%74Representative value: $389,200 | Transactions: 47 | Reliability: A75```7677---7879## 2. Dataset8081`province_transactions.csv` — ~745,119 transactions, 291 weeks, 2021-01-04 → 2026-07-27, 1,100+ cities.8283Columns: `id, date, amount, street, zipCode, city, lat, lng, propertyType, yearBuilt, floorArea, buildingType, previousValue, totalArValue, ownerType`8485Property types: `unifamilial` (~443k), `condo` (~100k), `plex` (~94k), `indéterminé` (~108k). `indéterminé` must never contaminate type-specific indexes; study what it represents and document it in an appendix.8687Several hedonic characteristics have missing values. Handle missingness explicitly — never `dropna()` the sample.8889---9091## 3. Repository Structure (Monorepo)9293```94qwhpi-platform/95├── CLAUDE.md96├── README.md97├── Makefile                     # one-command targets: make pipeline, make api, make web98├── docker-compose.yml           # db + api + web + scheduler99├── .pre-commit-config.yaml100│101├── engine/                      # Index Engine (Python)102│   ├── pyproject.toml103│   ├── src/qwhpi/104│   │   ├── config.py  ingest.py  clean.py  geography.py  features.py105│   │   ├── hedonic.py  hierarchy.py  state_space.py  repeat_sales.py106│   │   ├── index.py  uncertainty.py  reliability.py  validation.py107│   │   ├── seasonal.py  nowcast.py  vintages.py  export.py108│   ├── scripts/                 # 01_profile … 10_build_report (ordered, idempotent)109│   ├── tests/110│   └── notebooks/exploratory_only/111│112├── data/113│   ├── raw/  external/  interim/  processed/   # raw is immutable114│115├── db/116│   ├── migrations/              # Alembic117│   └── schema.sql118│119├── api/                         # FastAPI service120│   ├── pyproject.toml121│   ├── app/122│   │   ├── main.py  deps.py  models.py  schemas.py123│   │   ├── routers/ (index.py, geographies.py, liquidity.py, maps.py, meta.py)124│   │   └── services/125│   └── tests/126│127├── web/                         # Frontend (Next.js + TypeScript)128│   ├── package.json129│   ├── app/                     # routes: /, /explore, /compare, /map, /methodology, /api-docs130│   ├── components/  lib/  styles/131│   └── tests/132│133├── ops/134│   ├── scheduler/               # weekly refresh job135│   └── ci/136│137├── outputs/                     # figures, tables, maps, reports138├── paper/                       # LaTeX methodology paper139└── scripts/check_headers.py140```141142Production logic lives in `engine/src`, `api/app`, `web/` — never scattered in notebooks.143144---145146## 4. Index Engine — Methodology (non-negotiable core)147148### 4.1 Weekly calendar149Monday→Sunday weeks, each labeled by its Monday. Continuous grid — never drop zero-transaction weeks; flag them.150151### 4.2 Geography152Hierarchy: Quebec → 17 administrative regions → municipality → (optional) local zone. Spatial-join `lat/lng` against an authoritative Quebec/StatCan boundary dataset (document source, URL, version, CRS). Persist the join once in `data/processed/`. Do not trust the `city` text field alone.153154### 4.3 Cleaning155Reproducible pipeline; raw CSV untouched. Use `log(amount)`. Investigate zeros, implausible prices, non-arm's-length transfers, duplicates (flag with `duplicate_flag`/`duplicate_reason`, never silently delete). Produce a full exclusion table. Economically justified filters only — no unreported winsorizing.156157### 4.4 Hedonic model158Pool 2021–present. Do NOT run independent weekly regressions.159160- **Model A (headline):** structural characteristics only — type, log(floorArea), age (spline/bins, data-driven vintage breakpoints), buildingType, fine location controls.161- **Model B (robustness):** adds `totalArValue`, `previousValue` — beware valuation leakage; never the headline.162163Missingness: compare missing-indicator, conditional-median imputation, MICE, ML imputation. No future-price leakage. Choose stability + interpretability.164165### 4.5 Location control — critical166Compare municipality FE, postal-code FE, H3/geohash grid, 2D splines on lat/lng, hierarchical spatial effects. Westmount ≠ rest of Montreal. Diagnose residual spatial autocorrelation; improve until it is acceptable.167168### 4.6 Hierarchical weekly state169Latent path per `geography × property_type`:170171```172Quebec trend + region deviation + municipality deviation + type deviation + local week deviation173```174175Thin cells shrink toward parent trends; liquid cells (Montreal condo ~145/wk, Quebec City condo ~42/wk) are dominated by local data. Evaluate: Bayesian multilevel, empirical Bayes, mixed effects, state-space/Kalman (`μ_t = μ_{t-1} + η_t`), penalized splines. Benchmark stability and compute cost; do not choose complexity for its own sake. Also benchmark a two-stage architecture (structural hedonic residualization → hierarchical weekly time-series) against a unified time-dummy regression.176177### 4.7 Validation (mandatory)178- Repeat-sales index (~85k repeated addresses; beware condo/plex address sharing) as directional check.179- Downsampling experiment: use Montreal condos as lab; thin to 100/50/25/15/10/5 tx/week; measure RMSE, bias, volatility, turning-point accuracy, CI coverage. This empirically justifies liquidity tiers (starting hypotheses: ≥50 very strong, 20–49 strong, 10–19 shrinkage, 5–9 heavy shrinkage, <5 model-implied).180- Composition-shock simulation: raw median must move, hedonic index must not.181- Temporal + geographic holdouts; alternative specs (GAM, mixed effects, hierarchical Bayes, gradient-boosting residualization). ML (LightGBM/CatBoost/XGBoost) may estimate the cross-sectional component only — the time effect stays interpretable.182- Weekly vs monthly comparison from the same methodology: quantify signal-to-noise, turning-point detection, revision magnitude.183184### 4.8 Outputs per series185Base: 2021 average = 100 (keep raw latent log series). Both `index` (raw weekly) and `index_smoothed` (one-sided real-time version required alongside any two-sided smoother). Representative dollar value per segment from a documented property basket. Growth: 1w, 4w, 13w, 26w, YoY, YTD, since-2021. 95% CIs (cluster/block bootstrap or posterior). Reliability grades A–E from n, effective N, SE, shrinkage weight, missingness. NSA always; SA only if stable seasonality is demonstrated. `is_partial_week` nowcast handling for the latest week. Vintage framework: `first_release`, `current_vintage`, `revision`.186187### 4.9 Canonical dataset188189`data/processed/qwhpi_weekly.parquet`:190191```192week, geography_level, geography_id, geography_name, property_type,193index, index_smoothed, representative_value,194transactions, effective_sample_size,195weekly_pct, four_week_pct, thirteen_week_pct, yoy_pct,196lower_95, upper_95, reliability_grade, shrinkage_weight,197is_partial_week, model_version, data_vintage198```199200This single table powers the database, API, and dashboard.201202### 4.10 Index families203- **QWHPI-QC** (province), **QWHPI-REG** (17 regions), **QWHPI-CITY** (Montréal, Québec, Laval, Gatineau, Longueuil, Sherbrooke, Trois-Rivières, Saguenay, Lévis, Drummondville — as liquidity permits), **QWHPI-TYPE** (All / Single-family / Condo / Plex per supported geography).204- Secondary module: **Assessment Gap Index** from `amount / totalArValue` — separate concept, never mixed with the price index.205- Coverage matrix declaring, per municipality × type: published / conditional / not published.206207---208209## 5. Data Store210211- Parquet lake in `data/processed/` (partitioned by geography_level / year) is the source of truth.212- PostgreSQL (via docker-compose) serves the API: tables `series`, `observations`, `geographies`, `liquidity`, `vintages`, `model_runs`. Alembic migrations in `db/migrations/`.213- Loader `engine/src/qwhpi/export.py` upserts each pipeline run into the DB with `model_version` and `data_vintage`.214- Never let the API read raw CSVs.215216---217218## 6. API (FastAPI)219220Endpoints:221222```223GET /v1/index?geography=quebec-city&type=condo&from=2021-01-04&to=latest224GET /v1/index/latest?geography=...&type=...225GET /v1/geographies                 # hierarchy + coverage matrix226GET /v1/liquidity?geography=...&type=...227GET /v1/compare?series=montreal:condo,quebec-city:condo228GET /v1/map?metric=yoy&level=region229GET /v1/vintages?geography=...&type=...&week=...230GET /v1/meta                        # model_version, data_vintage, methodology link, author credit231GET /health232```233234Sample response:235236```json237{238  "geography": "Quebec City",239  "property_type": "condo",240  "frequency": "weekly",241  "latest_index": 137.42,242  "representative_value": 389200,243  "weekly_change": 0.31,244  "yoy_change": 6.74,245  "transactions": 47,246  "reliability": "A",247  "lower_95": 135.9,248  "upper_95": 138.9,249  "is_partial_week": false,250  "author": "Simon-Pierre Boucher",251  "contact": "contact@spboucher.ai"252}253```254255Requirements: Pydantic schemas, OpenAPI docs at `/docs`, pagination for full histories, CSV/JSON export toggle, ETag caching, rate limiting, CORS for the web app, tests for every router. Reliability and CIs are always returned — the API never hides uncertainty.256257---258259## 7. Dashboard (Next.js + TypeScript)260261Pages:262- **/** — headline: Quebec aggregate, latest week, YoY, sparkline, top movers.263- **/explore** — series picker (geography tree × property type), weekly chart with CI band, raw-median overlay toggle, smoothed toggle, growth-horizon selector, transaction-volume subchart, reliability badge, CSV download.264- **/compare** — multi-series comparison (e.g., condo across major cities), rebasing tool.265- **/map** — choropleth of the 17 regions (YoY, 13w, index level, assessment gap); H3 heat map for major urban areas where density allows.266- **/methodology** — rendered methodology summary; link to the paper.267- **/api-docs** — link/embed of OpenAPI docs.268269Requirements: charting via a solid library (e.g., ECharts/Recharts/Plotly), responsive, dark/light, consistent color config shared with engine figures, loading/empty/error states, low-reliability series visually flagged (never hidden), footer credit "Simon-Pierre Boucher — contact@spboucher.ai".270271---272273## 8. Automation & Ops274275- `make pipeline` → full engine run; `make refresh` → incremental update when new transactions are appended; `make api` / `make web` / `make up` (docker-compose).276- Scheduler (`ops/scheduler/`) runs the weekly refresh: ingest new rows → append-only clean → re-estimate weekly states → write new `data_vintage` → load DB → invalidate API cache.277- CI: lint, type-check (mypy + tsc), tests, **header check**, small-sample smoke run of the pipeline.278- Structured logging + run manifest (input hash, row counts, model version, timings) for every pipeline execution.279280---281282## 9. Engineering Standards283284- Python: pandas/polars, numpy, statsmodels, scikit-learn, geopandas, pyarrow; PyMC/CmdStanPy or statsmodels state-space where justified. Vectorized; no per-transaction Python loops; cache spatial joins, features, model matrices; Parquet everywhere.285- Diagnostics for every production model: residual distribution, heteroskedasticity, residuals-vs-fitted, temporal/spatial residual patterns, coefficient stability, weekly SEs, effective N.286- Model selection optimizes interpretability + stability + low bias + responsiveness + calibrated uncertainty + reproducibility — NOT transaction-level RMSE alone. This is economic measurement, not a prediction contest.287- Core principle everywhere: separate **price movement** from **composition movement**. Never present a weekly average price as a price level. Never fake precision — expose n, effective N, CI, shrinkage weight, reliability.288289---290291## 10. Research Outputs292293- `outputs/figures/`: Quebec aggregate; region comparison; big-4 city comparison; condo/single-family/plex by city; raw median vs hedonic; volumes; CI bands; reliability map; repeat-sales vs hedonic; downsampling stability; assessment-gap evolution; YoY appreciation map. One shared plotting config.294- `outputs/tables/`: data_summary, missingness, exclusions, weekly_liquidity, coverage, model_comparison, index_latest, repeat_sales_comparison, downsampling_results, revision_statistics.295- `paper/`: full LaTeX paper — *A High-Frequency Hedonic Housing Price Index for Quebec* — real, researched citations only (hedonic indexes, repeat sales, Case-Shiller, high-frequency measurement, hierarchical indexes, state-space indexes, spatial hedonics, index-number theory). No fabricated references.296- Research questions to actually investigate: regional heterogeneity post-2021; synchronization of turning points; Montreal lead/lag; type-level response speeds; weekly-vs-monthly information gain; minimum transactions for reliability; value of hierarchical pooling; assessment lag dynamics.297298---299300## 11. Execution Order3013021. **Audit** — schema, descriptives, missingness, outliers, coverage, weekly liquidity matrices. No index yet.3032. **Geography** — boundary acquisition, spatial join, validation, persistence.3043. **Cleaning** — research sample with documented exclusions.3054. **Baseline index** — transparent pooled hedonic time-dummy for Quebec + Montreal/Québec/Laval/Gatineau × All/Unifamilial/Condo/Plex.3065. **Hierarchical weekly model** — shrinkage for thin segments.3076. **Validation** — repeat sales, downsampling, monthly comparison, alternative specs.3087. **Scale** — all supported region and municipality series; coverage matrix.3098. **Data store + API** — DB schema, loader, FastAPI, tests.3109. **Dashboard** — all pages against the live API.31110. **Automation + research outputs** — scheduler, CI, figures, tables, paper.312313Work autonomously: inspect data → research accepted methodology → implement the most defensible option → document → benchmark alternatives → preserve reversibility. Do not settle for the first model that runs. Do not stop to ask permission on minor modeling choices.314315---316317## 12. Definition of Success318319One reproducible command (`make up` after `make pipeline`) yields:320321- a trustworthy weekly, quality-adjusted, hierarchically pooled index for every statistically defensible `geography × property_type`, 2021 → latest week, with CIs, reliability grades, representative dollar values, validation, and vintages;322- a documented API serving it;323- an interactive dashboard displaying it;324- publication-quality figures, tables, and a methodology paper;325- **every file in the repository headed with: Simon-Pierre Boucher — contact@spboucher.ai.**326