# CLAUDE.md — QWHPI Platform
# Quebec Weekly Housing Price Index — Full Platform Build

---

## 0. MANDATORY FILE HEADERS — READ FIRST

Every single file created in this project — Python, TypeScript, JavaScript, SQL, YAML, TOML, shell scripts, LaTeX, CSS, Dockerfiles, config files, notebooks — MUST begin with an author header.

**Python / shell / YAML / TOML / Dockerfile:**

```python
# =============================================================================
# QWHPI — Quebec Weekly Housing Price Index
# Author  : Simon-Pierre Boucher
# Contact : contact@spboucher.ai
# File    : <relative/path/to/file.py>
# Purpose : <one-line description>
# =============================================================================
```

**TypeScript / JavaScript / TSX / CSS:**

```ts
/**
 * =============================================================================
 * QWHPI — Quebec Weekly Housing Price Index
 * Author  : Simon-Pierre Boucher
 * Contact : contact@spboucher.ai
 * File    : <relative/path/to/file.ts>
 * Purpose : <one-line description>
 * =============================================================================
 */
```

**SQL:**

```sql
-- =============================================================================
-- QWHPI — Quebec Weekly Housing Price Index
-- Author  : Simon-Pierre Boucher
-- Contact : contact@spboucher.ai
-- File    : <relative/path/to/file.sql>
-- Purpose : <one-line description>
-- =============================================================================
```

**Markdown / LaTeX:** include an equivalent comment or front-matter block at the top.

Rules:
- No file ships without this header. Add it at file creation time, not as a cleanup pass.
- The `File` and `Purpose` lines must be accurate and kept up to date.
- Add a pre-commit hook (`scripts/check_headers.py`) that fails CI if any tracked source file is missing the header.
- API responses and the frontend footer must credit: **Simon-Pierre Boucher — contact@spboucher.ai**.

---

## 1. Mission

Build a complete, production-grade **platform** around the Quebec Weekly Housing Price Index — not just a research pipeline. The platform has four layers:

1. **Index Engine** — reproducible econometric pipeline producing quality-adjusted weekly indexes from `province_transactions.csv`.
2. **Data Store** — versioned Parquet lake + relational database serving the API.
3. **API** — FastAPI service exposing every published series with metadata, uncertainty, and reliability.
4. **Dashboard** — interactive web frontend (charts, maps, comparisons, downloads).

The index must NOT be a median-price tracker. It is a **hierarchical hedonic time-dummy index**: pooled hedonic estimation of characteristics, plus a latent weekly market state per `geography × property_type`, with partial pooling/shrinkage toward regional trends when weekly liquidity is thin.

Target queryable observation:

```
Geography: Quebec City | Type: Condo | Week: 2024-05-06
Index: 137.42 | 1w: +0.31% | 4w: +1.18% | YoY: +6.74%
Representative value: $389,200 | Transactions: 47 | Reliability: A
```

---

## 2. Dataset

`province_transactions.csv` — ~745,119 transactions, 291 weeks, 2021-01-04 → 2026-07-27, 1,100+ cities.

Columns: `id, date, amount, street, zipCode, city, lat, lng, propertyType, yearBuilt, floorArea, buildingType, previousValue, totalArValue, ownerType`

Property types: `unifamilial` (~443k), `condo` (~100k), `plex` (~94k), `indéterminé` (~108k). `indéterminé` must never contaminate type-specific indexes; study what it represents and document it in an appendix.

Several hedonic characteristics have missing values. Handle missingness explicitly — never `dropna()` the sample.

---

## 3. Repository Structure (Monorepo)

```
qwhpi-platform/
├── CLAUDE.md
├── README.md
├── Makefile                     # one-command targets: make pipeline, make api, make web
├── docker-compose.yml           # db + api + web + scheduler
├── .pre-commit-config.yaml
│
├── engine/                      # Index Engine (Python)
│   ├── pyproject.toml
│   ├── src/qwhpi/
│   │   ├── config.py  ingest.py  clean.py  geography.py  features.py
│   │   ├── hedonic.py  hierarchy.py  state_space.py  repeat_sales.py
│   │   ├── index.py  uncertainty.py  reliability.py  validation.py
│   │   ├── seasonal.py  nowcast.py  vintages.py  export.py
│   ├── scripts/                 # 01_profile … 10_build_report (ordered, idempotent)
│   ├── tests/
│   └── notebooks/exploratory_only/
│
├── data/
│   ├── raw/  external/  interim/  processed/   # raw is immutable
│
├── db/
│   ├── migrations/              # Alembic
│   └── schema.sql
│
├── api/                         # FastAPI service
│   ├── pyproject.toml
│   ├── app/
│   │   ├── main.py  deps.py  models.py  schemas.py
│   │   ├── routers/ (index.py, geographies.py, liquidity.py, maps.py, meta.py)
│   │   └── services/
│   └── tests/
│
├── web/                         # Frontend (Next.js + TypeScript)
│   ├── package.json
│   ├── app/                     # routes: /, /explore, /compare, /map, /methodology, /api-docs
│   ├── components/  lib/  styles/
│   └── tests/
│
├── ops/
│   ├── scheduler/               # weekly refresh job
│   └── ci/
│
├── outputs/                     # figures, tables, maps, reports
├── paper/                       # LaTeX methodology paper
└── scripts/check_headers.py
```

Production logic lives in `engine/src`, `api/app`, `web/` — never scattered in notebooks.

---

## 4. Index Engine — Methodology (non-negotiable core)

### 4.1 Weekly calendar
Monday→Sunday weeks, each labeled by its Monday. Continuous grid — never drop zero-transaction weeks; flag them.

### 4.2 Geography
Hierarchy: Quebec → 17 administrative regions → municipality → (optional) local zone. Spatial-join `lat/lng` against an authoritative Quebec/StatCan boundary dataset (document source, URL, version, CRS). Persist the join once in `data/processed/`. Do not trust the `city` text field alone.

### 4.3 Cleaning
Reproducible pipeline; raw CSV untouched. Use `log(amount)`. Investigate zeros, implausible prices, non-arm's-length transfers, duplicates (flag with `duplicate_flag`/`duplicate_reason`, never silently delete). Produce a full exclusion table. Economically justified filters only — no unreported winsorizing.

### 4.4 Hedonic model
Pool 2021–present. Do NOT run independent weekly regressions.

- **Model A (headline):** structural characteristics only — type, log(floorArea), age (spline/bins, data-driven vintage breakpoints), buildingType, fine location controls.
- **Model B (robustness):** adds `totalArValue`, `previousValue` — beware valuation leakage; never the headline.

Missingness: compare missing-indicator, conditional-median imputation, MICE, ML imputation. No future-price leakage. Choose stability + interpretability.

### 4.5 Location control — critical
Compare municipality FE, postal-code FE, H3/geohash grid, 2D splines on lat/lng, hierarchical spatial effects. Westmount ≠ rest of Montreal. Diagnose residual spatial autocorrelation; improve until it is acceptable.

### 4.6 Hierarchical weekly state
Latent path per `geography × property_type`:

```
Quebec trend + region deviation + municipality deviation + type deviation + local week deviation
```

Thin cells shrink toward parent trends; liquid cells (Montreal condo ~145/wk, Quebec City condo ~42/wk) are dominated by local data. Evaluate: Bayesian multilevel, empirical Bayes, mixed effects, state-space/Kalman (`μ_t = μ_{t-1} + η_t`), penalized splines. Benchmark stability and compute cost; do not choose complexity for its own sake. Also benchmark a two-stage architecture (structural hedonic residualization → hierarchical weekly time-series) against a unified time-dummy regression.

### 4.7 Validation (mandatory)
- Repeat-sales index (~85k repeated addresses; beware condo/plex address sharing) as directional check.
- Downsampling experiment: use Montreal condos as lab; thin to 100/50/25/15/10/5 tx/week; measure RMSE, bias, volatility, turning-point accuracy, CI coverage. This empirically justifies liquidity tiers (starting hypotheses: ≥50 very strong, 20–49 strong, 10–19 shrinkage, 5–9 heavy shrinkage, <5 model-implied).
- Composition-shock simulation: raw median must move, hedonic index must not.
- Temporal + geographic holdouts; alternative specs (GAM, mixed effects, hierarchical Bayes, gradient-boosting residualization). ML (LightGBM/CatBoost/XGBoost) may estimate the cross-sectional component only — the time effect stays interpretable.
- Weekly vs monthly comparison from the same methodology: quantify signal-to-noise, turning-point detection, revision magnitude.

### 4.8 Outputs per series
Base: 2021 average = 100 (keep raw latent log series). Both `index` (raw weekly) and `index_smoothed` (one-sided real-time version required alongside any two-sided smoother). Representative dollar value per segment from a documented property basket. Growth: 1w, 4w, 13w, 26w, YoY, YTD, since-2021. 95% CIs (cluster/block bootstrap or posterior). Reliability grades A–E from n, effective N, SE, shrinkage weight, missingness. NSA always; SA only if stable seasonality is demonstrated. `is_partial_week` nowcast handling for the latest week. Vintage framework: `first_release`, `current_vintage`, `revision`.

### 4.9 Canonical dataset

`data/processed/qwhpi_weekly.parquet`:

```
week, geography_level, geography_id, geography_name, property_type,
index, index_smoothed, representative_value,
transactions, effective_sample_size,
weekly_pct, four_week_pct, thirteen_week_pct, yoy_pct,
lower_95, upper_95, reliability_grade, shrinkage_weight,
is_partial_week, model_version, data_vintage
```

This single table powers the database, API, and dashboard.

### 4.10 Index families
- **QWHPI-QC** (province), **QWHPI-REG** (17 regions), **QWHPI-CITY** (Montréal, Québec, Laval, Gatineau, Longueuil, Sherbrooke, Trois-Rivières, Saguenay, Lévis, Drummondville — as liquidity permits), **QWHPI-TYPE** (All / Single-family / Condo / Plex per supported geography).
- Secondary module: **Assessment Gap Index** from `amount / totalArValue` — separate concept, never mixed with the price index.
- Coverage matrix declaring, per municipality × type: published / conditional / not published.

---

## 5. Data Store

- Parquet lake in `data/processed/` (partitioned by geography_level / year) is the source of truth.
- PostgreSQL (via docker-compose) serves the API: tables `series`, `observations`, `geographies`, `liquidity`, `vintages`, `model_runs`. Alembic migrations in `db/migrations/`.
- Loader `engine/src/qwhpi/export.py` upserts each pipeline run into the DB with `model_version` and `data_vintage`.
- Never let the API read raw CSVs.

---

## 6. API (FastAPI)

Endpoints:

```
GET /v1/index?geography=quebec-city&type=condo&from=2021-01-04&to=latest
GET /v1/index/latest?geography=...&type=...
GET /v1/geographies                 # hierarchy + coverage matrix
GET /v1/liquidity?geography=...&type=...
GET /v1/compare?series=montreal:condo,quebec-city:condo
GET /v1/map?metric=yoy&level=region
GET /v1/vintages?geography=...&type=...&week=...
GET /v1/meta                        # model_version, data_vintage, methodology link, author credit
GET /health
```

Sample response:

```json
{
  "geography": "Quebec City",
  "property_type": "condo",
  "frequency": "weekly",
  "latest_index": 137.42,
  "representative_value": 389200,
  "weekly_change": 0.31,
  "yoy_change": 6.74,
  "transactions": 47,
  "reliability": "A",
  "lower_95": 135.9,
  "upper_95": 138.9,
  "is_partial_week": false,
  "author": "Simon-Pierre Boucher",
  "contact": "contact@spboucher.ai"
}
```

Requirements: Pydantic schemas, OpenAPI docs at `/docs`, pagination for full histories, CSV/JSON export toggle, ETag caching, rate limiting, CORS for the web app, tests for every router. Reliability and CIs are always returned — the API never hides uncertainty.

---

## 7. Dashboard (Next.js + TypeScript)

Pages:
- **/** — headline: Quebec aggregate, latest week, YoY, sparkline, top movers.
- **/explore** — series picker (geography tree × property type), weekly chart with CI band, raw-median overlay toggle, smoothed toggle, growth-horizon selector, transaction-volume subchart, reliability badge, CSV download.
- **/compare** — multi-series comparison (e.g., condo across major cities), rebasing tool.
- **/map** — choropleth of the 17 regions (YoY, 13w, index level, assessment gap); H3 heat map for major urban areas where density allows.
- **/methodology** — rendered methodology summary; link to the paper.
- **/api-docs** — link/embed of OpenAPI docs.

Requirements: charting via a solid library (e.g., ECharts/Recharts/Plotly), responsive, dark/light, consistent color config shared with engine figures, loading/empty/error states, low-reliability series visually flagged (never hidden), footer credit "Simon-Pierre Boucher — contact@spboucher.ai".

---

## 8. Automation & Ops

- `make pipeline` → full engine run; `make refresh` → incremental update when new transactions are appended; `make api` / `make web` / `make up` (docker-compose).
- Scheduler (`ops/scheduler/`) runs the weekly refresh: ingest new rows → append-only clean → re-estimate weekly states → write new `data_vintage` → load DB → invalidate API cache.
- CI: lint, type-check (mypy + tsc), tests, **header check**, small-sample smoke run of the pipeline.
- Structured logging + run manifest (input hash, row counts, model version, timings) for every pipeline execution.

---

## 9. Engineering Standards

- Python: pandas/polars, numpy, statsmodels, scikit-learn, geopandas, pyarrow; PyMC/CmdStanPy or statsmodels state-space where justified. Vectorized; no per-transaction Python loops; cache spatial joins, features, model matrices; Parquet everywhere.
- Diagnostics for every production model: residual distribution, heteroskedasticity, residuals-vs-fitted, temporal/spatial residual patterns, coefficient stability, weekly SEs, effective N.
- Model selection optimizes interpretability + stability + low bias + responsiveness + calibrated uncertainty + reproducibility — NOT transaction-level RMSE alone. This is economic measurement, not a prediction contest.
- Core principle everywhere: separate **price movement** from **composition movement**. Never present a weekly average price as a price level. Never fake precision — expose n, effective N, CI, shrinkage weight, reliability.

---

## 10. Research Outputs

- `outputs/figures/`: Quebec aggregate; region comparison; big-4 city comparison; condo/single-family/plex by city; raw median vs hedonic; volumes; CI bands; reliability map; repeat-sales vs hedonic; downsampling stability; assessment-gap evolution; YoY appreciation map. One shared plotting config.
- `outputs/tables/`: data_summary, missingness, exclusions, weekly_liquidity, coverage, model_comparison, index_latest, repeat_sales_comparison, downsampling_results, revision_statistics.
- `paper/`: full LaTeX paper — *A High-Frequency Hedonic Housing Price Index for Quebec* — real, researched citations only (hedonic indexes, repeat sales, Case-Shiller, high-frequency measurement, hierarchical indexes, state-space indexes, spatial hedonics, index-number theory). No fabricated references.
- Research questions to actually investigate: regional heterogeneity post-2021; synchronization of turning points; Montreal lead/lag; type-level response speeds; weekly-vs-monthly information gain; minimum transactions for reliability; value of hierarchical pooling; assessment lag dynamics.

---

## 11. Execution Order

1. **Audit** — schema, descriptives, missingness, outliers, coverage, weekly liquidity matrices. No index yet.
2. **Geography** — boundary acquisition, spatial join, validation, persistence.
3. **Cleaning** — research sample with documented exclusions.
4. **Baseline index** — transparent pooled hedonic time-dummy for Quebec + Montreal/Québec/Laval/Gatineau × All/Unifamilial/Condo/Plex.
5. **Hierarchical weekly model** — shrinkage for thin segments.
6. **Validation** — repeat sales, downsampling, monthly comparison, alternative specs.
7. **Scale** — all supported region and municipality series; coverage matrix.
8. **Data store + API** — DB schema, loader, FastAPI, tests.
9. **Dashboard** — all pages against the live API.
10. **Automation + research outputs** — scheduler, CI, figures, tables, paper.

Work autonomously: inspect data → research accepted methodology → implement the most defensible option → document → benchmark alternatives → preserve reversibility. Do not settle for the first model that runs. Do not stop to ask permission on minor modeling choices.

---

## 12. Definition of Success

One reproducible command (`make up` after `make pipeline`) yields:

- a trustworthy weekly, quality-adjusted, hierarchically pooled index for every statistically defensible `geography × property_type`, 2021 → latest week, with CIs, reliability grades, representative dollar values, validation, and vintages;
- a documented API serving it;
- an interactive dashboard displaying it;
- publication-quality figures, tables, and a methodology paper;
- **every file in the repository headed with: Simon-Pierre Boucher — contact@spboucher.ai.**