# Table of contents

• What this is
• Key numbers
• Data policy — no microdata in this repo
• Platform architecture
• Methodology
• Validation
• Index families & coverage
• Canonical dataset
• API
• Dashboard
• Repository map
• Quickstart
• Engine pipeline
• Engineering standards
• Research outputs & paper
• Reproducibility
• Author & license

# 🎯 What this is

QHPI is a production-grade economic-measurement platform, not a median-price tracker. It separates price movement from composition movement: a week where only mansions sell must not register as a price increase.

The platform has four layers:

Every published observation looks like this:

Geography: Quebec City | Type: Condo | Period: 2026-06
Index: 163.5 (2021 = 100) | 1m: +0.4% | YoY: +8.1%
Representative value: $389,200 | Transactions: 47 | Reliability: A
95% CI: [160.9, 166.1] | Vintage: 2026-08-08 | Model: 2.1.0

Uncertainty is never hidden: every row carries a 95% confidence interval, an A–E reliability grade, effective sample size and shrinkage weight.

# 📊 Key numbers

# 🔒 Data policy — no microdata in this repo

This repository contains NO individual transaction data.

The .gitignore enforces this at the repo boundary. Aggregates are cell-level (geography × type × period) with transaction counts, never transaction rows. To rerun the full pipeline you must supply your own data/raw/province_transactions.csv (see Reproducibility).

# 🏗 Platform architecture

                     ┌─────────────────────────────────────────────┐
                     │              province_transactions.csv       │
                     │        (local only — NOT in this repo)       │
                     └──────────────────────┬──────────────────────┘
                                            ▼
 ┌───────────────────────────── ENGINE (Python) ─────────────────────────────┐
 │  01 profile → 02 geography (spatial join, SDA boundaries) → 03 clean      │
 │  → 04–08 weekly research module (baseline, hierarchical, validation)      │
 │  → 09 monthly estimation (rolling time-dummy, Huber-IRLS)                 │
 │  → 10 monthly validation → 11 canonical parquet                           │
 └──────────────────────────────────┬────────────────────────────────────────┘
                                    ▼
              data/processed/qhpi_monthly.parquet   ← source of truth
              (+ liquidity, coverage, vintages, assessment gap)
                                    ▼
        ┌────────────── PostgreSQL (docker-compose, Alembic) ──────────────┐
        │   series · observations · geographies · liquidity · vintages     │
        └──────────────────────────────┬────────────────────────────────────┘
                                       ▼
                      FastAPI  /v1/*  (uncertainty always returned)
                                       ▼
                 Next.js dashboard — www.indexqc.house
                 (explore · compare · map · PDF reports · methodology)

Weekly refresh: ops/scheduler/ ingests new rows → append-only clean → re-estimation → new data_vintage → DB load → API cache invalidation.

# 🧪 Methodology

The index is a robust hedonic time-dummy index with hierarchical pooling — three estimation regimes arbitrated by empirical validation (repeat sales + stratified matched-cell medians), not by convenience:

# 1. Province × type backbone — rolling time-dummy (RTD)

• Pooled robust regressions (log(price) on structural characteristics + fine location fixed effects + period dummies) over 13-month rolling windows with Huber-IRLS weighting.
• Windows are combined by mean splice, making published history revision-free by construction.
• Coefficient drift is monitored and published (outputs/tables/rtd_coefficient_drift.csv) — e.g. the floor-area elasticity moves 0.556 → 0.586 across windows, which a fixed pooled model would have hidden.

# 2. Liquid cells (median ≥ 40 tx/month) — direct local estimation

Each of the 50 liquid geography × type cells gets its own robust time-dummy regression (own hedonic coefficients, own FSA fixed effects), lightly smoothed by a state-space filter. This was arbitrated against deviation-from-pooled-surface modelling, which compressed genuine local divergence — Quebec City condos moved ~+55% (confirmed independently by repeat sales and stratified matched-cell medians) while pooled deviations reported only +42%.

# 3. Thin cells — hierarchical Kalman shrinkage

The remaining 31 thin cells follow a latent random-walk deviation from their parent's published path, estimated by Kalman filter/smoother with observation noise σ²/nₜ. Thin months shrink toward the parent trend; the shrinkage weight is published on every row.

# Hedonic specification (headline = structural only)

• log(amount) on log(floorArea) (+ time-stable missingness indicator), age from yearBuilt via conditional-median imputation (its missingness drifts 15% → 2% over the sample, so a "missing" bin would leak time), and fine location fixed effects (FSA).
• buildingType is banned from all hedonic models. The provider backfilled it in region-staggered waves (most regions 2022-01, Montreal 2023-01); the time-correlated missingness fabricated a −38 log-point cliff in local Montreal indexes. The ban is documented in engine/src/qwhpi/features.py.
• Assessment values (totalArValue, previousValue) never enter the headline (valuation leakage); they power the separate Assessment Gap Index (amount / totalArValue).

# Cleaning

Raw data is immutable; every exclusion is economically justified and counted in outputs/tables/exclusions.csv (assessment-ratio band, exact duplicates, non-arm's-length transfers, …). Duplicates are flagged, never silently deleted. The indéterminé property type never contaminates type-specific indexes.

# Geography

Transactions are spatially joined (lat/lng) against the authoritative Quebec SDA cadastral boundaries (source, version and CRS documented in data/external/SDA_SOURCE.md) — the free-text city field alone is not trusted.

# ✅ Validation

Every claim below has a table or figure in outputs/:

Full write-ups: outputs/reports/ (audit, geography, cleaning, hierarchical, validation v1 & v2, methodology research).

# 🗺 Index families & coverage

The coverage matrix (data/processed/coverage_matrix_monthly.parquet, outputs/tables/coverage_monthly.csv) declares per municipality × type: published / conditional / not published.

# 🗃 Canonical dataset

data/processed/qhpi_monthly.parquet (7,504 rows) powers the DB, API and dashboard. Weekly research module: data/processed/qwhpi_weekly.parquet (32,592 rows).

period, geography_level, geography_id, geography_name, property_type,
index, index_smoothed, representative_value,
transactions, effective_sample_size,
monthly_pct, three_month_pct, six_month_pct, yoy_pct,
lower_95, upper_95, reliability_grade, shrinkage_weight,
is_partial_month, model_version, data_vintage

• index = raw estimate, index_smoothed = one-sided (real-time safe) smoother — no look-ahead in the published real-time series.
• representative_value = dollar value of a documented, fixed property basket per segment.
• Vintage framework: first releases preserved under data/processed/vintages*/, revisions queryable via the API.

# 🔌 API

FastAPI service (OpenAPI docs at /docs), CSV/JSON export toggle, ETag caching, pagination, rate limiting, CORS. Reliability and CIs are always returned.

Sample response:

{
  "geography": "Quebec City",
  "property_type": "condo",
  "frequency": "monthly",
  "latest_index": 163.5,
  "representative_value": 389200,
  "monthly_change": 0.4,
  "yoy_change": 8.1,
  "transactions": 47,
  "reliability": "A",
  "lower_95": 160.9,
  "upper_95": 166.1,
  "is_partial_month": false,
  "author": "Simon-Pierre Boucher",
  "contact": "contact@spboucher.ai"
}

# 💻 Dashboard

Live at www.indexqc.house — Next.js + TypeScript, dark/light themes, fully responsive (audited at 390/360 px), prefers-reduced-motion respected, zero-dependency custom hero chart, ⌘K command palette over all 112 series.

Low-reliability series are visually flagged, never hidden. Footer credit: Simon-Pierre Boucher — contact@spboucher.ai.

# 📁 Repository map

├── engine/                  # Index engine (Python)
│   ├── src/qwhpi/           #   ingest · clean · geography · features · hedonic
│   │                        #   rtd · hierarchy · state_space · repeat_sales
│   │                        #   index · uncertainty · seasonal · nowcast
│   │                        #   vintages · export · plotting
│   ├── scripts/             #   01_profile … 11_monthly_canonical (ordered, idempotent)
│   └── tests/               #   smoke pipeline, features/clean, state-space
├── data/
│   ├── external/            # boundary-source documentation (files excluded by size)
│   └── processed/           # ✅ aggregated series only (see Data policy)
├── db/                      # PostgreSQL schema + Alembic migrations
├── api/                     # FastAPI app: routers/ services/ tests/
├── web/                     # Next.js dashboard: app/ components/ lib/ styles/
├── ops/                     # weekly-refresh scheduler + CI pipeline
├── outputs/
│   ├── figures/             # publication figures (shared plotting config)
│   ├── tables/              # 35+ audit/validation/model tables (aggregates)
│   └── reports/             # markdown reports for every pipeline stage
├── paper/                   # LaTeX methodology paper (+ compiled PDF)
├── scripts/check_headers.py # CI gate: every file carries the author header
├── Makefile                 # one-command targets
└── docker-compose.yml       # postgres + api + web + scheduler

# 🚀 Quickstart

Requires the raw transaction CSV (not distributed — see Data policy) for make pipeline. The API and dashboard run off the included aggregated Parquet without it.

# Engine (full run ~ audit → geography → clean → estimate → validate → canonical)
make pipeline

# Incremental monthly refresh (clean → estimate → canonical, ~2 min)
make refresh

# Services
make api          # FastAPI on :8080 — docs at /docs
make web          # Next.js dashboard on :3000
make up           # docker compose: postgres + api + web + scheduler
make down

# Quality
make test         # engine + api test suites
make lint         # ruff + mypy + tsc
make headers      # author-header check (also a pre-commit hook + CI gate)
make figures      # publication figures → outputs/figures/
make paper        # compile the LaTeX paper

# ⚙️ Engine pipeline

Every run emits a structured run manifest (input hash, row counts, model version, timings).

# 📐 Engineering standards

• Every file starts with the author header — enforced by scripts/check_headers.py in pre-commit and CI.
• Vectorized pandas/polars + numpy; no per-transaction Python loops; spatial joins, features and model matrices cached as Parquet.
• Diagnostics for every production model: residual distribution, heteroskedasticity, temporal/spatial residual patterns, coefficient stability, effective N.
• Model selection optimizes interpretability + stability + calibrated uncertainty + reproducibility — never transaction-level RMSE alone.
• CI (ops/ci/ci.yml): lint, mypy + tsc, tests, header check, small-sample smoke run of the pipeline.

# 📄 Research outputs & paper

• Figures (outputs/figures/): province aggregate, region comparison, big-4 cities, condo by city, raw median vs hedonic, volumes, reliability matrix, downsampling stability, assessment gap, YoY appreciation map.
• Tables (outputs/tables/): audit suite, exclusions, liquidity, coverage, model comparison, repeat-sales comparison, downsampling, seasonality, coefficient drift, weekly-vs-monthly.
• Paper (paper/qwhpi.pdf): A High-Frequency Hedonic Housing Price Index for Quebec — real, researched citations only (hedonic and repeat-sales index theory, Case-Shiller, hierarchical and state-space indexes, spatial hedonics, index-number theory).

# 🔁 Reproducibility

1. Place your transaction extract at data/raw/province_transactions.csv (columns: id, date, amount, street, zipCode, city, lat, lng, propertyType, yearBuilt, floorArea, buildingType, previousValue, totalArValue, ownerType).
2. Download the SDA boundary file per data/external/SDA_SOURCE.md.
3. make pipeline — deterministic given the same inputs; every run is stamped with model_version + data_vintage and manifested.

Without microdata, everything downstream of the canonical Parquet (DB load, API, dashboard, figures) is fully runnable from the aggregates included here.

# 👤 Author & license

Code, methodology, figures and documentation © Simon-Pierre Boucher. All rights reserved — contact the author for reuse, data questions or collaboration. The repository intentionally distributes no individual transaction records; published aggregates carry full uncertainty metadata so they are never mistaken for more precision than the data supports.