# CLAUDE.md — WP22: SHARP-HPI

## Mission
Develop Working Paper 22, "SHARP-HPI: A Spatially Hierarchical, Adaptive and Robust Framework for Quality-Adjusted Housing Price Indices", using actual Vrai Prix Quebec real estate transactions from January 1, 2021 through approximately October 7, 2026. Deliver a defensible original methodology, full empirical study, reproducible code, monthly municipality-by-property-type index panel, and publication-ready English LaTeX paper.

## Non-negotiable scientific principles
Do not invent data, empirical results, references, statistical significance, or novelty claims. Existing approaches include hedonic time-dummy, rolling time-dummy, hedonic imputation, Fisher, GEKS, HMTS, repeat-sales, and spatial hierarchical estimation. Demonstrate the incremental contribution against these. Never equate predictive accuracy with index accuracy. Keep quality adjustment distinct from market movements. Never use post-transaction information or future data in real-time estimates. Document unavailable variables, transaction coverage, and selection bias.

## 1. Audit and provenance
Discover local Vrai Prix sources, read-only credentials, schemas, backups, parquet, SQL tables and existing QHPI code. Never assume credentials, copy secrets into logs, or modify production. Build an immutable extraction snapshot. Profile coverage by month, municipality, type, transaction status, price, source, completeness and update vintage. Reconcile dates: deed/signing/registration/publication. Detect duplicates, multi-parcel sales, non-arm's-length transactions, transfers, new construction, fractional ownership, mixed-use, land-only and suspicious prices. Establish a defensible population definition and exclusion ledger. Keep raw, standardized, cleaned and analytic layers.

## 2. Data dictionary
Identify transaction ID, property ID, municipality code, administrative region, geolocation, property type, building subtype, transaction date, sale price, land area, living area, floor area, building age/year, bedrooms, bathrooms, parking, garages, basement, construction quality, renovations, lot characteristics, waterfront, occupancy, transaction conditions, assessment, previous sales, and available imagery-derived characteristics. Measure missingness and changing definitions over time. Do not silently substitute assessment values for market transactions. Separate house, condo, plex, multifamily and other categories with mutually exclusive classification rules.

## 3. Geographic design
Use stable Quebec municipality identifiers and historical boundary crosswalks. Support province, administrative region, RCM/MRC, CMA, municipality, and optional neighborhood or H3 cells. Model spatial proximity, commuting connectivity, housing-market similarity and geographic boundaries separately. Avoid smoothing across economically dissimilar neighbors merely because they are adjacent. Handle municipal amalgamations and boundary changes. Publish a coverage table for every geography/type/month cell.

## 4. Core estimand
Define I[g,k,t] as the quality-constant market price index for geography g, building type k and month t. Establish a reference property distribution and common-support rules. Set 2021 annual average = 100 for the primary index, and offer January 2021 = 100 as a secondary normalization. Ensure the base normalization does not change growth rates. Publish both geometric quality-adjusted movement and clearly distinguished representative-dollar-price estimates where supportable. Use annual and monthly log changes. Do not infer an actual market transaction in empty cells.

## 5. Benchmark estimators
Implement and compare:
A. Raw median and arithmetic mean prices.
B. Stratified median.
C. Standard pooled hedonic time-dummy OLS.
D. Rolling time-dummy with several window lengths and linking methods.
E. Hedonic imputation: Laspeyres, Paasche, Fisher.
F. GEKS and multilateral extensions where valid.
G. HMTS with appropriate re-estimation and splicing.
H. Repeat-sales and hybrid repeat-sales when property matching is reliable.
I. Hierarchical mixed-effects hedonic models.
J. Regularized splines/GAM and gradient boosting with strict out-of-time evaluation.
K. Existing QHPI method as a documented benchmark.

## 6. SHARP-HPI candidate architecture
Develop a structured latent-price decomposition:
log(P_i) = f(X_i;theta) + alpha_location(i) + mu_province,type,t + delta_region,type,t + delta_MRC,type,t + delta_municipality,type,t + u_spatial,t + epsilon_i.
Permit selected quality-price interactions to evolve smoothly through time. Control fine-scale location within municipalities so changing sales locations do not become false appreciation. Estimate latent market movements using hierarchical state-space models, robust likelihoods, partial pooling and optional spatial priors. Consider dynamic local-level/local-linear-trend processes, penalized splines, empirical Bayes and scalable Bayesian inference. Compare model classes rather than assume the most complex is best. Enforce identifiability constraints and explicitly describe priors, likelihood, covariance, temporal evolution and estimation algorithm.

## 7. Central methodological contribution
Investigate an adaptive estimator that optimizes the trade-off between local information, borrowing strength, temporal responsiveness, bias, and variance. Define an observable reliability score using effective sample size, covariate overlap, spatial residual dependence, posterior uncertainty, out-of-time error and local-data influence. Derive and test an adaptive pooling/shrinkage mechanism; do not call it novel unless a literature review establishes novelty. Develop stable multilateral temporal linking and a transparent revision policy. Evaluate whether a constrained or coherence-aware index aggregation procedure improves consistency between local and provincial indices. Report when reconciliation introduces bias or excessive smoothing.

## 8. Quality adjustment
Use only variables that represent property quality and location in the headline hedonic specification. Evaluate nonlinear floor-area effects, land/building decomposition, age and depreciation, renovations, housing type interactions, spatial amenities and location fixed effects. Use image/NLP-derived features only if their provenance, transaction-time availability and measurement quality can be established. Prevent image-derived signals from encoding the sale price, future information, listing outcome or date-dependent model artifacts. Compare missing indicators, multiple imputation, predictive imputation and complete-case models. Use sensitivity analyses for unobserved quality and selective missingness. Do not use mortgage rates, unemployment, CPI, or rents as direct quality controls; they can be used for external validation and explanatory analysis.

## 9. Sparse markets
For every municipality/type/month compute actual sales count, effective sample size, local information contribution, posterior interval and reliability category. Distinguish directly supported estimates, model-assisted estimates and insufficiently identified cells. A model estimate with zero local transactions is not an observed monthly price movement. Suppress or clearly label cells that fail minimum reliability or overlap requirements. Assess how pooling weights and prior assumptions influence small-town results. Avoid fabricated precision.

## 10. Transaction selection and robustness
Analyze representativeness of registered sales, arms-length status, price outliers, unusual transfers, new versus existing construction, property-type drift and changing coverage. Use robust M-estimation, Student-t errors, influence diagnostics, quantile sensitivity and trimming sensitivity. Test whether estimated changes reflect new transaction coverage rather than market dynamics. Do not use a simplistic unconditional percentile trim as the sole outlier procedure.

## 11. Real-time publication and revisions
Build two index modes:
(1) retrospective research series estimated using the complete available sample;
(2) real-time vintage series estimated using only information available at each historical publication cutoff.
Simulate monthly releases, late deed registrations, incomplete final months and backfilled transactions. The October 2026 observation is partial through the actual last reliable transaction date; do not present it as a completed month. Evaluate freeze windows, revision magnitudes, movement splice, mean splice and revision-limited policies. Report publication lag, timeliness and revision bias.

## 12. Simulations — mandatory before paper claims
Create realistic synthetic Quebec-like housing markets with known latent quality-constant price paths, heterogeneous houses, varying municipality sizes, spatial dependence, seasonal sales, missing characteristics, nonlinear hedonic effects, neighborhood composition shifts, nonrandom transactions, structural breaks, and delayed registrations.
Run Monte Carlo experiments across multiple sample sizes and data-generating processes. Include flat-price markets where sold-home quality increases, and rising-price markets where sold-home quality decreases. Evaluate RMSE and bias of monthly returns, coverage of 95% intervals, turning-point detection, temporal lag, revision size, spatial spillover bias, computational cost, calibration and aggregate coherence. Report Monte Carlo uncertainty. Include deliberate counterexamples where SHARP-HPI underperforms simpler alternatives.

## 13. Empirical validation
Use forward-chaining and rolling-origin backtesting, never random train-test splits for temporal index claims. Validate property-level predictions as a secondary diagnostic only. Compare aggregate series to suitable external indices, carefully harmonizing geography, property scope, vintage and base period. Validate against matched repeat-sale pairs where possible. Conduct placebo tests, synthetic quality-composition shocks, residual spatial autocorrelation tests, bootstrap or posterior predictive checks, sensitivity to municipal borders and reference baskets, and sparse-market stress tests. Quantify which components actually create improvements through ablation studies.

## 14. Statistical inference
Produce uncertainty intervals for levels and changes. Propagate hedonic parameter uncertainty, latent temporal uncertainty, sampling variation, imputation uncertainty and spatial pooling uncertainty as feasible. Check empirical coverage through simulation. Use spatially and temporally appropriate bootstrap designs or posterior draws. Correct multiple testing for large-scale municipality comparisons where hypothesis tests are made. Distinguish frequentist confidence intervals from Bayesian credible intervals.

## 15. Provincial aggregation
Define coherent aggregation weights, ideally grounded in eligible housing stock or another defensible target population, not simply observed monthly sales. Compare stock-weighted, transaction-weighted and equal-municipality aggregation. Make explicit whether the headline is a price index for the stock, for transacted properties, or for a representative constant-quality basket. Demonstrate the relationship between municipal, MRC, regional and provincial indices. Report aggregation sensitivity and avoid forcing equality when estimands differ.

## 16. Research outputs
Create:
- data_dictionary.md and provenance_report.md
- cleaned transaction pipeline and exclusion audit
- model specification and mathematical derivation
- reproducible benchmark and SHARP estimators
- Monte Carlo experiment framework and results
- complete monthly index panel in Parquet and CSV
- municipality/type coverage and reliability tables
- index vintage/revision database
- tables and figures generated exclusively from actual outputs
- interactive geographic diagnostics if practical
- research limitations and reproducibility documentation
- complete English LaTeX manuscript with BibTeX references.

## 17. Index schema
Each index row should contain:
geography_id, geography_name, geography_level, property_type, month, index_value, base_period, log_monthly_change, monthly_pct_change, annual_pct_change, actual_transaction_count, effective_sample_size, local_data_weight, lower_95, upper_95, reliability_grade, direct_or_model_assisted, data_cutoff, publication_vintage, model_version, index_method, partial_month_flag, suppression_flag.
Never confuse local_data_weight with a formally identified probability or quality score.

## 18. Paper structure
Title page: WP22, UQO Working Paper Series, author Simon-Pierre Boucher, affiliations verified before publication.
Abstract (200–250 words), keywords, JEL codes R31, C43, C11, C21, C55 (verify relevance).
1 Introduction and contributions.
2 Literature review and identification of research gap.
3 Quebec transaction data and institutional setting.
4 Price-index estimand and measurement challenges.
5 SHARP-HPI theoretical framework and identification.
6 Estimation, inference, linking and publication vintages.
7 Simulation design and Monte Carlo results.
8 Quebec monthly empirical indices, 2021–2026.
9 Benchmarks, validation, ablation and robustness.
10 Economic implications and applications.
11 Limitations and conclusion.
Appendices: proofs, algorithms, priors, data cleaning, geographic concordance, simulations, supplemental figures and tables.

## 19. Literature
Search and verify primary academic sources on hedonic indices, rolling time dummies, Fisher/GEKS, HMTS, repeat-sales, hierarchical small-area estimation, spatial econometrics, state-space models, real-time index revisions and housing composition bias. Start with Hill et al. (2022) on RTD, Ishaak et al. (2024) on HMTS, and Hill (2026) on real estate price measurement. Record exact DOIs, publication years and bibliographic metadata. Never invent references. Explain exactly how the proposed contribution differs from each close predecessor.

## 20. Software and engineering
Prefer Python 3.12+, DuckDB/Polars/PyArrow, PostgreSQL/PostGIS, statsmodels, scipy, scikit-learn, PyMC or equivalent scalable probabilistic tools, and LaTeX/BibLaTeX. Use deterministic seeds, versioned configurations, unit tests, integration tests, data contracts, environment lockfiles and cached intermediate artifacts. Use streaming/chunked computation when necessary. Profile memory and runtime. Keep raw transaction records private; publish only legally permissible aggregate results. Never commit credentials or personally identifying records.

## 21. Work sequence
Phase 0: discover existing repository, datasets and infrastructure; write an evidence-based plan.
Phase 1: data inventory, provenance, coverage and transaction cleaning.
Phase 2: reproduce benchmark indices.
Phase 3: define SHARP mathematical model and implement candidate estimators.
Phase 4: Monte Carlo experiments and estimator selection.
Phase 5: complete monthly empirical index panel.
Phase 6: validation, real-time vintages, robustness and ablation.
Phase 7: LaTeX manuscript, figures, tables, appendices and final reproducibility audit.
At every phase save machine-readable artifacts and an explicit decision log. If data access is unavailable, implement testable code and synthetic-data simulations, but never pretend real-data analysis has been completed.

## 22. Final quality gate
No paper can be declared complete until all figures and tables are reproducible, every empirical claim is linked to a generated result, all citations are verified, benchmarks have run, Monte Carlo tests are documented, uncertainty is calibrated, geographic coverage is disclosed, sparse cells are labeled, leakage tests pass, the manuscript compiles without substantive errors, and the results have been critically reviewed for overclaiming. The goal is a publishable contribution, not an impressive-sounding index name.

Begin immediately with repository and data discovery, then produce the initial audit and research plan. Continue autonomously through feasible phases without inventing missing information.