# 📑 Table of Contents

• Overview
• Key Findings
• Repository Structure
• Data
• Methodology at a Glance
• The Analysis Pipeline
• Installation & Setup
• Reproducing the Results
• Building the Paper
• Figures
• Results Summary
• Verification & Integrity
• Project History & Provenance
• Citation
• Contact & License

# 🔎 Overview

This repository contains the complete research compendium — analysis code, LaTeX source, publication-ready figures, and machine-readable result tables — for UQO Working Paper No. 3, which compares three frameworks for hedonic housing valuation on a single large dataset of 788,842 active Zillow listings spanning all 50 U.S. states and the District of Columbia:

The paper's core methodological contribution is a systematic study of spatial leakage in hedonic model evaluation: random train/test splits let geographically proximate listings appear on both sides of the split, inflating performance metrics. Three complementary designs quantify the damage — a 10-state geographic holdout, a 6-stage feature-ablation cascade, and a latitude/longitude augmentation experiment.

⚠️ Interpretation caveat. All estimates are listing-price capitalization gradients — conditional associations between attributes and asking prices. They are not causal willingness-to-pay parameters and not transaction-price implicit prices.

# 🏆 Key Findings

1. Geography does the heavy lifting in OLS. Moving from 4 Census-region dummies to state fixed effects to 886 ZIP3 fixed effects raises out-of-sample $R^2$ from 0.630 → 0.678 → 0.725 — a 9.5 pp gain from spatial granularity alone.

2. Strong, stable residual spatial autocorrelation. Moran's $I$ on OLS residuals averages 0.2745 (SD 0.0076) across three independent 5,000-listing subsamples (row-standardized KNN weights, $k = 8$; all $p < 0.001$).

3. Mean effects mask distributional heterogeneity. The garage gradient is 10× larger at $\tau = 0.10$ than at $\tau = 0.90$ ($z = 28.9$); the pool premium only emerges above the median; the age penalty is concentrated in lower-priced homes.

4. Random validation flatters machine learning. XGBoost reaches $R^2 = 0.833$ under a random 80/20 split but only 0.425–0.547 when 10 entire states (438,315 listings) are held out.

5. Geographic features can hurt generalization. Removing all geographic features improves geographic-holdout $R^2$ from 0.425 to 0.519, while adding raw lat/lon coordinates boosts random $R^2$ (+3.7 pp) but degrades geographic holdout. Region dummies memorize training-set price levels.

6. SHAP rankings are model-stable but not structural. Spearman rank correlations of mean $|\phi|$ across XGBoost / LightGBM / Random Forest range from 0.89 to 0.99, with the same six features on top — yet these remain predictive decompositions, not implicit prices.

7. Imputation matters. The lot-size gradient triples (0.018 → 0.059) when state-median-imputed lot sizes are dropped — a warning for hedonic work on scraped listing data.

# 🗂 Repository Structure

wp3_uqo/
├── README.md                      ← you are here
├── AUDIT.md                       ← forensic audit of the original project archive
├── CHANGES.md                     ← every move/refactor/fix made during restructuring
├── requirements.txt               ← pinned Python dependencies (validated 2026-08-05)
│
├── data/                          ⚠ NOT in the Git repo (3.2 GB — see "Data" below)
│   ├── raw/
│   │   └── us_housing.duckdb      # raw Zillow extract (839,313 × 116 + 4 aux tables)
│   └── processed/
│       ├── analytical_sample.pkl  # final sample, 788,842 × 68 engineered columns
│       ├── model_data.pkl         # standardized X (788,842 × 63), y, OLS fit, residuals
│       └── shap_data.pkl          # TreeSHAP values (10,000 × 62) + explanation sample
│
├── src/wp3/                       # shared library code
│   ├── config.py                  # paths (relative, WP3_ROOT-overridable), constants,
│   │                              #   feature-block definitions, holdout states, seeds
│   ├── data.py                    # loaders + 62-regressor design-matrix builder
│   ├── models.py                  # XGBoost training helper (dual validation schemes)
│   └── plotting.py                # journal figure style + human-readable variable labels
│
├── scripts/                       # numbered pipeline entry points
│   ├── 01_spatial_robustness.py   # ZIP3 FE OLS · Moran's I ·  XGB ± geography · ablation
│   ├── 02_qr_imputation.py        # QR stability · inter-quantile Wald · imputation ·
│   │                              #   winsorization sensitivity
│   ├── 03_make_figures.py         # regenerates all 11 figures + verification stats
│   └── 04_export_tables.py        # exports paper tables → results/tables/*.csv
│
├── figures/                       # all 11 paper figures (PNG, 300 dpi, unified style)
│
├── results/
│   ├── *.pkl                      ⚠ NOT in the Git repo (up to 437 MB)
│   └── tables/                    # 13 CSVs mirroring the paper's tables (in repo)
│
└── paper/                         # LaTeX source
    ├── main.tex                   # preamble + metadata + \input skeleton
    ├── references.bib             # 37 BibTeX entries (natbib + apalike)
    ├── sections/                  # one file per section (IMRaD + robustness)
    │   ├── titlepage.tex          ├── introduction.tex   ├── literature.tex
    │   ├── data.tex               ├── methodology.tex    ├── results.tex
    │   ├── robustness.tex         ├── discussion.tex     ├── limitations.tex
    │   └── conclusion.tex
    ├── appendix/appendix.tex      # extra figures, full OLS table, reproducibility checklist
    ├── Makefile                   # latexmk targets (all / clean / distclean)
    ├── uq_logo.jpg
    └── main.pdf                   # compiled paper — 52 pages, zero warnings

# 💾 Data

# What the data are

• Source: Zillow active for-sale listings, 2025–2026 snapshot.
• Raw: 839,313 residential properties × 116 variables, stored in a DuckDB database (properties, price_history, schools, tax_history, meta_columns).
• Analytical sample: 788,842 listings after five sequential filters (valid price $10K–$10M, living area 200–20,000 sqft, 1–10 beds/baths, valid coordinates, US states + DC only — 6.0 % attrition).
• 68 engineered columns: log transforms, age & age², ratios, amenity dummies, neighborhood scores, market-status flags, 6 interaction terms, and simplified categoricals (roof / construction / foundation / region).

# Why the data are not in this Git repository

GitHub rejects files above 100 MB. The DuckDB file is 1.6 GB and the processed pickles reach 867 MB, so data/ and results/*.pkl are excluded via .gitignore. The canonical copies live in the local research archive:

📬 Data access: available on request (subject to Zillow Terms of Service) — contact@spboucher.ai.

# 🧪 Methodology at a Glance

# 🔁 The Analysis Pipeline

us_housing.duckdb  (raw, 839,313 listings)
        │
        │  [sample construction & feature engineering — original script lost;
        │   the stored analytical_sample.pkl is the canonical artifact]
        ▼
analytical_sample.pkl  (788,842 × 68)
        │
        ├─ [estimation — original scripts lost; stored artifacts are canonical]
        │      ├── model_data.pkl        (standardized X, OLS fit, residuals)
        │      ├── qr_results.pkl        (QR params at 5 quantiles)
        │      ├── ml_results.pkl        (fitted XGB/LGBM + test predictions)
        │      ├── shap_data.pkl         (TreeSHAP values, 10,000 × 62)
        │      ├── extended_results.pkl  (unstandardized OLS, geo holdout, RF)
        │      └── v3_shap_stability.pkl (XGB/LGB/RF SHAP rank correlations)
        │
        ├─ scripts/01_spatial_robustness.py ──► results/v3_spatial_results.pkl
        ├─ scripts/02_qr_imputation.py ───────► results/v3_qr_imputation_results.pkl
        ├─ scripts/03_make_figures.py ────────► figures/fig1…fig11 (+ verification)
        └─ scripts/04_export_tables.py ───────► results/tables/*.csv
                                                       │
                                                       ▼
                                          paper/main.tex ──► main.pdf (52 p.)

The bracketed upstream stages were produced before this restructuring by scripts that no longer exist (see Project History); their outputs are preserved and every downstream number is verified against them.

# ⚙️ Installation & Setup

Prerequisites: Python ≥ 3.11, a TeX Live distribution (with newtx, natbib, booktabs, threeparttable), and ~4 GB of free disk for the data artifacts.

git clone https://github.com/spboucher-ai/wp3-hedonic-housing-us.git
cd wp3-hedonic-housing-us

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Then place the data artifacts (request them at contact@spboucher.ai) into data/raw/, data/processed/, and results/ as shown in Repository Structure.

All paths resolve relative to the repository root. To point the code at another location, set the environment variable:

export WP3_ROOT=/path/to/artifacts

# ▶️ Reproducing the Results

* Apple Silicon, 32 GB RAM; scripts use all cores (n_jobs=-1).

Every script prints verification statistics comparing recomputed values with the numbers reported in the paper (see Verification).

# 📄 Building the Paper

cd paper
latexmk          # pdflatex ×N + bibtex, via Makefile/latexmkrc defaults
# or: make       # same thing
# or: make clean / make distclean

• Output: paper/main.pdf — 52 pages, compiles with zero errors, zero undefined references/citations.
• Figures are pulled from ../figures/ via \graphicspath.
• Bibliography: paper/references.bib (37 entries), natbib + apalike.
• Metadata (title, abstract, keywords R31/C21/C45/C52, version) is centralized in main.tex \WP* macros.

# 🖼 Figures

All figures are regenerated from the stored artifacts by scripts/03_make_figures.py in a unified publication style (serif typography matched to the paper's newtx text font, muted colorblind-safe palette, no chart junk, 300 dpi).

# 📊 Results Summary

Predictive performance (log-price scale):

Ablation cascade (XGBoost): structural 0.498 → +lot 0.546 → +amenities 0.638 → +neighborhood 0.814 (+17.6 pp) → +market 0.824 → full 0.833 (random $R^2$).

Machine-readable versions of these and all other paper tables are in results/tables/ (13 CSV files: OLS coefficients, QR paths, inter-quantile Wald tests, stability CVs, Moran's I, imputation/winsorization sensitivity, SHAP importance & cross-model correlations, ZIP3 FE, ablation).

# ✅ Verification & Integrity

The restructuring included a systematic audit reconciling the manuscript against the stored result artifacts. Highlights:

• Exact matches — OLS residual diagnostics (skewness 0.18, kurtosis 5.55, Jarque–Bera 217,078), test-set $R^2$ (OLS 0.6301, XGBoost 0.8330, LightGBM 0.8088), Moran's $I$ subsample values and mean/SD, QR coefficients at all five quantiles, and the OLS reference line (0.312) in Figure 3.
• End-to-end reproduction — re-running the refactored Moran's $I$ pipeline reproduces the stored values to 4 decimal places (0.2742 / 0.2839 / 0.2652).
• Manuscript corrections — six internal inconsistencies were found and fixed (wrong KNN $k$ in the text, mislabeled inter-quantile sign convention, winsorization description not matching the actual procedure, a duplicated table row, mixed in/out-of-sample $R^2$ in the summary matrix). Every fix aligns the text to the stored results and is itemized in CHANGES.md §4. No scientific result was altered.
• Completed placeholders — 13 [TODO] table cells in the original manuscript were filled with the exact values from the result pickles.

See AUDIT.md for the full audit and CHANGES.md for the change log.

# 🕰 Project History & Provenance

• The project was completed in May 2026 in a flat working folder (immo-wp3-spb-20260519), then restructured into this compendium on 2026-08-05. The original archive is preserved untouched.
• The scripts that built the analytical sample and estimated the primary models (OLS/QR/ML/SHAP) were lost prior to restructuring; only two robustness scripts survived and were refactored into scripts/01 and scripts/02. The stored pickles are the canonical record of those stages and every downstream number is validated against them.
• Five paper figures had been deleted from the archive; they were rebuilt from the stored data, and subsequently all 11 figures were regenerated in a unified journal style.
• Three earlier monolithic drafts (paper_hedonic_pricing{,_v2,_v3}.tex) precede the sectioned wp3/ source that became paper/.

# 📖 Citation

If you use this code, the figures, or the results, please cite:

@techreport{boucher2026hedonic,
  author      = {Boucher, Simon-Pierre},
  title       = {Hedonic Housing Price Models for the United States:
                 A Multi-Method Comparison of Parametric, Quantile,
                 and Machine Learning Approaches},
  institution = {Universit\'e du Qu\'ebec en Outaouais,
                 D\'epartement des sciences administratives},
  type        = {Working Paper},
  number      = {3},
  year        = {2026},
  month       = {May}
}

# 📬 Contact & License

Simon-Pierre Boucher Département des sciences administratives Université du Québec en Outaouais — Gatineau (Québec), Canada 📧 contact@spboucher.ai · 🎓 simon-pierre.boucher@uqo.ca

© 2026 Simon-Pierre Boucher. Analysis code is shared for replication and review purposes. The underlying Zillow data are subject to Zillow's Terms of Service and are not redistributed in this repository; the paper (paper/main.pdf) is a working paper — please cite rather than redistribute.