| 1 |
|
−<div align="center"> |
| 2 |
|
− |
|
1 |
+<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca --> |
| 3 |
2 |
# 🏠 Hedonic Housing Price Models for the United States |
| 4 |
3 |
|
| 5 |
|
−### A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches |
| 6 |
|
− |
| 7 |
|
−**UQO Working Paper No. 3** |
| 8 |
|
− |
| 9 |
|
−[](https://www.python.org/) |
| 10 |
|
−[](https://www.latex-project.org/) |
| 11 |
|
−[](https://xgboost.readthedocs.io/) |
| 12 |
|
−[](https://lightgbm.readthedocs.io/) |
| 13 |
|
−[](https://shap.readthedocs.io/) |
| 14 |
|
−[](https://www.statsmodels.org/) |
| 15 |
|
−[](https://duckdb.org/) |
|
4 |
+**UQO Working Paper No. 3** — *A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches* |
| 16 |
5 |
|
| 17 |
|
−[](paper/main.pdf) |
| 18 |
|
−[](#-data) |
| 19 |
|
−[](#-data) |
| 20 |
|
−[](#-figures) |
| 21 |
|
−[-2E8B57)](paper/references.bib) |
| 22 |
|
−[](#-verification--integrity) |
| 23 |
|
−[](#-citation) |
|
6 |
+[-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) |
|
7 |
+[](requirements.txt) |
|
8 |
+[](#-data) |
|
9 |
+[](src/wp3/) |
|
10 |
+[](scripts/) |
|
11 |
+[](https://uqo.ca) |
|
12 |
+[](mailto:simon-pierre.boucher@uqo.ca) |
| 24 |
13 |
|
| 25 |
|
−**Author : [Simon-Pierre Boucher](mailto:contact@spboucher.ai)** |
| 26 |
|
−Département des sciences administratives — Université du Québec en Outaouais (UQO) |
| 27 |
|
−📧 **contact@spboucher.ai** |
| 28 |
|
− |
| 29 |
|
−</div> |
|
14 |
+> **TL;DR** — Three hedonic frameworks are compared on **788,842 active Zillow listings** covering all 50 U.S. states and |
|
15 |
+> DC. Semi-log OLS with 62 regressors gives R² = 0.634 (0.725 with 886 ZIP3 fixed effects; Moran's *I* = 0.27 on |
|
16 |
+> residuals). Quantile regression rejects coefficient equality between τ = 0.10 and 0.90 for **11 of 13** key variables. |
|
17 |
+> XGBoost reaches R² = 0.833 under a random split but only **0.425–0.547** when ten entire states are held out, and |
|
18 |
+> *removing* geographic features improves geographic-holdout R² to 0.519 — random validation flatters machine learning |
|
19 |
+> through spatial leakage. |
| 30 |
20 |
|
| 31 |
21 |
--- |
| 32 |
22 |
|
| 33 |
|
−## 📑 Table of Contents |
| 34 |
|
− |
| 35 |
|
−- [Overview](#-overview) |
| 36 |
|
−- [Key Findings](#-key-findings) |
| 37 |
|
−- [Repository Structure](#-repository-structure) |
| 38 |
|
−- [Data](#-data) |
| 39 |
|
−- [Methodology at a Glance](#-methodology-at-a-glance) |
| 40 |
|
−- [The Analysis Pipeline](#-the-analysis-pipeline) |
| 41 |
|
−- [Installation & Setup](#-installation--setup) |
| 42 |
|
−- [Reproducing the Results](#-reproducing-the-results) |
| 43 |
|
−- [Building the Paper](#-building-the-paper) |
| 44 |
|
−- [Figures](#-figures) |
| 45 |
|
−- [Results Summary](#-results-summary) |
| 46 |
|
−- [Verification & Integrity](#-verification--integrity) |
| 47 |
|
−- [Project History & Provenance](#-project-history--provenance) |
| 48 |
|
−- [Citation](#-citation) |
| 49 |
|
−- [Contact & License](#-contact--license) |
|
23 |
+## 📌 Status |
| 50 |
24 |
|
| 51 |
|
−--- |
|
25 |
+| Item | Value | |
|
26 |
+|---|---| |
|
27 |
+| Version | 1.1 (May 2026) — cover page harmonised to the UQO WP standard, October 2026 | |
|
28 |
+| SSRN | not yet submitted | |
|
29 |
+| Journal | not yet submitted | |
|
30 |
+| Code & data | figures and tables reproducible from stored artifacts; raw Zillow extract (3.2 GB) available on request | |
| 52 |
31 |
|
| 53 |
|
−## 🔎 Overview |
|
32 |
+## 🎯 What this paper does |
| 54 |
33 |
|
| 55 |
|
−This repository contains the complete research compendium — analysis code, LaTeX |
| 56 |
|
−source, publication-ready figures, and machine-readable result tables — for |
| 57 |
|
−**UQO Working Paper No. 3**, which compares three frameworks for hedonic housing |
| 58 |
|
−valuation on a single large dataset of **788,842 active Zillow listings** spanning |
| 59 |
|
−all 50 U.S. states and the District of Columbia: |
|
34 |
+The paper compares three frameworks for hedonic housing valuation on a single large dataset: |
| 60 |
35 |
|
| 61 |
36 |
| Framework | Question it answers | Headline result | |
| 62 |
37 |
|---|---|---| |
| 63 |
|
−| **Semi-log OLS** (62 regressors, HC3 SE) | What is the *average* conditional association between an attribute and log listing price? | $R^2 = 0.634$; price-to-area elasticity 0.63 | |
| 64 |
|
−| **Quantile regression** ($\tau \in \{0.10, 0.25, 0.50, 0.75, 0.90\}$) | How do attribute gradients *vary across the price distribution*? | Coefficient equality rejected for **11 of 13** variables (inter-quantile Wald tests) | |
| 65 |
|
−| **Gradient boosting + SHAP** (XGBoost, LightGBM) | How well can prices be *predicted*, and which features drive predictions? | $R^2 = 0.833$ (random split) vs. **0.425–0.547** (geographic holdout) | |
| 66 |
|
− |
| 67 |
|
−The paper's core methodological contribution is a systematic study of |
| 68 |
|
−**spatial leakage** in hedonic model evaluation: random train/test splits let |
| 69 |
|
−geographically proximate listings appear on both sides of the split, inflating |
| 70 |
|
−performance metrics. Three complementary designs quantify the damage — a 10-state |
| 71 |
|
−geographic holdout, a 6-stage feature-ablation cascade, and a latitude/longitude |
| 72 |
|
−augmentation experiment. |
| 73 |
|
− |
| 74 |
|
−> ⚠️ **Interpretation caveat.** All estimates are *listing-price capitalization |
| 75 |
|
−> gradients* — conditional associations between attributes and *asking* prices. |
| 76 |
|
−> They are not causal willingness-to-pay parameters and not transaction-price |
| 77 |
|
−> implicit prices. |
| 78 |
|
− |
| 79 |
|
−--- |
| 80 |
|
− |
| 81 |
|
−## 🏆 Key Findings |
| 82 |
|
− |
| 83 |
|
−1. **Geography does the heavy lifting in OLS.** Moving from 4 Census-region |
| 84 |
|
− dummies to state fixed effects to 886 ZIP3 fixed effects raises out-of-sample |
| 85 |
|
− $R^2$ from 0.630 → 0.678 → 0.725 — a 9.5 pp gain from spatial granularity alone. |
| 86 |
|
− |
| 87 |
|
−2. **Strong, stable residual spatial autocorrelation.** Moran's $I$ on OLS |
| 88 |
|
− residuals averages **0.2745** (SD 0.0076) across three independent 5,000-listing |
| 89 |
|
− subsamples (row-standardized KNN weights, $k = 8$; all $p < 0.001$). |
| 90 |
|
− |
| 91 |
|
−3. **Mean effects mask distributional heterogeneity.** The garage gradient is |
| 92 |
|
− **10× larger** at $\tau = 0.10$ than at $\tau = 0.90$ ($z = 28.9$); the pool |
| 93 |
|
− premium only emerges above the median; the age penalty is concentrated in |
| 94 |
|
− lower-priced homes. |
| 95 |
|
− |
| 96 |
|
−4. **Random validation flatters machine learning.** XGBoost reaches $R^2 = 0.833$ |
| 97 |
|
− under a random 80/20 split but only **0.425–0.547** when 10 entire states |
| 98 |
|
− (438,315 listings) are held out. |
| 99 |
|
− |
| 100 |
|
−5. **Geographic features can *hurt* generalization.** Removing all geographic |
| 101 |
|
− features *improves* geographic-holdout $R^2$ from 0.425 to **0.519**, while |
| 102 |
|
− adding raw lat/lon coordinates boosts random $R^2$ (+3.7 pp) but *degrades* |
| 103 |
|
− geographic holdout. Region dummies memorize training-set price levels. |
| 104 |
|
− |
| 105 |
|
−6. **SHAP rankings are model-stable but not structural.** Spearman rank |
| 106 |
|
− correlations of mean $|\phi|$ across XGBoost / LightGBM / Random Forest range |
| 107 |
|
− from **0.89 to 0.99**, with the same six features on top — yet these remain |
| 108 |
|
− predictive decompositions, not implicit prices. |
| 109 |
|
− |
| 110 |
|
−7. **Imputation matters.** The lot-size gradient **triples** (0.018 → 0.059) when |
| 111 |
|
− state-median-imputed lot sizes are dropped — a warning for hedonic work on |
| 112 |
|
− scraped listing data. |
| 113 |
|
− |
| 114 |
|
−--- |
| 115 |
|
− |
| 116 |
|
−## 🗂 Repository Structure |
|
38 |
+| **Semi-log OLS** (62 regressors, HC3 SE) | Average conditional association between an attribute and log listing price | R² = 0.634; price-to-area elasticity 0.63 | |
|
39 |
+| **Quantile regression** (τ ∈ {0.10, 0.25, 0.50, 0.75, 0.90}) | How attribute gradients vary across the price distribution | Equality rejected for 11 of 13 variables | |
|
40 |
+| **Gradient boosting + SHAP** (XGBoost, LightGBM) | How well prices can be predicted, and which features drive predictions | R² = 0.833 (random) vs. 0.425–0.547 (geographic holdout) | |
| 117 |
41 |
|
| 118 |
42 |
``` |
| 119 |
|
−wp3_uqo/ |
| 120 |
|
−├── README.md ← you are here |
| 121 |
|
−├── AUDIT.md ← forensic audit of the original project archive |
| 122 |
|
−├── CHANGES.md ← every move/refactor/fix made during restructuring |
| 123 |
|
−├── requirements.txt ← pinned Python dependencies (validated 2026-08-05) |
| 124 |
|
−│ |
| 125 |
|
−├── data/ ⚠ NOT in the Git repo (3.2 GB — see "Data" below) |
| 126 |
|
−│ ├── raw/ |
| 127 |
|
−│ │ └── us_housing.duckdb # raw Zillow extract (839,313 × 116 + 4 aux tables) |
| 128 |
|
−│ └── processed/ |
| 129 |
|
−│ ├── analytical_sample.pkl # final sample, 788,842 × 68 engineered columns |
| 130 |
|
−│ ├── model_data.pkl # standardized X (788,842 × 63), y, OLS fit, residuals |
| 131 |
|
−│ └── shap_data.pkl # TreeSHAP values (10,000 × 62) + explanation sample |
| 132 |
|
−│ |
| 133 |
|
−├── src/wp3/ # shared library code |
| 134 |
|
−│ ├── config.py # paths (relative, WP3_ROOT-overridable), constants, |
| 135 |
|
−│ │ # feature-block definitions, holdout states, seeds |
| 136 |
|
−│ ├── data.py # loaders + 62-regressor design-matrix builder |
| 137 |
|
−│ ├── models.py # XGBoost training helper (dual validation schemes) |
| 138 |
|
−│ └── plotting.py # journal figure style + human-readable variable labels |
| 139 |
|
−│ |
| 140 |
|
−├── scripts/ # numbered pipeline entry points |
| 141 |
|
−│ ├── 01_spatial_robustness.py # ZIP3 FE OLS · Moran's I · XGB ± geography · ablation |
| 142 |
|
−│ ├── 02_qr_imputation.py # QR stability · inter-quantile Wald · imputation · |
| 143 |
|
−│ │ # winsorization sensitivity |
| 144 |
|
−│ ├── 03_make_figures.py # regenerates all 11 figures + verification stats |
| 145 |
|
−│ └── 04_export_tables.py # exports paper tables → results/tables/*.csv |
| 146 |
|
−│ |
| 147 |
|
−├── figures/ # all 11 paper figures (PNG, 300 dpi, unified style) |
| 148 |
|
−│ |
| 149 |
|
−├── results/ |
| 150 |
|
−│ ├── *.pkl ⚠ NOT in the Git repo (up to 437 MB) |
| 151 |
|
−│ └── tables/ # 13 CSVs mirroring the paper's tables (in repo) |
| 152 |
|
−│ |
| 153 |
|
−└── paper/ # LaTeX source |
| 154 |
|
− ├── main.tex # preamble + metadata + \input skeleton |
| 155 |
|
− ├── references.bib # 37 BibTeX entries (natbib + apalike) |
| 156 |
|
− ├── sections/ # one file per section (IMRaD + robustness) |
| 157 |
|
− │ ├── titlepage.tex ├── introduction.tex ├── literature.tex |
| 158 |
|
− │ ├── data.tex ├── methodology.tex ├── results.tex |
| 159 |
|
− │ ├── robustness.tex ├── discussion.tex ├── limitations.tex |
| 160 |
|
− │ └── conclusion.tex |
| 161 |
|
− ├── appendix/appendix.tex # extra figures, full OLS table, reproducibility checklist |
| 162 |
|
− ├── Makefile # latexmk targets (all / clean / distclean) |
| 163 |
|
− ├── uq_logo.jpg |
| 164 |
|
− └── main.pdf # compiled paper — 52 pages, zero warnings |
|
43 |
+ln P_i = α + x_i'β + μ_g(i) + ε_i g = Census region → state → ZIP3 |
| 165 |
44 |
``` |
| 166 |
45 |
|
| 167 |
|
−--- |
| 168 |
|
− |
| 169 |
|
−## 💾 Data |
| 170 |
|
− |
| 171 |
|
−### What the data are |
| 172 |
|
− |
| 173 |
|
−- **Source:** Zillow active for-sale listings, 2025–2026 snapshot. |
| 174 |
|
−- **Raw:** 839,313 residential properties × 116 variables, stored in a DuckDB |
| 175 |
|
− database (`properties`, `price_history`, `schools`, `tax_history`, `meta_columns`). |
| 176 |
|
−- **Analytical sample:** 788,842 listings after five sequential filters |
| 177 |
|
− (valid price \$10K–\$10M, living area 200–20,000 sqft, 1–10 beds/baths, |
| 178 |
|
− valid coordinates, US states + DC only — 6.0 % attrition). |
| 179 |
|
−- **68 engineered columns:** log transforms, age & age², ratios, amenity dummies, |
| 180 |
|
− neighborhood scores, market-status flags, 6 interaction terms, and simplified |
| 181 |
|
− categoricals (roof / construction / foundation / region). |
| 182 |
|
− |
| 183 |
|
−### Why the data are not in this Git repository |
| 184 |
|
− |
| 185 |
|
−GitHub rejects files above 100 MB. The DuckDB file is **1.6 GB** and the processed |
| 186 |
|
−pickles reach **867 MB**, so `data/` and `results/*.pkl` are excluded via |
| 187 |
|
−`.gitignore`. The canonical copies live in the local research archive: |
| 188 |
|
− |
| 189 |
|
−| Artifact | Size | Location | |
| 190 |
|
−|---|---|---| |
| 191 |
|
−| `us_housing.duckdb` | 1.6 GB | `data/raw/` (local) | |
| 192 |
|
−| `analytical_sample.pkl` | 457 MB | `data/processed/` (local) | |
| 193 |
|
−| `model_data.pkl` | 867 MB | `data/processed/` (local) | |
| 194 |
|
−| `ml_results.pkl` | 437 MB | `results/` (local) | |
| 195 |
|
−| `shap_data.pkl`, `qr_results.pkl`, `extended_results.pkl`, `v3_*.pkl` | < 10 MB each | `results/` (local; small ones could be added on request) | |
| 196 |
|
− |
| 197 |
|
−📬 **Data access:** available on request (subject to Zillow Terms of Service) — |
| 198 |
|
−**contact@spboucher.ai**. |
| 199 |
|
− |
| 200 |
|
−--- |
| 201 |
|
− |
| 202 |
|
−## 🧪 Methodology at a Glance |
| 203 |
|
− |
| 204 |
|
−| Component | Specification | |
| 205 |
|
−|---|---| |
| 206 |
|
−| Dependent variable | $\ln(\text{listing price})$ | |
| 207 |
|
−| OLS | 62 regressors, standardized + unstandardized variants, **HC3** robust SE | |
| 208 |
|
−| Fixed effects | Census region (baseline) → state (50) → ZIP3 (886) | |
| 209 |
|
−| Quantile regression | statsmodels `QuantReg`, $\tau \in \{0.10,\ldots,0.90\}$, 150,000-obs subsample, 10-seed stability check | |
| 210 |
|
−| Inter-quantile tests | Wald $z$-tests on $\hat\beta(0.10)-\hat\beta(0.90)$ | |
| 211 |
|
−| XGBoost | 1,000 trees, depth 8, lr 0.05, subsample/colsample 0.8, early stopping 50 | |
| 212 |
|
−| LightGBM / RF / Ridge / Lasso / Elastic Net | benchmark suite | |
| 213 |
|
−| Validation | random 80/20 (seed 42) **and** 10-state geographic holdout (CA NY TX FL OH CO NC WA IL GA — 438,315 test obs) | |
| 214 |
|
−| Ablation | 6 nested feature sets: structural → +lot → +amenities → +neighborhood → +market → full | |
| 215 |
|
−| Interpretation | TreeSHAP on 10,000 test obs; cross-model Spearman stability (XGB/LGB/RF) | |
| 216 |
|
−| Spatial diagnostics | Moran's $I$, row-standardized KNN ($k=8$), 3 × 5,000-obs subsamples, 999 permutations | |
| 217 |
|
−| Sensitivity | state-median-imputation drop-out, lot-size winsorization (p99.5), Bike Score cap | |
| 218 |
|
− |
| 219 |
|
−--- |
| 220 |
|
− |
| 221 |
|
−## 🔁 The Analysis Pipeline |
| 222 |
|
− |
| 223 |
|
−``` |
| 224 |
|
−us_housing.duckdb (raw, 839,313 listings) |
| 225 |
|
− │ |
| 226 |
|
− │ [sample construction & feature engineering — original script lost; |
| 227 |
|
− │ the stored analytical_sample.pkl is the canonical artifact] |
| 228 |
|
− ▼ |
| 229 |
|
−analytical_sample.pkl (788,842 × 68) |
| 230 |
|
− │ |
| 231 |
|
− ├─ [estimation — original scripts lost; stored artifacts are canonical] |
| 232 |
|
− │ ├── model_data.pkl (standardized X, OLS fit, residuals) |
| 233 |
|
− │ ├── qr_results.pkl (QR params at 5 quantiles) |
| 234 |
|
− │ ├── ml_results.pkl (fitted XGB/LGBM + test predictions) |
| 235 |
|
− │ ├── shap_data.pkl (TreeSHAP values, 10,000 × 62) |
| 236 |
|
− │ ├── extended_results.pkl (unstandardized OLS, geo holdout, RF) |
| 237 |
|
− │ └── v3_shap_stability.pkl (XGB/LGB/RF SHAP rank correlations) |
| 238 |
|
− │ |
| 239 |
|
− ├─ scripts/01_spatial_robustness.py ──► results/v3_spatial_results.pkl |
| 240 |
|
− ├─ scripts/02_qr_imputation.py ───────► results/v3_qr_imputation_results.pkl |
| 241 |
|
− ├─ scripts/03_make_figures.py ────────► figures/fig1…fig11 (+ verification) |
| 242 |
|
− └─ scripts/04_export_tables.py ───────► results/tables/*.csv |
| 243 |
|
− │ |
| 244 |
|
− ▼ |
| 245 |
|
− paper/main.tex ──► main.pdf (52 p.) |
| 246 |
|
−``` |
| 247 |
|
− |
| 248 |
|
−The bracketed upstream stages were produced before this restructuring by scripts |
| 249 |
|
−that no longer exist (see [Project History](#-project-history--provenance)); their |
| 250 |
|
−outputs are preserved and every downstream number is verified against them. |
| 251 |
|
− |
| 252 |
|
−--- |
| 253 |
|
− |
| 254 |
|
−## ⚙️ Installation & Setup |
|
46 |
+The core methodological contribution is a systematic study of **spatial leakage** in hedonic model evaluation: random |
|
47 |
+train/test splits let geographically proximate listings appear on both sides of the split, inflating performance |
|
48 |
+metrics. Three designs quantify the damage — a 10-state geographic holdout, a 6-stage feature-ablation cascade, and a |
|
49 |
+latitude/longitude augmentation experiment. All estimates are *listing-price* gradients (asking prices), not causal |
|
50 |
+willingness-to-pay parameters. |
| 255 |
51 |
|
| 256 |
|
−**Prerequisites:** Python ≥ 3.11, a TeX Live distribution (with `newtx`, `natbib`, |
| 257 |
|
−`booktabs`, `threeparttable`), and ~4 GB of free disk for the data artifacts. |
|
52 |
+## 🏆 Headline results |
| 258 |
53 |
|
| 259 |
|
−```bash |
| 260 |
|
−git clone https://github.com/spboucher-ai/wp3-hedonic-housing-us.git |
| 261 |
|
−cd wp3-hedonic-housing-us |
| 262 |
|
− |
| 263 |
|
−python3 -m venv .venv |
| 264 |
|
−source .venv/bin/activate |
| 265 |
|
−pip install -r requirements.txt |
| 266 |
|
−``` |
| 267 |
|
− |
| 268 |
|
−Then place the data artifacts (request them at **contact@spboucher.ai**) into |
| 269 |
|
−`data/raw/`, `data/processed/`, and `results/` as shown in |
| 270 |
|
−[Repository Structure](#-repository-structure). |
| 271 |
|
− |
| 272 |
|
−All paths resolve relative to the repository root. To point the code at another |
| 273 |
|
−location, set the environment variable: |
| 274 |
|
− |
| 275 |
|
−```bash |
| 276 |
|
−export WP3_ROOT=/path/to/artifacts |
| 277 |
|
−``` |
| 278 |
|
− |
| 279 |
|
−--- |
| 280 |
|
− |
| 281 |
|
−## ▶️ Reproducing the Results |
| 282 |
|
− |
| 283 |
|
−| Step | Command | Runtime* | Needs | |
| 284 |
|
−|---|---|---|---| |
| 285 |
|
−| Spatial & robustness suite | `python scripts/01_spatial_robustness.py` | ~1–2 h (trains 16 XGBoost models) | `analytical_sample.pkl` | |
| 286 |
|
−| QR & sensitivity suite | `python scripts/02_qr_imputation.py` | ~1–3 h (12 quantile regressions on 150k obs) | `model_data.pkl`, DuckDB | |
| 287 |
|
−| **All 11 figures** | `python scripts/03_make_figures.py` | ~5 min | processed pickles | |
| 288 |
|
−| Single figures | `python scripts/03_make_figures.py --figs fig2 fig4` | seconds–minutes | idem | |
| 289 |
|
−| **CSV table exports** | `python scripts/04_export_tables.py` | ~2 min | result pickles | |
| 290 |
|
− |
| 291 |
|
−\* Apple Silicon, 32 GB RAM; scripts use all cores (`n_jobs=-1`). |
| 292 |
|
− |
| 293 |
|
−Every script prints **verification statistics** comparing recomputed values with |
| 294 |
|
−the numbers reported in the paper (see [Verification](#-verification--integrity)). |
| 295 |
|
− |
| 296 |
|
−--- |
| 297 |
|
− |
| 298 |
|
−## 📄 Building the Paper |
| 299 |
|
− |
| 300 |
|
−```bash |
| 301 |
|
−cd paper |
| 302 |
|
−latexmk # pdflatex ×N + bibtex, via Makefile/latexmkrc defaults |
| 303 |
|
−# or: make # same thing |
| 304 |
|
−# or: make clean / make distclean |
| 305 |
|
−``` |
| 306 |
|
− |
| 307 |
|
−- Output: `paper/main.pdf` — **52 pages**, compiles with **zero errors, zero |
| 308 |
|
− undefined references/citations**. |
| 309 |
|
−- Figures are pulled from `../figures/` via `\graphicspath`. |
| 310 |
|
−- Bibliography: `paper/references.bib` (37 entries), `natbib` + `apalike`. |
| 311 |
|
−- Metadata (title, abstract, keywords R31/C21/C45/C52, version) is centralized in |
| 312 |
|
− `main.tex` `\WP*` macros. |
| 313 |
|
− |
| 314 |
|
−--- |
| 315 |
|
− |
| 316 |
|
−## 🖼 Figures |
| 317 |
|
− |
| 318 |
|
−All figures are regenerated from the stored artifacts by `scripts/03_make_figures.py` |
| 319 |
|
−in a unified publication style (serif typography matched to the paper's `newtx` |
| 320 |
|
−text font, muted colorblind-safe palette, no chart junk, 300 dpi). |
| 321 |
|
− |
| 322 |
|
−| # | File | Content | Built from | |
| 323 |
|
−|---|---|---|---| |
| 324 |
|
−| 1 | `fig1_price_distribution.png` | Price & log-price histograms with medians | `analytical_sample.pkl` | |
| 325 |
|
−| 2 | `fig2_ols_diagnostics.png` | Residuals-vs-fitted, Q-Q, density, scale-location | `model_data.pkl` | |
| 326 |
|
−| 3 | `fig3_quantile_coefficients.png` | QR coefficient paths vs. OLS benchmark (8 panels) | `qr_results.pkl` + `model_data.pkl` | |
| 327 |
|
−| 4 | `fig4_model_comparison.png` | Predicted vs. actual: OLS / XGBoost / LightGBM | `ml_results.pkl` | |
| 328 |
|
−| 5 | `fig5_shap_summary.png` | SHAP beeswarm, top 20 features | `shap_data.pkl` | |
| 329 |
|
−| 6 | `fig6_shap_importance.png` | Mean \|SHAP\| bar chart, top 20 | `shap_data.pkl` | |
| 330 |
|
−| 7 | `fig7_geographic_prices.png` | 50,000-listing national price map (lat/lon, viridis) | `analytical_sample.pkl` | |
| 331 |
|
−| 8 | `fig8_regional_prices.png` | Log-price boxplots by Census region (with N) | `analytical_sample.pkl` | |
| 332 |
|
−| 9 | `fig9_marginal_effects.png` | Bivariate scatters + binned means (6 attributes) | `analytical_sample.pkl` | |
| 333 |
|
−| 10 | `fig10_shap_dependence.png` | SHAP dependence, 8 key features | `shap_data.pkl` | |
| 334 |
|
−| 11 | `fig11_shap_interactions.png` | SHAP for interaction terms (waterfront×sqft, pool×South) | `shap_data.pkl` | |
| 335 |
|
− |
| 336 |
|
−--- |
| 337 |
|
− |
| 338 |
|
−## 📊 Results Summary |
| 339 |
|
− |
| 340 |
|
−**Predictive performance (log-price scale):** |
| 341 |
|
− |
| 342 |
|
−| Model | Random $R^2$ | Geo-holdout $R^2$ | RMSE (random) | |
|
54 |
+| Model | Random R² | Geo-holdout R² | RMSE (random) | |
| 343 |
55 |
|---|---:|---:|---:| |
| 344 |
56 |
| OLS (62 regressors) | 0.630 | < 0 | 0.491 | |
| 345 |
57 |
| OLS + State FE | 0.678 | — | 0.459 | |
| 349 |
61 |
| **XGBoost** | **0.833** | **0.547** | **0.330** | |
| 350 |
62 |
| XGBoost, no geographic features | 0.830 | **0.519** | — | |
| 351 |
63 |
| XGBoost + lat/lon | 0.870 | 0.464 | — | |
| 352 |
|
−| XGBoost (ablation variant, full 62) | 0.833 | 0.425 | — | |
| 353 |
64 |
|
| 354 |
|
−**Ablation cascade (XGBoost):** structural 0.498 → +lot 0.546 → +amenities 0.638 → |
| 355 |
|
−+neighborhood **0.814 (+17.6 pp)** → +market 0.824 → full 0.833 (random $R^2$). |
|
65 |
+- **Geography does the heavy lifting in OLS**: region dummies → state FE → 886 ZIP3 FE raises out-of-sample R² from |
|
66 |
+ 0.630 → 0.678 → 0.725. |
|
67 |
+- **Residual spatial autocorrelation**: Moran's *I* = 0.2745 (SD 0.0076) across three 5,000-listing subsamples, all |
|
68 |
+ *p* < 0.001. |
|
69 |
+- **Distributional heterogeneity**: the garage gradient is 10× larger at τ = 0.10 than at τ = 0.90 (*z* = 28.9); the pool |
|
70 |
+ premium only emerges above the median. |
|
71 |
+- **Ablation cascade (XGBoost)**: structural 0.498 → +lot 0.546 → +amenities 0.638 → +neighbourhood **0.814** → +market |
|
72 |
+ 0.824 → full 0.833. |
|
73 |
+- **SHAP rankings are model-stable** (Spearman ρ = 0.89–0.99 across XGBoost / LightGBM / RF) but not structural. |
|
74 |
+- **Imputation matters**: the lot-size gradient triples (0.018 → 0.059) when state-median-imputed lot sizes are dropped. |
| 356 |
75 |
|
| 357 |
|
−**Machine-readable versions** of these and all other paper tables are in |
| 358 |
|
−[`results/tables/`](results/tables/) (13 CSV files: OLS coefficients, QR paths, |
| 359 |
|
−inter-quantile Wald tests, stability CVs, Moran's I, imputation/winsorization |
| 360 |
|
−sensitivity, SHAP importance & cross-model correlations, ZIP3 FE, ablation). |
|
76 |
+## 🗂 Repository layout |
| 361 |
77 |
|
| 362 |
|
−--- |
| 363 |
|
− |
| 364 |
|
−## ✅ Verification & Integrity |
| 365 |
|
− |
| 366 |
|
−The restructuring included a systematic audit reconciling the manuscript against |
| 367 |
|
−the stored result artifacts. Highlights: |
| 368 |
|
− |
| 369 |
|
−- **Exact matches** — OLS residual diagnostics (skewness 0.18, kurtosis 5.55, |
| 370 |
|
− Jarque–Bera 217,078), test-set $R^2$ (OLS 0.6301, XGBoost 0.8330, |
| 371 |
|
− LightGBM 0.8088), Moran's $I$ subsample values and mean/SD, QR coefficients at |
| 372 |
|
− all five quantiles, and the OLS reference line (0.312) in Figure 3. |
| 373 |
|
−- **End-to-end reproduction** — re-running the refactored Moran's $I$ pipeline |
| 374 |
|
− reproduces the stored values to 4 decimal places (0.2742 / 0.2839 / 0.2652). |
| 375 |
|
−- **Manuscript corrections** — six internal inconsistencies were found and fixed |
| 376 |
|
− (wrong KNN $k$ in the text, mislabeled inter-quantile sign convention, |
| 377 |
|
− winsorization description not matching the actual procedure, a duplicated table |
| 378 |
|
− row, mixed in/out-of-sample $R^2$ in the summary matrix). Every fix aligns the |
| 379 |
|
− text to the stored results and is itemized in [`CHANGES.md`](CHANGES.md) §4. |
| 380 |
|
− **No scientific result was altered.** |
| 381 |
|
−- **Completed placeholders** — 13 `[TODO]` table cells in the original manuscript |
| 382 |
|
− were filled with the exact values from the result pickles. |
| 383 |
|
− |
| 384 |
|
−See [`AUDIT.md`](AUDIT.md) for the full audit and [`CHANGES.md`](CHANGES.md) for |
| 385 |
|
−the change log. |
|
78 |
+``` |
|
79 |
+wp3_uqo/ |
|
80 |
+├── README.md |
|
81 |
+├── requirements.txt # pinned dependencies (validated 2026-08-05) |
|
82 |
+├── docs/ # AUDIT.md · CHANGES.md · PAPER_REVIEW.md · UPGRADE_REPORT.md |
|
83 |
+├── data/ # NOT tracked (3.2 GB) |
|
84 |
+│ ├── raw/us_housing.duckdb # raw Zillow extract (839,313 × 116 + 4 aux tables) |
|
85 |
+│ └── processed/ # analytical_sample.pkl · model_data.pkl · shap_data.pkl |
|
86 |
+├── src/wp3/ # shared library |
|
87 |
+│ ├── config.py # paths (WP3_ROOT-overridable), constants, feature blocks, holdout states, seeds |
|
88 |
+│ ├── data.py # loaders + 62-regressor design-matrix builder |
|
89 |
+│ ├── models.py # XGBoost training helper (dual validation schemes) |
|
90 |
+│ └── plotting.py # journal figure style + variable labels |
|
91 |
+├── scripts/ # the pipeline, in order |
|
92 |
+│ ├── 01_spatial_robustness.py # ZIP3 FE OLS · Moran's I · XGB ± geography · ablation |
|
93 |
+│ ├── 02_qr_imputation.py # QR stability · inter-quantile Wald · imputation · winsorisation |
|
94 |
+│ ├── 03_make_figures.py # all 11 figures + verification stats |
|
95 |
+│ └── 04_export_tables.py # paper tables → results/tables/*.csv |
|
96 |
+├── results/ |
|
97 |
+│ ├── *.pkl # NOT tracked (up to 437 MB) |
|
98 |
+│ └── tables/ # 13 CSVs mirroring the paper's tables |
|
99 |
+├── figures/ # 11 paper figures (PNG, 300 dpi) |
|
100 |
+└── paper/ # main.tex + sections/ + appendix/ + references.bib → main.pdf |
|
101 |
+``` |
| 386 |
102 |
|
| 387 |
|
−--- |
|
103 |
+## 🚀 Quick start |
| 388 |
104 |
|
| 389 |
|
−## 🕰 Project History & Provenance |
| 390 |
|
− |
| 391 |
|
−- The project was completed in **May 2026** in a flat working folder |
| 392 |
|
− (`immo-wp3-spb-20260519`), then restructured into this compendium on |
| 393 |
|
− **2026-08-05**. The original archive is preserved untouched. |
| 394 |
|
−- The scripts that built the analytical sample and estimated the primary models |
| 395 |
|
− (OLS/QR/ML/SHAP) were **lost** prior to restructuring; only two robustness |
| 396 |
|
− scripts survived and were refactored into `scripts/01` and `scripts/02`. The |
| 397 |
|
− stored pickles are the canonical record of those stages and every downstream |
| 398 |
|
− number is validated against them. |
| 399 |
|
−- Five paper figures had been deleted from the archive; they were rebuilt from the |
| 400 |
|
− stored data, and subsequently **all 11 figures** were regenerated in a unified |
| 401 |
|
− journal style. |
| 402 |
|
−- Three earlier monolithic drafts (`paper_hedonic_pricing{,_v2,_v3}.tex`) precede |
| 403 |
|
− the sectioned `wp3/` source that became `paper/`. |
|
105 |
+```bash |
|
106 |
+python3 -m venv .venv && source .venv/bin/activate |
|
107 |
+pip install -r requirements.txt |
|
108 |
+# place the artifacts (request them, see Data) in data/raw/, data/processed/, results/ |
|
109 |
+export WP3_ROOT=/path/to/artifacts # optional, defaults to the repo root |
| 404 |
110 |
|
| 405 |
|
−--- |
|
111 |
+python scripts/01_spatial_robustness.py # ~1–2 h (16 XGBoost models) |
|
112 |
+python scripts/02_qr_imputation.py # ~1–3 h (12 quantile regressions on 150k obs) |
|
113 |
+python scripts/03_make_figures.py # ~5 min, all 11 figures (--figs fig2 fig4 for a subset) |
|
114 |
+python scripts/04_export_tables.py # ~2 min, 13 CSV tables |
| 406 |
115 |
|
| 407 |
|
−## 📖 Citation |
|
116 |
+cd paper && make # latexmk → main.pdf (60 pages) |
|
117 |
+``` |
| 408 |
118 |
|
| 409 |
|
−If you use this code, the figures, or the results, please cite: |
|
119 |
+Apple Silicon, 32 GB RAM; scripts use all cores. Every script prints verification statistics comparing recomputed values |
|
120 |
+with the numbers reported in the paper. |
|
121 |
+ |
|
122 |
+## 🗃 Data |
|
123 |
+ |
|
124 |
+- **Source**: Zillow active for-sale listings, 2025–2026 snapshot; 839,313 residential properties × 116 variables in a |
|
125 |
+ DuckDB database (`properties`, `price_history`, `schools`, `tax_history`, `meta_columns`). |
|
126 |
+- **Analytical sample**: 788,842 listings after five filters (price $10K–$10M, living area 200–20,000 sqft, 1–10 |
|
127 |
+ beds/baths, valid coordinates, US states + DC; 6.0 % attrition); 68 engineered columns (logs, age and age², ratios, |
|
128 |
+ amenity dummies, neighbourhood scores, market-status flags, 6 interactions, simplified categoricals). |
|
129 |
+- **Not in git**: `us_housing.duckdb` (1.6 GB), `analytical_sample.pkl` (457 MB), `model_data.pkl` (867 MB), |
|
130 |
+ `ml_results.pkl` (437 MB). Available on request (subject to Zillow Terms of Service): |
|
131 |
+ simon-pierre.boucher@uqo.ca. |
|
132 |
+- The scripts that built the analytical sample and estimated the primary models were lost before the 2026-08-05 |
|
133 |
+ restructuring; the stored pickles are the canonical record of those stages and every downstream number is validated |
|
134 |
+ against them (`docs/AUDIT.md`). |
|
135 |
+ |
|
136 |
+## 🔬 Methodology |
|
137 |
+ |
|
138 |
+1. **OLS** — 62 regressors, standardised and unstandardised variants, HC3 robust SE; fixed effects at region (4), state |
|
139 |
+ (50) and ZIP3 (886) levels (`scripts/01`). |
|
140 |
+2. **Spatial diagnostics** — Moran's *I* on OLS residuals, row-standardised KNN weights (k = 8), 3 × 5,000-obs |
|
141 |
+ subsamples, 999 permutations. |
|
142 |
+3. **Quantile regression** — statsmodels `QuantReg`, τ ∈ {0.10, …, 0.90}, 150,000-obs subsample, 10-seed stability |
|
143 |
+ check; inter-quantile Wald *z*-tests on β̂(0.10) − β̂(0.90) (`scripts/02`). |
|
144 |
+4. **Machine learning** — XGBoost (1,000 trees, depth 8, lr 0.05, early stopping 50), LightGBM, RF, Ridge, Lasso, |
|
145 |
+ Elastic Net; random 80/20 split (seed 42) **and** 10-state geographic holdout (CA NY TX FL OH CO NC WA IL GA, |
|
146 |
+ 438,315 test obs) (`src/wp3/models.py`). |
|
147 |
+5. **Ablation** — 6 nested feature sets: structural → +lot → +amenities → +neighbourhood → +market → full. |
|
148 |
+6. **Interpretation** — TreeSHAP on 10,000 test obs; cross-model Spearman stability. |
|
149 |
+7. **Sensitivity** — state-median-imputation drop-out, lot-size winsorisation (p99.5), Bike Score cap. |
|
150 |
+ |
|
151 |
+## 🖼 Figures & tables inventory |
|
152 |
+ |
|
153 |
+| Figure | Content | |
|
154 |
+|---|---| |
|
155 |
+| `fig1_price_distribution` | Price & log-price histograms with medians | |
|
156 |
+| `fig2_ols_diagnostics` | Residuals-vs-fitted, Q-Q, density, scale-location | |
|
157 |
+| `fig3_quantile_coefficients` | QR coefficient paths vs. OLS benchmark (8 panels) | |
|
158 |
+| `fig4_model_comparison` | Predicted vs. actual: OLS / XGBoost / LightGBM | |
|
159 |
+| `fig5_shap_summary` | SHAP beeswarm, top 20 features | |
|
160 |
+| `fig6_shap_importance` | Mean \|SHAP\| bar chart, top 20 | |
|
161 |
+| `fig7_geographic_prices` | 50,000-listing national price map | |
|
162 |
+| `fig8_regional_prices` | Log-price boxplots by Census region | |
|
163 |
+| `fig9_marginal_effects` | Bivariate scatters + binned means (6 attributes) | |
|
164 |
+| `fig10_shap_dependence` | SHAP dependence, 8 key features | |
|
165 |
+| `fig11_shap_interactions` | SHAP for interaction terms (waterfront × sqft, pool × South) | |
|
166 |
+ |
|
167 |
+Tables (`results/tables/*.csv`, 13 files): OLS coefficients, QR paths, inter-quantile Wald tests, stability CVs, |
|
168 |
+Moran's *I*, imputation/winsorisation sensitivity, SHAP importance and cross-model correlations, ZIP3 FE, ablation. |
|
169 |
+ |
|
170 |
+## ⚠️ Limitations |
|
171 |
+ |
|
172 |
+- Listing (asking) prices, not transactions; gradients are conditional associations, not willingness to pay. |
|
173 |
+- A single 2025–2026 snapshot: no time dimension, no repeat-sales control. |
|
174 |
+- Lot size is state-median-imputed for part of the sample; the gradient is sensitive to that choice. |
|
175 |
+- The upstream sample-construction and primary-estimation scripts are lost; those stages are reproducible only from the |
|
176 |
+ stored artifacts. |
|
177 |
+ |
|
178 |
+## 📝 Citation |
| 410 |
179 |
|
| 411 |
180 |
```bibtex |
| 412 |
|
−@techreport{boucher2026hedonic, |
|
181 |
+@techreport{boucher2026hedonicus, |
| 413 |
182 |
author = {Boucher, Simon-Pierre}, |
| 414 |
|
− title = {Hedonic Housing Price Models for the United States: |
| 415 |
|
− A Multi-Method Comparison of Parametric, Quantile, |
| 416 |
|
− and Machine Learning Approaches}, |
| 417 |
|
− institution = {Universit\'e du Qu\'ebec en Outaouais, |
| 418 |
|
− D\'epartement des sciences administratives}, |
|
183 |
+ title = {Hedonic Housing Price Models for the United States: A Multi-Method |
|
184 |
+ Comparison of Parametric, Quantile, and Machine Learning Approaches}, |
|
185 |
+ institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, |
| 419 |
186 |
type = {Working Paper}, |
| 420 |
187 |
number = {3}, |
| 421 |
188 |
year = {2026}, |
| 423 |
190 |
} |
| 424 |
191 |
``` |
| 425 |
192 |
|
| 426 |
|
−--- |
| 427 |
|
− |
| 428 |
|
−## 📬 Contact & License |
|
193 |
+## 👤 Author & contact |
| 429 |
194 |
|
| 430 |
|
−**Simon-Pierre Boucher** |
| 431 |
|
−Département des sciences administratives |
| 432 |
|
−Université du Québec en Outaouais — Gatineau (Québec), Canada |
| 433 |
|
−📧 **contact@spboucher.ai** · 🎓 simon-pierre.boucher@uqo.ca |
|
195 |
+**Simon-Pierre Boucher** — Département des sciences administratives, |
|
196 |
+Université du Québec en Outaouais (UQO), Gatineau, QC. |
|
197 |
+📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) |
| 434 |
198 |
|
| 435 |
|
−© 2026 Simon-Pierre Boucher. Analysis code is shared for **replication and review** |
| 436 |
|
−purposes. The underlying Zillow data are subject to Zillow's Terms of Service and |
| 437 |
|
−are **not redistributed** in this repository; the paper (`paper/main.pdf`) is a |
| 438 |
|
−working paper — please cite rather than redistribute. |
|
199 |
+## 🗒 Notes |
| 439 |
200 |
|
| 440 |
|
−<div align="center"> |
| 441 |
|
−<sub>Built with Python · XGBoost · LightGBM · statsmodels · SHAP · DuckDB · LaTeX — reproducible from stored artifacts end to end.</sub> |
| 442 |
|
−</div> |
|
201 |
+- Format follows `Recherche-UQO/HARMONISATION.md` (2026-10-17): template title page, `natbib` + `aer` bibliography style, |
|
202 |
+ `docs/` for the restructuring reports. |
|
203 |
+- The original flat May 2026 project (`immo-wp3-spb-20260519`, 3.2 GB) is archived in |
|
204 |
+ `Recherche-UQO/_archives/versions-mai-2026/hedonic-models-united-states--immo-wp3-20260519/`; `docs/CHANGES.md` lists |
|
205 |
+ the six manuscript inconsistencies corrected on 2026-08-05 (no scientific result altered). |
|
206 |
+- GitHub mirror: `spboucher-ai/wp3-hedonic-housing-us` (remote `github`); canonical remote is spbgit `wp3_uqo.git`. |
|
207 |
+- Build verified 2026-10-17: 0 errors, 0 undefined references, 70 references rendered. |