SPB Git
spboucher.ai

spb/wp3_uqo

Public

UQO Working Paper No. 3 — Hedonic housing price models for the US: parametric, quantile, and machine-learning approaches.

researcheconomicshousinghedonicmachine-learninguqo

harmonisation standard UQO WP (2026-10-17)

Page titre du gabarit, natbib + aer, docs/ pour les rapports d'audit, README au standard (Status, citation WP 3), .gitignore et Makefile/.latexmkrc du gabarit, main.pdf recompilé.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 17, 2026) parent 4b91e7e Browse files

10 changed files +196 −414

10 changed files
Unified Split
modified .gitignore +23 −7
@@ -1,23 +1,39 @@
1−# Large data artifacts (> GitHub's 100 MB limit) — kept in the local archive, 1+# Raw & processed data — never committed (see data/raw/README.md for provenance)
2−# available on request: contact@spboucher.ai 2+data/raw/*
3−data/ 3+!data/raw/README.md
4−results/*.pkl 4+data/processed/*
5+!data/processed/.gitkeep
5 6
6 # Python 7 # Python
7 __pycache__/ 8 __pycache__/
8 *.pyc 9 *.pyc
10+.pytest_cache/
9 .venv/ 11 .venv/
12+*.egg-info/
10 13
11−# LaTeX build artifacts 14+# LaTeX build artifacts (paper/main.pdf IS committed)
12 paper/*.aux 15 paper/*.aux
13 paper/*.log 16 paper/*.log
14 paper/*.out 17 paper/*.out
15 paper/*.toc 18 paper/*.toc
19+paper/*.lof
20+paper/*.lot
16 paper/*.fls 21 paper/*.fls
17 paper/*.fdb_latexmk 22 paper/*.fdb_latexmk
18−paper/*.synctex.gz
19−paper/*.blg
20 paper/*.bbl 23 paper/*.bbl
24+paper/*.blg
25+paper/*.spl
26+paper/*.synctex.gz
27+paper/*.run.xml
28+paper/*.bcf
29+
30+# Logs
31+logs/
32+*.log
21 33
22 # macOS 34 # macOS
23 .DS_Store 35 .DS_Store
36+
37+# Project-specific: DuckDB + pickles (3.2 GB), available on request
38+data/
39+results/*.pkl
modified README.md +158 −393
@@ -1,345 +1,57 @@
1−<div align="center"> 1+<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
2−
3 # 🏠 Hedonic Housing Price Models for the United States 2 # 🏠 Hedonic Housing Price Models for the United States
4 3
5−### A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches 4+**UQO Working Paper No. 3** — *A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches*
6−
7−**UQO Working Paper No. 3**
8−
9−[![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/)
10−[![LaTeX](https://img.shields.io/badge/LaTeX-pdflatex%20%2B%20BibTeX-008080?logo=latex&logoColor=white)](https://www.latex-project.org/)
11−[![XGBoost](https://img.shields.io/badge/XGBoost-3.2.0-EB5E28)](https://xgboost.readthedocs.io/)
12−[![LightGBM](https://img.shields.io/badge/LightGBM-4.6.0-9ACD32)](https://lightgbm.readthedocs.io/)
13−[![SHAP](https://img.shields.io/badge/SHAP-0.48.0-blueviolet)](https://shap.readthedocs.io/)
14−[![statsmodels](https://img.shields.io/badge/statsmodels-0.14.6-4051B5)](https://www.statsmodels.org/)
15−[![DuckDB](https://img.shields.io/badge/DuckDB-1.5.2-FFF000?logo=duckdb&logoColor=black)](https://duckdb.org/)
16 5
17−[![Paper](https://img.shields.io/badge/Paper-60%20pages%20PDF-B31B1B?logo=adobeacrobatreader&logoColor=white)](paper/main.pdf) 6+[![Paper](https://img.shields.io/badge/paper-PDF%20(60%20p.)-b31b1b?logo=latex&logoColor=white)](paper/main.pdf)
18−[![Sample](https://img.shields.io/badge/Sample-788%2C842%20listings-2B5C8A)](#-data) 7+[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)](requirements.txt)
19−[![Coverage](https://img.shields.io/badge/Coverage-50%20states%20%2B%20DC-2B5C8A)](#-data) 8+[![Data](https://img.shields.io/badge/data-788%2C842%20Zillow%20listings%20·%20DuckDB-4051b5)](#-data)
20−[![Figures](https://img.shields.io/badge/Figures-11%20publication--ready-A63E38)](#-figures) 9+[![Estimation](https://img.shields.io/badge/estimation-OLS%20HC3%20·%20QuantReg%20·%20XGBoost%20%2B%20SHAP-4051b5)](src/wp3/)
21−[![References](https://img.shields.io/badge/References-70%20verified%20(OpenAlex%2FDOI)-2E8B57)](paper/references.bib) 10+[![Reproducible](https://img.shields.io/badge/reproducible-from%20stored%20artifacts-2e7d32)](scripts/)
22−[![Verified](https://img.shields.io/badge/Results-verified%20against%20stored%20artifacts-2E8B57)](#-verification--integrity) 11+[![Institution](https://img.shields.io/badge/UQO-D%C3%A9pt.%20des%20sciences%20administratives-16365c)](https://uqo.ca)
23−[![Status](https://img.shields.io/badge/Status-Working%20Paper%20v1.1-orange)](#-citation) 12+[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-16365c)](mailto:simon-pierre.boucher@uqo.ca)
24 13
25−**Author : [Simon-Pierre Boucher](mailto:contact@spboucher.ai)** 14+> **TL;DR** — Three hedonic frameworks are compared on **788,842 active Zillow listings** covering all 50 U.S. states and
26−Département des sciences administratives — Université du Québec en Outaouais (UQO) 15+> DC. Semi-log OLS with 62 regressors gives R² = 0.634 (0.725 with 886 ZIP3 fixed effects; Moran's *I* = 0.27 on
27−📧 **contact@spboucher.ai** 16+> residuals). Quantile regression rejects coefficient equality between τ = 0.10 and 0.90 for **11 of 13** key variables.
28− 17+> XGBoost reaches R² = 0.833 under a random split but only **0.425–0.547** when ten entire states are held out, and
29−</div> 18+> *removing* geographic features improves geographic-holdout R² to 0.519 — random validation flatters machine learning
19+> through spatial leakage.
30 20
31 --- 21 ---
32 22
33−## 📑 Table of Contents 23+## 📌 Status
34−
35−- [Overview](#-overview)
36−- [Key Findings](#-key-findings)
37−- [Repository Structure](#-repository-structure)
38−- [Data](#-data)
39−- [Methodology at a Glance](#-methodology-at-a-glance)
40−- [The Analysis Pipeline](#-the-analysis-pipeline)
41−- [Installation & Setup](#-installation--setup)
42−- [Reproducing the Results](#-reproducing-the-results)
43−- [Building the Paper](#-building-the-paper)
44−- [Figures](#-figures)
45−- [Results Summary](#-results-summary)
46−- [Verification & Integrity](#-verification--integrity)
47−- [Project History & Provenance](#-project-history--provenance)
48−- [Citation](#-citation)
49−- [Contact & License](#-contact--license)
50 24
51−--- 25+| Item | Value |
26+|---|---|
27+| Version | 1.1 (May 2026) — cover page harmonised to the UQO WP standard, October 2026 |
28+| SSRN | not yet submitted |
29+| Journal | not yet submitted |
30+| Code & data | figures and tables reproducible from stored artifacts; raw Zillow extract (3.2 GB) available on request |
52 31
53−## 🔎 Overview 32+## 🎯 What this paper does
54 33
55−This repository contains the complete research compendium — analysis code, LaTeX 34+The paper compares three frameworks for hedonic housing valuation on a single large dataset:
56−source, publication-ready figures, and machine-readable result tables — for
57−**UQO Working Paper No. 3**, which compares three frameworks for hedonic housing
58−valuation on a single large dataset of **788,842 active Zillow listings** spanning
59−all 50 U.S. states and the District of Columbia:
60 35
61 | Framework | Question it answers | Headline result | 36 | Framework | Question it answers | Headline result |
62 |---|---|---| 37 |---|---|---|
63−| **Semi-log OLS** (62 regressors, HC3 SE) | What is the *average* conditional association between an attribute and log listing price? | $R^2 = 0.634$; price-to-area elasticity 0.63 | 38+| **Semi-log OLS** (62 regressors, HC3 SE) | Average conditional association between an attribute and log listing price | R² = 0.634; price-to-area elasticity 0.63 |
64−| **Quantile regression** ($\tau \in \{0.10, 0.25, 0.50, 0.75, 0.90\}$) | How do attribute gradients *vary across the price distribution*? | Coefficient equality rejected for **11 of 13** variables (inter-quantile Wald tests) | 39+| **Quantile regression** (τ ∈ {0.10, 0.25, 0.50, 0.75, 0.90}) | How attribute gradients vary across the price distribution | Equality rejected for 11 of 13 variables |
65−| **Gradient boosting + SHAP** (XGBoost, LightGBM) | How well can prices be *predicted*, and which features drive predictions? | $R^2 = 0.833$ (random split) vs. **0.425–0.547** (geographic holdout) | 40+| **Gradient boosting + SHAP** (XGBoost, LightGBM) | How well prices can be predicted, and which features drive predictions | R² = 0.833 (random) vs. 0.425–0.547 (geographic holdout) |
66−
67−The paper's core methodological contribution is a systematic study of
68−**spatial leakage** in hedonic model evaluation: random train/test splits let
69−geographically proximate listings appear on both sides of the split, inflating
70−performance metrics. Three complementary designs quantify the damage — a 10-state
71−geographic holdout, a 6-stage feature-ablation cascade, and a latitude/longitude
72−augmentation experiment.
73−
74−> ⚠️ **Interpretation caveat.** All estimates are *listing-price capitalization
75−> gradients* — conditional associations between attributes and *asking* prices.
76−> They are not causal willingness-to-pay parameters and not transaction-price
77−> implicit prices.
78−
79−---
80−
81−## 🏆 Key Findings
82−
83−1. **Geography does the heavy lifting in OLS.** Moving from 4 Census-region
84− dummies to state fixed effects to 886 ZIP3 fixed effects raises out-of-sample
85− $R^2$ from 0.630 → 0.678 → 0.725 — a 9.5 pp gain from spatial granularity alone.
86−
87−2. **Strong, stable residual spatial autocorrelation.** Moran's $I$ on OLS
88− residuals averages **0.2745** (SD 0.0076) across three independent 5,000-listing
89− subsamples (row-standardized KNN weights, $k = 8$; all $p < 0.001$).
90−
91−3. **Mean effects mask distributional heterogeneity.** The garage gradient is
92− **10× larger** at $\tau = 0.10$ than at $\tau = 0.90$ ($z = 28.9$); the pool
93− premium only emerges above the median; the age penalty is concentrated in
94− lower-priced homes.
95−
96−4. **Random validation flatters machine learning.** XGBoost reaches $R^2 = 0.833$
97− under a random 80/20 split but only **0.425–0.547** when 10 entire states
98− (438,315 listings) are held out.
99−
100−5. **Geographic features can *hurt* generalization.** Removing all geographic
101− features *improves* geographic-holdout $R^2$ from 0.425 to **0.519**, while
102− adding raw lat/lon coordinates boosts random $R^2$ (+3.7 pp) but *degrades*
103− geographic holdout. Region dummies memorize training-set price levels.
104−
105−6. **SHAP rankings are model-stable but not structural.** Spearman rank
106− correlations of mean $|\phi|$ across XGBoost / LightGBM / Random Forest range
107− from **0.89 to 0.99**, with the same six features on top — yet these remain
108− predictive decompositions, not implicit prices.
109−
110−7. **Imputation matters.** The lot-size gradient **triples** (0.018 → 0.059) when
111− state-median-imputed lot sizes are dropped — a warning for hedonic work on
112− scraped listing data.
113−
114−---
115−
116−## 🗂 Repository Structure
117 41
118 ``` 42 ```
119−wp3_uqo/ 43+ln P_i = α + x_i'β + μ_g(i) + ε_i g = Census region → state → ZIP3
120−├── README.md ← you are here
121−├── AUDIT.md ← forensic audit of the original project archive
122−├── CHANGES.md ← every move/refactor/fix made during restructuring
123−├── requirements.txt ← pinned Python dependencies (validated 2026-08-05)
124−│
125−├── data/ ⚠ NOT in the Git repo (3.2 GB — see "Data" below)
126−│ ├── raw/
127−│ │ └── us_housing.duckdb # raw Zillow extract (839,313 × 116 + 4 aux tables)
128−│ └── processed/
129−│ ├── analytical_sample.pkl # final sample, 788,842 × 68 engineered columns
130−│ ├── model_data.pkl # standardized X (788,842 × 63), y, OLS fit, residuals
131−│ └── shap_data.pkl # TreeSHAP values (10,000 × 62) + explanation sample
132−│
133−├── src/wp3/ # shared library code
134−│ ├── config.py # paths (relative, WP3_ROOT-overridable), constants,
135−│ │ # feature-block definitions, holdout states, seeds
136−│ ├── data.py # loaders + 62-regressor design-matrix builder
137−│ ├── models.py # XGBoost training helper (dual validation schemes)
138−│ └── plotting.py # journal figure style + human-readable variable labels
139−│
140−├── scripts/ # numbered pipeline entry points
141−│ ├── 01_spatial_robustness.py # ZIP3 FE OLS · Moran's I · XGB ± geography · ablation
142−│ ├── 02_qr_imputation.py # QR stability · inter-quantile Wald · imputation ·
143−│ │ # winsorization sensitivity
144−│ ├── 03_make_figures.py # regenerates all 11 figures + verification stats
145−│ └── 04_export_tables.py # exports paper tables → results/tables/*.csv
146−│
147−├── figures/ # all 11 paper figures (PNG, 300 dpi, unified style)
148−│
149−├── results/
150−│ ├── *.pkl ⚠ NOT in the Git repo (up to 437 MB)
151−│ └── tables/ # 13 CSVs mirroring the paper's tables (in repo)
152−│
153−└── paper/ # LaTeX source
154− ├── main.tex # preamble + metadata + \input skeleton
155− ├── references.bib # 37 BibTeX entries (natbib + apalike)
156− ├── sections/ # one file per section (IMRaD + robustness)
157− │ ├── titlepage.tex ├── introduction.tex ├── literature.tex
158− │ ├── data.tex ├── methodology.tex ├── results.tex
159− │ ├── robustness.tex ├── discussion.tex ├── limitations.tex
160− │ └── conclusion.tex
161− ├── appendix/appendix.tex # extra figures, full OLS table, reproducibility checklist
162− ├── Makefile # latexmk targets (all / clean / distclean)
163− ├── uq_logo.jpg
164− └── main.pdf # compiled paper — 52 pages, zero warnings
165 ``` 44 ```
166 45
167−--- 46+The core methodological contribution is a systematic study of **spatial leakage** in hedonic model evaluation: random
168− 47+train/test splits let geographically proximate listings appear on both sides of the split, inflating performance
169−## 💾 Data 48+metrics. Three designs quantify the damage — a 10-state geographic holdout, a 6-stage feature-ablation cascade, and a
170− 49+latitude/longitude augmentation experiment. All estimates are *listing-price* gradients (asking prices), not causal
171−### What the data are 50+willingness-to-pay parameters.
172−
173−- **Source:** Zillow active for-sale listings, 2025–2026 snapshot.
174−- **Raw:** 839,313 residential properties × 116 variables, stored in a DuckDB
175− database (`properties`, `price_history`, `schools`, `tax_history`, `meta_columns`).
176−- **Analytical sample:** 788,842 listings after five sequential filters
177− (valid price \$10K–\$10M, living area 200–20,000 sqft, 1–10 beds/baths,
178− valid coordinates, US states + DC only — 6.0 % attrition).
179−- **68 engineered columns:** log transforms, age & age², ratios, amenity dummies,
180− neighborhood scores, market-status flags, 6 interaction terms, and simplified
181− categoricals (roof / construction / foundation / region).
182−
183−### Why the data are not in this Git repository
184−
185−GitHub rejects files above 100 MB. The DuckDB file is **1.6 GB** and the processed
186−pickles reach **867 MB**, so `data/` and `results/*.pkl` are excluded via
187−`.gitignore`. The canonical copies live in the local research archive:
188−
189−| Artifact | Size | Location |
190−|---|---|---|
191−| `us_housing.duckdb` | 1.6 GB | `data/raw/` (local) |
192−| `analytical_sample.pkl` | 457 MB | `data/processed/` (local) |
193−| `model_data.pkl` | 867 MB | `data/processed/` (local) |
194−| `ml_results.pkl` | 437 MB | `results/` (local) |
195−| `shap_data.pkl`, `qr_results.pkl`, `extended_results.pkl`, `v3_*.pkl` | < 10 MB each | `results/` (local; small ones could be added on request) |
196−
197−📬 **Data access:** available on request (subject to Zillow Terms of Service) —
198−**contact@spboucher.ai**.
199−
200−---
201−
202−## 🧪 Methodology at a Glance
203−
204−| Component | Specification |
205−|---|---|
206−| Dependent variable | $\ln(\text{listing price})$ |
207−| OLS | 62 regressors, standardized + unstandardized variants, **HC3** robust SE |
208−| Fixed effects | Census region (baseline) → state (50) → ZIP3 (886) |
209−| Quantile regression | statsmodels `QuantReg`, $\tau \in \{0.10,\ldots,0.90\}$, 150,000-obs subsample, 10-seed stability check |
210−| Inter-quantile tests | Wald $z$-tests on $\hat\beta(0.10)-\hat\beta(0.90)$ |
211−| XGBoost | 1,000 trees, depth 8, lr 0.05, subsample/colsample 0.8, early stopping 50 |
212−| LightGBM / RF / Ridge / Lasso / Elastic Net | benchmark suite |
213−| Validation | random 80/20 (seed 42) **and** 10-state geographic holdout (CA NY TX FL OH CO NC WA IL GA — 438,315 test obs) |
214−| Ablation | 6 nested feature sets: structural → +lot → +amenities → +neighborhood → +market → full |
215−| Interpretation | TreeSHAP on 10,000 test obs; cross-model Spearman stability (XGB/LGB/RF) |
216−| Spatial diagnostics | Moran's $I$, row-standardized KNN ($k=8$), 3 × 5,000-obs subsamples, 999 permutations |
217−| Sensitivity | state-median-imputation drop-out, lot-size winsorization (p99.5), Bike Score cap |
218−
219−---
220−
221−## 🔁 The Analysis Pipeline
222−
223−```
224−us_housing.duckdb (raw, 839,313 listings)
225− │
226− │ [sample construction & feature engineering — original script lost;
227− │ the stored analytical_sample.pkl is the canonical artifact]
228− ▼
229−analytical_sample.pkl (788,842 × 68)
230− │
231− ├─ [estimation — original scripts lost; stored artifacts are canonical]
232− │ ├── model_data.pkl (standardized X, OLS fit, residuals)
233− │ ├── qr_results.pkl (QR params at 5 quantiles)
234− │ ├── ml_results.pkl (fitted XGB/LGBM + test predictions)
235− │ ├── shap_data.pkl (TreeSHAP values, 10,000 × 62)
236− │ ├── extended_results.pkl (unstandardized OLS, geo holdout, RF)
237− │ └── v3_shap_stability.pkl (XGB/LGB/RF SHAP rank correlations)
238− │
239− ├─ scripts/01_spatial_robustness.py ──► results/v3_spatial_results.pkl
240− ├─ scripts/02_qr_imputation.py ───────► results/v3_qr_imputation_results.pkl
241− ├─ scripts/03_make_figures.py ────────► figures/fig1…fig11 (+ verification)
242− └─ scripts/04_export_tables.py ───────► results/tables/*.csv
243− │
244− ▼
245− paper/main.tex ──► main.pdf (52 p.)
246−```
247−
248−The bracketed upstream stages were produced before this restructuring by scripts
249−that no longer exist (see [Project History](#-project-history--provenance)); their
250−outputs are preserved and every downstream number is verified against them.
251−
252−---
253−
254−## ⚙️ Installation & Setup
255 51
256−**Prerequisites:** Python ≥ 3.11, a TeX Live distribution (with `newtx`, `natbib`, 52+## 🏆 Headline results
257−`booktabs`, `threeparttable`), and ~4 GB of free disk for the data artifacts.
258 53
259−```bash 54+| Model | Random R² | Geo-holdout R² | RMSE (random) |
260−git clone https://github.com/spboucher-ai/wp3-hedonic-housing-us.git
261−cd wp3-hedonic-housing-us
262−
263−python3 -m venv .venv
264−source .venv/bin/activate
265−pip install -r requirements.txt
266−```
267−
268−Then place the data artifacts (request them at **contact@spboucher.ai**) into
269−`data/raw/`, `data/processed/`, and `results/` as shown in
270−[Repository Structure](#-repository-structure).
271−
272−All paths resolve relative to the repository root. To point the code at another
273−location, set the environment variable:
274−
275−```bash
276−export WP3_ROOT=/path/to/artifacts
277−```
278−
279−---
280−
281−## ▶️ Reproducing the Results
282−
283−| Step | Command | Runtime* | Needs |
284−|---|---|---|---|
285−| Spatial & robustness suite | `python scripts/01_spatial_robustness.py` | ~1–2 h (trains 16 XGBoost models) | `analytical_sample.pkl` |
286−| QR & sensitivity suite | `python scripts/02_qr_imputation.py` | ~1–3 h (12 quantile regressions on 150k obs) | `model_data.pkl`, DuckDB |
287−| **All 11 figures** | `python scripts/03_make_figures.py` | ~5 min | processed pickles |
288−| Single figures | `python scripts/03_make_figures.py --figs fig2 fig4` | seconds–minutes | idem |
289−| **CSV table exports** | `python scripts/04_export_tables.py` | ~2 min | result pickles |
290−
291−\* Apple Silicon, 32 GB RAM; scripts use all cores (`n_jobs=-1`).
292−
293−Every script prints **verification statistics** comparing recomputed values with
294−the numbers reported in the paper (see [Verification](#-verification--integrity)).
295−
296−---
297−
298−## 📄 Building the Paper
299−
300−```bash
301−cd paper
302−latexmk # pdflatex ×N + bibtex, via Makefile/latexmkrc defaults
303−# or: make # same thing
304−# or: make clean / make distclean
305−```
306−
307−- Output: `paper/main.pdf` — **52 pages**, compiles with **zero errors, zero
308− undefined references/citations**.
309−- Figures are pulled from `../figures/` via `\graphicspath`.
310−- Bibliography: `paper/references.bib` (37 entries), `natbib` + `apalike`.
311−- Metadata (title, abstract, keywords R31/C21/C45/C52, version) is centralized in
312− `main.tex` `\WP*` macros.
313−
314−---
315−
316−## 🖼 Figures
317−
318−All figures are regenerated from the stored artifacts by `scripts/03_make_figures.py`
319−in a unified publication style (serif typography matched to the paper's `newtx`
320−text font, muted colorblind-safe palette, no chart junk, 300 dpi).
321−
322−| # | File | Content | Built from |
323−|---|---|---|---|
324−| 1 | `fig1_price_distribution.png` | Price & log-price histograms with medians | `analytical_sample.pkl` |
325−| 2 | `fig2_ols_diagnostics.png` | Residuals-vs-fitted, Q-Q, density, scale-location | `model_data.pkl` |
326−| 3 | `fig3_quantile_coefficients.png` | QR coefficient paths vs. OLS benchmark (8 panels) | `qr_results.pkl` + `model_data.pkl` |
327−| 4 | `fig4_model_comparison.png` | Predicted vs. actual: OLS / XGBoost / LightGBM | `ml_results.pkl` |
328−| 5 | `fig5_shap_summary.png` | SHAP beeswarm, top 20 features | `shap_data.pkl` |
329−| 6 | `fig6_shap_importance.png` | Mean \|SHAP\| bar chart, top 20 | `shap_data.pkl` |
330−| 7 | `fig7_geographic_prices.png` | 50,000-listing national price map (lat/lon, viridis) | `analytical_sample.pkl` |
331−| 8 | `fig8_regional_prices.png` | Log-price boxplots by Census region (with N) | `analytical_sample.pkl` |
332−| 9 | `fig9_marginal_effects.png` | Bivariate scatters + binned means (6 attributes) | `analytical_sample.pkl` |
333−| 10 | `fig10_shap_dependence.png` | SHAP dependence, 8 key features | `shap_data.pkl` |
334−| 11 | `fig11_shap_interactions.png` | SHAP for interaction terms (waterfront×sqft, pool×South) | `shap_data.pkl` |
335−
336−---
337−
338−## 📊 Results Summary
339−
340−**Predictive performance (log-price scale):**
341−
342−| Model | Random $R^2$ | Geo-holdout $R^2$ | RMSE (random) |
343 |---|---:|---:|---:| 55 |---|---:|---:|---:|
344 | OLS (62 regressors) | 0.630 | < 0 | 0.491 | 56 | OLS (62 regressors) | 0.630 | < 0 | 0.491 |
345 | OLS + State FE | 0.678 | — | 0.459 | 57 | OLS + State FE | 0.678 | — | 0.459 |
@@ -349,73 +61,128 @@ text font, muted colorblind-safe palette, no chart junk, 300 dpi).
349 | **XGBoost** | **0.833** | **0.547** | **0.330** | 61 | **XGBoost** | **0.833** | **0.547** | **0.330** |
350 | XGBoost, no geographic features | 0.830 | **0.519** | — | 62 | XGBoost, no geographic features | 0.830 | **0.519** | — |
351 | XGBoost + lat/lon | 0.870 | 0.464 | — | 63 | XGBoost + lat/lon | 0.870 | 0.464 | — |
352−| XGBoost (ablation variant, full 62) | 0.833 | 0.425 | — |
353 64
354−**Ablation cascade (XGBoost):** structural 0.498 → +lot 0.546 → +amenities 0.638 → 65+- **Geography does the heavy lifting in OLS**: region dummies → state FE → 886 ZIP3 FE raises out-of-sample R² from
355−+neighborhood **0.814 (+17.6 pp)** → +market 0.824 → full 0.833 (random $R^2$). 66+ 0.630 → 0.678 → 0.725.
67+- **Residual spatial autocorrelation**: Moran's *I* = 0.2745 (SD 0.0076) across three 5,000-listing subsamples, all
68+ *p* < 0.001.
69+- **Distributional heterogeneity**: the garage gradient is 10× larger at τ = 0.10 than at τ = 0.90 (*z* = 28.9); the pool
70+ premium only emerges above the median.
71+- **Ablation cascade (XGBoost)**: structural 0.498 → +lot 0.546 → +amenities 0.638 → +neighbourhood **0.814** → +market
72+ 0.824 → full 0.833.
73+- **SHAP rankings are model-stable** (Spearman ρ = 0.89–0.99 across XGBoost / LightGBM / RF) but not structural.
74+- **Imputation matters**: the lot-size gradient triples (0.018 → 0.059) when state-median-imputed lot sizes are dropped.
356 75
357−**Machine-readable versions** of these and all other paper tables are in 76+## 🗂 Repository layout
358−[`results/tables/`](results/tables/) (13 CSV files: OLS coefficients, QR paths,
359−inter-quantile Wald tests, stability CVs, Moran's I, imputation/winsorization
360−sensitivity, SHAP importance & cross-model correlations, ZIP3 FE, ablation).
361 77
362−--- 78+```
363− 79+wp3_uqo/
364−## ✅ Verification & Integrity 80+├── README.md
365− 81+├── requirements.txt # pinned dependencies (validated 2026-08-05)
366−The restructuring included a systematic audit reconciling the manuscript against 82+├── docs/ # AUDIT.md · CHANGES.md · PAPER_REVIEW.md · UPGRADE_REPORT.md
367−the stored result artifacts. Highlights: 83+├── data/ # NOT tracked (3.2 GB)
368− 84+│ ├── raw/us_housing.duckdb # raw Zillow extract (839,313 × 116 + 4 aux tables)
369−- **Exact matches** — OLS residual diagnostics (skewness 0.18, kurtosis 5.55, 85+│ └── processed/ # analytical_sample.pkl · model_data.pkl · shap_data.pkl
370− Jarque–Bera 217,078), test-set $R^2$ (OLS 0.6301, XGBoost 0.8330, 86+├── src/wp3/ # shared library
371− LightGBM 0.8088), Moran's $I$ subsample values and mean/SD, QR coefficients at 87+│ ├── config.py # paths (WP3_ROOT-overridable), constants, feature blocks, holdout states, seeds
372− all five quantiles, and the OLS reference line (0.312) in Figure 3. 88+│ ├── data.py # loaders + 62-regressor design-matrix builder
373−- **End-to-end reproduction** — re-running the refactored Moran's $I$ pipeline 89+│ ├── models.py # XGBoost training helper (dual validation schemes)
374− reproduces the stored values to 4 decimal places (0.2742 / 0.2839 / 0.2652). 90+│ └── plotting.py # journal figure style + variable labels
375−- **Manuscript corrections** — six internal inconsistencies were found and fixed 91+├── scripts/ # the pipeline, in order
376− (wrong KNN $k$ in the text, mislabeled inter-quantile sign convention, 92+│ ├── 01_spatial_robustness.py # ZIP3 FE OLS · Moran's I · XGB ± geography · ablation
377− winsorization description not matching the actual procedure, a duplicated table 93+│ ├── 02_qr_imputation.py # QR stability · inter-quantile Wald · imputation · winsorisation
378− row, mixed in/out-of-sample $R^2$ in the summary matrix). Every fix aligns the 94+│ ├── 03_make_figures.py # all 11 figures + verification stats
379− text to the stored results and is itemized in [`CHANGES.md`](CHANGES.md) §4. 95+│ └── 04_export_tables.py # paper tables → results/tables/*.csv
380− **No scientific result was altered.** 96+├── results/
381−- **Completed placeholders** — 13 `[TODO]` table cells in the original manuscript 97+│ ├── *.pkl # NOT tracked (up to 437 MB)
382− were filled with the exact values from the result pickles. 98+│ └── tables/ # 13 CSVs mirroring the paper's tables
383− 99+├── figures/ # 11 paper figures (PNG, 300 dpi)
384−See [`AUDIT.md`](AUDIT.md) for the full audit and [`CHANGES.md`](CHANGES.md) for 100+└── paper/ # main.tex + sections/ + appendix/ + references.bib → main.pdf
385−the change log. 101+```
386 102
387−--- 103+## 🚀 Quick start
388 104
389−## 🕰 Project History & Provenance 105+```bash
390− 106+python3 -m venv .venv && source .venv/bin/activate
391−- The project was completed in **May 2026** in a flat working folder 107+pip install -r requirements.txt
392− (`immo-wp3-spb-20260519`), then restructured into this compendium on 108+# place the artifacts (request them, see Data) in data/raw/, data/processed/, results/
393− **2026-08-05**. The original archive is preserved untouched. 109+export WP3_ROOT=/path/to/artifacts # optional, defaults to the repo root
394−- The scripts that built the analytical sample and estimated the primary models
395− (OLS/QR/ML/SHAP) were **lost** prior to restructuring; only two robustness
396− scripts survived and were refactored into `scripts/01` and `scripts/02`. The
397− stored pickles are the canonical record of those stages and every downstream
398− number is validated against them.
399−- Five paper figures had been deleted from the archive; they were rebuilt from the
400− stored data, and subsequently **all 11 figures** were regenerated in a unified
401− journal style.
402−- Three earlier monolithic drafts (`paper_hedonic_pricing{,_v2,_v3}.tex`) precede
403− the sectioned `wp3/` source that became `paper/`.
404 110
405−--- 111+python scripts/01_spatial_robustness.py # ~1–2 h (16 XGBoost models)
112+python scripts/02_qr_imputation.py # ~1–3 h (12 quantile regressions on 150k obs)
113+python scripts/03_make_figures.py # ~5 min, all 11 figures (--figs fig2 fig4 for a subset)
114+python scripts/04_export_tables.py # ~2 min, 13 CSV tables
406 115
407−## 📖 Citation 116+cd paper && make # latexmk → main.pdf (60 pages)
117+```
408 118
409−If you use this code, the figures, or the results, please cite: 119+Apple Silicon, 32 GB RAM; scripts use all cores. Every script prints verification statistics comparing recomputed values
120+with the numbers reported in the paper.
121+
122+## 🗃 Data
123+
124+- **Source**: Zillow active for-sale listings, 2025–2026 snapshot; 839,313 residential properties × 116 variables in a
125+ DuckDB database (`properties`, `price_history`, `schools`, `tax_history`, `meta_columns`).
126+- **Analytical sample**: 788,842 listings after five filters (price $10K–$10M, living area 200–20,000 sqft, 1–10
127+ beds/baths, valid coordinates, US states + DC; 6.0 % attrition); 68 engineered columns (logs, age and age², ratios,
128+ amenity dummies, neighbourhood scores, market-status flags, 6 interactions, simplified categoricals).
129+- **Not in git**: `us_housing.duckdb` (1.6 GB), `analytical_sample.pkl` (457 MB), `model_data.pkl` (867 MB),
130+ `ml_results.pkl` (437 MB). Available on request (subject to Zillow Terms of Service):
131+ simon-pierre.boucher@uqo.ca.
132+- The scripts that built the analytical sample and estimated the primary models were lost before the 2026-08-05
133+ restructuring; the stored pickles are the canonical record of those stages and every downstream number is validated
134+ against them (`docs/AUDIT.md`).
135+
136+## 🔬 Methodology
137+
138+1. **OLS** — 62 regressors, standardised and unstandardised variants, HC3 robust SE; fixed effects at region (4), state
139+ (50) and ZIP3 (886) levels (`scripts/01`).
140+2. **Spatial diagnostics** — Moran's *I* on OLS residuals, row-standardised KNN weights (k = 8), 3 × 5,000-obs
141+ subsamples, 999 permutations.
142+3. **Quantile regression** — statsmodels `QuantReg`, τ ∈ {0.10, …, 0.90}, 150,000-obs subsample, 10-seed stability
143+ check; inter-quantile Wald *z*-tests on β̂(0.10) − β̂(0.90) (`scripts/02`).
144+4. **Machine learning** — XGBoost (1,000 trees, depth 8, lr 0.05, early stopping 50), LightGBM, RF, Ridge, Lasso,
145+ Elastic Net; random 80/20 split (seed 42) **and** 10-state geographic holdout (CA NY TX FL OH CO NC WA IL GA,
146+ 438,315 test obs) (`src/wp3/models.py`).
147+5. **Ablation** — 6 nested feature sets: structural → +lot → +amenities → +neighbourhood → +market → full.
148+6. **Interpretation** — TreeSHAP on 10,000 test obs; cross-model Spearman stability.
149+7. **Sensitivity** — state-median-imputation drop-out, lot-size winsorisation (p99.5), Bike Score cap.
150+
151+## 🖼 Figures & tables inventory
152+
153+| Figure | Content |
154+|---|---|
155+| `fig1_price_distribution` | Price & log-price histograms with medians |
156+| `fig2_ols_diagnostics` | Residuals-vs-fitted, Q-Q, density, scale-location |
157+| `fig3_quantile_coefficients` | QR coefficient paths vs. OLS benchmark (8 panels) |
158+| `fig4_model_comparison` | Predicted vs. actual: OLS / XGBoost / LightGBM |
159+| `fig5_shap_summary` | SHAP beeswarm, top 20 features |
160+| `fig6_shap_importance` | Mean \|SHAP\| bar chart, top 20 |
161+| `fig7_geographic_prices` | 50,000-listing national price map |
162+| `fig8_regional_prices` | Log-price boxplots by Census region |
163+| `fig9_marginal_effects` | Bivariate scatters + binned means (6 attributes) |
164+| `fig10_shap_dependence` | SHAP dependence, 8 key features |
165+| `fig11_shap_interactions` | SHAP for interaction terms (waterfront × sqft, pool × South) |
166+
167+Tables (`results/tables/*.csv`, 13 files): OLS coefficients, QR paths, inter-quantile Wald tests, stability CVs,
168+Moran's *I*, imputation/winsorisation sensitivity, SHAP importance and cross-model correlations, ZIP3 FE, ablation.
169+
170+## ⚠️ Limitations
171+
172+- Listing (asking) prices, not transactions; gradients are conditional associations, not willingness to pay.
173+- A single 2025–2026 snapshot: no time dimension, no repeat-sales control.
174+- Lot size is state-median-imputed for part of the sample; the gradient is sensitive to that choice.
175+- The upstream sample-construction and primary-estimation scripts are lost; those stages are reproducible only from the
176+ stored artifacts.
177+
178+## 📝 Citation
410 179
411 ```bibtex 180 ```bibtex
412−@techreport{boucher2026hedonic, 181+@techreport{boucher2026hedonicus,
413 author = {Boucher, Simon-Pierre}, 182 author = {Boucher, Simon-Pierre},
414− title = {Hedonic Housing Price Models for the United States: 183+ title = {Hedonic Housing Price Models for the United States: A Multi-Method
415− A Multi-Method Comparison of Parametric, Quantile, 184+ Comparison of Parametric, Quantile, and Machine Learning Approaches},
416− and Machine Learning Approaches}, 185+ institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives},
417− institution = {Universit\'e du Qu\'ebec en Outaouais,
418− D\'epartement des sciences administratives},
419 type = {Working Paper}, 186 type = {Working Paper},
420 number = {3}, 187 number = {3},
421 year = {2026}, 188 year = {2026},
@@ -423,20 +190,18 @@ If you use this code, the figures, or the results, please cite:
423 } 190 }
424 ``` 191 ```
425 192
426−--- 193+## 👤 Author & contact
427−
428−## 📬 Contact & License
429 194
430−**Simon-Pierre Boucher** 195+**Simon-Pierre Boucher** — Département des sciences administratives,
431−Département des sciences administratives 196+Université du Québec en Outaouais (UQO), Gatineau, QC.
432−Université du Québec en Outaouais — Gatineau (Québec), Canada 197+📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca)
433−📧 **contact@spboucher.ai** · 🎓 simon-pierre.boucher@uqo.ca
434 198
435−© 2026 Simon-Pierre Boucher. Analysis code is shared for **replication and review** 199+## 🗒 Notes
436−purposes. The underlying Zillow data are subject to Zillow's Terms of Service and
437−are **not redistributed** in this repository; the paper (`paper/main.pdf`) is a
438−working paper — please cite rather than redistribute.
439 200
440−<div align="center"> 201+- Format follows `Recherche-UQO/HARMONISATION.md` (2026-10-17): template title page, `natbib` + `aer` bibliography style,
441−<sub>Built with Python · XGBoost · LightGBM · statsmodels · SHAP · DuckDB · LaTeX — reproducible from stored artifacts end to end.</sub> 202+ `docs/` for the restructuring reports.
442−</div> 203+- The original flat May 2026 project (`immo-wp3-spb-20260519`, 3.2 GB) is archived in
204+ `Recherche-UQO/_archives/versions-mai-2026/hedonic-models-united-states--immo-wp3-20260519/`; `docs/CHANGES.md` lists
205+ the six manuscript inconsistencies corrected on 2026-08-05 (no scientific result altered).
206+- GitHub mirror: `spboucher-ai/wp3-hedonic-housing-us` (remote `github`); canonical remote is spbgit `wp3_uqo.git`.
207+- Build verified 2026-10-17: 0 errors, 0 undefined references, 70 references rendered.
renamed AUDIT.md → docs/AUDIT.md +0 −0
renamed CHANGES.md → docs/CHANGES.md +0 −0
renamed PAPER_REVIEW.md → docs/PAPER_REVIEW.md +0 −0
renamed UPGRADE_REPORT.md → docs/UPGRADE_REPORT.md +0 −0
added paper/.latexmkrc +5 −0
@@ -0,0 +1,5 @@
1+$pdf_mode = 1;
2+$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S';
3+$bibtex_use = 2;
4+$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg';
5+@default_files = ('main.tex');
modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/main.tex +2 −3
@@ -67,9 +67,8 @@
67 } 67 }
68 68
69 % --- Bibliography --- 69 % --- Bibliography ---
70−\usepackage[round, authoryear, comma]{natbib} 70+\usepackage[authoryear,round,semicolon]{natbib}
71−\setcitestyle{aysep={,}} 71+\bibliographystyle{aer}
72−\bibliographystyle{apalike}
73 72
74 % --- Headers & Footers --- 73 % --- Headers & Footers ---
75 \usepackage{fancyhdr} 74 \usepackage{fancyhdr}
modified paper/sections/titlepage.tex +8 −11
@@ -1,5 +1,3 @@
1−% Author: Simon-Pierre Boucher — contact@spboucher.ai
2−%
3 % ============================================================================ 1 % ============================================================================
4 % Title Page 2 % Title Page
5 % ============================================================================ 3 % ============================================================================
@@ -12,8 +10,8 @@
12 10
13 \vspace{0.6cm} 11 \vspace{0.6cm}
14 12
15−{\footnotesize\textsc{Universit\'e du Qu\'ebec en Outaouais}}\\[0.15cm] 13+{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm]
16−{\footnotesize\textsc{D\'epartement des sciences administratives}} 14+{\footnotesize\textsc{Département des sciences administratives}}
17 15
18 \vspace{0.8cm} 16 \vspace{0.8cm}
19 17
@@ -32,17 +30,16 @@
32 \vspace{1.2cm} 30 \vspace{1.2cm}
33 31
34 % --- Author --- 32 % --- Author ---
35−{\large \WPauthor\footnotemark[1]}\\[0.3cm] 33+{\large \WPauthor}\\[0.3cm]
36−{\normalsize\itshape \WPaffiliation} 34+{\normalsize \WPaffiliation}\\[0.15cm]
37− 35+{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm]
38−\footnotetext[1]{D\'epartement des sciences administratives, Universit\'e du Qu\'ebec en Outaouais (UQO), 283 boulevard Alexandre-Tach\'e, Gatineau, QC J9A 1L8, Canada. Email: \href{mailto:\WPemail}{\WPemail}. All errors are my own.} 36+{\small \WPaddress}
39 37
40 \vspace{0.8cm} 38 \vspace{0.8cm}
41 39
42 % --- Date & Version --- 40 % --- Date & Version ---
43−{\normalsize First draft: May 2026}\\[0.1cm] 41+{\normalsize \WPdate}\\[0.1cm]
44−{\normalsize This version: August 5, 2026}\\[0.1cm] 42+{\small Version~\WPversion}
45−{\small\itshape Version~\WPversion}
46 43
47 \end{center} 44 \end{center}
48 45
49 46
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere