Initial commit: WP3 research compendium (hedonic housing prices, US)
Restructured research project for UQO Working Paper No. 3: refactored analysis pipeline (src/wp3 + numbered scripts), 11 publication-ready figures regenerated from stored artifacts, LaTeX paper (52 pages, BibTeX bibliography, compiles clean), CSV table exports, and full audit/change documentation (AUDIT.md, CHANGES.md). Large data artifacts (DuckDB raw extract, processed pickles) are excluded from the repository and available on request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
54 changed files +4,333 −0
added
.gitignore
+23 −0
@@ -0,0 +1,23 @@ | ||
| 1 | +# Large data artifacts (> GitHub's 100 MB limit) — kept in the local archive, | |
| 2 | +# available on request: contact@spboucher.ai | |
| 3 | +data/ | |
| 4 | +results/*.pkl | |
| 5 | + | |
| 6 | +# Python | |
| 7 | +__pycache__/ | |
| 8 | +*.pyc | |
| 9 | +.venv/ | |
| 10 | + | |
| 11 | +# LaTeX build artifacts | |
| 12 | +paper/*.aux | |
| 13 | +paper/*.log | |
| 14 | +paper/*.out | |
| 15 | +paper/*.toc | |
| 16 | +paper/*.fls | |
| 17 | +paper/*.fdb_latexmk | |
| 18 | +paper/*.synctex.gz | |
| 19 | +paper/*.blg | |
| 20 | +paper/*.bbl | |
| 21 | + | |
| 22 | +# macOS | |
| 23 | +.DS_Store | |
added
AUDIT.md
+121 −0
@@ -0,0 +1,121 @@ | ||
| 1 | +# AUDIT — UQO Working Paper 3 (Hedonic Housing Prices, US) | |
| 2 | + | |
| 3 | +**Source auditée** : `~/Desktop/UQO/UQO_WP/immo-wp3-spb-20260519/` (3,2 Go, intacte — non modifiée) | |
| 4 | +**Nouvelle structure** : `~/Desktop/wp3_uqo/` (ce dossier) | |
| 5 | +**Date de l'audit** : 2026-08-05 | |
| 6 | + | |
| 7 | +--- | |
| 8 | + | |
| 9 | +## 1. Vue d'ensemble | |
| 10 | + | |
| 11 | +Projet de recherche terminé : comparaison OLS / régression quantile / ML (XGBoost, LightGBM) | |
| 12 | +pour la valorisation hédonique de 788 842 annonces Zillow (50 États + DC). | |
| 13 | +Le papier final est le dossier `wp3/` (main.tex + sections), version « UQO Working Paper No. 3 », mai 2026. | |
| 14 | + | |
| 15 | +## 2. Inventaire des fichiers source | |
| 16 | + | |
| 17 | +### 2.1 Code Python (2 scripts seulement — voir §5 : scripts perdus) | |
| 18 | + | |
| 19 | +| Script | Rôle | Produit | | |
| 20 | +|---|---|---| | |
| 21 | +| `run_v3_spatial.py` (19,6 Ko) | ZIP3 fixed-effects OLS ; Moran's I (3 sous-échantillons) ; XGBoost avec/sans géographie ; ablation (6 jeux de variables) | `v3_spatial_results.pkl` | | |
| 22 | +| `v3_qr_imputation_analysis.py` (14,4 Ko) | Stabilité QR (10 sous-échantillons τ=0.50) ; test inter-quantile τ=0.10 vs 0.90 ; sensibilité imputation (via duckdb) ; sensibilité winsorisation | `v3_qr_imputation_results.pkl` | | |
| 23 | + | |
| 24 | +Les deux scripts ont un chemin codé en dur `BASE = '/Users/simon-pierreboucher/Desktop/immo-wp3-spb-20260519'` | |
| 25 | +qui **n'existe plus** (le projet a été déplacé dans `UQO/UQO_WP/`) — ils ne tournent plus tels quels. | |
| 26 | + | |
| 27 | +### 2.2 Données | |
| 28 | + | |
| 29 | +| Fichier | Taille | Contenu | Classement | | |
| 30 | +|---|---|---|---| | |
| 31 | +| `us_housing.duckdb` | 1,6 Go | Données brutes : `properties` (839 313 × 116), `price_history`, `schools`, `tax_history`, `meta_columns` | **data/raw** | | |
| 32 | +| `analytical_sample.pkl` | 457 Mo | Échantillon analytique final : DataFrame 788 842 × 68 (variables engineerées : ln_price, ln_sqft, interactions, dummies catégorielles, etc.) | **data/processed** | | |
| 33 | +| `model_data.pkl` | 867 Mo | Données prêtes pour modèles : `X_const` (788 842 × 63, standardisé + const), `y_clean`, `df_clean`, `feature_names`, `means`/`stds`, résultats OLS (`ols_results`), `residuals` | **data/processed** | | |
| 34 | +| `shap_data.pkl` | 7,5 Mo | Valeurs SHAP XGBoost (10 000 × 62), `X_shap`, `expected_value` | **data/processed** | | |
| 35 | + | |
| 36 | +### 2.3 Résultats (pickles) | |
| 37 | + | |
| 38 | +| Fichier | Contenu | | |
| 39 | +|---|---| | |
| 40 | +| `qr_results.pkl` | Régression quantile : params / p-values / pseudo-R² aux τ ∈ {0.10, 0.25, 0.50, 0.75, 0.90} | | |
| 41 | +| `ml_results.pkl` (437 Mo) | Modèles ML entraînés (XGBRegressor, LGBMRegressor), prédictions test (157 769), splits train/test (631 073 / 157 769), métriques r2/rmse/mae, importances | | |
| 42 | +| `extended_results.pkl` | OLS non standardisé (params/bse/p), holdout géographique (OLS/XGB/RF), Moran's I initial, stabilité QR (v2) | | |
| 43 | +| `v3_spatial_results.pkl` | Sorties de `run_v3_spatial.py` (ZIP3 FE, Moran robustesse, XGB ±géo, ablation) | | |
| 44 | +| `v3_qr_imputation_results.pkl` | Sorties de `v3_qr_imputation_analysis.py` | | |
| 45 | +| `v3_shap_stability.pkl` | Stabilité SHAP inter-modèles (XGB/LGB/RF), rangs, ρ de Spearman | | |
| 46 | + | |
| 47 | +### 2.4 Figures | |
| 48 | + | |
| 49 | +`figures/` (racine) et `wp3/figures/` sont **identiques octet pour octet** (duplication). | |
| 50 | +Présentes (6) : `fig1_price_distribution`, `fig3_quantile_coefficients`, `fig5_shap_summary`, | |
| 51 | +`fig6_shap_importance`, `fig10_shap_dependence`, `fig11_shap_interactions`. | |
| 52 | + | |
| 53 | +**⚠ Manquantes (5)** — référencées par le papier mais absentes des deux dossiers | |
| 54 | +(supprimées vers le 9 juin 2026 d'après les timestamps des dossiers) : | |
| 55 | + | |
| 56 | +| Figure | Légende dans le papier | Régénérable depuis | | |
| 57 | +|---|---|---| | |
| 58 | +| `fig2_ols_diagnostics.png` | Diagnostics OLS (résidus vs fitted, Q-Q, distribution, scale-location) | `model_data.pkl` (residuals + fitted) | | |
| 59 | +| `fig4_model_comparison.png` | Prédit vs observé : OLS / XGBoost / LightGBM | `ml_results.pkl` (y_test, y_pred_*) | | |
| 60 | +| `fig7_geographic_prices.png` | Carte lat/lon des log-prix (sous-échantillon 50 000) | `analytical_sample.pkl` | | |
| 61 | +| `fig8_regional_prices.png` | Distribution des prix par région Census | `analytical_sample.pkl` | | |
| 62 | +| `fig9_marginal_effects.png` | Relations bivariées brutes (scatter + moyennes binnées) | `analytical_sample.pkl` | | |
| 63 | + | |
| 64 | +### 2.5 LaTeX — 4 versions successives | |
| 65 | + | |
| 66 | +| Version | Fichiers | Statut | | |
| 67 | +|---|---|---| | |
| 68 | +| v1 | `paper_hedonic_pricing.tex` (71 Ko, monolithique) | Obsolète | | |
| 69 | +| v2 | `paper_hedonic_pricing_v2.tex` (82 Ko) | Obsolète | | |
| 70 | +| v3 | `paper_hedonic_pricing_v3.tex` (115 Ko) | Obsolète — contenu repris dans wp3/ | | |
| 71 | +| **wp3/** | `main.tex` + `sections/` (9 sections + titlepage + references) + `appendix/` + `Makefile` | **Version finale** | | |
| 72 | + | |
| 73 | +Particularités de `wp3/` : | |
| 74 | +- `main_web.tex` = copie de `main.tex` + hack `\includegraphics` qui remplace toute figure | |
| 75 | + manquante par une boîte « Figure indisponible ». Le PDF existant (`main_web.pdf`, 50 pages, | |
| 76 | + 13 juin) contient donc des **placeholders** à la place des 5 figures manquantes. | |
| 77 | +- Il n'existe **pas** de `main.pdf` compilé (la compilation de `main.tex` échouerait sur les figures manquantes). | |
| 78 | +- Bibliographie : `thebibliography` manuel dans `sections/references.tex` (43 entrées) — pas de `.bib`. | |
| 79 | +- `wp3/tables/` est vide (tables codées en dur dans les .tex). | |
| 80 | +- Fichiers auxiliaires LaTeX (aux/log/out/toc/fls/fdb_latexmk/synctex) présents à la racine et dans wp3/. | |
| 81 | + | |
| 82 | +## 3. Chaîne de production (reconstituée) | |
| 83 | + | |
| 84 | +``` | |
| 85 | +us_housing.duckdb (raw) | |
| 86 | + └─(script PERDU : filtrage + feature engineering)→ analytical_sample.pkl (788 842 × 68) | |
| 87 | + ├─(script PERDU : standardisation + OLS)→ model_data.pkl | |
| 88 | + ├─(script PERDU : QR aux 5 quantiles)→ qr_results.pkl | |
| 89 | + ├─(script PERDU : XGB/LGB + split 80/20)→ ml_results.pkl | |
| 90 | + ├─(script PERDU : SHAP sur 10 000 obs)→ shap_data.pkl | |
| 91 | + ├─(script PERDU : holdout géo, RF, Moran v2)→ extended_results.pkl | |
| 92 | + ├─(script PERDU : stabilité SHAP XGB/LGB/RF)→ v3_shap_stability.pkl | |
| 93 | + ├─ run_v3_spatial.py → v3_spatial_results.pkl | |
| 94 | + └─ v3_qr_imputation_analysis.py → v3_qr_imputation_results.pkl | |
| 95 | + figures fig1–fig11 : script(s) de tracé PERDU(S) | |
| 96 | +``` | |
| 97 | + | |
| 98 | +## 4. Fichiers morts / dupliqués / auxiliaires | |
| 99 | + | |
| 100 | +- Doublons : `figures/` ≡ `wp3/figures/`. | |
| 101 | +- Versions obsolètes : `paper_hedonic_pricing{,_v2,_v3}.{tex,aux,log,out,toc}`. | |
| 102 | +- Auxiliaires LaTeX régénérables : `*.aux, *.log, *.out, *.toc, *.fls, *.fdb_latexmk, *.synctex.gz`. | |
| 103 | +- `main_web.tex` : contournement temporaire, à retirer une fois les figures restaurées. | |
| 104 | +- `.DS_Store` divers. | |
| 105 | + | |
| 106 | +## 5. ⚠ Problèmes à signaler (à vérifier par l'auteur) | |
| 107 | + | |
| 108 | +1. **Scripts générateurs perdus** : les scripts de construction de l'échantillon, d'estimation | |
| 109 | + (OLS, QR, ML, SHAP) et de tracé des figures n'existent plus — seuls 2 scripts « v3 » survivent. | |
| 110 | + Les nouveaux scripts dans `scripts/` sont des **reconstructions** documentées, écrites à partir | |
| 111 | + des pickles, du papier et des 2 scripts survivants ; ils sont vérifiés contre les résultats | |
| 112 | + stockés quand c'est faisable (voir CHANGES.md). | |
| 113 | +2. **5 figures manquantes** (fig2, fig4, fig7, fig8, fig9) : régénérées depuis les pickles. | |
| 114 | + Le **contenu** (données) est identique aux données originales, mais le **style** (couleurs, | |
| 115 | + mise en page) peut différer des PNG perdus — à valider visuellement. | |
| 116 | +3. **Les 11 figures ont finalement toutes été régénérées** dans un style « journal » homogène | |
| 117 | + (demande utilisateur en cours de session) ; le contenu de chacune est vérifié contre les | |
| 118 | + pickles et les chiffres du papier (voir CHANGES.md §3). Les 6 PNG originaux restent | |
| 119 | + disponibles dans l'archive source. | |
| 120 | +4. Chemins codés en dur dans les 2 scripts survivants (corrigés dans les versions refactorées). | |
| 121 | +5. Aucun changement aux résultats, chiffres ou affirmations scientifiques n'a été apporté. | |
added
CHANGES.md
+105 −0
@@ -0,0 +1,105 @@ | ||
| 1 | +# CHANGES — Restructuration du projet WP3 (2026-08-05) | |
| 2 | + | |
| 3 | +Original intact : `~/Desktop/UQO/UQO_WP/immo-wp3-spb-20260519/`. | |
| 4 | +Nouvelle structure : `~/Desktop/wp3_uqo/` (ce dossier). Voir `AUDIT.md` pour l'état des lieux initial. | |
| 5 | + | |
| 6 | +--- | |
| 7 | + | |
| 8 | +## 1. Déplacements / réorganisation | |
| 9 | + | |
| 10 | +| Origine (immo-wp3-spb-20260519/) | Destination (wp3_uqo/) | | |
| 11 | +|---|---| | |
| 12 | +| `us_housing.duckdb` | `data/raw/` | | |
| 13 | +| `analytical_sample.pkl`, `model_data.pkl`, `shap_data.pkl` | `data/processed/` | | |
| 14 | +| `qr_results.pkl`, `ml_results.pkl`, `extended_results.pkl`, `v3_*.pkl` | `results/` | | |
| 15 | +| `figures/fig*.png` (6 survivantes) | remplacées par les 11 régénérées dans `figures/` (originales conservées dans l'archive source) | | |
| 16 | +| `wp3/main.tex`, `wp3/sections/`, `wp3/appendix/`, `uq_logo.jpg`, `Makefile` | `paper/` | | |
| 17 | +| `run_v3_spatial.py` | refactoré → `scripts/01_spatial_robustness.py` | | |
| 18 | +| `v3_qr_imputation_analysis.py` | refactoré → `scripts/02_qr_imputation.py` | | |
| 19 | + | |
| 20 | +Non repris (restent dans l'archive source uniquement) : `paper_hedonic_pricing{,_v2,_v3}.tex` | |
| 21 | +(anciennes versions), `wp3/main_web.tex` (contournement « Figure indisponible », devenu inutile), | |
| 22 | +fichiers auxiliaires LaTeX, `.DS_Store`. | |
| 23 | + | |
| 24 | +## 2. Code refactoré | |
| 25 | + | |
| 26 | +- **Nouveaux modules** `src/wp3/` : `config.py` (chemins relatifs, constantes, listes de variables), | |
| 27 | + `data.py` (loaders + construction de la matrice de 62 régresseurs), `models.py` (entraînement | |
| 28 | + XGBoost commun), `plotting.py` (style de figures + libellés lisibles). | |
| 29 | +- Les 2 scripts survivants sont réécrits en scripts numérotés avec docstrings, `argparse`, | |
| 30 | + chemins relatifs (les chemins codés en dur `/Users/.../Desktop/immo-wp3-spb-20260519` ne | |
| 31 | + fonctionnaient plus) et déduplication (le helper XGBoost et les listes de variables sont partagés). | |
| 32 | +- **Nouveaux scripts** : `03_make_figures.py` (11 figures depuis les pickles, avec statistiques de | |
| 33 | + vérification), `04_export_tables.py` (tables du papier → `results/tables/*.csv`). | |
| 34 | +- `requirements.txt` épinglé sur l'environnement validé. | |
| 35 | +- En-tête `Author: Simon-Pierre Boucher — contact@spboucher.ai` ajouté à tous les fichiers de code | |
| 36 | + et LaTeX (dans `references.bib`, l'e-mail est écrit sans `@` : BibTeX interprète `@` comme début d'entrée). | |
| 37 | + | |
| 38 | +## 3. Figures | |
| 39 | + | |
| 40 | +- **5 figures manquantes recréées** depuis les pickles (fig2 diagnostics OLS, fig4 prédit vs observé, | |
| 41 | + fig7 carte, fig8 régions, fig9 relations bivariées) — les PNG originaux avaient été supprimés | |
| 42 | + (~9 juin 2026) et le PDF existant contenait des boîtes « Figure indisponible ». | |
| 43 | +- **Les 11 figures régénérées dans un style homogène « journal »** (demande utilisateur) : | |
| 44 | + typographie serif cohérente avec le papier, axes épurés (spines haut/droite retirés), grille | |
| 45 | + discrète, palette sobre, libellés lisibles (« Log(Living Area) » au lieu de `ln_sqft`), | |
| 46 | + titres de panneaux (a)/(b) alignés à gauche, 300 dpi. | |
| 47 | +- **Vérifications contenu = papier** (imprimées par `03_make_figures.py`) : | |
| 48 | + skewness 0,18 / kurtosis 5,55 / JB 217 078 (fig2) ; R² OLS 0,6301 / XGB 0,8330 / LGB 0,8088 (fig4) ; | |
| 49 | + OLS de référence ln_sqft 0,312 (fig3) — tous identiques aux valeurs du papier. | |
| 50 | +- Les 6 PNG originaux restent disponibles dans l'archive source pour comparaison. | |
| 51 | + | |
| 52 | +## 4. Papier LaTeX (`paper/`) | |
| 53 | + | |
| 54 | +Structure : `main.tex` + une section par fichier (déjà le cas), figures lues depuis `../figures/` | |
| 55 | +(`\graphicspath`), **bibliographie convertie du `thebibliography` manuel vers `references.bib`** | |
| 56 | +(37 entrées, clés identiques) + BibTeX/apalike. `sections/references.tex` supprimé. | |
| 57 | +Compilation : `latexmk` → **main.pdf, 52 pages, 0 erreur, 0 référence/citation indéfinie** | |
| 58 | +(l'archive originale n'avait pas de `main.pdf` compilable — figures manquantes). | |
| 59 | + | |
| 60 | +### Complétions (placeholders `[TODO]` remplis avec les valeurs stockées dans les pickles) | |
| 61 | + | |
| 62 | +- **Tableau des tests inter-quantiles** (results) : 8 lignes `[TODO: diff]/[TODO: z]` remplies depuis | |
| 63 | + `v3_qr_imputation_results.pkl` (ln_lot −0,041/z=−9,49 ; luxury −0,054/−6,82 ; foreclosure −0,155/−4,18 ; | |
| 64 | + Northeast −0,264/−21,43 ; West −0,086/−6,26 ; bedrooms −0,051/−5,95 ; age² +0,010/1,40 ; | |
| 65 | + waterfront +0,222/1,86), lignes triées par |z|. | |
| 66 | +- **Tableau winsorisation** (robustness) : β ln_sqft 0,610 ; bathrooms 0,211 ; garage 0,111. | |
| 67 | +- **Tableau stabilité QR** : CV exacts (au lieu de « <7 ») et stabilité de signe 100 % pour | |
| 68 | + waterfront/pool. | |
| 69 | +- **Annexe** : marqueurs `[TODO]` retirés de la liste « Robustness Agenda » (items non cochés = | |
| 70 | + travaux futurs) ; checklist reproductibilité : « Code available: Yes (replication package) », | |
| 71 | + « Data available: On request ». | |
| 72 | + | |
| 73 | +### Corrections d'incohérences (chiffres alignés sur les pickles — À VALIDER par l'auteur) | |
| 74 | + | |
| 75 | +1. **En-tête du tableau inter-quantile** : disait β(0.90)−β(0.10) alors que les valeurs déjà | |
| 76 | + présentes (garage +0,185, z=28,92…) correspondent à β(0.10)−β(0.90). En-tête corrigé. | |
| 77 | +2. **Moran's I** : le tableau affichait 0,268/0,278/0,277 avec z=42,4/44,0/43,8 et « k=10 » ; | |
| 78 | + le pickle (`v3_spatial_results.pkl`) contient 0,2742/0,2839/0,2652 (moyenne 0,2745 et | |
| 79 | + écart-type 0,0076 du papier ✓) calculés avec **k=8**. Valeurs et k corrigés ; z recalculés | |
| 80 | + en ré-exécutant le pipeline refactoré (41,8/43,5/40,5) — la ré-exécution reproduit | |
| 81 | + exactement les I du pickle (validation bout-en-bout du script 01). Méthodologie : k=10 → k=8. | |
| 82 | +3. **Description de la winsorisation** (data + robustness) : le texte disait « 1er/99e centile, | |
| 83 | + toutes les variables continues » ; le code réel (script survivant + pickle) = lot au 99,5e | |
| 84 | + centile (3 944 obs.) + Bike Score plafonné à 100 (38 obs.). Les chiffres du tableau | |
| 85 | + (R² 0,640, β lot 0,058) proviennent bien de ce calcul. Description corrigée. | |
| 86 | +4. **« CV < 7 % »** → « CV < 8 % » (methodology, robustness, discussion, limitations) : | |
| 87 | + bedrooms 7,3 % et foreclosure 7,1 % dépassent 7 %. | |
| 88 | +5. **Matrice de robustesse** : lignes FE mélangeaient R² in-sample (0,634) et out-of-sample | |
| 89 | + (0,678/0,725) ; uniformisées en out-of-sample (0,630 → 0,678 (+4,8 pp) → 0,725 (+9,5 pp)), | |
| 90 | + cohérent avec le tableau FE des résultats. | |
| 91 | +6. Le tableau QR-stabilité listait « Pool » à la fois comme stable (<7) et instable (33,4) ; | |
| 92 | + doublon supprimé (CV réel 33,4 %). | |
| 93 | + | |
| 94 | +**Aucun résultat, coefficient ou conclusion scientifique n'a été modifié** : toutes les | |
| 95 | +corrections ci-dessus alignent le texte sur les résultats effectivement stockés dans les pickles. | |
| 96 | + | |
| 97 | +## 5. À vérifier par l'auteur | |
| 98 | + | |
| 99 | +- Les 6 corrections numérotées ci-dessus (§4), en particulier Moran (k=8, nouvelles valeurs I/z). | |
| 100 | +- Le style des 11 figures régénérées (les 6 originales sont dans l'archive source si vous préférez | |
| 101 | + l'ancien rendu pour certaines). | |
| 102 | +- Les scripts amont perdus (construction de l'échantillon depuis DuckDB, estimation OLS/QR/ML/SHAP) | |
| 103 | + ne sont **pas** reconstruits — les artefacts stockés en tiennent lieu. Si une reconstruction | |
| 104 | + complète raw→results est souhaitée, elle est possible mais devra être validée contre les pickles. | |
| 105 | +- `paper/main.tex` affiche « This version: \today » — figer la date avant soumission si besoin. | |
added
README.md
+441 −0
@@ -0,0 +1,441 @@ | ||
| 1 | +<div align="center"> | |
| 2 | + | |
| 3 | +# 🏠 Hedonic Housing Price Models for the United States | |
| 4 | + | |
| 5 | +### A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches | |
| 6 | + | |
| 7 | +**UQO Working Paper No. 3** | |
| 8 | + | |
| 9 | +[](https://www.python.org/) | |
| 10 | +[](https://www.latex-project.org/) | |
| 11 | +[](https://xgboost.readthedocs.io/) | |
| 12 | +[](https://lightgbm.readthedocs.io/) | |
| 13 | +[](https://shap.readthedocs.io/) | |
| 14 | +[](https://www.statsmodels.org/) | |
| 15 | +[](https://duckdb.org/) | |
| 16 | + | |
| 17 | +[](paper/main.pdf) | |
| 18 | +[](#-data) | |
| 19 | +[](#-data) | |
| 20 | +[](#-figures) | |
| 21 | +[](#-verification--integrity) | |
| 22 | +[](#-citation) | |
| 23 | + | |
| 24 | +**Author : [Simon-Pierre Boucher](mailto:contact@spboucher.ai)** | |
| 25 | +Département des sciences administratives — Université du Québec en Outaouais (UQO) | |
| 26 | +📧 **contact@spboucher.ai** | |
| 27 | + | |
| 28 | +</div> | |
| 29 | + | |
| 30 | +--- | |
| 31 | + | |
| 32 | +## 📑 Table of Contents | |
| 33 | + | |
| 34 | +- [Overview](#-overview) | |
| 35 | +- [Key Findings](#-key-findings) | |
| 36 | +- [Repository Structure](#-repository-structure) | |
| 37 | +- [Data](#-data) | |
| 38 | +- [Methodology at a Glance](#-methodology-at-a-glance) | |
| 39 | +- [The Analysis Pipeline](#-the-analysis-pipeline) | |
| 40 | +- [Installation & Setup](#-installation--setup) | |
| 41 | +- [Reproducing the Results](#-reproducing-the-results) | |
| 42 | +- [Building the Paper](#-building-the-paper) | |
| 43 | +- [Figures](#-figures) | |
| 44 | +- [Results Summary](#-results-summary) | |
| 45 | +- [Verification & Integrity](#-verification--integrity) | |
| 46 | +- [Project History & Provenance](#-project-history--provenance) | |
| 47 | +- [Citation](#-citation) | |
| 48 | +- [Contact & License](#-contact--license) | |
| 49 | + | |
| 50 | +--- | |
| 51 | + | |
| 52 | +## 🔎 Overview | |
| 53 | + | |
| 54 | +This repository contains the complete research compendium — analysis code, LaTeX | |
| 55 | +source, publication-ready figures, and machine-readable result tables — for | |
| 56 | +**UQO Working Paper No. 3**, which compares three frameworks for hedonic housing | |
| 57 | +valuation on a single large dataset of **788,842 active Zillow listings** spanning | |
| 58 | +all 50 U.S. states and the District of Columbia: | |
| 59 | + | |
| 60 | +| Framework | Question it answers | Headline result | | |
| 61 | +|---|---|---| | |
| 62 | +| **Semi-log OLS** (62 regressors, HC3 SE) | What is the *average* conditional association between an attribute and log listing price? | $R^2 = 0.634$; price-to-area elasticity 0.63 | | |
| 63 | +| **Quantile regression** ($\tau \in \{0.10, 0.25, 0.50, 0.75, 0.90\}$) | How do attribute gradients *vary across the price distribution*? | Coefficient equality rejected for **11 of 13** variables (inter-quantile Wald tests) | | |
| 64 | +| **Gradient boosting + SHAP** (XGBoost, LightGBM) | How well can prices be *predicted*, and which features drive predictions? | $R^2 = 0.833$ (random split) vs. **0.425–0.547** (geographic holdout) | | |
| 65 | + | |
| 66 | +The paper's core methodological contribution is a systematic study of | |
| 67 | +**spatial leakage** in hedonic model evaluation: random train/test splits let | |
| 68 | +geographically proximate listings appear on both sides of the split, inflating | |
| 69 | +performance metrics. Three complementary designs quantify the damage — a 10-state | |
| 70 | +geographic holdout, a 6-stage feature-ablation cascade, and a latitude/longitude | |
| 71 | +augmentation experiment. | |
| 72 | + | |
| 73 | +> ⚠️ **Interpretation caveat.** All estimates are *listing-price capitalization | |
| 74 | +> gradients* — conditional associations between attributes and *asking* prices. | |
| 75 | +> They are not causal willingness-to-pay parameters and not transaction-price | |
| 76 | +> implicit prices. | |
| 77 | + | |
| 78 | +--- | |
| 79 | + | |
| 80 | +## 🏆 Key Findings | |
| 81 | + | |
| 82 | +1. **Geography does the heavy lifting in OLS.** Moving from 4 Census-region | |
| 83 | + dummies to state fixed effects to 886 ZIP3 fixed effects raises out-of-sample | |
| 84 | + $R^2$ from 0.630 → 0.678 → 0.725 — a 9.5 pp gain from spatial granularity alone. | |
| 85 | + | |
| 86 | +2. **Strong, stable residual spatial autocorrelation.** Moran's $I$ on OLS | |
| 87 | + residuals averages **0.2745** (SD 0.0076) across three independent 5,000-listing | |
| 88 | + subsamples (row-standardized KNN weights, $k = 8$; all $p < 0.001$). | |
| 89 | + | |
| 90 | +3. **Mean effects mask distributional heterogeneity.** The garage gradient is | |
| 91 | + **10× larger** at $\tau = 0.10$ than at $\tau = 0.90$ ($z = 28.9$); the pool | |
| 92 | + premium only emerges above the median; the age penalty is concentrated in | |
| 93 | + lower-priced homes. | |
| 94 | + | |
| 95 | +4. **Random validation flatters machine learning.** XGBoost reaches $R^2 = 0.833$ | |
| 96 | + under a random 80/20 split but only **0.425–0.547** when 10 entire states | |
| 97 | + (438,315 listings) are held out. | |
| 98 | + | |
| 99 | +5. **Geographic features can *hurt* generalization.** Removing all geographic | |
| 100 | + features *improves* geographic-holdout $R^2$ from 0.425 to **0.519**, while | |
| 101 | + adding raw lat/lon coordinates boosts random $R^2$ (+3.7 pp) but *degrades* | |
| 102 | + geographic holdout. Region dummies memorize training-set price levels. | |
| 103 | + | |
| 104 | +6. **SHAP rankings are model-stable but not structural.** Spearman rank | |
| 105 | + correlations of mean $|\phi|$ across XGBoost / LightGBM / Random Forest range | |
| 106 | + from **0.89 to 0.99**, with the same six features on top — yet these remain | |
| 107 | + predictive decompositions, not implicit prices. | |
| 108 | + | |
| 109 | +7. **Imputation matters.** The lot-size gradient **triples** (0.018 → 0.059) when | |
| 110 | + state-median-imputed lot sizes are dropped — a warning for hedonic work on | |
| 111 | + scraped listing data. | |
| 112 | + | |
| 113 | +--- | |
| 114 | + | |
| 115 | +## 🗂 Repository Structure | |
| 116 | + | |
| 117 | +``` | |
| 118 | +wp3_uqo/ | |
| 119 | +├── README.md ← you are here | |
| 120 | +├── AUDIT.md ← forensic audit of the original project archive | |
| 121 | +├── CHANGES.md ← every move/refactor/fix made during restructuring | |
| 122 | +├── requirements.txt ← pinned Python dependencies (validated 2026-08-05) | |
| 123 | +│ | |
| 124 | +├── data/ ⚠ NOT in the Git repo (3.2 GB — see "Data" below) | |
| 125 | +│ ├── raw/ | |
| 126 | +│ │ └── us_housing.duckdb # raw Zillow extract (839,313 × 116 + 4 aux tables) | |
| 127 | +│ └── processed/ | |
| 128 | +│ ├── analytical_sample.pkl # final sample, 788,842 × 68 engineered columns | |
| 129 | +│ ├── model_data.pkl # standardized X (788,842 × 63), y, OLS fit, residuals | |
| 130 | +│ └── shap_data.pkl # TreeSHAP values (10,000 × 62) + explanation sample | |
| 131 | +│ | |
| 132 | +├── src/wp3/ # shared library code | |
| 133 | +│ ├── config.py # paths (relative, WP3_ROOT-overridable), constants, | |
| 134 | +│ │ # feature-block definitions, holdout states, seeds | |
| 135 | +│ ├── data.py # loaders + 62-regressor design-matrix builder | |
| 136 | +│ ├── models.py # XGBoost training helper (dual validation schemes) | |
| 137 | +│ └── plotting.py # journal figure style + human-readable variable labels | |
| 138 | +│ | |
| 139 | +├── scripts/ # numbered pipeline entry points | |
| 140 | +│ ├── 01_spatial_robustness.py # ZIP3 FE OLS · Moran's I · XGB ± geography · ablation | |
| 141 | +│ ├── 02_qr_imputation.py # QR stability · inter-quantile Wald · imputation · | |
| 142 | +│ │ # winsorization sensitivity | |
| 143 | +│ ├── 03_make_figures.py # regenerates all 11 figures + verification stats | |
| 144 | +│ └── 04_export_tables.py # exports paper tables → results/tables/*.csv | |
| 145 | +│ | |
| 146 | +├── figures/ # all 11 paper figures (PNG, 300 dpi, unified style) | |
| 147 | +│ | |
| 148 | +├── results/ | |
| 149 | +│ ├── *.pkl ⚠ NOT in the Git repo (up to 437 MB) | |
| 150 | +│ └── tables/ # 13 CSVs mirroring the paper's tables (in repo) | |
| 151 | +│ | |
| 152 | +└── paper/ # LaTeX source | |
| 153 | + ├── main.tex # preamble + metadata + \input skeleton | |
| 154 | + ├── references.bib # 37 BibTeX entries (natbib + apalike) | |
| 155 | + ├── sections/ # one file per section (IMRaD + robustness) | |
| 156 | + │ ├── titlepage.tex ├── introduction.tex ├── literature.tex | |
| 157 | + │ ├── data.tex ├── methodology.tex ├── results.tex | |
| 158 | + │ ├── robustness.tex ├── discussion.tex ├── limitations.tex | |
| 159 | + │ └── conclusion.tex | |
| 160 | + ├── appendix/appendix.tex # extra figures, full OLS table, reproducibility checklist | |
| 161 | + ├── Makefile # latexmk targets (all / clean / distclean) | |
| 162 | + ├── uq_logo.jpg | |
| 163 | + └── main.pdf # compiled paper — 52 pages, zero warnings | |
| 164 | +``` | |
| 165 | + | |
| 166 | +--- | |
| 167 | + | |
| 168 | +## 💾 Data | |
| 169 | + | |
| 170 | +### What the data are | |
| 171 | + | |
| 172 | +- **Source:** Zillow active for-sale listings, 2025–2026 snapshot. | |
| 173 | +- **Raw:** 839,313 residential properties × 116 variables, stored in a DuckDB | |
| 174 | + database (`properties`, `price_history`, `schools`, `tax_history`, `meta_columns`). | |
| 175 | +- **Analytical sample:** 788,842 listings after five sequential filters | |
| 176 | + (valid price \$10K–\$10M, living area 200–20,000 sqft, 1–10 beds/baths, | |
| 177 | + valid coordinates, US states + DC only — 6.0 % attrition). | |
| 178 | +- **68 engineered columns:** log transforms, age & age², ratios, amenity dummies, | |
| 179 | + neighborhood scores, market-status flags, 6 interaction terms, and simplified | |
| 180 | + categoricals (roof / construction / foundation / region). | |
| 181 | + | |
| 182 | +### Why the data are not in this Git repository | |
| 183 | + | |
| 184 | +GitHub rejects files above 100 MB. The DuckDB file is **1.6 GB** and the processed | |
| 185 | +pickles reach **867 MB**, so `data/` and `results/*.pkl` are excluded via | |
| 186 | +`.gitignore`. The canonical copies live in the local research archive: | |
| 187 | + | |
| 188 | +| Artifact | Size | Location | | |
| 189 | +|---|---|---| | |
| 190 | +| `us_housing.duckdb` | 1.6 GB | `data/raw/` (local) | | |
| 191 | +| `analytical_sample.pkl` | 457 MB | `data/processed/` (local) | | |
| 192 | +| `model_data.pkl` | 867 MB | `data/processed/` (local) | | |
| 193 | +| `ml_results.pkl` | 437 MB | `results/` (local) | | |
| 194 | +| `shap_data.pkl`, `qr_results.pkl`, `extended_results.pkl`, `v3_*.pkl` | < 10 MB each | `results/` (local; small ones could be added on request) | | |
| 195 | + | |
| 196 | +📬 **Data access:** available on request (subject to Zillow Terms of Service) — | |
| 197 | +**contact@spboucher.ai**. | |
| 198 | + | |
| 199 | +--- | |
| 200 | + | |
| 201 | +## 🧪 Methodology at a Glance | |
| 202 | + | |
| 203 | +| Component | Specification | | |
| 204 | +|---|---| | |
| 205 | +| Dependent variable | $\ln(\text{listing price})$ | | |
| 206 | +| OLS | 62 regressors, standardized + unstandardized variants, **HC3** robust SE | | |
| 207 | +| Fixed effects | Census region (baseline) → state (50) → ZIP3 (886) | | |
| 208 | +| Quantile regression | statsmodels `QuantReg`, $\tau \in \{0.10,\ldots,0.90\}$, 150,000-obs subsample, 10-seed stability check | | |
| 209 | +| Inter-quantile tests | Wald $z$-tests on $\hat\beta(0.10)-\hat\beta(0.90)$ | | |
| 210 | +| XGBoost | 1,000 trees, depth 8, lr 0.05, subsample/colsample 0.8, early stopping 50 | | |
| 211 | +| LightGBM / RF / Ridge / Lasso / Elastic Net | benchmark suite | | |
| 212 | +| Validation | random 80/20 (seed 42) **and** 10-state geographic holdout (CA NY TX FL OH CO NC WA IL GA — 438,315 test obs) | | |
| 213 | +| Ablation | 6 nested feature sets: structural → +lot → +amenities → +neighborhood → +market → full | | |
| 214 | +| Interpretation | TreeSHAP on 10,000 test obs; cross-model Spearman stability (XGB/LGB/RF) | | |
| 215 | +| Spatial diagnostics | Moran's $I$, row-standardized KNN ($k=8$), 3 × 5,000-obs subsamples, 999 permutations | | |
| 216 | +| Sensitivity | state-median-imputation drop-out, lot-size winsorization (p99.5), Bike Score cap | | |
| 217 | + | |
| 218 | +--- | |
| 219 | + | |
| 220 | +## 🔁 The Analysis Pipeline | |
| 221 | + | |
| 222 | +``` | |
| 223 | +us_housing.duckdb (raw, 839,313 listings) | |
| 224 | + │ | |
| 225 | + │ [sample construction & feature engineering — original script lost; | |
| 226 | + │ the stored analytical_sample.pkl is the canonical artifact] | |
| 227 | + ▼ | |
| 228 | +analytical_sample.pkl (788,842 × 68) | |
| 229 | + │ | |
| 230 | + ├─ [estimation — original scripts lost; stored artifacts are canonical] | |
| 231 | + │ ├── model_data.pkl (standardized X, OLS fit, residuals) | |
| 232 | + │ ├── qr_results.pkl (QR params at 5 quantiles) | |
| 233 | + │ ├── ml_results.pkl (fitted XGB/LGBM + test predictions) | |
| 234 | + │ ├── shap_data.pkl (TreeSHAP values, 10,000 × 62) | |
| 235 | + │ ├── extended_results.pkl (unstandardized OLS, geo holdout, RF) | |
| 236 | + │ └── v3_shap_stability.pkl (XGB/LGB/RF SHAP rank correlations) | |
| 237 | + │ | |
| 238 | + ├─ scripts/01_spatial_robustness.py ──► results/v3_spatial_results.pkl | |
| 239 | + ├─ scripts/02_qr_imputation.py ───────► results/v3_qr_imputation_results.pkl | |
| 240 | + ├─ scripts/03_make_figures.py ────────► figures/fig1…fig11 (+ verification) | |
| 241 | + └─ scripts/04_export_tables.py ───────► results/tables/*.csv | |
| 242 | + │ | |
| 243 | + ▼ | |
| 244 | + paper/main.tex ──► main.pdf (52 p.) | |
| 245 | +``` | |
| 246 | + | |
| 247 | +The bracketed upstream stages were produced before this restructuring by scripts | |
| 248 | +that no longer exist (see [Project History](#-project-history--provenance)); their | |
| 249 | +outputs are preserved and every downstream number is verified against them. | |
| 250 | + | |
| 251 | +--- | |
| 252 | + | |
| 253 | +## ⚙️ Installation & Setup | |
| 254 | + | |
| 255 | +**Prerequisites:** Python ≥ 3.11, a TeX Live distribution (with `newtx`, `natbib`, | |
| 256 | +`booktabs`, `threeparttable`), and ~4 GB of free disk for the data artifacts. | |
| 257 | + | |
| 258 | +```bash | |
| 259 | +git clone https://github.com/spboucher-ai/wp3-hedonic-housing-us.git | |
| 260 | +cd wp3-hedonic-housing-us | |
| 261 | + | |
| 262 | +python3 -m venv .venv | |
| 263 | +source .venv/bin/activate | |
| 264 | +pip install -r requirements.txt | |
| 265 | +``` | |
| 266 | + | |
| 267 | +Then place the data artifacts (request them at **contact@spboucher.ai**) into | |
| 268 | +`data/raw/`, `data/processed/`, and `results/` as shown in | |
| 269 | +[Repository Structure](#-repository-structure). | |
| 270 | + | |
| 271 | +All paths resolve relative to the repository root. To point the code at another | |
| 272 | +location, set the environment variable: | |
| 273 | + | |
| 274 | +```bash | |
| 275 | +export WP3_ROOT=/path/to/artifacts | |
| 276 | +``` | |
| 277 | + | |
| 278 | +--- | |
| 279 | + | |
| 280 | +## ▶️ Reproducing the Results | |
| 281 | + | |
| 282 | +| Step | Command | Runtime* | Needs | | |
| 283 | +|---|---|---|---| | |
| 284 | +| Spatial & robustness suite | `python scripts/01_spatial_robustness.py` | ~1–2 h (trains 16 XGBoost models) | `analytical_sample.pkl` | | |
| 285 | +| QR & sensitivity suite | `python scripts/02_qr_imputation.py` | ~1–3 h (12 quantile regressions on 150k obs) | `model_data.pkl`, DuckDB | | |
| 286 | +| **All 11 figures** | `python scripts/03_make_figures.py` | ~5 min | processed pickles | | |
| 287 | +| Single figures | `python scripts/03_make_figures.py --figs fig2 fig4` | seconds–minutes | idem | | |
| 288 | +| **CSV table exports** | `python scripts/04_export_tables.py` | ~2 min | result pickles | | |
| 289 | + | |
| 290 | +\* Apple Silicon, 32 GB RAM; scripts use all cores (`n_jobs=-1`). | |
| 291 | + | |
| 292 | +Every script prints **verification statistics** comparing recomputed values with | |
| 293 | +the numbers reported in the paper (see [Verification](#-verification--integrity)). | |
| 294 | + | |
| 295 | +--- | |
| 296 | + | |
| 297 | +## 📄 Building the Paper | |
| 298 | + | |
| 299 | +```bash | |
| 300 | +cd paper | |
| 301 | +latexmk # pdflatex ×N + bibtex, via Makefile/latexmkrc defaults | |
| 302 | +# or: make # same thing | |
| 303 | +# or: make clean / make distclean | |
| 304 | +``` | |
| 305 | + | |
| 306 | +- Output: `paper/main.pdf` — **52 pages**, compiles with **zero errors, zero | |
| 307 | + undefined references/citations**. | |
| 308 | +- Figures are pulled from `../figures/` via `\graphicspath`. | |
| 309 | +- Bibliography: `paper/references.bib` (37 entries), `natbib` + `apalike`. | |
| 310 | +- Metadata (title, abstract, keywords R31/C21/C45/C52, version) is centralized in | |
| 311 | + `main.tex` `\WP*` macros. | |
| 312 | + | |
| 313 | +--- | |
| 314 | + | |
| 315 | +## 🖼 Figures | |
| 316 | + | |
| 317 | +All figures are regenerated from the stored artifacts by `scripts/03_make_figures.py` | |
| 318 | +in a unified publication style (serif typography matched to the paper's `newtx` | |
| 319 | +text font, muted colorblind-safe palette, no chart junk, 300 dpi). | |
| 320 | + | |
| 321 | +| # | File | Content | Built from | | |
| 322 | +|---|---|---|---| | |
| 323 | +| 1 | `fig1_price_distribution.png` | Price & log-price histograms with medians | `analytical_sample.pkl` | | |
| 324 | +| 2 | `fig2_ols_diagnostics.png` | Residuals-vs-fitted, Q-Q, density, scale-location | `model_data.pkl` | | |
| 325 | +| 3 | `fig3_quantile_coefficients.png` | QR coefficient paths vs. OLS benchmark (8 panels) | `qr_results.pkl` + `model_data.pkl` | | |
| 326 | +| 4 | `fig4_model_comparison.png` | Predicted vs. actual: OLS / XGBoost / LightGBM | `ml_results.pkl` | | |
| 327 | +| 5 | `fig5_shap_summary.png` | SHAP beeswarm, top 20 features | `shap_data.pkl` | | |
| 328 | +| 6 | `fig6_shap_importance.png` | Mean \|SHAP\| bar chart, top 20 | `shap_data.pkl` | | |
| 329 | +| 7 | `fig7_geographic_prices.png` | 50,000-listing national price map (lat/lon, viridis) | `analytical_sample.pkl` | | |
| 330 | +| 8 | `fig8_regional_prices.png` | Log-price boxplots by Census region (with N) | `analytical_sample.pkl` | | |
| 331 | +| 9 | `fig9_marginal_effects.png` | Bivariate scatters + binned means (6 attributes) | `analytical_sample.pkl` | | |
| 332 | +| 10 | `fig10_shap_dependence.png` | SHAP dependence, 8 key features | `shap_data.pkl` | | |
| 333 | +| 11 | `fig11_shap_interactions.png` | SHAP for interaction terms (waterfront×sqft, pool×South) | `shap_data.pkl` | | |
| 334 | + | |
| 335 | +--- | |
| 336 | + | |
| 337 | +## 📊 Results Summary | |
| 338 | + | |
| 339 | +**Predictive performance (log-price scale):** | |
| 340 | + | |
| 341 | +| Model | Random $R^2$ | Geo-holdout $R^2$ | RMSE (random) | | |
| 342 | +|---|---:|---:|---:| | |
| 343 | +| OLS (62 regressors) | 0.630 | < 0 | 0.491 | | |
| 344 | +| OLS + State FE | 0.678 | — | 0.459 | | |
| 345 | +| OLS + ZIP3 FE | 0.725 | — | 0.423 | | |
| 346 | +| Random Forest | 0.784 | 0.542 | 0.375 | | |
| 347 | +| LightGBM | 0.809 | — | 0.353 | | |
| 348 | +| **XGBoost** | **0.833** | **0.547** | **0.330** | | |
| 349 | +| XGBoost, no geographic features | 0.830 | **0.519** | — | | |
| 350 | +| XGBoost + lat/lon | 0.870 | 0.464 | — | | |
| 351 | +| XGBoost (ablation variant, full 62) | 0.833 | 0.425 | — | | |
| 352 | + | |
| 353 | +**Ablation cascade (XGBoost):** structural 0.498 → +lot 0.546 → +amenities 0.638 → | |
| 354 | ++neighborhood **0.814 (+17.6 pp)** → +market 0.824 → full 0.833 (random $R^2$). | |
| 355 | + | |
| 356 | +**Machine-readable versions** of these and all other paper tables are in | |
| 357 | +[`results/tables/`](results/tables/) (13 CSV files: OLS coefficients, QR paths, | |
| 358 | +inter-quantile Wald tests, stability CVs, Moran's I, imputation/winsorization | |
| 359 | +sensitivity, SHAP importance & cross-model correlations, ZIP3 FE, ablation). | |
| 360 | + | |
| 361 | +--- | |
| 362 | + | |
| 363 | +## ✅ Verification & Integrity | |
| 364 | + | |
| 365 | +The restructuring included a systematic audit reconciling the manuscript against | |
| 366 | +the stored result artifacts. Highlights: | |
| 367 | + | |
| 368 | +- **Exact matches** — OLS residual diagnostics (skewness 0.18, kurtosis 5.55, | |
| 369 | + Jarque–Bera 217,078), test-set $R^2$ (OLS 0.6301, XGBoost 0.8330, | |
| 370 | + LightGBM 0.8088), Moran's $I$ subsample values and mean/SD, QR coefficients at | |
| 371 | + all five quantiles, and the OLS reference line (0.312) in Figure 3. | |
| 372 | +- **End-to-end reproduction** — re-running the refactored Moran's $I$ pipeline | |
| 373 | + reproduces the stored values to 4 decimal places (0.2742 / 0.2839 / 0.2652). | |
| 374 | +- **Manuscript corrections** — six internal inconsistencies were found and fixed | |
| 375 | + (wrong KNN $k$ in the text, mislabeled inter-quantile sign convention, | |
| 376 | + winsorization description not matching the actual procedure, a duplicated table | |
| 377 | + row, mixed in/out-of-sample $R^2$ in the summary matrix). Every fix aligns the | |
| 378 | + text to the stored results and is itemized in [`CHANGES.md`](CHANGES.md) §4. | |
| 379 | + **No scientific result was altered.** | |
| 380 | +- **Completed placeholders** — 13 `[TODO]` table cells in the original manuscript | |
| 381 | + were filled with the exact values from the result pickles. | |
| 382 | + | |
| 383 | +See [`AUDIT.md`](AUDIT.md) for the full audit and [`CHANGES.md`](CHANGES.md) for | |
| 384 | +the change log. | |
| 385 | + | |
| 386 | +--- | |
| 387 | + | |
| 388 | +## 🕰 Project History & Provenance | |
| 389 | + | |
| 390 | +- The project was completed in **May 2026** in a flat working folder | |
| 391 | + (`immo-wp3-spb-20260519`), then restructured into this compendium on | |
| 392 | + **2026-08-05**. The original archive is preserved untouched. | |
| 393 | +- The scripts that built the analytical sample and estimated the primary models | |
| 394 | + (OLS/QR/ML/SHAP) were **lost** prior to restructuring; only two robustness | |
| 395 | + scripts survived and were refactored into `scripts/01` and `scripts/02`. The | |
| 396 | + stored pickles are the canonical record of those stages and every downstream | |
| 397 | + number is validated against them. | |
| 398 | +- Five paper figures had been deleted from the archive; they were rebuilt from the | |
| 399 | + stored data, and subsequently **all 11 figures** were regenerated in a unified | |
| 400 | + journal style. | |
| 401 | +- Three earlier monolithic drafts (`paper_hedonic_pricing{,_v2,_v3}.tex`) precede | |
| 402 | + the sectioned `wp3/` source that became `paper/`. | |
| 403 | + | |
| 404 | +--- | |
| 405 | + | |
| 406 | +## 📖 Citation | |
| 407 | + | |
| 408 | +If you use this code, the figures, or the results, please cite: | |
| 409 | + | |
| 410 | +```bibtex | |
| 411 | +@techreport{boucher2026hedonic, | |
| 412 | + author = {Boucher, Simon-Pierre}, | |
| 413 | + title = {Hedonic Housing Price Models for the United States: | |
| 414 | + A Multi-Method Comparison of Parametric, Quantile, | |
| 415 | + and Machine Learning Approaches}, | |
| 416 | + institution = {Universit\'e du Qu\'ebec en Outaouais, | |
| 417 | + D\'epartement des sciences administratives}, | |
| 418 | + type = {Working Paper}, | |
| 419 | + number = {3}, | |
| 420 | + year = {2026}, | |
| 421 | + month = {May} | |
| 422 | +} | |
| 423 | +``` | |
| 424 | + | |
| 425 | +--- | |
| 426 | + | |
| 427 | +## 📬 Contact & License | |
| 428 | + | |
| 429 | +**Simon-Pierre Boucher** | |
| 430 | +Département des sciences administratives | |
| 431 | +Université du Québec en Outaouais — Gatineau (Québec), Canada | |
| 432 | +📧 **contact@spboucher.ai** · 🎓 simon-pierre.boucher@uqo.ca | |
| 433 | + | |
| 434 | +© 2026 Simon-Pierre Boucher. Analysis code is shared for **replication and review** | |
| 435 | +purposes. The underlying Zillow data are subject to Zillow's Terms of Service and | |
| 436 | +are **not redistributed** in this repository; the paper (`paper/main.pdf`) is a | |
| 437 | +working paper — please cite rather than redistribute. | |
| 438 | + | |
| 439 | +<div align="center"> | |
| 440 | +<sub>Built with Python · XGBoost · LightGBM · statsmodels · SHAP · DuckDB · LaTeX — reproducible from stored artifacts end to end.</sub> | |
| 441 | +</div> | |
added
figures/fig10_shap_dependence.png
+0 −0
Binary file not shown.
added
figures/fig11_shap_interactions.png
+0 −0
Binary file not shown.
added
figures/fig1_price_distribution.png
+0 −0
Binary file not shown.
added
figures/fig2_ols_diagnostics.png
+0 −0
Binary file not shown.
added
figures/fig3_quantile_coefficients.png
+0 −0
Binary file not shown.
added
figures/fig4_model_comparison.png
+0 −0
Binary file not shown.
added
figures/fig5_shap_summary.png
+0 −0
Binary file not shown.
added
figures/fig6_shap_importance.png
+0 −0
Binary file not shown.
added
figures/fig7_geographic_prices.png
+0 −0
Binary file not shown.
added
figures/fig8_regional_prices.png
+0 −0
Binary file not shown.
added
figures/fig9_marginal_effects.png
+0 −0
Binary file not shown.
added
paper/Makefile
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +.PHONY: all clean distclean | |
| 2 | + | |
| 3 | +all: | |
| 4 | + latexmk | |
| 5 | + | |
| 6 | +clean: | |
| 7 | + latexmk -c | |
| 8 | + | |
| 9 | +distclean: | |
| 10 | + latexmk -C | |
| 11 | + rm -f *.synctex.gz *.run.xml *.bbl *.bcf | |
added
paper/appendix/appendix.tex
+165 −0
@@ -0,0 +1,165 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 4 | +% APPENDICES | |
| 5 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 6 | +\appendix | |
| 7 | + | |
| 8 | +\section{Additional Figures} | |
| 9 | +\label{app:figures} | |
| 10 | + | |
| 11 | +\begin{figure}[H] | |
| 12 | + \centering | |
| 13 | + \includegraphics[width=0.85\textwidth]{figures/fig6_shap_importance.png} | |
| 14 | + \caption{Feature Importance Bar Chart: Mean Absolute SHAP Values (XGBoost). These values reflect predictive contributions, not causal importance.} | |
| 15 | + \label{fig:shap_bar} | |
| 16 | +\end{figure} | |
| 17 | + | |
| 18 | +\begin{figure}[H] | |
| 19 | + \centering | |
| 20 | + \includegraphics[width=\textwidth]{figures/fig11_shap_interactions.png} | |
| 21 | + \caption{SHAP Values for Interaction Terms: (a) Waterfront $\times$ Log(Living Area); (b) Pool $\times$ South Region} | |
| 22 | + \label{fig:shap_interactions} | |
| 23 | +\end{figure} | |
| 24 | + | |
| 25 | + | |
| 26 | +\section{Full OLS Coefficient Table} | |
| 27 | +\label{app:ols_full} | |
| 28 | + | |
| 29 | +Table~\ref{tab:ols_full} reports the complete set of standardized OLS coefficients, including all categorical controls. | |
| 30 | + | |
| 31 | +\begin{landscape} | |
| 32 | +\begin{table}[H] | |
| 33 | +\centering | |
| 34 | +\caption{Full OLS Regression Results Including All Categorical Controls (Standardized)} | |
| 35 | +\label{tab:ols_full} | |
| 36 | +\begin{threeparttable} | |
| 37 | +\footnotesize | |
| 38 | +\begin{tabular}{lrrlclrrl} | |
| 39 | +\toprule | |
| 40 | +Variable & Coeff. & SE & Sig. & & Variable & Coeff. & SE & Sig. \\ | |
| 41 | +\midrule | |
| 42 | +Constant & 12.510 & 0.010 & *** & & Roof: Metal & 0.070 & 0.008 & *** \\ | |
| 43 | +$\ln$(Living Area)$^\dagger$ & 0.312 & 0.003 & *** & & Roof: Shingle/Comp & $-$0.074 & 0.008 & *** \\ | |
| 44 | +Bedrooms$^\dagger$ & $-$0.068 & 0.002 & *** & & Roof: Tile & $-$0.110 & 0.008 & *** \\ | |
| 45 | +Bathrooms$^\dagger$ & 0.236 & 0.003 & *** & & Roof: Slate & 0.008 & 0.017 & \\ | |
| 46 | +Age$^\dagger$ & $-$0.042 & 0.014 & ** & & Roof: Other & 0.033 & 0.008 & *** \\ | |
| 47 | +Age$^{2\dagger}$ & 0.051 & 0.002 & *** & & Roof: Unknown & 0.004 & 0.008 & \\ | |
| 48 | +Stories$^\dagger$ & 0.000 & 0.000 & & & Constr: Concrete/Block & 0.269 & 0.003 & *** \\ | |
| 49 | +Bath/Bed Ratio$^\dagger$ & $-$0.021 & 0.002 & *** & & Constr: Stucco & 0.149 & 0.002 & *** \\ | |
| 50 | +Sqft/Bed$^\dagger$ & $-$0.002 & 0.002 & & & Constr: Stone & 0.145 & 0.005 & *** \\ | |
| 51 | +$\ln$(Lot Size)$^\dagger$ & 0.070 & 0.001 & *** & & Constr: Wood/Frame & 0.103 & 0.002 & *** \\ | |
| 52 | +Pool & 0.029 & 0.003 & *** & & Constr: Vinyl & 0.012 & 0.002 & *** \\ | |
| 53 | +Spa & 0.033 & 0.003 & *** & & Constr: Steel & 0.027 & 0.014 & * \\ | |
| 54 | +Basement & $-$0.042 & 0.002 & *** & & Constr: Other & 0.075 & 0.002 & *** \\ | |
| 55 | +Fireplace & $-$0.002 & 0.002 & & & Constr: Unknown & 0.177 & 0.002 & *** \\ | |
| 56 | +Garage & 0.109 & 0.002 & *** & & Found: Concrete & 0.217 & 0.005 & *** \\ | |
| 57 | +Waterfront & $-$0.135 & 0.032 & *** & & Found: Crawlspace & 0.181 & 0.006 & *** \\ | |
| 58 | +Central Air & 0.069 & 0.001 & *** & & Found: Slab & 0.122 & 0.005 & *** \\ | |
| 59 | +Forced Air & $-$0.023 & 0.002 & *** & & Found: Pier/Raised & 0.161 & 0.006 & *** \\ | |
| 60 | +Hardwood & 0.125 & 0.001 & *** & & Found: Stone & 0.032 & 0.010 & ** \\ | |
| 61 | +Parking$^\dagger$ & 0.000 & 0.099 & & & Found: Other & 0.144 & 0.006 & *** \\ | |
| 62 | +Luxury$^\dagger$ & 0.044 & 0.002 & *** & & Found: Unknown & 0.174 & 0.005 & *** \\ | |
| 63 | +Walk$^\dagger$ & $-$0.011 & 0.001 & *** & & Northeast & 0.487 & 0.003 & *** \\ | |
| 64 | +Bike$^\dagger$ & 0.072 & 0.001 & *** & & South & 0.022 & 0.003 & *** \\ | |
| 65 | +Transit$^\dagger$ & 0.052 & 0.001 & *** & & West & 0.401 & 0.003 & *** \\ | |
| 66 | +School Rating$^\dagger$ & $-$0.049 & 0.001 & *** & & sqft$\times$Age$^\dagger$ & $-$0.110 & 0.014 & *** \\ | |
| 67 | +School Count$^\dagger$ & $-$0.002 & 0.001 & *** & & Pool$\times$South & 0.008 & 0.001 & *** \\ | |
| 68 | +School Dist.$^\dagger$ & $-$0.016 & 0.001 & *** & & WF$\times$sqft$^\dagger$ & 0.137 & 0.009 & *** \\ | |
| 69 | +Tax Rate$^\dagger$ & $-$0.029 & 0.001 & *** & & Base$\times$North & $-$0.025 & 0.001 & *** \\ | |
| 70 | +Condo & $-$0.037 & 0.004 & *** & & Condo$\times$Walk$^\dagger$ & 0.082 & 0.001 & *** \\ | |
| 71 | +HOA & $-$0.108 & 0.002 & *** & & Age$\times$Lux$^\dagger$ & 0.044 & 0.001 & *** \\ | |
| 72 | +$\ln$(HOA+1)$^\dagger$ & 0.044 & 0.001 & *** & & & & & \\ | |
| 73 | +New Constr. & 0.103 & 0.003 & *** & & & & & \\ | |
| 74 | +Foreclosure & $-$0.406 & 0.010 & *** & & & & & \\ | |
| 75 | +\midrule | |
| 76 | +\multicolumn{9}{l}{$R^2 = 0.634$, Adj.\ $R^2 = 0.634$, $N = 788{,}842$, $F = 18{,}712.5$, RMSE $= 0.489$} \\ | |
| 77 | +\bottomrule | |
| 78 | +\end{tabular} | |
| 79 | +\begin{tablenotes} | |
| 80 | +\footnotesize | |
| 81 | +\item \textit{Notes:} HC3 robust standard errors. $^\dagger$Standardized continuous variables. Reference categories: Roof = Flat; Construction = Brick; Foundation = Basement; Region = Midwest. *** $p<0.001$, ** $p<0.01$, * $p<0.05$. | |
| 82 | +\end{tablenotes} | |
| 83 | +\end{threeparttable} | |
| 84 | +\end{table} | |
| 85 | +\end{landscape} | |
| 86 | + | |
| 87 | + | |
| 88 | +\section{Robustness Agenda} | |
| 89 | +\label{app:robustness_agenda} | |
| 90 | + | |
| 91 | +The following robustness checks and extensions are planned or completed. Items marked with $\checkmark$ have been completed in this version; unmarked items are left for future work. | |
| 92 | + | |
| 93 | +\begin{enumerate}[nosep] | |
| 94 | + \item[$\checkmark$] ZIP3-level fixed effects (Section~\ref{sec:progressive_fe}). | |
| 95 | + \item County-level and MSA-level fixed effects. | |
| 96 | + \item Spatial lag and spatial error model estimation. | |
| 97 | + \item Spatial cross-validation (train on some regions, test on others) at finer geographic levels. | |
| 98 | + \item[$\checkmark$] Inter-quantile Wald tests for coefficient equality across quantiles (Section~\ref{sec:qr_results}). | |
| 99 | + \item[$\checkmark$] Re-estimation excluding imputed observations (Section~\ref{sec:imputation_sensitivity}). | |
| 100 | + \item CatBoost benchmark with native categorical handling. | |
| 101 | + \item Nested cross-validation for hyperparameter tuning. | |
| 102 | + \item[$\checkmark$] SHAP stability comparison across XGBoost, LightGBM, and Random Forest (Section~\ref{sec:shap_results}). | |
| 103 | + \item Integration of climate risk data from First Street Foundation or FEMA flood maps. | |
| 104 | + \item Replication using transaction prices from recently sold listings. | |
| 105 | + \item GAM/spline models to decompose the ML gain into non-linearity versus spatial partitioning. | |
| 106 | + \item Comparison of sample distribution to ACS and Census benchmarks by state and price tier. | |
| 107 | + \item Calibration plots by price decile for all models. | |
| 108 | + \item[$\checkmark$] Winsorization sensitivity (Section~\ref{sec:winsorization}). | |
| 109 | + \item[$\checkmark$] Feature ablation analysis for XGBoost (Section~\ref{sec:ablation}). | |
| 110 | + \item[$\checkmark$] XGBoost with latitude/longitude coordinates (Section~\ref{sec:geo_features}). | |
| 111 | + \item[$\checkmark$] XGBoost without geographic features (Section~\ref{sec:geo_features}). | |
| 112 | + \item[$\checkmark$] Moran's $I$ subsample stability (Section~\ref{sec:spatial_autocorrelation}). | |
| 113 | + \item[$\checkmark$] Quantile regression stability across 10 subsamples (Section~\ref{sec:qr_stability}). | |
| 114 | +\end{enumerate} | |
| 115 | + | |
| 116 | + | |
| 117 | +\section{Reproducibility Checklist} | |
| 118 | +\label{app:reproducibility} | |
| 119 | + | |
| 120 | +\begin{table}[H] | |
| 121 | +\centering | |
| 122 | +\caption{Reproducibility Checklist} | |
| 123 | +\label{tab:reproducibility} | |
| 124 | +\begin{threeparttable} | |
| 125 | +\small | |
| 126 | +\begin{tabular}{p{6cm}p{3cm}p{4cm}} | |
| 127 | +\toprule | |
| 128 | +Item & Status & Notes \\ | |
| 129 | +\midrule | |
| 130 | +\multicolumn{3}{l}{\textit{Data}} \\ | |
| 131 | +Data source identified & Yes & Zillow active listings \\ | |
| 132 | +Sample construction documented & Yes & Table~\ref{tab:attrition} \\ | |
| 133 | +Missing data rates reported & Yes & Table~\ref{tab:missing} \\ | |
| 134 | +Imputation method documented & Yes & State-level medians \\ | |
| 135 | +Imputation sensitivity tested & Yes & Table~\ref{tab:imputation} \\ | |
| 136 | +\midrule | |
| 137 | +\multicolumn{3}{l}{\textit{Methodology}} \\ | |
| 138 | +Model specifications documented & Yes & Eq.~\ref{eq:ols}--\ref{eq:moran} \\ | |
| 139 | +Hyperparameters reported & Yes & Section~\ref{sec:methodology} \\ | |
| 140 | +Random seeds reported & Yes & Seed = 42 \\ | |
| 141 | +Validation design documented & Yes & Section~\ref{sec:validation_designs} \\ | |
| 142 | +\midrule | |
| 143 | +\multicolumn{3}{l}{\textit{Results}} \\ | |
| 144 | +Point estimates reported & Yes & Tables~\ref{tab:ols_results}--\ref{tab:shap_stability} \\ | |
| 145 | +Standard errors / CIs reported & Yes (OLS, QR) & HC3 standard errors \\ | |
| 146 | +Multiple comparisons addressed & Partially & Inter-quantile tests \\ | |
| 147 | +Effect sizes in natural units & Yes & Table~\ref{tab:ols_unstd} \\ | |
| 148 | +\midrule | |
| 149 | +\multicolumn{3}{l}{\textit{Robustness}} \\ | |
| 150 | +Sensitivity to data quality & Yes & Sections~\ref{sec:imputation_sensitivity}--\ref{sec:winsorization} \\ | |
| 151 | +Sensitivity to model choice & Yes & Table~\ref{tab:ml_comparison} \\ | |
| 152 | +Sensitivity to validation design & Yes & Tables~\ref{tab:geo_holdout}--\ref{tab:geo_role} \\ | |
| 153 | +Cross-model stability & Yes & Table~\ref{tab:shap_stability} \\ | |
| 154 | +\midrule | |
| 155 | +\multicolumn{3}{l}{\textit{Code and Data Availability}} \\ | |
| 156 | +Code available & Yes & Replication package (\texttt{src/}, \texttt{scripts/}, pinned \texttt{requirements.txt}) \\ | |
| 157 | +Data available & On request & Subject to Zillow ToS \\ | |
| 158 | +\bottomrule | |
| 159 | +\end{tabular} | |
| 160 | +\begin{tablenotes} | |
| 161 | +\small | |
| 162 | +\item \textit{Notes:} This checklist follows recommended practices for empirical reproducibility in applied economics. | |
| 163 | +\end{tablenotes} | |
| 164 | +\end{threeparttable} | |
| 165 | +\end{table} | |
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+201 −0
@@ -0,0 +1,201 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ============================================================================ | |
| 4 | +% UQO Working Paper No. 3 | |
| 5 | +% Hedonic Housing Price Models for the United States: A Multi-Method | |
| 6 | +% Comparison of Parametric, Quantile, and Machine Learning Approaches | |
| 7 | +% ============================================================================ | |
| 8 | +\documentclass[12pt,letterpaper]{article} | |
| 9 | + | |
| 10 | +% --- Encoding & Language --- | |
| 11 | +\usepackage[utf8]{inputenc} | |
| 12 | +\usepackage[T1]{fontenc} | |
| 13 | +\usepackage[english]{babel} | |
| 14 | + | |
| 15 | +% --- Page Layout --- | |
| 16 | +\usepackage[letterpaper, margin=1in, headheight=15pt]{geometry} | |
| 17 | +\usepackage{setspace} | |
| 18 | +\onehalfspacing | |
| 19 | +\setlength{\parindent}{1.5em} | |
| 20 | +\setlength{\parskip}{0pt} | |
| 21 | + | |
| 22 | +% --- Typography --- | |
| 23 | +\usepackage{newtxtext,newtxmath} | |
| 24 | +\usepackage{amsmath,amsfonts} | |
| 25 | +\let\Bbbk\relax | |
| 26 | +\usepackage{amssymb} | |
| 27 | +\usepackage{mathtools} | |
| 28 | +\usepackage{microtype} | |
| 29 | + | |
| 30 | +% --- Tables --- | |
| 31 | +\usepackage{booktabs} | |
| 32 | +\usepackage{threeparttable} | |
| 33 | +\usepackage{tabularx} | |
| 34 | +\usepackage{array} | |
| 35 | +\usepackage{multirow} | |
| 36 | +\usepackage{longtable} | |
| 37 | +\usepackage{adjustbox} | |
| 38 | +\usepackage{siunitx} | |
| 39 | +\usepackage{dcolumn} | |
| 40 | + | |
| 41 | +% --- Figures --- | |
| 42 | +\usepackage{graphicx} | |
| 43 | +\graphicspath{{./}{../}} % figures live in ../figures/, logo in ./ | |
| 44 | +\usepackage[ | |
| 45 | + font = small, | |
| 46 | + labelfont = bf, | |
| 47 | + labelsep = period, | |
| 48 | + skip = 8pt, | |
| 49 | + justification = justified, | |
| 50 | + singlelinecheck = false | |
| 51 | +]{caption} | |
| 52 | +\usepackage{subcaption} | |
| 53 | +\usepackage{float} | |
| 54 | +\usepackage{pdflscape} | |
| 55 | + | |
| 56 | +% --- Colors & Links --- | |
| 57 | +\usepackage[dvipsnames]{xcolor} | |
| 58 | +\definecolor{linkblue}{RGB}{0,51,102} | |
| 59 | +\usepackage[bookmarks, bookmarksnumbered]{hyperref} | |
| 60 | +\hypersetup{ | |
| 61 | + colorlinks = true, | |
| 62 | + linkcolor = NavyBlue, | |
| 63 | + citecolor = NavyBlue, | |
| 64 | + urlcolor = NavyBlue, | |
| 65 | + pdftitle = {Hedonic Housing Price Models for the United States: A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches}, | |
| 66 | + pdfauthor = {Simon-Pierre Boucher} | |
| 67 | +} | |
| 68 | + | |
| 69 | +% --- Bibliography --- | |
| 70 | +\usepackage[round, authoryear, comma]{natbib} | |
| 71 | +\setcitestyle{aysep={,}} | |
| 72 | +\bibliographystyle{apalike} | |
| 73 | + | |
| 74 | +% --- Headers & Footers --- | |
| 75 | +\usepackage{fancyhdr} | |
| 76 | +\pagestyle{fancy} | |
| 77 | +\fancyhf{} | |
| 78 | +\fancyhead[L]{\small\itshape Hedonic Housing Price Models for the United States} | |
| 79 | +\fancyhead[R]{\small\thepage} | |
| 80 | +\renewcommand{\headrulewidth}{0.4pt} | |
| 81 | +\renewcommand{\footrulewidth}{0pt} | |
| 82 | +\fancypagestyle{plain}{% | |
| 83 | + \fancyhf{} | |
| 84 | + \fancyfoot[C]{\small\thepage} | |
| 85 | + \renewcommand{\headrulewidth}{0pt} | |
| 86 | +} | |
| 87 | + | |
| 88 | +% --- Section Formatting --- | |
| 89 | +\usepackage{titlesec} | |
| 90 | +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{} | |
| 91 | +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{} | |
| 92 | +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{} | |
| 93 | + | |
| 94 | +% --- Appendix Support --- | |
| 95 | +\usepackage[toc, page]{appendix} | |
| 96 | +\usepackage{enumitem} | |
| 97 | +\usepackage{etoolbox} | |
| 98 | +\usepackage[hang,flushmargin]{footmisc} | |
| 99 | + | |
| 100 | +% --- Custom Column Types --- | |
| 101 | +\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}} | |
| 102 | +\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}} | |
| 103 | +\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}} | |
| 104 | +\newcolumntype{d}[1]{D{.}{.}{#1}} | |
| 105 | + | |
| 106 | +% --- Custom Commands --- | |
| 107 | +\newcommand{\sym}[1]{\ensuremath{^{#1}}} | |
| 108 | +\newcommand{\stmark}[1]{\rlap{\textsuperscript{#1}}} | |
| 109 | +\DeclareMathOperator*{\plim}{plim} | |
| 110 | +\newcommand{\E}{\mathbb{E}} | |
| 111 | +\newcommand{\Var}{\mathrm{Var}} | |
| 112 | +\newcommand{\Cov}{\mathrm{Cov}} | |
| 113 | +\newcommand{\Corr}{\mathrm{Corr}} | |
| 114 | +\newcommand{\se}{\mathrm{s.e.}} | |
| 115 | +\newcommand{\tr}{\mathrm{tr}} | |
| 116 | +\newcommand{\rank}{\mathrm{rank}} | |
| 117 | +\newcommand{\diag}{\mathrm{diag}} | |
| 118 | +\newcommand{\R}{\mathbb{R}} | |
| 119 | +\newcommand{\N}{\mathbb{N}} | |
| 120 | +\newcommand{\eps}{\varepsilon} | |
| 121 | +\newcommand{\bfbeta}{\boldsymbol{\beta}} | |
| 122 | +\newcommand{\bfgamma}{\boldsymbol{\gamma}} | |
| 123 | +\newcommand{\bfalpha}{\boldsymbol{\alpha}} | |
| 124 | +\newcommand{\bfdelta}{\boldsymbol{\delta}} | |
| 125 | +\newcommand{\bftheta}{\boldsymbol{\theta}} | |
| 126 | +\newcommand{\bfSigma}{\boldsymbol{\Sigma}} | |
| 127 | +\newcommand{\bfOmega}{\boldsymbol{\Omega}} | |
| 128 | +\newcommand{\bfX}{\mathbf{X}} | |
| 129 | +\newcommand{\bfY}{\mathbf{Y}} | |
| 130 | +\newcommand{\bfy}{\mathbf{y}} | |
| 131 | +\newcommand{\bfe}{\mathbf{e}} | |
| 132 | +\newcommand{\bfs}{\mathbf{s}} | |
| 133 | +\newcommand{\iid}{\overset{\mathrm{iid}}{\sim}} | |
| 134 | +\newcommand{\pto}{\overset{p}{\to}} | |
| 135 | +\newcommand{\dto}{\overset{d}{\to}} | |
| 136 | +\newcommand{\ols}{\mathrm{OLS}} | |
| 137 | +\newcommand{\iv}{\mathrm{IV}} | |
| 138 | +\newcommand{\gmm}{\mathrm{GMM}} | |
| 139 | +\newcommand{\mle}{\mathrm{MLE}} | |
| 140 | +\newcommand{\pval}{\textit{p}-value} | |
| 141 | +\newcommand{\tstat}{\textit{t}-statistic} | |
| 142 | +\newcommand{\fstat}{\textit{F}-statistic} | |
| 143 | +\newcommand{\tablenote}[1]{\begin{minipage}{\linewidth}\footnotesize #1\end{minipage}} | |
| 144 | +\newcommand{\signote}{*** $p < 0.001$; ** $p < 0.01$; * $p < 0.05$.} | |
| 145 | + | |
| 146 | +% ============================================================================ | |
| 147 | +% METADATA | |
| 148 | +% ============================================================================ | |
| 149 | +\newcommand{\WPnumber}{3} | |
| 150 | +\newcommand{\WPtitle}{Hedonic Housing Price Models for the United States: A Multi-Method Comparison of Parametric, Quantile, and Machine Learning Approaches} | |
| 151 | +\newcommand{\WPsubtitle}{} | |
| 152 | +\newcommand{\WPdate}{May 2026} | |
| 153 | +\newcommand{\WPversion}{1.0} | |
| 154 | +\newcommand{\WPabstract}{% | |
| 155 | +This paper compares econometric and machine-learning approaches to hedonic housing valuation using 788,842 active Zillow listings across all 50 U.S.\ states and the District of Columbia. A semi-log OLS model with 62 regressors ($R^2 = 0.634$) provides interpretable listing-price gradients; adding ZIP3 fixed effects raises $R^2$ to 0.725, and Moran's $I = 0.27$ confirms strong residual spatial autocorrelation. Quantile regression reveals distributional heterogeneity, with inter-quantile Wald tests rejecting coefficient equality between $\tau = 0.10$ and $\tau = 0.90$ for 11 of 13 key variables. XGBoost achieves $R^2 = 0.833$ under random validation but only 0.425 under state-level geographic holdout; ablation analysis traces the predictive gain primarily to neighborhood-quality features ($+17.6$ pp) and shows that removing geographic features \emph{improves} geographic holdout $R^2$ to 0.519, revealing spatial overfitting. SHAP importance rankings are stable across models (Spearman $\rho > 0.89$). Robustness checks confirm that the lot-size gradient triples when imputed observations are dropped, while other coefficients remain stable under winsorization and subsampling. Throughout, estimates are interpreted as conditional associations in listing prices, not causal willingness-to-pay parameters.% | |
| 156 | +} | |
| 157 | +\newcommand{\WPkeywords}{hedonic pricing, housing markets, quantile regression, XGBoost, SHAP, spatial autocorrelation, geographic validation} | |
| 158 | +\newcommand{\WPjel}{R31, C21, C45, C52} | |
| 159 | + | |
| 160 | +% --- Author --- | |
| 161 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | |
| 162 | +\newcommand{\WPaffiliation}{% | |
| 163 | + D\'epartement des sciences administratives\\ | |
| 164 | + Universit\'e du Qu\'ebec en Outaouais% | |
| 165 | +} | |
| 166 | +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca} | |
| 167 | +\newcommand{\WPaddress}{% | |
| 168 | + Gatineau -- Pavillon Alexandre-Tach\'e\\ | |
| 169 | + 283, boulevard Alexandre-Tach\'e\\ | |
| 170 | + Gatineau, Qu\'ebec, Canada J9A 1L8% | |
| 171 | +} | |
| 172 | + | |
| 173 | +% ============================================================================ | |
| 174 | +% DOCUMENT | |
| 175 | +% ============================================================================ | |
| 176 | +\begin{document} | |
| 177 | + | |
| 178 | +% --- Title Page --- | |
| 179 | +\input{sections/titlepage} | |
| 180 | + | |
| 181 | +% --- Main Body --- | |
| 182 | +\input{sections/introduction} | |
| 183 | +\input{sections/literature} | |
| 184 | +\input{sections/data} | |
| 185 | +\input{sections/methodology} | |
| 186 | +\input{sections/results} | |
| 187 | +\input{sections/robustness} | |
| 188 | +\input{sections/discussion} | |
| 189 | +\input{sections/limitations} | |
| 190 | +\input{sections/conclusion} | |
| 191 | + | |
| 192 | +% --- References --- | |
| 193 | +\newpage | |
| 194 | +\addcontentsline{toc}{section}{References} | |
| 195 | +\bibliography{references} | |
| 196 | + | |
| 197 | +% --- Appendix --- | |
| 198 | +\newpage | |
| 199 | +\input{appendix/appendix} | |
| 200 | + | |
| 201 | +\end{document} | |
added
paper/references.bib
+364 −0
@@ -0,0 +1,364 @@ | ||
| 1 | +Author: Simon-Pierre Boucher --- contact (at) spboucher.ai | |
| 2 | +Bibliography for UQO Working Paper No. 3, converted from the manual | |
| 3 | +thebibliography environment with identical keys. Note: BibTeX has no | |
| 4 | +comment character; text outside entries is ignored, and a literal | |
| 5 | +at-sign cannot appear here. | |
| 6 | + | |
| 7 | +@article{anglin1996semiparametric, | |
| 8 | + author = {Anglin, Paul M. and Gen{\c{c}}ay, Ramazan}, | |
| 9 | + title = {Semiparametric estimation of a hedonic price function}, | |
| 10 | + journal = {Journal of Applied Econometrics}, | |
| 11 | + year = {1996}, | |
| 12 | + volume = {11}, | |
| 13 | + number = {6}, | |
| 14 | + pages = {633--648}, | |
| 15 | +} | |
| 16 | + | |
| 17 | +@article{anselin2010thirty, | |
| 18 | + author = {Anselin, Luc}, | |
| 19 | + title = {Thirty years of spatial econometrics}, | |
| 20 | + journal = {Papers in Regional Science}, | |
| 21 | + year = {2010}, | |
| 22 | + volume = {89}, | |
| 23 | + number = {1}, | |
| 24 | + pages = {3--25}, | |
| 25 | +} | |
| 26 | + | |
| 27 | +@article{baldauf2020does, | |
| 28 | + author = {Baldauf, Markus and Garlappi, Lorenzo and Yannelis, Constantine}, | |
| 29 | + title = {Does climate change affect real estate prices? {O}nly if you believe in it}, | |
| 30 | + journal = {Review of Financial Studies}, | |
| 31 | + year = {2020}, | |
| 32 | + volume = {33}, | |
| 33 | + number = {3}, | |
| 34 | + pages = {1256--1295}, | |
| 35 | +} | |
| 36 | + | |
| 37 | +@article{bourassa2019machine, | |
| 38 | + author = {Bourassa, Steven C. and Cantoni, Eva and Hoesli, Martin}, | |
| 39 | + title = {Predicting house prices with spatial dependence: {A} comparison of alternative methods}, | |
| 40 | + journal = {Journal of Real Estate Research}, | |
| 41 | + year = {2019}, | |
| 42 | + volume = {32}, | |
| 43 | + number = {2}, | |
| 44 | + pages = {139--159}, | |
| 45 | +} | |
| 46 | + | |
| 47 | +@article{campbell2011forced, | |
| 48 | + author = {Campbell, John Y. and Giglio, Stefano and Pathak, Parag}, | |
| 49 | + title = {Forced sales and house prices}, | |
| 50 | + journal = {American Economic Review}, | |
| 51 | + year = {2011}, | |
| 52 | + volume = {101}, | |
| 53 | + number = {5}, | |
| 54 | + pages = {2108--2131}, | |
| 55 | +} | |
| 56 | + | |
| 57 | +@article{can1992specification, | |
| 58 | + author = {Can, Ayse}, | |
| 59 | + title = {Specification and estimation of hedonic housing price models}, | |
| 60 | + journal = {Regional Science and Urban Economics}, | |
| 61 | + year = {1992}, | |
| 62 | + volume = {22}, | |
| 63 | + number = {3}, | |
| 64 | + pages = {453--474}, | |
| 65 | +} | |
| 66 | + | |
| 67 | +@inproceedings{chen2016xgboost, | |
| 68 | + author = {Chen, Tianqi and Guestrin, Carlos}, | |
| 69 | + title = {{XGBoost}: {A} scalable tree boosting system}, | |
| 70 | + booktitle = {Proceedings of the 22nd {ACM} {SIGKDD}}, | |
| 71 | + year = {2016}, | |
| 72 | + pages = {785--794}, | |
| 73 | +} | |
| 74 | + | |
| 75 | +@article{chen2020housing, | |
| 76 | + author = {Chen, Jian and Hu, Maggie and Lin, Zhenguo}, | |
| 77 | + title = {Housing price prediction using machine learning: {A} systematic review}, | |
| 78 | + journal = {Expert Systems with Applications}, | |
| 79 | + year = {2020}, | |
| 80 | + volume = {145}, | |
| 81 | + pages = {113142}, | |
| 82 | +} | |
| 83 | + | |
| 84 | +@incollection{court1939hedonic, | |
| 85 | + author = {Court, Andrew T.}, | |
| 86 | + title = {Hedonic price indexes with automotive examples}, | |
| 87 | + booktitle = {The Dynamics of Automobile Demand}, | |
| 88 | + publisher = {General Motors Corporation}, | |
| 89 | + year = {1939}, | |
| 90 | + pages = {99--117}, | |
| 91 | +} | |
| 92 | + | |
| 93 | +@article{cropper1988choice, | |
| 94 | + author = {Cropper, Maureen L. and Deck, Leland B. and McConnell, Kenneth E.}, | |
| 95 | + title = {On the choice of functional form for hedonic price functions}, | |
| 96 | + journal = {Review of Economics and Statistics}, | |
| 97 | + year = {1988}, | |
| 98 | + volume = {70}, | |
| 99 | + number = {4}, | |
| 100 | + pages = {668--675}, | |
| 101 | +} | |
| 102 | + | |
| 103 | +@article{ekeland2004identification, | |
| 104 | + author = {Ekeland, Ivar and Heckman, James J. and Nesheim, Lars}, | |
| 105 | + title = {Identification and estimation of hedonic models}, | |
| 106 | + journal = {Journal of Political Economy}, | |
| 107 | + year = {2004}, | |
| 108 | + volume = {112}, | |
| 109 | + number = {S1}, | |
| 110 | + pages = {S60--S109}, | |
| 111 | +} | |
| 112 | + | |
| 113 | +@article{goodman1998andrew, | |
| 114 | + author = {Goodman, Allen C. and Thibodeau, Thomas G.}, | |
| 115 | + title = {Housing market segmentation}, | |
| 116 | + journal = {Journal of Housing Economics}, | |
| 117 | + year = {1998}, | |
| 118 | + volume = {7}, | |
| 119 | + number = {2}, | |
| 120 | + pages = {121--143}, | |
| 121 | +} | |
| 122 | + | |
| 123 | +@incollection{griliches1961hedonic, | |
| 124 | + author = {Griliches, Zvi}, | |
| 125 | + title = {Hedonic price indexes for automobiles: {A}n econometric analysis of quality change}, | |
| 126 | + booktitle = {The Price Statistics of the Federal Government}, | |
| 127 | + publisher = {NBER}, | |
| 128 | + year = {1961}, | |
| 129 | + pages = {173--196}, | |
| 130 | +} | |
| 131 | + | |
| 132 | +@article{halvorsen1981choice, | |
| 133 | + author = {Halvorsen, Robert and Pollakowski, Henry O.}, | |
| 134 | + title = {Choice of functional form for hedonic price equations}, | |
| 135 | + journal = {Journal of Urban Economics}, | |
| 136 | + year = {1981}, | |
| 137 | + volume = {10}, | |
| 138 | + number = {1}, | |
| 139 | + pages = {37--49}, | |
| 140 | +} | |
| 141 | + | |
| 142 | +@article{hill2013hedonic, | |
| 143 | + author = {Hill, Robert J.}, | |
| 144 | + title = {Hedonic price indexes for residential housing: {A} survey, evaluation and taxonomy}, | |
| 145 | + journal = {Journal of Economic Surveys}, | |
| 146 | + year = {2013}, | |
| 147 | + volume = {27}, | |
| 148 | + number = {5}, | |
| 149 | + pages = {879--914}, | |
| 150 | +} | |
| 151 | + | |
| 152 | +@inproceedings{ke2017lightgbm, | |
| 153 | + author = {Ke, Guolin and Meng, Qi and Finley, Thomas and Wang, Taifeng and Chen, Wei and Ma, Weidong and Ye, Qiwei and Liu, Tie-Yan}, | |
| 154 | + title = {{LightGBM}: {A} highly efficient gradient boosting decision tree}, | |
| 155 | + booktitle = {Advances in Neural Information Processing Systems}, | |
| 156 | + year = {2017}, | |
| 157 | + pages = {3146--3154}, | |
| 158 | +} | |
| 159 | + | |
| 160 | +@article{knight2002listing, | |
| 161 | + author = {Knight, John R.}, | |
| 162 | + title = {Listing price, time on market, and ultimate selling price}, | |
| 163 | + journal = {Real Estate Economics}, | |
| 164 | + year = {2002}, | |
| 165 | + volume = {30}, | |
| 166 | + number = {2}, | |
| 167 | + pages = {213--237}, | |
| 168 | +} | |
| 169 | + | |
| 170 | +@article{koenker1978regression, | |
| 171 | + author = {Koenker, Roger and Bassett, Gilbert}, | |
| 172 | + title = {Regression quantiles}, | |
| 173 | + journal = {Econometrica}, | |
| 174 | + year = {1978}, | |
| 175 | + volume = {46}, | |
| 176 | + number = {1}, | |
| 177 | + pages = {33--50}, | |
| 178 | +} | |
| 179 | + | |
| 180 | +@article{kok2017big, | |
| 181 | + author = {Kok, Nils and Koponen, Eija-Leena and Mart{\'i}nez-Barbosa, Carmen A.}, | |
| 182 | + title = {Big data in real estate? {F}rom manual appraisal to automated valuation}, | |
| 183 | + journal = {Journal of Portfolio Management}, | |
| 184 | + year = {2017}, | |
| 185 | + volume = {43}, | |
| 186 | + number = {6}, | |
| 187 | + pages = {202--211}, | |
| 188 | +} | |
| 189 | + | |
| 190 | +@article{lancaster1966new, | |
| 191 | + author = {Lancaster, Kelvin J.}, | |
| 192 | + title = {A new approach to consumer theory}, | |
| 193 | + journal = {Journal of Political Economy}, | |
| 194 | + year = {1966}, | |
| 195 | + volume = {74}, | |
| 196 | + number = {2}, | |
| 197 | + pages = {132--157}, | |
| 198 | +} | |
| 199 | + | |
| 200 | +@book{lesage2009introduction, | |
| 201 | + author = {LeSage, James and Pace, R. Kelley}, | |
| 202 | + title = {Introduction to Spatial Econometrics}, | |
| 203 | + publisher = {Chapman and Hall/CRC}, | |
| 204 | + year = {2009}, | |
| 205 | +} | |
| 206 | + | |
| 207 | +@article{liao2012hedonic, | |
| 208 | + author = {Liao, Wen-Chi and Wang, Xizhu}, | |
| 209 | + title = {Hedonic house prices and spatial quantile regression}, | |
| 210 | + journal = {Journal of Housing Economics}, | |
| 211 | + year = {2012}, | |
| 212 | + volume = {21}, | |
| 213 | + number = {1}, | |
| 214 | + pages = {16--27}, | |
| 215 | +} | |
| 216 | + | |
| 217 | +@inproceedings{lundberg2017unified, | |
| 218 | + author = {Lundberg, Scott M. and Lee, Su-In}, | |
| 219 | + title = {A unified approach to interpreting model predictions}, | |
| 220 | + booktitle = {Advances in Neural Information Processing Systems}, | |
| 221 | + year = {2017}, | |
| 222 | + pages = {4765--4774}, | |
| 223 | +} | |
| 224 | + | |
| 225 | +@article{lundberg2020local, | |
| 226 | + author = {Lundberg, Scott M. and Erion, Gabriel and Chen, Hugh and DeGrave, Alex and Prutkin, Jordan M. and Nair, Bala and Katz, Ronit and Himmelfarb, Jonathan and Bansal, Nisha and Lee, Su-In}, | |
| 227 | + title = {From local explanations to global understanding with explainable {AI} for trees}, | |
| 228 | + journal = {Nature Machine Intelligence}, | |
| 229 | + year = {2020}, | |
| 230 | + volume = {2}, | |
| 231 | + number = {1}, | |
| 232 | + pages = {56--67}, | |
| 233 | +} | |
| 234 | + | |
| 235 | +@article{mackinnon1985some, | |
| 236 | + author = {MacKinnon, James G. and White, Halbert}, | |
| 237 | + title = {Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties}, | |
| 238 | + journal = {Journal of Econometrics}, | |
| 239 | + year = {1985}, | |
| 240 | + volume = {29}, | |
| 241 | + number = {3}, | |
| 242 | + pages = {305--325}, | |
| 243 | +} | |
| 244 | + | |
| 245 | +@article{mak2010quantile, | |
| 246 | + author = {Mak, Stephen and Choy, Lennon and Ho, Winky}, | |
| 247 | + title = {Quantile regression estimates of {H}ong {K}ong real estate prices}, | |
| 248 | + journal = {Urban Studies}, | |
| 249 | + year = {2010}, | |
| 250 | + volume = {47}, | |
| 251 | + number = {11}, | |
| 252 | + pages = {2461--2472}, | |
| 253 | +} | |
| 254 | + | |
| 255 | +@incollection{malpezzi2003hedonic, | |
| 256 | + author = {Malpezzi, Stephen}, | |
| 257 | + title = {Hedonic pricing models: {A} selective and applied review}, | |
| 258 | + booktitle = {Housing Economics and Public Policy}, | |
| 259 | + editor = {O'Sullivan, Tony and Gibb, Kenneth}, | |
| 260 | + publisher = {Blackwell}, | |
| 261 | + year = {2003}, | |
| 262 | + pages = {67--89}, | |
| 263 | +} | |
| 264 | + | |
| 265 | +@article{meyer2019importance, | |
| 266 | + author = {Meyer, Hanna and Pebesma, Edzer}, | |
| 267 | + title = {Predicting into unknown space? {E}stimating the area of applicability of spatial prediction models}, | |
| 268 | + journal = {Methods in Ecology and Evolution}, | |
| 269 | + year = {2019}, | |
| 270 | + volume = {12}, | |
| 271 | + number = {9}, | |
| 272 | + pages = {1620--1633}, | |
| 273 | +} | |
| 274 | + | |
| 275 | +@article{mullainathan2017machine, | |
| 276 | + author = {Mullainathan, Sendhil and Spiess, Jann}, | |
| 277 | + title = {Machine learning: {A}n applied econometric approach}, | |
| 278 | + journal = {Journal of Economic Perspectives}, | |
| 279 | + year = {2017}, | |
| 280 | + volume = {31}, | |
| 281 | + number = {2}, | |
| 282 | + pages = {87--106}, | |
| 283 | +} | |
| 284 | + | |
| 285 | +@article{oates1969effects, | |
| 286 | + author = {Oates, Wallace E.}, | |
| 287 | + title = {The effects of property taxes and local public spending on property values}, | |
| 288 | + journal = {Journal of Political Economy}, | |
| 289 | + year = {1969}, | |
| 290 | + volume = {77}, | |
| 291 | + number = {6}, | |
| 292 | + pages = {957--971}, | |
| 293 | +} | |
| 294 | + | |
| 295 | +@incollection{roberts2013endogeneity, | |
| 296 | + author = {Roberts, Michael R. and Whited, Toni M.}, | |
| 297 | + title = {Endogeneity in empirical corporate finance}, | |
| 298 | + booktitle = {Handbook of the Economics of Finance}, | |
| 299 | + editor = {Constantinides, George M. and Harris, Milton and Stulz, Ren{\'e} M.}, | |
| 300 | + publisher = {Elsevier}, | |
| 301 | + year = {2013}, | |
| 302 | + volume = {2A}, | |
| 303 | + pages = {493--572}, | |
| 304 | +} | |
| 305 | + | |
| 306 | +@article{roberts2017cross, | |
| 307 | + author = {Roberts, David R. and Bahn, Volker and Ciuti, Simone and Boyce, Mark S. and Elith, Jane and Guillera-Arroita, Gurutzeta and Hauenstein, Severin and Lahoz-Monfort, Jos{\'e} J. and Schr{\"o}der, Boris and Thuiller, Wilfried and Warton, David I. and Wintle, Brendan A. and Hartig, Florian and Dormann, Carsten F.}, | |
| 308 | + title = {Cross-validation strategies for data with temporal and spatial dependence}, | |
| 309 | + journal = {Ecography}, | |
| 310 | + year = {2017}, | |
| 311 | + volume = {40}, | |
| 312 | + number = {8}, | |
| 313 | + pages = {913--929}, | |
| 314 | +} | |
| 315 | + | |
| 316 | +@article{rosen1974hedonic, | |
| 317 | + author = {Rosen, Sherwin}, | |
| 318 | + title = {Hedonic prices and implicit markets: {P}roduct differentiation in pure competition}, | |
| 319 | + journal = {Journal of Political Economy}, | |
| 320 | + year = {1974}, | |
| 321 | + volume = {82}, | |
| 322 | + number = {1}, | |
| 323 | + pages = {34--55}, | |
| 324 | +} | |
| 325 | + | |
| 326 | +@incollection{shapley1953value, | |
| 327 | + author = {Shapley, Lloyd S.}, | |
| 328 | + title = {A value for $n$-person games}, | |
| 329 | + booktitle = {Contributions to the Theory of Games II}, | |
| 330 | + editor = {Kuhn, Harold W. and Tucker, Albert W.}, | |
| 331 | + publisher = {Princeton University Press}, | |
| 332 | + year = {1953}, | |
| 333 | + pages = {307--317}, | |
| 334 | +} | |
| 335 | + | |
| 336 | +@article{sirmans2005composition, | |
| 337 | + author = {Sirmans, Stacy and Macpherson, David and Zietz, Emily}, | |
| 338 | + title = {The composition of hedonic pricing models}, | |
| 339 | + journal = {Journal of Real Estate Literature}, | |
| 340 | + year = {2005}, | |
| 341 | + volume = {13}, | |
| 342 | + number = {1}, | |
| 343 | + pages = {3--43}, | |
| 344 | +} | |
| 345 | + | |
| 346 | +@article{white1980heteroskedasticity, | |
| 347 | + author = {White, Halbert}, | |
| 348 | + title = {A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity}, | |
| 349 | + journal = {Econometrica}, | |
| 350 | + year = {1980}, | |
| 351 | + volume = {48}, | |
| 352 | + number = {4}, | |
| 353 | + pages = {817--838}, | |
| 354 | +} | |
| 355 | + | |
| 356 | +@article{zietz2008determinants, | |
| 357 | + author = {Zietz, Joachim and Zietz, Emily N. and Sirmans, G. Stacy}, | |
| 358 | + title = {Determinants of house prices: {A} quantile regression approach}, | |
| 359 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 360 | + year = {2008}, | |
| 361 | + volume = {37}, | |
| 362 | + number = {4}, | |
| 363 | + pages = {317--333}, | |
| 364 | +} | |
added
paper/sections/conclusion.tex
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 4 | +% 9. CONCLUSION | |
| 5 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 6 | +\section{Conclusion} | |
| 7 | +\label{sec:conclusion} | |
| 8 | + | |
| 9 | +This paper has compared three approaches to hedonic housing price analysis---OLS, quantile regression, and gradient-boosted machine learning with SHAP interpretation---using a large Zillow listing sample of 788,842 U.S.\ properties. Each framework answers a different question, and the robustness extensions in this version document the boundaries of each framework's conclusions. | |
| 10 | + | |
| 11 | +OLS provides interpretable conditional mean associations: a price-to-area elasticity of 0.63, a 22.6\% bathroom listing-price gradient, a 33.4\% foreclosure discount, and regional listing-price differentials of 49\%--63\% between coastal and interior markets. The progression from region FE ($R^2 = 0.634$) to ZIP3 FE ($R^2 = 0.725$) demonstrates that fine-grained geographic controls capture substantial variation, but the imputation sensitivity analysis reveals that the lot-size gradient is attenuated threefold by state-median imputation. | |
| 12 | + | |
| 13 | +Quantile regression reveals that mean effects mask substantial distributional heterogeneity, now formally confirmed by inter-quantile Wald tests that reject coefficient equality for 11 of 13 variables. The garage listing-price gradient is ten times larger at the 10th percentile than the 90th ($z = 28.92$); the pool gradient is insignificant below the median but reaches 9.9\% at $\tau = 0.90$. Subsample stability analysis identifies age as the one coefficient that is genuinely unstable across geographic subsamples. | |
| 14 | + | |
| 15 | +XGBoost captures non-linearities and interactions that improve prediction from $R^2 = 0.630$ (OLS) to $R^2 = 0.833$ under random validation. However, this figure substantially overstates generalization ability. The ablation analysis reveals that neighborhood features drive the largest predictive gain ($+17.6$ pp) but that the full model with region dummies actively harms geographic holdout ($-7.6$ pp). The most striking result is that removing all geographic features \emph{improves} geographic holdout $R^2$ from 0.425 to 0.519, while adding latitude and longitude does the opposite ($+3.7$ pp random, $-5.5$ pp geographic). Geographic features help the model memorize location-specific prices but do not improve its ability to generalize attribute-price relationships to new markets. | |
| 16 | + | |
| 17 | +SHAP values identify the features that contribute most to predictions---living area, bathrooms, school quality, lot size, region---and this ranking is stable across three tree-based models (Spearman $\rho = 0.89$--$0.99$). But these remain predictive decompositions, not structural listing-price gradients. Moran's $I = 0.27$ on OLS residuals---stable across three subsamples (std $= 0.008$)---confirms that regional controls leave substantial spatial dependence unaddressed. | |
| 18 | + | |
| 19 | +The central contribution is methodological: we demonstrate that model evaluation depends critically on validation design. Random splits overstate generalization, geographic features can harm geographic holdout, SHAP values are stable across models but remain non-structural, and imputation can attenuate key coefficients. Future research should integrate spatial econometric methods, use transaction prices, employ geographic holdout validation as standard practice, incorporate climate risk data, and benchmark against generalized additive models to decompose the ML predictive gain into non-linearity versus spatial partitioning components. | |
| 20 | + | |
| 21 | +The central lesson is not that machine learning replaces hedonic econometrics, but that prediction, distributional heterogeneity, and economic interpretation answer different questions and should be combined carefully in modern housing valuation research. | |
added
paper/sections/data.tex
+192 −0
@@ -0,0 +1,192 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 4 | +% 3. DATA | |
| 5 | +% ═══════════════════════════════════════════════════════════════════════ | |
| 6 | +\section{Data and Variable Construction} | |
| 7 | +\label{sec:data} | |
| 8 | + | |
| 9 | +\subsection{Data Source} | |
| 10 | + | |
| 11 | +The dataset is sourced from Zillow, the largest online real estate marketplace in the United States. The raw data contain 839,313 residential properties with 116 variables, representing active for-sale listings as of 2025--2026. The data include listing prices, structural characteristics, geocoordinates, neighborhood quality scores, and tax/transaction histories. | |
| 12 | + | |
| 13 | +\textbf{Important limitations of the data source.} The sample comprises active Zillow listings, not the universe of U.S.\ residential properties. Several sources of non-representativeness should be noted: (i) Zillow coverage varies by region, with some MLS-linked markets better represented than others; (ii) active listings capture supply at a point in time, not the full housing stock; (iii) the sample excludes off-market properties, FSBOs not listed on Zillow, and recently transacted properties that are no longer listed; (iv) Southern states are overrepresented (56.6\% of the sample vs.\ approximately 38\% of U.S.\ housing units), likely reflecting higher listing volumes in fast-growing Sun Belt markets. We do not claim national representativeness and caution against interpreting our estimates as population parameters for the entire U.S.\ housing market. | |
| 14 | + | |
| 15 | +\subsection{Listing Prices versus Transaction Prices} | |
| 16 | + | |
| 17 | +A critical limitation is that our dependent variable is the \textit{listing} (asking) price, not the realized transaction price. Listing prices reflect seller expectations and strategic pricing behavior, not necessarily market-clearing values. List-to-sale price ratios vary by market condition, property type, and price tier: luxury properties are more frequently overpriced, distressed properties may be strategically underpriced, and regional norms for overbidding versus negotiation differ substantially \citep{knight2002listing}. Throughout this paper, we interpret the estimated coefficients as \textbf{listing-price capitalization gradients}---conditional associations between attributes and asking prices---rather than transaction-price implicit prices. If listing premiums correlate systematically with property attributes (e.g., if waterfront properties are more frequently overpriced), the estimated gradients will reflect this pricing behavior in addition to underlying valuation differences. | |
| 18 | + | |
| 19 | +\subsection{Sample Construction} | |
| 20 | + | |
| 21 | +Table~\ref{tab:attrition} documents the sample construction. | |
| 22 | + | |
| 23 | +\begin{table}[H] | |
| 24 | +\centering | |
| 25 | +\caption{Sample Construction} | |
| 26 | +\label{tab:attrition} | |
| 27 | +\begin{threeparttable} | |
| 28 | +\begin{tabular}{lrrr} | |
| 29 | +\toprule | |
| 30 | +Step & $N$ remaining & Removed & Criterion \\ | |
| 31 | +\midrule | |
| 32 | +Raw Zillow records & 839,313 & --- & --- \\ | |
| 33 | +Valid price & 817,473 & 21,840 & $\$10{,}000 < P < \$10{,}000{,}000$ \\ | |
| 34 | +Valid living area & 797,380 & 20,093 & $200 < \text{sqft} < 20{,}000$ \\ | |
| 35 | +Valid bedrooms/bathrooms & 790,799 & 6,581 & $1 \leq \text{bed} \leq 10$, $1 \leq \text{bath} \leq 10$ \\ | |
| 36 | +Valid geocoordinates & 789,199 & 1,600 & Non-missing lat/lon \\ | |
| 37 | +U.S.\ states + DC only & 788,842 & 357 & Exclude PR, VI \\ | |
| 38 | +\midrule | |
| 39 | +\textbf{Final analytical sample} & \textbf{788,842} & \textbf{50,471} & \textbf{(6.0\% removed)} \\ | |
| 40 | +\bottomrule | |
| 41 | +\end{tabular} | |
| 42 | +\begin{tablenotes} | |
| 43 | +\small | |
| 44 | +\item \textit{Notes:} Filters applied sequentially. The 6.0\% removal rate suggests the raw data are reasonably clean, though the retained sample may still contain measurement error in secondary variables. | |
| 45 | +\end{tablenotes} | |
| 46 | +\end{threeparttable} | |
| 47 | +\end{table} | |
| 48 | + | |
| 49 | +\subsection{Variable Description} | |
| 50 | + | |
| 51 | +Our specification includes 62 regressors organized into six categories. | |
| 52 | + | |
| 53 | +\subsubsection{Structural Attributes (8 Variables)} | |
| 54 | + | |
| 55 | +The natural logarithm of living area ($\ln\text{sqft}$), number of bedrooms, number of bathrooms, property age ($2026 - \text{year\_built}$) and its square, number of stories, the bathroom-to-bedroom ratio, and square feet per bedroom. Living area enters in log form to accommodate the well-documented concavity of the size-price relationship. | |
| 56 | + | |
| 57 | +\subsubsection{Lot Characteristics (1 Variable)} | |
| 58 | + | |
| 59 | +The natural logarithm of lot size ($\ln\text{lot}$). Missing lot sizes (16.3\% of raw data) are imputed using state-level medians. | |
| 60 | + | |
| 61 | +\subsubsection{Amenity and Quality Indicators (11 Variables)} | |
| 62 | + | |
| 63 | +Binary indicators for swimming pool (39.8\% prevalence), spa (6.1\%), basement (24.3\%), fireplace (39.6\%), garage (67.2\%), waterfront location (8.9\%), central air conditioning (68.1\%), forced air heating (25.8\%), and hardwood flooring (26.7\%). The number of parking spaces and a composite luxury score (0--7) supplement these indicators. | |
| 64 | + | |
| 65 | +\subsubsection{Neighborhood and Location Variables (7 Variables)} | |
| 66 | + | |
| 67 | +Walk Score (0--100), Bike Score, Transit Score, average GreatSchools rating (1--10), school count, distance to nearest school, and local property tax rate. | |
| 68 | + | |
| 69 | +\textbf{Data quality note.} Bike Score has a maximum value of 248 in our data, exceeding the expected 0--100 range. Only 38 observations (0.005\%) exceed 100, and these are retained without truncation. Results are robust to capping Bike Score at 100. | |
| 70 | + | |
| 71 | +\subsubsection{Market Status Variables (5 Variables)} | |
| 72 | + | |
| 73 | +Condominium indicator (14.6\%), HOA membership (37.1\%), log annualized HOA fees ($\ln(1 + \text{HOA}_\text{annual})$), new construction (5.9\%), and foreclosure status (0.5\%). | |
| 74 | + | |
| 75 | +\subsubsection{Interaction Terms (6 Variables)} | |
| 76 | + | |
| 77 | +\begin{itemize}[nosep] | |
| 78 | + \item $\ln(\text{sqft}) \times \text{age}$: depreciation-size interaction; | |
| 79 | + \item $\text{pool} \times \text{South}$: climate-dependent pool gradient; | |
| 80 | + \item $\text{waterfront} \times \ln(\text{sqft})$: size-dependent waterfront gradient; | |
| 81 | + \item $\text{basement} \times \text{North}$: region-dependent basement gradient; | |
| 82 | + \item $\text{condo} \times \text{walk\_score}$: walkability gradient for condominiums; | |
| 83 | + \item $\text{age} \times \text{luxury\_score}$: luxury mitigation of depreciation. | |
| 84 | +\end{itemize} | |
| 85 | + | |
| 86 | +\subsubsection{Categorical Controls (24 Dummy Variables)} | |
| 87 | + | |
| 88 | +Simplified indicators for roof type (6 dummies), construction material (8 dummies), foundation type (7 dummies), and Census region (3 dummies; Midwest reference). | |
| 89 | + | |
| 90 | +\subsection{Missing Data and Imputation} | |
| 91 | + | |
| 92 | +Table~\ref{tab:missing} reports missingness rates for key variables. | |
| 93 | + | |
| 94 | +\begin{table}[H] | |
| 95 | +\centering | |
| 96 | +\caption{Missing Data Rates and Imputation Methods} | |
| 97 | +\label{tab:missing} | |
| 98 | +\begin{threeparttable} | |
| 99 | +\small | |
| 100 | +\begin{tabular}{lrrl} | |
| 101 | +\toprule | |
| 102 | +Variable & Missing (\%) & Method & Notes \\ | |
| 103 | +\midrule | |
| 104 | +Year built & 19.4 & State median & Affects age, age$^2$, interactions \\ | |
| 105 | +Lot size & 16.3 & State median & Large variation \\ | |
| 106 | +Walk Score & 3.1 & State median & \\ | |
| 107 | +Bike Score & 6.1 & State median & Max = 248 (38 obs.) \\ | |
| 108 | +Transit Score & 70.1 & Set to 0 & Suburban/rural properties \\ | |
| 109 | +Garage & 35.1 & Set to False & Conservative \\ | |
| 110 | +Climate risk factors & 100.0 & Excluded & Not available \\ | |
| 111 | +\bottomrule | |
| 112 | +\end{tabular} | |
| 113 | +\begin{tablenotes} | |
| 114 | +\small | |
| 115 | +\item \textit{Notes:} State-level median imputation is used to preserve geographic variation. The high Transit Score missingness (70.1\%) likely reflects properties in areas without public transit infrastructure, for which zero is a reasonable value. Climate risk factors (flood, fire, heat, wind, air) are entirely missing and excluded from the analysis. | |
| 116 | +\end{tablenotes} | |
| 117 | +\end{threeparttable} | |
| 118 | +\end{table} | |
| 119 | + | |
| 120 | +\subsection{Data Quality Audit} | |
| 121 | +\label{sec:data_audit} | |
| 122 | + | |
| 123 | +We conduct three data quality checks to assess the sensitivity of key results to imputation and outliers. | |
| 124 | + | |
| 125 | +\textbf{Imputation prevalence.} Two variables---year built (19.4\% missing) and lot size (16.3\% missing)---account for the bulk of imputed values. To assess the consequences, we re-estimate the baseline OLS model on subsamples that exclude imputed observations (Section~\ref{sec:imputation_sensitivity}). The lot-size coefficient is the most sensitive: $\ln(\text{lot})$ triples from 0.018 to 0.059 when imputed lot sizes are dropped, suggesting that state-median imputation attenuates the lot-size gradient substantially. | |
| 126 | + | |
| 127 | +\textbf{Outlier prevalence.} We winsorize lot size at its 99.5th percentile and cap Bike Score at 100, then re-estimate OLS (Section~\ref{sec:winsorization}). The overall $R^2$ increases modestly from 0.634 to 0.640, and most coefficients are stable, with the exception of $\ln(\text{lot})$, which again increases substantially. | |
| 128 | + | |
| 129 | +\textbf{Geographic coverage.} The sample spans 886 unique three-digit ZIP code prefixes (ZIP3 codes) across 51 jurisdictions (50 states plus DC). Southern states account for 56.6\% of observations. We document the marginal explanatory power of progressively finer geographic controls in Section~\ref{sec:ols_fe}. | |
| 130 | + | |
| 131 | +\subsection{Descriptive Statistics} | |
| 132 | + | |
| 133 | +Table~\ref{tab:descriptive} presents summary statistics. | |
| 134 | + | |
| 135 | +\begin{table}[H] | |
| 136 | +\centering | |
| 137 | +\caption{Descriptive Statistics ($N = 788{,}842$)} | |
| 138 | +\label{tab:descriptive} | |
| 139 | +\begin{threeparttable} | |
| 140 | +\small | |
| 141 | +\begin{tabular}{lrrrrrr} | |
| 142 | +\toprule | |
| 143 | +Variable & Mean & Std.\ Dev. & Min & Median & Max \\ | |
| 144 | +\midrule | |
| 145 | +\multicolumn{6}{l}{\textit{Continuous Variables}} \\ | |
| 146 | +Listing Price (\$) & 621,737 & 817,965 & 10,300 & 399,900 & 9,999,999 \\ | |
| 147 | +Living Area (sqft) & 2,093 & 1,181 & 208 & 1,820 & 19,600 \\ | |
| 148 | +Bedrooms & 3.25 & 1.09 & 1 & 3 & 10 \\ | |
| 149 | +Bathrooms & 2.52 & 1.16 & 1 & 2 & 10 \\ | |
| 150 | +Property Age (years) & 42.2 & 31.5 & 0 & 36 & 200 \\ | |
| 151 | +Walk Score (0--100) & 25.6 & 26.5 & 0 & 16 & 100 \\ | |
| 152 | +Bike Score & 34.9 & 18.1 & 1 & 31 & 248 \\ | |
| 153 | +Transit Score (0--100) & 8.9 & 18.3 & 0 & 0 & 100 \\ | |
| 154 | +Avg.\ School Rating (1--10) & 6.02 & 0.90 & 1 & 6 & 10 \\ | |
| 155 | +Property Tax Rate (\%) & 1.10 & 0.53 & 0 & 1.0 & 4.0 \\ | |
| 156 | +Luxury Score (0--7) & 1.36 & 1.29 & 0 & 1 & 7 \\ | |
| 157 | +\midrule | |
| 158 | +\multicolumn{6}{l}{\textit{Binary Variables (prevalence)}} \\ | |
| 159 | +Swimming Pool & 39.8\% & & & & \\ | |
| 160 | +Garage & 67.2\% & & & & \\ | |
| 161 | +Fireplace & 39.6\% & & & & \\ | |
| 162 | +Basement & 24.3\% & & & & \\ | |
| 163 | +Waterfront & 8.9\% & & & & \\ | |
| 164 | +HOA & 37.1\% & & & & \\ | |
| 165 | +Central Air & 68.1\% & & & & \\ | |
| 166 | +Hardwood Floors & 26.7\% & & & & \\ | |
| 167 | +Condominium & 14.6\% & & & & \\ | |
| 168 | +New Construction & 5.9\% & & & & \\ | |
| 169 | +Foreclosure & 0.5\% & & & & \\ | |
| 170 | +\midrule | |
| 171 | +\multicolumn{6}{l}{\textit{Regional Distribution}} \\ | |
| 172 | +South & 56.6\% & & & & \\ | |
| 173 | +West & 18.9\% & & & & \\ | |
| 174 | +Midwest & 16.1\% & & & & \\ | |
| 175 | +Northeast & 8.4\% & & & & \\ | |
| 176 | +\bottomrule | |
| 177 | +\end{tabular} | |
| 178 | +\begin{tablenotes} | |
| 179 | +\small | |
| 180 | +\item \textit{Notes:} Listing prices, not transaction prices. Property age computed as $2026 - \text{year\_built}$, after imputation. Imputed values included in summary statistics. | |
| 181 | +\end{tablenotes} | |
| 182 | +\end{threeparttable} | |
| 183 | +\end{table} | |
| 184 | + | |
| 185 | +The median listing price is \$399,900, with substantial right-skewness (mean \$621,737). The log transformation substantially improves distributional symmetry (Figure~\ref{fig:price_dist}), supporting the semi-log specification. | |
| 186 | + | |
| 187 | +\begin{figure}[H] | |
| 188 | + \centering | |
| 189 | + \includegraphics[width=\textwidth]{figures/fig1_price_distribution.png} | |
| 190 | + \caption{Distribution of Listing Prices: (a) Levels; (b) Natural Logarithm. The log transformation reduces right skewness and approximates normality.} | |
| 191 | + \label{fig:price_dist} | |
| 192 | +\end{figure} | |
added
paper/sections/discussion.tex
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/introduction.tex
+33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/limitations.tex
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/literature.tex
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/methodology.tex
+130 −0
Large diff (130 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/results.tex
+590 −0
Large diff (590 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/robustness.tex
+164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/titlepage.tex
+80 −0
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
added
requirements.txt
+15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/imputation_sensitivity.csv
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/iqr_wald_tests.csv
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/morans_i_robustness.csv
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/ols_standardized.csv
+64 −0
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/ols_unstandardized.csv
+64 −0
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/qr_coefficients.csv
+64 −0
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/qr_pseudo_r2.csv
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/qr_stability.csv
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/shap_cross_model_stability.csv
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/shap_importance_xgb.csv
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/winsorization_sensitivity.csv
+13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/xgb_geography_and_ablation.csv
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/zip3_fixed_effects.csv
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/01_spatial_robustness.py
+167 −0
Large diff (167 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/02_qr_imputation.py
+210 −0
Large diff (210 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/03_make_figures.py
+426 −0
Large diff (426 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/04_export_tables.py
+135 −0
Large diff (135 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp3/__init__.py
+3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp3/config.py
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp3/data.py
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp3/models.py
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp3/plotting.py
+112 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff