Ultra-detailed README with badges, full pipeline/model/robustness documentation · 8a97ae0 · wp5_uqo

1

2

3

−# Airbnb, Residential Rents, and Housing Market Pressure (UQO WP5)

3

+<div align="center">

4

5

−A hedonic and spatial econometric analysis of the relationship between Airbnb

6

−activity and residential rental prices in Quebec, Canada.

7

−UQO Working Paper No. 5 — Simon-Pierre Boucher, Université du Québec en Outaouais.

5

+# 🏠 Airbnb, Residential Rents, and Housing Market Pressure

8

6

9

−## Repository layout

7

+### A Hedonic and Spatial Econometric Analysis — UQO Working Paper No. 5

8

+

9

+[![Python](https://img.shields.io/badge/Python-3.14-3776AB?logo=python&logoColor=white)](https://www.python.org/)

10

+[![pandas](https://img.shields.io/badge/pandas-3.0.2-150458?logo=pandas&logoColor=white)](https://pandas.pydata.org/)

11

+[![statsmodels](https://img.shields.io/badge/statsmodels-0.14.6-4051B5)](https://www.statsmodels.org/)

12

+[![scikit--learn](https://img.shields.io/badge/scikit--learn-1.6.1-F7931E?logo=scikitlearn&logoColor=white)](https://scikit-learn.org/)

13

+[![PySAL/spreg](https://img.shields.io/badge/PySAL%2Fspreg-1.9.0-8A2BE2)](https://pysal.org/)

14

+[![SHAP](https://img.shields.io/badge/SHAP-0.48.0-FF4081)](https://shap.readthedocs.io/)

15

+[![LaTeX](https://img.shields.io/badge/LaTeX-latexmk-008080?logo=latex&logoColor=white)](https://www.latex-project.org/)

16

+

17

+[![Status](https://img.shields.io/badge/status-working%20paper-blue)]()

18

+[![Reproducible](https://img.shields.io/badge/pipeline-fully%20reproducible-brightgreen)]()

19

+[![Verified](https://img.shields.io/badge/tables-14%2F16%20byte--identical-brightgreen)]()

20

+[![Paper](https://img.shields.io/badge/PDF-43%20pages%2C%200%20unresolved%20refs-success)]()

21

+[![Figures](https://img.shields.io/badge/figures-16%20regenerated-informational)]()

22

+[![Models](https://img.shields.io/badge/models-OLS%20%7C%20SAR%2FSEM%20%7C%20Quantile%20%7C%20ML-orange)]()

23

+

24

+**Simon-Pierre Boucher** · Département des sciences administratives, Université du Québec en Outaouais (UQO)

25

+📧 [contact@spboucher.ai](mailto:contact@spboucher.ai) · 📄 [`paper/main.pdf`](paper/main.pdf)

26

+

27

+</div>

28

+

29

+---

30

+

31

+## 📋 Table of Contents

32

+

33

+1. [Overview](#-overview)

34

+2. [Key Findings](#-key-findings)

35

+3. [Repository Structure](#-repository-structure)

36

+4. [Data](#-data)

37

+5. [Analysis Pipeline](#-analysis-pipeline)

38

+6. [Econometric Models](#-econometric-models)

39

+7. [Robustness Programme](#-robustness-programme)

40

+8. [Reproducing Everything](#-reproducing-everything)

41

+9. [Environment & Dependencies](#-environment--dependencies)

42

+10. [Outputs Inventory](#-outputs-inventory)

43

+11. [Verification & Provenance](#-verification--provenance)

44

+12. [Citation](#-citation)

45

+13. [Contact](#-contact)

46

+

47

+---

48

+

49

+## 🔍 Overview

50

+

51

+This repository contains the **complete, reproducible research compendium** for UQO

52

+Working Paper No. 5: data pipeline, econometric analysis, machine-learning

53

+robustness suite, all figures/tables, and the LaTeX source of the paper.

54

+

55

+The paper studies the relationship between **Airbnb short-term rental activity**

56

+and **residential rents** in Quebec, Canada, using cross-sectional microdata:

57

+~5,000 Airbnb listings and 8,356 Realtor.ca rental listings, linked by an exact

58

+**Haversine spatial buffer merge** (250 m / 500 m / 1 km / 2 km). The empirical

59

+strategy combines hedonic OLS with city fixed effects, spatial econometrics

60

+(SAR / SEM via GMM), quantile regressions, and machine-learning benchmarks

61

+(LASSO, elastic net, random forest, gradient boosting, SHAP).

62

+

63

+> ⚠️ **Interpretation note.** The design is cross-sectional. All estimates are

64

+> *conditional correlations*, not causal effects — the paper is explicit about

65

+> reverse causality, omitted amenities, and spatial sorting.

66

+

67

+---

68

+

69

+## 📊 Key Findings

70

+

71

+| # | Finding | Evidence |

72

+|---|---------|----------|

73

+| 1 | Each additional Airbnb listing within **500 m** is associated with **≈ +0.4 %** monthly rent (0.3–0.5 % across specifications), conditional on dwelling characteristics, building type, and city fixed effects | Table 3, Model 1c: β = 0.0039, HC1 SE = 0.0002 |

74

+| 2 | The per-listing association **decays with distance**: 0.93 % (250 m) → 0.38 % (500 m) → 0.16 % (1 km) → 0.06 % (2 km) | Table 8, Figure 4 |

75

+| 3 | The association is **strongest at the top of the rent distribution**: β rises from ≈ 0.0034–0.0037 (τ ≤ 0.50) to **0.0047 at τ = 0.90** | Table 7, Figure 5 |

76

+| 4 | **Spatial dependence is substantial** (SAR ρ̂ = 0.137***, SEM λ̂ = 0.539) and the Airbnb coefficient survives with mild attenuation (0.0034 SAR) | Table 6 |

77

+| 5 | Airbnb **nightly prices** are driven by listing characteristics; city-level mean rent carries **no significant premium** | Table 4 |

78

+| 6 | **ML benchmarks**: random forest test R² = 0.71, gradient boosting 0.69 vs OLS 0.51 — nonlinearities exist, but the linear hedonic remains adequate for inference | Table 9, SHAP Figures 6–7 |

79

+| 7 | The estimate is stable under **city-clustered SEs**, an **interior-size control**, a **log(1+count)** functional form, **ring-exposure** decomposition, and **leave-one-city-out** exclusions (incl. Montreal) | Table 12, Figure 9 |

80

+

81

+Economic magnitude: at the median rent (≈ $1,950/month), one extra nearby

82

+listing ≈ **$6–10/month**; a neighbourhood with 20 nearby listings ≈ 6–10 %

83

+higher rent, all else equal.

84

+

85

+---

86

+

87

+## 📁 Repository Structure

10

88

11

89

```

12

90

wp5_uqo/

13

−├── README.md # this file

14

−├── AUDIT.md # audit of the original project (pre-restructuring)

15

−├── CHANGES.md # everything that was moved, refactored, or rewritten

16

−├── requirements.txt # pinned Python dependencies

91

+├── README.md ← you are here

92

+├── AUDIT.md ← forensic audit of the original project

93

+├── CHANGES.md ← full restructuring & rewrite log

94

+├── requirements.txt ← pinned dependencies (exact reproduction)

95

+├── .gitignore

96

+│

17

97

├── data/

18

−│ ├── raw/ # airbnb.csv + rent.json (NOT distributed — see data/raw/README.md)

19

−│ └── processed/ # cleaned & merged parquet files (committed)

20

−├── src/ # shared modules (paths/config, geo, LaTeX tables, plot styles)

21

−├── scripts/ # numbered pipeline entry points (01–10)

22

−├── figures/ # all 15 paper figures (regenerated by the pipeline)

98

+│ ├── raw/ ← airbnb.csv + rent.json (NOT distributed)

99

+│ │ └── README.md ← what the raw files are & where they go

100

+│ └── processed/ ← committed, verified parquet datasets

101

+│ ├── airbnb_clean.parquet (3,456 × 21)

102

+│ ├── rent_clean.parquet (8,303 × 12)

103

+│ ├── merged_spatial.parquet (8,303 × 36)

104

+│ ├── merged_neighborhood.parquet (8,303 × 19)

105

+│ └── merged_analysis.parquet (8,303 × 43) ← main analysis file

106

+│

107

+├── src/ ← shared modules (imported by all scripts)

108

+│ ├── config.py ← every path & constant; output dirs auto-created

109

+│ ├── geo.py ← vectorised Haversine distance matrix

110

+│ ├── latex_tables.py ← significance stars + stargazer-style builder

111

+│ └── plotting.py ← the two publication matplotlib styles

112

+│

113

+├── scripts/ ← numbered pipeline entry points

114

+│ ├── 01_inspect_raw_data.py ├── 07_spatial_models.py

115

+│ ├── 02_clean_airbnb.py ├── 08_quantile_models.py

116

+│ ├── 03_clean_rent.py ├── 09_ml_robustness.py

117

+│ ├── 04_merge_data.py ├── 10_robustness_tables_figures.py

118

+│ ├── 05_descriptive_analysis.py └── 11_extended_robustness.py

119

+│ └── 06_hedonic_models.py

120

+│

121

+├── figures/ ← all 16 PDF figures (pipeline output)

23

122

├── results/

24

−│ ├── tables/ # LaTeX table fragments + CSV outputs

25

−│ └── logs/ # data-inspection log

26

−└── paper/ # LaTeX source (main.tex + sections/ + appendix/) → main.pdf

123

+│ ├── tables/ ← 13 LaTeX fragments + 4 CSVs

124

+│ └── logs/ ← data-inspection log

125

+│

126

+└── paper/ ← LaTeX source → main.pdf (43 pages)

127

+ ├── main.tex ← preamble + metadata + assembly

128

+ ├── Makefile / .latexmkrc ← `make` or `latexmk` builds the PDF

129

+ ├── references.bib ← 24 entries, all cited

130

+ ├── sections/ ← titlepage + 8 numbered sections

131

+ └── appendix/ ← data / methods / robustness appendices

27

132

```

28

133

29

−## Reproducing everything

134

+---

135

+

136

+## 💾 Data

137

+

138

+### Sources

139

+

140

141

+|---|---|---|---|---|

142

+| Airbnb listings | Airbnb (scraped, Quebec) | ~5,000 | **3,456** | nightly price, lat/lon, property type, rating, reviews, superhost |

143

+| Rental listings | Realtor.ca (scraped, Quebec) | 8,356 | **8,303** | monthly rent, lat/lon, bedrooms, bathrooms, building type, size |

144

+

145

+> 🔒 The **raw files are not distributed** (`data/raw/README.md`). The committed

146

+> parquets in `data/processed/` make steps 04–10 and the paper fully

147

+> reproducible without them; steps 01–03 fail with an explanatory message.

148

+

149

+### Cleaning highlights

150

+

151

+- City names standardised across accented/unaccented variants and boroughs

152

+ (Montréal arrondissements → *Montreal*, Québec arrondissements → *Québec*, …).

153

+- Prices/rents winsorised at p1/p99; log transforms (`log_price`, `log_rent`).

154

+- Ratings imputed at the median (unrated ≠ zero-quality); review counts at 0.

155

+- `is_entire_home` built from property types (House, Cabin/Chalet, Condo).

156

+

157

+### Spatial merge (the core construction)

158

+

159

+For every rental listing *i* and buffer radius *r* ∈ {250 m, 500 m, 1 km, 2 km},

160

+six exposure metrics are computed against all Airbnb listings via a **chunked,

161

+fully vectorised Haversine distance matrix** (`src/geo.py`, 500 rentals/chunk):

162

+

163

+`airbnb_count_r` · `airbnb_density_r` (count/πr²) · `mean_airbnb_price_r` ·

164

+`share_entire_home_r` · `mean_rating_r` · `superhost_share_r`

165

+

166

+plus city-level aggregates (`airbnb_count_city`, …) merged by standardised name.

167

+

168

+---

169

+

170

+## ⚙️ Analysis Pipeline

171

+

172

+Each script is standalone (`python3 scripts/NN_*.py`), reads/writes only through

173

+`src/config.py` paths, and prints a full execution log.

174

+

175

176

+|---|---|---|---|---|

177

+| 01 | `01_inspect_raw_data.py` | raw csv/json | `results/logs/data_inspection.txt` | s |

178

+| 02 | `02_clean_airbnb.py` | `airbnb.csv` | `airbnb_clean.parquet` | s |

179

+| 03 | `03_clean_rent.py` | `rent.json` | `rent_clean.parquet` | s |

180

+| 04 | `04_merge_data.py` | 2 clean parquets | 3 merged parquets | ~1 min |

181

+| 05 | `05_descriptive_analysis.py` | parquets | 3 tables + 7 figures | s |

182

+| 06 | `06_hedonic_models.py` | `merged_analysis` | Models 1–3 (3 tables) | s |

183

+| 07 | `07_spatial_models.py` | `merged_analysis` | SAR/SEM + buffer tables, 1 figure | ~1 min |

184

+| 08 | `08_quantile_models.py` | `merged_analysis` | quantile table + figure | ~1 min |

185

+| 09 | `09_ml_robustness.py` | `merged_analysis` | ML table + 3 figures (SHAP) | ~2 min |

186

+| 10 | `10_robustness_tables_figures.py` | `merged_analysis` | 2 tables + 3 figures | s |

187

+| 11 | `11_extended_robustness.py` | `merged_analysis` | extended table + 1 figure | ~1 min |

188

+

189

+<sub>*Apple Silicon, Python 3.14; “s” < 15 seconds.</sub>

190

+

191

+---

192

+

193

+## 📐 Econometric Models

194

+

195

196

+|---|---|---|---|

197

198

199

200

201

+| **5** Quantile | τ ∈ {.10, .25, .50, .75, .90} + fine grid for the coefficient path | IRLS (statsmodels) | 08 |

202

203

+

204

+---

30

205

31

−Requires Python ≥ 3.12 (developed on 3.14.4) and a TeX distribution with `latexmk`.

206

+## 🛡️ Robustness Programme

207

+

208

+- **Buffer radii** 250 m → 2 km (Table 8, Figure 4) — monotone spatial decay.

209

+- **Alternative exposure**: density (≡ count at fixed r); entire-home share (null → discussed).

210

+- **Subsamples**: Montreal / outside-Montreal / apartments / houses (Table 11).

211

+- **Outliers**: rent trimmed p5–p95; Airbnb count trimmed p1–p99 (Table 10).

212

+- **Extended** (Table 12, Figure 9): city-clustered SEs (tighter than HC1),

213

+ interior-size control, log(1+count), 500 m + ring 500 m–1 km decomposition,

214

+ leave-one-city-out over the 8 largest cities.

215

+

216

+---

217

+

218

+## 🚀 Reproducing Everything

219

+

220

+```bash

221

+git clone https://github.com/spboucher-ai/wp5_uqo.git

222

+cd wp5_uqo

223

+python3 -m pip install -r requirements.txt

224

+

225

+# (optional) steps 01–03 need data/raw/airbnb.csv + rent.json

226

+for s in scripts/0{4..9}_*.py scripts/1{0,1}_*.py; do python3 "$s"; done

227

+

228

+cd paper && latexmk # → paper/main.pdf

229

+```

230

+

231

+Or step by step:

32

232

33

233

```bash

34

−pip install -r requirements.txt

35

−

36

−# Steps 01–03 need the raw files in data/raw/ (see data/raw/README.md).

37

−# The committed parquets in data/processed/ make them optional.

38

−python3 scripts/01_inspect_raw_data.py # optional — raw data inspection log

39

−python3 scripts/02_clean_airbnb.py # optional — rebuilds airbnb_clean.parquet

40

−python3 scripts/03_clean_rent.py # optional — rebuilds rent_clean.parquet

41

−

42

−python3 scripts/04_merge_data.py # spatial buffer merge (Haversine 250m–2km)

43

−python3 scripts/05_descriptive_analysis.py # summary tables + descriptive figures

44

−python3 scripts/06_hedonic_models.py # Models 1–3 (hedonic OLS, HC1)

45

−python3 scripts/07_spatial_models.py # Model 4 (SAR/SEM via spreg) + buffer robustness

46

−python3 scripts/08_quantile_models.py # Model 5 (quantile regressions)

47

−python3 scripts/09_ml_robustness.py # Model 6 (LASSO/ENet/RF/GBM + SHAP)

48

−python3 scripts/10_robustness_tables_figures.py # robustness tables + extra figures

49

−python3 scripts/11_extended_robustness.py # clustered SEs, size control, log form,

50

− # ring exposure, leave-one-city-out

51

−

52

−cd paper && latexmk # builds paper/main.pdf

234

+python3 scripts/04_merge_data.py # spatial buffer merge

235

+python3 scripts/05_descriptive_analysis.py # summary stats + descriptive figures

236

+python3 scripts/06_hedonic_models.py # Models 1–3

237

+python3 scripts/07_spatial_models.py # Model 4 (SAR/SEM) + buffer robustness

238

+python3 scripts/08_quantile_models.py # Model 5

239

+python3 scripts/09_ml_robustness.py # Model 6 + SHAP

240

+python3 scripts/10_robustness_tables_figures.py

241

+python3 scripts/11_extended_robustness.py # clustered SEs, ring, leave-one-out

242

+cd paper && latexmk # or: make

243

+```

244

+

245

+Everything is deterministic (fixed seed 42 for ML; no other stochastic steps).

246

+

247

+---

248

+

249

+## 🧰 Environment & Dependencies

250

+

251

+Developed and verified on **macOS (Apple Silicon), Python 3.14.4**, TeX Live 2025.

252

+

253

+| Package | Pinned | Used for |

254

+|---|---|---|

255

+| pandas / pyarrow | 3.0.2 / 24.0.0 | data wrangling, parquet I/O |

256

+| numpy / scipy | 2.4.4 / 1.17.1 | vectorised Haversine, KD-tree |

257

+| statsmodels | 0.14.6 | OLS (HC1/cluster), quantile regression |

258

+| libpysal / spreg | 4.14.1 / 1.9.0 | KNN weights, SAR (GM_Lag), SEM (GM_Error) |

259

+| scikit-learn | 1.6.1 | LASSO, ENet, RF, GBM |

260

+| shap | 0.48.0 | SHAP values & plots |

261

+| matplotlib | 3.10.9 | all figures |

262

+

263

+`pip install -r requirements.txt` reproduces the exact environment.

264

+

265

+---

266

+

267

+## 📤 Outputs Inventory

268

+

269

+<details>

270

+<summary><b>16 figures</b> (click to expand)</summary>

271

+

272

+`map_airbnb` · `map_rent` · `dist_airbnb_price` · `dist_rent` · `airbnb_by_city` ·

273

+`rent_by_city` · `scatter_airbnb_rent` · `rent_airbnb_heatmap` ·

274

+`coefficient_buffer_comparison` · `quantile_coefficients` ·

275

+`ml_predicted_vs_actual` · `feature_importance` · `shap_summary` ·

276

+`coefficient_robustness` · `rent_by_airbnb_bins` · `leave_one_city_out`

277

+</details>

278

+

279

+<details>

280

+<summary><b>13 LaTeX tables + 4 CSVs</b> (click to expand)</summary>

281

+

282

+`summary_stats_airbnb` · `summary_stats_rent` · `correlation_matrix` ·

283

+`hedonic_rent_baseline` · `hedonic_airbnb_pricing` · `city_level_interaction` ·

284

+`spatial_models` · `buffer_robustness` · `quantile_regression` ·

285

+`ml_comparison` · `robustness_outliers` · `robustness_subsamples` ·

286

+`extended_robustness` (+ CSV mirrors of the summary/correlation/ML tables)

287

+

288

+Tables are emitted as **booktabs fragments** — the paper wraps each `\input`

289

+in its own `table` environment, so the pipeline feeds the PDF with no manual edits.

290

+</details>

291

+

292

+---

293

+

294

+## ✅ Verification & Provenance

295

+

296

+This repository was rebuilt on **2026-08-05** from the original project

297

+(`immo-wp5-spb-20260519`, kept untouched as backup) with a strict

298

+*no-result-changes* policy:

299

+

300

+- The 3 merged parquets regenerated from the committed clean data are

301

+ **value-identical** to the originals (`pandas.testing.assert_frame_equal`).

302

+- **14 / 16 tables byte-identical**; the 2 residual differences are float-level

303

+ and fully documented in [`AUDIT.md`](AUDIT.md) §8.

304

+- All figures regenerated, including 5 that were missing from the original

305

+ project (which previously broke the LaTeX build).

306

+- Every text↔table inconsistency found in the original manuscript was corrected

307

+ *in the prose* (never in the results) and is itemised in

308

+ [`CHANGES.md`](CHANGES.md) §4 — start there if you review the paper.

309

+

310

+---

311

+

312

+## 📖 Citation

313

+

314

+```bibtex

315

+@techreport{boucher2026airbnb,

316

+ author = {Boucher, Simon-Pierre},

317

+ title = {Airbnb, Residential Rents, and Housing Market Pressure:

318

+ A Hedonic and Spatial Econometric Analysis},

319

+ institution = {Universit\'e du Qu\'ebec en Outaouais,

320

+ D\'epartement des sciences administratives},

321

+ type = {Working Paper},

322

+ number = {5},

323

+ year = {2026},

324

+ month = {May},

325

+ address = {Gatineau, QC, Canada}

326

+}

53

327

```

54

328

55

−Every script reads and writes paths defined in `src/config.py`; the pipeline can

56

−be run from any working directory.

57

−

58

−## Key findings

59

−

60

−- Each additional Airbnb listing within 500 m is associated with ≈ 0.4% higher

61

− monthly rent (0.3–0.5% across specifications), controlling for dwelling

62

− characteristics, building type, and city fixed effects.

63

−- The association decays with buffer radius (250 m → 2 km) and is strongest at

64

− the upper quantiles of the rent distribution.

65

−- SAR/SEM spatial models confirm substantial spatial autocorrelation in rents;

66

− the Airbnb coefficient survives with mild attenuation.

67

−- The estimate is stable under city-clustered standard errors, an interior-size

68

− control, a log(1+count) functional form, ring-exposure decomposition, and

69

− leave-one-city-out exclusions (including Montreal).

70

−- ML benchmarks (random forest, gradient boosting) confirm the predictive

71

− relevance of Airbnb exposure; the linear specification remains adequate.

72

−- Cross-sectional design — associations, not causal effects.

73

−

74

−## Provenance

75

−

76

−Restructured from `~/Desktop/UQO/UQO_WP/immo-wp5-spb-20260519` (left untouched

77

−as backup) on 2026-08-05. See `AUDIT.md` for the audit of the original project

78

−and `CHANGES.md` for the full list of changes; regenerated tables were verified

79

−against the originals (14/16 byte-identical, 2 residual float-level differences

80

−documented in `AUDIT.md` §8).

329

+---

330

+

331

+## 📬 Contact

332

+

333

+**Simon-Pierre Boucher**

334

+Département des sciences administratives · Université du Québec en Outaouais

335

+283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada

336

+

337

+- 📧 **contact@spboucher.ai**

338

+- 🏛️ simon-pierre.boucher@uqo.ca

339

+

340

+*Issues and pull requests are welcome — please do not open issues asking for the

341

+raw scraped data, which cannot be redistributed.*

342

+

343

+---

344

+

345

+<div align="center">

346

347

+</div>

81

348