# CHANGES — Restructuring of `immo-wp5-spb-20260519` → `wp5_uqo` Date: 2026-08-05. The original project at `~/Desktop/UQO/UQO_WP/immo-wp5-spb-20260519` was **left completely untouched** and serves as the backup (in place of an `_old/` copy inside this repo). Per request, every file created in this repo carries the header `Author: Simon-Pierre Boucher — contact@spboucher.ai` (comment syntax adapted per language: `#` Python/Makefile, `%` LaTeX/BibTeX, `` Markdown; in `references.bib` the email is written `contact (at) spboucher.ai` because a literal `@` inside a `.bib` comment breaks BibTeX). ## 1. Moved / renamed | Original | New | |---|---| | `airbnb.csv`, `rent.json` (project root) | `data/raw/` (files missing — see §5) | | `data_clean/*.parquet` (5 files) | `data/processed/` (bit-identical copies, checksummed) | | `scripts/01_load_inspect_data.py` | `scripts/01_inspect_raw_data.py` | | `scripts/02…09_*.py` | same names under `scripts/` (refactored) | | `scripts/10_generate_tables_figures.py` | `scripts/10_robustness_tables_figures.py` | | `outputs/tables/` | `results/tables/` | | `outputs/logs/` | `results/logs/` | | `outputs/figures/` + `wp5/figures/` (duplicates) | `figures/` (single source, regenerated) | | `wp5/` (canonical LaTeX) | `paper/` | | `paper/` (old LaTeX copy), `wp5/main_web.tex` + fallback hack | **dropped** (superseded; the fallback is unnecessary now that all figures exist) | | `scripts/__pycache__/`, `.DS_Store`, LaTeX build artifacts | dropped / gitignored | New files: `README.md`, `AUDIT.md`, `CHANGES.md`, `requirements.txt` (pinned), `data/raw/README.md`, `.gitignore`, `src/` package. ## 2. Code refactoring (results preserved — see §4) - **`src/config.py`** — all paths and constants (buffer radii, chunk size, Earth radius, random seed) in one place; output dirs auto-created; `require()` gives a clear error message when an input is missing instead of a traceback. - **`src/geo.py`** — the vectorised Haversine matrix (was duplicated in script 04). - **`src/latex_tables.py`** — significance stars (previously copy-pasted in 4 scripts) and the stargazer-style `results_to_latex` builder (was in script 06). - **`src/plotting.py`** — the two matplotlib style blocks (were duplicated across scripts 05, 09, 10). Scripts 07/08 intentionally keep matplotlib defaults, as in the original. - All scripts now have a `main()` entry point, docstrings, and import shared helpers; scripts 01–03 fail with an explanatory message when the raw data are absent. Script 10's two near-identical OLS helpers were merged into one (`run_ols_city_fe`, parameterised by the Airbnb exposure variable). - **Table fragments:** the published `outputs/tables/*.tex` had been *hand-edited* after generation (the `\begin{table}…\end{table}` wrappers were stripped so the paper could `\input` them inside its own table environments). The refactored generators emit exactly those fragments, so pipeline output now feeds the paper directly with no manual post-processing. - Deliberately **kept as-is** (to reproduce identical outputs): the `size_sqft`/`guests_count`/etc. column names requested by script 05 that don't exist in the processed data (the summary-stats helper skips them, matching the published tables) — noted in `AUDIT.md` §5.4. ## 3. Pipeline verification (Phase 2b) Full re-run of steps 04→10 in this repo, compared against the originals: - 3/3 merged parquet files **value-identical**. - 14/16 tables **byte-identical**. Two residual, numerically negligible differences (documented, not corrected): the τ=0.10 quantile-regression column (3 control coefficients at the 4th decimal — IRLS convergence jitter; the Airbnb coefficient and all conclusions unchanged) and 2 values of `ml_comparison.csv` at the 16th significant digit. Details: `AUDIT.md` §8. - All **15 figures regenerated**, including the 5 missing from the original project (`map_airbnb`, `map_rent`, `quantile_coefficients`, `rent_by_city`, `rent_by_airbnb_bins`) — this un-breaks the paper build, which previously failed on the missing maps. ## 4. Paper rewrite (`paper/`) Structure was already `main.tex` + `sections/` + `appendix/` + `references.bib`; it was kept, with figure paths pointed at `../figures/` and table inputs at `../results/tables/`. `latexmk` builds `main.pdf` (41 pages) with **zero unresolved references or citations**. All 24 BibTeX entries are cited; every figure/table in the paper is referenced in the text and captioned. ### Editorial corrections that ALIGN THE TEXT WITH THE PAPER'S OWN TABLES **⚠️ These need your review — the previous prose contradicted the (unchanged) tables:** 1. **Abstract, §5.2, Conclusion — Airbnb pricing model (Model 2).** The text claimed listings in higher-rent cities "command significant price premia" (with a "$100 → 2–4%" magnitude). Table 4 shows `mean_rent_city` is *insignificant in every column* (negative point estimates in 2a/2b). The text now reports the null result and its interpretation. Also corrected: the claimed positive rating/superhost premia (rating is not in the model; superhost is significantly **negative**). 2. **§5.3 City-level model.** "Strong positive correlation … fewer than 30 cities" → modest but significant forward association (0.0001**), essentially zero explanatory power in the reverse regression, and the actual 153 cities. 3. **§5.5 Quantile regressions.** "Insignificant at the 10th percentile, monotonically increasing" → significant at *all* quantiles, roughly flat over the lower half, rising to its maximum (0.0047) at τ=0.90. 4. **§6 Alternative exposure / §7 Commercialisation.** The text claimed the entire-home share is positively associated with rents (supporting the commercialisation hypothesis). Model 1e shows a small *negative, insignificant* coefficient; the text now reports this and discusses why. The phantom "mean Airbnb price" alternative measure (never in the table) was removed from the list. 5. **§5.6 ML results.** "Gradient boosting best, all ML beat OLS; bedrooms and city indicators most important" → random forest is best (test R²=0.71), the regularised linear models tie OLS, and the top features are bathrooms, coordinates, and bedrooms (city indicators are not ML features). 6. **§5.1 Baseline description.** Column description now matches the actual (1a)–(1e) layout; bedroom effect corrected to 11–13% (was "15–25%"), bathrooms 23–30%; adjusted R² 0.56 (was "0.40–0.55"). 7. **§6 Outlier sensitivity.** Text described winsorisation at 1/99; the code *trims* at rent 5/95 and Airbnb count 1/99. Text now matches, and the "negligible effect" claim was corrected (the rent-trimmed estimate is ~20% smaller, still significant). ### Methods descriptions aligned with the actual implementation 8. **Model 4 / methodological appendix.** Radius-based leave-one-out spatial lag estimated by OLS → the implemented KNN(k=5) row-standardised weights with SAR estimated by `GM_Lag` and SEM by `GM_Error` (spreg). 9. **Quantile SEs.** "Bootstrap, 1,000 replications" → the asymptotic kernel-based SEs actually produced by `statsmodels.QuantReg`. 10. **Clustered SEs.** Claims that clustered-SE results are "reported" were removed (none were computed); replaced by an honest inference caveat. 11. **Model 2/6 covariate lists, GBM "early stopping", RF "permutation importance"** → corrected to the actual controls, fixed 500 iterations, and SHAP values. 12. **Figure captions.** The two distribution histograms were captioned "(log scale)" but plot levels in CAD → captions corrected. 13. **Appendix sample-attrition numbers** updated to the actual counts (Airbnb 5,000 → 3,456; rent 8,356 → 8,303; regression sample 7,925). Sections 1 (introduction), 2 (literature) and most of 7–8 needed only the consistency fixes above; the prose was already in polished academic English and was otherwise preserved. ## 4bis. Extended robustness added on request ("make paper more robust") New analyses (2026-08-05, after the initial restructuring commit) — the existing tables and figures are untouched; everything below is **additive**: - **`scripts/11_extended_robustness.py`** → `results/tables/extended_robustness.tex` (Table 12) and `figures/leave_one_city_out.pdf` (Figure 9): - (R1) baseline (replicates Model 1c exactly: β = 0.0039, HC1); - (R2) **city-clustered standard errors** (153 clusters) — clustering *tightens* the SE on the Airbnb coefficient (0.0001 vs 0.0002), so baseline significance is conservative on this dimension; - (R3) **interior-size control** (per 100 sq ft, N = 3,531) — β = 0.0032***; - (R4) **log(1+count) functional form** — 0.048***: doubling the nearby count ≈ +3.4% rent; mild concavity; - (R5) **ring decomposition** (500m count + 500m–1km annulus jointly) — both positive and significant (0.0013 / 0.0018); reported honestly: the association extends to the kilometre scale rather than being confined to 500m; - **leave-one-city-out** over the 8 largest cities — β ranges 0.0038–0.0040; excluding Montreal (half the sample) leaves it at 0.0038. - Paper: new §6.5 "Additional Specification Checks" and §6.6 "Leave-One-City-Out Sensitivity"; the speculative "Inference Caveats" subsection was replaced by the actual clustered-SE result; §4 (Standard Errors) and §6.7 updated. - Cosmetic: `\small` + tighter `\tabcolsep` on the three widest tables removed the pre-existing overfull-hbox warnings (1 minor one remains, in a text line). - PDF: 43 pages, 0 unresolved references/citations. ## 5. Items requiring your review - **Raw data missing** (`airbnb.csv`, `rent.json`): absent from the original folder and the whole disk. Steps 01–03 are ready but cannot run until the files are restored to `data/raw/`. Everything else reproduces from `data/processed/`. - **The text↔table contradictions in §4 above** (especially item 1, which changes the abstract's fourth claim, and item 4). The tables were and remain the original results; if you believe the *tables* are wrong instead, the code paths to investigate are `scripts/06_hedonic_models.py` (`mean_rent_city`, `share_entire_home_500m`). - The two numerically negligible reproduction differences (`AUDIT.md` §8). - The paper still says "Version 1.0, May 2026" — bump `\WPversion`/`\WPdate` in `paper/main.tex` if you consider this revision a new version. - `figures/ml_predicted_vs_actual.pdf`, `airbnb_by_city.pdf`, `rent_by_city.pdf`, `scatter_airbnb_rent.pdf`, `rent_airbnb_heatmap.pdf`, `rent_by_airbnb_bins.pdf` are generated but not included in the paper (same as the original); include them if desired.