# CHANGES — Restructuring of immo-wp5-spb-20260519 → wp5_uqo

Date: 2026-08-05. The original project at ~/Desktop/UQO/UQO_WP/immo-wp5-spb-20260519 was left completely untouched and serves as the backup (in place of an _old/ copy inside this repo).

Per request, every file created in this repo carries the header Author: Simon-Pierre Boucher — contact@spboucher.ai (comment syntax adapted per language: # Python/Makefile, % LaTeX/BibTeX, <!-- --> Markdown; in references.bib the email is written contact (at) spboucher.ai because a literal @ inside a .bib comment breaks BibTeX).

# 1. Moved / renamed

New files: README.md, AUDIT.md, CHANGES.md, requirements.txt (pinned), data/raw/README.md, .gitignore, src/ package.

# 2. Code refactoring (results preserved — see §4)

• src/config.py — all paths and constants (buffer radii, chunk size, Earth radius, random seed) in one place; output dirs auto-created; require() gives a clear error message when an input is missing instead of a traceback.
• src/geo.py — the vectorised Haversine matrix (was duplicated in script 04).
• src/latex_tables.py — significance stars (previously copy-pasted in 4 scripts) and the stargazer-style results_to_latex builder (was in script 06).
• src/plotting.py — the two matplotlib style blocks (were duplicated across scripts 05, 09, 10). Scripts 07/08 intentionally keep matplotlib defaults, as in the original.
• All scripts now have a main() entry point, docstrings, and import shared helpers; scripts 01–03 fail with an explanatory message when the raw data are absent. Script 10's two near-identical OLS helpers were merged into one (run_ols_city_fe, parameterised by the Airbnb exposure variable).
• Table fragments: the published outputs/tables/*.tex had been hand-edited after generation (the \begin{table}…\end{table} wrappers were stripped so the paper could \input them inside its own table environments). The refactored generators emit exactly those fragments, so pipeline output now feeds the paper directly with no manual post-processing.
• Deliberately kept as-is (to reproduce identical outputs): the size_sqft/guests_count/etc. column names requested by script 05 that don't exist in the processed data (the summary-stats helper skips them, matching the published tables) — noted in AUDIT.md §5.4.

# 3. Pipeline verification (Phase 2b)

Full re-run of steps 04→10 in this repo, compared against the originals:

• 3/3 merged parquet files value-identical.
• 14/16 tables byte-identical. Two residual, numerically negligible differences (documented, not corrected): the τ=0.10 quantile-regression column (3 control coefficients at the 4th decimal — IRLS convergence jitter; the Airbnb coefficient and all conclusions unchanged) and 2 values of ml_comparison.csv at the 16th significant digit. Details: AUDIT.md §8.
• All 15 figures regenerated, including the 5 missing from the original project (map_airbnb, map_rent, quantile_coefficients, rent_by_city, rent_by_airbnb_bins) — this un-breaks the paper build, which previously failed on the missing maps.

# 4. Paper rewrite (paper/)

Structure was already main.tex + sections/ + appendix/ + references.bib; it was kept, with figure paths pointed at ../figures/ and table inputs at ../results/tables/. latexmk builds main.pdf (41 pages) with zero unresolved references or citations. All 24 BibTeX entries are cited; every figure/table in the paper is referenced in the text and captioned.

# Editorial corrections that ALIGN THE TEXT WITH THE PAPER'S OWN TABLES

⚠️ These need your review — the previous prose contradicted the (unchanged) tables:

1. Abstract, §5.2, Conclusion — Airbnb pricing model (Model 2). The text claimed listings in higher-rent cities "command significant price premia" (with a "$100 → 2–4%" magnitude). Table 4 shows mean_rent_city is insignificant in every column (negative point estimates in 2a/2b). The text now reports the null result and its interpretation. Also corrected: the claimed positive rating/superhost premia (rating is not in the model; superhost is significantly negative).
2. §5.3 City-level model. "Strong positive correlation … fewer than 30 cities" → modest but significant forward association (0.0001**), essentially zero explanatory power in the reverse regression, and the actual 153 cities.
3. §5.5 Quantile regressions. "Insignificant at the 10th percentile, monotonically increasing" → significant at all quantiles, roughly flat over the lower half, rising to its maximum (0.0047) at τ=0.90.
4. §6 Alternative exposure / §7 Commercialisation. The text claimed the entire-home share is positively associated with rents (supporting the commercialisation hypothesis). Model 1e shows a small negative, insignificant coefficient; the text now reports this and discusses why. The phantom "mean Airbnb price" alternative measure (never in the table) was removed from the list.
5. §5.6 ML results. "Gradient boosting best, all ML beat OLS; bedrooms and city indicators most important" → random forest is best (test R²=0.71), the regularised linear models tie OLS, and the top features are bathrooms, coordinates, and bedrooms (city indicators are not ML features).
6. §5.1 Baseline description. Column description now matches the actual (1a)–(1e) layout; bedroom effect corrected to 11–13% (was "15–25%"), bathrooms 23–30%; adjusted R² 0.56 (was "0.40–0.55").
7. §6 Outlier sensitivity. Text described winsorisation at 1/99; the code trims at rent 5/95 and Airbnb count 1/99. Text now matches, and the "negligible effect" claim was corrected (the rent-trimmed estimate is ~20% smaller, still significant).

# Methods descriptions aligned with the actual implementation

8. Model 4 / methodological appendix. Radius-based leave-one-out spatial lag estimated by OLS → the implemented KNN(k=5) row-standardised weights with SAR estimated by GM_Lag and SEM by GM_Error (spreg).
9. Quantile SEs. "Bootstrap, 1,000 replications" → the asymptotic kernel-based SEs actually produced by statsmodels.QuantReg.
10. Clustered SEs. Claims that clustered-SE results are "reported" were removed (none were computed); replaced by an honest inference caveat.
11. Model 2/6 covariate lists, GBM "early stopping", RF "permutation importance" → corrected to the actual controls, fixed 500 iterations, and SHAP values.
12. Figure captions. The two distribution histograms were captioned "(log scale)" but plot levels in CAD → captions corrected.
13. Appendix sample-attrition numbers updated to the actual counts (Airbnb 5,000 → 3,456; rent 8,356 → 8,303; regression sample 7,925).

Sections 1 (introduction), 2 (literature) and most of 7–8 needed only the consistency fixes above; the prose was already in polished academic English and was otherwise preserved.

# 4bis. Extended robustness added on request ("make paper more robust")

New analyses (2026-08-05, after the initial restructuring commit) — the existing tables and figures are untouched; everything below is additive:

• scripts/11_extended_robustness.py → results/tables/extended_robustness.tex (Table 12) and figures/leave_one_city_out.pdf (Figure 9):
  • (R1) baseline (replicates Model 1c exactly: β = 0.0039, HC1);
  • (R2) city-clustered standard errors (153 clusters) — clustering tightens the SE on the Airbnb coefficient (0.0001 vs 0.0002), so baseline significance is conservative on this dimension;
  • (R3) interior-size control (per 100 sq ft, N = 3,531) — β = 0.0032***;
  • (R4) log(1+count) functional form — 0.048***: doubling the nearby count ≈ +3.4% rent; mild concavity;
  • (R5) ring decomposition (500m count + 500m–1km annulus jointly) — both positive and significant (0.0013 / 0.0018); reported honestly: the association extends to the kilometre scale rather than being confined to 500m;
  • leave-one-city-out over the 8 largest cities — β ranges 0.0038–0.0040; excluding Montreal (half the sample) leaves it at 0.0038.
• Paper: new §6.5 "Additional Specification Checks" and §6.6 "Leave-One-City-Out Sensitivity"; the speculative "Inference Caveats" subsection was replaced by the actual clustered-SE result; §4 (Standard Errors) and §6.7 updated.
• Cosmetic: \small + tighter \tabcolsep on the three widest tables removed the pre-existing overfull-hbox warnings (1 minor one remains, in a text line).
• PDF: 43 pages, 0 unresolved references/citations.

# 5. Items requiring your review

• Raw data missing (airbnb.csv, rent.json): absent from the original folder and the whole disk. Steps 01–03 are ready but cannot run until the files are restored to data/raw/. Everything else reproduces from data/processed/.
• The text↔table contradictions in §4 above (especially item 1, which changes the abstract's fourth claim, and item 4). The tables were and remain the original results; if you believe the tables are wrong instead, the code paths to investigate are scripts/06_hedonic_models.py (mean_rent_city, share_entire_home_500m).
• The two numerically negligible reproduction differences (AUDIT.md §8).
• The paper still says "Version 1.0, May 2026" — bump \WPversion/\WPdate in paper/main.tex if you consider this revision a new version.
• figures/ml_predicted_vs_actual.pdf, airbnb_by_city.pdf, rent_by_city.pdf, scatter_airbnb_rent.pdf, rent_airbnb_heatmap.pdf, rent_by_airbnb_bins.pdf are generated but not included in the paper (same as the original); include them if desired.