spb/wp5_uqo Public

1<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai -->23# CHANGES — Restructuring of `immo-wp5-spb-20260519` → `wp5_uqo`45Date: 2026-08-05.  The original project at6`~/Desktop/UQO/UQO_WP/immo-wp5-spb-20260519` was **left completely untouched** and7serves as the backup (in place of an `_old/` copy inside this repo).89Per request, every file created in this repo carries the header10`Author: Simon-Pierre Boucher — contact@spboucher.ai` (comment syntax adapted per11language: `#` Python/Makefile, `%` LaTeX/BibTeX, `<!-- -->` Markdown; in12`references.bib` the email is written `contact (at) spboucher.ai` because a literal13`@` inside a `.bib` comment breaks BibTeX).1415## 1. Moved / renamed1617| Original | New |18|---|---|19| `airbnb.csv`, `rent.json` (project root) | `data/raw/` (files missing — see §5) |20| `data_clean/*.parquet` (5 files) | `data/processed/` (bit-identical copies, checksummed) |21| `scripts/01_load_inspect_data.py` | `scripts/01_inspect_raw_data.py` |22| `scripts/02…09_*.py` | same names under `scripts/` (refactored) |23| `scripts/10_generate_tables_figures.py` | `scripts/10_robustness_tables_figures.py` |24| `outputs/tables/` | `results/tables/` |25| `outputs/logs/` | `results/logs/` |26| `outputs/figures/` + `wp5/figures/` (duplicates) | `figures/` (single source, regenerated) |27| `wp5/` (canonical LaTeX) | `paper/` |28| `paper/` (old LaTeX copy), `wp5/main_web.tex` + fallback hack | **dropped** (superseded; the fallback is unnecessary now that all figures exist) |29| `scripts/__pycache__/`, `.DS_Store`, LaTeX build artifacts | dropped / gitignored |3031New files: `README.md`, `AUDIT.md`, `CHANGES.md`, `requirements.txt` (pinned),32`data/raw/README.md`, `.gitignore`, `src/` package.3334## 2. Code refactoring (results preserved — see §4)3536- **`src/config.py`** — all paths and constants (buffer radii, chunk size, Earth37  radius, random seed) in one place; output dirs auto-created; `require()` gives a38  clear error message when an input is missing instead of a traceback.39- **`src/geo.py`** — the vectorised Haversine matrix (was duplicated in script 04).40- **`src/latex_tables.py`** — significance stars (previously copy-pasted in 441  scripts) and the stargazer-style `results_to_latex` builder (was in script 06).42- **`src/plotting.py`** — the two matplotlib style blocks (were duplicated across43  scripts 05, 09, 10). Scripts 07/08 intentionally keep matplotlib defaults, as44  in the original.45- All scripts now have a `main()` entry point, docstrings, and import shared46  helpers; scripts 01–03 fail with an explanatory message when the raw data are47  absent. Script 10's two near-identical OLS helpers were merged into one48  (`run_ols_city_fe`, parameterised by the Airbnb exposure variable).49- **Table fragments:** the published `outputs/tables/*.tex` had been *hand-edited*50  after generation (the `\begin{table}…\end{table}` wrappers were stripped so the51  paper could `\input` them inside its own table environments). The refactored52  generators emit exactly those fragments, so pipeline output now feeds the paper53  directly with no manual post-processing.54- Deliberately **kept as-is** (to reproduce identical outputs): the55  `size_sqft`/`guests_count`/etc. column names requested by script 05 that don't56  exist in the processed data (the summary-stats helper skips them, matching the57  published tables) — noted in `AUDIT.md` §5.4.5859## 3. Pipeline verification (Phase 2b)6061Full re-run of steps 04→10 in this repo, compared against the originals:6263- 3/3 merged parquet files **value-identical**.64- 14/16 tables **byte-identical**. Two residual, numerically negligible65  differences (documented, not corrected): the τ=0.10 quantile-regression column66  (3 control coefficients at the 4th decimal — IRLS convergence jitter; the67  Airbnb coefficient and all conclusions unchanged) and 2 values of68  `ml_comparison.csv` at the 16th significant digit. Details: `AUDIT.md` §8.69- All **15 figures regenerated**, including the 5 missing from the original70  project (`map_airbnb`, `map_rent`, `quantile_coefficients`, `rent_by_city`,71  `rent_by_airbnb_bins`) — this un-breaks the paper build, which previously72  failed on the missing maps.7374## 4. Paper rewrite (`paper/`)7576Structure was already `main.tex` + `sections/` + `appendix/` + `references.bib`;77it was kept, with figure paths pointed at `../figures/` and table inputs at78`../results/tables/`. `latexmk` builds `main.pdf` (41 pages) with **zero79unresolved references or citations**. All 24 BibTeX entries are cited; every80figure/table in the paper is referenced in the text and captioned.8182### Editorial corrections that ALIGN THE TEXT WITH THE PAPER'S OWN TABLES83**⚠️ These need your review — the previous prose contradicted the (unchanged) tables:**84851. **Abstract, §5.2, Conclusion — Airbnb pricing model (Model 2).** The text86   claimed listings in higher-rent cities "command significant price premia"87   (with a "$100 → 2–4%" magnitude). Table 4 shows `mean_rent_city` is88   *insignificant in every column* (negative point estimates in 2a/2b). The text89   now reports the null result and its interpretation. Also corrected: the90   claimed positive rating/superhost premia (rating is not in the model;91   superhost is significantly **negative**).922. **§5.3 City-level model.** "Strong positive correlation … fewer than 3093   cities" → modest but significant forward association (0.0001**), essentially94   zero explanatory power in the reverse regression, and the actual 153 cities.953. **§5.5 Quantile regressions.** "Insignificant at the 10th percentile,96   monotonically increasing" → significant at *all* quantiles, roughly flat over97   the lower half, rising to its maximum (0.0047) at τ=0.90.984. **§6 Alternative exposure / §7 Commercialisation.** The text claimed the99   entire-home share is positively associated with rents (supporting the100   commercialisation hypothesis). Model 1e shows a small *negative,101   insignificant* coefficient; the text now reports this and discusses why. The102   phantom "mean Airbnb price" alternative measure (never in the table) was103   removed from the list.1045. **§5.6 ML results.** "Gradient boosting best, all ML beat OLS; bedrooms and105   city indicators most important" → random forest is best (test R²=0.71), the106   regularised linear models tie OLS, and the top features are bathrooms,107   coordinates, and bedrooms (city indicators are not ML features).1086. **§5.1 Baseline description.** Column description now matches the actual109   (1a)–(1e) layout; bedroom effect corrected to 11–13% (was "15–25%"),110   bathrooms 23–30%; adjusted R² 0.56 (was "0.40–0.55").1117. **§6 Outlier sensitivity.** Text described winsorisation at 1/99; the code112   *trims* at rent 5/95 and Airbnb count 1/99. Text now matches, and the113   "negligible effect" claim was corrected (the rent-trimmed estimate is ~20%114   smaller, still significant).115116### Methods descriptions aligned with the actual implementation1178. **Model 4 / methodological appendix.** Radius-based leave-one-out spatial lag118   estimated by OLS → the implemented KNN(k=5) row-standardised weights with SAR119   estimated by `GM_Lag` and SEM by `GM_Error` (spreg).1209. **Quantile SEs.** "Bootstrap, 1,000 replications" → the asymptotic121   kernel-based SEs actually produced by `statsmodels.QuantReg`.12210. **Clustered SEs.** Claims that clustered-SE results are "reported" were123    removed (none were computed); replaced by an honest inference caveat.12411. **Model 2/6 covariate lists, GBM "early stopping", RF "permutation125    importance"** → corrected to the actual controls, fixed 500 iterations, and126    SHAP values.12712. **Figure captions.** The two distribution histograms were captioned "(log128    scale)" but plot levels in CAD → captions corrected.12913. **Appendix sample-attrition numbers** updated to the actual counts130    (Airbnb 5,000 → 3,456; rent 8,356 → 8,303; regression sample 7,925).131132Sections 1 (introduction), 2 (literature) and most of 7–8 needed only the133consistency fixes above; the prose was already in polished academic English and134was otherwise preserved.135136## 4bis. Extended robustness added on request ("make paper more robust")137138New analyses (2026-08-05, after the initial restructuring commit) — the existing139tables and figures are untouched; everything below is **additive**:140141- **`scripts/11_extended_robustness.py`** → `results/tables/extended_robustness.tex`142  (Table 12) and `figures/leave_one_city_out.pdf` (Figure 9):143  - (R1) baseline (replicates Model 1c exactly: β = 0.0039, HC1);144  - (R2) **city-clustered standard errors** (153 clusters) — clustering *tightens*145    the SE on the Airbnb coefficient (0.0001 vs 0.0002), so baseline significance146    is conservative on this dimension;147  - (R3) **interior-size control** (per 100 sq ft, N = 3,531) — β = 0.0032***;148  - (R4) **log(1+count) functional form** — 0.048***: doubling the nearby count149    ≈ +3.4% rent; mild concavity;150  - (R5) **ring decomposition** (500m count + 500m–1km annulus jointly) — both151    positive and significant (0.0013 / 0.0018); reported honestly: the152    association extends to the kilometre scale rather than being confined to153    500m;154  - **leave-one-city-out** over the 8 largest cities — β ranges 0.0038–0.0040;155    excluding Montreal (half the sample) leaves it at 0.0038.156- Paper: new §6.5 "Additional Specification Checks" and §6.6 "Leave-One-City-Out157  Sensitivity"; the speculative "Inference Caveats" subsection was replaced by158  the actual clustered-SE result; §4 (Standard Errors) and §6.7 updated.159- Cosmetic: `\small` + tighter `\tabcolsep` on the three widest tables removed160  the pre-existing overfull-hbox warnings (1 minor one remains, in a text line).161- PDF: 43 pages, 0 unresolved references/citations.162163## 5. Items requiring your review164165- **Raw data missing** (`airbnb.csv`, `rent.json`): absent from the original166  folder and the whole disk. Steps 01–03 are ready but cannot run until the files167  are restored to `data/raw/`. Everything else reproduces from `data/processed/`.168- **The text↔table contradictions in §4 above** (especially item 1, which changes169  the abstract's fourth claim, and item 4). The tables were and remain the170  original results; if you believe the *tables* are wrong instead, the code paths171  to investigate are `scripts/06_hedonic_models.py` (`mean_rent_city`,172  `share_entire_home_500m`).173- The two numerically negligible reproduction differences (`AUDIT.md` §8).174- The paper still says "Version 1.0, May 2026" — bump `\WPversion`/`\WPdate` in175  `paper/main.tex` if you consider this revision a new version.176- `figures/ml_predicted_vs_actual.pdf`, `airbnb_by_city.pdf`, `rent_by_city.pdf`,177  `scatter_airbnb_rent.pdf`, `rent_airbnb_heatmap.pdf`, `rent_by_airbnb_bins.pdf`178  are generated but not included in the paper (same as the original); include179  them if desired.180