spb/wp5_uqo Public

1<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai -->23# AUDIT — Original project `immo-wp5-spb-20260519`45Audit of `~/Desktop/UQO/UQO_WP/immo-wp5-spb-20260519` performed on 2026-08-05, **before** any restructuring.6The original folder is left **untouched**; the clean repository is rebuilt at `~/Desktop/wp5_uqo`.78## 1. Project summary910Hedonic and spatial econometric analysis of the relationship between Airbnb activity and11residential rents in Quebec, Canada (UQO Working Paper No. 5, Simon-Pierre Boucher, May 2026).12~5,000 Airbnb listings + 8,356 Realtor.ca rental listings, spatial buffer merge (Haversine,13250 m–2 km), OLS/hedonic models, spatial models (SAR/SEM via spreg), quantile regressions,14ML robustness (LASSO, Elastic Net, RF, GBM, SHAP).1516## 2. Scripts (`scripts/`, 10 files, ~2,930 lines)1718| Script | Role | Inputs | Outputs |19|---|---|---|---|20| `01_load_inspect_data.py` | Inspect raw data, write data dictionary log | `airbnb.csv`, `rent.json` (**missing**, see §5.1) | `outputs/logs/data_inspection.txt` |21| `02_clean_airbnb.py` | Standardize cities, winsorize price p1/p99, `log_price`, impute rating (median) & reviews (0), `is_entire_home` | `airbnb.csv` (**missing**) | `data_clean/airbnb_clean.parquet` |22| `03_clean_rent.py` | Filter Single Family + Monthly, parse rent/city/borough/lat/lon/beds/baths/size, winsorize p1/p99, `log_rent` | `rent.json` (**missing**) | `data_clean/rent_clean.parquet` |23| `04_merge_data.py` | Spatial buffer merge (Haversine 250 m/500 m/1 km/2 km, chunked), city/borough aggregation, combined analysis file | the 2 clean parquets | `merged_spatial.parquet`, `merged_neighborhood.parquet`, `merged_analysis.parquet` |24| `05_descriptive_analysis.py` | Summary-stat tables, correlation matrix, 7 descriptive figures | clean + merged parquets | 3 tables, 7 figures |25| `06_hedonic_models.py` | Models 1a–1e (rent), 2a–2c (Airbnb pricing), 3 fwd/rev (city level), OLS HC1 | `merged_analysis`, `airbnb_clean` | 3 `.tex` tables |26| `07_spatial_models.py` | Model 4: SAR (GM_Lag) + SEM (GM_Error) via spreg, KNN(5) weights; buffer robustness 250 m–2 km | `merged_analysis` | 2 tables + `coefficient_buffer_comparison.pdf` |27| `08_quantile_models.py` | Model 5: quantile regressions τ∈{.10,.25,.50,.75,.90} + fine grid plot | `merged_analysis` | 1 table + `quantile_coefficients.pdf` |28| `09_ml_robustness.py` | Model 6: OLS/LASSO/ENet/RF/GBM (seed 42, 80/20 split), SHAP | `merged_analysis` | `ml_comparison.{tex,csv}` + 3 figures |29| `10_generate_tables_figures.py` | Outlier/winsorization + subsample robustness tables, hexbin heatmap, coefficient-robustness plot, quintile bar chart | `merged_analysis` | 2 tables + 3 figures |3031All paths are already relative to the project root via `Path(__file__).resolve().parent.parent` — good.32Scripts are standalone monoliths with heavy duplication (see §5.4).3334## 3. Data3536- `data_clean/` (5 parquet files, all present and readable):37  `airbnb_clean.parquet` (4,950 rows expected), `rent_clean.parquet` (8,303 rows),38  `merged_spatial.parquet`, `merged_neighborhood.parquet`, `merged_analysis.parquet` (8,258 rows × 42 cols).39- **Raw data (`airbnb.csv`, `rent.json`) are absent** from the folder and from the whole disk40  (searched `~/Desktop/UQO` and Spotlight). See §5.1.4142## 4. Figure inventory4344Figures **generated by the code** (15) vs figures **present** in `outputs/figures/` (10):4546| Figure | Generated by | In `outputs/figures/` | Referenced by paper |47|---|---|---|---|48| dist_airbnb_price.pdf | 05 | yes | yes |49| dist_rent.pdf | 05 | yes | yes |50| airbnb_by_city.pdf | 05 | yes | no |51| rent_by_city.pdf | 05 | **no** | no |52| scatter_airbnb_rent.pdf | 05 | yes | no |53| map_airbnb.pdf | 05 | **no** | **yes → broken** |54| map_rent.pdf | 05 | **no** | **yes → broken** |55| coefficient_buffer_comparison.pdf | 07 | yes | yes (×2) |56| quantile_coefficients.pdf | 08 | **no** | **yes → broken** |57| ml_predicted_vs_actual.pdf | 09 | yes | no |58| feature_importance.pdf | 09 | yes | yes |59| shap_summary.pdf | 09 | yes | yes |60| rent_airbnb_heatmap.pdf | 10 | yes | no |61| coefficient_robustness.pdf | 10 | yes | yes |62| rent_by_airbnb_bins.pdf | 10 | **no** | no |6364The 5 missing PDFs were evidently deleted (or the run predates those functions); the code that65produces them is intact. **Consequence:** `wp5/main.tex` fails to compile (fatal error, no PDF);66only `wp5/main_web.tex` compiles because it replaces missing figures with placeholder boxes.67Re-running the pipeline regenerates all 15 figures and fixes the paper build.6869`wp5/figures/` contains a stale copy of the same 10 PDFs (duplicate of `outputs/figures/`).7071## 5. Problems found (nothing was changed; fixes happen in the new repo)7273### 5.1 Raw data missing (blocking for steps 01–03 only)74`airbnb.csv` and `rent.json` are referenced by scripts 01–03 and by the original README but do75not exist anywhere on disk. The pipeline is therefore reproducible **from `data_clean/` onward**76(steps 04–10 + paper), which covers every number and figure in the paper. Steps 01–03 are kept77in the new repo and fail with an explicit message if the raw files are absent.78**→ Requires user review: locate/restore the raw files if full from-scratch reproduction is needed.**7980### 5.2 Two divergent copies of the paper81- `paper/` — same sections, figures referenced as `../outputs/figures/…`, no title page, compiled OK.82- `wp5/` — adds UQO title page (`uq_logo.jpg`), `Makefile`, `.latexmkrc`, `\graphicspath{{figures/}}`,83  plus `main_web.tex` (variant with a "Figure indisponible" fallback macro).84- Only real content differences: figure paths in `03_data.tex`, `05_results.tex`, `06_robustness.tex`.85- `wp5/` is the canonical/most recent version (June 2026) → used as the base for the rewrite.8687### 5.3 Figure caption vs content mismatch88`03_data.tex` captions the two distribution histograms as "(log scale)" but scripts plot **levels**89(CAD). Caption corrected in the rewrite (editorial fix; figures unchanged).9091### 5.4 Code-quality issues (fixed by refactor, results preserved)92- Significance-star helper duplicated in 4 scripts; LaTeX regression-table builder duplicated in 4 scripts;93  matplotlib style blocks duplicated in 3 scripts; city-FE OLS helper duplicated within script 10.94- Scripts 05 requests columns `size_sqft`, `guests_count`, `amenities_count`, `num_images`,95  `quality_score` that don't exist in the parquets (`rent_clean` has `size_interior_sqft`); the96  helper silently skips them, so the published summary tables lack those rows. **Kept as-is** to97  reproduce identical tables; noted for future work.98- Scripts 06–08 are top-level scripts (no `main()`); 02/03 likewise.99- `scripts/__pycache__/` committed; `.DS_Store` files scattered.100101### 5.5 Dead / duplicate / unused files102- `paper/` entire directory superseded by `wp5/` (kept only as reference in the original folder).103- `wp5/main_web.tex` + `main_web.*` build artifacts: web variant with placeholder hack — obsolete104  once figures are regenerated.105- LaTeX build artifacts (`.aux`, `.log`, `.out`, `.bbl`, `.blg`, `.fls`, `.fdb_latexmk`, `.toc`, `.synctex.gz`).106- `scripts/__pycache__/`, `.DS_Store` (×6).107- `outputs/figures/` vs `wp5/figures/`: duplicated PDFs.108109### 5.6 Environment110Python 3.14.4 (Homebrew) with pandas 3.0.2, numpy 2.4.4, statsmodels 0.14.6, scikit-learn 1.6.1,111scipy 1.17.1, matplotlib 3.10.9, pyarrow 24.0.0, shap 0.48.0, libpysal 4.14.1, spreg 1.9.0.112TeX Live (`pdflatex`, `latexmk`, `bibtex`) available. Note: with libpysal+spreg installed, script 07113takes the SAR/SEM branch — consistent with the published `spatial_models.tex`.114115## 6. Paper (`wp5/`)116117- `main.tex` (184 lines): clean preamble (newtx, booktabs, natbib/apalike, fancyhdr, hyperref),118  metadata macros, inputs 8 sections + title page + 3 appendices; 24 BibTeX entries, all cited.119- Sections: 01 introduction, 02 literature, 03 data, 04 methodology (models 1–6 + identification),120  05 results, 06 robustness, 07 discussion, 08 conclusion; appendices: data, methods, robustness.121- Prose already in good academic English; rewrite pass = flow/consistency polish, caption fix122  (§5.3), figure/table path normalization to the new layout, and making every figure resolvable.123- Tables are `\input` from `../outputs/tables/*.tex` → becomes `../results/tables/` in the new repo.124125## 7. Verification plan (Phase 2b)1261271. Copy the 5 parquets to `data/processed/` (bit-identical, checksummed).1282. Re-run refactored steps 04→10; compare regenerated `merged_*.parquet` (DataFrame equality)129   and every `.tex`/`.csv` table against the originals (`diff`). Figures: confirm regeneration and130   spot-check values; PDF bytes differ (timestamps) so tables are the numeric ground truth.1313. Any discrepancy is recorded below — never silently corrected.132133## 8. Discrepancies found during verification134135Full pipeline (steps 04–10) re-run in the new repo on 2026-08-05 and compared to the originals:136137- **Merged parquets (3/3): identical** — `merged_spatial`, `merged_neighborhood`,138  `merged_analysis` regenerated from the committed clean parquets are value-identical139  to the originals (`pandas.testing.assert_frame_equal`).140- **Tables: 14/16 byte-identical** after aligning the generators with the published141  fragments (the original `outputs/tables/*.tex` had been hand-stripped of their142  `\begin{table}…\end{table}` wrappers after generation; the new scripts emit those143  fragments directly — see `CHANGES.md`). Residual differences, both preserved and144  NOT corrected:145  1. `quantile_regression.tex` — in the τ=0.10 column only, three control146     coefficients differ in the 4th decimal (bathrooms 0.2410→0.2411,147     bt_House 0.0607→0.0608, bt_Row/Townhouse 0.1302→0.1310, SE 0.0469→0.0468).148     The Airbnb coefficient, all other quantiles, and the OLS column are identical.149     Cause: IRLS convergence jitter of `statsmodels.QuantReg` at the extreme150     quantile; no significance level or claim in the paper changes.151  2. `ml_comparison.csv` — two values differ at the 16th significant digit152     (~1e-16 relative; float representation). The 4-decimal `ml_comparison.tex`153     is byte-identical.154- **Figures: all 15 regenerated**, including the 5 missing from the original155  (`map_airbnb`, `map_rent`, `quantile_coefficients`, `rent_by_city`,156  `rent_by_airbnb_bins`), which un-breaks the paper build. PDF bytes differ from157  the surviving originals (embedded timestamps/IDs); the numeric ground truth is158  the table set above, plus the underlying data verified identical.159