spb/wp2_uqo Public

1<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai -->23# CHANGES — restructuring of immo-wp2-spb-20260519 → wp2_uqo45Original project: `~/Desktop/UQO/UQO_WP/immo-wp2-spb-20260519` (untouched, serves as backup).6New repository: `~/Desktop/wp2_uqo`. Date: 2026-08-05.78## 1. What was moved / renamed910| Original | New location |11|---|---|12| `louka.db` | `data/raw/louka.db` |13| `embeddings_maisons.npy`, `sim_matrix_maisons.npy` | `data/processed/` (pipeline cache) |14| `hedonic_maison_results.csv` (+ earlier-approach CSVs/NPYs) | `data/processed/legacy/` (originals kept for comparison) |15| `hedonic_maison.py` (818 lines, monolithic) | split into `src/` modules + `scripts/01–04` |16| `paper_figures_v2.py` | `scripts/05_figures.py` (ports all 11 figures) |17| `wp2/main.tex` + `wp2/sections/` + `wp2/tables/` | `paper/` (preamble extracted to `paper/preamble.tex`) |18| `hedonic_semantic.py`, `hedonic_cosine_references.py`, `paper_figures.py`, root LaTeX tree, `paper.tex` | **not migrated** (superseded; documented in `AUDIT.md`) |1920## 2. Code refactoring2122- All hardcoded paths to the defunct `~/Desktop/d/` replaced by repository-relative23  paths in `src/config.py` (overridable via `WP2_DB_PATH`).24- The DB-extraction block that was copy-pasted across 5 scripts now lives once in25  `src/data.py`; the 20 reference descriptions (pasted 3×, in two languages) live26  once in `src/references.py` (slugs for data columns, English labels for display).27- Models A–E defined once in `src/models.py`; robustness checks ported faithfully28  (same estimators, same seeds: bootstrap `np.random.seed(42)`, CV `random_state=42`).29- New numbered pipeline: `01_prepare_data` → `02_similarities` → `03_models` →30  `04_robustness` → `05_figures` → `06_tables`. Everything runs end-to-end from31  `data/raw/louka.db`; embeddings are cached and `--force` re-encodes.32- **New `scripts/06_tables.py` generates the paper's numeric tables directly from33  the pipeline results** — this closes the transcription gap that produced the34  wrong numbers documented in §4 below.35- `requirements.txt` pins the exact package versions used for verification.3637## 3. Verification (regenerated vs. original artifacts)3839- Sample: 17,087 houses — **identical** (all 30 columns of40  `hedonic_maison_results.csv` equal to the original; similarity matrix41  bit-identical).42- Models A–E: R², adj. R², AIC, BIC, k — **identical to the paper's Table 4**43  (e.g. R² 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118; joint F = 99.53).44- Model D coefficients: **identical to the paper's Table 5** (all 27 coefficients,45  SEs, t-stats, impacts).46- Figures: regenerated fig1, fig4–fig11 match the shipped ones (same content,47  PDF sizes within ±2 bytes). **fig2 and fig3, referenced by the paper but missing48  from the original repo (the compiled PDF showed "Figure indisponible" boxes),49  are regenerated and now included.**50- `robustness_results.csv`, lost in the original, is regenerated (seeded, reproducible).5152## 4. Numbers corrected in the paper ⚠️ (review recommended)5354The original LaTeX contained statistics that **do not match the actual pipeline55outputs** (the project's own `figures/table1/table2` CSVs confirm the pipeline56values). They appear to have been transcribed from memory/drafted text rather57than from the analysis output. All were replaced by the reproducible values;58the paper's tables are now machine-generated (`scripts/06_tables.py`).5960| Claim | Old (paper) | New (verified) |61|---|---|---|62| Price mean | $764,508 | **$807,259** (median $589,900 unchanged) |63| Description length | mean 683, sd 367, max 5,399 | **mean 510, sd 151, max 703** (field truncated ~700 chars — now noted in the text) |64| Parking mean | 2.3 | **5.5** (counts all spots incl. driveway — now explained) |65| log-price sd | 0.68 | **0.70** |66| Similarity stats & correlations (Table 3) | e.g. Luxury corr −0.178 | **regenerated** (e.g. Luxury −0.234; all in [−0.29, −0.17]) |67| Breusch–Pagan | LM = 2,847.3 | **LM = 768.0** (same conclusion) |68| VIF | "none > 10, mean ≈ 4.2" | **mean 17.2, 16/20 > 10** — subsection rewritten honestly (block-level inference unaffected; O'Brien 2007 argument) |69| Inter-reference correlations | "0.4–0.8", "Motivated Seller lowest" | **0.66–0.97, mean 0.84**; top pair Entry-Level/Motivated Seller r = 0.97 |70| Winsorized model | 342 obs dropped, adj R² 0.498, shifts Luxury −0.012 / Motivated +0.008 | **341 obs, adj R² 0.486, largest shifts Entry-Level −0.024 / Motivated Seller +0.023** (no sign reversals, all 16 stay significant — unchanged conclusions) |71| Bootstrap | "SE within 5% of HC3" | **within 9% (15/20 within 5%)** |72| Lasso / Elastic Net | "Lasso keeps 15/20, ENet 17/20, high concordance with OLS" | **both keep 4/20** — subsection rewritten: sparse block-representative selection under strong collinearity, complementary rather than concordant |73| Quantile table | e.g. Luxury 0.098/0.125/0.176; "Modern & Land increasing" | **regenerated** (Luxury 0.093/0.130/0.160 increasing ✓; Modern high at both tails; Land & Nature stable) |74| PCA comparison | "PCA yields R² 0.496 < 0.512 → reference approach outperforms" | **PCA(20) actually fits better: ΔR² +0.070 vs +0.044** (49.5 % variance ✓). Reframed as an explicit interpretability-vs-fit trade-off (Table 8 + Discussion §PCA rewritten) |7576Qualitative conclusions that survive unchanged: heteroskedasticity → HC3;77no sign reversals under trimming; bootstrap ≈ HC3; Luxury premium increasing in78quantiles; discounts attenuating; joint semantic block highly significant.79Claims that had to change substance: **VIF/multicollinearity** (now acknowledged80as substantial), **Lasso concordance** (now honest about sparse selection), and81**PCA fit comparison** (now favors PCA on fit, reference approach on82interpretability). These are flagged because they alter the robustness83narrative, not the headline results.8485## 5. Paper rewrite (Phase 3)8687- `paper/main.tex` slimmed (metadata + inputs); packages and macros extracted to88  `paper/preamble.tex`; figures pulled from `../figures/` via `\graphicspath`;89  the `main_web.tex` variant with the silent figure-fallback hack was dropped.90- All **[TODO] blocks removed** (robustness §7.7–7.9, methodology, summary table)91  and converted into completed analyses (PCA comparison), honest limitations92  (Discussion), or future-research items (Conclusion).93- New appendix with real content: **A. verbatim French reference descriptions**94  (replication) and **B. Model E coefficient table** — both machine-generated.95- Editorial pass on Data/Results/Robustness/Discussion: corrected figure-3 caption96  (boxplots, not KDE), documented the description-truncation and parking-field97  quirks, tightened the collinearity narrative, consistent notation preserved.98- Fig 10 y-labels now use readable names instead of raw variable names.99- Title page: "First draft: May 2025" corrected to **May 2026** (the data were100  scraped January 2026); version bumped to 1.1.101- Abstract: "robust to bootstrap inference, winsorization, and Lasso selection"102  → "robust to bootstrap inference and outlier trimming" (accurate).103- Compiles clean: `latexmk -pdf main.tex` → **47 pages, 0 errors, 0 undefined104  references/citations** (24 minor overfull hboxes). Bibliography complete105  (59/59 cited keys resolve).106107## 6. Items for your review1081091. **§4 corrections** — especially VIF, Lasso, and PCA, which change the110   robustness narrative (the old text claimed low collinearity and high111   selection concordance; the data say otherwise). Verify you are comfortable112   with the new framing before circulating.1132. `data/raw/louka.db` contains a `users` table with plaintext credentials114   (application leftover, unused by the analysis). Consider stripping it:115   `sqlite3 data/raw/louka.db "DROP TABLE users; VACUUM;"`.1163. The description-length truncation (~700 chars) is now disclosed in the paper;117   if a non-truncated export exists, re-running the pipeline on it would be easy.1184. The quantile-regression standard errors use the statsmodels kernel-based119   estimator (the old table note claimed bootstrap SEs, which was never the case).1205. Original folder `immo-wp2-spb-20260519` was left untouched; delete or archive121   it once you're satisfied with `wp2_uqo`.122