spb/wp2_uqo Public

1# AUDIT — immo-wp2-spb-2026051923Audit of the original project at `~/Desktop/UQO/UQO_WP/immo-wp2-spb-20260519`4(354 MB), performed 2026-08-05 before restructuring into `~/Desktop/wp2_uqo`.5The original folder is left untouched and serves as the backup.67## 1. What the project is89Working Paper No. 2 (UQO): *"Decoding Real Estate Descriptions: Semantic10Embeddings and Hedonic Pricing of Residential Properties in Quebec"*11(Simon-Pierre Boucher). The analysis embeds the free-text `PublicRemarks` of1217,087 Quebec single-family house listings with `all-MiniLM-L6-v2`13(sentence-transformers), computes cosine similarities to 20 researcher-defined14French reference descriptions ("luxury", "needs renovation", …), and adds15those 20 similarity features to a log-price hedonic OLS model (models A–E,16HC3 errors), plus robustness checks (VIF, Breusch–Pagan, quantile regression,17winsorization, 1,000-rep bootstrap, Lasso/Elastic Net).1819Headline numbers (paper, model D vs A): adjusted R² 0.452 → 0.511,20F = 99.53 (p < 0.001), n = 17,087.2122## 2. Data2324| File | Size | Role |25|---|---|---|26| `louka.db` | 268 MB | **Raw data.** SQLite; table `properties` (id, mls_number, category, price_value, bedrooms, bathrooms, data JSON blob). Counts: house 17,089, condo 8,586, land 8,506, rent 8,240, plex 4,058. Sample = houses with price > 0 and remarks ≥ 20 chars → 17,087. |27| `hedonic_maison_results.csv` | 17,087 rows | Processed: houses + 20 similarity features (main analysis dataset). |28| `hedonic_results.csv` | 29,731 rows | Processed (earlier PCA approach, house/condo/plex). |29| `hedonic_cosine_results.csv` | 29,731 rows | Processed (earlier 16-reference approach, all categories). |30| `embeddings_maisons.npy` | 25 MB | 17,087 × 384 embeddings (houses; used by the paper). |31| `embeddings_remarks.npy` | 44 MB | 29,731 × 384 embeddings (all categories; earlier approach). |32| `sim_matrix_maisons.npy` | 1.3 MB | 17,087 × 20 similarity matrix (paper). |33| `similarity_matrix.npy` | 1.8 MB | 29,731 × 16 (earlier approach). |34| `reference_embeddings.npz` | 43 KB | 16 reference embeddings (earlier approach). |3536⚠️ Notes:37- `louka.db` also contains a `users` table with **plaintext usernames/passwords**38  (application leftover). It is not used by any script. Not copied onward.39- All scripts hardcode `DB_PATH = "/Users/simon-pierreboucher/Desktop/d/louka.db"`40  and write outputs to `~/Desktop/d/` — a folder that **no longer exists**. The41  project copy of `louka.db` is what remains; outputs were copied into the42  project at some point.43- `robustness_results.csv` (written by `hedonic_maison.py` step 12) was saved44  to `~/Desktop/d/` and is **not present anywhere** — regenerated in the new repo.4546## 3. Scripts (chronological / by role)4748| Script | Role | Outputs | Status |49|---|---|---|---|50| `hedonic_semantic.py` (348 l.) | Exploration #1: PCA(20) of embeddings, house/condo/plex, models 1–4 | `hedonic_results.csv`, `embeddings_remarks.npy` | Superseded; not used by the paper |51| `hedonic_cosine_references.py` (447 l.) | Exploration #2: 16 reference similarities, all categories, models A–C | `hedonic_cosine_results.csv`, `reference_embeddings.npz`, `similarity_matrix.npy` | Superseded; not used by the paper |52| `hedonic_maison.py` (818 l.) | **Main analysis**: 20 references, houses only, models A–E + all robustness checks | `hedonic_maison_results.csv`, `embeddings_maisons.npy`, `sim_matrix_maisons.npy`, `robustness_results.csv` (lost) | **Paper source** (tables 1–8) |53| `paper_figures.py` (578 l.) | Figures v1 (figs 1–8) + `table1–4_*.csv` | `figures/` | Superseded by v2 for figures; still the only producer of the 4 table CSVs |54| `paper_figures_v2.py` (386 l.) | **Figures v2 (final)**: figs 1–11, tighter layout | `figures/fig1…fig11` | **Paper source** (all figures) |5556Duplication: the DB-extraction block (~70 lines) and the OLS-fit helper are57copy-pasted in all 5 scripts, with small drifts (`half_baths` and lat/lon only58in some; 16 vs 20 references; French vs English reference names). The 2059reference descriptions are pasted 3× (French keys in `hedonic_maison.py`,60English display keys in both figure scripts — same French texts).6162## 4. Figures6364`figures/` and `wp2/figures/` (identical copies) contain fig1, fig4–fig1165(PDF, some PNG) + `table1–4_*.csv`.6667**Missing: `fig2_coefficient_plot.pdf` and `fig3_similarity_distributions.pdf`**,68although both papers reference them (`sections/results.tex`, `sections/data.tex`).69`wp2/main_web.tex` silently masks this with a custom `\includegraphics`70fallback that prints a "Figure indisponible" box — the compiled71`wp2/main_web.pdf` therefore ships with two placeholder boxes instead of72Figures 2 (semantic coefficient forest plot) and 3 (similarity distributions).73Both are regenerated by `paper_figures_v2.py`. **This is the main defect fixed74by the restructuring.**7576Figure → script map (all from `paper_figures_v2.py`):77fig1 model comparison; fig2 semantic coefficient forest plot (was missing);78fig3 similarity boxplots (was missing); fig4 price-quintile heatmap;79fig5 similarity correlation matrix; fig6 scatter plots (4 dims);80fig7 methodology diagram; fig8 R² decomposition; fig9 price distributions;81fig10 structural coefficients; fig11 residual diagnostics.8283## 5. LaTeX8485Two generations coexist:8687- **Root (obsolete, May 20)**: monolithic `paper.tex` (112 KB, first draft),88  then `main.tex` + `preamble.tex` + `titlepage.tex` + `sections/` + `tables/`89  + `appendix/` + `references.bib`.90- **`wp2/` (canonical, May 20 – June 13)**: `main.tex` (self-contained91  preamble, UQO WP style), `main_web.tex` (same + figure-fallback hack, source92  of the compiled `main_web.pdf`), `sections/` (9 files incl. titlepage),93  `tables/` (8 hand-written booktabs tables), `appendix/appendix.tex`,94  `references.bib` (551 lines, ~40 entries), `Makefile`, `uq_logo.jpg`.9596`diff -r` confirms root `sections/`, `tables/`, `appendix/`, `references.bib`97are **byte-identical** to the `wp2/` ones (root just lacks `titlepage.tex`).98So the root LaTeX tree is a pure duplicate → dropped.99100Numbers in `tables/*.tex` were hand-transcribed from `hedonic_maison.py`101console output (no automated table generation).102103## 6. Dead / duplicate / unused files104105- `paper.tex`, root `main.tex`, `preamble.tex`, `titlepage.tex`, root106  `sections/`, `tables/`, `appendix/`, root `figures/` — superseded duplicates.107- Build artifacts: `*.aux, *.log, *.out, *.toc, *.bbl, *.blg, *.fls,108  *.fdb_latexmk, *.synctex.gz` at root and in `wp2/`.109- `.DS_Store` files.110- Earlier-approach outputs (`hedonic_results.csv`, `hedonic_cosine_results.csv`,111  `embeddings_remarks.npy`, `similarity_matrix.npy`, `reference_embeddings.npz`)112  are kept in `data/processed/legacy/` for traceability but are not part of the113  pipeline.114115## 7. Discrepancies & reproduction status116117- **fig2/fig3 missing** in the original (see §4) — regenerated here.118- **`robustness_results.csv` lost** in the original — regenerated here.119- Reproduction check (new pipeline vs original artifacts) — results recorded120  in `CHANGES.md` §Verification:121  - similarity features vs `hedonic_maison_results.csv` (17,087 × 20),122  - model A–E statistics vs `tables/tab_model_comparison.tex`123    (R² 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118),124  - regenerated figures vs shipped fig1, fig4–fig11.125- Bootstrap (`np.random.seed(42)`) and Lasso/ENet CV (`random_state=42`) are126  seeded, so robustness numbers are reproducible on the same package versions.127