spb/wp9_uqo Public

1# CHANGES — Restructuring report (2026-08-05)23Original project: `~/Desktop/UQO/UQO_WP/immo-wp9-spb-20260529` (**left untouched**).4New clean repository: `~/Desktop/wp9_uqo` (this folder).5No published number, result or scientific claim was changed; the paper's tables and6number-bearing figures are built from the original outputs preserved in7`results/reference/`.89## 1. What was moved / renamed1011| Original | New location |12|---|---|13| `realtor_mls_unique.duckdb` (repo root) | `data/raw/` (byte-identical copy) |14| `analysis/*.csv`, `*.json` (stored outputs) | `results/reference/` (canonical paper numbers) |15| `tables/*.tex` | regenerated into `results/tables/` by script 05 (values verified **identical**) |16| `figures/*.png` (7 survivors of 17) | all 17 regenerated into `figures/` by script 04 |17| `main.tex` + `sections/` + `references.bib` + `uq_logo.jpg` | `paper/` |18| `Makefile`, `.latexmkrc` | `paper/` |19| LaTeX build artifacts (`.aux/.log/.fls/...`), `.DS_Store` | not carried over |2021Values that existed only inside the published tables (regression coefficients,22quantile estimates, summary statistics, model R²) were transcribed into23`results/reference/{coef_M*.csv, quantile.csv, summary_stats.csv, fit.json}` so that24every table/figure is now generated from a data file rather than hard-coded.2526## 2. Code refactoring2728The three live scripts (`estimate_ext.py`, `estimate_ext2.py`, `make_assets.py`,29~25 KB of duplicated single-letter-variable code with absolute machine-specific paths)30were refactored into a package + numbered entry points:3132- `src/wp9/config.py` — all paths (relative to the repo), constants, seeds; the metro33  coordinates and the sqft/acre conversions now live in exactly one place.34- `src/wp9/parsing.py` — documented parsers for every semi-structured raw field.35- `src/wp9/sample.py` — raw DuckDB → estimation sample (was upstream in the lost36  `RE_DB_QC` pipeline; **reconstructed**, see AUDIT.md §5).37- `src/wp9/models.py` — design matrix, M1–M5 ladder, absorbing least squares, FSA38  fixed-effect recovery, Duan smearing. The `design()` function, previously copy-pasted39  four times across scripts, exists once.40- `src/wp9/plotstyle.py` — shared figure style/palette.41- `scripts/01_build_sample.py` → `data/processed/analysis.parquet`42- `scripts/02_estimate_core.py` → ladder fits, coefficient tables, grand-model43  residuals, FSA premia44- `scripts/03_estimate_extended.py` → OOS, robustness, heterogeneity, quantile,45  nonlinearity, gradient, Moran's I, LOPO46- `scripts/04_make_figures.py` / `05_make_tables.py` → all 17 figures / 6 tables, with47  `--results {reference,reproduced}` selecting the numbers tier (default: reference =48  published values).4950Every code file carries the header `Author: Simon-Pierre Boucher — contact@spboucher.ai`.51`analysis/ml_benchmark.py` was dropped (dead code: outputs absent, never cited by the52paper). `main_web.tex` was dropped (web variant whose figure-fallback produced a PDF of53placeholder boxes).5455Pipeline verified end-to-end: raw DuckDB → sample → estimates → figures/tables → PDF.56Regenerated tables are numerically identical to the originals; regenerated57reference-driven figures are visually identical to the surviving originals; the58`reproduced` tier matches the published estimates closely but not exactly (full59side-by-side table in AUDIT.md §5).6061## 3. Paper rewrite (`paper/`)6263- Structure kept as `main.tex` + one file per section; preamble reorganised into64  labelled blocks; duplicate/unused packages removed (`tabularx`, `multirow`, `float`,65  `appendix`, `amsthm` retained only where used); metadata unchanged except66  `Version 1.0 → 1.1`.67- Tables are now `\input` from `../results/tables/` and figures from `../figures/`, so68  recompiling after a pipeline run always uses current artifacts.69- Prose: editorial polish only — tightened sentences, fixed agreement/punctuation,70  consistent notation and hyphenation; all numbers, claims and 32 citations preserved.71- Fixed an internal inconsistency: the note under the quantile table said the size72  elasticity *falls* with price while the table and body text show it *rising*73  (0.557 → 0.598); the note now agrees with the table.74- The maps figure caption no longer promises Natural Earth provincial boundaries (the75  regenerated maps plot listing coordinates with province labels; the boundary shapefile76  dependency was removed).77- Compiles cleanly: `paper/main.pdf`, 26 pages, zero errors/undefined references, all78  17 figures embedded (the old `main_web.pdf` contained none).7980## 4. Items requiring your review81821. **Condominium size-gradient sentence (results §5.2).** The original83   `fig_size_gradient.png` is lost. Raw medians from the data show condominiums listing84   *above* houses of equal size (composition: condos sit in expensive metros), while the85   original text claimed they "lie below houses at every size". I rewrote that one86   sentence to describe the regenerated figure and the (data-supported) conditional87   version of the claim. Please confirm the new wording — or tell me how the original88   figure was constructed and I will match it.892. **Reproduced tier differences** (AUDIT.md §5): sample +0.86%; M5 living-area90   elasticity 0.530 vs 0.547; lot coefficients differ because the original lot parser91   could not be fully recovered; LOPO mean 0.315 vs 0.362. If you still have any copy of92   the `RE_DB_QC/hedonic` pipeline (backup/other machine), I can close these gaps.933. **Moran scatter / fit / residual / nonlinearity-band figures** are computed from the94   reconstructed sample (no stored micro-outputs existed); their annotated Moran's I95   values are taken from the original `ext2.json` while the scatter clouds are96   reproduced data.974. **`ml_benchmark.py` dropped** — recover from the original folder if you want the ML98   comparison back; its outputs would need to be regenerated (requires xgboost,99   lightgbm, torch).1005. The paper still lists `simon-pierre.boucher@uqo.ca` as contact (unchanged); code101   headers use `contact@spboucher.ai` per your instruction.102