# CHANGES — Restructuring report (2026-08-05)

Original project: `~/Desktop/UQO/UQO_WP/immo-wp9-spb-20260529` (**left untouched**).
New clean repository: `~/Desktop/wp9_uqo` (this folder).
No published number, result or scientific claim was changed; the paper's tables and
number-bearing figures are built from the original outputs preserved in
`results/reference/`.

## 1. What was moved / renamed

| Original | New location |
|---|---|
| `realtor_mls_unique.duckdb` (repo root) | `data/raw/` (byte-identical copy) |
| `analysis/*.csv`, `*.json` (stored outputs) | `results/reference/` (canonical paper numbers) |
| `tables/*.tex` | regenerated into `results/tables/` by script 05 (values verified **identical**) |
| `figures/*.png` (7 survivors of 17) | all 17 regenerated into `figures/` by script 04 |
| `main.tex` + `sections/` + `references.bib` + `uq_logo.jpg` | `paper/` |
| `Makefile`, `.latexmkrc` | `paper/` |
| LaTeX build artifacts (`.aux/.log/.fls/...`), `.DS_Store` | not carried over |

Values that existed only inside the published tables (regression coefficients,
quantile estimates, summary statistics, model R²) were transcribed into
`results/reference/{coef_M*.csv, quantile.csv, summary_stats.csv, fit.json}` so that
every table/figure is now generated from a data file rather than hard-coded.

## 2. Code refactoring

The three live scripts (`estimate_ext.py`, `estimate_ext2.py`, `make_assets.py`,
~25 KB of duplicated single-letter-variable code with absolute machine-specific paths)
were refactored into a package + numbered entry points:

- `src/wp9/config.py` — all paths (relative to the repo), constants, seeds; the metro
  coordinates and the sqft/acre conversions now live in exactly one place.
- `src/wp9/parsing.py` — documented parsers for every semi-structured raw field.
- `src/wp9/sample.py` — raw DuckDB → estimation sample (was upstream in the lost
  `RE_DB_QC` pipeline; **reconstructed**, see AUDIT.md §5).
- `src/wp9/models.py` — design matrix, M1–M5 ladder, absorbing least squares, FSA
  fixed-effect recovery, Duan smearing. The `design()` function, previously copy-pasted
  four times across scripts, exists once.
- `src/wp9/plotstyle.py` — shared figure style/palette.
- `scripts/01_build_sample.py` → `data/processed/analysis.parquet`
- `scripts/02_estimate_core.py` → ladder fits, coefficient tables, grand-model
  residuals, FSA premia
- `scripts/03_estimate_extended.py` → OOS, robustness, heterogeneity, quantile,
  nonlinearity, gradient, Moran's I, LOPO
- `scripts/04_make_figures.py` / `05_make_tables.py` → all 17 figures / 6 tables, with
  `--results {reference,reproduced}` selecting the numbers tier (default: reference =
  published values).

Every code file carries the header `Author: Simon-Pierre Boucher — contact@spboucher.ai`.
`analysis/ml_benchmark.py` was dropped (dead code: outputs absent, never cited by the
paper). `main_web.tex` was dropped (web variant whose figure-fallback produced a PDF of
placeholder boxes).

Pipeline verified end-to-end: raw DuckDB → sample → estimates → figures/tables → PDF.
Regenerated tables are numerically identical to the originals; regenerated
reference-driven figures are visually identical to the surviving originals; the
`reproduced` tier matches the published estimates closely but not exactly (full
side-by-side table in AUDIT.md §5).

## 3. Paper rewrite (`paper/`)

- Structure kept as `main.tex` + one file per section; preamble reorganised into
  labelled blocks; duplicate/unused packages removed (`tabularx`, `multirow`, `float`,
  `appendix`, `amsthm` retained only where used); metadata unchanged except
  `Version 1.0 → 1.1`.
- Tables are now `\input` from `../results/tables/` and figures from `../figures/`, so
  recompiling after a pipeline run always uses current artifacts.
- Prose: editorial polish only — tightened sentences, fixed agreement/punctuation,
  consistent notation and hyphenation; all numbers, claims and 32 citations preserved.
- Fixed an internal inconsistency: the note under the quantile table said the size
  elasticity *falls* with price while the table and body text show it *rising*
  (0.557 → 0.598); the note now agrees with the table.
- The maps figure caption no longer promises Natural Earth provincial boundaries (the
  regenerated maps plot listing coordinates with province labels; the boundary shapefile
  dependency was removed).
- Compiles cleanly: `paper/main.pdf`, 26 pages, zero errors/undefined references, all
  17 figures embedded (the old `main_web.pdf` contained none).

## 4. Items requiring your review

1. **Condominium size-gradient sentence (results §5.2).** The original
   `fig_size_gradient.png` is lost. Raw medians from the data show condominiums listing
   *above* houses of equal size (composition: condos sit in expensive metros), while the
   original text claimed they "lie below houses at every size". I rewrote that one
   sentence to describe the regenerated figure and the (data-supported) conditional
   version of the claim. Please confirm the new wording — or tell me how the original
   figure was constructed and I will match it.
2. **Reproduced tier differences** (AUDIT.md §5): sample +0.86%; M5 living-area
   elasticity 0.530 vs 0.547; lot coefficients differ because the original lot parser
   could not be fully recovered; LOPO mean 0.315 vs 0.362. If you still have any copy of
   the `RE_DB_QC/hedonic` pipeline (backup/other machine), I can close these gaps.
3. **Moran scatter / fit / residual / nonlinearity-band figures** are computed from the
   reconstructed sample (no stored micro-outputs existed); their annotated Moran's I
   values are taken from the original `ext2.json` while the scatter clouds are
   reproduced data.
4. **`ml_benchmark.py` dropped** — recover from the original folder if you want the ML
   comparison back; its outputs would need to be regenerated (requires xgboost,
   lightgbm, torch).
5. The paper still lists `simon-pierre.boucher@uqo.ca` as contact (unchanged); code
   headers use `contact@spboucher.ai` per your instruction.