| 1 |
|
−<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai --> |
| 2 |
|
− |
| 3 |
|
−<div align="center"> |
| 4 |
|
− |
|
1 |
+<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca --> |
| 5 |
2 |
# 🏠 Decoding Real Estate Descriptions |
| 6 |
3 |
|
| 7 |
|
−### Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec |
| 8 |
|
− |
| 9 |
|
−**UQO Working Paper No. 2** |
| 10 |
|
− |
| 11 |
|
−[](mailto:contact@spboucher.ai) |
| 12 |
|
−[](mailto:contact@spboucher.ai) |
| 13 |
|
−[](https://uqo.ca) |
| 14 |
|
− |
| 15 |
|
−[](https://www.python.org/) |
| 16 |
|
−[](https://www.latex-project.org/) |
| 17 |
|
−[](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) |
| 18 |
|
−[](https://www.statsmodels.org/) |
| 19 |
|
−[](https://scikit-learn.org/) |
|
4 |
+**UQO Working Paper No. 2** — *Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec* |
| 20 |
5 |
|
| 21 |
|
−[](#-reproducing-everything) |
| 22 |
|
−[](#-data) |
| 23 |
|
−[](#-key-results) |
| 24 |
|
−[](paper/main.pdf) |
| 25 |
|
−[-00695C?style=flat-square&logo=googlescholar&logoColor=white)](paper/references.bib) |
| 26 |
|
−[](#-citation) |
|
6 |
+[-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) |
|
7 |
+[](requirements.txt) |
|
8 |
+[](data/README.md) |
|
9 |
+[](src/) |
|
10 |
+[](scripts/) |
|
11 |
+[](https://uqo.ca) |
|
12 |
+[](mailto:simon-pierre.boucher@uqo.ca) |
| 27 |
13 |
|
| 28 |
|
−</div> |
|
14 |
+> **TL;DR** — Listing descriptions written by real-estate agents contain price-relevant information that classic hedonic |
|
15 |
+> variables miss. The free-text `PublicRemarks` of **17,087 Quebec single-family house listings** are embedded with a |
|
16 |
+> sentence transformer, projected onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs |
|
17 |
+> Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and the 20 interpretable scores are added to a |
|
18 |
+> log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001); each semantic |
|
19 |
+> dimension carries a named, signed, economically meaningful implicit price (Modern/Contemporary +16.4 %, Luxury +14.2 %, |
|
20 |
+> Needs Renovation −7.9 %, Motivated Seller −8.5 % per SD). |
| 29 |
21 |
|
| 30 |
22 |
--- |
| 31 |
23 |
|
| 32 |
|
−> **TL;DR** — Listing descriptions written by real estate agents contain price-relevant information that classic hedonic variables miss. This project embeds the free-text `PublicRemarks` of **17,087 Quebec single-family house listings** with a sentence transformer, projects each embedding onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and adds those 20 interpretable scores to a log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001), and each semantic dimension carries a named, signed, economically meaningful implicit price. |
|
24 |
+## 📌 Status |
| 33 |
25 |
|
| 34 |
|
−--- |
| 35 |
|
− |
| 36 |
|
−## 📖 Table of Contents |
| 37 |
|
− |
| 38 |
|
−- [Overview](#-overview) |
| 39 |
|
−- [Key Results](#-key-results) |
| 40 |
|
−- [Methodology](#-methodology) |
| 41 |
|
−- [Repository Structure](#-repository-structure) |
| 42 |
|
−- [The Pipeline, Script by Script](#-the-pipeline-script-by-script) |
| 43 |
|
−- [Data](#-data) |
| 44 |
|
−- [Reproducing Everything](#-reproducing-everything) |
| 45 |
|
−- [Building the Paper](#-building-the-paper) |
| 46 |
|
−- [Verification & Provenance](#-verification--provenance) |
| 47 |
|
−- [Figures Gallery](#-figures-gallery) |
| 48 |
|
−- [Requirements](#-requirements) |
| 49 |
|
−- [Citation](#-citation) |
| 50 |
|
−- [Author & Contact](#-author--contact) |
| 51 |
|
− |
| 52 |
|
−--- |
| 53 |
|
− |
| 54 |
|
−## 🔍 Overview |
|
26 |
+| Item | Value | |
|
27 |
+|---|---| |
|
28 |
+| Version | 1.1 (May 2026) — cover page harmonised to the UQO WP standard, October 2026 | |
|
29 |
+| SSRN | Abstract ID **7626640** (submitted 2026-10-17, preliminary upload) | |
|
30 |
+| Journal | not yet submitted | |
|
31 |
+| Code & data | fully reproducible from `scripts/`; raw database not redistributed (see Data) | |
| 55 |
32 |
|
| 56 |
|
−Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, bathrooms, lot size. But the *narrative* of a listing ("cuisine gastronomique, comptoirs de quartz…" vs. "besoin de rénovation, vendu tel quel…") carries quality information that no structured field captures. |
|
33 |
+## 🎯 What this paper does |
| 57 |
34 |
|
| 58 |
|
−The catch, historically, is a **depth-vs-interpretability trade-off**: |
|
35 |
+Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, |
|
36 |
+bathrooms, lot size. But the *narrative* of a listing («cuisine gastronomique, comptoirs de quartz…» vs. «besoin de |
|
37 |
+rénovation, vendu tel quel…») carries quality information that no structured field captures. Historically there has been |
|
38 |
+a depth-vs-interpretability trade-off: |
| 59 |
39 |
|
| 60 |
40 |
| Approach | Semantic depth | Economic interpretability | |
| 61 |
41 |
|---|---|---| |
| 62 |
42 |
| Keyword counts / bag-of-words | ❌ shallow | ✅ high | |
| 63 |
43 |
| Sentiment scores | ❌ one dimension | ✅ high | |
| 64 |
44 |
| LDA topics | 🟡 moderate | ❌ unstable | |
| 65 |
|
−| Raw BERT/transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) | |
|
45 |
+| Raw transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) | |
| 66 |
46 |
| **Reference-based cosine projection (this paper)** | ✅ **deep** | ✅ **high** | |
| 67 |
47 |
|
| 68 |
|
−**The idea:** instead of feeding 384 anonymous embedding dimensions into a regression, define 20 short *reference descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embed them with the same model, and use the **cosine similarity between each listing and each reference** as 20 named regressors. Every coefficient then reads directly as "the implicit price of sounding more *luxury* / more *fixer-upper* / more *waterfront*", per standard deviation. |
| 69 |
|
− |
| 70 |
|
−This is conceptually analogous to factor-mimicking portfolios in asset pricing, and to concept-bottleneck models in interpretable ML: the deep representation is channeled through human-named concepts before inference. |
| 71 |
|
− |
| 72 |
|
−--- |
|
48 |
+Instead of feeding 384 anonymous embedding dimensions into a regression, the paper defines 20 short *reference |
|
49 |
+descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embeds them with the same |
|
50 |
+model, and uses the **cosine similarity between each listing and each reference** as 20 named regressors: |
| 73 |
51 |
|
| 74 |
|
−## 📊 Key Results |
|
52 |
+``` |
|
53 |
+log(P_i) = α + β'X_i + δ·len_i + γ'S_i + ε_i S_i = E_i · Rᵀ (17,087 × 20) |
|
54 |
+``` |
| 75 |
55 |
|
| 76 |
|
−### Model comparison (n = 17,087) |
|
56 |
+Every coefficient then reads directly as «the implicit price of sounding more *luxury* / more *fixer-upper* / more |
|
57 |
+*waterfront*», per standard deviation — analogous to factor-mimicking portfolios in asset pricing and to concept-bottleneck |
|
58 |
+models in interpretable ML. |
| 77 |
59 |
|
| 78 |
|
−| Model | Specification | R² | Adj. R² | AIC | BIC | k | |
| 79 |
|
−|:---:|---|---:|---:|---:|---:|---:| |
| 80 |
|
−| A | Structural only | 0.4525 | 0.4523 | 26,165 | 26,219 | 6 | |
| 81 |
|
−| B | A + description length | 0.4647 | 0.4645 | 25,779 | 25,841 | 7 | |
| 82 |
|
−| C | A + 20 semantic similarities | 0.4962 | 0.4954 | 24,783 | 24,992 | 26 | |
| 83 |
|
−| **D** | **Full (B + C)** | **0.5122** | **0.5115** | **24,232** | 24,448 | 27 | |
| 84 |
|
−| E | Parsimonious (16 significant dims) | 0.5118 | 0.5111 | 24,239 | **24,425** | 23 | |
|
60 |
+## 🏆 Headline results |
| 85 |
61 |
|
| 86 |
|
−Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: structural 88.3 %, text length 2.4 %, semantics 9.3 %. |
|
62 |
+| Model (n = 17,087) | Specification | Adj. R² | AIC | k | |
|
63 |
+|:---:|---|---:|---:|---:| |
|
64 |
+| A | Structural only | 0.4523 | 26,165 | 6 | |
|
65 |
+| B | A + description length | 0.4645 | 25,779 | 7 | |
|
66 |
+| C | A + 20 semantic similarities | 0.4954 | 24,783 | 26 | |
|
67 |
+| **D** | **Full (B + C)** | **0.5115** | **24,232** | 27 | |
|
68 |
+| E | Parsimonious (16 significant dims) | 0.5111 | 24,239 | 23 | |
| 87 |
69 |
|
| 88 |
|
−### Implicit semantic price gradients (Model D, per +1 SD, HC3 errors) |
|
70 |
+Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: |
|
71 |
+structural 88.3 %, text length 2.4 %, semantics 9.3 %. |
| 89 |
72 |
|
| 90 |
|
−| 📈 Positive | Impact | 📉 Negative | Impact | |
|
73 |
+| 📈 Positive (per +1 SD, HC3) | Impact | 📉 Negative | Impact | |
| 91 |
74 |
|---|---:|---|---:| |
| 92 |
75 |
| Modern/Contemporary | **+16.4 %*** | Family-Friendly | **−11.6 %*** | |
| 93 |
76 |
| Luxury | **+14.2 %*** | New Construction | **−10.5 %*** | |
| 96 |
79 |
| Waterfront | +4.7 %*** | Motivated Seller | **−8.5 %*** | |
| 97 |
80 |
| Panoramic View | +3.0 %** | Needs Renovation | **−7.9 %*** | |
| 98 |
81 |
|
| 99 |
|
−<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (see paper §6). Impact = (e^β − 1) × 100 %.</sub> |
|
82 |
+<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (paper §6). Impact = (e^β − 1) × 100 %.</sub> |
| 100 |
83 |
|
| 101 |
|
−### Heterogeneity & robustness (highlights) |
|
84 |
+- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from 9.7 % to 17.3 %; urgency |
|
85 |
+ and condition discounts attenuate at the top of the market. |
|
86 |
+- **Bootstrap** (1,000 replications): SEs within 9 % of HC3. **Trimming** (1st/99th percentiles): no sign reversal. |
|
87 |
+- **Multicollinearity disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2); block inference unaffected. |
|
88 |
+- **PCA benchmark**: 20 principal components fit better (ΔR² +0.070 vs. +0.044) but are economically unreadable. |
| 102 |
89 |
|
| 103 |
|
−- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from **9.7 %** at the 25th percentile to **17.3 %** at the 75th; urgency and condition discounts attenuate at the top of the market. |
| 104 |
|
−- **Bootstrap** (1,000 replications, seed 42): SEs within 9 % of HC3 (15/20 within 5 %). |
| 105 |
|
−- **Outlier trimming** (1st/99th price percentiles): no sign reversal, all 16 significant dimensions stay significant. |
| 106 |
|
−- **Multicollinearity is real and disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2). Block-level inference is unaffected; individual coefficients are partial associations within a correlated block. |
| 107 |
|
−- **PCA benchmark**: 20 principal components of the raw embeddings fit *better* (ΔR² +0.070 vs. +0.044) but are economically unreadable — the paper quantifies the interpretability-vs-fit trade-off explicitly. |
| 108 |
|
− |
| 109 |
|
−--- |
| 110 |
|
− |
| 111 |
|
−## 🧪 Methodology |
| 112 |
|
− |
| 113 |
|
−``` |
| 114 |
|
−┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ |
| 115 |
|
−│ Property │ │ Sentence │ │ Cosine │ │ Hedonic │ │ Implicit │ |
| 116 |
|
−│ Listings ├──▶│ Embeddings ├──▶│ Similarity ├──▶│ OLS Model ├──▶│ Price │ |
| 117 |
|
−│ (n=17,087) │ │ (384-d) │ │ (20 refs) │ │ (HC3 s.e.) │ │ Estimates │ |
| 118 |
|
−└─────────────┘ └──────────────┘ └──────────────┘ └─────────────┘ └──────────────┘ |
| 119 |
|
− PublicRemarks all-MiniLM-L6-v2 S = E · Rᵀ log(P) = α + β'X + δℓ + γ's + ε |
| 120 |
|
−``` |
| 121 |
|
− |
| 122 |
|
−1. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (the transformer handles raw text). |
| 123 |
|
−2. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalized), batch 256; the whole corpus encodes in seconds on Apple Silicon. |
| 124 |
|
−3. **Projection** — 20 French reference descriptions (146–257 chars each, full verbatim text in paper Appendix A), designed for *semantic saturation*, *dimensional specificity*, *linguistic consistency*. One matrix product yields the 17,087 × 20 similarity matrix. |
| 125 |
|
−4. **Estimation** — nested OLS models A–E on log price, all covariates standardized, HC3 robust errors; joint F-tests; quantile regressions; bootstrap; winsorization; VIF; Lasso/Elastic-Net selection; PCA benchmark. |
| 126 |
|
− |
| 127 |
|
−The 20 dimensions span six domains: **Quality & Standing** (Luxury, Entry-Level) · **Condition** (Renovated, Needs Renovation) · **Physical** (Bright & Spacious, Garage & Parking, Finished Basement, Pool & Landscaping) · **Location & Setting** (Land & Nature, Panoramic View, Premium Location, Quiet & Peaceful, Waterfront) · **Style & Character** (Modern/Contemporary, Heritage/Character, New Construction, Energy Efficient, Family-Friendly) · **Market Signals** (Income/Investment, Motivated Seller). |
| 128 |
|
− |
| 129 |
|
−--- |
| 130 |
|
− |
| 131 |
|
−## 📁 Repository Structure |
|
90 |
+## 🗂 Repository layout |
| 132 |
91 |
|
| 133 |
92 |
``` |
| 134 |
93 |
wp2_uqo/ |
| 135 |
|
−├── README.md ← you are here |
| 136 |
|
−├── AUDIT.md ← forensic audit of the original (pre-restructuring) project |
| 137 |
|
−├── CHANGES.md ← everything that was moved / refactored / corrected, with diffs |
| 138 |
|
−├── PAPER_REVIEW.md ← critical pre-upgrade assessment of the paper's scholarship |
| 139 |
|
−├── UPGRADE_REPORT.md ← literature expansion: refs added/replaced, claims flagged |
| 140 |
|
−├── requirements.txt ← pinned dependencies (exact versions used for verification) |
| 141 |
|
−│ |
| 142 |
|
−├── data/ ← NOT tracked by git (see data/README.md) |
| 143 |
|
−│ ├── raw/louka.db │ SQLite, 268 MB, 46,479 listings (5 categories) |
| 144 |
|
−│ └── processed/ │ parquet sample, embeddings cache, similarity matrix, |
| 145 |
|
−│ └── legacy/ │ analysis CSV + pre-refactor artifacts kept for comparison |
| 146 |
|
−│ |
| 147 |
|
−├── src/ ← analysis package (imported by scripts/) |
| 148 |
|
−│ ├── config.py │ repo-relative paths, model name, seed, structural vars |
| 149 |
|
−│ ├── references.py │ the 20 French reference descriptions + English labels |
| 150 |
|
−│ ├── data.py │ SQLite → DataFrame extraction (n = 17,087) |
| 151 |
|
−│ ├── embeddings.py │ encoding, caching, cosine-similarity features |
| 152 |
|
−│ └── models.py │ standardized-design OLS A–E, HC3 |
| 153 |
|
−│ |
| 154 |
|
−├── scripts/ ← numbered pipeline entry points (run in order) |
| 155 |
|
−│ ├── 01_prepare_data.py │ louka.db → data/processed/houses.parquet |
| 156 |
|
−│ ├── 02_similarities.py │ embeddings (cached) + 20 similarity features → CSV |
| 157 |
|
−│ ├── 03_models.py │ models A–E → results/*.csv |
| 158 |
|
−│ ├── 04_robustness.py │ VIF · BP · quantile · winsor · bootstrap · Lasso/ENet |
| 159 |
|
−│ ├── 05_figures.py │ figures fig1–fig11 (PDF + PNG) |
| 160 |
|
−│ └── 06_tables.py │ LaTeX tables generated from results (no hand transcription) |
| 161 |
|
−│ |
| 162 |
|
−├── figures/ ← all 11 paper figures, regenerated from the pipeline |
| 163 |
|
−├── results/ ← model & robustness outputs (CSV, versioned) |
| 164 |
|
−│ |
| 165 |
|
−└── paper/ ← LaTeX source (UQO working-paper style) |
| 166 |
|
− ├── main.tex │ metadata + section inputs (53-page PDF) |
| 167 |
|
− ├── preamble.tex │ packages, layout, custom macros |
| 168 |
|
− ├── main.pdf │ compiled paper (committed for convenience) |
| 169 |
|
− ├── Makefile │ latexmk build |
| 170 |
|
− ├── references.bib │ 86 entries, every one verified (OpenAlex + DOI) |
| 171 |
|
− ├── sections/ │ titlepage · intro · literature · methodology · data · |
| 172 |
|
− │ │ results · robustness · discussion · conclusion · appendix |
| 173 |
|
− └── tables/ │ 7 machine-generated + 3 hand-maintained booktabs tables |
|
94 |
+├── README.md |
|
95 |
+├── requirements.txt # pinned dependencies (Python 3.14, macOS arm64) |
|
96 |
+├── docs/ # AUDIT.md · CHANGES.md · PAPER_REVIEW.md · UPGRADE_REPORT.md (restructuring history) |
|
97 |
+├── data/ # NOT tracked (see data/README.md) |
|
98 |
+│ ├── raw/louka.db # SQLite, 268 MB, 46,479 listings |
|
99 |
+│ └── processed/ # parquet sample, embeddings cache, similarity matrix |
|
100 |
+├── src/ # analysis package |
|
101 |
+│ ├── config.py # repo-relative paths, model name, seed, structural vars |
|
102 |
+│ ├── references.py # the 20 French reference descriptions + English labels |
|
103 |
+│ ├── data.py # SQLite → DataFrame extraction (n = 17,087) |
|
104 |
+│ ├── embeddings.py # encoding, caching, cosine-similarity features |
|
105 |
+│ └── models.py # standardized-design OLS A–E, HC3 |
|
106 |
+├── scripts/ # the pipeline, in order |
|
107 |
+│ ├── 01_prepare_data.py # louka.db → data/processed/houses.parquet |
|
108 |
+│ ├── 02_similarities.py # embeddings (cached) + 20 similarity features |
|
109 |
+│ ├── 03_models.py # models A–E → results/*.csv |
|
110 |
+│ ├── 04_robustness.py # VIF · BP · quantile · winsor · bootstrap · Lasso/ENet |
|
111 |
+│ ├── 05_figures.py # fig1–fig11 (PDF + PNG) |
|
112 |
+│ └── 06_tables.py # LaTeX tables generated from results |
|
113 |
+├── results/ # model & robustness outputs (CSV) |
|
114 |
+├── figures/ # 11 paper figures |
|
115 |
+└── paper/ # main.tex + preamble.tex + sections/ + tables/ + references.bib → main.pdf |
| 174 |
116 |
``` |
| 175 |
117 |
|
| 176 |
|
−--- |
| 177 |
|
− |
| 178 |
|
−## ⚙️ The Pipeline, Script by Script |
| 179 |
|
− |
| 180 |
|
−| # | Script | Input | Output | Runtime* | |
| 181 |
|
−|---|---|---|---|---| |
| 182 |
|
−| 1 | `01_prepare_data.py` | `data/raw/louka.db` | `houses.parquet` (17,087 × 14) | ~15 s | |
| 183 |
|
−| 2 | `02_similarities.py` | parquet + refs | embeddings `.npy` (cached), `sim_matrix.npy`, analysis CSV | ~5 s cached / ~3 min fresh | |
| 184 |
|
−| 3 | `03_models.py` | analysis CSV | `model_comparison.csv`, coefficient tables, descriptive stats | ~5 s | |
| 185 |
|
−| 4 | `04_robustness.py` | analysis CSV | `robustness_results.csv` (163 rows: VIF, BP, QR, winsor, bootstrap, Lasso) | ~4 min (bootstrap) | |
| 186 |
|
−| 5 | `05_figures.py` | analysis CSV | `figures/fig1…fig11.{pdf,png}` | ~30 s | |
| 187 |
|
−| 6 | `06_tables.py` | `results/*.csv` | `paper/tables/*.tex` (7 tables) | ~2 s | |
| 188 |
|
− |
| 189 |
|
−<sub>*Apple M-series laptop. All stochastic steps are seeded (`seed = 42`): bootstrap resampling, CV folds, scatter subsampling.</sub> |
| 190 |
|
− |
| 191 |
|
−**Design principles applied during the refactor** (see `CHANGES.md` for the full story): |
| 192 |
|
− |
| 193 |
|
−- 🚫 **No hardcoded paths** — everything resolves from the repo root (`src/config.py`); the DB location can be overridden with `WP2_DB_PATH`. |
| 194 |
|
−- 🧬 **Single source of truth** — the extraction logic and the 20 reference texts, previously copy-pasted across five scripts (in two languages), now live in exactly one place each. |
| 195 |
|
−- 🔁 **Deterministic & cached** — embeddings are deterministic for a given model version and cached; `--force` re-encodes from scratch. |
| 196 |
|
−- 📋 **Tables are compiled artifacts** — every number in the paper's statistical tables is written by `06_tables.py` from the results CSVs. The original project transcribed numbers by hand, which introduced errors that this restructuring caught and fixed (documented, with old→new values, in `CHANGES.md` §4). |
| 197 |
|
− |
| 198 |
|
−--- |
| 199 |
|
− |
| 200 |
|
−## 💾 Data |
| 201 |
|
− |
| 202 |
|
−The raw data is a **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: **46,479 listings** across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). Each row carries structured fields (price, bedrooms, bathrooms) plus a full JSON blob including the agent-written `PublicRemarks`. |
| 203 |
|
− |
| 204 |
|
−**The analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. Median listing price $589,900 (mean $807,259, max $25M). Descriptions average 510 characters and are truncated at ~700 characters in the export (disclosed in the paper). |
| 205 |
|
− |
| 206 |
|
−> ⚠️ **`data/` is not tracked by git**: the raw database exceeds GitHub's 100 MB file limit, contains a leftover application `users` table, and the listing texts are proprietary platform content. `data/README.md` explains how to place a copy of `louka.db` to re-run the pipeline from scratch. All *derived, aggregate* outputs (results CSVs, figures, tables, paper) **are** versioned, so every number in the paper is inspectable without the raw data. |
| 207 |
|
− |
| 208 |
|
−--- |
| 209 |
|
− |
| 210 |
|
−## 🔄 Reproducing Everything |
|
118 |
+## 🚀 Quick start |
| 211 |
119 |
|
| 212 |
120 |
```bash |
| 213 |
|
−git clone https://github.com/spboucher-ai/wp2_uqo.git |
| 214 |
|
−cd wp2_uqo |
| 215 |
121 |
python3 -m pip install -r requirements.txt |
| 216 |
122 |
|
| 217 |
123 |
# place louka.db in data/raw/ (see data/README.md), then: |
| 218 |
|
−python3 scripts/01_prepare_data.py # extract the house sample |
|
124 |
+python3 scripts/01_prepare_data.py # 17,087 houses with a valid price and description |
| 219 |
125 |
python3 scripts/02_similarities.py # embed + project (add --force to re-encode) |
| 220 |
126 |
python3 scripts/03_models.py # hedonic models A–E |
| 221 |
|
−python3 scripts/04_robustness.py # full robustness battery (~4 min) |
| 222 |
|
−python3 scripts/05_figures.py # all 11 figures |
|
127 |
+python3 scripts/04_robustness.py # robustness battery (~4 min, bootstrap) |
|
128 |
+python3 scripts/05_figures.py # 11 figures |
| 223 |
129 |
python3 scripts/06_tables.py # LaTeX tables from results |
| 224 |
130 |
|
| 225 |
|
−cd paper && latexmk -pdf main.tex # 53-page PDF, 0 errors |
|
131 |
+cd paper && make # latexmk → main.pdf (53 pages) |
| 226 |
132 |
``` |
| 227 |
133 |
|
| 228 |
|
−Expected checkpoints along the way: |
|
134 |
+Total runtime ≈ 8 minutes on an Apple-silicon laptop (embeddings cached after the first run). All stochastic steps are |
|
135 |
+seeded (`seed = 42`). Expected checkpoints: step 3 prints `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`. |
| 229 |
136 |
|
| 230 |
|
−- Step 1 prints `17,087 houses with a valid price and description`. |
| 231 |
|
−- Step 3 prints the model table with `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`. |
| 232 |
|
−- Step 4 prints `Winsorized R2=0.4867` and the Lasso selection (4/20 at the CV penalty). |
|
137 |
+## 🗃 Data |
| 233 |
138 |
|
| 234 |
|
−--- |
| 235 |
|
− |
| 236 |
|
−## 📝 Building the Paper |
|
139 |
+A **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: |
|
140 |
+46,479 listings across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). |
|
141 |
+Each row carries structured fields plus the agent-written `PublicRemarks`. |
| 237 |
142 |
|
| 238 |
|
−The paper (`paper/`) follows the UQO working-paper format: title page with logo, abstract page with keywords + JEL codes, IMRaD body (Introduction · Literature · Methodology · Data · Results · Robustness · Discussion · Conclusion), `natbib`/`apalike` bibliography, and two appendices (verbatim reference texts; parsimonious-model estimates). |
|
143 |
+**Analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. |
|
144 |
+Median listing price $589,900 (mean $807,259). Descriptions average 510 characters, truncated at ~700 characters in the |
|
145 |
+export (disclosed in the paper). |
| 239 |
146 |
|
| 240 |
|
−```bash |
| 241 |
|
−cd paper |
| 242 |
|
−make # or: latexmk -pdf main.tex |
| 243 |
|
−``` |
|
147 |
+`data/` is not tracked: the database exceeds 100 MB and listing texts are proprietary platform content. All derived, |
|
148 |
+aggregate outputs (results CSVs, figures, tables, paper) are versioned, so every number in the paper is inspectable |
|
149 |
+without the raw data. `data/README.md` explains how to place a copy of `louka.db`; the location can be overridden with |
|
150 |
+`WP2_DB_PATH`. |
| 244 |
151 |
|
| 245 |
|
−Figures are pulled from `../figures/` via `\graphicspath`, so the paper always reflects the latest pipeline run. The build is clean: **0 errors, 0 undefined references, 0 missing citations** (59/59 bib keys resolve). |
|
152 |
+## 🔬 Methodology |
| 246 |
153 |
|
| 247 |
|
−--- |
|
154 |
+1. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (`src/data.py`). |
|
155 |
+2. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalised), batch 256 (`src/embeddings.py`). |
|
156 |
+3. **Projection** — 20 French reference descriptions (146–257 chars, verbatim in paper Appendix A), designed for semantic |
|
157 |
+ saturation, dimensional specificity and linguistic consistency; one matrix product yields the 17,087 × 20 similarity |
|
158 |
+ matrix (`src/references.py`). |
|
159 |
+4. **Estimation** — nested OLS models A–E on log price, standardised covariates, HC3 errors; joint F-tests; quantile |
|
160 |
+ regressions; bootstrap; winsorisation; VIF; Lasso/Elastic-Net selection; PCA benchmark (`src/models.py`, |
|
161 |
+ `scripts/04_robustness.py`). |
| 248 |
162 |
|
| 249 |
|
−## ✅ Verification & Provenance |
|
163 |
+The 20 dimensions span six domains: Quality & Standing · Condition · Physical · Location & Setting · Style & Character · |
|
164 |
+Market Signals. |
| 250 |
165 |
|
| 251 |
|
−This repository is a restructuring of an earlier research folder. Before anything was rewritten, the pipeline was validated against the original artifacts: |
|
166 |
+## 🖼 Figures & tables inventory |
| 252 |
167 |
|
| 253 |
|
−| Check | Result | |
|
168 |
+| Figure | Content | |
| 254 |
169 |
|---|---| |
| 255 |
|
−| Analysis dataset (17,087 × 30) | ✅ identical, column by column | |
| 256 |
|
−| Similarity matrix (17,087 × 20) | ✅ bit-identical | |
| 257 |
|
−| Models A–E fit statistics | ✅ identical to the paper's Table 4 | |
| 258 |
|
−| Model D coefficients (27 params) | ✅ identical to the paper's Table 5 | |
| 259 |
|
−| Figures fig1, fig4–fig11 | ✅ regenerated, byte-size within ±2 B | |
| 260 |
|
−| Figures fig2, fig3 | 🔧 were *missing* from the original repo (placeholder boxes in the old PDF) — regenerated | |
| 261 |
|
−| `robustness_results.csv` | 🔧 lost in the original — regenerated (seeded) | |
| 262 |
|
−| Descriptive/robustness numbers in the old LaTeX | ⚠️ several did not match the code's own outputs — corrected and now machine-generated; full old→new list in [`CHANGES.md`](CHANGES.md) §4 | |
| 263 |
|
− |
| 264 |
|
−`AUDIT.md` documents the original project's layout, dead files, and defects; `CHANGES.md` documents every move, refactor, and numeric correction. |
| 265 |
|
− |
| 266 |
|
−--- |
| 267 |
|
− |
| 268 |
|
−## 🖼️ Figures Gallery |
| 269 |
|
− |
| 270 |
|
−| # | Figure | What it shows | |
| 271 |
|
−|---|---|---| |
| 272 |
|
−| 1 | `fig1_model_comparison` | R² / adj. R² across models A–E | |
| 273 |
|
−| 2 | `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) | |
| 274 |
|
−| 3 | `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features | |
| 275 |
|
−| 4 | `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension | |
| 276 |
|
−| 5 | `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) | |
| 277 |
|
−| 6 | `fig6_scatter_plots` | Four illustrative similarity–price scatters | |
| 278 |
|
−| 7 | `fig7_methodology` | Pipeline diagram | |
| 279 |
|
−| 8 | `fig8_r2_decomposition` | Stacked decomposition of Model D's R² | |
| 280 |
|
−| 9 | `fig9_price_distribution` | Price and log-price histograms | |
| 281 |
|
−| 10 | `fig10_structural_coefficients` | Structural coefficients with 95 % CIs | |
| 282 |
|
−| 11 | `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q | |
| 283 |
|
− |
| 284 |
|
−All figures exist as publication PDF (vector) and PNG (300 dpi). |
| 285 |
|
− |
| 286 |
|
−--- |
| 287 |
|
− |
| 288 |
|
−## 📦 Requirements |
| 289 |
|
− |
| 290 |
|
−| Package | Version | Used for | |
| 291 |
|
−|---|---|---| |
| 292 |
|
−| Python | 3.14 | — | |
| 293 |
|
−| numpy | 2.4.4 | linear algebra | |
| 294 |
|
−| pandas | 3.0.2 | data wrangling | |
| 295 |
|
−| pyarrow | 24.0.0 | parquet I/O | |
| 296 |
|
−| scipy | 1.17.1 | tests, distributions | |
| 297 |
|
−| scikit-learn | 1.6.1 | PCA, Lasso/ElasticNet CV, scaling | |
| 298 |
|
−| statsmodels | 0.14.6 | OLS/HC3, QuantReg, BP test, VIF | |
| 299 |
|
−| matplotlib | 3.10.9 | figures | |
| 300 |
|
−| sentence-transformers | 5.5.0 | embeddings | |
| 301 |
|
−| torch | 2.12.0 | transformer backend (MPS on Apple Silicon) | |
| 302 |
|
− |
| 303 |
|
−LaTeX: TeX Live 2026 with `latexmk` (newtx, booktabs, threeparttable, natbib, hyperref…). |
| 304 |
|
− |
| 305 |
|
−--- |
| 306 |
|
− |
| 307 |
|
−## 📚 Citation |
|
170 |
+| `fig1_model_comparison` | R² / adj. R² across models A–E | |
|
171 |
+| `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) | |
|
172 |
+| `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features | |
|
173 |
+| `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension | |
|
174 |
+| `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) | |
|
175 |
+| `fig6_scatter_plots` | Four illustrative similarity–price scatters | |
|
176 |
+| `fig7_methodology` | Pipeline diagram | |
|
177 |
+| `fig8_r2_decomposition` | Stacked decomposition of Model D's R² | |
|
178 |
+| `fig9_price_distribution` | Price and log-price histograms | |
|
179 |
+| `fig10_structural_coefficients` | Structural coefficients with 95 % CIs | |
|
180 |
+| `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q | |
|
181 |
+ |
|
182 |
+Tables (`paper/tables/*.tex`): descriptive statistics, similarity statistics, model comparison, full Model D results, |
|
183 |
+parsimonious model, quantile regressions, robustness summary, reference texts, text comparison (7 machine-generated by |
|
184 |
+`06_tables.py`, 3 hand-maintained). |
|
185 |
+ |
|
186 |
+## ⚠️ Limitations |
|
187 |
+ |
|
188 |
+- Cross-sectional listing (asking) prices, not transaction prices; coefficients are conditional associations. |
|
189 |
+- Descriptions truncated at ~700 characters in the export. |
|
190 |
+- Similarity dimensions are highly collinear (mean VIF 17.2): individual coefficients are partial associations within a |
|
191 |
+ correlated block; block-level inference is the robust object. |
|
192 |
+- Reference texts embed the researcher's definitional choices; the PCA benchmark quantifies the fit cost of |
|
193 |
+ interpretability. |
|
194 |
+ |
|
195 |
+## 📝 Citation |
| 308 |
196 |
|
| 309 |
197 |
```bibtex |
| 310 |
198 |
@techreport{boucher2026decoding, |
| 311 |
199 |
author = {Boucher, Simon-Pierre}, |
| 312 |
200 |
title = {Decoding Real Estate Descriptions: Semantic Embeddings and |
| 313 |
201 |
Hedonic Pricing of Residential Properties in Quebec}, |
| 314 |
|
− institution = {Universit\'e du Qu\'ebec en Outaouais, |
| 315 |
|
− D\'epartement des sciences administratives}, |
|
202 |
+ institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, |
| 316 |
203 |
type = {Working Paper}, |
| 317 |
204 |
number = {2}, |
| 318 |
205 |
year = {2026}, |
| 320 |
207 |
} |
| 321 |
208 |
``` |
| 322 |
209 |
|
| 323 |
|
−--- |
| 324 |
|
− |
| 325 |
|
−## 👤 Author & Contact |
| 326 |
|
− |
| 327 |
|
−**Simon-Pierre Boucher** |
| 328 |
|
−Département des sciences administratives |
| 329 |
|
−Université du Québec en Outaouais (UQO) |
| 330 |
|
−283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada |
|
210 |
+## 👤 Author & contact |
| 331 |
211 |
|
| 332 |
|
−📧 **contact@spboucher.ai** |
|
212 |
+**Simon-Pierre Boucher** — Département des sciences administratives, |
|
213 |
+Université du Québec en Outaouais (UQO), Gatineau, QC. |
|
214 |
+📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) |
| 333 |
215 |
|
| 334 |
|
−*Comments and suggestions are welcome. All errors are my own.* |
| 335 |
|
− |
| 336 |
|
−--- |
|
216 |
+## 🗒 Notes |
| 337 |
217 |
|
| 338 |
|
−<div align="center"> |
| 339 |
|
−<sub>© 2026 Simon-Pierre Boucher. Paper, code, and figures — all rights reserved.</sub> |
| 340 |
|
−</div> |
|
218 |
+- Format follows `Recherche-UQO/HARMONISATION.md` (2026-10-17): template title page, `natbib` + `aer` bibliography style, |
|
219 |
+ `docs/` for the restructuring reports. |
|
220 |
+- The original flat May 2026 project (`immo-wp2-spb-20260519`) is archived in |
|
221 |
+ `Recherche-UQO/_archives/versions-mai-2026/decoding-real-estate-descriptions--immo-wp2-20260519/`; `docs/AUDIT.md` and |
|
222 |
+ `docs/CHANGES.md` document the 2026-08-05 restructuring and the numeric corrections it caught (old → new values). |
|
223 |
+- Build verified 2026-10-17: 0 errors, 0 undefined references, 86 references rendered. |