| 1 | −<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai --> |
1 | +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca --> |
| 2 | − |
| |
| 3 | −<div align="center"> |
| |
| 4 | − |
| |
| 5 | # 🏠 Decoding Real Estate Descriptions |
2 | # 🏠 Decoding Real Estate Descriptions |
| 6 | |
3 | |
| 7 | −### Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec |
4 | +**UQO Working Paper No. 2** — *Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec* |
| 8 | − |
| |
| 9 | −**UQO Working Paper No. 2** |
| |
| 10 | − |
| |
| 11 | −[](mailto:contact@spboucher.ai) |
| |
| 12 | −[](mailto:contact@spboucher.ai) |
| |
| 13 | −[](https://uqo.ca) |
| |
| 14 | − |
| |
| 15 | −[](https://www.python.org/) |
| |
| 16 | −[](https://www.latex-project.org/) |
| |
| 17 | −[](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) |
| |
| 18 | −[](https://www.statsmodels.org/) |
| |
| 19 | −[](https://scikit-learn.org/) |
| |
| 20 | |
5 | |
| 21 | −[](#-reproducing-everything) |
6 | +[-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) |
| 22 | −[](#-data) |
7 | +[](requirements.txt) |
| 23 | −[](#-key-results) |
8 | +[](data/README.md) |
| 24 | −[](paper/main.pdf) |
9 | +[](src/) |
| 25 | −[-00695C?style=flat-square&logo=googlescholar&logoColor=white)](paper/references.bib) |
10 | +[](scripts/) |
| 26 | −[](#-citation) |
11 | +[](https://uqo.ca) |
| |
12 | +[](mailto:simon-pierre.boucher@uqo.ca) |
| 27 | |
13 | |
| 28 | −</div> |
14 | +> **TL;DR** — Listing descriptions written by real-estate agents contain price-relevant information that classic hedonic |
| |
15 | +> variables miss. The free-text `PublicRemarks` of **17,087 Quebec single-family house listings** are embedded with a |
| |
16 | +> sentence transformer, projected onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs |
| |
17 | +> Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and the 20 interpretable scores are added to a |
| |
18 | +> log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001); each semantic |
| |
19 | +> dimension carries a named, signed, economically meaningful implicit price (Modern/Contemporary +16.4 %, Luxury +14.2 %, |
| |
20 | +> Needs Renovation −7.9 %, Motivated Seller −8.5 % per SD). |
| 29 | |
21 | |
| 30 | --- |
22 | --- |
| 31 | |
23 | |
| 32 | −> **TL;DR** — Listing descriptions written by real estate agents contain price-relevant information that classic hedonic variables miss. This project embeds the free-text `PublicRemarks` of **17,087 Quebec single-family house listings** with a sentence transformer, projects each embedding onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and adds those 20 interpretable scores to a log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001), and each semantic dimension carries a named, signed, economically meaningful implicit price. |
24 | +## 📌 Status |
| 33 | |
25 | |
| 34 | −--- |
26 | +| Item | Value | |
| 35 | − |
27 | +|---|---| |
| 36 | −## 📖 Table of Contents |
28 | +| Version | 1.1 (May 2026) — cover page harmonised to the UQO WP standard, October 2026 | |
| 37 | − |
29 | +| SSRN | Abstract ID **7626640** (submitted 2026-10-17, preliminary upload) | |
| 38 | −- [Overview](#-overview) |
30 | +| Journal | not yet submitted | |
| 39 | −- [Key Results](#-key-results) |
31 | +| Code & data | fully reproducible from `scripts/`; raw database not redistributed (see Data) | |
| 40 | −- [Methodology](#-methodology) |
| |
| 41 | −- [Repository Structure](#-repository-structure) |
| |
| 42 | −- [The Pipeline, Script by Script](#-the-pipeline-script-by-script) |
| |
| 43 | −- [Data](#-data) |
| |
| 44 | −- [Reproducing Everything](#-reproducing-everything) |
| |
| 45 | −- [Building the Paper](#-building-the-paper) |
| |
| 46 | −- [Verification & Provenance](#-verification--provenance) |
| |
| 47 | −- [Figures Gallery](#-figures-gallery) |
| |
| 48 | −- [Requirements](#-requirements) |
| |
| 49 | −- [Citation](#-citation) |
| |
| 50 | −- [Author & Contact](#-author--contact) |
| |
| 51 | − |
| |
| 52 | −--- |
| |
| 53 | − |
| |
| 54 | −## 🔍 Overview |
| |
| 55 | |
32 | |
| 56 | −Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, bathrooms, lot size. But the *narrative* of a listing ("cuisine gastronomique, comptoirs de quartz…" vs. "besoin de rénovation, vendu tel quel…") carries quality information that no structured field captures. |
33 | +## 🎯 What this paper does |
| 57 | |
34 | |
| 58 | −The catch, historically, is a **depth-vs-interpretability trade-off**: |
35 | +Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, |
| |
36 | +bathrooms, lot size. But the *narrative* of a listing («cuisine gastronomique, comptoirs de quartz…» vs. «besoin de |
| |
37 | +rénovation, vendu tel quel…») carries quality information that no structured field captures. Historically there has been |
| |
38 | +a depth-vs-interpretability trade-off: |
| 59 | |
39 | |
| 60 | | Approach | Semantic depth | Economic interpretability | |
40 | | Approach | Semantic depth | Economic interpretability | |
| 61 | |---|---|---| |
41 | |---|---|---| |
| 62 | | Keyword counts / bag-of-words | ❌ shallow | ✅ high | |
42 | | Keyword counts / bag-of-words | ❌ shallow | ✅ high | |
| 63 | | Sentiment scores | ❌ one dimension | ✅ high | |
43 | | Sentiment scores | ❌ one dimension | ✅ high | |
| 64 | | LDA topics | 🟡 moderate | ❌ unstable | |
44 | | LDA topics | 🟡 moderate | ❌ unstable | |
| 65 | −| Raw BERT/transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) | |
45 | +| Raw transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) | |
| 66 | | **Reference-based cosine projection (this paper)** | ✅ **deep** | ✅ **high** | |
46 | | **Reference-based cosine projection (this paper)** | ✅ **deep** | ✅ **high** | |
| 67 | |
47 | |
| 68 | −**The idea:** instead of feeding 384 anonymous embedding dimensions into a regression, define 20 short *reference descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embed them with the same model, and use the **cosine similarity between each listing and each reference** as 20 named regressors. Every coefficient then reads directly as "the implicit price of sounding more *luxury* / more *fixer-upper* / more *waterfront*", per standard deviation. |
48 | +Instead of feeding 384 anonymous embedding dimensions into a regression, the paper defines 20 short *reference |
| 69 | − |
49 | +descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embeds them with the same |
| 70 | −This is conceptually analogous to factor-mimicking portfolios in asset pricing, and to concept-bottleneck models in interpretable ML: the deep representation is channeled through human-named concepts before inference. |
50 | +model, and uses the **cosine similarity between each listing and each reference** as 20 named regressors: |
| 71 | − |
| |
| 72 | −--- |
| |
| 73 | |
51 | |
| 74 | −## 📊 Key Results |
52 | +``` |
| |
53 | +log(P_i) = α + β'X_i + δ·len_i + γ'S_i + ε_i S_i = E_i · Rᵀ (17,087 × 20) |
| |
54 | +``` |
| 75 | |
55 | |
| 76 | −### Model comparison (n = 17,087) |
56 | +Every coefficient then reads directly as «the implicit price of sounding more *luxury* / more *fixer-upper* / more |
| |
57 | +*waterfront*», per standard deviation — analogous to factor-mimicking portfolios in asset pricing and to concept-bottleneck |
| |
58 | +models in interpretable ML. |
| 77 | |
59 | |
| 78 | −| Model | Specification | R² | Adj. R² | AIC | BIC | k | |
60 | +## 🏆 Headline results |
| 79 | −|:---:|---|---:|---:|---:|---:|---:| |
| |
| 80 | −| A | Structural only | 0.4525 | 0.4523 | 26,165 | 26,219 | 6 | |
| |
| 81 | −| B | A + description length | 0.4647 | 0.4645 | 25,779 | 25,841 | 7 | |
| |
| 82 | −| C | A + 20 semantic similarities | 0.4962 | 0.4954 | 24,783 | 24,992 | 26 | |
| |
| 83 | −| **D** | **Full (B + C)** | **0.5122** | **0.5115** | **24,232** | 24,448 | 27 | |
| |
| 84 | −| E | Parsimonious (16 significant dims) | 0.5118 | 0.5111 | 24,239 | **24,425** | 23 | |
| |
| 85 | |
61 | |
| 86 | −Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: structural 88.3 %, text length 2.4 %, semantics 9.3 %. |
62 | +| Model (n = 17,087) | Specification | Adj. R² | AIC | k | |
| |
63 | +|:---:|---|---:|---:|---:| |
| |
64 | +| A | Structural only | 0.4523 | 26,165 | 6 | |
| |
65 | +| B | A + description length | 0.4645 | 25,779 | 7 | |
| |
66 | +| C | A + 20 semantic similarities | 0.4954 | 24,783 | 26 | |
| |
67 | +| **D** | **Full (B + C)** | **0.5115** | **24,232** | 27 | |
| |
68 | +| E | Parsimonious (16 significant dims) | 0.5111 | 24,239 | 23 | |
| 87 | |
69 | |
| 88 | −### Implicit semantic price gradients (Model D, per +1 SD, HC3 errors) |
70 | +Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: |
| |
71 | +structural 88.3 %, text length 2.4 %, semantics 9.3 %. |
| 89 | |
72 | |
| 90 | −| 📈 Positive | Impact | 📉 Negative | Impact | |
73 | +| 📈 Positive (per +1 SD, HC3) | Impact | 📉 Negative | Impact | |
| 91 | |---|---:|---|---:| |
74 | |---|---:|---|---:| |
| 92 | | Modern/Contemporary | **+16.4 %*** | Family-Friendly | **−11.6 %*** | |
75 | | Modern/Contemporary | **+16.4 %*** | Family-Friendly | **−11.6 %*** | |
| 93 | | Luxury | **+14.2 %*** | New Construction | **−10.5 %*** | |
76 | | Luxury | **+14.2 %*** | New Construction | **−10.5 %*** | |
| 96 | | Waterfront | +4.7 %*** | Motivated Seller | **−8.5 %*** | |
79 | | Waterfront | +4.7 %*** | Motivated Seller | **−8.5 %*** | |
| 97 | | Panoramic View | +3.0 %** | Needs Renovation | **−7.9 %*** | |
80 | | Panoramic View | +3.0 %** | Needs Renovation | **−7.9 %*** | |
| 98 | |
81 | |
| 99 | −<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (see paper §6). Impact = (e^β − 1) × 100 %.</sub> |
82 | +<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (paper §6). Impact = (e^β − 1) × 100 %.</sub> |
| 100 | |
83 | |
| 101 | −### Heterogeneity & robustness (highlights) |
84 | +- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from 9.7 % to 17.3 %; urgency |
| |
85 | + and condition discounts attenuate at the top of the market. |
| |
86 | +- **Bootstrap** (1,000 replications): SEs within 9 % of HC3. **Trimming** (1st/99th percentiles): no sign reversal. |
| |
87 | +- **Multicollinearity disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2); block inference unaffected. |
| |
88 | +- **PCA benchmark**: 20 principal components fit better (ΔR² +0.070 vs. +0.044) but are economically unreadable. |
| 102 | |
89 | |
| 103 | −- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from **9.7 %** at the 25th percentile to **17.3 %** at the 75th; urgency and condition discounts attenuate at the top of the market. |
90 | +## 🗂 Repository layout |
| 104 | −- **Bootstrap** (1,000 replications, seed 42): SEs within 9 % of HC3 (15/20 within 5 %). |
| |
| 105 | −- **Outlier trimming** (1st/99th price percentiles): no sign reversal, all 16 significant dimensions stay significant. |
| |
| 106 | −- **Multicollinearity is real and disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2). Block-level inference is unaffected; individual coefficients are partial associations within a correlated block. |
| |
| 107 | −- **PCA benchmark**: 20 principal components of the raw embeddings fit *better* (ΔR² +0.070 vs. +0.044) but are economically unreadable — the paper quantifies the interpretability-vs-fit trade-off explicitly. |
| |
| 108 | − |
| |
| 109 | −--- |
| |
| 110 | − |
| |
| 111 | −## 🧪 Methodology |
| |
| 112 | − |
| |
| 113 | −``` |
| |
| 114 | −┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ |
| |
| 115 | −│ Property │ │ Sentence │ │ Cosine │ │ Hedonic │ │ Implicit │ |
| |
| 116 | −│ Listings ├──▶│ Embeddings ├──▶│ Similarity ├──▶│ OLS Model ├──▶│ Price │ |
| |
| 117 | −│ (n=17,087) │ │ (384-d) │ │ (20 refs) │ │ (HC3 s.e.) │ │ Estimates │ |
| |
| 118 | −└─────────────┘ └──────────────┘ └──────────────┘ └─────────────┘ └──────────────┘ |
| |
| 119 | − PublicRemarks all-MiniLM-L6-v2 S = E · Rᵀ log(P) = α + β'X + δℓ + γ's + ε |
| |
| 120 | −``` |
| |
| 121 | − |
| |
| 122 | −1. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (the transformer handles raw text). |
| |
| 123 | −2. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalized), batch 256; the whole corpus encodes in seconds on Apple Silicon. |
| |
| 124 | −3. **Projection** — 20 French reference descriptions (146–257 chars each, full verbatim text in paper Appendix A), designed for *semantic saturation*, *dimensional specificity*, *linguistic consistency*. One matrix product yields the 17,087 × 20 similarity matrix. |
| |
| 125 | −4. **Estimation** — nested OLS models A–E on log price, all covariates standardized, HC3 robust errors; joint F-tests; quantile regressions; bootstrap; winsorization; VIF; Lasso/Elastic-Net selection; PCA benchmark. |
| |
| 126 | − |
| |
| 127 | −The 20 dimensions span six domains: **Quality & Standing** (Luxury, Entry-Level) · **Condition** (Renovated, Needs Renovation) · **Physical** (Bright & Spacious, Garage & Parking, Finished Basement, Pool & Landscaping) · **Location & Setting** (Land & Nature, Panoramic View, Premium Location, Quiet & Peaceful, Waterfront) · **Style & Character** (Modern/Contemporary, Heritage/Character, New Construction, Energy Efficient, Family-Friendly) · **Market Signals** (Income/Investment, Motivated Seller). |
| |
| 128 | − |
| |
| 129 | −--- |
| |
| 130 | − |
| |
| 131 | −## 📁 Repository Structure |
| |
| 132 | |
91 | |
| 133 | ``` |
92 | ``` |
| 134 | wp2_uqo/ |
93 | wp2_uqo/ |
| 135 | −├── README.md ← you are here |
94 | +├── README.md |
| 136 | −├── AUDIT.md ← forensic audit of the original (pre-restructuring) project |
95 | +├── requirements.txt # pinned dependencies (Python 3.14, macOS arm64) |
| 137 | −├── CHANGES.md ← everything that was moved / refactored / corrected, with diffs |
96 | +├── docs/ # AUDIT.md · CHANGES.md · PAPER_REVIEW.md · UPGRADE_REPORT.md (restructuring history) |
| 138 | −├── PAPER_REVIEW.md ← critical pre-upgrade assessment of the paper's scholarship |
97 | +├── data/ # NOT tracked (see data/README.md) |
| 139 | −├── UPGRADE_REPORT.md ← literature expansion: refs added/replaced, claims flagged |
98 | +│ ├── raw/louka.db # SQLite, 268 MB, 46,479 listings |
| 140 | −├── requirements.txt ← pinned dependencies (exact versions used for verification) |
99 | +│ └── processed/ # parquet sample, embeddings cache, similarity matrix |
| 141 | −│ |
100 | +├── src/ # analysis package |
| 142 | −├── data/ ← NOT tracked by git (see data/README.md) |
101 | +│ ├── config.py # repo-relative paths, model name, seed, structural vars |
| 143 | −│ ├── raw/louka.db │ SQLite, 268 MB, 46,479 listings (5 categories) |
102 | +│ ├── references.py # the 20 French reference descriptions + English labels |
| 144 | −│ └── processed/ │ parquet sample, embeddings cache, similarity matrix, |
103 | +│ ├── data.py # SQLite → DataFrame extraction (n = 17,087) |
| 145 | −│ └── legacy/ │ analysis CSV + pre-refactor artifacts kept for comparison |
104 | +│ ├── embeddings.py # encoding, caching, cosine-similarity features |
| 146 | −│ |
105 | +│ └── models.py # standardized-design OLS A–E, HC3 |
| 147 | −├── src/ ← analysis package (imported by scripts/) |
106 | +├── scripts/ # the pipeline, in order |
| 148 | −│ ├── config.py │ repo-relative paths, model name, seed, structural vars |
107 | +│ ├── 01_prepare_data.py # louka.db → data/processed/houses.parquet |
| 149 | −│ ├── references.py │ the 20 French reference descriptions + English labels |
108 | +│ ├── 02_similarities.py # embeddings (cached) + 20 similarity features |
| 150 | −│ ├── data.py │ SQLite → DataFrame extraction (n = 17,087) |
109 | +│ ├── 03_models.py # models A–E → results/*.csv |
| 151 | −│ ├── embeddings.py │ encoding, caching, cosine-similarity features |
110 | +│ ├── 04_robustness.py # VIF · BP · quantile · winsor · bootstrap · Lasso/ENet |
| 152 | −│ └── models.py │ standardized-design OLS A–E, HC3 |
111 | +│ ├── 05_figures.py # fig1–fig11 (PDF + PNG) |
| 153 | −│ |
112 | +│ └── 06_tables.py # LaTeX tables generated from results |
| 154 | −├── scripts/ ← numbered pipeline entry points (run in order) |
113 | +├── results/ # model & robustness outputs (CSV) |
| 155 | −│ ├── 01_prepare_data.py │ louka.db → data/processed/houses.parquet |
114 | +├── figures/ # 11 paper figures |
| 156 | −│ ├── 02_similarities.py │ embeddings (cached) + 20 similarity features → CSV |
115 | +└── paper/ # main.tex + preamble.tex + sections/ + tables/ + references.bib → main.pdf |
| 157 | −│ ├── 03_models.py │ models A–E → results/*.csv |
| |
| 158 | −│ ├── 04_robustness.py │ VIF · BP · quantile · winsor · bootstrap · Lasso/ENet |
| |
| 159 | −│ ├── 05_figures.py │ figures fig1–fig11 (PDF + PNG) |
| |
| 160 | −│ └── 06_tables.py │ LaTeX tables generated from results (no hand transcription) |
| |
| 161 | −│ |
| |
| 162 | −├── figures/ ← all 11 paper figures, regenerated from the pipeline |
| |
| 163 | −├── results/ ← model & robustness outputs (CSV, versioned) |
| |
| 164 | −│ |
| |
| 165 | −└── paper/ ← LaTeX source (UQO working-paper style) |
| |
| 166 | − ├── main.tex │ metadata + section inputs (53-page PDF) |
| |
| 167 | − ├── preamble.tex │ packages, layout, custom macros |
| |
| 168 | − ├── main.pdf │ compiled paper (committed for convenience) |
| |
| 169 | − ├── Makefile │ latexmk build |
| |
| 170 | − ├── references.bib │ 86 entries, every one verified (OpenAlex + DOI) |
| |
| 171 | − ├── sections/ │ titlepage · intro · literature · methodology · data · |
| |
| 172 | − │ │ results · robustness · discussion · conclusion · appendix |
| |
| 173 | − └── tables/ │ 7 machine-generated + 3 hand-maintained booktabs tables |
| |
| 174 | ``` |
116 | ``` |
| 175 | |
117 | |
| 176 | −--- |
118 | +## 🚀 Quick start |
| 177 | − |
| |
| 178 | −## ⚙️ The Pipeline, Script by Script |
| |
| 179 | − |
| |
| 180 | −| # | Script | Input | Output | Runtime* | |
| |
| 181 | −|---|---|---|---|---| |
| |
| 182 | −| 1 | `01_prepare_data.py` | `data/raw/louka.db` | `houses.parquet` (17,087 × 14) | ~15 s | |
| |
| 183 | −| 2 | `02_similarities.py` | parquet + refs | embeddings `.npy` (cached), `sim_matrix.npy`, analysis CSV | ~5 s cached / ~3 min fresh | |
| |
| 184 | −| 3 | `03_models.py` | analysis CSV | `model_comparison.csv`, coefficient tables, descriptive stats | ~5 s | |
| |
| 185 | −| 4 | `04_robustness.py` | analysis CSV | `robustness_results.csv` (163 rows: VIF, BP, QR, winsor, bootstrap, Lasso) | ~4 min (bootstrap) | |
| |
| 186 | −| 5 | `05_figures.py` | analysis CSV | `figures/fig1…fig11.{pdf,png}` | ~30 s | |
| |
| 187 | −| 6 | `06_tables.py` | `results/*.csv` | `paper/tables/*.tex` (7 tables) | ~2 s | |
| |
| 188 | − |
| |
| 189 | −<sub>*Apple M-series laptop. All stochastic steps are seeded (`seed = 42`): bootstrap resampling, CV folds, scatter subsampling.</sub> |
| |
| 190 | − |
| |
| 191 | −**Design principles applied during the refactor** (see `CHANGES.md` for the full story): |
| |
| 192 | − |
| |
| 193 | −- 🚫 **No hardcoded paths** — everything resolves from the repo root (`src/config.py`); the DB location can be overridden with `WP2_DB_PATH`. |
| |
| 194 | −- 🧬 **Single source of truth** — the extraction logic and the 20 reference texts, previously copy-pasted across five scripts (in two languages), now live in exactly one place each. |
| |
| 195 | −- 🔁 **Deterministic & cached** — embeddings are deterministic for a given model version and cached; `--force` re-encodes from scratch. |
| |
| 196 | −- 📋 **Tables are compiled artifacts** — every number in the paper's statistical tables is written by `06_tables.py` from the results CSVs. The original project transcribed numbers by hand, which introduced errors that this restructuring caught and fixed (documented, with old→new values, in `CHANGES.md` §4). |
| |
| 197 | − |
| |
| 198 | −--- |
| |
| 199 | − |
| |
| 200 | −## 💾 Data |
| |
| 201 | − |
| |
| 202 | −The raw data is a **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: **46,479 listings** across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). Each row carries structured fields (price, bedrooms, bathrooms) plus a full JSON blob including the agent-written `PublicRemarks`. |
| |
| 203 | − |
| |
| 204 | −**The analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. Median listing price $589,900 (mean $807,259, max $25M). Descriptions average 510 characters and are truncated at ~700 characters in the export (disclosed in the paper). |
| |
| 205 | − |
| |
| 206 | −> ⚠️ **`data/` is not tracked by git**: the raw database exceeds GitHub's 100 MB file limit, contains a leftover application `users` table, and the listing texts are proprietary platform content. `data/README.md` explains how to place a copy of `louka.db` to re-run the pipeline from scratch. All *derived, aggregate* outputs (results CSVs, figures, tables, paper) **are** versioned, so every number in the paper is inspectable without the raw data. |
| |
| 207 | − |
| |
| 208 | −--- |
| |
| 209 | − |
| |
| 210 | −## 🔄 Reproducing Everything |
| |
| 211 | |
119 | |
| 212 | ```bash |
120 | ```bash |
| 213 | −git clone https://github.com/spboucher-ai/wp2_uqo.git |
| |
| 214 | −cd wp2_uqo |
| |
| 215 | python3 -m pip install -r requirements.txt |
121 | python3 -m pip install -r requirements.txt |
| 216 | |
122 | |
| 217 | # place louka.db in data/raw/ (see data/README.md), then: |
123 | # place louka.db in data/raw/ (see data/README.md), then: |
| 218 | −python3 scripts/01_prepare_data.py # extract the house sample |
124 | +python3 scripts/01_prepare_data.py # 17,087 houses with a valid price and description |
| 219 | python3 scripts/02_similarities.py # embed + project (add --force to re-encode) |
125 | python3 scripts/02_similarities.py # embed + project (add --force to re-encode) |
| 220 | python3 scripts/03_models.py # hedonic models A–E |
126 | python3 scripts/03_models.py # hedonic models A–E |
| 221 | −python3 scripts/04_robustness.py # full robustness battery (~4 min) |
127 | +python3 scripts/04_robustness.py # robustness battery (~4 min, bootstrap) |
| 222 | −python3 scripts/05_figures.py # all 11 figures |
128 | +python3 scripts/05_figures.py # 11 figures |
| 223 | python3 scripts/06_tables.py # LaTeX tables from results |
129 | python3 scripts/06_tables.py # LaTeX tables from results |
| 224 | |
130 | |
| 225 | −cd paper && latexmk -pdf main.tex # 53-page PDF, 0 errors |
131 | +cd paper && make # latexmk → main.pdf (53 pages) |
| 226 | ``` |
132 | ``` |
| 227 | |
133 | |
| 228 | −Expected checkpoints along the way: |
134 | +Total runtime ≈ 8 minutes on an Apple-silicon laptop (embeddings cached after the first run). All stochastic steps are |
| |
135 | +seeded (`seed = 42`). Expected checkpoints: step 3 prints `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`. |
| 229 | |
136 | |
| 230 | −- Step 1 prints `17,087 houses with a valid price and description`. |
137 | +## 🗃 Data |
| 231 | −- Step 3 prints the model table with `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`. |
| |
| 232 | −- Step 4 prints `Winsorized R2=0.4867` and the Lasso selection (4/20 at the CV penalty). |
| |
| 233 | |
138 | |
| 234 | −--- |
139 | +A **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: |
| 235 | − |
140 | +46,479 listings across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). |
| 236 | −## 📝 Building the Paper |
141 | +Each row carries structured fields plus the agent-written `PublicRemarks`. |
| 237 | |
142 | |
| 238 | −The paper (`paper/`) follows the UQO working-paper format: title page with logo, abstract page with keywords + JEL codes, IMRaD body (Introduction · Literature · Methodology · Data · Results · Robustness · Discussion · Conclusion), `natbib`/`apalike` bibliography, and two appendices (verbatim reference texts; parsimonious-model estimates). |
143 | +**Analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. |
| |
144 | +Median listing price $589,900 (mean $807,259). Descriptions average 510 characters, truncated at ~700 characters in the |
| |
145 | +export (disclosed in the paper). |
| 239 | |
146 | |
| 240 | −```bash |
147 | +`data/` is not tracked: the database exceeds 100 MB and listing texts are proprietary platform content. All derived, |
| 241 | −cd paper |
148 | +aggregate outputs (results CSVs, figures, tables, paper) are versioned, so every number in the paper is inspectable |
| 242 | −make # or: latexmk -pdf main.tex |
149 | +without the raw data. `data/README.md` explains how to place a copy of `louka.db`; the location can be overridden with |
| 243 | −``` |
150 | +`WP2_DB_PATH`. |
| 244 | |
151 | |
| 245 | −Figures are pulled from `../figures/` via `\graphicspath`, so the paper always reflects the latest pipeline run. The build is clean: **0 errors, 0 undefined references, 0 missing citations** (59/59 bib keys resolve). |
152 | +## 🔬 Methodology |
| 246 | |
153 | |
| 247 | −--- |
154 | +1. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (`src/data.py`). |
| |
155 | +2. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalised), batch 256 (`src/embeddings.py`). |
| |
156 | +3. **Projection** — 20 French reference descriptions (146–257 chars, verbatim in paper Appendix A), designed for semantic |
| |
157 | + saturation, dimensional specificity and linguistic consistency; one matrix product yields the 17,087 × 20 similarity |
| |
158 | + matrix (`src/references.py`). |
| |
159 | +4. **Estimation** — nested OLS models A–E on log price, standardised covariates, HC3 errors; joint F-tests; quantile |
| |
160 | + regressions; bootstrap; winsorisation; VIF; Lasso/Elastic-Net selection; PCA benchmark (`src/models.py`, |
| |
161 | + `scripts/04_robustness.py`). |
| 248 | |
162 | |
| 249 | −## ✅ Verification & Provenance |
163 | +The 20 dimensions span six domains: Quality & Standing · Condition · Physical · Location & Setting · Style & Character · |
| |
164 | +Market Signals. |
| 250 | |
165 | |
| 251 | −This repository is a restructuring of an earlier research folder. Before anything was rewritten, the pipeline was validated against the original artifacts: |
166 | +## 🖼 Figures & tables inventory |
| 252 | |
167 | |
| 253 | −| Check | Result | |
168 | +| Figure | Content | |
| 254 | |---|---| |
169 | |---|---| |
| 255 | −| Analysis dataset (17,087 × 30) | ✅ identical, column by column | |
170 | +| `fig1_model_comparison` | R² / adj. R² across models A–E | |
| 256 | −| Similarity matrix (17,087 × 20) | ✅ bit-identical | |
171 | +| `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) | |
| 257 | −| Models A–E fit statistics | ✅ identical to the paper's Table 4 | |
172 | +| `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features | |
| 258 | −| Model D coefficients (27 params) | ✅ identical to the paper's Table 5 | |
173 | +| `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension | |
| 259 | −| Figures fig1, fig4–fig11 | ✅ regenerated, byte-size within ±2 B | |
174 | +| `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) | |
| 260 | −| Figures fig2, fig3 | 🔧 were *missing* from the original repo (placeholder boxes in the old PDF) — regenerated | |
175 | +| `fig6_scatter_plots` | Four illustrative similarity–price scatters | |
| 261 | −| `robustness_results.csv` | 🔧 lost in the original — regenerated (seeded) | |
176 | +| `fig7_methodology` | Pipeline diagram | |
| 262 | −| Descriptive/robustness numbers in the old LaTeX | ⚠️ several did not match the code's own outputs — corrected and now machine-generated; full old→new list in [`CHANGES.md`](CHANGES.md) §4 | |
177 | +| `fig8_r2_decomposition` | Stacked decomposition of Model D's R² | |
| 263 | − |
178 | +| `fig9_price_distribution` | Price and log-price histograms | |
| 264 | −`AUDIT.md` documents the original project's layout, dead files, and defects; `CHANGES.md` documents every move, refactor, and numeric correction. |
179 | +| `fig10_structural_coefficients` | Structural coefficients with 95 % CIs | |
| 265 | − |
180 | +| `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q | |
| 266 | −--- |
181 | + |
| 267 | − |
182 | +Tables (`paper/tables/*.tex`): descriptive statistics, similarity statistics, model comparison, full Model D results, |
| 268 | −## 🖼️ Figures Gallery |
183 | +parsimonious model, quantile regressions, robustness summary, reference texts, text comparison (7 machine-generated by |
| 269 | − |
184 | +`06_tables.py`, 3 hand-maintained). |
| 270 | −| # | Figure | What it shows | |
185 | + |
| 271 | −|---|---|---| |
186 | +## ⚠️ Limitations |
| 272 | −| 1 | `fig1_model_comparison` | R² / adj. R² across models A–E | |
187 | + |
| 273 | −| 2 | `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) | |
188 | +- Cross-sectional listing (asking) prices, not transaction prices; coefficients are conditional associations. |
| 274 | −| 3 | `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features | |
189 | +- Descriptions truncated at ~700 characters in the export. |
| 275 | −| 4 | `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension | |
190 | +- Similarity dimensions are highly collinear (mean VIF 17.2): individual coefficients are partial associations within a |
| 276 | −| 5 | `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) | |
191 | + correlated block; block-level inference is the robust object. |
| 277 | −| 6 | `fig6_scatter_plots` | Four illustrative similarity–price scatters | |
192 | +- Reference texts embed the researcher's definitional choices; the PCA benchmark quantifies the fit cost of |
| 278 | −| 7 | `fig7_methodology` | Pipeline diagram | |
193 | + interpretability. |
| 279 | −| 8 | `fig8_r2_decomposition` | Stacked decomposition of Model D's R² | |
194 | + |
| 280 | −| 9 | `fig9_price_distribution` | Price and log-price histograms | |
195 | +## 📝 Citation |
| 281 | −| 10 | `fig10_structural_coefficients` | Structural coefficients with 95 % CIs | |
| |
| 282 | −| 11 | `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q | |
| |
| 283 | − |
| |
| 284 | −All figures exist as publication PDF (vector) and PNG (300 dpi). |
| |
| 285 | − |
| |
| 286 | −--- |
| |
| 287 | − |
| |
| 288 | −## 📦 Requirements |
| |
| 289 | − |
| |
| 290 | −| Package | Version | Used for | |
| |
| 291 | −|---|---|---| |
| |
| 292 | −| Python | 3.14 | — | |
| |
| 293 | −| numpy | 2.4.4 | linear algebra | |
| |
| 294 | −| pandas | 3.0.2 | data wrangling | |
| |
| 295 | −| pyarrow | 24.0.0 | parquet I/O | |
| |
| 296 | −| scipy | 1.17.1 | tests, distributions | |
| |
| 297 | −| scikit-learn | 1.6.1 | PCA, Lasso/ElasticNet CV, scaling | |
| |
| 298 | −| statsmodels | 0.14.6 | OLS/HC3, QuantReg, BP test, VIF | |
| |
| 299 | −| matplotlib | 3.10.9 | figures | |
| |
| 300 | −| sentence-transformers | 5.5.0 | embeddings | |
| |
| 301 | −| torch | 2.12.0 | transformer backend (MPS on Apple Silicon) | |
| |
| 302 | − |
| |
| 303 | −LaTeX: TeX Live 2026 with `latexmk` (newtx, booktabs, threeparttable, natbib, hyperref…). |
| |
| 304 | − |
| |
| 305 | −--- |
| |
| 306 | − |
| |
| 307 | −## 📚 Citation |
| |
| 308 | |
196 | |
| 309 | ```bibtex |
197 | ```bibtex |
| 310 | @techreport{boucher2026decoding, |
198 | @techreport{boucher2026decoding, |
| 311 | author = {Boucher, Simon-Pierre}, |
199 | author = {Boucher, Simon-Pierre}, |
| 312 | title = {Decoding Real Estate Descriptions: Semantic Embeddings and |
200 | title = {Decoding Real Estate Descriptions: Semantic Embeddings and |
| 313 | Hedonic Pricing of Residential Properties in Quebec}, |
201 | Hedonic Pricing of Residential Properties in Quebec}, |
| 314 | − institution = {Universit\'e du Qu\'ebec en Outaouais, |
202 | + institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, |
| 315 | − D\'epartement des sciences administratives}, |
| |
| 316 | type = {Working Paper}, |
203 | type = {Working Paper}, |
| 317 | number = {2}, |
204 | number = {2}, |
| 318 | year = {2026}, |
205 | year = {2026}, |
| 320 | } |
207 | } |
| 321 | ``` |
208 | ``` |
| 322 | |
209 | |
| 323 | −--- |
210 | +## 👤 Author & contact |
| 324 | − |
| |
| 325 | −## 👤 Author & Contact |
| |
| 326 | − |
| |
| 327 | −**Simon-Pierre Boucher** |
| |
| 328 | −Département des sciences administratives |
| |
| 329 | −Université du Québec en Outaouais (UQO) |
| |
| 330 | −283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada |
| |
| 331 | |
211 | |
| 332 | −📧 **contact@spboucher.ai** |
212 | +**Simon-Pierre Boucher** — Département des sciences administratives, |
| |
213 | +Université du Québec en Outaouais (UQO), Gatineau, QC. |
| |
214 | +📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) |
| 333 | |
215 | |
| 334 | −*Comments and suggestions are welcome. All errors are my own.* |
216 | +## 🗒 Notes |
| 335 | − |
| |
| 336 | −--- |
| |
| 337 | |
217 | |
| 338 | −<div align="center"> |
218 | +- Format follows `Recherche-UQO/HARMONISATION.md` (2026-10-17): template title page, `natbib` + `aer` bibliography style, |
| 339 | −<sub>© 2026 Simon-Pierre Boucher. Paper, code, and figures — all rights reserved.</sub> |
219 | + `docs/` for the restructuring reports. |
| 340 | −</div> |
220 | +- The original flat May 2026 project (`immo-wp2-spb-20260519`) is archived in |
| |
221 | + `Recherche-UQO/_archives/versions-mai-2026/decoding-real-estate-descriptions--immo-wp2-20260519/`; `docs/AUDIT.md` and |
| |
222 | + `docs/CHANGES.md` document the 2026-08-05 restructuring and the numeric corrections it caught (old → new values). |
| |
223 | +- Build verified 2026-10-17: 0 errors, 0 undefined references, 86 references rendered. |