spb/wp2_uqo Public
UQO Working Paper No. 2 — Decoding Real Estate Descriptions: text-based hedonic analysis of housing listings.
TeX 73.8%
Python 26%
1<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai -->23<div align="center">45# 🏠 Decoding Real Estate Descriptions67### Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec89**UQO Working Paper No. 2**1011[](mailto:contact@spboucher.ai)12[](mailto:contact@spboucher.ai)13[](https://uqo.ca)1415[](https://www.python.org/)16[](https://www.latex-project.org/)17[](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)18[](https://www.statsmodels.org/)19[](https://scikit-learn.org/)2021[](#-reproducing-everything)22[](#-data)23[](#-key-results)24[](paper/main.pdf)25[-00695C?style=flat-square&logo=googlescholar&logoColor=white)](paper/references.bib)26[](#-citation)2728</div>2930---3132> **TL;DR** — Listing descriptions written by real estate agents contain price-relevant information that classic hedonic variables miss. This project embeds the free-text `PublicRemarks` of **17,087 Quebec single-family house listings** with a sentence transformer, projects each embedding onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and adds those 20 interpretable scores to a log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001), and each semantic dimension carries a named, signed, economically meaningful implicit price.3334---3536## 📖 Table of Contents3738- [Overview](#-overview)39- [Key Results](#-key-results)40- [Methodology](#-methodology)41- [Repository Structure](#-repository-structure)42- [The Pipeline, Script by Script](#-the-pipeline-script-by-script)43- [Data](#-data)44- [Reproducing Everything](#-reproducing-everything)45- [Building the Paper](#-building-the-paper)46- [Verification & Provenance](#-verification--provenance)47- [Figures Gallery](#-figures-gallery)48- [Requirements](#-requirements)49- [Citation](#-citation)50- [Author & Contact](#-author--contact)5152---5354## 🔍 Overview5556Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, bathrooms, lot size. But the *narrative* of a listing ("cuisine gastronomique, comptoirs de quartz…" vs. "besoin de rénovation, vendu tel quel…") carries quality information that no structured field captures.5758The catch, historically, is a **depth-vs-interpretability trade-off**:5960| Approach | Semantic depth | Economic interpretability |61|---|---|---|62| Keyword counts / bag-of-words | ❌ shallow | ✅ high |63| Sentiment scores | ❌ one dimension | ✅ high |64| LDA topics | 🟡 moderate | ❌ unstable |65| Raw BERT/transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) |66| **Reference-based cosine projection (this paper)** | ✅ **deep** | ✅ **high** |6768**The idea:** instead of feeding 384 anonymous embedding dimensions into a regression, define 20 short *reference descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embed them with the same model, and use the **cosine similarity between each listing and each reference** as 20 named regressors. Every coefficient then reads directly as "the implicit price of sounding more *luxury* / more *fixer-upper* / more *waterfront*", per standard deviation.6970This is conceptually analogous to factor-mimicking portfolios in asset pricing, and to concept-bottleneck models in interpretable ML: the deep representation is channeled through human-named concepts before inference.7172---7374## 📊 Key Results7576### Model comparison (n = 17,087)7778| Model | Specification | R² | Adj. R² | AIC | BIC | k |79|:---:|---|---:|---:|---:|---:|---:|80| A | Structural only | 0.4525 | 0.4523 | 26,165 | 26,219 | 6 |81| B | A + description length | 0.4647 | 0.4645 | 25,779 | 25,841 | 7 |82| C | A + 20 semantic similarities | 0.4962 | 0.4954 | 24,783 | 24,992 | 26 |83| **D** | **Full (B + C)** | **0.5122** | **0.5115** | **24,232** | 24,448 | 27 |84| E | Parsimonious (16 significant dims) | 0.5118 | 0.5111 | 24,239 | **24,425** | 23 |8586Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: structural 88.3 %, text length 2.4 %, semantics 9.3 %.8788### Implicit semantic price gradients (Model D, per +1 SD, HC3 errors)8990| 📈 Positive | Impact | 📉 Negative | Impact |91|---|---:|---|---:|92| Modern/Contemporary | **+16.4 %*** | Family-Friendly | **−11.6 %*** |93| Luxury | **+14.2 %*** | New Construction | **−10.5 %*** |94| Land & Nature | **+13.0 %*** | Quiet & Peaceful | −9.5 %*** |95| Entry-Level | +10.4 %*** | Bright & Spacious | −8.9 %*** |96| Waterfront | +4.7 %*** | Motivated Seller | **−8.5 %*** |97| Panoramic View | +3.0 %** | Needs Renovation | **−7.9 %*** |9899<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (see paper §6). Impact = (e^β − 1) × 100 %.</sub>100101### Heterogeneity & robustness (highlights)102103- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from **9.7 %** at the 25th percentile to **17.3 %** at the 75th; urgency and condition discounts attenuate at the top of the market.104- **Bootstrap** (1,000 replications, seed 42): SEs within 9 % of HC3 (15/20 within 5 %).105- **Outlier trimming** (1st/99th price percentiles): no sign reversal, all 16 significant dimensions stay significant.106- **Multicollinearity is real and disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2). Block-level inference is unaffected; individual coefficients are partial associations within a correlated block.107- **PCA benchmark**: 20 principal components of the raw embeddings fit *better* (ΔR² +0.070 vs. +0.044) but are economically unreadable — the paper quantifies the interpretability-vs-fit trade-off explicitly.108109---110111## 🧪 Methodology112113```114┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐115│ Property │ │ Sentence │ │ Cosine │ │ Hedonic │ │ Implicit │116│ Listings ├──▶│ Embeddings ├──▶│ Similarity ├──▶│ OLS Model ├──▶│ Price │117│ (n=17,087) │ │ (384-d) │ │ (20 refs) │ │ (HC3 s.e.) │ │ Estimates │118└─────────────┘ └──────────────┘ └──────────────┘ └─────────────┘ └──────────────┘119 PublicRemarks all-MiniLM-L6-v2 S = E · Rᵀ log(P) = α + β'X + δℓ + γ's + ε120```1211221. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (the transformer handles raw text).1232. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalized), batch 256; the whole corpus encodes in seconds on Apple Silicon.1243. **Projection** — 20 French reference descriptions (146–257 chars each, full verbatim text in paper Appendix A), designed for *semantic saturation*, *dimensional specificity*, *linguistic consistency*. One matrix product yields the 17,087 × 20 similarity matrix.1254. **Estimation** — nested OLS models A–E on log price, all covariates standardized, HC3 robust errors; joint F-tests; quantile regressions; bootstrap; winsorization; VIF; Lasso/Elastic-Net selection; PCA benchmark.126127The 20 dimensions span six domains: **Quality & Standing** (Luxury, Entry-Level) · **Condition** (Renovated, Needs Renovation) · **Physical** (Bright & Spacious, Garage & Parking, Finished Basement, Pool & Landscaping) · **Location & Setting** (Land & Nature, Panoramic View, Premium Location, Quiet & Peaceful, Waterfront) · **Style & Character** (Modern/Contemporary, Heritage/Character, New Construction, Energy Efficient, Family-Friendly) · **Market Signals** (Income/Investment, Motivated Seller).128129---130131## 📁 Repository Structure132133```134wp2_uqo/135├── README.md ← you are here136├── AUDIT.md ← forensic audit of the original (pre-restructuring) project137├── CHANGES.md ← everything that was moved / refactored / corrected, with diffs138├── PAPER_REVIEW.md ← critical pre-upgrade assessment of the paper's scholarship139├── UPGRADE_REPORT.md ← literature expansion: refs added/replaced, claims flagged140├── requirements.txt ← pinned dependencies (exact versions used for verification)141│142├── data/ ← NOT tracked by git (see data/README.md)143│ ├── raw/louka.db │ SQLite, 268 MB, 46,479 listings (5 categories)144│ └── processed/ │ parquet sample, embeddings cache, similarity matrix,145│ └── legacy/ │ analysis CSV + pre-refactor artifacts kept for comparison146│147├── src/ ← analysis package (imported by scripts/)148│ ├── config.py │ repo-relative paths, model name, seed, structural vars149│ ├── references.py │ the 20 French reference descriptions + English labels150│ ├── data.py │ SQLite → DataFrame extraction (n = 17,087)151│ ├── embeddings.py │ encoding, caching, cosine-similarity features152│ └── models.py │ standardized-design OLS A–E, HC3153│154├── scripts/ ← numbered pipeline entry points (run in order)155│ ├── 01_prepare_data.py │ louka.db → data/processed/houses.parquet156│ ├── 02_similarities.py │ embeddings (cached) + 20 similarity features → CSV157│ ├── 03_models.py │ models A–E → results/*.csv158│ ├── 04_robustness.py │ VIF · BP · quantile · winsor · bootstrap · Lasso/ENet159│ ├── 05_figures.py │ figures fig1–fig11 (PDF + PNG)160│ └── 06_tables.py │ LaTeX tables generated from results (no hand transcription)161│162├── figures/ ← all 11 paper figures, regenerated from the pipeline163├── results/ ← model & robustness outputs (CSV, versioned)164│165└── paper/ ← LaTeX source (UQO working-paper style)166 ├── main.tex │ metadata + section inputs (53-page PDF)167 ├── preamble.tex │ packages, layout, custom macros168 ├── main.pdf │ compiled paper (committed for convenience)169 ├── Makefile │ latexmk build170 ├── references.bib │ 86 entries, every one verified (OpenAlex + DOI)171 ├── sections/ │ titlepage · intro · literature · methodology · data ·172 │ │ results · robustness · discussion · conclusion · appendix173 └── tables/ │ 7 machine-generated + 3 hand-maintained booktabs tables174```175176---177178## ⚙️ The Pipeline, Script by Script179180| # | Script | Input | Output | Runtime* |181|---|---|---|---|---|182| 1 | `01_prepare_data.py` | `data/raw/louka.db` | `houses.parquet` (17,087 × 14) | ~15 s |183| 2 | `02_similarities.py` | parquet + refs | embeddings `.npy` (cached), `sim_matrix.npy`, analysis CSV | ~5 s cached / ~3 min fresh |184| 3 | `03_models.py` | analysis CSV | `model_comparison.csv`, coefficient tables, descriptive stats | ~5 s |185| 4 | `04_robustness.py` | analysis CSV | `robustness_results.csv` (163 rows: VIF, BP, QR, winsor, bootstrap, Lasso) | ~4 min (bootstrap) |186| 5 | `05_figures.py` | analysis CSV | `figures/fig1…fig11.{pdf,png}` | ~30 s |187| 6 | `06_tables.py` | `results/*.csv` | `paper/tables/*.tex` (7 tables) | ~2 s |188189<sub>*Apple M-series laptop. All stochastic steps are seeded (`seed = 42`): bootstrap resampling, CV folds, scatter subsampling.</sub>190191**Design principles applied during the refactor** (see `CHANGES.md` for the full story):192193- 🚫 **No hardcoded paths** — everything resolves from the repo root (`src/config.py`); the DB location can be overridden with `WP2_DB_PATH`.194- 🧬 **Single source of truth** — the extraction logic and the 20 reference texts, previously copy-pasted across five scripts (in two languages), now live in exactly one place each.195- 🔁 **Deterministic & cached** — embeddings are deterministic for a given model version and cached; `--force` re-encodes from scratch.196- 📋 **Tables are compiled artifacts** — every number in the paper's statistical tables is written by `06_tables.py` from the results CSVs. The original project transcribed numbers by hand, which introduced errors that this restructuring caught and fixed (documented, with old→new values, in `CHANGES.md` §4).197198---199200## 💾 Data201202The raw data is a **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: **46,479 listings** across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). Each row carries structured fields (price, bedrooms, bathrooms) plus a full JSON blob including the agent-written `PublicRemarks`.203204**The analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. Median listing price $589,900 (mean $807,259, max $25M). Descriptions average 510 characters and are truncated at ~700 characters in the export (disclosed in the paper).205206> ⚠️ **`data/` is not tracked by git**: the raw database exceeds GitHub's 100 MB file limit, contains a leftover application `users` table, and the listing texts are proprietary platform content. `data/README.md` explains how to place a copy of `louka.db` to re-run the pipeline from scratch. All *derived, aggregate* outputs (results CSVs, figures, tables, paper) **are** versioned, so every number in the paper is inspectable without the raw data.207208---209210## 🔄 Reproducing Everything211212```bash213git clone https://github.com/spboucher-ai/wp2_uqo.git214cd wp2_uqo215python3 -m pip install -r requirements.txt216217# place louka.db in data/raw/ (see data/README.md), then:218python3 scripts/01_prepare_data.py # extract the house sample219python3 scripts/02_similarities.py # embed + project (add --force to re-encode)220python3 scripts/03_models.py # hedonic models A–E221python3 scripts/04_robustness.py # full robustness battery (~4 min)222python3 scripts/05_figures.py # all 11 figures223python3 scripts/06_tables.py # LaTeX tables from results224225cd paper && latexmk -pdf main.tex # 53-page PDF, 0 errors226```227228Expected checkpoints along the way:229230- Step 1 prints `17,087 houses with a valid price and description`.231- Step 3 prints the model table with `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`.232- Step 4 prints `Winsorized R2=0.4867` and the Lasso selection (4/20 at the CV penalty).233234---235236## 📝 Building the Paper237238The paper (`paper/`) follows the UQO working-paper format: title page with logo, abstract page with keywords + JEL codes, IMRaD body (Introduction · Literature · Methodology · Data · Results · Robustness · Discussion · Conclusion), `natbib`/`apalike` bibliography, and two appendices (verbatim reference texts; parsimonious-model estimates).239240```bash241cd paper242make # or: latexmk -pdf main.tex243```244245Figures are pulled from `../figures/` via `\graphicspath`, so the paper always reflects the latest pipeline run. The build is clean: **0 errors, 0 undefined references, 0 missing citations** (59/59 bib keys resolve).246247---248249## ✅ Verification & Provenance250251This repository is a restructuring of an earlier research folder. Before anything was rewritten, the pipeline was validated against the original artifacts:252253| Check | Result |254|---|---|255| Analysis dataset (17,087 × 30) | ✅ identical, column by column |256| Similarity matrix (17,087 × 20) | ✅ bit-identical |257| Models A–E fit statistics | ✅ identical to the paper's Table 4 |258| Model D coefficients (27 params) | ✅ identical to the paper's Table 5 |259| Figures fig1, fig4–fig11 | ✅ regenerated, byte-size within ±2 B |260| Figures fig2, fig3 | 🔧 were *missing* from the original repo (placeholder boxes in the old PDF) — regenerated |261| `robustness_results.csv` | 🔧 lost in the original — regenerated (seeded) |262| Descriptive/robustness numbers in the old LaTeX | ⚠️ several did not match the code's own outputs — corrected and now machine-generated; full old→new list in [`CHANGES.md`](CHANGES.md) §4 |263264`AUDIT.md` documents the original project's layout, dead files, and defects; `CHANGES.md` documents every move, refactor, and numeric correction.265266---267268## 🖼️ Figures Gallery269270| # | Figure | What it shows |271|---|---|---|272| 1 | `fig1_model_comparison` | R² / adj. R² across models A–E |273| 2 | `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) |274| 3 | `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features |275| 4 | `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension |276| 5 | `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) |277| 6 | `fig6_scatter_plots` | Four illustrative similarity–price scatters |278| 7 | `fig7_methodology` | Pipeline diagram |279| 8 | `fig8_r2_decomposition` | Stacked decomposition of Model D's R² |280| 9 | `fig9_price_distribution` | Price and log-price histograms |281| 10 | `fig10_structural_coefficients` | Structural coefficients with 95 % CIs |282| 11 | `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q |283284All figures exist as publication PDF (vector) and PNG (300 dpi).285286---287288## 📦 Requirements289290| Package | Version | Used for |291|---|---|---|292| Python | 3.14 | — |293| numpy | 2.4.4 | linear algebra |294| pandas | 3.0.2 | data wrangling |295| pyarrow | 24.0.0 | parquet I/O |296| scipy | 1.17.1 | tests, distributions |297| scikit-learn | 1.6.1 | PCA, Lasso/ElasticNet CV, scaling |298| statsmodels | 0.14.6 | OLS/HC3, QuantReg, BP test, VIF |299| matplotlib | 3.10.9 | figures |300| sentence-transformers | 5.5.0 | embeddings |301| torch | 2.12.0 | transformer backend (MPS on Apple Silicon) |302303LaTeX: TeX Live 2026 with `latexmk` (newtx, booktabs, threeparttable, natbib, hyperref…).304305---306307## 📚 Citation308309```bibtex310@techreport{boucher2026decoding,311 author = {Boucher, Simon-Pierre},312 title = {Decoding Real Estate Descriptions: Semantic Embeddings and313 Hedonic Pricing of Residential Properties in Quebec},314 institution = {Universit\'e du Qu\'ebec en Outaouais,315 D\'epartement des sciences administratives},316 type = {Working Paper},317 number = {2},318 year = {2026},319 month = {May}320}321```322323---324325## 👤 Author & Contact326327**Simon-Pierre Boucher**328Département des sciences administratives329Université du Québec en Outaouais (UQO)330283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada331332📧 **contact@spboucher.ai**333334*Comments and suggestions are welcome. All errors are my own.*335336---337338<div align="center">339<sub>© 2026 Simon-Pierre Boucher. Paper, code, and figures — all rights reserved.</sub>340</div>341