TL;DR — Listing descriptions written by real estate agents contain price-relevant information that classic hedonic variables miss. This project embeds the free-text PublicRemarks of 17,087 Quebec single-family house listings with a sentence transformer, projects each embedding onto 20 researcher-defined semantic reference descriptions (Luxury, Needs Renovation, Waterfront, Motivated Seller, …) via cosine similarity, and adds those 20 interpretable scores to a log-price hedonic OLS model. Adjusted R² rises from 0.452 to 0.511 (joint F = 99.53, p < 0.001), and each semantic dimension carries a named, signed, economically meaningful implicit price.

# 📖 Table of Contents

• Overview
• Key Results
• Methodology
• Repository Structure
• The Pipeline, Script by Script
• Data
• Reproducing Everything
• Building the Paper
• Verification & Provenance
• Figures Gallery
• Requirements
• Citation
• Author & Contact

# 🔍 Overview

Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, bathrooms, lot size. But the narrative of a listing ("cuisine gastronomique, comptoirs de quartz…" vs. "besoin de rénovation, vendu tel quel…") carries quality information that no structured field captures.

The catch, historically, is a depth-vs-interpretability trade-off:

The idea: instead of feeding 384 anonymous embedding dimensions into a regression, define 20 short reference descriptions — synthetic French paragraphs, each embodying one qualitative housing dimension — embed them with the same model, and use the cosine similarity between each listing and each reference as 20 named regressors. Every coefficient then reads directly as "the implicit price of sounding more luxury / more fixer-upper / more waterfront", per standard deviation.

This is conceptually analogous to factor-mimicking portfolios in asset pricing, and to concept-bottleneck models in interpretable ML: the deep representation is channeled through human-named concepts before inference.

# 📊 Key Results

# Model comparison (n = 17,087)

Joint significance of the 21 text variables (D vs. A): F = 99.53, p < 0.001. Variance decomposition of Model D: structural 88.3 %, text length 2.4 %, semantics 9.3 %.

# Implicit semantic price gradients (Model D, per +1 SD, HC3 errors)

_{*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (see paper §6). Impact = (e^β − 1) × 100 %.}

# Heterogeneity & robustness (highlights)

• Quantile regressions (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from 9.7 % at the 25th percentile to 17.3 % at the 75th; urgency and condition discounts attenuate at the top of the market.
• Bootstrap (1,000 replications, seed 42): SEs within 9 % of HC3 (15/20 within 5 %).
• Outlier trimming (1st/99th price percentiles): no sign reversal, all 16 significant dimensions stay significant.
• Multicollinearity is real and disclosed: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2). Block-level inference is unaffected; individual coefficients are partial associations within a correlated block.
• PCA benchmark: 20 principal components of the raw embeddings fit better (ΔR² +0.070 vs. +0.044) but are economically unreadable — the paper quantifies the interpretability-vs-fit trade-off explicitly.

# 🧪 Methodology

┌─────────────┐   ┌──────────────┐   ┌──────────────┐   ┌─────────────┐   ┌──────────────┐
│  Property   │   │   Sentence   │   │    Cosine    │   │   Hedonic   │   │   Implicit   │
│  Listings   ├──▶│  Embeddings  ├──▶│  Similarity  ├──▶│  OLS Model  ├──▶│    Price     │
│ (n=17,087)  │   │   (384-d)    │   │  (20 refs)   │   │ (HC3 s.e.)  │   │  Estimates   │
└─────────────┘   └──────────────┘   └──────────────┘   └─────────────┘   └──────────────┘
 PublicRemarks     all-MiniLM-L6-v2    S = E · Rᵀ          log(P) = α + β'X + δℓ + γ's + ε

1. Extraction — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (the transformer handles raw text).
2. Embedding — all-MiniLM-L6-v2 (22.7M params, 384-d, L2-normalized), batch 256; the whole corpus encodes in seconds on Apple Silicon.
3. Projection — 20 French reference descriptions (146–257 chars each, full verbatim text in paper Appendix A), designed for semantic saturation, dimensional specificity, linguistic consistency. One matrix product yields the 17,087 × 20 similarity matrix.
4. Estimation — nested OLS models A–E on log price, all covariates standardized, HC3 robust errors; joint F-tests; quantile regressions; bootstrap; winsorization; VIF; Lasso/Elastic-Net selection; PCA benchmark.

The 20 dimensions span six domains: Quality & Standing (Luxury, Entry-Level) · Condition (Renovated, Needs Renovation) · Physical (Bright & Spacious, Garage & Parking, Finished Basement, Pool & Landscaping) · Location & Setting (Land & Nature, Panoramic View, Premium Location, Quiet & Peaceful, Waterfront) · Style & Character (Modern/Contemporary, Heritage/Character, New Construction, Energy Efficient, Family-Friendly) · Market Signals (Income/Investment, Motivated Seller).

# 📁 Repository Structure

wp2_uqo/
├── README.md                ← you are here
├── AUDIT.md                 ← forensic audit of the original (pre-restructuring) project
├── CHANGES.md               ← everything that was moved / refactored / corrected, with diffs
├── PAPER_REVIEW.md          ← critical pre-upgrade assessment of the paper's scholarship
├── UPGRADE_REPORT.md        ← literature expansion: refs added/replaced, claims flagged
├── requirements.txt         ← pinned dependencies (exact versions used for verification)
│
├── data/                    ← NOT tracked by git (see data/README.md)
│   ├── raw/louka.db         │  SQLite, 268 MB, 46,479 listings (5 categories)
│   └── processed/           │  parquet sample, embeddings cache, similarity matrix,
│       └── legacy/          │  analysis CSV + pre-refactor artifacts kept for comparison
│
├── src/                     ← analysis package (imported by scripts/)
│   ├── config.py            │  repo-relative paths, model name, seed, structural vars
│   ├── references.py        │  the 20 French reference descriptions + English labels
│   ├── data.py              │  SQLite → DataFrame extraction (n = 17,087)
│   ├── embeddings.py        │  encoding, caching, cosine-similarity features
│   └── models.py            │  standardized-design OLS A–E, HC3
│
├── scripts/                 ← numbered pipeline entry points (run in order)
│   ├── 01_prepare_data.py   │  louka.db → data/processed/houses.parquet
│   ├── 02_similarities.py   │  embeddings (cached) + 20 similarity features → CSV
│   ├── 03_models.py         │  models A–E → results/*.csv
│   ├── 04_robustness.py     │  VIF · BP · quantile · winsor · bootstrap · Lasso/ENet
│   ├── 05_figures.py        │  figures fig1–fig11 (PDF + PNG)
│   └── 06_tables.py         │  LaTeX tables generated from results (no hand transcription)
│
├── figures/                 ← all 11 paper figures, regenerated from the pipeline
├── results/                 ← model & robustness outputs (CSV, versioned)
│
└── paper/                   ← LaTeX source (UQO working-paper style)
    ├── main.tex             │  metadata + section inputs (53-page PDF)
    ├── preamble.tex         │  packages, layout, custom macros
    ├── main.pdf             │  compiled paper (committed for convenience)
    ├── Makefile             │  latexmk build
    ├── references.bib       │  86 entries, every one verified (OpenAlex + DOI)
    ├── sections/            │  titlepage · intro · literature · methodology · data ·
    │                        │  results · robustness · discussion · conclusion · appendix
    └── tables/              │  7 machine-generated + 3 hand-maintained booktabs tables

# ⚙️ The Pipeline, Script by Script

_{*Apple M-series laptop. All stochastic steps are seeded (seed = 42): bootstrap resampling, CV folds, scatter subsampling.}

Design principles applied during the refactor (see CHANGES.md for the full story):

• 🚫 No hardcoded paths — everything resolves from the repo root (src/config.py); the DB location can be overridden with WP2_DB_PATH.
• 🧬 Single source of truth — the extraction logic and the 20 reference texts, previously copy-pasted across five scripts (in two languages), now live in exactly one place each.
• 🔁 Deterministic & cached — embeddings are deterministic for a given model version and cached; --force re-encodes from scratch.
• 📋 Tables are compiled artifacts — every number in the paper's statistical tables is written by 06_tables.py from the results CSVs. The original project transcribed numbers by hand, which introduced errors that this restructuring caught and fixed (documented, with old→new values, in CHANGES.md §4).

# 💾 Data

The raw data is a SQLite database (louka.db, 268 MB) of Quebec residential listings collected from Realtor.ca / Centris: 46,479 listings across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). Each row carries structured fields (price, bedrooms, bathrooms) plus a full JSON blob including the agent-written PublicRemarks.

The analysis sample: single-family houses with a positive price and a description of ≥ 20 characters → n = 17,087. Median listing price $589,900 (mean $807,259, max $25M). Descriptions average 510 characters and are truncated at ~700 characters in the export (disclosed in the paper).

⚠️ data/ is not tracked by git: the raw database exceeds GitHub's 100 MB file limit, contains a leftover application users table, and the listing texts are proprietary platform content. data/README.md explains how to place a copy of louka.db to re-run the pipeline from scratch. All derived, aggregate outputs (results CSVs, figures, tables, paper) are versioned, so every number in the paper is inspectable without the raw data.

# 🔄 Reproducing Everything

git clone https://github.com/spboucher-ai/wp2_uqo.git
cd wp2_uqo
python3 -m pip install -r requirements.txt

# place louka.db in data/raw/ (see data/README.md), then:
python3 scripts/01_prepare_data.py     # extract the house sample
python3 scripts/02_similarities.py     # embed + project (add --force to re-encode)
python3 scripts/03_models.py           # hedonic models A–E
python3 scripts/04_robustness.py       # full robustness battery (~4 min)
python3 scripts/05_figures.py          # all 11 figures
python3 scripts/06_tables.py           # LaTeX tables from results

cd paper && latexmk -pdf main.tex      # 53-page PDF, 0 errors

Expected checkpoints along the way:

• Step 1 prints 17,087 houses with a valid price and description.
• Step 3 prints the model table with R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118 and F = 99.53.
• Step 4 prints Winsorized R2=0.4867 and the Lasso selection (4/20 at the CV penalty).

# 📝 Building the Paper

The paper (paper/) follows the UQO working-paper format: title page with logo, abstract page with keywords + JEL codes, IMRaD body (Introduction · Literature · Methodology · Data · Results · Robustness · Discussion · Conclusion), natbib/apalike bibliography, and two appendices (verbatim reference texts; parsimonious-model estimates).

cd paper
make            # or: latexmk -pdf main.tex

Figures are pulled from ../figures/ via \graphicspath, so the paper always reflects the latest pipeline run. The build is clean: 0 errors, 0 undefined references, 0 missing citations (59/59 bib keys resolve).

# ✅ Verification & Provenance

This repository is a restructuring of an earlier research folder. Before anything was rewritten, the pipeline was validated against the original artifacts:

AUDIT.md documents the original project's layout, dead files, and defects; CHANGES.md documents every move, refactor, and numeric correction.

# 🖼️ Figures Gallery

All figures exist as publication PDF (vector) and PNG (300 dpi).

# 📦 Requirements

LaTeX: TeX Live 2026 with latexmk (newtx, booktabs, threeparttable, natbib, hyperref…).

# 📚 Citation

@techreport{boucher2026decoding,
  author      = {Boucher, Simon-Pierre},
  title       = {Decoding Real Estate Descriptions: Semantic Embeddings and
                 Hedonic Pricing of Residential Properties in Quebec},
  institution = {Universit\'e du Qu\'ebec en Outaouais,
                 D\'epartement des sciences administratives},
  type        = {Working Paper},
  number      = {2},
  year        = {2026},
  month       = {May}
}

# 👤 Author & Contact

Simon-Pierre Boucher Département des sciences administratives Université du Québec en Outaouais (UQO) 283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada

📧 contact@spboucher.ai

Comments and suggestions are welcome. All errors are my own.