Initial commit: semantic hedonic pricing pipeline, figures, and paper (UQO WP no. 2)
Restructured from immo-wp2-spb-20260519: modular src/ package, numbered reproduction pipeline (01-06), machine-generated LaTeX tables, regenerated figures (incl. previously missing fig2/fig3), corrected paper statistics (see CHANGES.md), 47-page PDF compiling clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
72 changed files +4,180 −0
added
.gitignore
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +# Data — too large / sensitive for git (see data/README.md) | |
| 2 | +data/* | |
| 3 | +!data/README.md | |
| 4 | + | |
| 5 | +# LaTeX build artifacts | |
| 6 | +paper/*.aux | |
| 7 | +paper/*.log | |
| 8 | +paper/*.out | |
| 9 | +paper/*.toc | |
| 10 | +paper/*.bbl | |
| 11 | +paper/*.blg | |
| 12 | +paper/*.fls | |
| 13 | +paper/*.fdb_latexmk | |
| 14 | +paper/*.synctex.gz | |
| 15 | + | |
| 16 | +# macOS | |
| 17 | +.DS_Store | |
| 18 | + | |
| 19 | +# Python | |
| 20 | +__pycache__/ | |
| 21 | +*.pyc | |
added
AUDIT.md
+126 −0
@@ -0,0 +1,126 @@ | ||
| 1 | +# AUDIT — immo-wp2-spb-20260519 | |
| 2 | + | |
| 3 | +Audit of the original project at `~/Desktop/UQO/UQO_WP/immo-wp2-spb-20260519` | |
| 4 | +(354 MB), performed 2026-08-05 before restructuring into `~/Desktop/wp2_uqo`. | |
| 5 | +The original folder is left untouched and serves as the backup. | |
| 6 | + | |
| 7 | +## 1. What the project is | |
| 8 | + | |
| 9 | +Working Paper No. 2 (UQO): *"Decoding Real Estate Descriptions: Semantic | |
| 10 | +Embeddings and Hedonic Pricing of Residential Properties in Quebec"* | |
| 11 | +(Simon-Pierre Boucher). The analysis embeds the free-text `PublicRemarks` of | |
| 12 | +17,087 Quebec single-family house listings with `all-MiniLM-L6-v2` | |
| 13 | +(sentence-transformers), computes cosine similarities to 20 researcher-defined | |
| 14 | +French reference descriptions ("luxury", "needs renovation", …), and adds | |
| 15 | +those 20 similarity features to a log-price hedonic OLS model (models A–E, | |
| 16 | +HC3 errors), plus robustness checks (VIF, Breusch–Pagan, quantile regression, | |
| 17 | +winsorization, 1,000-rep bootstrap, Lasso/Elastic Net). | |
| 18 | + | |
| 19 | +Headline numbers (paper, model D vs A): adjusted R² 0.452 → 0.511, | |
| 20 | +F = 99.53 (p < 0.001), n = 17,087. | |
| 21 | + | |
| 22 | +## 2. Data | |
| 23 | + | |
| 24 | +| File | Size | Role | | |
| 25 | +|---|---|---| | |
| 26 | +| `louka.db` | 268 MB | **Raw data.** SQLite; table `properties` (id, mls_number, category, price_value, bedrooms, bathrooms, data JSON blob). Counts: house 17,089, condo 8,586, land 8,506, rent 8,240, plex 4,058. Sample = houses with price > 0 and remarks ≥ 20 chars → 17,087. | | |
| 27 | +| `hedonic_maison_results.csv` | 17,087 rows | Processed: houses + 20 similarity features (main analysis dataset). | | |
| 28 | +| `hedonic_results.csv` | 29,731 rows | Processed (earlier PCA approach, house/condo/plex). | | |
| 29 | +| `hedonic_cosine_results.csv` | 29,731 rows | Processed (earlier 16-reference approach, all categories). | | |
| 30 | +| `embeddings_maisons.npy` | 25 MB | 17,087 × 384 embeddings (houses; used by the paper). | | |
| 31 | +| `embeddings_remarks.npy` | 44 MB | 29,731 × 384 embeddings (all categories; earlier approach). | | |
| 32 | +| `sim_matrix_maisons.npy` | 1.3 MB | 17,087 × 20 similarity matrix (paper). | | |
| 33 | +| `similarity_matrix.npy` | 1.8 MB | 29,731 × 16 (earlier approach). | | |
| 34 | +| `reference_embeddings.npz` | 43 KB | 16 reference embeddings (earlier approach). | | |
| 35 | + | |
| 36 | +⚠️ Notes: | |
| 37 | +- `louka.db` also contains a `users` table with **plaintext usernames/passwords** | |
| 38 | + (application leftover). It is not used by any script. Not copied onward. | |
| 39 | +- All scripts hardcode `DB_PATH = "/Users/simon-pierreboucher/Desktop/d/louka.db"` | |
| 40 | + and write outputs to `~/Desktop/d/` — a folder that **no longer exists**. The | |
| 41 | + project copy of `louka.db` is what remains; outputs were copied into the | |
| 42 | + project at some point. | |
| 43 | +- `robustness_results.csv` (written by `hedonic_maison.py` step 12) was saved | |
| 44 | + to `~/Desktop/d/` and is **not present anywhere** — regenerated in the new repo. | |
| 45 | + | |
| 46 | +## 3. Scripts (chronological / by role) | |
| 47 | + | |
| 48 | +| Script | Role | Outputs | Status | | |
| 49 | +|---|---|---|---| | |
| 50 | +| `hedonic_semantic.py` (348 l.) | Exploration #1: PCA(20) of embeddings, house/condo/plex, models 1–4 | `hedonic_results.csv`, `embeddings_remarks.npy` | Superseded; not used by the paper | | |
| 51 | +| `hedonic_cosine_references.py` (447 l.) | Exploration #2: 16 reference similarities, all categories, models A–C | `hedonic_cosine_results.csv`, `reference_embeddings.npz`, `similarity_matrix.npy` | Superseded; not used by the paper | | |
| 52 | +| `hedonic_maison.py` (818 l.) | **Main analysis**: 20 references, houses only, models A–E + all robustness checks | `hedonic_maison_results.csv`, `embeddings_maisons.npy`, `sim_matrix_maisons.npy`, `robustness_results.csv` (lost) | **Paper source** (tables 1–8) | | |
| 53 | +| `paper_figures.py` (578 l.) | Figures v1 (figs 1–8) + `table1–4_*.csv` | `figures/` | Superseded by v2 for figures; still the only producer of the 4 table CSVs | | |
| 54 | +| `paper_figures_v2.py` (386 l.) | **Figures v2 (final)**: figs 1–11, tighter layout | `figures/fig1…fig11` | **Paper source** (all figures) | | |
| 55 | + | |
| 56 | +Duplication: the DB-extraction block (~70 lines) and the OLS-fit helper are | |
| 57 | +copy-pasted in all 5 scripts, with small drifts (`half_baths` and lat/lon only | |
| 58 | +in some; 16 vs 20 references; French vs English reference names). The 20 | |
| 59 | +reference descriptions are pasted 3× (French keys in `hedonic_maison.py`, | |
| 60 | +English display keys in both figure scripts — same French texts). | |
| 61 | + | |
| 62 | +## 4. Figures | |
| 63 | + | |
| 64 | +`figures/` and `wp2/figures/` (identical copies) contain fig1, fig4–fig11 | |
| 65 | +(PDF, some PNG) + `table1–4_*.csv`. | |
| 66 | + | |
| 67 | +**Missing: `fig2_coefficient_plot.pdf` and `fig3_similarity_distributions.pdf`**, | |
| 68 | +although both papers reference them (`sections/results.tex`, `sections/data.tex`). | |
| 69 | +`wp2/main_web.tex` silently masks this with a custom `\includegraphics` | |
| 70 | +fallback that prints a "Figure indisponible" box — the compiled | |
| 71 | +`wp2/main_web.pdf` therefore ships with two placeholder boxes instead of | |
| 72 | +Figures 2 (semantic coefficient forest plot) and 3 (similarity distributions). | |
| 73 | +Both are regenerated by `paper_figures_v2.py`. **This is the main defect fixed | |
| 74 | +by the restructuring.** | |
| 75 | + | |
| 76 | +Figure → script map (all from `paper_figures_v2.py`): | |
| 77 | +fig1 model comparison; fig2 semantic coefficient forest plot (was missing); | |
| 78 | +fig3 similarity boxplots (was missing); fig4 price-quintile heatmap; | |
| 79 | +fig5 similarity correlation matrix; fig6 scatter plots (4 dims); | |
| 80 | +fig7 methodology diagram; fig8 R² decomposition; fig9 price distributions; | |
| 81 | +fig10 structural coefficients; fig11 residual diagnostics. | |
| 82 | + | |
| 83 | +## 5. LaTeX | |
| 84 | + | |
| 85 | +Two generations coexist: | |
| 86 | + | |
| 87 | +- **Root (obsolete, May 20)**: monolithic `paper.tex` (112 KB, first draft), | |
| 88 | + then `main.tex` + `preamble.tex` + `titlepage.tex` + `sections/` + `tables/` | |
| 89 | + + `appendix/` + `references.bib`. | |
| 90 | +- **`wp2/` (canonical, May 20 – June 13)**: `main.tex` (self-contained | |
| 91 | + preamble, UQO WP style), `main_web.tex` (same + figure-fallback hack, source | |
| 92 | + of the compiled `main_web.pdf`), `sections/` (9 files incl. titlepage), | |
| 93 | + `tables/` (8 hand-written booktabs tables), `appendix/appendix.tex`, | |
| 94 | + `references.bib` (551 lines, ~40 entries), `Makefile`, `uq_logo.jpg`. | |
| 95 | + | |
| 96 | +`diff -r` confirms root `sections/`, `tables/`, `appendix/`, `references.bib` | |
| 97 | +are **byte-identical** to the `wp2/` ones (root just lacks `titlepage.tex`). | |
| 98 | +So the root LaTeX tree is a pure duplicate → dropped. | |
| 99 | + | |
| 100 | +Numbers in `tables/*.tex` were hand-transcribed from `hedonic_maison.py` | |
| 101 | +console output (no automated table generation). | |
| 102 | + | |
| 103 | +## 6. Dead / duplicate / unused files | |
| 104 | + | |
| 105 | +- `paper.tex`, root `main.tex`, `preamble.tex`, `titlepage.tex`, root | |
| 106 | + `sections/`, `tables/`, `appendix/`, root `figures/` — superseded duplicates. | |
| 107 | +- Build artifacts: `*.aux, *.log, *.out, *.toc, *.bbl, *.blg, *.fls, | |
| 108 | + *.fdb_latexmk, *.synctex.gz` at root and in `wp2/`. | |
| 109 | +- `.DS_Store` files. | |
| 110 | +- Earlier-approach outputs (`hedonic_results.csv`, `hedonic_cosine_results.csv`, | |
| 111 | + `embeddings_remarks.npy`, `similarity_matrix.npy`, `reference_embeddings.npz`) | |
| 112 | + are kept in `data/processed/legacy/` for traceability but are not part of the | |
| 113 | + pipeline. | |
| 114 | + | |
| 115 | +## 7. Discrepancies & reproduction status | |
| 116 | + | |
| 117 | +- **fig2/fig3 missing** in the original (see §4) — regenerated here. | |
| 118 | +- **`robustness_results.csv` lost** in the original — regenerated here. | |
| 119 | +- Reproduction check (new pipeline vs original artifacts) — results recorded | |
| 120 | + in `CHANGES.md` §Verification: | |
| 121 | + - similarity features vs `hedonic_maison_results.csv` (17,087 × 20), | |
| 122 | + - model A–E statistics vs `tables/tab_model_comparison.tex` | |
| 123 | + (R² 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118), | |
| 124 | + - regenerated figures vs shipped fig1, fig4–fig11. | |
| 125 | +- Bootstrap (`np.random.seed(42)`) and Lasso/ENet CV (`random_state=42`) are | |
| 126 | + seeded, so robustness numbers are reproducible on the same package versions. | |
added
CHANGES.md
+121 −0
@@ -0,0 +1,121 @@ | ||
| 1 | +<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai --> | |
| 2 | + | |
| 3 | +# CHANGES — restructuring of immo-wp2-spb-20260519 → wp2_uqo | |
| 4 | + | |
| 5 | +Original project: `~/Desktop/UQO/UQO_WP/immo-wp2-spb-20260519` (untouched, serves as backup). | |
| 6 | +New repository: `~/Desktop/wp2_uqo`. Date: 2026-08-05. | |
| 7 | + | |
| 8 | +## 1. What was moved / renamed | |
| 9 | + | |
| 10 | +| Original | New location | | |
| 11 | +|---|---| | |
| 12 | +| `louka.db` | `data/raw/louka.db` | | |
| 13 | +| `embeddings_maisons.npy`, `sim_matrix_maisons.npy` | `data/processed/` (pipeline cache) | | |
| 14 | +| `hedonic_maison_results.csv` (+ earlier-approach CSVs/NPYs) | `data/processed/legacy/` (originals kept for comparison) | | |
| 15 | +| `hedonic_maison.py` (818 lines, monolithic) | split into `src/` modules + `scripts/01–04` | | |
| 16 | +| `paper_figures_v2.py` | `scripts/05_figures.py` (ports all 11 figures) | | |
| 17 | +| `wp2/main.tex` + `wp2/sections/` + `wp2/tables/` | `paper/` (preamble extracted to `paper/preamble.tex`) | | |
| 18 | +| `hedonic_semantic.py`, `hedonic_cosine_references.py`, `paper_figures.py`, root LaTeX tree, `paper.tex` | **not migrated** (superseded; documented in `AUDIT.md`) | | |
| 19 | + | |
| 20 | +## 2. Code refactoring | |
| 21 | + | |
| 22 | +- All hardcoded paths to the defunct `~/Desktop/d/` replaced by repository-relative | |
| 23 | + paths in `src/config.py` (overridable via `WP2_DB_PATH`). | |
| 24 | +- The DB-extraction block that was copy-pasted across 5 scripts now lives once in | |
| 25 | + `src/data.py`; the 20 reference descriptions (pasted 3×, in two languages) live | |
| 26 | + once in `src/references.py` (slugs for data columns, English labels for display). | |
| 27 | +- Models A–E defined once in `src/models.py`; robustness checks ported faithfully | |
| 28 | + (same estimators, same seeds: bootstrap `np.random.seed(42)`, CV `random_state=42`). | |
| 29 | +- New numbered pipeline: `01_prepare_data` → `02_similarities` → `03_models` → | |
| 30 | + `04_robustness` → `05_figures` → `06_tables`. Everything runs end-to-end from | |
| 31 | + `data/raw/louka.db`; embeddings are cached and `--force` re-encodes. | |
| 32 | +- **New `scripts/06_tables.py` generates the paper's numeric tables directly from | |
| 33 | + the pipeline results** — this closes the transcription gap that produced the | |
| 34 | + wrong numbers documented in §4 below. | |
| 35 | +- `requirements.txt` pins the exact package versions used for verification. | |
| 36 | + | |
| 37 | +## 3. Verification (regenerated vs. original artifacts) | |
| 38 | + | |
| 39 | +- Sample: 17,087 houses — **identical** (all 30 columns of | |
| 40 | + `hedonic_maison_results.csv` equal to the original; similarity matrix | |
| 41 | + bit-identical). | |
| 42 | +- Models A–E: R², adj. R², AIC, BIC, k — **identical to the paper's Table 4** | |
| 43 | + (e.g. R² 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118; joint F = 99.53). | |
| 44 | +- Model D coefficients: **identical to the paper's Table 5** (all 27 coefficients, | |
| 45 | + SEs, t-stats, impacts). | |
| 46 | +- Figures: regenerated fig1, fig4–fig11 match the shipped ones (same content, | |
| 47 | + PDF sizes within ±2 bytes). **fig2 and fig3, referenced by the paper but missing | |
| 48 | + from the original repo (the compiled PDF showed "Figure indisponible" boxes), | |
| 49 | + are regenerated and now included.** | |
| 50 | +- `robustness_results.csv`, lost in the original, is regenerated (seeded, reproducible). | |
| 51 | + | |
| 52 | +## 4. Numbers corrected in the paper ⚠️ (review recommended) | |
| 53 | + | |
| 54 | +The original LaTeX contained statistics that **do not match the actual pipeline | |
| 55 | +outputs** (the project's own `figures/table1/table2` CSVs confirm the pipeline | |
| 56 | +values). They appear to have been transcribed from memory/drafted text rather | |
| 57 | +than from the analysis output. All were replaced by the reproducible values; | |
| 58 | +the paper's tables are now machine-generated (`scripts/06_tables.py`). | |
| 59 | + | |
| 60 | +| Claim | Old (paper) | New (verified) | | |
| 61 | +|---|---|---| | |
| 62 | +| Price mean | $764,508 | **$807,259** (median $589,900 unchanged) | | |
| 63 | +| Description length | mean 683, sd 367, max 5,399 | **mean 510, sd 151, max 703** (field truncated ~700 chars — now noted in the text) | | |
| 64 | +| Parking mean | 2.3 | **5.5** (counts all spots incl. driveway — now explained) | | |
| 65 | +| log-price sd | 0.68 | **0.70** | | |
| 66 | +| Similarity stats & correlations (Table 3) | e.g. Luxury corr −0.178 | **regenerated** (e.g. Luxury −0.234; all in [−0.29, −0.17]) | | |
| 67 | +| Breusch–Pagan | LM = 2,847.3 | **LM = 768.0** (same conclusion) | | |
| 68 | +| VIF | "none > 10, mean ≈ 4.2" | **mean 17.2, 16/20 > 10** — subsection rewritten honestly (block-level inference unaffected; O'Brien 2007 argument) | | |
| 69 | +| Inter-reference correlations | "0.4–0.8", "Motivated Seller lowest" | **0.66–0.97, mean 0.84**; top pair Entry-Level/Motivated Seller r = 0.97 | | |
| 70 | +| Winsorized model | 342 obs dropped, adj R² 0.498, shifts Luxury −0.012 / Motivated +0.008 | **341 obs, adj R² 0.486, largest shifts Entry-Level −0.024 / Motivated Seller +0.023** (no sign reversals, all 16 stay significant — unchanged conclusions) | | |
| 71 | +| Bootstrap | "SE within 5% of HC3" | **within 9% (15/20 within 5%)** | | |
| 72 | +| Lasso / Elastic Net | "Lasso keeps 15/20, ENet 17/20, high concordance with OLS" | **both keep 4/20** — subsection rewritten: sparse block-representative selection under strong collinearity, complementary rather than concordant | | |
| 73 | +| Quantile table | e.g. Luxury 0.098/0.125/0.176; "Modern & Land increasing" | **regenerated** (Luxury 0.093/0.130/0.160 increasing ✓; Modern high at both tails; Land & Nature stable) | | |
| 74 | +| PCA comparison | "PCA yields R² 0.496 < 0.512 → reference approach outperforms" | **PCA(20) actually fits better: ΔR² +0.070 vs +0.044** (49.5 % variance ✓). Reframed as an explicit interpretability-vs-fit trade-off (Table 8 + Discussion §PCA rewritten) | | |
| 75 | + | |
| 76 | +Qualitative conclusions that survive unchanged: heteroskedasticity → HC3; | |
| 77 | +no sign reversals under trimming; bootstrap ≈ HC3; Luxury premium increasing in | |
| 78 | +quantiles; discounts attenuating; joint semantic block highly significant. | |
| 79 | +Claims that had to change substance: **VIF/multicollinearity** (now acknowledged | |
| 80 | +as substantial), **Lasso concordance** (now honest about sparse selection), and | |
| 81 | +**PCA fit comparison** (now favors PCA on fit, reference approach on | |
| 82 | +interpretability). These are flagged because they alter the robustness | |
| 83 | +narrative, not the headline results. | |
| 84 | + | |
| 85 | +## 5. Paper rewrite (Phase 3) | |
| 86 | + | |
| 87 | +- `paper/main.tex` slimmed (metadata + inputs); packages and macros extracted to | |
| 88 | + `paper/preamble.tex`; figures pulled from `../figures/` via `\graphicspath`; | |
| 89 | + the `main_web.tex` variant with the silent figure-fallback hack was dropped. | |
| 90 | +- All **[TODO] blocks removed** (robustness §7.7–7.9, methodology, summary table) | |
| 91 | + and converted into completed analyses (PCA comparison), honest limitations | |
| 92 | + (Discussion), or future-research items (Conclusion). | |
| 93 | +- New appendix with real content: **A. verbatim French reference descriptions** | |
| 94 | + (replication) and **B. Model E coefficient table** — both machine-generated. | |
| 95 | +- Editorial pass on Data/Results/Robustness/Discussion: corrected figure-3 caption | |
| 96 | + (boxplots, not KDE), documented the description-truncation and parking-field | |
| 97 | + quirks, tightened the collinearity narrative, consistent notation preserved. | |
| 98 | +- Fig 10 y-labels now use readable names instead of raw variable names. | |
| 99 | +- Title page: "First draft: May 2025" corrected to **May 2026** (the data were | |
| 100 | + scraped January 2026); version bumped to 1.1. | |
| 101 | +- Abstract: "robust to bootstrap inference, winsorization, and Lasso selection" | |
| 102 | + → "robust to bootstrap inference and outlier trimming" (accurate). | |
| 103 | +- Compiles clean: `latexmk -pdf main.tex` → **47 pages, 0 errors, 0 undefined | |
| 104 | + references/citations** (24 minor overfull hboxes). Bibliography complete | |
| 105 | + (59/59 cited keys resolve). | |
| 106 | + | |
| 107 | +## 6. Items for your review | |
| 108 | + | |
| 109 | +1. **§4 corrections** — especially VIF, Lasso, and PCA, which change the | |
| 110 | + robustness narrative (the old text claimed low collinearity and high | |
| 111 | + selection concordance; the data say otherwise). Verify you are comfortable | |
| 112 | + with the new framing before circulating. | |
| 113 | +2. `data/raw/louka.db` contains a `users` table with plaintext credentials | |
| 114 | + (application leftover, unused by the analysis). Consider stripping it: | |
| 115 | + `sqlite3 data/raw/louka.db "DROP TABLE users; VACUUM;"`. | |
| 116 | +3. The description-length truncation (~700 chars) is now disclosed in the paper; | |
| 117 | + if a non-truncated export exists, re-running the pipeline on it would be easy. | |
| 118 | +4. The quantile-regression standard errors use the statsmodels kernel-based | |
| 119 | + estimator (the old table note claimed bootstrap SEs, which was never the case). | |
| 120 | +5. Original folder `immo-wp2-spb-20260519` was left untouched; delete or archive | |
| 121 | + it once you're satisfied with `wp2_uqo`. | |
added
README.md
+337 −0
@@ -0,0 +1,337 @@ | ||
| 1 | +<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai --> | |
| 2 | + | |
| 3 | +<div align="center"> | |
| 4 | + | |
| 5 | +# 🏠 Decoding Real Estate Descriptions | |
| 6 | + | |
| 7 | +### Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec | |
| 8 | + | |
| 9 | +**UQO Working Paper No. 2** | |
| 10 | + | |
| 11 | +[](mailto:contact@spboucher.ai) | |
| 12 | +[](mailto:contact@spboucher.ai) | |
| 13 | +[](https://uqo.ca) | |
| 14 | + | |
| 15 | +[](https://www.python.org/) | |
| 16 | +[](https://www.latex-project.org/) | |
| 17 | +[](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | |
| 18 | +[](https://www.statsmodels.org/) | |
| 19 | +[](https://scikit-learn.org/) | |
| 20 | + | |
| 21 | +[](#-reproducing-everything) | |
| 22 | +[](#-data) | |
| 23 | +[](#-key-results) | |
| 24 | +[](paper/main.pdf) | |
| 25 | +[](#-citation) | |
| 26 | + | |
| 27 | +</div> | |
| 28 | + | |
| 29 | +--- | |
| 30 | + | |
| 31 | +> **TL;DR** — Listing descriptions written by real estate agents contain price-relevant information that classic hedonic variables miss. This project embeds the free-text `PublicRemarks` of **17,087 Quebec single-family house listings** with a sentence transformer, projects each embedding onto **20 researcher-defined semantic reference descriptions** (*Luxury*, *Needs Renovation*, *Waterfront*, *Motivated Seller*, …) via cosine similarity, and adds those 20 interpretable scores to a log-price hedonic OLS model. Adjusted R² rises from **0.452 to 0.511** (joint *F* = 99.53, *p* < 0.001), and each semantic dimension carries a named, signed, economically meaningful implicit price. | |
| 32 | + | |
| 33 | +--- | |
| 34 | + | |
| 35 | +## 📖 Table of Contents | |
| 36 | + | |
| 37 | +- [Overview](#-overview) | |
| 38 | +- [Key Results](#-key-results) | |
| 39 | +- [Methodology](#-methodology) | |
| 40 | +- [Repository Structure](#-repository-structure) | |
| 41 | +- [The Pipeline, Script by Script](#-the-pipeline-script-by-script) | |
| 42 | +- [Data](#-data) | |
| 43 | +- [Reproducing Everything](#-reproducing-everything) | |
| 44 | +- [Building the Paper](#-building-the-paper) | |
| 45 | +- [Verification & Provenance](#-verification--provenance) | |
| 46 | +- [Figures Gallery](#-figures-gallery) | |
| 47 | +- [Requirements](#-requirements) | |
| 48 | +- [Citation](#-citation) | |
| 49 | +- [Author & Contact](#-author--contact) | |
| 50 | + | |
| 51 | +--- | |
| 52 | + | |
| 53 | +## 🔍 Overview | |
| 54 | + | |
| 55 | +Hedonic pricing models decompose a property's price into the implicit prices of its characteristics — bedrooms, bathrooms, lot size. But the *narrative* of a listing ("cuisine gastronomique, comptoirs de quartz…" vs. "besoin de rénovation, vendu tel quel…") carries quality information that no structured field captures. | |
| 56 | + | |
| 57 | +The catch, historically, is a **depth-vs-interpretability trade-off**: | |
| 58 | + | |
| 59 | +| Approach | Semantic depth | Economic interpretability | | |
| 60 | +|---|---|---| | |
| 61 | +| Keyword counts / bag-of-words | ❌ shallow | ✅ high | | |
| 62 | +| Sentiment scores | ❌ one dimension | ✅ high | | |
| 63 | +| LDA topics | 🟡 moderate | ❌ unstable | | |
| 64 | +| Raw BERT/transformer embeddings | ✅ deep | ❌ opaque (384–768 anonymous dims) | | |
| 65 | +| **Reference-based cosine projection (this paper)** | ✅ **deep** | ✅ **high** | | |
| 66 | + | |
| 67 | +**The idea:** instead of feeding 384 anonymous embedding dimensions into a regression, define 20 short *reference descriptions* — synthetic French paragraphs, each embodying one qualitative housing dimension — embed them with the same model, and use the **cosine similarity between each listing and each reference** as 20 named regressors. Every coefficient then reads directly as "the implicit price of sounding more *luxury* / more *fixer-upper* / more *waterfront*", per standard deviation. | |
| 68 | + | |
| 69 | +This is conceptually analogous to factor-mimicking portfolios in asset pricing, and to concept-bottleneck models in interpretable ML: the deep representation is channeled through human-named concepts before inference. | |
| 70 | + | |
| 71 | +--- | |
| 72 | + | |
| 73 | +## 📊 Key Results | |
| 74 | + | |
| 75 | +### Model comparison (n = 17,087) | |
| 76 | + | |
| 77 | +| Model | Specification | R² | Adj. R² | AIC | BIC | k | | |
| 78 | +|:---:|---|---:|---:|---:|---:|---:| | |
| 79 | +| A | Structural only | 0.4525 | 0.4523 | 26,165 | 26,219 | 6 | | |
| 80 | +| B | A + description length | 0.4647 | 0.4645 | 25,779 | 25,841 | 7 | | |
| 81 | +| C | A + 20 semantic similarities | 0.4962 | 0.4954 | 24,783 | 24,992 | 26 | | |
| 82 | +| **D** | **Full (B + C)** | **0.5122** | **0.5115** | **24,232** | 24,448 | 27 | | |
| 83 | +| E | Parsimonious (16 significant dims) | 0.5118 | 0.5111 | 24,239 | **24,425** | 23 | | |
| 84 | + | |
| 85 | +Joint significance of the 21 text variables (D vs. A): ***F* = 99.53, *p* < 0.001**. Variance decomposition of Model D: structural 88.3 %, text length 2.4 %, semantics 9.3 %. | |
| 86 | + | |
| 87 | +### Implicit semantic price gradients (Model D, per +1 SD, HC3 errors) | |
| 88 | + | |
| 89 | +| 📈 Positive | Impact | 📉 Negative | Impact | | |
| 90 | +|---|---:|---|---:| | |
| 91 | +| Modern/Contemporary | **+16.4 %*** | Family-Friendly | **−11.6 %*** | | |
| 92 | +| Luxury | **+14.2 %*** | New Construction | **−10.5 %*** | | |
| 93 | +| Land & Nature | **+13.0 %*** | Quiet & Peaceful | −9.5 %*** | | |
| 94 | +| Entry-Level | +10.4 %*** | Bright & Spacious | −8.9 %*** | | |
| 95 | +| Waterfront | +4.7 %*** | Motivated Seller | **−8.5 %*** | | |
| 96 | +| Panoramic View | +3.0 %** | Needs Renovation | **−7.9 %*** | | |
| 97 | + | |
| 98 | +<sub>*** p < 0.001, ** p < 0.01. Negative coefficients on Family-Friendly / New Construction largely proxy suburban location (see paper §6). Impact = (e^β − 1) × 100 %.</sub> | |
| 99 | + | |
| 100 | +### Heterogeneity & robustness (highlights) | |
| 101 | + | |
| 102 | +- **Quantile regressions** (τ = 0.25 / 0.50 / 0.75): the Luxury premium rises monotonically from **9.7 %** at the 25th percentile to **17.3 %** at the 75th; urgency and condition discounts attenuate at the top of the market. | |
| 103 | +- **Bootstrap** (1,000 replications, seed 42): SEs within 9 % of HC3 (15/20 within 5 %). | |
| 104 | +- **Outlier trimming** (1st/99th price percentiles): no sign reversal, all 16 significant dimensions stay significant. | |
| 105 | +- **Multicollinearity is real and disclosed**: similarity dimensions correlate at 0.66–0.97 (mean VIF 17.2). Block-level inference is unaffected; individual coefficients are partial associations within a correlated block. | |
| 106 | +- **PCA benchmark**: 20 principal components of the raw embeddings fit *better* (ΔR² +0.070 vs. +0.044) but are economically unreadable — the paper quantifies the interpretability-vs-fit trade-off explicitly. | |
| 107 | + | |
| 108 | +--- | |
| 109 | + | |
| 110 | +## 🧪 Methodology | |
| 111 | + | |
| 112 | +``` | |
| 113 | +┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ | |
| 114 | +│ Property │ │ Sentence │ │ Cosine │ │ Hedonic │ │ Implicit │ | |
| 115 | +│ Listings ├──▶│ Embeddings ├──▶│ Similarity ├──▶│ OLS Model ├──▶│ Price │ | |
| 116 | +│ (n=17,087) │ │ (384-d) │ │ (20 refs) │ │ (HC3 s.e.) │ │ Estimates │ | |
| 117 | +└─────────────┘ └──────────────┘ └──────────────┘ └─────────────┘ └──────────────┘ | |
| 118 | + PublicRemarks all-MiniLM-L6-v2 S = E · Rᵀ log(P) = α + β'X + δℓ + γ's + ε | |
| 119 | +``` | |
| 120 | + | |
| 121 | +1. **Extraction** — houses only, price > 0, description ≥ 20 characters; minimal preprocessing (the transformer handles raw text). | |
| 122 | +2. **Embedding** — `all-MiniLM-L6-v2` (22.7M params, 384-d, L2-normalized), batch 256; the whole corpus encodes in seconds on Apple Silicon. | |
| 123 | +3. **Projection** — 20 French reference descriptions (146–257 chars each, full verbatim text in paper Appendix A), designed for *semantic saturation*, *dimensional specificity*, *linguistic consistency*. One matrix product yields the 17,087 × 20 similarity matrix. | |
| 124 | +4. **Estimation** — nested OLS models A–E on log price, all covariates standardized, HC3 robust errors; joint F-tests; quantile regressions; bootstrap; winsorization; VIF; Lasso/Elastic-Net selection; PCA benchmark. | |
| 125 | + | |
| 126 | +The 20 dimensions span six domains: **Quality & Standing** (Luxury, Entry-Level) · **Condition** (Renovated, Needs Renovation) · **Physical** (Bright & Spacious, Garage & Parking, Finished Basement, Pool & Landscaping) · **Location & Setting** (Land & Nature, Panoramic View, Premium Location, Quiet & Peaceful, Waterfront) · **Style & Character** (Modern/Contemporary, Heritage/Character, New Construction, Energy Efficient, Family-Friendly) · **Market Signals** (Income/Investment, Motivated Seller). | |
| 127 | + | |
| 128 | +--- | |
| 129 | + | |
| 130 | +## 📁 Repository Structure | |
| 131 | + | |
| 132 | +``` | |
| 133 | +wp2_uqo/ | |
| 134 | +├── README.md ← you are here | |
| 135 | +├── AUDIT.md ← forensic audit of the original (pre-restructuring) project | |
| 136 | +├── CHANGES.md ← everything that was moved / refactored / corrected, with diffs | |
| 137 | +├── requirements.txt ← pinned dependencies (exact versions used for verification) | |
| 138 | +│ | |
| 139 | +├── data/ ← NOT tracked by git (see data/README.md) | |
| 140 | +│ ├── raw/louka.db │ SQLite, 268 MB, 46,479 listings (5 categories) | |
| 141 | +│ └── processed/ │ parquet sample, embeddings cache, similarity matrix, | |
| 142 | +│ └── legacy/ │ analysis CSV + pre-refactor artifacts kept for comparison | |
| 143 | +│ | |
| 144 | +├── src/ ← analysis package (imported by scripts/) | |
| 145 | +│ ├── config.py │ repo-relative paths, model name, seed, structural vars | |
| 146 | +│ ├── references.py │ the 20 French reference descriptions + English labels | |
| 147 | +│ ├── data.py │ SQLite → DataFrame extraction (n = 17,087) | |
| 148 | +│ ├── embeddings.py │ encoding, caching, cosine-similarity features | |
| 149 | +│ └── models.py │ standardized-design OLS A–E, HC3 | |
| 150 | +│ | |
| 151 | +├── scripts/ ← numbered pipeline entry points (run in order) | |
| 152 | +│ ├── 01_prepare_data.py │ louka.db → data/processed/houses.parquet | |
| 153 | +│ ├── 02_similarities.py │ embeddings (cached) + 20 similarity features → CSV | |
| 154 | +│ ├── 03_models.py │ models A–E → results/*.csv | |
| 155 | +│ ├── 04_robustness.py │ VIF · BP · quantile · winsor · bootstrap · Lasso/ENet | |
| 156 | +│ ├── 05_figures.py │ figures fig1–fig11 (PDF + PNG) | |
| 157 | +│ └── 06_tables.py │ LaTeX tables generated from results (no hand transcription) | |
| 158 | +│ | |
| 159 | +├── figures/ ← all 11 paper figures, regenerated from the pipeline | |
| 160 | +├── results/ ← model & robustness outputs (CSV, versioned) | |
| 161 | +│ | |
| 162 | +└── paper/ ← LaTeX source (UQO working-paper style) | |
| 163 | + ├── main.tex │ metadata + section inputs (47-page PDF) | |
| 164 | + ├── preamble.tex │ packages, layout, custom macros | |
| 165 | + ├── main.pdf │ compiled paper (committed for convenience) | |
| 166 | + ├── Makefile │ latexmk build | |
| 167 | + ├── references.bib │ 59 entries, all cited keys resolve | |
| 168 | + ├── sections/ │ titlepage · intro · literature · methodology · data · | |
| 169 | + │ │ results · robustness · discussion · conclusion · appendix | |
| 170 | + └── tables/ │ 7 machine-generated + 3 hand-maintained booktabs tables | |
| 171 | +``` | |
| 172 | + | |
| 173 | +--- | |
| 174 | + | |
| 175 | +## ⚙️ The Pipeline, Script by Script | |
| 176 | + | |
| 177 | +| # | Script | Input | Output | Runtime* | | |
| 178 | +|---|---|---|---|---| | |
| 179 | +| 1 | `01_prepare_data.py` | `data/raw/louka.db` | `houses.parquet` (17,087 × 14) | ~15 s | | |
| 180 | +| 2 | `02_similarities.py` | parquet + refs | embeddings `.npy` (cached), `sim_matrix.npy`, analysis CSV | ~5 s cached / ~3 min fresh | | |
| 181 | +| 3 | `03_models.py` | analysis CSV | `model_comparison.csv`, coefficient tables, descriptive stats | ~5 s | | |
| 182 | +| 4 | `04_robustness.py` | analysis CSV | `robustness_results.csv` (163 rows: VIF, BP, QR, winsor, bootstrap, Lasso) | ~4 min (bootstrap) | | |
| 183 | +| 5 | `05_figures.py` | analysis CSV | `figures/fig1…fig11.{pdf,png}` | ~30 s | | |
| 184 | +| 6 | `06_tables.py` | `results/*.csv` | `paper/tables/*.tex` (7 tables) | ~2 s | | |
| 185 | + | |
| 186 | +<sub>*Apple M-series laptop. All stochastic steps are seeded (`seed = 42`): bootstrap resampling, CV folds, scatter subsampling.</sub> | |
| 187 | + | |
| 188 | +**Design principles applied during the refactor** (see `CHANGES.md` for the full story): | |
| 189 | + | |
| 190 | +- 🚫 **No hardcoded paths** — everything resolves from the repo root (`src/config.py`); the DB location can be overridden with `WP2_DB_PATH`. | |
| 191 | +- 🧬 **Single source of truth** — the extraction logic and the 20 reference texts, previously copy-pasted across five scripts (in two languages), now live in exactly one place each. | |
| 192 | +- 🔁 **Deterministic & cached** — embeddings are deterministic for a given model version and cached; `--force` re-encodes from scratch. | |
| 193 | +- 📋 **Tables are compiled artifacts** — every number in the paper's statistical tables is written by `06_tables.py` from the results CSVs. The original project transcribed numbers by hand, which introduced errors that this restructuring caught and fixed (documented, with old→new values, in `CHANGES.md` §4). | |
| 194 | + | |
| 195 | +--- | |
| 196 | + | |
| 197 | +## 💾 Data | |
| 198 | + | |
| 199 | +The raw data is a **SQLite database (`louka.db`, 268 MB)** of Quebec residential listings collected from Realtor.ca / Centris: **46,479 listings** across five categories — houses (17,089), condos (8,586), land (8,506), rentals (8,240), plexes (4,058). Each row carries structured fields (price, bedrooms, bathrooms) plus a full JSON blob including the agent-written `PublicRemarks`. | |
| 200 | + | |
| 201 | +**The analysis sample**: single-family houses with a positive price and a description of ≥ 20 characters → **n = 17,087**. Median listing price $589,900 (mean $807,259, max $25M). Descriptions average 510 characters and are truncated at ~700 characters in the export (disclosed in the paper). | |
| 202 | + | |
| 203 | +> ⚠️ **`data/` is not tracked by git**: the raw database exceeds GitHub's 100 MB file limit, contains a leftover application `users` table, and the listing texts are proprietary platform content. `data/README.md` explains how to place a copy of `louka.db` to re-run the pipeline from scratch. All *derived, aggregate* outputs (results CSVs, figures, tables, paper) **are** versioned, so every number in the paper is inspectable without the raw data. | |
| 204 | + | |
| 205 | +--- | |
| 206 | + | |
| 207 | +## 🔄 Reproducing Everything | |
| 208 | + | |
| 209 | +```bash | |
| 210 | +git clone https://github.com/spboucher-ai/wp2_uqo.git | |
| 211 | +cd wp2_uqo | |
| 212 | +python3 -m pip install -r requirements.txt | |
| 213 | + | |
| 214 | +# place louka.db in data/raw/ (see data/README.md), then: | |
| 215 | +python3 scripts/01_prepare_data.py # extract the house sample | |
| 216 | +python3 scripts/02_similarities.py # embed + project (add --force to re-encode) | |
| 217 | +python3 scripts/03_models.py # hedonic models A–E | |
| 218 | +python3 scripts/04_robustness.py # full robustness battery (~4 min) | |
| 219 | +python3 scripts/05_figures.py # all 11 figures | |
| 220 | +python3 scripts/06_tables.py # LaTeX tables from results | |
| 221 | + | |
| 222 | +cd paper && latexmk -pdf main.tex # 47-page PDF, 0 errors | |
| 223 | +``` | |
| 224 | + | |
| 225 | +Expected checkpoints along the way: | |
| 226 | + | |
| 227 | +- Step 1 prints `17,087 houses with a valid price and description`. | |
| 228 | +- Step 3 prints the model table with `R² = 0.4525 / 0.4647 / 0.4962 / 0.5122 / 0.5118` and `F = 99.53`. | |
| 229 | +- Step 4 prints `Winsorized R2=0.4867` and the Lasso selection (4/20 at the CV penalty). | |
| 230 | + | |
| 231 | +--- | |
| 232 | + | |
| 233 | +## 📝 Building the Paper | |
| 234 | + | |
| 235 | +The paper (`paper/`) follows the UQO working-paper format: title page with logo, abstract page with keywords + JEL codes, IMRaD body (Introduction · Literature · Methodology · Data · Results · Robustness · Discussion · Conclusion), `natbib`/`apalike` bibliography, and two appendices (verbatim reference texts; parsimonious-model estimates). | |
| 236 | + | |
| 237 | +```bash | |
| 238 | +cd paper | |
| 239 | +make # or: latexmk -pdf main.tex | |
| 240 | +``` | |
| 241 | + | |
| 242 | +Figures are pulled from `../figures/` via `\graphicspath`, so the paper always reflects the latest pipeline run. The build is clean: **0 errors, 0 undefined references, 0 missing citations** (59/59 bib keys resolve). | |
| 243 | + | |
| 244 | +--- | |
| 245 | + | |
| 246 | +## ✅ Verification & Provenance | |
| 247 | + | |
| 248 | +This repository is a restructuring of an earlier research folder. Before anything was rewritten, the pipeline was validated against the original artifacts: | |
| 249 | + | |
| 250 | +| Check | Result | | |
| 251 | +|---|---| | |
| 252 | +| Analysis dataset (17,087 × 30) | ✅ identical, column by column | | |
| 253 | +| Similarity matrix (17,087 × 20) | ✅ bit-identical | | |
| 254 | +| Models A–E fit statistics | ✅ identical to the paper's Table 4 | | |
| 255 | +| Model D coefficients (27 params) | ✅ identical to the paper's Table 5 | | |
| 256 | +| Figures fig1, fig4–fig11 | ✅ regenerated, byte-size within ±2 B | | |
| 257 | +| Figures fig2, fig3 | 🔧 were *missing* from the original repo (placeholder boxes in the old PDF) — regenerated | | |
| 258 | +| `robustness_results.csv` | 🔧 lost in the original — regenerated (seeded) | | |
| 259 | +| Descriptive/robustness numbers in the old LaTeX | ⚠️ several did not match the code's own outputs — corrected and now machine-generated; full old→new list in [`CHANGES.md`](CHANGES.md) §4 | | |
| 260 | + | |
| 261 | +`AUDIT.md` documents the original project's layout, dead files, and defects; `CHANGES.md` documents every move, refactor, and numeric correction. | |
| 262 | + | |
| 263 | +--- | |
| 264 | + | |
| 265 | +## 🖼️ Figures Gallery | |
| 266 | + | |
| 267 | +| # | Figure | What it shows | | |
| 268 | +|---|---|---| | |
| 269 | +| 1 | `fig1_model_comparison` | R² / adj. R² across models A–E | | |
| 270 | +| 2 | `fig2_coefficient_plot` | Forest plot of the 20 semantic implicit prices (Model D) | | |
| 271 | +| 3 | `fig3_similarity_distributions` | Box plots of the 20 cosine-similarity features | | |
| 272 | +| 4 | `fig4_quintile_heatmap` | Mean similarity by price quintile × dimension | | |
| 273 | +| 5 | `fig5_correlation_matrix` | Inter-dimension Pearson correlations (0.66–0.97) | | |
| 274 | +| 6 | `fig6_scatter_plots` | Four illustrative similarity–price scatters | | |
| 275 | +| 7 | `fig7_methodology` | Pipeline diagram | | |
| 276 | +| 8 | `fig8_r2_decomposition` | Stacked decomposition of Model D's R² | | |
| 277 | +| 9 | `fig9_price_distribution` | Price and log-price histograms | | |
| 278 | +| 10 | `fig10_structural_coefficients` | Structural coefficients with 95 % CIs | | |
| 279 | +| 11 | `fig11_residual_diagnostics` | Residuals vs. fitted + normal Q-Q | | |
| 280 | + | |
| 281 | +All figures exist as publication PDF (vector) and PNG (300 dpi). | |
| 282 | + | |
| 283 | +--- | |
| 284 | + | |
| 285 | +## 📦 Requirements | |
| 286 | + | |
| 287 | +| Package | Version | Used for | | |
| 288 | +|---|---|---| | |
| 289 | +| Python | 3.14 | — | | |
| 290 | +| numpy | 2.4.4 | linear algebra | | |
| 291 | +| pandas | 3.0.2 | data wrangling | | |
| 292 | +| pyarrow | 24.0.0 | parquet I/O | | |
| 293 | +| scipy | 1.17.1 | tests, distributions | | |
| 294 | +| scikit-learn | 1.6.1 | PCA, Lasso/ElasticNet CV, scaling | | |
| 295 | +| statsmodels | 0.14.6 | OLS/HC3, QuantReg, BP test, VIF | | |
| 296 | +| matplotlib | 3.10.9 | figures | | |
| 297 | +| sentence-transformers | 5.5.0 | embeddings | | |
| 298 | +| torch | 2.12.0 | transformer backend (MPS on Apple Silicon) | | |
| 299 | + | |
| 300 | +LaTeX: TeX Live 2026 with `latexmk` (newtx, booktabs, threeparttable, natbib, hyperref…). | |
| 301 | + | |
| 302 | +--- | |
| 303 | + | |
| 304 | +## 📚 Citation | |
| 305 | + | |
| 306 | +```bibtex | |
| 307 | +@techreport{boucher2026decoding, | |
| 308 | + author = {Boucher, Simon-Pierre}, | |
| 309 | + title = {Decoding Real Estate Descriptions: Semantic Embeddings and | |
| 310 | + Hedonic Pricing of Residential Properties in Quebec}, | |
| 311 | + institution = {Universit\'e du Qu\'ebec en Outaouais, | |
| 312 | + D\'epartement des sciences administratives}, | |
| 313 | + type = {Working Paper}, | |
| 314 | + number = {2}, | |
| 315 | + year = {2026}, | |
| 316 | + month = {May} | |
| 317 | +} | |
| 318 | +``` | |
| 319 | + | |
| 320 | +--- | |
| 321 | + | |
| 322 | +## 👤 Author & Contact | |
| 323 | + | |
| 324 | +**Simon-Pierre Boucher** | |
| 325 | +Département des sciences administratives | |
| 326 | +Université du Québec en Outaouais (UQO) | |
| 327 | +283, boulevard Alexandre-Taché, Gatineau (Québec) J9A 1L8, Canada | |
| 328 | + | |
| 329 | +📧 **contact@spboucher.ai** | |
| 330 | + | |
| 331 | +*Comments and suggestions are welcome. All errors are my own.* | |
| 332 | + | |
| 333 | +--- | |
| 334 | + | |
| 335 | +<div align="center"> | |
| 336 | +<sub>© 2026 Simon-Pierre Boucher. Paper, code, and figures — all rights reserved.</sub> | |
| 337 | +</div> | |
added
data/README.md
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai --> | |
| 2 | + | |
| 3 | +# data/ — not tracked by git | |
| 4 | + | |
| 5 | +This directory is excluded from version control because: | |
| 6 | + | |
| 7 | +1. `raw/louka.db` (268 MB) exceeds GitHub's 100 MB per-file limit; | |
| 8 | +2. the database contains a leftover application `users` table (credentials) that | |
| 9 | + must not be published; | |
| 10 | +3. the listing texts are proprietary platform content (Realtor.ca / Centris). | |
| 11 | + | |
| 12 | +## Restoring the data locally | |
| 13 | + | |
| 14 | +Place the SQLite database at: | |
| 15 | + | |
| 16 | +``` | |
| 17 | +data/raw/louka.db # table `properties`: 46,479 listings, 5 categories | |
| 18 | +``` | |
| 19 | + | |
| 20 | +(or set the environment variable `WP2_DB_PATH` to its location), then run the | |
| 21 | +pipeline from the repository root: | |
| 22 | + | |
| 23 | +```bash | |
| 24 | +python3 scripts/01_prepare_data.py | |
| 25 | +python3 scripts/02_similarities.py | |
| 26 | +python3 scripts/03_models.py | |
| 27 | +python3 scripts/04_robustness.py | |
| 28 | +python3 scripts/05_figures.py | |
| 29 | +python3 scripts/06_tables.py | |
| 30 | +``` | |
| 31 | + | |
| 32 | +The pipeline recreates everything under `data/processed/`: | |
| 33 | + | |
| 34 | +| File | Content | | |
| 35 | +|---|---| | |
| 36 | +| `houses.parquet` | analysis sample (17,087 × 14, incl. remarks text) | | |
| 37 | +| `embeddings_maisons.npy` | 17,087 × 384 sentence embeddings (cache) | | |
| 38 | +| `sim_matrix_maisons.npy` | 17,087 × 20 cosine-similarity matrix | | |
| 39 | +| `hedonic_maison_results.csv` | analysis dataset used by all models/figures | | |
| 40 | +| `legacy/` | pre-refactor artifacts kept for byte-level comparison | | |
| 41 | + | |
| 42 | +Embeddings are deterministic for a given `all-MiniLM-L6-v2` model version, so a | |
| 43 | +fresh run reproduces the committed results exactly (see `CHANGES.md` §3). | |
added
figures/fig10_structural_coefficients.pdf
+0 −0
Binary file not shown.
added
figures/fig10_structural_coefficients.png
+0 −0
Binary file not shown.
added
figures/fig11_residual_diagnostics.pdf
+0 −0
Binary file not shown.
added
figures/fig11_residual_diagnostics.png
+0 −0
Binary file not shown.
added
figures/fig1_model_comparison.pdf
+0 −0
Binary file not shown.
added
figures/fig1_model_comparison.png
+0 −0
Binary file not shown.
added
figures/fig2_coefficient_plot.pdf
+0 −0
Binary file not shown.
added
figures/fig2_coefficient_plot.png
+0 −0
Binary file not shown.
added
figures/fig3_similarity_distributions.pdf
+0 −0
Binary file not shown.
added
figures/fig3_similarity_distributions.png
+0 −0
Binary file not shown.
added
figures/fig4_quintile_heatmap.pdf
+0 −0
Binary file not shown.
added
figures/fig4_quintile_heatmap.png
+0 −0
Binary file not shown.
added
figures/fig5_correlation_matrix.pdf
+0 −0
Binary file not shown.
added
figures/fig5_correlation_matrix.png
+0 −0
Binary file not shown.
added
figures/fig6_scatter_plots.pdf
+0 −0
Binary file not shown.
added
figures/fig6_scatter_plots.png
+0 −0
Binary file not shown.
added
figures/fig7_methodology.pdf
+0 −0
Binary file not shown.
added
figures/fig7_methodology.png
+0 −0
Binary file not shown.
added
figures/fig8_r2_decomposition.pdf
+0 −0
Binary file not shown.
added
figures/fig8_r2_decomposition.png
+0 −0
Binary file not shown.
added
figures/fig9_price_distribution.pdf
+0 −0
Binary file not shown.
added
figures/fig9_price_distribution.png
+0 −0
Binary file not shown.
added
paper/Makefile
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +# Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +# | |
| 3 | +# Build the paper. Figures are read from ../figures/ (run the pipeline first). | |
| 4 | + | |
| 5 | +all: main.pdf | |
| 6 | + | |
| 7 | +main.pdf: main.tex preamble.tex sections/*.tex tables/*.tex references.bib | |
| 8 | + latexmk -pdf -interaction=nonstopmode main.tex | |
| 9 | + | |
| 10 | +clean: | |
| 11 | + latexmk -C | |
| 12 | + rm -f *.bbl *.blg *.fdb_latexmk *.fls *.out *.toc | |
| 13 | + | |
| 14 | +.PHONY: all clean | |
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+73 −0
@@ -0,0 +1,73 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ============================================================================ | |
| 4 | +% UQO Working Paper No. 2 | |
| 5 | +% Decoding Real Estate Descriptions: Semantic Embeddings and Hedonic | |
| 6 | +% Pricing of Residential Properties in Quebec | |
| 7 | +% | |
| 8 | +% Build: latexmk -pdf main.tex (figures are read from ../figures/) | |
| 9 | +% ============================================================================ | |
| 10 | +\documentclass[12pt,letterpaper]{article} | |
| 11 | + | |
| 12 | +\input{preamble} | |
| 13 | + | |
| 14 | +% ============================================================================ | |
| 15 | +% METADATA | |
| 16 | +% ============================================================================ | |
| 17 | +\newcommand{\WPnumber}{2} | |
| 18 | +\newcommand{\WPtitle}{Decoding Real Estate Descriptions: Semantic Embeddings and Hedonic Pricing of Residential Properties in Quebec} | |
| 19 | +\newcommand{\WPsubtitle}{} | |
| 20 | +\newcommand{\WPdate}{May 2026} | |
| 21 | +\newcommand{\WPversion}{1.1} | |
| 22 | +\newcommand{\WPabstract}{% | |
| 23 | +This paper develops an interpretable NLP approach to hedonic pricing that projects sentence-transformer embeddings of property listing descriptions onto 20 researcher-defined semantic references, producing economically meaningful features. Using 17,087 single-family listings from Quebec, Canada, I show that adding cosine-similarity measures to a standard hedonic model increases adjusted $R^2$ from 0.452 to 0.511 ($F=99.53$, $p<0.001$). Modern/contemporary language is associated with $+$16.4\% prices per standard deviation; luxury $+$14.2\%; land/nature $+$13.0\%. Renovation-need language is associated with $-$7.9\% and seller urgency $-$8.5\%. Quantile regressions reveal heterogeneous effects across the price distribution, and the estimates are robust to bootstrap inference and outlier trimming.% | |
| 24 | +} | |
| 25 | +\newcommand{\WPkeywords}{hedonic pricing, NLP, sentence embeddings, cosine similarity, real estate valuation, Quebec} | |
| 26 | +\newcommand{\WPjel}{R31, C45, C21, R21} | |
| 27 | + | |
| 28 | +% --- Author --- | |
| 29 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | |
| 30 | +\newcommand{\WPaffiliation}{% | |
| 31 | + D\'epartement des sciences administratives\\ | |
| 32 | + Universit\'e du Qu\'ebec en Outaouais% | |
| 33 | +} | |
| 34 | +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca} | |
| 35 | +\newcommand{\WPaddress}{% | |
| 36 | + Gatineau -- Pavillon Alexandre-Tach\'e\\ | |
| 37 | + 283, boulevard Alexandre-Tach\'e\\ | |
| 38 | + Gatineau, Qu\'ebec, Canada J9A 1L8% | |
| 39 | +} | |
| 40 | + | |
| 41 | +% --- PDF metadata --- | |
| 42 | +\hypersetup{ | |
| 43 | + pdftitle = {\WPtitle}, | |
| 44 | + pdfauthor = {\WPauthor}, | |
| 45 | +} | |
| 46 | + | |
| 47 | +% ============================================================================ | |
| 48 | +% DOCUMENT | |
| 49 | +% ============================================================================ | |
| 50 | +\begin{document} | |
| 51 | + | |
| 52 | +% --- Title Page & Abstract --- | |
| 53 | +\input{sections/titlepage} | |
| 54 | + | |
| 55 | +% --- Main Body --- | |
| 56 | +\input{sections/introduction} | |
| 57 | +\input{sections/literature} | |
| 58 | +\input{sections/methodology} | |
| 59 | +\input{sections/data} | |
| 60 | +\input{sections/results} | |
| 61 | +\input{sections/robustness} | |
| 62 | +\input{sections/discussion} | |
| 63 | +\input{sections/conclusion} | |
| 64 | + | |
| 65 | +% --- References --- | |
| 66 | +\clearpage | |
| 67 | +\bibliography{references} | |
| 68 | + | |
| 69 | +% --- Appendix --- | |
| 70 | +\clearpage | |
| 71 | +\input{sections/appendix} | |
| 72 | + | |
| 73 | +\end{document} | |
added
paper/preamble.tex
+137 −0
@@ -0,0 +1,137 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +% ============================================================================ | |
| 4 | +% PREAMBLE — packages, layout, custom commands (UQO working-paper style) | |
| 5 | +% ============================================================================ | |
| 6 | + | |
| 7 | +% --- Encoding & Language --- | |
| 8 | +\usepackage[utf8]{inputenc} | |
| 9 | +\usepackage[T1]{fontenc} | |
| 10 | +\usepackage[english]{babel} | |
| 11 | + | |
| 12 | +% --- Page Layout --- | |
| 13 | +\usepackage[letterpaper, margin=1in, headheight=15pt]{geometry} | |
| 14 | +\usepackage{setspace} | |
| 15 | +\onehalfspacing | |
| 16 | +\setlength{\parindent}{1.5em} | |
| 17 | +\setlength{\parskip}{0pt} | |
| 18 | + | |
| 19 | +% --- Typography --- | |
| 20 | +\usepackage{newtxtext,newtxmath} | |
| 21 | +\usepackage{amsmath,amsfonts} | |
| 22 | +\let\Bbbk\relax | |
| 23 | +\usepackage{amssymb} | |
| 24 | +\usepackage{mathtools} | |
| 25 | +\usepackage{microtype} | |
| 26 | + | |
| 27 | +% --- Tables --- | |
| 28 | +\usepackage{booktabs} | |
| 29 | +\usepackage{threeparttable} | |
| 30 | +\usepackage{tabularx} | |
| 31 | +\usepackage{array} | |
| 32 | +\usepackage{multirow} | |
| 33 | +\usepackage{longtable} | |
| 34 | +\usepackage{adjustbox} | |
| 35 | +\usepackage{siunitx} | |
| 36 | +\usepackage{dcolumn} | |
| 37 | + | |
| 38 | +% --- Figures --- | |
| 39 | +\usepackage{graphicx} | |
| 40 | +\graphicspath{{../figures/}{.}} | |
| 41 | +\usepackage[ | |
| 42 | + font = small, | |
| 43 | + labelfont = bf, | |
| 44 | + labelsep = period, | |
| 45 | + skip = 8pt, | |
| 46 | + justification = justified, | |
| 47 | + singlelinecheck = false | |
| 48 | +]{caption} | |
| 49 | +\usepackage{subcaption} | |
| 50 | +\usepackage{float} | |
| 51 | +\usepackage{pdflscape} | |
| 52 | + | |
| 53 | +% --- Colors & Links --- | |
| 54 | +\usepackage[dvipsnames]{xcolor} | |
| 55 | +\usepackage[bookmarks, bookmarksnumbered]{hyperref} | |
| 56 | +\hypersetup{ | |
| 57 | + colorlinks = true, | |
| 58 | + linkcolor = NavyBlue, | |
| 59 | + citecolor = NavyBlue, | |
| 60 | + urlcolor = NavyBlue, | |
| 61 | +} | |
| 62 | + | |
| 63 | +% --- Bibliography --- | |
| 64 | +\usepackage[round, authoryear, comma]{natbib} | |
| 65 | +\setcitestyle{aysep={,}} | |
| 66 | +\bibliographystyle{apalike} | |
| 67 | + | |
| 68 | +% --- Headers & Footers --- | |
| 69 | +\usepackage{fancyhdr} | |
| 70 | +\pagestyle{fancy} | |
| 71 | +\fancyhf{} | |
| 72 | +\fancyhead[L]{\small\itshape Semantic Embeddings and Hedonic Pricing} | |
| 73 | +\fancyhead[R]{\small\thepage} | |
| 74 | +\renewcommand{\headrulewidth}{0.4pt} | |
| 75 | +\renewcommand{\footrulewidth}{0pt} | |
| 76 | +\fancypagestyle{plain}{% | |
| 77 | + \fancyhf{} | |
| 78 | + \fancyfoot[C]{\small\thepage} | |
| 79 | + \renewcommand{\headrulewidth}{0pt} | |
| 80 | +} | |
| 81 | + | |
| 82 | +% --- Section Formatting --- | |
| 83 | +\usepackage{titlesec} | |
| 84 | +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{} | |
| 85 | +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{} | |
| 86 | +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{} | |
| 87 | + | |
| 88 | +% --- Appendix Support --- | |
| 89 | +\usepackage[toc, page]{appendix} | |
| 90 | +\usepackage{enumitem} | |
| 91 | +\usepackage{etoolbox} | |
| 92 | + | |
| 93 | +% --- Custom Column Types --- | |
| 94 | +\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}} | |
| 95 | +\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}} | |
| 96 | +\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}} | |
| 97 | +\newcolumntype{d}[1]{D{.}{.}{#1}} | |
| 98 | + | |
| 99 | +% --- Custom Commands --- | |
| 100 | +\newcommand{\sym}[1]{\ensuremath{^{#1}}} | |
| 101 | +\newcommand{\stmark}[1]{\rlap{\textsuperscript{#1}}} | |
| 102 | +\DeclareMathOperator*{\plim}{plim} | |
| 103 | +\newcommand{\E}{\mathbb{E}} | |
| 104 | +\newcommand{\Var}{\mathrm{Var}} | |
| 105 | +\newcommand{\Cov}{\mathrm{Cov}} | |
| 106 | +\newcommand{\Corr}{\mathrm{Corr}} | |
| 107 | +\newcommand{\se}{\mathrm{s.e.}} | |
| 108 | +\newcommand{\tr}{\mathrm{tr}} | |
| 109 | +\newcommand{\rank}{\mathrm{rank}} | |
| 110 | +\newcommand{\diag}{\mathrm{diag}} | |
| 111 | +\newcommand{\R}{\mathbb{R}} | |
| 112 | +\newcommand{\N}{\mathbb{N}} | |
| 113 | +\newcommand{\eps}{\varepsilon} | |
| 114 | +\newcommand{\bfbeta}{\boldsymbol{\beta}} | |
| 115 | +\newcommand{\bfgamma}{\boldsymbol{\gamma}} | |
| 116 | +\newcommand{\bfalpha}{\boldsymbol{\alpha}} | |
| 117 | +\newcommand{\bfdelta}{\boldsymbol{\delta}} | |
| 118 | +\newcommand{\bftheta}{\boldsymbol{\theta}} | |
| 119 | +\newcommand{\bfSigma}{\boldsymbol{\Sigma}} | |
| 120 | +\newcommand{\bfOmega}{\boldsymbol{\Omega}} | |
| 121 | +\newcommand{\bfX}{\mathbf{X}} | |
| 122 | +\newcommand{\bfY}{\mathbf{Y}} | |
| 123 | +\newcommand{\bfy}{\mathbf{y}} | |
| 124 | +\newcommand{\bfe}{\mathbf{e}} | |
| 125 | +\newcommand{\bfs}{\mathbf{s}} | |
| 126 | +\newcommand{\iid}{\overset{\mathrm{iid}}{\sim}} | |
| 127 | +\newcommand{\pto}{\overset{p}{\to}} | |
| 128 | +\newcommand{\dto}{\overset{d}{\to}} | |
| 129 | +\newcommand{\ols}{\mathrm{OLS}} | |
| 130 | +\newcommand{\iv}{\mathrm{IV}} | |
| 131 | +\newcommand{\gmm}{\mathrm{GMM}} | |
| 132 | +\newcommand{\mle}{\mathrm{MLE}} | |
| 133 | +\newcommand{\pval}{\textit{p}-value} | |
| 134 | +\newcommand{\tstat}{\textit{t}-statistic} | |
| 135 | +\newcommand{\fstat}{\textit{F}-statistic} | |
| 136 | +\newcommand{\tablenote}[1]{\begin{minipage}{\linewidth}\footnotesize #1\end{minipage}} | |
| 137 | +\newcommand{\signote}{*** $p < 0.001$; ** $p < 0.01$; * $p < 0.05$.} | |
added
paper/references.bib
+551 −0
@@ -0,0 +1,551 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% BIBLIOGRAPHY — Working Paper | |
| 3 | +% ============================================================================ | |
| 4 | + | |
| 5 | +@article{athey2019machine, | |
| 6 | + author = {Athey, Susan and Imbens, Guido W.}, | |
| 7 | + title = {Machine Learning Methods That Economists Should Know About}, | |
| 8 | + journal = {Annual Review of Economics}, | |
| 9 | + year = {2019}, | |
| 10 | + volume = {11}, | |
| 11 | + pages = {685--725} | |
| 12 | +} | |
| 13 | + | |
| 14 | +@article{bayer2016racial, | |
| 15 | + author = {Bayer, Patrick and Casey, Marcus and Ferreira, Fernando and McMillan, Robert}, | |
| 16 | + title = {Racial and Ethnic Price Differentials in the Housing Market}, | |
| 17 | + journal = {Journal of Urban Economics}, | |
| 18 | + year = {2016}, | |
| 19 | + volume = {102}, | |
| 20 | + pages = {91--105} | |
| 21 | +} | |
| 22 | + | |
| 23 | +@article{blei2003latent, | |
| 24 | + author = {Blei, David M. and Ng, Andrew Y. and Jordan, Michael I.}, | |
| 25 | + title = {Latent {D}irichlet Allocation}, | |
| 26 | + journal = {Journal of Machine Learning Research}, | |
| 27 | + year = {2003}, | |
| 28 | + volume = {3}, | |
| 29 | + pages = {993--1022} | |
| 30 | +} | |
| 31 | + | |
| 32 | +@article{breusch1979simple, | |
| 33 | + author = {Breusch, Trevor S. and Pagan, Adrian R.}, | |
| 34 | + title = {A Simple Test for Heteroscedasticity and Random Coefficient Variation}, | |
| 35 | + journal = {Econometrica}, | |
| 36 | + year = {1979}, | |
| 37 | + volume = {47}, | |
| 38 | + number = {5}, | |
| 39 | + pages = {1287--1294} | |
| 40 | +} | |
| 41 | + | |
| 42 | +@article{cheshire2004capitalisation, | |
| 43 | + author = {Cheshire, Paul and Sheppard, Stephen}, | |
| 44 | + title = {Capitalising the Value of Free Schools: The Impact of Supply Characteristics and Uncertainty}, | |
| 45 | + journal = {The Economic Journal}, | |
| 46 | + year = {2004}, | |
| 47 | + volume = {114}, | |
| 48 | + number = {499}, | |
| 49 | + pages = {F397--F424} | |
| 50 | +} | |
| 51 | + | |
| 52 | +@inproceedings{conneau2020unsupervised, | |
| 53 | + author = {Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin}, | |
| 54 | + title = {Unsupervised Cross-Lingual Representation Learning at Scale}, | |
| 55 | + booktitle = {Proceedings of the 58th Annual Meeting of the ACL}, | |
| 56 | + year = {2020}, | |
| 57 | + pages = {8440--8451} | |
| 58 | +} | |
| 59 | + | |
| 60 | +@article{cropper1988choice, | |
| 61 | + author = {Cropper, Maureen L. and Deck, Leland B. and McConnell, Kenneth E.}, | |
| 62 | + title = {On the Choice of Functional Form for Hedonic Price Functions}, | |
| 63 | + journal = {Review of Economics and Statistics}, | |
| 64 | + year = {1988}, | |
| 65 | + volume = {70}, | |
| 66 | + number = {4}, | |
| 67 | + pages = {668--675} | |
| 68 | +} | |
| 69 | + | |
| 70 | +@unpublished{demers2018textual, | |
| 71 | + author = {Demers, Elizabeth and Eisfeldt, Andrea L.}, | |
| 72 | + title = {Textual Analysis of Real Estate Listings}, | |
| 73 | + year = {2018}, | |
| 74 | + note = {Working Paper, University of California, Los Angeles} | |
| 75 | +} | |
| 76 | + | |
| 77 | +@inproceedings{devlin2019bert, | |
| 78 | + author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina}, | |
| 79 | + title = {{BERT}: Pre-Training of Deep Bidirectional Transformers for Language Understanding}, | |
| 80 | + booktitle = {Proceedings of NAACL-HLT}, | |
| 81 | + year = {2019}, | |
| 82 | + pages = {4171--4186} | |
| 83 | +} | |
| 84 | + | |
| 85 | +@book{efron1993introduction, | |
| 86 | + author = {Efron, Bradley and Tibshirani, Robert J.}, | |
| 87 | + title = {An Introduction to the Bootstrap}, | |
| 88 | + publisher = {Chapman \& Hall/CRC}, | |
| 89 | + year = {1993} | |
| 90 | +} | |
| 91 | + | |
| 92 | +@article{epple1987hedonic, | |
| 93 | + author = {Epple, Dennis}, | |
| 94 | + title = {Hedonic Prices and Implicit Markets: Estimating Demand and Supply Functions for Differentiated Products}, | |
| 95 | + journal = {Journal of Political Economy}, | |
| 96 | + year = {1987}, | |
| 97 | + volume = {95}, | |
| 98 | + number = {1}, | |
| 99 | + pages = {59--80} | |
| 100 | +} | |
| 101 | + | |
| 102 | +@inproceedings{ethayarajh2019contextual, | |
| 103 | + author = {Ethayarajh, Kawin}, | |
| 104 | + title = {How Contextual Are Contextualized Word Representations? {C}omparing the Geometry of {BERT}, {ELMo}, and {GPT-2} Embeddings}, | |
| 105 | + booktitle = {Proceedings of EMNLP-IJCNLP}, | |
| 106 | + year = {2019}, | |
| 107 | + pages = {55--65} | |
| 108 | +} | |
| 109 | + | |
| 110 | +@article{fama1993common, | |
| 111 | + author = {Fama, Eugene F. and French, Kenneth R.}, | |
| 112 | + title = {Common Risk Factors in the Returns on Stocks and Bonds}, | |
| 113 | + journal = {Journal of Financial Economics}, | |
| 114 | + year = {1993}, | |
| 115 | + volume = {33}, | |
| 116 | + number = {1}, | |
| 117 | + pages = {3--56} | |
| 118 | +} | |
| 119 | + | |
| 120 | +@book{frank2007falling, | |
| 121 | + author = {Frank, Robert H.}, | |
| 122 | + title = {Falling Behind: How Rising Inequality Harms the Middle Class}, | |
| 123 | + publisher = {University of California Press}, | |
| 124 | + year = {2007} | |
| 125 | +} | |
| 126 | + | |
| 127 | +@inproceedings{gao2021simcse, | |
| 128 | + author = {Gao, Tianyu and Yao, Xingcheng and Chen, Danqi}, | |
| 129 | + title = {{SimCSE}: Simple Contrastive Learning of Sentence Embeddings}, | |
| 130 | + booktitle = {Proceedings of EMNLP}, | |
| 131 | + year = {2021}, | |
| 132 | + pages = {6894--6910} | |
| 133 | +} | |
| 134 | + | |
| 135 | +@article{gibbons2014costs, | |
| 136 | + author = {Gibbons, Stephen and Mourato, Susana and Resende, Guilherme M.}, | |
| 137 | + title = {The Amenity Value of {E}nglish Nature: A Hedonic Price Approach}, | |
| 138 | + journal = {Environmental and Resource Economics}, | |
| 139 | + year = {2014}, | |
| 140 | + volume = {57}, | |
| 141 | + number = {2}, | |
| 142 | + pages = {175--196} | |
| 143 | +} | |
| 144 | + | |
| 145 | +@article{goodman1998housing, | |
| 146 | + author = {Goodman, Allen C. and Thibodeau, Thomas G.}, | |
| 147 | + title = {Housing Market Segmentation}, | |
| 148 | + journal = {Journal of Housing Economics}, | |
| 149 | + year = {1998}, | |
| 150 | + volume = {7}, | |
| 151 | + number = {2}, | |
| 152 | + pages = {121--143} | |
| 153 | +} | |
| 154 | + | |
| 155 | +@article{goodwin2020feature, | |
| 156 | + author = {Goodwin, Kimberly and Sirmans, Stacy and Nanda, Anupam}, | |
| 157 | + title = {Feature and Textual Sentiment Analysis of Online Real Estate Listings}, | |
| 158 | + journal = {Journal of Housing Research}, | |
| 159 | + year = {2020}, | |
| 160 | + volume = {29}, | |
| 161 | + number = {sup1}, | |
| 162 | + pages = {S75--S92} | |
| 163 | +} | |
| 164 | + | |
| 165 | +@article{halvorsen1981choice, | |
| 166 | + author = {Halvorsen, Robert and Palmquist, Raymond}, | |
| 167 | + title = {The Interpretation of Dummy Variables in Semilogarithmic Equations}, | |
| 168 | + journal = {American Economic Review}, | |
| 169 | + year = {1980}, | |
| 170 | + volume = {70}, | |
| 171 | + number = {3}, | |
| 172 | + pages = {474--475} | |
| 173 | +} | |
| 174 | + | |
| 175 | +@article{haurin2010list, | |
| 176 | + author = {Haurin, Donald R. and McGreal, Stanley and Adair, Alastair and Brown, Louise and Webb, James R.}, | |
| 177 | + title = {List Price and Sales Prices of Residential Properties During Booms and Busts}, | |
| 178 | + journal = {Journal of Housing Economics}, | |
| 179 | + year = {2010}, | |
| 180 | + volume = {22}, | |
| 181 | + number = {1}, | |
| 182 | + pages = {1--10} | |
| 183 | +} | |
| 184 | + | |
| 185 | +@article{hong2020text, | |
| 186 | + author = {Hong, Jongho and Choi, Hyunjoong and Kim, Woojin}, | |
| 187 | + title = {A House Price Valuation Based on the Random Forest Approach: The Mass Appraisal of Residential Property in South Korea}, | |
| 188 | + journal = {International Journal of Strategic Property Management}, | |
| 189 | + year = {2020}, | |
| 190 | + volume = {24}, | |
| 191 | + number = {3}, | |
| 192 | + pages = {140--152} | |
| 193 | +} | |
| 194 | + | |
| 195 | +@article{huang2022house, | |
| 196 | + author = {Huang, Dashan and Ni, Yangtian}, | |
| 197 | + title = {Textual Sentiment Analysis of Real Estate Listings and House Prices}, | |
| 198 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 199 | + year = {2022}, | |
| 200 | + volume = {65}, | |
| 201 | + number = {3}, | |
| 202 | + pages = {395--419} | |
| 203 | +} | |
| 204 | + | |
| 205 | +@article{irwin2002interacting, | |
| 206 | + author = {Irwin, Elena G. and Bockstael, Nancy E.}, | |
| 207 | + title = {Interacting Agents, Spatial Externalities and the Evolution of Residential Land Use Patterns}, | |
| 208 | + journal = {Journal of Economic Geography}, | |
| 209 | + year = {2002}, | |
| 210 | + volume = {2}, | |
| 211 | + number = {1}, | |
| 212 | + pages = {31--54} | |
| 213 | +} | |
| 214 | + | |
| 215 | +@article{koenker1978regression, | |
| 216 | + author = {Koenker, Roger and Bassett, Gilbert W.}, | |
| 217 | + title = {Regression Quantiles}, | |
| 218 | + journal = {Econometrica}, | |
| 219 | + year = {1978}, | |
| 220 | + volume = {46}, | |
| 221 | + number = {1}, | |
| 222 | + pages = {33--50} | |
| 223 | +} | |
| 224 | + | |
| 225 | +@book{koenker2005quantile, | |
| 226 | + author = {Koenker, Roger}, | |
| 227 | + title = {Quantile Regression}, | |
| 228 | + publisher = {Cambridge University Press}, | |
| 229 | + year = {2005} | |
| 230 | +} | |
| 231 | + | |
| 232 | +@inproceedings{koh2020concept, | |
| 233 | + author = {Koh, Pang Wei and Nguyen, Thao and Tang, Yew Siang and Mussmann, Stephen and Pierson, Emma and Kim, Been and Liang, Percy}, | |
| 234 | + title = {Concept Bottleneck Models}, | |
| 235 | + booktitle = {Proceedings of ICML}, | |
| 236 | + year = {2020}, | |
| 237 | + pages = {5338--5348} | |
| 238 | +} | |
| 239 | + | |
| 240 | +@article{kok2017big, | |
| 241 | + author = {Kok, Nils and Koponen, Eija-Leena and Mart{\'i}nez-Barbosa, Carlos A.}, | |
| 242 | + title = {Big Data in Real Estate? {F}rom Manual Appraisal to Automated Valuation}, | |
| 243 | + journal = {Journal of Portfolio Management}, | |
| 244 | + year = {2017}, | |
| 245 | + volume = {43}, | |
| 246 | + number = {6}, | |
| 247 | + pages = {202--211} | |
| 248 | +} | |
| 249 | + | |
| 250 | +@article{lam2022textual, | |
| 251 | + author = {Lam, Ka Chi and Yu, Chin Ying and Lam, Ka Yun}, | |
| 252 | + title = {An Investigation of the Effects of Textual Information on House Prices Using NLP Methods}, | |
| 253 | + journal = {Journal of Real Estate Research}, | |
| 254 | + year = {2022}, | |
| 255 | + volume = {44}, | |
| 256 | + number = {2}, | |
| 257 | + pages = {178--205} | |
| 258 | +} | |
| 259 | + | |
| 260 | +@article{lancaster1966new, | |
| 261 | + author = {Lancaster, Kelvin J.}, | |
| 262 | + title = {A New Approach to Consumer Theory}, | |
| 263 | + journal = {Journal of Political Economy}, | |
| 264 | + year = {1966}, | |
| 265 | + volume = {74}, | |
| 266 | + number = {2}, | |
| 267 | + pages = {132--157} | |
| 268 | +} | |
| 269 | + | |
| 270 | +@inproceedings{le2020flaubert, | |
| 271 | + author = {Le, Hang and Vial, Lo{\"i}c and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabb{\'e}, Beno{\^i}t and Besacier, Laurent and Schwab, Didier}, | |
| 272 | + title = {{FlauBERT}: Unsupervised Language Model Pre-Training for {F}rench}, | |
| 273 | + booktitle = {Proceedings of LREC}, | |
| 274 | + year = {2020}, | |
| 275 | + pages = {2479--2490} | |
| 276 | +} | |
| 277 | + | |
| 278 | +@book{lesage2009introduction, | |
| 279 | + author = {LeSage, James P. and Pace, R. Kelley}, | |
| 280 | + title = {Introduction to Spatial Econometrics}, | |
| 281 | + publisher = {Chapman \& Hall/CRC}, | |
| 282 | + year = {2009} | |
| 283 | +} | |
| 284 | + | |
| 285 | +@article{levitt2008information, | |
| 286 | + author = {Levitt, Steven D. and Syverson, Chad}, | |
| 287 | + title = {Market Distortions When Agents Are Better Informed: The Value of Information in Real Estate Transactions}, | |
| 288 | + journal = {Review of Economics and Statistics}, | |
| 289 | + year = {2008}, | |
| 290 | + volume = {90}, | |
| 291 | + number = {4}, | |
| 292 | + pages = {599--611} | |
| 293 | +} | |
| 294 | + | |
| 295 | +@inproceedings{li2020sentence, | |
| 296 | + author = {Li, Bohan and Zhou, Hao and He, Junxian and Wang, Mingxuan and Yang, Yiming and Li, Lei}, | |
| 297 | + title = {On the Sentence Embeddings from Pre-Trained Language Models}, | |
| 298 | + booktitle = {Proceedings of EMNLP}, | |
| 299 | + year = {2020}, | |
| 300 | + pages = {9119--9130} | |
| 301 | +} | |
| 302 | + | |
| 303 | +@article{li2023deep, | |
| 304 | + author = {Li, Jing and Xu, Yilan and Zhu, Haishan}, | |
| 305 | + title = {Deep Learning-Based Housing Valuation: A Comprehensive Text and Image Approach}, | |
| 306 | + journal = {Real Estate Economics}, | |
| 307 | + year = {2023}, | |
| 308 | + volume = {51}, | |
| 309 | + number = {4}, | |
| 310 | + pages = {1062--1094} | |
| 311 | +} | |
| 312 | + | |
| 313 | +@article{loughran2011liability, | |
| 314 | + author = {Loughran, Tim and McDonald, Bill}, | |
| 315 | + title = {When Is a Liability Not a Liability? {T}extual Analysis, Dictionaries, and 10-{K}s}, | |
| 316 | + journal = {Journal of Finance}, | |
| 317 | + year = {2011}, | |
| 318 | + volume = {66}, | |
| 319 | + number = {1}, | |
| 320 | + pages = {35--65} | |
| 321 | +} | |
| 322 | + | |
| 323 | +@article{mackinnon1985some, | |
| 324 | + author = {MacKinnon, James G. and White, Halbert}, | |
| 325 | + title = {Some Heteroskedasticity-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties}, | |
| 326 | + journal = {Journal of Econometrics}, | |
| 327 | + year = {1985}, | |
| 328 | + volume = {29}, | |
| 329 | + number = {3}, | |
| 330 | + pages = {305--325} | |
| 331 | +} | |
| 332 | + | |
| 333 | +@incollection{malpezzi2003hedonic, | |
| 334 | + author = {Malpezzi, Stephen}, | |
| 335 | + title = {Hedonic Pricing Models: A Selective and Applied Review}, | |
| 336 | + booktitle = {Housing Economics and Public Policy}, | |
| 337 | + editor = {O'Sullivan, Tony and Gibb, Kenneth}, | |
| 338 | + publisher = {Blackwell Science}, | |
| 339 | + year = {2003}, | |
| 340 | + pages = {67--89} | |
| 341 | +} | |
| 342 | + | |
| 343 | +@article{marinescu2020opening, | |
| 344 | + author = {Marinescu, Ioana and Wolthoff, Ronald}, | |
| 345 | + title = {Opening the Black Box of the Matching Function: The Power of Words}, | |
| 346 | + journal = {Journal of Labor Economics}, | |
| 347 | + year = {2020}, | |
| 348 | + volume = {38}, | |
| 349 | + number = {2}, | |
| 350 | + pages = {535--568} | |
| 351 | +} | |
| 352 | + | |
| 353 | +@inproceedings{martin2020camembert, | |
| 354 | + author = {Martin, Louis and Muller, Benjamin and Ortiz~Su{\'a}rez, Pedro Javier and Dupont, Yoann and Romary, Laurent and de~la~Clergerie, {\'E}ric and Seddah, Djam{\'e} and Sagot, Beno{\^i}t}, | |
| 355 | + title = {{CamemBERT}: A Tasty {F}rench Language Model}, | |
| 356 | + booktitle = {Proceedings of the 58th Annual Meeting of the ACL}, | |
| 357 | + year = {2020}, | |
| 358 | + pages = {7203--7219} | |
| 359 | +} | |
| 360 | + | |
| 361 | +@inproceedings{mikolov2013distributed, | |
| 362 | + author = {Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg S. and Dean, Jeff}, | |
| 363 | + title = {Distributed Representations of Words and Phrases and Their Compositionality}, | |
| 364 | + booktitle = {NeurIPS}, | |
| 365 | + year = {2013}, | |
| 366 | + volume = {26}, | |
| 367 | + pages = {3111--3119} | |
| 368 | +} | |
| 369 | + | |
| 370 | +@article{milgrom1981good, | |
| 371 | + author = {Milgrom, Paul R.}, | |
| 372 | + title = {Good News and Bad News: Representation Theorems and Applications}, | |
| 373 | + journal = {Bell Journal of Economics}, | |
| 374 | + year = {1981}, | |
| 375 | + volume = {12}, | |
| 376 | + number = {2}, | |
| 377 | + pages = {380--391} | |
| 378 | +} | |
| 379 | + | |
| 380 | +@article{mullainathan2017machine, | |
| 381 | + author = {Mullainathan, Sendhil and Spiess, Jann}, | |
| 382 | + title = {Machine Learning: An Applied Econometric Approach}, | |
| 383 | + journal = {Journal of Economic Perspectives}, | |
| 384 | + year = {2017}, | |
| 385 | + volume = {31}, | |
| 386 | + number = {2}, | |
| 387 | + pages = {87--106} | |
| 388 | +} | |
| 389 | + | |
| 390 | +@article{nowak2017quality, | |
| 391 | + author = {Nowak, Adam and Smith, Patrick}, | |
| 392 | + title = {Textual Analysis in Real Estate}, | |
| 393 | + journal = {Journal of Applied Econometrics}, | |
| 394 | + year = {2017}, | |
| 395 | + volume = {32}, | |
| 396 | + number = {4}, | |
| 397 | + pages = {788--803} | |
| 398 | +} | |
| 399 | + | |
| 400 | +@article{obrien2007caution, | |
| 401 | + author = {O'Brien, Robert M.}, | |
| 402 | + title = {A Caution Regarding Rules of Thumb for Variance Inflation Factors}, | |
| 403 | + journal = {Quality \& Quantity}, | |
| 404 | + year = {2007}, | |
| 405 | + volume = {41}, | |
| 406 | + number = {5}, | |
| 407 | + pages = {673--690} | |
| 408 | +} | |
| 409 | + | |
| 410 | +@article{ozdogan2020effect, | |
| 411 | + author = {Ozdogan, Ilker and Tukel, Oya Icmeli and Boran, Semih}, | |
| 412 | + title = {The Effect of Listing Descriptions on House Prices: An Empirical Assessment}, | |
| 413 | + journal = {Journal of Housing Economics}, | |
| 414 | + year = {2020}, | |
| 415 | + volume = {50}, | |
| 416 | + pages = {101726} | |
| 417 | +} | |
| 418 | + | |
| 419 | +@article{pace1998appraisal, | |
| 420 | + author = {Pace, R. Kelley and Barry, Ronald}, | |
| 421 | + title = {Quick Computation of Spatial Autoregressive Estimators}, | |
| 422 | + journal = {Geographical Analysis}, | |
| 423 | + year = {1998}, | |
| 424 | + volume = {29}, | |
| 425 | + number = {3}, | |
| 426 | + pages = {232--247} | |
| 427 | +} | |
| 428 | + | |
| 429 | +@article{pakes2003reconsideration, | |
| 430 | + author = {Pakes, Ariel}, | |
| 431 | + title = {A Reconsideration of Hedonic Price Indexes with an Application to {PC}'s}, | |
| 432 | + journal = {American Economic Review}, | |
| 433 | + year = {2003}, | |
| 434 | + volume = {93}, | |
| 435 | + number = {5}, | |
| 436 | + pages = {1578--1596} | |
| 437 | +} | |
| 438 | + | |
| 439 | +@article{palmquist1984estimating, | |
| 440 | + author = {Palmquist, Raymond B.}, | |
| 441 | + title = {Estimating the Demand for the Characteristics of Housing}, | |
| 442 | + journal = {Review of Economics and Statistics}, | |
| 443 | + year = {1984}, | |
| 444 | + volume = {66}, | |
| 445 | + number = {3}, | |
| 446 | + pages = {394--404} | |
| 447 | +} | |
| 448 | + | |
| 449 | +@inproceedings{peters2018deep, | |
| 450 | + author = {Peters, Matthew E. and Neumann, Mark and Iyyer, Mohit and Gardner, Matt and Clark, Christopher and Lee, Kenton and Zettlemoyer, Luke}, | |
| 451 | + title = {Deep Contextualized Word Representations}, | |
| 452 | + booktitle = {Proceedings of NAACL-HLT}, | |
| 453 | + year = {2018}, | |
| 454 | + pages = {2227--2237} | |
| 455 | +} | |
| 456 | + | |
| 457 | +@inproceedings{reimers2019sentence, | |
| 458 | + author = {Reimers, Nils and Gurevych, Iryna}, | |
| 459 | + title = {Sentence-{BERT}: Sentence Embeddings Using Siamese {BERT}-Networks}, | |
| 460 | + booktitle = {Proceedings of EMNLP-IJCNLP}, | |
| 461 | + year = {2019}, | |
| 462 | + pages = {3982--3992} | |
| 463 | +} | |
| 464 | + | |
| 465 | +@article{rosen1974hedonic, | |
| 466 | + author = {Rosen, Sherwin}, | |
| 467 | + title = {Hedonic Prices and Implicit Markets: Product Differentiation in Pure Competition}, | |
| 468 | + journal = {Journal of Political Economy}, | |
| 469 | + year = {1974}, | |
| 470 | + volume = {82}, | |
| 471 | + number = {1}, | |
| 472 | + pages = {34--55} | |
| 473 | +} | |
| 474 | + | |
| 475 | +@article{shen2020text, | |
| 476 | + author = {Shen, Lin and Ross, Stephen L.}, | |
| 477 | + title = {Information Value of Property Descriptions: A Machine Learning Approach}, | |
| 478 | + journal = {Journal of Urban Economics}, | |
| 479 | + year = {2020}, | |
| 480 | + volume = {121}, | |
| 481 | + pages = {103299} | |
| 482 | +} | |
| 483 | + | |
| 484 | +@article{sirmans2005composition, | |
| 485 | + author = {Sirmans, Stacy and Macpherson, David and Zietz, Emily}, | |
| 486 | + title = {The Composition of Hedonic Pricing Models}, | |
| 487 | + journal = {Journal of Real Estate Literature}, | |
| 488 | + year = {2005}, | |
| 489 | + volume = {13}, | |
| 490 | + number = {1}, | |
| 491 | + pages = {3--43} | |
| 492 | +} | |
| 493 | + | |
| 494 | +@article{sirmans2006empirical, | |
| 495 | + author = {Sirmans, G. Stacy and MacDonald, Lynn and Macpherson, David A. and Zietz, Emily N.}, | |
| 496 | + title = {The Value of Housing Characteristics: A Meta Analysis}, | |
| 497 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 498 | + year = {2006}, | |
| 499 | + volume = {33}, | |
| 500 | + number = {3}, | |
| 501 | + pages = {215--240} | |
| 502 | +} | |
| 503 | + | |
| 504 | +@article{tibshirani1996regression, | |
| 505 | + author = {Tibshirani, Robert}, | |
| 506 | + title = {Regression Shrinkage and Selection via the {L}asso}, | |
| 507 | + journal = {Journal of the Royal Statistical Society: Series B}, | |
| 508 | + year = {1996}, | |
| 509 | + volume = {58}, | |
| 510 | + number = {1}, | |
| 511 | + pages = {267--288} | |
| 512 | +} | |
| 513 | + | |
| 514 | +@article{tyrvainen2005benefits, | |
| 515 | + author = {Tyrv{\"a}inen, Liisa and Miettinen, Antti}, | |
| 516 | + title = {Property Prices and Urban Forest Amenities}, | |
| 517 | + journal = {Journal of Environmental Economics and Management}, | |
| 518 | + year = {2000}, | |
| 519 | + volume = {39}, | |
| 520 | + number = {2}, | |
| 521 | + pages = {205--223} | |
| 522 | +} | |
| 523 | + | |
| 524 | +@inproceedings{wang2020minilm, | |
| 525 | + author = {Wang, Wenhui and Wei, Furu and Dong, Li and Bao, Hangbo and Yang, Nan and Zhou, Ming}, | |
| 526 | + title = {{MiniLM}: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers}, | |
| 527 | + booktitle = {NeurIPS}, | |
| 528 | + year = {2020}, | |
| 529 | + volume = {33}, | |
| 530 | + pages = {5776--5788} | |
| 531 | +} | |
| 532 | + | |
| 533 | +@article{yoo2012variable, | |
| 534 | + author = {Yoo, Seyoung and Im, Jungho and Wagner, John E.}, | |
| 535 | + title = {Variable Selection for Hedonic Model Using Machine Learning Approaches: A Case Study in Onondaga County, NY}, | |
| 536 | + journal = {Landscape and Urban Planning}, | |
| 537 | + year = {2012}, | |
| 538 | + volume = {107}, | |
| 539 | + number = {3}, | |
| 540 | + pages = {293--306} | |
| 541 | +} | |
| 542 | + | |
| 543 | +@article{zou2005regularization, | |
| 544 | + author = {Zou, Hui and Hastie, Trevor}, | |
| 545 | + title = {Regularization and Variable Selection via the Elastic Net}, | |
| 546 | + journal = {Journal of the Royal Statistical Society: Series B}, | |
| 547 | + year = {2005}, | |
| 548 | + volume = {67}, | |
| 549 | + number = {2}, | |
| 550 | + pages = {301--320} | |
| 551 | +} | |
added
paper/sections/appendix.tex
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +%====================================================================== | |
| 4 | +% APPENDIX | |
| 5 | +%====================================================================== | |
| 6 | +\begin{appendices} | |
| 7 | + | |
| 8 | +\renewcommand{\thefigure}{A\arabic{figure}} | |
| 9 | +\renewcommand{\thetable}{A\arabic{table}} | |
| 10 | +\setcounter{figure}{0} | |
| 11 | +\setcounter{table}{0} | |
| 12 | + | |
| 13 | +\section{Reference Descriptions} \label{app:references} | |
| 14 | + | |
| 15 | +Table~\ref{tab:reference_texts} reproduces the verbatim French text of the 20 reference descriptions used to construct the semantic similarity features (Section~\ref{sec:methodology}). Publishing the exact reference texts makes the feature construction fully replicable and allows readers to assess the operationalization of each qualitative dimension directly. | |
| 16 | + | |
| 17 | +\input{tables/tab_reference_texts} | |
| 18 | + | |
| 19 | +\section{Parsimonious Model Estimates} \label{app:parsimonious} | |
| 20 | + | |
| 21 | +Table~\ref{tab:parsimonious} reports the complete coefficient estimates for the parsimonious specification (Model~E), which retains the 16 semantic dimensions that are individually significant at the 5\% level in the full model (Model~D). | |
| 22 | + | |
| 23 | +\input{tables/tab_parsimonious} | |
| 24 | + | |
| 25 | +\end{appendices} | |
added
paper/sections/conclusion.tex
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +%====================================================================== | |
| 4 | +\section{Conclusion} \label{sec:conclusion} | |
| 5 | +%====================================================================== | |
| 6 | + | |
| 7 | +This paper demonstrates that property listing descriptions contain economically meaningful information that can be systematically extracted and integrated into hedonic pricing models. Using sentence embeddings and cosine similarity with 20 interpretable reference descriptions, we show that semantic features from listing text improve the explanatory power of a standard hedonic model by 5.9 percentage points (adjusted $R^2$: 0.452 $\rightarrow$ 0.511) for 17,087 single-family homes in Quebec, Canada. The estimates are stable under bootstrap inference, outlier trimming, and quantile-based heterogeneity analysis, while the accompanying diagnostics candidly document the strong collinearity among semantic dimensions and its implications for variable-by-variable interpretation. | |
| 8 | + | |
| 9 | +Our approach bridges two literatures---hedonic pricing and natural language processing---by producing features that satisfy the requirements of both. They are semantically rich, capturing deep meaning rather than surface-level keyword overlap, and economically interpretable, with each feature corresponding to a named qualitative dimension carrying a clear coefficient estimate. Descriptions emphasizing luxury finishes (+14.2\%), modern design (+16.4\%), and natural settings (+13.0\%) command significant premiums, while language signaling renovation needs ($-$7.9\%), seller urgency ($-$8.5\%), and family-oriented marketing ($-$11.6\%) is associated with discounts. Importantly, the quantile regression analysis reveals that these effects are heterogeneous across the price distribution: the luxury premium is amplified for high-value properties, while renovation and urgency discounts are attenuated, consistent with quality complementarities and differential buyer price sensitivity. | |
| 10 | + | |
| 11 | +The general methodology---embedding domain-specific text, computing cosine similarities against researcher-designed references, and incorporating the resulting features into standard econometric models---is portable to any setting where free-text descriptions accompany structured economic data. Applications beyond real estate include: | |
| 12 | +\begin{itemize}[noitemsep, topsep=3pt] | |
| 13 | +\item \textbf{E-commerce}: Product listing text on platforms such as Amazon or eBay contains quality and condition signals that could be incorporated into hedonic price models for used goods, electronics, or collectibles. | |
| 14 | +\item \textbf{Labor economics}: Job posting descriptions embed information about workplace culture, benefits, and flexibility that are imperfectly captured by structured fields; reference-based similarity could quantify the implicit wage premiums associated with different workplace attributes \citep{marinescu2020opening}. | |
| 15 | +\item \textbf{Innovation economics}: Patent abstracts describe inventions in varying degrees of technical novelty and commercial potential; semantic features could augment patent valuation models. | |
| 16 | +\item \textbf{Corporate finance}: Earnings call transcripts and annual report narratives contain forward-looking information about firm prospects; reference-based sentiment dimensions could improve asset pricing models beyond the binary positive/negative sentiment used by \citet{loughran2011liability}. | |
| 17 | +\item \textbf{Hospitality}: Hotel and Airbnb listing descriptions convey amenity and ambiance information that complements structured data on room type, location, and ratings. | |
| 18 | +\end{itemize} | |
| 19 | + | |
| 20 | +In each case, the reference-based approach can translate qualitative textual information into quantitative features suitable for econometric analysis while preserving the interpretability required for inference on implicit prices. | |
| 21 | + | |
| 22 | +Future research should extend this framework along several dimensions. First, applying the method to transaction prices rather than listing prices would provide cleaner identification of text-price relationships and enable estimation of the listing premium as a function of semantic content. Second, temporal analysis using panel data could reveal how the relationship between listing language and prices evolves over market cycles---for instance, whether luxury premiums are amplified in bull markets and attenuated in recessions. Third, spatial econometric extensions incorporating municipality fixed effects and spatial autoregressive structures \citep{lesage2009introduction} could disentangle location-based from property-level semantic effects, addressing the concern that some dimensions (Family-Friendly, New Construction) proxy for suburban location. Fourth, multilingual embedding models specifically optimized for French---such as CamemBERT \citep{martin2020camembert} or domain-adapted variants---together with TF-IDF, keyword, and sentiment baselines, would complete the comparison of text representations begun in Section~\ref{sec:robustness}. Fifth, a systematic sensitivity analysis of the reference descriptions---multi-paraphrase averaging, bilingual variants, and length perturbations---would quantify the robustness of the semantic features to their exact phrasing. Finally, causal identification could be strengthened through within-property variation---comparing successive listings for the same property by different agents---or through A/B testing partnerships with real estate platforms. | |
added
paper/sections/data.tex
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +% Author: Simon-Pierre Boucher — contact@spboucher.ai | |
| 2 | +% | |
| 3 | +%====================================================================== | |
| 4 | +\section{Data} \label{sec:data} | |
| 5 | +%====================================================================== | |
| 6 | + | |
| 7 | +\subsection{Data Source and Sample Construction} | |
| 8 | + | |
| 9 | +The dataset is drawn from a database of residential property listings in Quebec, Canada, collected from the Realtor.ca and Centris platforms. Realtor.ca is the public-facing portal of the Canadian Real Estate Association (CREA), and Centris is the Quebec-specific MLS (Multiple Listing Service) platform operated by the Qu\'ebec Professional Association of Real Estate Brokers (APCIQ). Together, these platforms capture the near-universe of listed residential properties in the province. | |
| 10 | + | |
| 11 | +The full database contains 46,479 listings across five property categories: houses (17,089), condominiums (8,586), land (8,506), rentals (8,240), and multiplexes (4,058). For this study, we restrict the sample to single-family houses to maintain a homogeneous product category, following standard practice in the hedonic pricing literature \citep{sirmans2005composition}. After excluding listings with missing or uninformative descriptions (fewer than 20 characters) and non-positive prices, the final sample comprises 17,087 observations. | |
| 12 | + | |
| 13 | +\subsection{Structural Variables} | |
| 14 | + | |
| 15 | +Table~\ref{tab:descriptive} presents descriptive statistics for the structural and textual variables used in the hedonic models. | |
| 16 | + | |
| 17 | +\input{tables/tab_descriptive} | |
| 18 | + | |
| 19 | +The median listing price is \$589,900, with substantial right-skew (mean \$807,259, maximum \$25 million). The log transformation reduces this skewness considerably (log-price standard deviation of 0.70). The typical house has 3 bedrooms and 2 bathrooms (sample medians). The parking-spaces field averages 5.5 with high variance; it records the total number of parking spots reported by the agent, including driveway and outdoor spaces, which explains its large mean and range. The lot-size field is recorded without standardized units at the source and is correspondingly noisy---a limitation reflected in its imprecise coefficient estimate in the regressions. Listing descriptions average 510 characters (standard deviation of 151) and are truncated at approximately 700 characters in the data export, so the length variable is bounded from above. | |
| 20 | + | |
| 21 | +Figure~\ref{fig:price_dist} displays the distributions of price and log-price, confirming the appropriateness of the semi-logarithmic specification. | |
| 22 | + | |
| 23 | +\begin{figure}[!htbp] | |
| 24 | +\centering | |
| 25 | +\includegraphics[width=0.85\textwidth]{fig9_price_distribution.pdf} | |
| 26 | +\caption{Distribution of listing prices and log-prices ($n = 17{,}087$).} | |
| 27 | +\label{fig:price_dist} | |
| 28 | +\begin{minipage}{0.9\textwidth} | |
| 29 | +\footnotesize | |
| 30 | +Notes: Left panel shows the distribution of raw listing prices; right panel shows the distribution of log-transformed prices. The log transformation substantially reduces right-skewness. | |
| 31 | +\end{minipage} | |
| 32 | +\end{figure} | |
| 33 | + | |
| 34 | +\subsection{Semantic Similarity Features} | |
| 35 | + | |
| 36 | +Table~\ref{tab:sim_stats} reports the distribution of cosine similarity scores across the 20 reference dimensions, along with their bivariate correlations with log-price. | |
| 37 | + | |
| 38 | +\input{tables/tab_similarity_stats} | |
| 39 | + | |
| 40 | +A notable pattern in Table~\ref{tab:sim_stats} is that all bivariate correlations between similarity scores and log-price are negative, ranging from $-0.29$ to $-0.17$. This initially counterintuitive result arises because longer, more detailed descriptions---which characterize lower-priced listings where agents invest more effort in textual marketing---produce higher similarity scores with \textit{all} references. This confound is addressed in the multivariate regression by controlling for description length, after which the theoretically expected positive and negative effects of different qualitative dimensions emerge clearly. | |
| 41 | + | |
| 42 | +Figure~\ref{fig:sim_dist} displays the distributions of cosine similarities, revealing substantial variation within each dimension. | |
| 43 | + | |
| 44 | +\begin{figure}[!htbp] | |
| 45 | +\centering | |
| 46 | +\includegraphics[width=0.85\textwidth]{fig3_similarity_distributions.pdf} | |
| 47 | +\caption{Distribution of cosine similarity scores across 20 reference dimensions.} | |
| 48 | +\label{fig:sim_dist} | |
| 49 | +\begin{minipage}{0.9\textwidth} | |
| 50 | +\footnotesize | |
| 51 | +Notes: Each box plot displays the distribution of cosine similarity between listing embeddings and the corresponding reference description embedding, ordered by median similarity. Boxes span the interquartile range; whiskers extend to 1.5 times the interquartile range. | |
| 52 | +\end{minipage} | |
| 53 | +\end{figure} | |
added
paper/sections/discussion.tex
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/introduction.tex
+23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/literature.tex
+79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/methodology.tex
+102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/results.tex
+169 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/robustness.tex
+57 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/titlepage.tex
+80 −0
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_descriptive.tex
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_full_results.tex
+49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_model_comparison.tex
+27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_parsimonious.tex
+43 −0
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_quantile.tex
+23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_reference_texts.tex
+38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_references.tex
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_robustness_summary.tex
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_similarity_stats.tex
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/tab_text_comparison.tex
+22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
added
requirements.txt
+13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added
results/coefficients_model_D.csv
+28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added
results/coefficients_model_E.csv
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
results/descriptive_stats.csv
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
results/model_comparison.csv
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
results/robustness_results.csv
+164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added
results/similarity_stats.csv
+21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/01_prepare_data.py
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/02_similarities.py
+55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/03_models.py
+104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/04_robustness.py
+146 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/05_figures.py
+357 −0
Large diff (357 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/06_tables.py
+385 −0
Large diff (385 lines) is not loaded by default to keep the page fast.Load diff
added
src/__init__.py
+3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added
src/config.py
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
src/data.py
+101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added
src/embeddings.py
+59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added
src/models.py
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
src/references.py
+149 −0
Large diff (150 lines) is not loaded by default to keep the page fast.Load diff