harmonisation standard UQO WP (2026-10-17)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
61 changed files +7,212 −0
added
.env.example
+2 −0
@@ -0,0 +1,2 @@ | ||
| 1 | +OPENAI_API_KEY= | |
| 2 | +ANTHROPIC_API_KEY= | |
added
.gitignore
+42 −0
@@ -0,0 +1,42 @@ | ||
| 1 | +# Raw & processed data — never committed (see data/raw/README.md for provenance) | |
| 2 | +data/raw/* | |
| 3 | +!data/raw/README.md | |
| 4 | +data/processed/* | |
| 5 | +!data/processed/.gitkeep | |
| 6 | + | |
| 7 | +# Python | |
| 8 | +__pycache__/ | |
| 9 | +*.pyc | |
| 10 | +.pytest_cache/ | |
| 11 | +.venv/ | |
| 12 | +*.egg-info/ | |
| 13 | + | |
| 14 | +# LaTeX build artifacts (paper/main.pdf IS committed) | |
| 15 | +paper/*.aux | |
| 16 | +paper/*.log | |
| 17 | +paper/*.out | |
| 18 | +paper/*.toc | |
| 19 | +paper/*.lof | |
| 20 | +paper/*.lot | |
| 21 | +paper/*.fls | |
| 22 | +paper/*.fdb_latexmk | |
| 23 | +paper/*.bbl | |
| 24 | +paper/*.blg | |
| 25 | +paper/*.spl | |
| 26 | +paper/*.synctex.gz | |
| 27 | +paper/*.run.xml | |
| 28 | +paper/*.bcf | |
| 29 | + | |
| 30 | +# Logs | |
| 31 | +logs/ | |
| 32 | +*.log | |
| 33 | + | |
| 34 | +# macOS | |
| 35 | +.DS_Store | |
| 36 | + | |
| 37 | +# Project-specific | |
| 38 | +# OPENAI_API_KEY / ANTHROPIC_API_KEY — never commit | |
| 39 | +.env | |
| 40 | +# LLM/vision caches (specs.jsonl, facade_specs.jsonl), merged datasets, OOS predictions | |
| 41 | +data/processed/* | |
| 42 | +!data/processed/README.md | |
added
README.md
+164 −0
@@ -0,0 +1,164 @@ | ||
| 1 | +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca --> | |
| 2 | +# 🏠 Hedonic House Pricing with Language-Model and Vision Features | |
| 3 | + | |
| 4 | +**UQO Working Paper No. 14** — *Evidence from Quebec Real-Estate Listings* | |
| 5 | + | |
| 6 | +[-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) | |
| 7 | +[](requirements.txt) | |
| 8 | +[](data/raw/README.md) | |
| 9 | +[](scripts/) | |
| 10 | +[](https://uqo.ca) | |
| 11 | +[](mailto:simon-pierre.boucher@uqo.ca) | |
| 12 | + | |
| 13 | +> **TL;DR** — Much of a home's value is recorded only in the free text and the photographs of its listing. We use a | |
| 14 | +> large language model to turn each description into structured specifications and a vision–language model to turn the | |
| 15 | +> façade photo into objective visual attributes, both under strict JSON schemas. On 19,123 Quebec house listings the two | |
| 16 | +> AI layers lift the R² of an already rich log-linear hedonic model from 0.717 to 0.755; the vision layer is the largest | |
| 17 | +> out-of-sample gain of any feature block for every estimator (OLS, gradient boosting, neural nets); the best ensemble | |
| 18 | +> reaches out-of-sample R² 0.779 with a 14.0% median absolute error. Implicit prices are economically sensible (+2.3% per | |
| 19 | +> point of curb appeal, +3–4% for brick or stone) and survive municipality-clustered inference. | |
| 20 | + | |
| 21 | +--- | |
| 22 | + | |
| 23 | +## 📌 Status | |
| 24 | + | |
| 25 | +| Item | Value | | |
| 26 | +|---|---| | |
| 27 | +| Version | 2.0 (July 2026) — revised draft after an internal review round (`scripts/13_revision_extras.py`) | | |
| 28 | +| SSRN | not yet submitted | | |
| 29 | +| Journal | Elsevier-format highlights prepared (`docs/highlights_elsevier.tex`); target to be decided | | |
| 30 | +| Code & data | reproducible from `scripts/` run from the repository root; raw crawls and LLM caches not committed | | |
| 31 | + | |
| 32 | +## 🎯 What this paper does | |
| 33 | + | |
| 34 | +Hedonic models are bounded by the characteristics an analyst can measure. We treat general-purpose AI models as a | |
| 35 | +**feature-engineering technology**: each listing description is parsed by an LLM into a fixed schema of property | |
| 36 | +specifications (renovations, finishes, systems, extras), and each façade photograph is parsed by a vision–language model | |
| 37 | +into objective attributes (cladding, storeys, garage, curb appeal). The outputs are ordinary econometric variables and | |
| 38 | +enter a log-linear hedonic regression with city fixed effects: | |
| 39 | + | |
| 40 | +``` | |
| 41 | +ln P_i = α_city(i) + X_i'β + S_i'γ (LLM text layer) + V_i'δ (vision layer) + ε_i | |
| 42 | +``` | |
| 43 | + | |
| 44 | +We then measure, under identical 5-fold splits, how much each layer adds for OLS, histogram gradient boosting, a | |
| 45 | +residual MLP with city/type embeddings, and a stacked blend. | |
| 46 | + | |
| 47 | +## 🏆 Headline results | |
| 48 | + | |
| 49 | +| Object | Estimate | | |
| 50 | +|---|---| | |
| 51 | +| Base hedonic OLS (structural + location), in-sample R² | 0.717 | | |
| 52 | +| + LLM text layer + vision layer | 0.755 | | |
| 53 | +| Out-of-sample R², OLS: base → +text → +text+vision (5-fold) | 0.52 → 0.62 → … (see `paper/tables/ablation.tex`) | | |
| 54 | +| Out-of-sample R², GBM: base → +text | 0.69 → 0.73 | | |
| 55 | +| Best ensemble (robust NN + GBM blend) | R²_OOS = 0.779, median absolute error 14.0% | | |
| 56 | +| Curb-appeal score (1–5) | +2.3% per point | | |
| 57 | +| Brick / stone façade | +3–4% | | |
| 58 | +| Inference | HC3 and municipality-clustered standard errors agree | | |
| 59 | + | |
| 60 | +## 🗂 Repository layout | |
| 61 | + | |
| 62 | +``` | |
| 63 | +hedonic-llm-vision-features/ | |
| 64 | +├── data/ | |
| 65 | +│ ├── raw/ # two Apify crawls (Realtor.ca, DuProprio), 2026-07-03 — not committed | |
| 66 | +│ └── processed/ # LLM/vision caches, merged datasets, OOS predictions — not committed | |
| 67 | +├── scripts/ # run from the repository root, in order | |
| 68 | +│ ├── 00a_test_openai_extraction.py / 00b_test_vision_claude.py # API smoke tests | |
| 69 | +│ ├── 01_extract_specs_llm.py # description → JSON specs (parallel, resumable) | |
| 70 | +│ ├── 02_facade_vision.py # façade photo → visual attributes | |
| 71 | +│ ├── 03_merge_llm_features.py | |
| 72 | +│ ├── 04_hedonic_ols_base.py / 05_hedonic_ols_v4_text.py | |
| 73 | +│ ├── 06_hedonic_ml_cv.py / 07_torch_nn.py / 08_torch_nn_robust.py | |
| 74 | +│ ├── 09_v5_ml_text_vision.py / 10_final_blend.py | |
| 75 | +│ ├── 11_extras_v5_robustness.py / 12_ablation_extras.py / 13_revision_extras.py | |
| 76 | +│ └── 14_figures_v5.py / 15_make_tables.py / 16_make_tables_v5.py / 17_gen_regression_table.py | |
| 77 | +├── results/ # hedonic summaries (txt), ablation_extras.json, revision_extras.json | |
| 78 | +├── docs/ # METHODOLOGIE.md + methodologie.tex/pdf (French data & method note), Elsevier highlights | |
| 79 | +└── paper/ # main.tex + sections/ + tables/ + figures/ + references.bib → main.pdf | |
| 80 | +``` | |
| 81 | + | |
| 82 | +## 🚀 Quick start | |
| 83 | + | |
| 84 | +```bash | |
| 85 | +python3 -m pip install -r requirements.txt | |
| 86 | +cp .env.example .env # OPENAI_API_KEY=…, ANTHROPIC_API_KEY=… (only steps 00–02 call the APIs) | |
| 87 | +# place the two Apify crawls in data/raw/ and build data/processed/maisons_avec_prix.json (docs/METHODOLOGIE.md §2–4), then: | |
| 88 | +WORKERS=24 python3 scripts/01_extract_specs_llm.py | |
| 89 | +python3 scripts/02_facade_vision.py | |
| 90 | +python3 scripts/03_merge_llm_features.py | |
| 91 | +python3 scripts/05_hedonic_ols_v4_text.py | |
| 92 | +python3 scripts/08_torch_nn_robust.py && python3 scripts/09_v5_ml_text_vision.py && python3 scripts/10_final_blend.py | |
| 93 | +python3 scripts/11_extras_v5_robustness.py && python3 scripts/12_ablation_extras.py && python3 scripts/13_revision_extras.py | |
| 94 | +python3 scripts/14_figures_v5.py && python3 scripts/16_make_tables_v5.py && python3 scripts/17_gen_regression_table.py | |
| 95 | +cd paper && make # latexmk → main.pdf | |
| 96 | +``` | |
| 97 | + | |
| 98 | +LLM extraction ≈ 600 tokens in / 110 out per listing (24 threads); vision ≈ 2,470 tokens in per image (16 threads); | |
| 99 | +both are resumable. Econometrics and ML ≈ 20 minutes on an Apple-silicon laptop (PyTorch on MPS). | |
| 100 | + | |
| 101 | +## 🗃 Data | |
| 102 | + | |
| 103 | +27,302 Realtor.ca and 7,693 DuProprio listings crawled on 2026-07-03; houses (single-family and townhouse) with a valid | |
| 104 | +price retained, outliers removed by the rules in `docs/METHODOLOGIE.md` §3, giving 19,123 observations. Structural | |
| 105 | +variables (living area, lot, rooms, year built, type, city), text-derived specifications and façade attributes. See | |
| 106 | +`data/raw/README.md` and `data/processed/README.md`. | |
| 107 | + | |
| 108 | +## 🔬 Methodology | |
| 109 | + | |
| 110 | +1. **AI feature layers** with strict JSON schemas and retry/backoff — `scripts/01_…`, `scripts/02_…`. | |
| 111 | +2. **Log-linear hedonic OLS** with city fixed effects, HC3 and municipality-clustered SEs — `scripts/05_…`, `scripts/13_…`. | |
| 112 | +3. **Ablation by feature block × estimator** under identical folds (OLS, GBM, NN) — `scripts/09_…`, `scripts/12_…`. | |
| 113 | +4. **Stacked blend** of the robust NN and HistGradientBoosting — `scripts/10_final_blend.py`. | |
| 114 | +5. **Robustness** by segment, extreme city effects, calibration and importance plots — `scripts/11_…`, `scripts/14_…`. | |
| 115 | + | |
| 116 | +## 🖼 Figures & tables inventory | |
| 117 | + | |
| 118 | +| Figure | Content | | |
| 119 | +|---|---| | |
| 120 | +| `fig_ablation` | out-of-sample gain by feature block and estimator | | |
| 121 | +| `fig_calibration` | predicted vs actual, blend | | |
| 122 | +| `fig_importance` | feature importance | | |
| 123 | +| `fig_premia` | location premia | | |
| 124 | +| `fig_validation` | validation of AI-extracted attributes | | |
| 125 | +| `nn_diagram` (TikZ) | architecture of the robust NN | | |
| 126 | + | |
| 127 | +Tables (`paper/tables/*.tex`): summary stats, regression (A/B panels), models, ablation, robustness, AI summary. | |
| 128 | + | |
| 129 | +## ⚠️ Limitations | |
| 130 | + | |
| 131 | +- Asking prices from listings, single cross-section (July 2026). | |
| 132 | +- Realtor.ca listings lack coordinates; location enters through city fixed effects and postal codes. | |
| 133 | +- LLM/vision outputs are model-dependent; schemas and prompts are versioned in the scripts. | |
| 134 | +- API costs make the extraction layers the expensive part of replication. | |
| 135 | + | |
| 136 | +## 📝 Citation | |
| 137 | + | |
| 138 | +```bibtex | |
| 139 | +@techreport{boucher2026llmvisionhedonic, | |
| 140 | + author = {Boucher, Simon-Pierre}, | |
| 141 | + title = {Hedonic House Pricing with Language-Model and Vision Features: Evidence from Quebec Real-Estate Listings}, | |
| 142 | + institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, | |
| 143 | + type = {Working Paper}, | |
| 144 | + number = {14}, | |
| 145 | + year = {2026}, | |
| 146 | + month = {July} | |
| 147 | +} | |
| 148 | +``` | |
| 149 | + | |
| 150 | +## 👤 Author & contact | |
| 151 | + | |
| 152 | +**Simon-Pierre Boucher** — Département des sciences administratives, | |
| 153 | +Université du Québec en Outaouais (UQO), Gatineau, QC. | |
| 154 | +📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) | |
| 155 | + | |
| 156 | +## 🗒 Notes | |
| 157 | + | |
| 158 | +- Numbering: this paper was first issued as "Working Paper No. 11" (July 2026); number 11 belongs to *Half a Million | |
| 159 | + Prices, Twenty Models*. Renumbered **14** on 2026-10-17; cover date fixed to July 2026 (was `\today`). | |
| 160 | +- Harmonised 2026-10-17: flat scripts moved to `scripts/` (numbered), data to `data/`, logs removed, summaries to | |
| 161 | + `results/`, methodology note to `docs/`; file paths inside the scripts now point to `data/…` and `results/…` | |
| 162 | + (run them from the repository root). Bibliography switched from `plainnat` to `aer`. | |
| 163 | +- The script `13_revision_extras.py` was written for a Journal of Housing Economics revision round (clustered SEs, | |
| 164 | + in-sample ablation, extreme city effects). | |
added
data/processed/README.md
+10 −0
@@ -0,0 +1,10 @@ | ||
| 1 | +# Processed data (not committed) | |
| 2 | + | |
| 3 | +| File | Produced by | Content | | |
| 4 | +|---|---|---| | |
| 5 | +| `maisons_avec_prix.json` | house filter + cleaning described in `docs/METHODOLOGIE.md` §2–4 (combined Realtor + DuProprio houses with price) | 19,123 house listings | | |
| 6 | +| `specs.jsonl` | `scripts/01_extract_specs_llm.py` (GPT-4o-mini, strict JSON schema, resumable cache keyed by uid) | structured specifications extracted from descriptions | | |
| 7 | +| `facade_specs.jsonl` | `scripts/02_facade_vision.py` (vision–language model on the façade photo) | objective visual attributes (curb appeal, cladding, …) | | |
| 8 | +| `maisons_specs.json` / `.csv` | `scripts/03_merge_llm_features.py` | listings merged with LLM features | | |
| 9 | +| `maisons_hedonic.csv`, `maisons_hedonic_v4.csv` | `scripts/04_…`, `scripts/05_…` | final datasets with fitted price and `resid_pct` | | |
| 10 | +| `nn_robust_oos.npy`, `v5_blend_oos.npy` | `scripts/08_…`, `scripts/09_…` | out-of-sample predictions (5-fold, seed 42) reused by steps 10–14 | | |
added
data/raw/README.md
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +# Raw data — Realtor.ca and DuProprio listings (Apify crawlers, 2026-07-03) | |
| 2 | + | |
| 3 | +| File | Source | Records | Size | | |
| 4 | +|---|---|---|---| | |
| 5 | +| `dataset_realtor-property-crawler_2026-07-03_10-36-26-001.json` | Realtor.ca (MLS) via Apify `realtor-property-crawler` | 27,302 listings | 161 MB | | |
| 6 | +| `dataset_duproprio-property-crawler_2026-07-03_00-37-49-449.json` | DuProprio via Apify `duproprio-property-crawler` | 7,693 listings | 87 MB | | |
| 7 | + | |
| 8 | +Cross-sectional snapshot extracted on 2026-07-03 (no time dimension). Common fields: `source`, `url`, `listingId`, | |
| 9 | +`price`, `address{…}`, `property{…}`, `features[]`, `description`, `images[]`. Realtor adds `rooms[]` (room dimensions) | |
| 10 | +but no coordinates; DuProprio adds `coordinates{lat,lon}` and captioned images. Unique key: `uid = source:listingId`. | |
| 11 | +Houses (single-family + townhouse) with a valid price: 19,123 listings used in the paper. | |
| 12 | + | |
| 13 | +Not committed (size and crawler terms). Re-run the two Apify actors or request the snapshot from the author. | |
| 14 | +Full data dictionary and cleaning rules: `docs/METHODOLOGIE.md` (French). | |
added
docs/METHODOLOGIE.md
+172 −0
@@ -0,0 +1,172 @@ | ||
| 1 | +# Méthodologie — Modèle hédonique de prix des maisons (Québec) | |
| 2 | + | |
| 3 | +*Pipeline : Realtor.ca + DuProprio → nettoyage → feature engineering (règles + LLM + vision) → régression hédonique log-linéaire à effets fixes.* | |
| 4 | + | |
| 5 | +--- | |
| 6 | + | |
| 7 | +## 1. Origine des données | |
| 8 | + | |
| 9 | +| Source | Fichier | Nb enreg. | Nature | | |
| 10 | +|---|---|---|---| | |
| 11 | +| **Realtor.ca** | `dataset_realtor-property-crawler_...json` (161 Mo) | 27 302 | Annonces MLS via crawler Realtor (Apify) | | |
| 12 | +| **DuProprio** | `dataset_duproprio-property-crawler_...json` (87 Mo) | 7 693 | Annonces vente sans intermédiaire (Apify) | | |
| 13 | + | |
| 14 | +- **Collecte** : scrapers Apify, extraction le **2026-07-03**. Chaque enregistrement est un objet JSON structuré. | |
| 15 | +- **Champs communs** : `source`, `url`, `listingId`, `title`, `price`/`priceNumber`, `currency`, `address{full,street,city,province,postalCode,...}`, `property{...}`, `features[]`, `description`, `images[]`, `scrapedAt`. | |
| 16 | +- **Spécificités** : | |
| 17 | + - Realtor : `property.buildingType`, `storeys`, `yearBuilt`, `parkingSpaces`, **`rooms[]`** (dimensions par pièce), pas de coordonnées GPS, images sans caption utile. | |
| 18 | + - DuProprio : `property.type`, `levels`, **`coordinates{lat,lon}`** (7 691 présentes), images **avec captions** (`Kitchen`, etc.), pas de `rooms`. | |
| 19 | +- **Instantané transversal** (cross-section) : un seul point dans le temps → pas de dimension temporelle, donc pas d'effet fixe temporel nécessaire. | |
| 20 | + | |
| 21 | +**Identifiant unique** construit : `uid = source + ":" + listingId` (évite toute collision entre les deux espaces d'ID). | |
| 22 | + | |
| 23 | +--- | |
| 24 | + | |
| 25 | +## 2. Construction du dataset combiné « maisons » | |
| 26 | + | |
| 27 | +1. **Fusion** des deux tableaux JSON (schéma harmonisé). | |
| 28 | +2. **Définition de « maison »** (choix utilisateur : unifamiliales + townhouses) : | |
| 29 | + - Realtor : `buildingType ∈ {House, Row / Townhouse}` | |
| 30 | + - DuProprio : `type ∈ {Home, Bungalow, Semi Detached, Intergenerational, Country Home, Cottage, Villa, Townhouse}` | |
| 31 | + - Exclus : condos, plex (duplex→quintuplex), terrains/lots, fermes, commercial, mobile homes. | |
| 32 | +3. **Filtre prix présent** : `priceNumber` non nul et > 0. | |
| 33 | + | |
| 34 | +→ **19 273 maisons avec prix** (15 470 Realtor + 3 803 DuProprio). | |
| 35 | + | |
| 36 | +--- | |
| 37 | + | |
| 38 | +## 3. Nettoyage des aberrations | |
| 39 | + | |
| 40 | +Les prix scrappés contiennent des erreurs de parsing (min 899 $, max 368 G$). Règles appliquées : | |
| 41 | + | |
| 42 | +| Variable | Règle | Justification | | |
| 43 | +|---|---|---| | |
| 44 | +| Prix | garder **50 k$ – 15 M$** | plage plausible d'une maison au QC ; retire loyers mal parsés et concaténations | | |
| 45 | +| Superficie habitable | 300 – 20 000 ft² (sinon → manquant) | valeurs physiquement possibles | | |
| 46 | +| Terrain | 200 – 5 000 000 ft² (sinon → manquant) | idem | | |
| 47 | +| Chambres / sdb | 0–15 / 0–12 (sinon → manquant) | plafonds réalistes | | |
| 48 | +| **Prix/pied²** | garder **P0,5 – P99,5** (≈ 73–1 276 $/ft²) sur les biens à surface réelle | retire les erreurs résiduelles de parsing prix ou surface | | |
| 49 | + | |
| 50 | +→ **19 115 maisons** conservées (158 retirées). | |
| 51 | + | |
| 52 | +**Principe** : on borne plutôt que de supprimer quand une seule variable est aberrante (on met la variable à *manquant* et on l'impute), pour ne pas perdre l'observation entière. | |
| 53 | + | |
| 54 | +--- | |
| 55 | + | |
| 56 | +## 4. Construction des variables | |
| 57 | + | |
| 58 | +### 4.1 Variable cible | |
| 59 | +`ln_price = ln(priceNumber)` — transformation logarithmique (rend la distribution symétrique, linéarise les effets multiplicatifs, coefficients ≈ **% d'impact sur le prix**). | |
| 60 | + | |
| 61 | +### 4.2 Variables structurelles (règles déterministes) | |
| 62 | +- **Surface & terrain** : parsing d'unités hétérogènes → tout en **pieds carrés**. | |
| 63 | + - `"1535 sqft"`, `"1,333.2 ft²"`, `"11 561.52 ft²"` (séparateurs de milliers variés), `"961.6 m2"` (m² × 10,7639). | |
| 64 | +- **Reconstruction de surface via les pièces (Realtor)** : dimensions `"23 ft ,4 in x 13 ft ,8 in"` → pieds-décimaux → aire ; somme sur toutes les pièces. Calibration sur le recouvrement (ratio médian surface réelle / somme pièces ≈ **0,98**) → sert à combler `squareFeet` manquant. | |
| 65 | + - Constat : les annonces à pièces détaillées ≈ celles qui ont déjà `squareFeet` → gain de couverture faible, mais les **comptes/types de pièces** restent informatifs. | |
| 66 | +- **Autres** : `beds`, `baths`, `parking` (Realtor), `storeys`/`levels`, `age = 2026 − yearBuilt` (+ `age²` pour la courbure), drapeaux `has_*` marquant les valeurs imputées. | |
| 67 | + | |
| 68 | +### 4.3 Aménités (règles par mots-clés bilingues) | |
| 69 | +17 dummies détectés dans `features[] + description + title` (FR/EN) : piscine, bord de l'eau, garage, foyer, climatisation/thermopompe, sous-sol fini, cul-de-sac, golf, ski, proximité parc/école/autoroute/transport, absence de voisin arrière. | |
| 70 | + | |
| 71 | +### 4.4 Détail des pièces (Realtor, `rooms[]`) | |
| 72 | +Nombre de pièces, nombre de niveaux, chambre principale, sous-sol, bureau, salle familiale, atelier, **cellier à vin**, solarium, mezzanine, bibliothèque, buanderie. | |
| 73 | + | |
| 74 | +### 4.5 Signaux de listing | |
| 75 | +Nombre de photos (`n_images`, écrêté à 60), présence de visite virtuelle — proxies de gamme/qualité de la mise en marché. | |
| 76 | + | |
| 77 | +### 4.6 Variables géographiques (effets fixes) | |
| 78 | +- **Ville** nettoyée (Realtor : on retire le quartier entre parenthèses ; minuscules ; normalisation). Villes avec **≥ 20 annonces** → dummy propre ; les rares → `__other__`. ≈ 194 villes en effets fixes. | |
| 79 | +- **Type de propriété** harmonisé, regroupé si < 30 annonces. | |
| 80 | +- **Source** (Realtor/DuProprio) → dummy captant les différences systématiques entre plateformes. | |
| 81 | + | |
| 82 | +### 4.7 Imputation | |
| 83 | +Valeurs manquantes après nettoyage : imputées par **médiane par type de propriété** (puis médiane globale), **avec un drapeau `has_*`** pour que le modèle distingue « valeur observée » de « valeur imputée ». | |
| 84 | + | |
| 85 | +--- | |
| 86 | + | |
| 87 | +## 5. Mécanismes IA pour le feature engineering | |
| 88 | + | |
| 89 | +Deux couches IA extraient des variables **impossibles à obtenir par règles** — l'une du texte, l'autre des images. | |
| 90 | + | |
| 91 | +### 5.1 Extraction de specs depuis la description — **GPT-4o-mini** (OpenAI) | |
| 92 | +- **But** : transformer la description libre en 20 attributs structurés (état : à rénover→neuf, clé en main, cuisine/sdb rénovée, revêtements de sol, comptoirs quartz/granit, chauffage, vue sur l'eau, revenus locatifs, grand terrain, secteur tranquille, + mots-clés de vente). | |
| 93 | +- **Mécanisme** : appel Chat Completions avec **`response_format = json_schema` strict** (schéma imposé, énumérations fermées) → sortie JSON valide garantie, pas de parsing fragile. `temperature = 0` (déterminisme). Système : « extracteur factuel, uniquement d'après le texte ; absence → false/inconnu/[] ». | |
| 94 | +- **Ingénierie** : ~600 tokens in / ~110 out par annonce ; **parallèle** (24 threads), **résumable** (cache `specs.jsonl` clé = uid), **retry** exponentiel sur 429/5xx. | |
| 95 | +- **Coût/temps** : ~$3 / ~19 k annonces. | |
| 96 | + | |
| 97 | +### 5.2 Attributs visuels de façade — **Claude Haiku 4.5** (Anthropic, vision) | |
| 98 | +- **But** : lire ce que le texte ne dit pas — **état visuel réel, attrait extérieur (curb appeal), niveau de rénovation, matériaux, aménagement paysager, cadre (urbain/banlieue/rural/bord d'eau/boisé), indice de luxe, qualité de la photo**. | |
| 99 | +- **Sélection d'image** : 1 photo/annonce = la façade. Realtor → 1re photo. DuProprio → photo dont la caption est l'adresse (pas une pièce), via exclusion de mots-clés de pièces. | |
| 100 | +- **Mécanisme** : Messages API, image passée par **URL** (`source.type = url`, pas de téléchargement), **`output_config.format = json_schema`** (12 attributs, énumérations fermées). Système : « évaluateur objectif ; scores fondés sur critères concrets (surfaces, matériaux, aménagement), pas sur un goût personnel ; si non-extérieur → is_exterior=false ». | |
| 101 | +- **Ingénierie** : ~2 470 tokens in / ~100 out par image ; **parallèle** (16 threads), **résumable** (cache `facade_specs.jsonl`), **retry** sur 429/5xx/529. | |
| 102 | +- **Coût/temps** : ~$57 / ~19 k images. | |
| 103 | +- **Pourquoi Haiku** : modèle vision le moins cher et rapide, largement suffisant pour de la classification d'attributs concrets. | |
| 104 | + | |
| 105 | +### 5.3 Pourquoi le JSON structuré strict est central | |
| 106 | +Le schéma imposé (types + `enum` + champs requis + `additionalProperties:false`) fait valider la sortie **au niveau de l'API** : le modèle est forcé de renvoyer exactement les champs attendus, dans le bon domaine de valeurs. On obtient des variables **directement tabulables**, reproductibles, sans post-traitement de texte libre. | |
| 107 | + | |
| 108 | +### 5.4 Intégration au dataset | |
| 109 | +Les sorties LLM (clé `uid`) sont converties en variables numériques : booléens → 0/1, `condition` → score ordinal 0-5, revêtements → dummies (bois franc, quartz/granit), vue → dummy eau, etc., **avec un drapeau `has_gpt` / `is_exterior`** pour marquer la disponibilité. | |
| 110 | + | |
| 111 | +--- | |
| 112 | + | |
| 113 | +## 6. Modèle économétrique | |
| 114 | + | |
| 115 | +### 6.1 Spécification — hédonique log-linéaire à effets fixes | |
| 116 | +$$\ln(P_i) = \alpha + \beta\,\ln(\text{surface}_i) + \gamma\,\ln(\text{terrain}_i) + \sum_k \delta_k X_{ki} + \sum_c \lambda_c \mathbb{1}[\text{ville}_i=c] + \varepsilon_i$$ | |
| 117 | + | |
| 118 | +- **Log-log** sur surface et terrain → coefficients = **élasticités** (% prix par % surface). | |
| 119 | +- **Log-niveau** sur le reste → un coefficient $b$ ≈ **$(e^b-1)\times100$ % d'impact** sur le prix. | |
| 120 | +- **Effets fixes ville** ($\lambda_c$) : absorbent tout le différentiel de localisation (le plus gros déterminant du prix). Chaque coefficient = prime/décote de la ville vs la ville de référence. | |
| 121 | +- **Effets fixes type de propriété** et **source**. | |
| 122 | + | |
| 123 | +### 6.2 Double spécification (couverture vs richesse) | |
| 124 | +Realtor ne rapporte `squareFeet` que pour ~38 % des annonces → deux modèles, prédictions combinées : | |
| 125 | +- **Modèle A (riche)** : inclut `ln(surface)`, estimé sur les biens à surface **mesurée** (réelle ou reconstruite via pièces). Meilleur ajustement. | |
| 126 | +- **Modèle B (large)** : sans surface, estimé sur **toutes** les maisons → sert à pricer celles sans surface (via beds/baths/terrain/pièces + specs). | |
| 127 | +- Prédiction finale : A si surface mesurée, sinon B. | |
| 128 | + | |
| 129 | +### 6.3 Estimation & inférence | |
| 130 | +- **MCO (OLS)** sur `ln_price`. | |
| 131 | +- **Erreurs-types robustes à l'hétéroscédasticité HC3** (les résidus immobiliers sont hétéroscédastiques ; HC3 est conservateur en petit/moyen échantillon). | |
| 132 | +- **Correction du biais de rétrotransformation** : $\hat{P} = e^{\hat{\ln P}} \times e^{\hat\sigma^2/2}$ (correction normale de Duan) — sinon $e^{\hat{\ln P}}$ sous-estime systématiquement le niveau. | |
| 133 | + | |
| 134 | +### 6.4 Interprétation | |
| 135 | +Coefficients rapportés en **% d'impact prix** + significativité (p-values sur SE HC3). Effets fixes ville présentés comme primes de localisation. | |
| 136 | + | |
| 137 | +--- | |
| 138 | + | |
| 139 | +## 7. Statistiques & diagnostics | |
| 140 | + | |
| 141 | +| Indicateur | Signification | | |
| 142 | +|---|---| | |
| 143 | +| **R² / R²-ajusté** | part de variance de `ln_price` expliquée | | |
| 144 | +| **Écart-type des résidus (log)** | → **erreur typique en %** = $(e^{\sigma}-1)\times100$ | | |
| 145 | +| **Erreur absolue médiane** | \|prix réel / prix prédit − 1\| médian sur toutes les prédictions | | |
| 146 | +| **% à ±15 % / ±25 %** | part des prédictions dans la fourchette | | |
| 147 | +| **`resid_pct`** par annonce | surprix/sous-prix vs modèle → repère aubaines & surévaluations | | |
| 148 | + | |
| 149 | +Progression observée : v2 (structurel) R²≈0,70 → v3 (+ pièces/photos) R²≈0,72 → v4 (+ specs GPT) → v5 (+ vision) attendu. | |
| 150 | + | |
| 151 | +--- | |
| 152 | + | |
| 153 | +## 8. Limites & biais (transparence) | |
| 154 | + | |
| 155 | +- **Endogénéité / non-causalité** : de belles photos ou des specs riches corrèlent avec des courtiers haut de gamme et des biens chers → coefficients **prédictifs**, pas causaux. À ne pas lire comme « ajouter une piscine ⇒ +X % ». | |
| 156 | +- **Biais de sélection** : uniquement des biens **en vente** (prix demandé, pas prix de transaction) ; les biens sans surface/pièces sont non aléatoires (annonces moins complètes). | |
| 157 | +- **Imputation** : les médianes par type réduisent la variance des variables imputées (atténuation possible des coefficients) — d'où les drapeaux `has_*`. | |
| 158 | +- **Extraction LLM** : erreurs/hallucinations possibles ; `temperature=0` + schéma strict + énumérations fermées les limitent, mais pas de validation humaine systématique. La vision juge sur **une seule** photo. | |
| 159 | +- **Instantané unique** : pas de dynamique de marché ; primes de localisation figées au 2026-07-03. | |
| 160 | +- **Couverture géographique** : villes < 20 annonces regroupées en `__other__` (moins de finesse spatiale là où les données sont rares). | |
| 161 | + | |
| 162 | +--- | |
| 163 | + | |
| 164 | +## 9. Fichiers produits | |
| 165 | + | |
| 166 | +| Fichier | Contenu | | |
| 167 | +|---|---| | |
| 168 | +| `hedonic.py`, `hedonic_v4.py` | pipelines reproductibles (nettoyage + modèles) | | |
| 169 | +| `extract_specs.py`, `facade_vision.py` | extracteurs LLM (texte / vision), parallèles & résumables | | |
| 170 | +| `specs.jsonl`, `facade_specs.jsonl` | sorties LLM brutes (clé uid) | | |
| 171 | +| `maisons_hedonic*.csv` | dataset final + prix prédit + `resid_pct` | | |
| 172 | +| `hedonic_*_summary.txt` | sorties statistiques complètes | | |
added
docs/highlights_elsevier.tex
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Highlights — separate file required at Elsevier submission | |
| 3 | +% (3-5 bullets, max 85 characters each, results-oriented) | |
| 4 | +% ============================================================================ | |
| 5 | +\documentclass[12pt]{article} | |
| 6 | +\usepackage[utf8]{inputenc} | |
| 7 | +\usepackage[T1]{fontenc} | |
| 8 | +\begin{document} | |
| 9 | +\section*{Highlights} | |
| 10 | +\begin{itemize} | |
| 11 | + \item LLM and vision models turn listing text and photos into hedonic regressors | |
| 12 | + \item Schema-constrained decoding makes AI outputs reproducible econometric variables | |
| 13 | + \item Two AI feature layers add four points of $R^2$ to a rich hedonic model | |
| 14 | + \item Façade-vision features give the largest out-of-sample gain of any block | |
| 15 | + \item Curb appeal, materials, and luxury cues carry sensible implicit prices | |
| 16 | +\end{itemize} | |
| 17 | +\end{document} | |
added
docs/methodologie.pdf
+0 −0
Binary file not shown.
added
docs/methodologie.tex
+372 −0
@@ -0,0 +1,372 @@ | ||
| 1 | +\documentclass[11pt,a4paper]{article} | |
| 2 | + | |
| 3 | +\usepackage[utf8]{inputenc} | |
| 4 | +\usepackage[T1]{fontenc} | |
| 5 | +\usepackage[french]{babel} | |
| 6 | +\usepackage{lmodern} | |
| 7 | +\usepackage{amsmath,amssymb} | |
| 8 | +\usepackage{booktabs} | |
| 9 | +\usepackage{longtable} | |
| 10 | +\usepackage{array} | |
| 11 | +\usepackage{geometry} | |
| 12 | +\usepackage{xcolor} | |
| 13 | +\usepackage{enumitem} | |
| 14 | +\usepackage{hyperref} | |
| 15 | +\usepackage{fancyhdr} | |
| 16 | + | |
| 17 | +\geometry{margin=2.4cm} | |
| 18 | +\hypersetup{colorlinks=true, linkcolor=blue!55!black, urlcolor=blue!55!black, | |
| 19 | + citecolor=blue!55!black} | |
| 20 | + | |
| 21 | +\newcommand{\code}[1]{\texttt{\small #1}} | |
| 22 | +\renewcommand{\arraystretch}{1.25} | |
| 23 | + | |
| 24 | +\pagestyle{fancy} | |
| 25 | +\fancyhf{} | |
| 26 | +\lhead{\small Modèle hédonique de prix — Québec} | |
| 27 | +\rhead{\small\thepage} | |
| 28 | +\renewcommand{\headrulewidth}{0.4pt} | |
| 29 | + | |
| 30 | +\title{\textbf{Méthodologie}\\[2mm] | |
| 31 | +\large Modèle hédonique de prix des maisons (Québec)\\[1mm] | |
| 32 | +\normalsize Realtor.ca $+$ DuProprio $\rightarrow$ nettoyage $\rightarrow$ ingénierie | |
| 33 | +de variables (règles $+$ LLM $+$ vision) $\rightarrow$ régression hédonique | |
| 34 | +log-linéaire à effets fixes} | |
| 35 | +\author{} | |
| 36 | +\date{Données extraites le 3 juillet 2026} | |
| 37 | + | |
| 38 | +\begin{document} | |
| 39 | +\maketitle | |
| 40 | +\thispagestyle{fancy} | |
| 41 | + | |
| 42 | +\begin{abstract} | |
| 43 | +\noindent Ce document décrit la méthodologie complète de construction d'un modèle | |
| 44 | +hédonique de prix pour les maisons unifamiliales et maisons en rangée du Québec. | |
| 45 | +Le pipeline combine deux jeux de données d'annonces immobilières, un nettoyage des | |
| 46 | +valeurs aberrantes fondé sur des règles, une ingénierie de variables à quatre | |
| 47 | +origines (règles déterministes, extraction de texte par grand modèle de langage, | |
| 48 | +analyse d'images par modèle de vision, effets fixes géographiques) et une | |
| 49 | +estimation par moindres carrés ordinaires du logarithme du prix avec erreurs-types | |
| 50 | +robustes à l'hétéroscédasticité. | |
| 51 | +\end{abstract} | |
| 52 | + | |
| 53 | +\tableofcontents | |
| 54 | +\bigskip | |
| 55 | + | |
| 56 | +%====================================================================== | |
| 57 | +\section{Origine des données} | |
| 58 | + | |
| 59 | +\begin{longtable}{@{}p{2.4cm}p{5.2cm}rp{4.6cm}@{}} | |
| 60 | +\toprule | |
| 61 | +\textbf{Source} & \textbf{Fichier} & \textbf{Nb} & \textbf{Nature}\\ | |
| 62 | +\midrule | |
| 63 | +\endhead | |
| 64 | +Realtor.ca & \code{dataset\_realtor-\dots.json} (161 Mo) & 27\,302 & | |
| 65 | +Annonces MLS via crawler Apify\\ | |
| 66 | +DuProprio & \code{dataset\_duproprio-\dots.json} (87 Mo) & 7\,693 & | |
| 67 | +Annonces de vente sans intermédiaire (Apify)\\ | |
| 68 | +\bottomrule | |
| 69 | +\end{longtable} | |
| 70 | + | |
| 71 | +\begin{itemize}[leftmargin=1.2em] | |
| 72 | + \item \textbf{Collecte} : scrapers Apify, extraction le \textbf{2026-07-03}. Chaque | |
| 73 | + enregistrement est un objet JSON structuré. | |
| 74 | + \item \textbf{Champs communs} : \code{source}, \code{url}, \code{listingId}, | |
| 75 | + \code{title}, \code{price}/\code{priceNumber}, \code{currency}, | |
| 76 | + \code{address\{...\}}, \code{property\{...\}}, \code{features[]}, | |
| 77 | + \code{description}, \code{images[]}, \code{scrapedAt}. | |
| 78 | + \item \textbf{Spécificités} : | |
| 79 | + \begin{itemize}[leftmargin=1.2em] | |
| 80 | + \item Realtor : \code{buildingType}, \code{storeys}, \code{yearBuilt}, | |
| 81 | + \code{parkingSpaces}, \textbf{\code{rooms[]}} (dimensions par pièce) ; | |
| 82 | + pas de coordonnées GPS ; images sans légende utile. | |
| 83 | + \item DuProprio : \code{type}, \code{levels}, | |
| 84 | + \textbf{\code{coordinates}} (7\,691 présentes) ; images \textbf{avec légendes} | |
| 85 | + (\emph{Kitchen}, etc.) ; pas de \code{rooms}. | |
| 86 | + \end{itemize} | |
| 87 | + \item \textbf{Instantané transversal} (une seule date) : pas de dimension | |
| 88 | + temporelle, donc pas d'effet fixe temporel requis. | |
| 89 | +\end{itemize} | |
| 90 | + | |
| 91 | +\noindent\textbf{Identifiant unique} : \code{uid = source + ":" + listingId} | |
| 92 | +(évite toute collision entre les deux espaces d'identifiants). | |
| 93 | + | |
| 94 | +%====================================================================== | |
| 95 | +\section{Construction du dataset «~maisons~»} | |
| 96 | + | |
| 97 | +\begin{enumerate}[leftmargin=1.4em] | |
| 98 | + \item \textbf{Fusion} des deux tableaux JSON (schéma harmonisé). | |
| 99 | + \item \textbf{Définition de «~maison~»} (unifamiliales $+$ maisons en rangée) : | |
| 100 | + \begin{itemize}[leftmargin=1.2em] | |
| 101 | + \item Realtor : \code{buildingType} $\in \{$House, Row/Townhouse$\}$ ; | |
| 102 | + \item DuProprio : \code{type} $\in \{$Home, Bungalow, Semi Detached, | |
| 103 | + Intergenerational, Country Home, Cottage, Villa, Townhouse$\}$ ; | |
| 104 | + \item \emph{Exclus} : condos, plex (duplex$\rightarrow$quintuplex), | |
| 105 | + terrains/lots, fermes, commercial, maisons mobiles. | |
| 106 | + \end{itemize} | |
| 107 | + \item \textbf{Filtre prix présent} : \code{priceNumber} non nul et $>0$. | |
| 108 | +\end{enumerate} | |
| 109 | + | |
| 110 | +\noindent$\rightarrow$ \textbf{19\,273 maisons avec prix} (15\,470 Realtor | |
| 111 | +$+$ 3\,803 DuProprio). | |
| 112 | + | |
| 113 | +%====================================================================== | |
| 114 | +\section{Nettoyage des valeurs aberrantes} | |
| 115 | + | |
| 116 | +Les prix extraits contiennent des erreurs d'analyse syntaxique (minimum | |
| 117 | +observé 899~\$, maximum 368~G\$). Règles appliquées : | |
| 118 | + | |
| 119 | +\begin{longtable}{@{}p{3.0cm}p{6.2cm}p{5.2cm}@{}} | |
| 120 | +\toprule | |
| 121 | +\textbf{Variable} & \textbf{Règle} & \textbf{Justification}\\ | |
| 122 | +\midrule | |
| 123 | +\endhead | |
| 124 | +Prix & garder \textbf{50\,k\$ -- 15\,M\$} & plage plausible au QC ; retire loyers | |
| 125 | +mal analysés et concaténations\\ | |
| 126 | +Surface habitable & 300 -- 20\,000 ft$^2$ (sinon $\rightarrow$ manquant) & | |
| 127 | +valeurs physiquement possibles\\ | |
| 128 | +Terrain & 200 -- 5\,000\,000 ft$^2$ (sinon $\rightarrow$ manquant) & idem\\ | |
| 129 | +Chambres / sdb & 0--15 / 0--12 (sinon $\rightarrow$ manquant) & plafonds réalistes\\ | |
| 130 | +Prix/pied$^2$ & garder \textbf{P0{,}5 -- P99{,}5} ($\approx 73$--$1276~\$$/ft$^2$) & | |
| 131 | +retire les erreurs résiduelles de prix ou surface\\ | |
| 132 | +\bottomrule | |
| 133 | +\end{longtable} | |
| 134 | + | |
| 135 | +\noindent$\rightarrow$ \textbf{19\,115 maisons} conservées (158 retirées). | |
| 136 | + | |
| 137 | +\paragraph{Principe.} On \emph{borne} plutôt que de supprimer quand une seule | |
| 138 | +variable est aberrante : la variable est mise à \emph{manquant} puis imputée, pour | |
| 139 | +ne pas perdre l'observation entière. | |
| 140 | + | |
| 141 | +%====================================================================== | |
| 142 | +\section{Construction des variables} | |
| 143 | + | |
| 144 | +\subsection{Variable cible} | |
| 145 | +\[ | |
| 146 | +\code{ln\_price} = \ln(\code{priceNumber}) | |
| 147 | +\] | |
| 148 | +La transformation logarithmique rend la distribution symétrique, linéarise les | |
| 149 | +effets multiplicatifs et donne des coefficients interprétables en pourcentage | |
| 150 | +d'impact sur le prix. | |
| 151 | + | |
| 152 | +\subsection{Variables structurelles (règles déterministes)} | |
| 153 | +\begin{itemize}[leftmargin=1.2em] | |
| 154 | + \item \textbf{Surface \& terrain} : analyse d'unités hétérogènes, tout converti en | |
| 155 | + \textbf{pieds carrés} : \code{"1535 sqft"}, \code{"1,333.2 ft²"}, | |
| 156 | + \code{"11 561.52 ft²"} (séparateurs de milliers variés), | |
| 157 | + \code{"961.6 m2"} ($\times 10{,}7639$). | |
| 158 | + \item \textbf{Reconstruction de surface via les pièces (Realtor)} : dimensions | |
| 159 | + \code{"23 ft ,4 in x 13 ft ,8 in"} $\rightarrow$ pieds décimaux $\rightarrow$ aire, | |
| 160 | + sommées sur toutes les pièces. Calibration sur le recouvrement (ratio médian | |
| 161 | + surface réelle / somme des pièces $\approx 0{,}98$) pour combler | |
| 162 | + \code{squareFeet} manquant. | |
| 163 | + \item \textbf{Autres} : \code{beds}, \code{baths}, \code{parking}, | |
| 164 | + \code{storeys}/\code{levels}, \code{age} $= 2026 - \code{yearBuilt}$ (avec | |
| 165 | + \code{age}$^2$ pour la courbure), drapeaux \code{has\_*} marquant les valeurs | |
| 166 | + imputées. | |
| 167 | +\end{itemize} | |
| 168 | + | |
| 169 | +\subsection{Aménités (règles par mots-clés bilingues)} | |
| 170 | +17 variables indicatrices détectées dans \code{features[]} $+$ \code{description} | |
| 171 | +$+$ \code{title} (FR/EN) : piscine, bord de l'eau, garage, foyer, climatisation/ | |
| 172 | +thermopompe, sous-sol fini, cul-de-sac, golf, ski, proximité parc/école/autoroute/ | |
| 173 | +transport, absence de voisin arrière. | |
| 174 | + | |
| 175 | +\subsection{Détail des pièces (Realtor)} | |
| 176 | +Nombre de pièces, nombre de niveaux, chambre principale, sous-sol, bureau, salle | |
| 177 | +familiale, atelier, cellier à vin, solarium, mezzanine, bibliothèque, buanderie. | |
| 178 | + | |
| 179 | +\subsection{Signaux de mise en marché} | |
| 180 | +Nombre de photos (\code{n\_images}, écrêté à 60), présence de visite virtuelle — | |
| 181 | +indicateurs indirects de gamme et de qualité de l'annonce. | |
| 182 | + | |
| 183 | +\subsection{Variables géographiques (effets fixes)} | |
| 184 | +\begin{itemize}[leftmargin=1.2em] | |
| 185 | + \item \textbf{Ville} nettoyée (Realtor : retrait du quartier entre parenthèses ; | |
| 186 | + minuscules ; normalisation). Villes avec $\ge 20$ annonces $\rightarrow$ | |
| 187 | + indicatrice propre, les rares $\rightarrow$ \code{\_\_other\_\_} | |
| 188 | + ($\approx 194$ villes en effets fixes). | |
| 189 | + \item \textbf{Type de propriété} harmonisé, regroupé si $<30$ annonces. | |
| 190 | + \item \textbf{Source} (Realtor/DuProprio) $\rightarrow$ indicatrice captant les | |
| 191 | + différences systématiques entre plateformes. | |
| 192 | +\end{itemize} | |
| 193 | + | |
| 194 | +\subsection{Imputation} | |
| 195 | +Valeurs manquantes après nettoyage : imputées par \textbf{médiane par type de | |
| 196 | +propriété} (puis médiane globale), \textbf{avec un drapeau \code{has\_*}} pour que | |
| 197 | +le modèle distingue valeur observée et valeur imputée. | |
| 198 | + | |
| 199 | +%====================================================================== | |
| 200 | +\section{Mécanismes d'IA pour l'ingénierie de variables} | |
| 201 | + | |
| 202 | +Deux couches d'IA extraient des variables impossibles à obtenir par règles~: l'une | |
| 203 | +depuis le texte, l'autre depuis les images. | |
| 204 | + | |
| 205 | +\subsection{Extraction de specs depuis la description --- GPT-4o-mini} | |
| 206 | +\begin{itemize}[leftmargin=1.2em] | |
| 207 | + \item \textbf{But} : transformer la description libre en 20 attributs structurés | |
| 208 | + (état~: à rénover$\rightarrow$neuf ; clé en main ; cuisine/sdb rénovée ; | |
| 209 | + revêtements de sol ; comptoirs quartz/granit ; chauffage ; vue sur l'eau ; | |
| 210 | + revenus locatifs ; grand terrain ; secteur tranquille ; mots-clés de vente). | |
| 211 | + \item \textbf{Mécanisme} : appel \emph{Chat Completions} avec | |
| 212 | + \code{response\_format = json\_schema} \textbf{strict} (schéma imposé, | |
| 213 | + énumérations fermées) $\rightarrow$ sortie JSON valide garantie, sans analyse | |
| 214 | + fragile. \code{temperature = 0} (déterminisme). Instruction système : extracteur | |
| 215 | + factuel, uniquement d'après le texte ; absence $\rightarrow$ false/inconnu/[]. | |
| 216 | + \item \textbf{Ingénierie} : $\sim$600 tokens en entrée / $\sim$110 en sortie par | |
| 217 | + annonce ; \textbf{parallèle} (24 fils), \textbf{résumable} (cache | |
| 218 | + \code{specs.jsonl}, clé \code{uid}), \textbf{réessais} exponentiels sur 429/5xx. | |
| 219 | + \item \textbf{Coût / temps} : $\sim$\$3 pour $\sim$19\,k annonces. | |
| 220 | +\end{itemize} | |
| 221 | + | |
| 222 | +\subsection{Attributs visuels de façade --- Claude Haiku 4.5 (vision)} | |
| 223 | +\begin{itemize}[leftmargin=1.2em] | |
| 224 | + \item \textbf{But} : lire ce que le texte ne dit pas --- état visuel réel, | |
| 225 | + attrait extérieur (\emph{curb appeal}), niveau de rénovation, matériaux, | |
| 226 | + aménagement paysager, cadre (urbain/banlieue/rural/bord d'eau/boisé), indice de | |
| 227 | + luxe, qualité de la photo. | |
| 228 | + \item \textbf{Sélection d'image} : une photo/annonce $=$ la façade. Realtor | |
| 229 | + $\rightarrow$ 1\up{re} photo ; DuProprio $\rightarrow$ photo dont la légende est | |
| 230 | + l'adresse (pas une pièce), par exclusion de mots-clés de pièces. | |
| 231 | + \item \textbf{Mécanisme} : \emph{Messages API}, image passée par \textbf{URL} | |
| 232 | + (\code{source.type = url}, sans téléchargement), | |
| 233 | + \code{output\_config.format = json\_schema} (12 attributs, énumérations fermées). | |
| 234 | + Instruction système : évaluateur objectif ; scores fondés sur des critères | |
| 235 | + concrets (surfaces, matériaux, aménagement), pas sur un goût personnel. | |
| 236 | + \item \textbf{Ingénierie} : $\sim$2\,470 tokens en entrée / $\sim$100 en sortie | |
| 237 | + par image ; \textbf{parallèle} (16 fils), \textbf{résumable} (cache | |
| 238 | + \code{facade\_specs.jsonl}), \textbf{réessais} sur 429/5xx/529. | |
| 239 | + \item \textbf{Coût / temps} : $\sim$\$57 pour $\sim$19\,k images. | |
| 240 | + \item \textbf{Choix de Haiku} : modèle de vision le moins cher et rapide, | |
| 241 | + suffisant pour de la classification d'attributs concrets. | |
| 242 | +\end{itemize} | |
| 243 | + | |
| 244 | +\subsection{Rôle central du JSON structuré strict} | |
| 245 | +Le schéma imposé (types $+$ \code{enum} $+$ champs requis $+$ | |
| 246 | +\code{additionalProperties:false}) fait valider la sortie \textbf{au niveau de | |
| 247 | +l'API}~: le modèle renvoie exactement les champs attendus, dans le bon domaine de | |
| 248 | +valeurs. On obtient des variables directement tabulables et reproductibles, sans | |
| 249 | +post-traitement de texte libre. | |
| 250 | + | |
| 251 | +\subsection{Intégration au dataset} | |
| 252 | +Les sorties des modèles (clé \code{uid}) sont converties en variables numériques~: | |
| 253 | +booléens $\rightarrow$ 0/1, état $\rightarrow$ score ordinal 0--5, revêtements | |
| 254 | +$\rightarrow$ indicatrices (bois franc, quartz/granit), vue $\rightarrow$ | |
| 255 | +indicatrice eau, etc., avec un drapeau \code{has\_gpt} / \code{is\_exterior} | |
| 256 | +marquant la disponibilité. | |
| 257 | + | |
| 258 | +%====================================================================== | |
| 259 | +\section{Modèle économétrique} | |
| 260 | + | |
| 261 | +\subsection{Spécification --- hédonique log-linéaire à effets fixes} | |
| 262 | +\begin{equation} | |
| 263 | +\ln(P_i) = \alpha + \beta\,\ln(\text{surface}_i) + \gamma\,\ln(\text{terrain}_i) | |
| 264 | ++ \sum_k \delta_k X_{ki} | |
| 265 | ++ \sum_c \lambda_c\,\mathbb{1}[\text{ville}_i=c] | |
| 266 | ++ \varepsilon_i | |
| 267 | +\end{equation} | |
| 268 | + | |
| 269 | +\begin{itemize}[leftmargin=1.2em] | |
| 270 | + \item \textbf{Log--log} sur surface et terrain $\rightarrow$ coefficients $=$ | |
| 271 | + \textbf{élasticités} (\% de prix par \% de surface). | |
| 272 | + \item \textbf{Log--niveau} sur le reste $\rightarrow$ un coefficient $b$ vaut | |
| 273 | + $\approx (e^{b}-1)\times 100$~\% d'impact sur le prix. | |
| 274 | + \item \textbf{Effets fixes ville} ($\lambda_c$) : absorbent le différentiel de | |
| 275 | + localisation (premier déterminant du prix) ; chaque coefficient $=$ prime/décote | |
| 276 | + de la ville vs la référence. | |
| 277 | + \item \textbf{Effets fixes} type de propriété et source. | |
| 278 | +\end{itemize} | |
| 279 | + | |
| 280 | +\subsection{Double spécification (couverture vs richesse)} | |
| 281 | +Realtor ne rapporte \code{squareFeet} que pour $\sim$38~\% des annonces, d'où deux | |
| 282 | +modèles et des prédictions combinées~: | |
| 283 | +\begin{itemize}[leftmargin=1.2em] | |
| 284 | + \item \textbf{Modèle A (riche)} : inclut $\ln(\text{surface})$, estimé sur les | |
| 285 | + biens à surface \emph{mesurée} (réelle ou reconstruite via les pièces). Meilleur | |
| 286 | + ajustement. | |
| 287 | + \item \textbf{Modèle B (large)} : sans surface, estimé sur \emph{toutes} les | |
| 288 | + maisons $\rightarrow$ price celles sans surface (beds/baths/terrain/pièces $+$ | |
| 289 | + specs). | |
| 290 | + \item \textbf{Prédiction finale} : A si surface mesurée, sinon B. | |
| 291 | +\end{itemize} | |
| 292 | + | |
| 293 | +\subsection{Estimation et inférence} | |
| 294 | +\begin{itemize}[leftmargin=1.2em] | |
| 295 | + \item \textbf{Moindres carrés ordinaires} sur \code{ln\_price}. | |
| 296 | + \item \textbf{Erreurs-types robustes à l'hétéroscédasticité HC3} (les résidus | |
| 297 | + immobiliers sont hétéroscédastiques ; HC3 est conservateur en échantillon | |
| 298 | + moyen). | |
| 299 | + \item \textbf{Correction du biais de rétrotransformation} : | |
| 300 | + \[ | |
| 301 | + \hat{P} = e^{\widehat{\ln P}} \times e^{\hat\sigma^2/2} | |
| 302 | + \] | |
| 303 | + (correction normale de Duan) --- sinon $e^{\widehat{\ln P}}$ sous-estime | |
| 304 | + systématiquement le niveau de prix. | |
| 305 | +\end{itemize} | |
| 306 | + | |
| 307 | +\subsection{Interprétation} | |
| 308 | +Coefficients rapportés en \textbf{\% d'impact prix} $+$ significativité (p-values | |
| 309 | +sur SE HC3). Effets fixes ville présentés comme primes de localisation. | |
| 310 | + | |
| 311 | +%====================================================================== | |
| 312 | +\section{Statistiques et diagnostics} | |
| 313 | + | |
| 314 | +\begin{longtable}{@{}p{5.2cm}p{9.4cm}@{}} | |
| 315 | +\toprule | |
| 316 | +\textbf{Indicateur} & \textbf{Signification}\\ | |
| 317 | +\midrule | |
| 318 | +\endhead | |
| 319 | +$R^2$ / $R^2$ ajusté & part de variance de \code{ln\_price} expliquée\\ | |
| 320 | +Écart-type des résidus (log) & $\rightarrow$ erreur typique en \% $=(e^{\sigma}-1)\times100$\\ | |
| 321 | +Erreur absolue médiane & $|\,$prix réel $/$ prix prédit $-1\,|$ médian\\ | |
| 322 | +\% à $\pm15$~\% / $\pm25$~\% & part des prédictions dans la fourchette\\ | |
| 323 | +\code{resid\_pct} par annonce & surprix/sous-prix vs modèle | |
| 324 | +$\rightarrow$ repère aubaines et surévaluations\\ | |
| 325 | +\bottomrule | |
| 326 | +\end{longtable} | |
| 327 | + | |
| 328 | +\noindent Progression observée : v2 (structurel) $R^2\approx0{,}70$ | |
| 329 | +$\rightarrow$ v3 ($+$ pièces/photos) $R^2\approx0{,}72$ | |
| 330 | +$\rightarrow$ v4 ($+$ specs GPT) $\rightarrow$ v5 ($+$ vision) attendu. | |
| 331 | + | |
| 332 | +%====================================================================== | |
| 333 | +\section{Limites et biais} | |
| 334 | + | |
| 335 | +\begin{itemize}[leftmargin=1.2em] | |
| 336 | + \item \textbf{Endogénéité / non-causalité} : de belles photos ou des specs riches | |
| 337 | + corrèlent avec des courtiers haut de gamme et des biens chers $\rightarrow$ | |
| 338 | + coefficients \emph{prédictifs}, pas causaux. | |
| 339 | + \item \textbf{Biais de sélection} : uniquement des biens \emph{en vente} (prix | |
| 340 | + demandé, pas prix de transaction) ; les biens sans surface/pièces sont non | |
| 341 | + aléatoires (annonces moins complètes). | |
| 342 | + \item \textbf{Imputation} : les médianes par type réduisent la variance des | |
| 343 | + variables imputées (atténuation possible des coefficients), d'où les drapeaux | |
| 344 | + \code{has\_*}. | |
| 345 | + \item \textbf{Extraction LLM} : erreurs/hallucinations possibles ; | |
| 346 | + \code{temperature=0} $+$ schéma strict $+$ énumérations fermées les limitent, | |
| 347 | + sans validation humaine systématique. La vision juge sur \emph{une seule} photo. | |
| 348 | + \item \textbf{Instantané unique} : pas de dynamique de marché ; primes de | |
| 349 | + localisation figées au 2026-07-03. | |
| 350 | + \item \textbf{Couverture géographique} : villes $<20$ annonces regroupées en | |
| 351 | + \code{\_\_other\_\_}. | |
| 352 | +\end{itemize} | |
| 353 | + | |
| 354 | +%====================================================================== | |
| 355 | +\section{Fichiers produits} | |
| 356 | + | |
| 357 | +\begin{longtable}{@{}p{5.4cm}p{9.2cm}@{}} | |
| 358 | +\toprule | |
| 359 | +\textbf{Fichier} & \textbf{Contenu}\\ | |
| 360 | +\midrule | |
| 361 | +\endhead | |
| 362 | +\code{hedonic.py}, \code{hedonic\_v4.py} & pipelines reproductibles (nettoyage $+$ | |
| 363 | +modèles)\\ | |
| 364 | +\code{extract\_specs.py}, \code{facade\_vision.py} & extracteurs LLM (texte / | |
| 365 | +vision), parallèles et résumables\\ | |
| 366 | +\code{specs.jsonl}, \code{facade\_specs.jsonl} & sorties LLM brutes (clé \code{uid})\\ | |
| 367 | +\code{maisons\_hedonic*.csv} & dataset final $+$ prix prédit $+$ \code{resid\_pct}\\ | |
| 368 | +\code{hedonic\_*\_summary.txt} & sorties statistiques complètes\\ | |
| 369 | +\bottomrule | |
| 370 | +\end{longtable} | |
| 371 | + | |
| 372 | +\end{document} | |
added
paper/.latexmkrc
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +$pdf_mode = 1; | |
| 2 | +$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S'; | |
| 3 | +$bibtex_use = 2; | |
| 4 | +$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg'; | |
| 5 | +@default_files = ('main.tex'); | |
added
paper/Makefile
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +.PHONY: all clean distclean | |
| 2 | + | |
| 3 | +all: | |
| 4 | + latexmk | |
| 5 | + | |
| 6 | +clean: | |
| 7 | + latexmk -c | |
| 8 | + | |
| 9 | +distclean: | |
| 10 | + latexmk -C | |
| 11 | + rm -f *.synctex.gz *.run.xml *.bbl *.bcf | |
added
paper/appendix/appendix.tex
+92 −0
@@ -0,0 +1,92 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Appendix | |
| 3 | +% ============================================================================ | |
| 4 | +\newpage | |
| 5 | +\begin{appendices} | |
| 6 | + | |
| 7 | +\renewcommand{\thetable}{A\arabic{table}} | |
| 8 | +\renewcommand{\thefigure}{A\arabic{figure}} | |
| 9 | +\setcounter{table}{0} | |
| 10 | +\setcounter{figure}{0} | |
| 11 | + | |
| 12 | +\section{Machine-Generated Feature Definitions} | |
| 13 | +\label{app:features} | |
| 14 | + | |
| 15 | +Table~\ref{tab:app_features} lists the attributes extracted by the language and | |
| 16 | +vision models under their respective strict JSON schemas. All fields use closed | |
| 17 | +enumerations or booleans; absent information is coded as \texttt{false}, | |
| 18 | +\texttt{unknown}, or an empty list. | |
| 19 | + | |
| 20 | +\begin{table}[htbp] | |
| 21 | + \centering | |
| 22 | + \begin{threeparttable} | |
| 23 | + \caption{Attributes Extracted by the AI Models} | |
| 24 | + \label{tab:app_features} | |
| 25 | + \small | |
| 26 | + \begin{tabular}{@{}p{2.9cm}p{4.4cm}p{6.3cm}@{}} | |
| 27 | + \toprule | |
| 28 | + Layer & Field & Domain \\ | |
| 29 | + \midrule | |
| 30 | + \multicolumn{3}{@{}l}{\textit{Panel A: Text (LLM, \texttt{gpt-4o-mini})}} \\[3pt] | |
| 31 | + Condition & \texttt{condition} & to renovate / original / good / renovated / like-new / new \\ | |
| 32 | + Turnkey & \texttt{turnkey} & boolean \\ | |
| 33 | + Renovated kitchen & \texttt{renovated\_kitchen} & boolean \\ | |
| 34 | + Renovated bathroom & \texttt{renovated\_bathroom} & boolean \\ | |
| 35 | + Flooring & \texttt{flooring} & hardwood / ceramic / laminate / carpet / vinyl / concrete \\ | |
| 36 | + Countertop & \texttt{countertop} & granite / quartz / marble / laminate / wood \\ | |
| 37 | + Heating & \texttt{heating} & electric / heat pump / wood / gas / oil / radiant / geothermal \\ | |
| 38 | + Water view & \texttt{view} & water / mountain / city / none \\ | |
| 39 | + Income property & \texttt{income\_property} & boolean \\ | |
| 40 | + Large lot; quiet area & \texttt{large\_lot}, \texttt{quiet\_area} & boolean \\[6pt] | |
| 41 | + \multicolumn{3}{@{}l}{\textit{Panel B: Vision (\texttt{claude-haiku-4-5}, façade)}} \\[3pt] | |
| 42 | + Building type (visual) & \texttt{building\_type\_visual} & detached / semi / row / condo / cottage / land \\ | |
| 43 | + Condition score & \texttt{condition\_score} & integer 1--10 \\ | |
| 44 | + Curb appeal & \texttt{curb\_appeal} & integer 1--10 \\ | |
| 45 | + Renovation level & \texttt{renovation\_level} & to renovate / dated / standard / renovated / high-end \\ | |
| 46 | + Exterior materials & \texttt{exterior\_materials} & brick / stone / vinyl / wood / aluminum / stucco / concrete \\ | |
| 47 | + Landscaping & \texttt{landscaping} & none / minimal / maintained / manicured \\ | |
| 48 | + Setting & \texttt{setting} & dense urban / suburban / rural / waterfront / wooded \\ | |
| 49 | + Luxury indicator & \texttt{curb\_luxury} & boolean \\ | |
| 50 | + Photo quality & \texttt{photo\_quality} & integer 1--10 \\ | |
| 51 | + \bottomrule | |
| 52 | + \end{tabular} | |
| 53 | + \begin{tablenotes}[flushleft] | |
| 54 | + \footnotesize | |
| 55 | + \item \textit{Notes:} Both extractions use schema-constrained decoding with | |
| 56 | + closed enumerations, required fields, and \texttt{additionalProperties:false}; | |
| 57 | + decoding temperature is zero. The text layer reads the free-text description; | |
| 58 | + the vision layer analyzes one façade photograph per listing. Fields are mapped | |
| 59 | + to numeric regressors (booleans to $0/1$, ordinal scales to integers, categories | |
| 60 | + to indicators), each with an availability flag. | |
| 61 | + \end{tablenotes} | |
| 62 | + \end{threeparttable} | |
| 63 | +\end{table} | |
| 64 | + | |
| 65 | +\section{Sample Construction Flow} | |
| 66 | +\label{app:flow} | |
| 67 | + | |
| 68 | +\begin{table}[htbp] | |
| 69 | + \centering | |
| 70 | + \begin{threeparttable} | |
| 71 | + \caption{From Raw Records to Estimation Sample} | |
| 72 | + \label{tab:app_flow} | |
| 73 | + \begin{tabular}{@{}lr@{}} | |
| 74 | + \toprule | |
| 75 | + Step & Records \\ | |
| 76 | + \midrule | |
| 77 | + Realtor.ca raw & 27{,}302 \\ | |
| 78 | + DuProprio raw & 7{,}693 \\ | |
| 79 | + Houses (single-family + townhouse) with price & 19{,}273 \\ | |
| 80 | + After outlier cleaning & 19{,}123 \\ | |
| 81 | + \quad of which with measured living area (Model~A) & 9{,}446 \\ | |
| 82 | + \bottomrule | |
| 83 | + \end{tabular} | |
| 84 | + \begin{tablenotes}[flushleft] | |
| 85 | + \footnotesize | |
| 86 | + \item \textit{Notes:} See Section~\ref{sec:data}. ``Measured living area'' means | |
| 87 | + reported area or area reconstructed from room-level dimensions. | |
| 88 | + \end{tablenotes} | |
| 89 | + \end{threeparttable} | |
| 90 | +\end{table} | |
| 91 | + | |
| 92 | +\end{appendices} | |
added
paper/figures/fig_ablation.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_calibration.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_importance.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_premia.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_validation.pdf
+0 −0
Binary file not shown.
added
paper/figures/nn_diagram.tex
+80 −0
@@ -0,0 +1,80 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% TikZ: Robust tabular neural network architecture | |
| 3 | +% ============================================================================ | |
| 4 | +\begin{tikzpicture}[ | |
| 5 | + font=\small, | |
| 6 | + >=Latex, | |
| 7 | + node distance=6mm and 6mm, | |
| 8 | + box/.style={rounded corners=2pt, draw=black!55, align=center, | |
| 9 | + minimum height=8mm, inner sep=3pt, text width=24mm}, | |
| 10 | + cat/.style={box, fill=NavyBlue!8}, | |
| 11 | + num/.style={box, fill=OliveGreen!10, text width=30mm}, | |
| 12 | + emb/.style={box, fill=BurntOrange!14, text width=22mm}, | |
| 13 | + op/.style={box, fill=black!5}, | |
| 14 | + spine/.style={box, fill=black!6, text width=52mm}, | |
| 15 | + res/.style={box, fill=Plum!10, text width=52mm, minimum height=11mm}, | |
| 16 | + outb/.style={box, fill=NavyBlue!16, text width=34mm, font=\bfseries}, | |
| 17 | + detail/.style={rounded corners=2pt, draw=black!40, dashed, fill=Plum!5, | |
| 18 | + align=left, text width=40mm, inner sep=4pt, font=\footnotesize}, | |
| 19 | + ar/.style={->, thick, black!65} | |
| 20 | +] | |
| 21 | + | |
| 22 | +% ---- inputs ---- | |
| 23 | +\node[cat] (city) {City index\\{\footnotesize (195 cats)}}; | |
| 24 | +\node[cat, right=of city] (type) {Type index}; | |
| 25 | +\node[cat, right=of type] (src) {Source index}; | |
| 26 | +\node[num, right=10mm of src] (numin) {Numeric features\\{\footnotesize $\approx$45: area, lot, beds,\\baths, age, amenities,\\rooms, GPT specs}}; | |
| 27 | + | |
| 28 | +% ---- embeddings / bn ---- | |
| 29 | +\node[emb, below=of city] (ecity) {Embedding\\{\footnotesize dim 29}}; | |
| 30 | +\node[emb, below=of type] (etype) {Embedding\\{\footnotesize dim 4}}; | |
| 31 | +\node[emb, below=of src] (esrc) {Embedding\\{\footnotesize dim 2}}; | |
| 32 | +\node[op, below=of numin, text width=30mm] (bn) {BatchNorm\\{\footnotesize + missingness flags}}; | |
| 33 | + | |
| 34 | +\draw[ar] (city)--(ecity); \draw[ar] (type)--(etype); | |
| 35 | +\draw[ar] (src)--(esrc); \draw[ar] (numin)--(bn); | |
| 36 | + | |
| 37 | +% ---- concat ---- | |
| 38 | +\node[spine, below=12mm of etype.south, xshift=8mm] (concat) | |
| 39 | + {Concatenate \;$\rightarrow$\; $\mathbb{R}^{\,\text{emb}+\text{num}}$}; | |
| 40 | +\draw[ar] (ecity) |- (concat.west); | |
| 41 | +\draw[ar] (etype) -- (concat); | |
| 42 | +\draw[ar] (esrc) -- (concat); | |
| 43 | +\draw[ar] (bn) -| (concat.east); | |
| 44 | + | |
| 45 | +% ---- projection ---- | |
| 46 | +\node[spine, below=of concat] (proj) {Linear $\rightarrow$ 256\;\; (trunk)}; | |
| 47 | +\draw[ar] (concat)--(proj); | |
| 48 | + | |
| 49 | +% ---- residual blocks ---- | |
| 50 | +\node[res, below=of proj] (resb) | |
| 51 | + {Residual block \;$\times\,3$\\{\footnotesize BN $\to$ GELU $\to$ Lin$_{512}$ $\to$ Dropout $\to$ Lin$_{256}$ \;$\oplus$\; skip}}; | |
| 52 | +\draw[ar] (proj)--(resb); | |
| 53 | + | |
| 54 | +% ---- head ---- | |
| 55 | +\node[spine, below=of resb] (head) | |
| 56 | + {Head:\; BatchNorm $\to$ GELU $\to$ Linear$_{1}$}; | |
| 57 | +\draw[ar] (resb)--(head); | |
| 58 | + | |
| 59 | +% ---- output ---- | |
| 60 | +\node[outb, below=of head] (out) {$\widehat{\ln(\text{Price})}$}; | |
| 61 | +\draw[ar] (head)--(out); | |
| 62 | + | |
| 63 | +% ---- residual detail callout ---- | |
| 64 | +\node[detail, right=10mm of resb] (dtl) | |
| 65 | + {\textbf{Res block:} $x \mapsto x + f(x)$\\ | |
| 66 | + $f=$ Lin$_{256}\!\circ$Drop$\circ$Lin$_{512}\!\circ$GELU$\circ$BN}; | |
| 67 | +\draw[->, dashed, black!45] (resb.east)--(dtl.west); | |
| 68 | + | |
| 69 | +% ---- training / ensemble annotation ---- | |
| 70 | +\node[detail, right=10mm of head, text width=42mm, fill=OliveGreen!6, draw=black!40] (train) | |
| 71 | + {\textbf{Training (robust):}\\ | |
| 72 | + Huber loss $\cdot$ AdamW $\cdot$ cosine LR\\ | |
| 73 | + grad-clip 5 $\cdot$ dropout $\cdot$ early stop}; | |
| 74 | + | |
| 75 | +% ---- ensemble + blend ---- | |
| 76 | +\node[op, below=8mm of out, text width=64mm, fill=NavyBlue!7] (ens) | |
| 77 | + {Ensemble of 3 seeds (average) \;$\Rightarrow$\; blend 50/50 with gradient boosting}; | |
| 78 | +\draw[ar] (out)--(ens); | |
| 79 | + | |
| 80 | +\end{tikzpicture} | |
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+150 −0
@@ -0,0 +1,150 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% UQO Working Paper — Hedonic House Pricing with LLM & Vision Features | |
| 3 | +% ============================================================================ | |
| 4 | +\documentclass[12pt,letterpaper]{article} | |
| 5 | + | |
| 6 | +% --- Encoding & Language --- | |
| 7 | +\usepackage[utf8]{inputenc} | |
| 8 | +\usepackage[T1]{fontenc} | |
| 9 | +\usepackage[english]{babel} | |
| 10 | + | |
| 11 | +% --- Page Layout --- | |
| 12 | +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry} | |
| 13 | + | |
| 14 | +% --- Typography --- | |
| 15 | +\usepackage{mathptmx} | |
| 16 | +\usepackage{microtype} | |
| 17 | +\usepackage{setspace} | |
| 18 | +\onehalfspacing | |
| 19 | + | |
| 20 | +% --- Math --- | |
| 21 | +\usepackage{amsmath,amssymb,amsthm} | |
| 22 | +\usepackage{bm} | |
| 23 | + | |
| 24 | +% --- Tables --- | |
| 25 | +\usepackage{booktabs} | |
| 26 | +\usepackage{tabularx} | |
| 27 | +\usepackage{multirow} | |
| 28 | +\usepackage{threeparttable} | |
| 29 | + | |
| 30 | +% --- Figures --- | |
| 31 | +\usepackage{graphicx} | |
| 32 | +\usepackage{subcaption} | |
| 33 | +\usepackage{float} | |
| 34 | +\usepackage{tikz} | |
| 35 | +\usetikzlibrary{arrows.meta,positioning,fit,backgrounds,calc,shapes.geometric} | |
| 36 | + | |
| 37 | +% --- Colors & Links --- | |
| 38 | +\usepackage[dvipsnames]{xcolor} | |
| 39 | +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true,pdftitle={Hedonic House Pricing with Language-Model and Vision Features: Evidence from Quebec Real-Estate Listings},pdfauthor={Simon-Pierre Boucher}]{hyperref} | |
| 40 | + | |
| 41 | +% --- Bibliography --- | |
| 42 | +\usepackage[authoryear,round,semicolon]{natbib} | |
| 43 | +\bibliographystyle{aer} | |
| 44 | + | |
| 45 | +% --- Captions --- | |
| 46 | +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption} | |
| 47 | + | |
| 48 | +% --- Headers & Footers --- | |
| 49 | +\usepackage{fancyhdr} | |
| 50 | +\pagestyle{fancy} | |
| 51 | +\fancyhf{} | |
| 52 | +\renewcommand{\headrulewidth}{0pt} | |
| 53 | +\fancyfoot[C]{\thepage} | |
| 54 | + | |
| 55 | +% --- Section Formatting --- | |
| 56 | +\usepackage{titlesec} | |
| 57 | +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{} | |
| 58 | +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{} | |
| 59 | +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{} | |
| 60 | + | |
| 61 | +% --- Appendix Support --- | |
| 62 | +\usepackage[toc,page]{appendix} | |
| 63 | + | |
| 64 | +% --- Misc --- | |
| 65 | +\usepackage{enumitem} | |
| 66 | +\setlist{nosep,leftmargin=*} | |
| 67 | + | |
| 68 | +% ============================================================================ | |
| 69 | +% METADATA | |
| 70 | +% ============================================================================ | |
| 71 | +\newcommand{\WPnumber}{14} | |
| 72 | +\newcommand{\WPtitle}{Hedonic House Pricing with Language-Model and Vision Features} | |
| 73 | +\newcommand{\WPsubtitle}{Evidence from Quebec Real-Estate Listings} | |
| 74 | +\newcommand{\WPdate}{July 2026} | |
| 75 | +\newcommand{\WPversion}{2.0} | |
| 76 | +\newcommand{\WPabstract}{% | |
| 77 | + Hedonic house-price models are bounded by the characteristics an analyst can | |
| 78 | + measure, yet much of a home's value is recorded only in the free-text description | |
| 79 | + and photographs of its listing. We treat general-purpose AI models as a | |
| 80 | + feature-engineering technology for hedonic analysis: a large language model | |
| 81 | + converts each description into structured property specifications, and a | |
| 82 | + vision--language model converts the façade photograph into objective visual | |
| 83 | + attributes, both under strict JSON schemas that render the outputs reproducible | |
| 84 | + econometric variables. Using 19{,}123 Quebec house listings, the two AI layers | |
| 85 | + raise the $R^2$ of an already rich log--linear hedonic specification from 0.717 to | |
| 86 | + 0.755, and the vision layer delivers the largest out-of-sample gain of any feature | |
| 87 | + block for every estimator, from OLS to gradient boosting and neural networks (best | |
| 88 | + ensemble: out-of-sample $R^2$ of 0.779, median absolute error 14.0\%). The | |
| 89 | + extracted attributes carry economically sensible implicit prices---2.3\% per point | |
| 90 | + of curb appeal, 3--4\% for brick or stone façades---that survive | |
| 91 | + municipality-clustered inference. | |
| 92 | +} | |
| 93 | +\newcommand{\WPkeywords}{hedonic pricing, house prices, large language models, computer vision, feature engineering, Quebec} | |
| 94 | +\newcommand{\WPjel}{R31, C21, C55, R21} | |
| 95 | + | |
| 96 | +% --- Author --- | |
| 97 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | |
| 98 | +\newcommand{\WPaffiliation}{% | |
| 99 | + Département des sciences administratives\\ | |
| 100 | + Université du Québec en Outaouais} | |
| 101 | +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca} | |
| 102 | +\newcommand{\WPaddress}{% | |
| 103 | + Gatineau -- Pavillon Alexandre-Taché\\ | |
| 104 | + 283, boulevard Alexandre-Taché\\ | |
| 105 | + Gatineau, Québec, Canada J9A 1L8} | |
| 106 | + | |
| 107 | +% ============================================================================ | |
| 108 | +\begin{document} | |
| 109 | + | |
| 110 | +\input{sections/titlepage} | |
| 111 | + | |
| 112 | +\input{sections/introduction} | |
| 113 | +\input{sections/literature} | |
| 114 | +\input{sections/data} | |
| 115 | +\input{sections/methodology} | |
| 116 | +\input{sections/results} | |
| 117 | +\input{sections/benchmark} | |
| 118 | +\input{sections/robustness} | |
| 119 | +\input{sections/conclusion} | |
| 120 | + | |
| 121 | +% ============================================================================ | |
| 122 | +% Declarations (Elsevier requirements) | |
| 123 | +% ============================================================================ | |
| 124 | +\section*{Declarations} | |
| 125 | + | |
| 126 | +\paragraph{Funding.} This research did not receive any specific grant from funding | |
| 127 | +agencies in the public, commercial, or not-for-profit sectors. | |
| 128 | + | |
| 129 | +\paragraph{Declaration of competing interest.} The author declares that he has no | |
| 130 | +known competing financial interests or personal relationships that could have | |
| 131 | +appeared to influence the work reported in this paper. | |
| 132 | + | |
| 133 | +\paragraph{Data availability.} The listing data were collected from publicly | |
| 134 | +accessible web platforms (Realtor.ca and DuProprio) on 3~July~2026. The full | |
| 135 | +data-construction code, the extraction schemas, and the derived analysis dataset | |
| 136 | +(excluding copyrighted listing text and images) are available from the author upon | |
| 137 | +reasonable request. | |
| 138 | + | |
| 139 | +\paragraph{Use of AI models.} Large language and vision--language models | |
| 140 | +(\texttt{gpt-4o-mini} and \texttt{claude-haiku-4-5}) are the object of study of this | |
| 141 | +paper and were used exclusively as documented in Section~\ref{sec:methodology} to | |
| 142 | +extract structured features from listing text and photographs. They were not used to | |
| 143 | +generate the analysis or the conclusions of the paper. | |
| 144 | + | |
| 145 | +\newpage | |
| 146 | +\bibliography{references} | |
| 147 | + | |
| 148 | +\input{appendix/appendix} | |
| 149 | + | |
| 150 | +\end{document} | |
added
paper/references.bib
+316 −0
@@ -0,0 +1,316 @@ | ||
| 1 | +@article{rosen1974hedonic, | |
| 2 | + title={Hedonic prices and implicit markets: product differentiation in pure competition}, | |
| 3 | + author={Rosen, Sherwin}, | |
| 4 | + journal={Journal of Political Economy}, | |
| 5 | + volume={82}, | |
| 6 | + number={1}, | |
| 7 | + pages={34--55}, | |
| 8 | + year={1974} | |
| 9 | +} | |
| 10 | + | |
| 11 | +@article{lancaster1966new, | |
| 12 | + title={A new approach to consumer theory}, | |
| 13 | + author={Lancaster, Kelvin J.}, | |
| 14 | + journal={Journal of Political Economy}, | |
| 15 | + volume={74}, | |
| 16 | + number={2}, | |
| 17 | + pages={132--157}, | |
| 18 | + year={1966} | |
| 19 | +} | |
| 20 | + | |
| 21 | +@article{sirmans2005composition, | |
| 22 | + title={The composition of hedonic pricing models}, | |
| 23 | + author={Sirmans, Stacy G. and Macpherson, David A. and Zietz, Emily N.}, | |
| 24 | + journal={Journal of Real Estate Literature}, | |
| 25 | + volume={13}, | |
| 26 | + number={1}, | |
| 27 | + pages={1--44}, | |
| 28 | + year={2005} | |
| 29 | +} | |
| 30 | + | |
| 31 | +@incollection{malpezzi2003hedonic, | |
| 32 | + title={Hedonic pricing models: a selective and applied review}, | |
| 33 | + author={Malpezzi, Stephen}, | |
| 34 | + booktitle={Housing Economics and Public Policy}, | |
| 35 | + editor={O'Sullivan, Tony and Gibb, Kenneth}, | |
| 36 | + pages={67--89}, | |
| 37 | + publisher={Blackwell Science}, | |
| 38 | + year={2003} | |
| 39 | +} | |
| 40 | + | |
| 41 | +@article{duan1983smearing, | |
| 42 | + title={Smearing estimate: a nonparametric retransformation method}, | |
| 43 | + author={Duan, Naihua}, | |
| 44 | + journal={Journal of the American Statistical Association}, | |
| 45 | + volume={78}, | |
| 46 | + number={383}, | |
| 47 | + pages={605--610}, | |
| 48 | + year={1983} | |
| 49 | +} | |
| 50 | + | |
| 51 | +@article{white1980heteroskedasticity, | |
| 52 | + title={A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity}, | |
| 53 | + author={White, Halbert}, | |
| 54 | + journal={Econometrica}, | |
| 55 | + volume={48}, | |
| 56 | + number={4}, | |
| 57 | + pages={817--838}, | |
| 58 | + year={1980} | |
| 59 | +} | |
| 60 | + | |
| 61 | +@article{mackinnon1985some, | |
| 62 | + title={Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties}, | |
| 63 | + author={MacKinnon, James G. and White, Halbert}, | |
| 64 | + journal={Journal of Econometrics}, | |
| 65 | + volume={29}, | |
| 66 | + number={3}, | |
| 67 | + pages={305--325}, | |
| 68 | + year={1985} | |
| 69 | +} | |
| 70 | + | |
| 71 | +@article{can1992spatial, | |
| 72 | + title={Specification and estimation of hedonic housing price models}, | |
| 73 | + author={Can, Ayse}, | |
| 74 | + journal={Regional Science and Urban Economics}, | |
| 75 | + volume={22}, | |
| 76 | + number={3}, | |
| 77 | + pages={453--474}, | |
| 78 | + year={1992} | |
| 79 | +} | |
| 80 | + | |
| 81 | +@book{anselin1988spatial, | |
| 82 | + title={Spatial Econometrics: Methods and Models}, | |
| 83 | + author={Anselin, Luc}, | |
| 84 | + publisher={Kluwer Academic Publishers}, | |
| 85 | + address={Dordrecht}, | |
| 86 | + year={1988} | |
| 87 | +} | |
| 88 | + | |
| 89 | +@article{gentzkow2019text, | |
| 90 | + title={Text as data}, | |
| 91 | + author={Gentzkow, Matthew and Kelly, Bryan and Taddy, Matt}, | |
| 92 | + journal={Journal of Economic Literature}, | |
| 93 | + volume={57}, | |
| 94 | + number={3}, | |
| 95 | + pages={535--574}, | |
| 96 | + year={2019} | |
| 97 | +} | |
| 98 | + | |
| 99 | +@article{poursaeed2018vision, | |
| 100 | + title={Vision-based real estate price estimation}, | |
| 101 | + author={Poursaeed, Omid and Matera, Tomas and Belongie, Serge}, | |
| 102 | + journal={Machine Vision and Applications}, | |
| 103 | + volume={29}, | |
| 104 | + number={4}, | |
| 105 | + pages={667--676}, | |
| 106 | + year={2018} | |
| 107 | +} | |
| 108 | + | |
| 109 | +@inproceedings{ahmed2016house, | |
| 110 | + title={House price estimation from visual and textual features}, | |
| 111 | + author={Ahmed, Eman H. and Moustafa, Mohamed}, | |
| 112 | + booktitle={Proceedings of the 8th International Joint Conference on Computational Intelligence (NCTA)}, | |
| 113 | + pages={62--68}, | |
| 114 | + year={2016} | |
| 115 | +} | |
| 116 | + | |
| 117 | +@article{bin2004prediction, | |
| 118 | + title={A prediction comparison of housing sales prices by parametric versus semi-parametric regressions}, | |
| 119 | + author={Bin, Okmyung}, | |
| 120 | + journal={Journal of Housing Economics}, | |
| 121 | + volume={13}, | |
| 122 | + number={1}, | |
| 123 | + pages={68--84}, | |
| 124 | + year={2004} | |
| 125 | +} | |
| 126 | + | |
| 127 | +@article{mullainathan2017machine, | |
| 128 | + title={Machine learning: an applied econometric approach}, | |
| 129 | + author={Mullainathan, Sendhil and Spiess, Jann}, | |
| 130 | + journal={Journal of Economic Perspectives}, | |
| 131 | + volume={31}, | |
| 132 | + number={2}, | |
| 133 | + pages={87--106}, | |
| 134 | + year={2017} | |
| 135 | +} | |
| 136 | + | |
| 137 | +@article{halvorsen1980interpretation, | |
| 138 | + title={The interpretation of dummy variables in semilogarithmic equations}, | |
| 139 | + author={Halvorsen, Robert and Palmquist, Raymond}, | |
| 140 | + journal={American Economic Review}, | |
| 141 | + volume={70}, | |
| 142 | + number={3}, | |
| 143 | + pages={474--475}, | |
| 144 | + year={1980} | |
| 145 | +} | |
| 146 | + | |
| 147 | +@article{breiman2001random, | |
| 148 | + title={Random forests}, | |
| 149 | + author={Breiman, Leo}, | |
| 150 | + journal={Machine Learning}, | |
| 151 | + volume={45}, | |
| 152 | + number={1}, | |
| 153 | + pages={5--32}, | |
| 154 | + year={2001} | |
| 155 | +} | |
| 156 | + | |
| 157 | +@article{friedman2001greedy, | |
| 158 | + title={Greedy function approximation: a gradient boosting machine}, | |
| 159 | + author={Friedman, Jerome H.}, | |
| 160 | + journal={Annals of Statistics}, | |
| 161 | + volume={29}, | |
| 162 | + number={5}, | |
| 163 | + pages={1189--1232}, | |
| 164 | + year={2001} | |
| 165 | +} | |
| 166 | + | |
| 167 | +@article{guo2016entity, | |
| 168 | + title={Entity embeddings of categorical variables}, | |
| 169 | + author={Guo, Cheng and Berkhahn, Felix}, | |
| 170 | + journal={arXiv preprint arXiv:1604.06737}, | |
| 171 | + year={2016} | |
| 172 | +} | |
| 173 | + | |
| 174 | +@inproceedings{loshchilov2019decoupled, | |
| 175 | + title={Decoupled weight decay regularization}, | |
| 176 | + author={Loshchilov, Ilya and Hutter, Frank}, | |
| 177 | + booktitle={International Conference on Learning Representations (ICLR)}, | |
| 178 | + year={2019} | |
| 179 | +} | |
| 180 | + | |
| 181 | +@inproceedings{gorishniy2021revisiting, | |
| 182 | + title={Revisiting deep learning models for tabular data}, | |
| 183 | + author={Gorishniy, Yury and Rubachev, Ivan and Khrulkov, Valentin and Babenko, Artem}, | |
| 184 | + booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 185 | + year={2021} | |
| 186 | +} | |
| 187 | + | |
| 188 | +@inproceedings{grinsztajn2022why, | |
| 189 | + title={Why do tree-based models still outperform deep learning on tabular data?}, | |
| 190 | + author={Grinsztajn, L{\'e}o and Oyallon, Edouard and Varoquaux, Ga{\"e}l}, | |
| 191 | + booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 192 | + year={2022} | |
| 193 | +} | |
| 194 | + | |
| 195 | +@inproceedings{he2016deep, | |
| 196 | + title={Deep residual learning for image recognition}, | |
| 197 | + author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian}, | |
| 198 | + booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, | |
| 199 | + pages={770--778}, | |
| 200 | + year={2016} | |
| 201 | +} | |
| 202 | + | |
| 203 | +@article{nowak2017textual, | |
| 204 | + title={Textual analysis in real estate}, | |
| 205 | + author={Nowak, Adam and Smith, Patrick S.}, | |
| 206 | + journal={Journal of Applied Econometrics}, | |
| 207 | + volume={32}, | |
| 208 | + number={4}, | |
| 209 | + pages={896--918}, | |
| 210 | + year={2017} | |
| 211 | +} | |
| 212 | + | |
| 213 | +@article{shen2021information, | |
| 214 | + title={Information value of property description: A machine learning approach}, | |
| 215 | + author={Shen, Lily and Ross, Stephen L.}, | |
| 216 | + journal={Journal of Urban Economics}, | |
| 217 | + volume={121}, | |
| 218 | + pages={103299}, | |
| 219 | + year={2021} | |
| 220 | +} | |
| 221 | + | |
| 222 | +@article{johnson2020curb, | |
| 223 | + title={Valuing curb appeal}, | |
| 224 | + author={Johnson, Erik B. and Tidwell, Alan and Villupuram, Sriram V.}, | |
| 225 | + journal={Journal of Real Estate Finance and Economics}, | |
| 226 | + volume={60}, | |
| 227 | + number={1-2}, | |
| 228 | + pages={111--133}, | |
| 229 | + year={2020} | |
| 230 | +} | |
| 231 | + | |
| 232 | +@article{lindenthal2021architecture, | |
| 233 | + title={Machine learning, architectural styles and property values}, | |
| 234 | + author={Lindenthal, Thies and Johnson, Erik B.}, | |
| 235 | + journal={Journal of Real Estate Finance and Economics}, | |
| 236 | + year={2021} | |
| 237 | +} | |
| 238 | + | |
| 239 | +@techreport{glaeser2018vision, | |
| 240 | + title={Computer vision and real estate: Do looks matter and do incentives determine looks}, | |
| 241 | + author={Glaeser, Edward L. and Kincaid, Michael Scott and Naik, Nikhil}, | |
| 242 | + institution={National Bureau of Economic Research}, | |
| 243 | + type={Working Paper}, | |
| 244 | + number={25174}, | |
| 245 | + year={2018} | |
| 246 | +} | |
| 247 | + | |
| 248 | +@article{kok2017bigdata, | |
| 249 | + title={Big data in real estate? {F}rom manual appraisal to automated valuation}, | |
| 250 | + author={Kok, Nils and Koponen, Eija-Leena and Mart{\'i}nez-Barbosa, Carmen Adriana}, | |
| 251 | + journal={Journal of Portfolio Management}, | |
| 252 | + volume={43}, | |
| 253 | + number={6}, | |
| 254 | + pages={202--211}, | |
| 255 | + year={2017} | |
| 256 | +} | |
| 257 | + | |
| 258 | +@article{hendel2009mls, | |
| 259 | + title={The relative performance of real estate marketing platforms: {MLS} versus {FSBOMadison.com}}, | |
| 260 | + author={Hendel, Igal and Nevo, Aviv and Ortalo-Magn{\'e}, Fran{\c{c}}ois}, | |
| 261 | + journal={American Economic Review}, | |
| 262 | + volume={99}, | |
| 263 | + number={5}, | |
| 264 | + pages={1878--1898}, | |
| 265 | + year={2009} | |
| 266 | +} | |
| 267 | + | |
| 268 | +@article{horowitz1992list, | |
| 269 | + title={The role of the list price in housing markets: Theory and an econometric model}, | |
| 270 | + author={Horowitz, Joel L.}, | |
| 271 | + journal={Journal of Applied Econometrics}, | |
| 272 | + volume={7}, | |
| 273 | + number={2}, | |
| 274 | + pages={115--129}, | |
| 275 | + year={1992} | |
| 276 | +} | |
| 277 | + | |
| 278 | +@article{knight2002listing, | |
| 279 | + title={Listing price, time on market, and ultimate selling price: Causes and effects of listing price changes}, | |
| 280 | + author={Knight, John R.}, | |
| 281 | + journal={Real Estate Economics}, | |
| 282 | + volume={30}, | |
| 283 | + number={2}, | |
| 284 | + pages={213--237}, | |
| 285 | + year={2002} | |
| 286 | +} | |
| 287 | + | |
| 288 | +@incollection{han2015microstructure, | |
| 289 | + title={The microstructure of housing markets: Search, bargaining, and brokerage}, | |
| 290 | + author={Han, Lu and Strange, William C.}, | |
| 291 | + booktitle={Handbook of Regional and Urban Economics}, | |
| 292 | + volume={5}, | |
| 293 | + pages={813--886}, | |
| 294 | + publisher={Elsevier}, | |
| 295 | + year={2015} | |
| 296 | +} | |
| 297 | + | |
| 298 | +@article{hill2013hedonic, | |
| 299 | + title={Hedonic price indexes for residential housing: A survey, evaluation and taxonomy}, | |
| 300 | + author={Hill, Robert J.}, | |
| 301 | + journal={Journal of Economic Surveys}, | |
| 302 | + volume={27}, | |
| 303 | + number={5}, | |
| 304 | + pages={879--914}, | |
| 305 | + year={2013} | |
| 306 | +} | |
| 307 | + | |
| 308 | +@article{cameron2015cluster, | |
| 309 | + title={A practitioner's guide to cluster-robust inference}, | |
| 310 | + author={Cameron, A. Colin and Miller, Douglas L.}, | |
| 311 | + journal={Journal of Human Resources}, | |
| 312 | + volume={50}, | |
| 313 | + number={2}, | |
| 314 | + pages={317--372}, | |
| 315 | + year={2015} | |
| 316 | +} | |
added
paper/sections/benchmark.tex
+260 −0
@@ -0,0 +1,260 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Predictive Benchmark: From OLS to Deep Learning} | |
| 3 | +\label{sec:benchmark} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +The hedonic OLS of Section~\ref{sec:results} is designed for interpretation: its | |
| 7 | +coefficients are implicit prices. It is not, however, the most accurate predictor of | |
| 8 | +price. To map the accuracy frontier---and to gauge how much curvature and interaction | |
| 9 | +the linear model leaves on the table---we benchmark a ladder of estimators of | |
| 10 | +increasing flexibility on the \emph{identical} feature set, under a common | |
| 11 | +5-fold cross-validation protocol. All rows of Table~\ref{tab:models} share the same | |
| 12 | +folds and random seed, so differences reflect the estimator, not the split. | |
| 13 | + | |
| 14 | +\subsection{The Estimator Ladder} | |
| 15 | +\label{subsec:ladder} | |
| 16 | + | |
| 17 | +\paragraph{(1) Ordinary least squares.} The linear hedonic of | |
| 18 | +Equation~\eqref{eq:hedonic}, with one-hot city, property-type, and source indicators | |
| 19 | +and within-property-type median imputation of missing continuous attributes. It is | |
| 20 | +the interpretable baseline and, mechanically, the least flexible: it imposes constant | |
| 21 | +marginal effects and additive separability. | |
| 22 | + | |
| 23 | +\paragraph{(2) Random forest.} A bagged ensemble of regression trees | |
| 24 | +\citep{mullainathan2017machine}. Trees capture non-linearities and low-order | |
| 25 | +interactions automatically and handle missing values natively, but averaging | |
| 26 | +deep decorrelated trees tends to over-smooth sharp price thresholds. | |
| 27 | + | |
| 28 | +\paragraph{(3) Gradient boosting.} A histogram-based gradient-boosted tree ensemble | |
| 29 | +(600 boosting iterations, learning rate 0.05, up to 63 leaves, $L_2$ regularization). | |
| 30 | +Boosting fits trees sequentially to residuals, which lets it model both smooth | |
| 31 | +curvature (age, area) and abrupt thresholds; like the forest it ingests missing | |
| 32 | +values directly. It is the strongest single model in our comparison. | |
| 33 | + | |
| 34 | +\paragraph{(4) Neural networks.} A plain multilayer perceptron and a robust | |
| 35 | +residual network with entity embeddings, described in | |
| 36 | +Section~\ref{subsec:nn}. Neural networks excel when high-cardinality categoricals | |
| 37 | +(here, 195 municipalities) can be represented as dense learned vectors rather than | |
| 38 | +sparse indicators. | |
| 39 | + | |
| 40 | +\paragraph{(5) Blend.} A simple 50/50 average of the robust neural network and the | |
| 41 | +gradient-boosting predictions. Because the two models make partly decorrelated | |
| 42 | +errors---the network smooths through embeddings while the boosted trees cut sharp | |
| 43 | +thresholds---their average cancels idiosyncratic noise, a standard stacking result. | |
| 44 | + | |
| 45 | +\subsection{Formal Definitions} | |
| 46 | +\label{subsec:formal} | |
| 47 | + | |
| 48 | +Let $y_i=\ln P_i$ and let $\mathbf{x}_i$ denote the feature vector of | |
| 49 | +Equation~\eqref{eq:blocks}. Each estimator approximates the conditional mean | |
| 50 | +$f(\mathbf{x})=\mathbb{E}[y\mid\mathbf{x}]$. | |
| 51 | + | |
| 52 | +\paragraph{Linear (OLS).} $f(\mathbf{x})=\mathbf{x}^{\!\top}\bm{\theta}$ with the | |
| 53 | +closed form of Section~\ref{subsec:econometrics}; categoricals enter as one-hot | |
| 54 | +indicators. | |
| 55 | + | |
| 56 | +\paragraph{Random forest \citep{breiman2001random}.} An average of $B$ regression | |
| 57 | +trees, each grown on a bootstrap resample with a random feature subset at every split: | |
| 58 | +% | |
| 59 | +\begin{equation} | |
| 60 | + \hat f_{\text{RF}}(\mathbf{x})=\frac1B\sum_{b=1}^{B} T_b(\mathbf{x};\Theta_b), | |
| 61 | + \label{eq:rf} | |
| 62 | +\end{equation} | |
| 63 | +% | |
| 64 | +where $\Theta_b$ denotes the randomness (resample and split subsets) of tree~$b$. | |
| 65 | + | |
| 66 | +\paragraph{Gradient boosting \citep{friedman2001greedy}.} A forward stagewise additive | |
| 67 | +expansion built by fitting each new tree to the negative gradient of the loss: | |
| 68 | +% | |
| 69 | +\begin{equation} | |
| 70 | + \resizebox{.94\textwidth}{!}{$\displaystyle | |
| 71 | + F_M(\mathbf{x}) = F_0 + \nu\sum_{m=1}^{M} f_m(\mathbf{x}), | |
| 72 | + \quad | |
| 73 | + f_m = \arg\min_{f}\sum_{i=1}^{n} | |
| 74 | + \big(\!-g_{mi}-f(\mathbf{x}_i)\big)^2, | |
| 75 | + \quad | |
| 76 | + g_{mi}=\frac{\partial L\big(y_i,F_{m-1}(\mathbf{x}_i)\big)}{\partial F_{m-1}(\mathbf{x}_i)} | |
| 77 | + $} | |
| 78 | + \label{eq:gbm} | |
| 79 | +\end{equation} | |
| 80 | +% | |
| 81 | +with learning rate $\nu$ and $L_2$-regularized histogram trees $f_m$; missing values | |
| 82 | +are routed to a learned default direction at each split. | |
| 83 | + | |
| 84 | +\paragraph{Neural network (robust).} High-cardinality categoricals are mapped to | |
| 85 | +learned embeddings \citep{guo2016entity}, concatenated with the batch-normalized | |
| 86 | +numeric block and projected to a $d$-dimensional trunk: | |
| 87 | +% | |
| 88 | +\begin{equation} | |
| 89 | + \mathbf{h}_0 = \mathbf{W}_0 | |
| 90 | + \Big[\, \textstyle\bigoplus_{c\in\{\text{city},\text{type},\text{src}\}} | |
| 91 | + \mathbf{E}_c(\text{cat}_{ci}) | |
| 92 | + \;\Vert\; \mathrm{BN}(\mathbf{x}^{\text{num}}_i)\,\Big]. | |
| 93 | + \label{eq:nn_in} | |
| 94 | +\end{equation} | |
| 95 | +% | |
| 96 | +The trunk passes through $L=3$ pre-activation residual blocks | |
| 97 | +\citep{he2016deep,gorishniy2021revisiting}, | |
| 98 | +% | |
| 99 | +\begin{equation} | |
| 100 | + \mathbf{h}_{\ell} | |
| 101 | + = \mathbf{h}_{\ell-1} | |
| 102 | + + \mathrm{Drop}\!\Big(\mathbf{W}_2^{(\ell)}\, | |
| 103 | + \mathrm{Drop}\!\big(\phi(\mathbf{W}_1^{(\ell)}\, | |
| 104 | + \mathrm{BN}(\mathbf{h}_{\ell-1}))\big)\Big), | |
| 105 | + \qquad \phi=\text{GELU}, | |
| 106 | + \label{eq:resblock} | |
| 107 | +\end{equation} | |
| 108 | +% | |
| 109 | +and a normalization head yields $\hat y_i = \mathbf{w}^{\!\top}\phi(\mathrm{BN}(\mathbf{h}_L))+b$. | |
| 110 | +Parameters minimize the Huber (smooth-$L_1$) loss, robust to price outliers, | |
| 111 | +% | |
| 112 | +\begin{equation} | |
| 113 | + L_\delta(r)= | |
| 114 | + \begin{cases} | |
| 115 | + \tfrac12 r^2, & |r|\le \delta,\\[2pt] | |
| 116 | + \delta\big(|r|-\tfrac12\delta\big), & |r|>\delta, | |
| 117 | + \end{cases} | |
| 118 | + \qquad r_i=\hat y_i-y_i,\;\delta=1, | |
| 119 | + \label{eq:huber} | |
| 120 | +\end{equation} | |
| 121 | +% | |
| 122 | +optimized with AdamW (decoupled weight decay) \citep{loshchilov2019decoupled} under a | |
| 123 | +cosine-annealed learning rate $\eta_t=\tfrac12\eta_0\big(1+\cos(\pi t/T)\big)$, | |
| 124 | +gradient-norm clipping, and early stopping. Three seeds are averaged per fold. | |
| 125 | + | |
| 126 | +\paragraph{Blend.} A convex combination of the two strongest models in log space, | |
| 127 | +% | |
| 128 | +\begin{equation} | |
| 129 | + \widehat{\ln P}^{\,\text{blend}} | |
| 130 | + = w\,\widehat{\ln P}^{\,\text{GBM}} | |
| 131 | + + (1-w)\,\widehat{\ln P}^{\,\text{NN}}, | |
| 132 | + \qquad w=0.5. | |
| 133 | + \label{eq:blend} | |
| 134 | +\end{equation} | |
| 135 | + | |
| 136 | +\paragraph{Evaluation.} With $\hat y_{-k(i)}$ the prediction for $i$ from the model | |
| 137 | +trained on the folds excluding $i$'s fold, we report out-of-sample fit and level | |
| 138 | +accuracy, | |
| 139 | +% | |
| 140 | +\begin{equation} | |
| 141 | + R^2_{\text{OOS}} = 1-\frac{\sum_i\big(y_i-\hat y_{-k(i)}\big)^2} | |
| 142 | + {\sum_i\big(y_i-\bar y\big)^2}, | |
| 143 | + \qquad | |
| 144 | + \text{MdAPE}=\operatorname*{median}_i\Big|e^{\,\hat y_{-k(i)}-y_i}-1\Big|. | |
| 145 | + \label{eq:cvmetrics} | |
| 146 | +\end{equation} | |
| 147 | + | |
| 148 | +\subsection{Results} | |
| 149 | +\label{subsec:bench_results} | |
| 150 | + | |
| 151 | +\input{tables/models} | |
| 152 | + | |
| 153 | +Table~\ref{tab:models} reports the ladder in two panels. In Panel~A (structural, | |
| 154 | +amenity, and text features) out-of-sample $R^2$ rises monotonically with flexibility: | |
| 155 | +from 0.628 for OLS, to 0.677 for the random forest, 0.711 for the plain network, and a | |
| 156 | +near-tie at the top between the robust network (0.741) and gradient boosting (0.743); | |
| 157 | +their blend attains $R^2_{\text{OOS}}=0.760$. Panel~B adds the twelve façade vision | |
| 158 | +attributes. Vision lifts \emph{every} estimator---OLS gains 5.8 points | |
| 159 | +(0.628 $\to$ 0.686), the random forest 2.7 (0.677 $\to$ 0.704), and gradient boosting | |
| 160 | +2.7 (0.743 $\to$ 0.770), the single largest increment from any one feature layer at | |
| 161 | +the frontier---and the \textbf{text\,+\,vision blend attains | |
| 162 | +$R^2_{\text{OOS}}=0.779$ with a median absolute error of 14.0\%}, the best model in | |
| 163 | +the study. | |
| 164 | + | |
| 165 | +Three lessons follow. First, on the full feature set roughly nine points of | |
| 166 | +out-of-sample $R^2$ separate the interpretable linear model (0.686) from the | |
| 167 | +predictive frontier (0.779)---the price of interpretability, and a measure of the | |
| 168 | +curvature and interactions the hedonic form omits. Notably, the vision layer buys the | |
| 169 | +linear model more accuracy (+5.8 points) than switching from OLS to a random forest | |
| 170 | +does (+4.9), so better measurement rivals better estimation. Second, on this tabular | |
| 171 | +problem gradient boosting and a carefully regularized neural network are | |
| 172 | +statistically comparable, consistent with the broader evidence that tree ensembles | |
| 173 | +remain strong baselines on tabular data \citep{grinsztajn2022why}; the gains come | |
| 174 | +from \emph{combining} them. Third, the façade-vision layer is not redundant with the | |
| 175 | +text and structural fields: it moves the frontier more than any other block we add, | |
| 176 | +confirming that image-derived quality is genuinely orthogonal information. | |
| 177 | + | |
| 178 | +\begin{figure}[htbp] | |
| 179 | + \centering | |
| 180 | + \includegraphics[width=.72\textwidth]{figures/fig_ablation.pdf} | |
| 181 | + \caption{Out-of-sample $R^2$ as feature layers are added (nested, left to right). | |
| 182 | + Each layer improves both the linear and boosted models; the façade-vision layer | |
| 183 | + delivers the largest single jump---about $+5.8$ points for OLS and $+2.7$ for | |
| 184 | + gradient boosting.} | |
| 185 | + \label{fig:ablation} | |
| 186 | +\end{figure} | |
| 187 | + | |
| 188 | +\subsection{Feature Ablation and Importance} | |
| 189 | +\label{subsec:ablation} | |
| 190 | + | |
| 191 | +Table~\ref{tab:ablation} and Figure~\ref{fig:ablation} decompose the accuracy gains | |
| 192 | +layer by layer. Reading down the table, room detail and listing signals add the first | |
| 193 | +large increment, the LLM text layer a smaller one, and the vision layer the single | |
| 194 | +largest jump for every estimator---raising OLS from 0.628 to 0.686 and gradient | |
| 195 | +boosting from 0.743 to 0.770 out of sample. The pattern is monotone: no layer degrades | |
| 196 | +fit, and the two AI layers together account for a substantial share of the distance | |
| 197 | +between the structural baseline and the frontier. | |
| 198 | + | |
| 199 | +\input{tables/ablation} | |
| 200 | + | |
| 201 | +Figure~\ref{fig:importance} aggregates the gradient-boosting permutation importances | |
| 202 | +into feature blocks. Structural attributes and location fixed effects dominate, as | |
| 203 | +expected, but the \textbf{vision block alone accounts for 13\% of total importance}--- | |
| 204 | +more than the rule-based amenities and the LLM text block combined---reinforcing that | |
| 205 | +the façade photograph is a first-order, not marginal, source of price information. | |
| 206 | +Figure~\ref{fig:calibration} shows the blend is well calibrated across the price range, | |
| 207 | +with an approximately symmetric, tight error distribution centered near zero. | |
| 208 | + | |
| 209 | +\begin{figure}[htbp] | |
| 210 | + \centering | |
| 211 | + \includegraphics[width=.72\textwidth]{figures/fig_importance.pdf} | |
| 212 | + \caption{Share of gradient-boosting permutation importance by feature block | |
| 213 | + (text\,+\,vision model). The vision block accounts for 13\% of total importance, | |
| 214 | + exceeding the rule-based amenity and LLM-text blocks.} | |
| 215 | + \label{fig:importance} | |
| 216 | +\end{figure} | |
| 217 | + | |
| 218 | +\begin{figure}[htbp] | |
| 219 | + \centering | |
| 220 | + \includegraphics[width=\textwidth]{figures/fig_calibration.pdf} | |
| 221 | + \caption{Out-of-sample calibration of the text\,+\,vision blend. Left: predicted | |
| 222 | + versus actual price (log--log; dashed line is the 45$^\circ$ identity). Right: | |
| 223 | + distribution of percentage prediction errors, with median absolute error of 14.0\%.} | |
| 224 | + \label{fig:calibration} | |
| 225 | +\end{figure} | |
| 226 | + | |
| 227 | +\subsection{A Robust Tabular Neural Network} | |
| 228 | +\label{subsec:nn} | |
| 229 | + | |
| 230 | +Figure~\ref{fig:nn} details the robust network. High-cardinality categoricals enter | |
| 231 | +as learned \textbf{entity embeddings}: each municipality, property type, and source is | |
| 232 | +mapped to a dense vector (dimension $\approx 1.6\,C^{0.56}$ in cardinality $C$, so | |
| 233 | +about 29 dimensions for the 195 cities). The embeddings are concatenated with the | |
| 234 | +batch-normalized numeric block and projected to a 256-dimensional trunk, which passes | |
| 235 | +through three \textbf{residual blocks} | |
| 236 | +(BatchNorm $\to$ GELU $\to$ Linear$_{512}$ $\to$ Dropout $\to$ Linear$_{256}$, with a | |
| 237 | +skip connection), followed by a normalization head that outputs $\ln(\text{Price})$. | |
| 238 | + | |
| 239 | +Robustness comes from the training recipe rather than raw capacity: | |
| 240 | +a \textbf{Huber (smooth-$L_1$) loss} that damps the influence of price outliers; | |
| 241 | +\textbf{AdamW} with decoupled weight decay and \textbf{cosine-annealed} learning rate; | |
| 242 | +\textbf{gradient-norm clipping} at 5; \textbf{dropout} and batch normalization | |
| 243 | +throughout; \textbf{learned missingness indicators} appended to the numeric block so | |
| 244 | +the network can exploit the non-random pattern of missing fields; and | |
| 245 | +\textbf{early stopping} on a held-out 10\% of each training fold. Finally, three | |
| 246 | +networks with different random seeds are trained per fold and \textbf{averaged}, which | |
| 247 | +reduces prediction variance. The result (0.741) is competitive with gradient boosting | |
| 248 | +and, once blended with it, produces the best model in the study. | |
| 249 | + | |
| 250 | +\begin{figure}[htbp] | |
| 251 | + \centering | |
| 252 | + \input{figures/nn_diagram} | |
| 253 | + \caption{Architecture of the robust tabular neural network. High-cardinality | |
| 254 | + categoricals enter as learned embeddings; numeric features are batch-normalized and | |
| 255 | + augmented with missingness indicators. The concatenated representation is projected | |
| 256 | + to a 256-dimensional trunk, refined by three residual blocks, and mapped to | |
| 257 | + $\ln(\text{Price})$. Three seeds are ensembled per fold; the final model blends this | |
| 258 | + network 50/50 with gradient boosting.} | |
| 259 | + \label{fig:nn} | |
| 260 | +\end{figure} | |
added
paper/sections/conclusion.tex
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Conclusion} | |
| 3 | +\label{sec:conclusion} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +We have estimated a hedonic price model for Quebec houses that augments the standard | |
| 7 | +set of structural, locational, and amenity attributes with two machine-generated | |
| 8 | +feature layers: structured specifications extracted from free-text descriptions by a | |
| 9 | +large language model, and objective visual attributes extracted from façade | |
| 10 | +photographs by a vision--language model. Schema-constrained decoding renders these | |
| 11 | +outputs as ordinary, reproducible econometric variables, obtained at a marginal cost | |
| 12 | +of a fraction of a cent per listing. The central message is methodological: | |
| 13 | +general-purpose AI models, disciplined by strict output schemas, function as a | |
| 14 | +low-cost feature-engineering technology that measurably expands the characteristics | |
| 15 | +set available to hedonic analysis. | |
| 16 | + | |
| 17 | +The gains are economically meaningful. The two AI layers add nearly four points of | |
| 18 | +in-sample $R^2$ to an already rich log--linear specification (0.717 to 0.755 with | |
| 19 | +living area observed), and they raise the out-of-sample accuracy of every estimator | |
| 20 | +in our ladder, from OLS to gradient boosting and an embedding-based residual neural | |
| 21 | +network; the best blend attains an out-of-sample $R^2$ of 0.779 with a median | |
| 22 | +absolute prediction error of 14.0\%. The façade-vision layer alone contributes the | |
| 23 | +largest increment of any feature block (2.7 points for gradient boosting, 5.8 for | |
| 24 | +OLS), confirming that image-derived quality is genuinely orthogonal to text and | |
| 25 | +structured fields. Crucially for hedonic practice, the extracted attributes carry | |
| 26 | +interpretable implicit prices---curb appeal, renovation level, luxury cues, and | |
| 27 | +exterior materials all price sensibly and survive municipality-clustered | |
| 28 | +inference---so the technology augments the classical regression rather than | |
| 29 | +replacing it with a black box. | |
| 30 | + | |
| 31 | +Three extensions are natural. First, the same schema-constrained approach scales | |
| 32 | +from a single façade image to a small panel of interior photographs (kitchen, | |
| 33 | +bathroom, living space), which our design already accommodates. Second, the | |
| 34 | +DuProprio coordinates permit continuous spatial features---distance to the urban | |
| 35 | +core, to water, or to amenities---that would refine the geography beyond discrete | |
| 36 | +city effects. Third, and most important for external validity, linking listings to | |
| 37 | +registry transaction prices would establish how much of the AI-measured quality | |
| 38 | +premium survives bargaining. For appraisal, automated valuation, and property-tax | |
| 39 | +assessment---applications that consume hedonic models at scale---the practical | |
| 40 | +implication is immediate: the unstructured majority of a listing's information | |
| 41 | +content is no longer out of reach. | |
added
paper/sections/data.tex
+74 −0
@@ -0,0 +1,74 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Data} | |
| 3 | +\label{sec:data} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{Data Sources} | |
| 7 | +\label{subsec:data_sources} | |
| 8 | + | |
| 9 | +We assemble a cross-section of residential listings from two complementary Quebec | |
| 10 | +platforms, both retrieved on 3~July~2026 via web-scraping actors. The first, | |
| 11 | +\textbf{Realtor.ca}, is the national Multiple Listing Service (MLS) portal and | |
| 12 | +provides broker-listed inventory (27{,}302 raw records). The second, | |
| 13 | +\textbf{DuProprio}, is the dominant for-sale-by-owner platform in Quebec (7{,}693 | |
| 14 | +raw records). Pooling the two sources broadens coverage across price segments and | |
| 15 | +sale channels---brokered and FSBO listings differ systematically in seller | |
| 16 | +composition and marketing \citep{hendel2009mls}---and a source indicator absorbs any | |
| 17 | +systematic level difference between them. The price we observe is the \emph{asking} | |
| 18 | +price; Section~\ref{subsec:limits} discusses the interpretation of hedonic | |
| 19 | +coefficients estimated on list rather than transaction prices | |
| 20 | +\citep{horowitz1992list,knight2002listing}. | |
| 21 | + | |
| 22 | +Each record is a structured JSON object containing a price, an address, a | |
| 23 | +property-attribute block, a free-text description, a list of features, and an array | |
| 24 | +of image URLs. The two sources differ in their auxiliary fields: Realtor.ca records | |
| 25 | +include room-by-room dimensions but no geographic coordinates, whereas DuProprio | |
| 26 | +records include coordinates and captioned images but no room block. We construct a | |
| 27 | +unique key by concatenating the source and the listing identifier. Because the data | |
| 28 | +form a single-date cross-section, no time dimension is modeled. | |
| 29 | + | |
| 30 | +\subsection{Sample Construction} | |
| 31 | +\label{subsec:sample} | |
| 32 | + | |
| 33 | +We restrict attention to \emph{houses}, defined as single-family dwellings and | |
| 34 | +townhouses. On Realtor.ca this corresponds to building types \textit{House} and | |
| 35 | +\textit{Row/Townhouse}; on DuProprio it corresponds to the property types | |
| 36 | +\textit{Home}, \textit{Bungalow}, \textit{Semi Detached}, \textit{Intergenerational}, | |
| 37 | +\textit{Country Home}, \textit{Cottage}, \textit{Villa}, and \textit{Townhouse}. | |
| 38 | +Condominiums, income properties (duplex through quintuplex), vacant land, farms, | |
| 39 | +commercial properties, and mobile homes are excluded. After requiring a non-missing, | |
| 40 | +positive price, the pooled sample contains 19{,}273 houses. | |
| 41 | + | |
| 42 | +\subsection{Outlier Cleaning} | |
| 43 | +\label{subsec:cleaning} | |
| 44 | + | |
| 45 | +Scraped prices contain parsing artifacts spanning several orders of magnitude (the | |
| 46 | +raw minimum and maximum are \$899 and \$368~billion). We apply conservative, | |
| 47 | +transparent bounds rather than model-based trimming: prices are retained in the | |
| 48 | +\$50{,}000--\$15{,}000{,}000 range; living area in 300--20{,}000~ft$^2$; lot area in | |
| 49 | +200--5{,}000{,}000~ft$^2$; bedrooms in 0--15 and bathrooms in 0--12. When a single | |
| 50 | +attribute violates its bound it is set to missing (and later imputed) rather than | |
| 51 | +deleting the observation, preserving sample size. Finally, we trim the | |
| 52 | +price-per-square-foot distribution at its 0.5th and 99.5th percentiles | |
| 53 | +(approximately \$73--\$1{,}276/ft$^2$) among listings with a measured area, removing | |
| 54 | +residual price- or area-parsing errors. The cleaned sample contains | |
| 55 | +\textbf{19{,}123 houses}, of which 80.3\% originate from Realtor.ca. | |
| 56 | + | |
| 57 | +\subsection{Summary Statistics} | |
| 58 | +\label{subsec:summary_stats} | |
| 59 | + | |
| 60 | +Table~\ref{tab:summary_stats} reports descriptive statistics for the key variables. | |
| 61 | +The median list price is \$599{,}900. The distribution of living area is right-skewed, | |
| 62 | +motivating the logarithmic transformation adopted in Section~\ref{sec:methodology}. | |
| 63 | +Living area is reported for only a minority of Realtor.ca listings; where it is | |
| 64 | +missing we reconstruct it from room-level dimensions (Section~\ref{subsec:structural}), | |
| 65 | +and where neither is available we rely on the area-free specification. | |
| 66 | + | |
| 67 | +\input{tables/summary_stats} | |
| 68 | + | |
| 69 | +Table~\ref{tab:ai_summary} previews the machine-generated attributes described in | |
| 70 | +Section~\ref{sec:methodology}: roughly a quarter of listings are described as recently | |
| 71 | +renovated, the mean visual condition and curb-appeal scores are 7.4 and 7.2 on a | |
| 72 | +10-point scale, and 43\% of façades display a visual luxury cue. | |
| 73 | + | |
| 74 | +\input{tables/ai_summary} | |
added
paper/sections/introduction.tex
+87 −0
@@ -0,0 +1,87 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Introduction} | |
| 3 | +\label{sec:introduction} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +The hedonic price framework decomposes the market value of a heterogeneous good | |
| 7 | +into the implicit prices of its characteristics \citep{lancaster1966new, | |
| 8 | +rosen1974hedonic}. In housing markets it remains the workhorse tool for appraisal, | |
| 9 | +index construction, property-tax assessment, and policy analysis. Yet the empirical | |
| 10 | +performance of hedonic models is bounded by the richness of the characteristics an | |
| 11 | +analyst can measure. A large share of what determines a home's value---its state of | |
| 12 | +repair, the quality of its finishes, its curb appeal, the character of its | |
| 13 | +street---is recorded not in structured fields but in the free-text description and | |
| 14 | +the photographs that accompany each listing. These modalities have historically been | |
| 15 | +discarded because they were costly to encode at scale, leaving hedonic estimates | |
| 16 | +exposed to omitted-quality bias and hedonic predictions short of their potential | |
| 17 | +accuracy. | |
| 18 | + | |
| 19 | +This paper treats general-purpose artificial-intelligence models as a | |
| 20 | +\emph{feature-engineering technology} for hedonic analysis, and quantifies the | |
| 21 | +explanatory power that technology adds. We assemble a cross-section of 19{,}123 | |
| 22 | +single-family houses and townhouses listed for sale in Quebec, pooling the | |
| 23 | +brokerage-listed MLS inventory from Realtor.ca with the for-sale-by-owner inventory | |
| 24 | +from DuProprio. On top of standard structural, locational, and amenity attributes, | |
| 25 | +we build two machine-generated feature layers. First, a large language model (LLM) | |
| 26 | +reads each free-text description and returns a fixed set of property | |
| 27 | +specifications---state of repair, renovation of the kitchen and bathroom, flooring | |
| 28 | +and countertop materials, heating system, view, and income potential---under a | |
| 29 | +strict JSON schema that guarantees valid, reproducible output. Second, a | |
| 30 | +vision--language model inspects the façade photograph of each property and returns | |
| 31 | +objective visual attributes: overall condition, curb appeal, renovation level, | |
| 32 | +exterior materials, landscaping, setting, and a luxury indicator. Because both | |
| 33 | +extractions are schema-constrained, deterministic, and blind to the listing price, | |
| 34 | +their outputs behave as ordinary tabulated regressors rather than as noisy text. | |
| 35 | + | |
| 36 | +Our contribution is threefold. First, we show that AI-based feature engineering | |
| 37 | +delivers economically meaningful gains in explanatory power over an already rich | |
| 38 | +hedonic specification: the two machine-generated layers raise in-sample $R^2$ from | |
| 39 | +0.717 to 0.755 (nearly four points) in our preferred specification, and raise the | |
| 40 | +out-of-sample $R^2$ of every estimator we consider, with the façade-vision layer | |
| 41 | +alone contributing the largest single increment of any feature block (2.7 points for | |
| 42 | +gradient boosting, 5.8 points for OLS). Second, we show that the extracted | |
| 43 | +attributes carry interpretable implicit prices---a one-point increase in visual curb | |
| 44 | +appeal is associated with a 2.3\% higher asking price, a visible luxury cue with | |
| 45 | +5.2\%, brick or stone façades with premia of 3--4\% relative to vinyl siding, and a | |
| 46 | +renovated kitchen (read from the text) with 2.6\%---so the approach augments rather | |
| 47 | +than replaces the classical hedonic regression. Third, we document a complete, | |
| 48 | +reproducible pipeline from raw scraped listings to an analysis-ready dataset, | |
| 49 | +including rules-based outlier cleaning, unit harmonization that reconstructs living | |
| 50 | +area from room-level dimensions, and the schema-constrained extraction protocol, | |
| 51 | +with a total AI cost of about \$60 for the full sample---roughly a third of a cent | |
| 52 | +per listing. | |
| 53 | + | |
| 54 | +Our main findings can be summarized as follows. A log--linear specification with 195 | |
| 55 | +municipality fixed effects, property-type fixed effects, and | |
| 56 | +municipality-clustered standard errors explains 75.5\% of the variation in log | |
| 57 | +asking prices when living area is observed, and 72.5\% on the full sample when it is | |
| 58 | +not. The estimated implicit prices are economically sensible: the living-area | |
| 59 | +elasticity is 0.36, each additional bathroom is associated with a 7.6\% premium, and | |
| 60 | +the age profile traces the familiar depreciation-then-heritage pattern. The | |
| 61 | +municipality effects span a wide range---from roughly $+231$\% in Westmount to about | |
| 62 | +$-30$\% in remote municipalities---and order the province's markets as expected. | |
| 63 | +Benchmarking a ladder of estimators on the identical feature set, an ensemble blend | |
| 64 | +of a robust embedding-based neural network and gradient boosting attains an | |
| 65 | +out-of-sample $R^2$ of 0.779 with a median absolute prediction error of 14.0\%. We | |
| 66 | +interpret all estimates as \emph{predictive} associations rather than causal | |
| 67 | +effects, since listing content is chosen by sellers and correlates with unobserved | |
| 68 | +determinants of value, and we discuss the asking-price nature of the dependent | |
| 69 | +variable at length in Section~\ref{sec:robustness}. | |
| 70 | + | |
| 71 | +Beyond the specific Quebec application, the exercise speaks to a general measurement | |
| 72 | +problem in housing economics. Text- and image-derived quality measures have until | |
| 73 | +now required bespoke supervised models trained on labeled data | |
| 74 | +\citep{nowak2017textual,shen2021information,johnson2020curb}. General-purpose | |
| 75 | +foundation models with schema-constrained decoding lower the fixed cost of that | |
| 76 | +measurement step to nearly zero, produce attributes that are human-readable rather | |
| 77 | +than latent, and can be audited field by field. Our results suggest that this | |
| 78 | +technology moves the effective characteristics frontier of hedonic analysis by | |
| 79 | +enough to matter for appraisal, automated valuation, and assessment applications. | |
| 80 | + | |
| 81 | +The remainder of the paper is organized as follows. Section~\ref{sec:literature} | |
| 82 | +reviews the related literature. Section~\ref{sec:data} describes the data sources | |
| 83 | +and cleaning. Section~\ref{sec:methodology} presents the feature-engineering | |
| 84 | +pipeline and the econometric specification. Section~\ref{sec:results} reports the | |
| 85 | +main results. Section~\ref{sec:benchmark} maps the predictive frontier across | |
| 86 | +estimators. Section~\ref{sec:robustness} discusses robustness and limitations. | |
| 87 | +Section~\ref{sec:conclusion} concludes. | |
added
paper/sections/literature.tex
+74 −0
@@ -0,0 +1,74 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Related Literature} | |
| 3 | +\label{sec:literature} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +This paper connects four strands of research. | |
| 7 | + | |
| 8 | +The first is the classical \textbf{hedonic pricing} literature. Building on | |
| 9 | +\citet{lancaster1966new}, \citet{rosen1974hedonic} formalized the market for | |
| 10 | +differentiated goods as a market for their underlying characteristics, providing the | |
| 11 | +theoretical foundation for interpreting regression coefficients as implicit prices. | |
| 12 | +The empirical housing literature that followed is vast; \citet{sirmans2005composition} | |
| 13 | +survey the characteristics most commonly included in hedonic house-price models and | |
| 14 | +document substantial heterogeneity in specification, \citet{malpezzi2003hedonic} | |
| 15 | +provides a critical review of functional-form and identification choices, and | |
| 16 | +\citet{hill2013hedonic} surveys the use of hedonic methods for house-price | |
| 17 | +measurement. A recurring theme in this literature is omitted-variable bias from | |
| 18 | +unmeasured quality: the state of repair, the standard of finishes, and the visual | |
| 19 | +quality of a dwelling are priced by the market but rarely recorded in the structured | |
| 20 | +fields available to the econometrician. Our contribution speaks directly to this | |
| 21 | +measurement problem. Our log--linear specification, with logged continuous attributes | |
| 22 | +and a dependent variable in logs, follows the mainstream of this literature. | |
| 23 | + | |
| 24 | +The second strand concerns \textbf{spatial structure, functional form, and | |
| 25 | +prediction}. \citet{can1992spatial} and \citet{anselin1988spatial} emphasize that | |
| 26 | +unobserved locational amenities induce spatial dependence that, if ignored, biases | |
| 27 | +coefficient estimates; we absorb the first-order component of this problem through a | |
| 28 | +rich set of municipality fixed effects and cluster inference at the same level. | |
| 29 | +On the prediction side, \citet{bin2004prediction} compares parametric and | |
| 30 | +semi-parametric hedonic predictors, \citet{mullainathan2017machine} frame | |
| 31 | +machine-learning methods as estimators of the conditional expectation | |
| 32 | +$\mathbb{E}[P\mid\mathbf{z}]$ well suited to prediction tasks, and | |
| 33 | +\citet{kok2017bigdata} document the rise of automated valuation models in industry. | |
| 34 | +This literature generally finds that flexible learners beat linear hedonics on | |
| 35 | +accuracy at some cost in interpretability; our estimator ladder | |
| 36 | +(Section~\ref{sec:benchmark}) reproduces that pattern on a common feature set, and | |
| 37 | +our focus is on how much of the gap is closed not by a better \emph{estimator} but by | |
| 38 | +better \emph{measurement}. | |
| 39 | + | |
| 40 | +The third strand is the use of \textbf{unstructured listing content}---text and | |
| 41 | +images---in real-estate valuation. \citet{gentzkow2019text} survey text-as-data | |
| 42 | +methods in economics. In real estate specifically, \citet{nowak2017textual} show that | |
| 43 | +words in MLS remarks predict transaction prices and reduce hedonic pricing errors, | |
| 44 | +and \citet{shen2021information} extract quality signals from property descriptions | |
| 45 | +with supervised machine learning, finding that the text carries information about | |
| 46 | +unobserved quality that is capitalized into prices. On the image side, | |
| 47 | +\citet{ahmed2016house} combine visual and textual features for price estimation; | |
| 48 | +\citet{poursaeed2018vision} show that learned measures of interior and exterior | |
| 49 | +quality carry independent predictive power; \citet{glaeser2018vision} use street-level | |
| 50 | +imagery of Boston houses to measure exterior quality and study its interaction with | |
| 51 | +incentives; \citet{johnson2020curb} construct a curb-appeal index from Google Street | |
| 52 | +View images and estimate its capitalization into sale prices; and | |
| 53 | +\citet{lindenthal2021architecture} classify architectural styles from photographs and | |
| 54 | +document style premia. Relative to this work, our approach differs in the | |
| 55 | +\emph{technology of measurement}: rather than training bespoke supervised models on | |
| 56 | +labeled data---which requires labels, task-specific engineering, and yields opaque | |
| 57 | +scores---we use general-purpose large language and vision--language models with | |
| 58 | +schema-constrained decoding to produce a fixed set of interpretable, human-readable | |
| 59 | +attributes (e.g., ``renovated kitchen,'' ``brick façade,'' a 1--10 curb-appeal score) | |
| 60 | +at a marginal cost of a fraction of a cent per listing, and we embed those attributes | |
| 61 | +in a transparent hedonic regression whose coefficients can be read as implicit | |
| 62 | +prices. The models never observe the price, so the extracted attributes cannot | |
| 63 | +mechanically encode the outcome. | |
| 64 | + | |
| 65 | +The fourth strand is the \textbf{microstructure of listing markets}, which frames two | |
| 66 | +institutional features of our data. Our sample pools a brokered MLS platform | |
| 67 | +(Realtor.ca) with a for-sale-by-owner platform (DuProprio); \citet{hendel2009mls} | |
| 68 | +compare exactly these two market forms and document differences in seller | |
| 69 | +composition, marketing, and outcomes, motivating our platform indicator and | |
| 70 | +platform-specific robustness checks. And because we observe asking rather than | |
| 71 | +transaction prices, the interpretation of our dependent variable rests on the | |
| 72 | +list-price literature: the asking price is a strategic choice that anchors search and | |
| 73 | +bargaining \citep{horowitz1992list,knight2002listing,han2015microstructure}. We | |
| 74 | +return to this limitation in Section~\ref{sec:robustness}. | |
added
paper/sections/methodology.tex
+241 −0
@@ -0,0 +1,241 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Methodology} | |
| 3 | +\label{sec:methodology} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +Our empirical strategy has two parts: a feature-engineering pipeline that turns raw | |
| 7 | +listings into an analysis-ready design matrix (Sections~\ref{subsec:structural}--% | |
| 8 | +\ref{subsec:vision}), and a hedonic regression that maps those features to price | |
| 9 | +(Section~\ref{subsec:econometrics}). | |
| 10 | + | |
| 11 | +\subsection{Theoretical Framework} | |
| 12 | +\label{subsec:theory} | |
| 13 | + | |
| 14 | +Following \citet{rosen1974hedonic}, a house is a bundle of $K$ characteristics | |
| 15 | +$\mathbf{z}=(z_1,\dots,z_K)$, and the market clears through a hedonic price function | |
| 16 | +$P(\mathbf{z})$ that equates the marginal bid of buyers to the marginal offer of | |
| 17 | +sellers. A buyer with income $y$ and preferences $U(\mathbf{x},\mathbf{z})$ over a | |
| 18 | +numéraire good $\mathbf{x}$ and housing attributes chooses $\mathbf{z}$ to solve | |
| 19 | +% | |
| 20 | +\begin{equation} | |
| 21 | + \max_{\mathbf{x},\,\mathbf{z}}\; U(\mathbf{x},\mathbf{z}) | |
| 22 | + \quad\text{s.t.}\quad y = \mathbf{x} + P(\mathbf{z}), | |
| 23 | + \label{eq:consumer} | |
| 24 | +\end{equation} | |
| 25 | +% | |
| 26 | +which yields the first-order condition that, at the optimum, the implicit | |
| 27 | +(\emph{marginal}) price of each attribute equals the buyer's marginal willingness to | |
| 28 | +pay: | |
| 29 | +% | |
| 30 | +\begin{equation} | |
| 31 | + p_k(\mathbf{z}) \;\equiv\; \frac{\partial P(\mathbf{z})}{\partial z_k} | |
| 32 | + \;=\; \frac{\partial U/\partial z_k}{\partial U/\partial x}, | |
| 33 | + \qquad k = 1,\dots,K. | |
| 34 | + \label{eq:implicit} | |
| 35 | +\end{equation} | |
| 36 | +% | |
| 37 | +Equation~\eqref{eq:implicit} is the object of interest: the slope of the hedonic | |
| 38 | +surface in the direction of attribute~$z_k$. The empirical task is to recover | |
| 39 | +$P(\mathbf{z})$---and hence its gradient---from observed prices and characteristics. | |
| 40 | +The linear model of Section~\ref{subsec:econometrics} imposes a parametric form on | |
| 41 | +$P(\cdot)$ and reads the implicit prices off its coefficients; the machine-learning | |
| 42 | +estimators of Section~\ref{sec:benchmark} approximate $P(\cdot)$ nonparametrically, | |
| 43 | +trading the closed-form gradient for predictive flexibility. | |
| 44 | + | |
| 45 | +\subsection{Structural and Locational Attributes} | |
| 46 | +\label{subsec:structural} | |
| 47 | + | |
| 48 | +Structural attributes are extracted by deterministic rules. Living area and lot area | |
| 49 | +are reported in heterogeneous units (\texttt{"1535 sqft"}, \texttt{"1,333.2 ft²"}, | |
| 50 | +\texttt{"11 561.52 ft²"}, \texttt{"961.6 m2"}); we parse each string and convert all | |
| 51 | +areas to square feet. For Realtor.ca listings that report room-by-room dimensions | |
| 52 | +(e.g.\ \texttt{"23 ft ,4 in x 13 ft ,8 in"}) but no aggregate living area, we sum the | |
| 53 | +parsed room areas and calibrate the total against listings where both are observed; | |
| 54 | +the median ratio of reported area to summed-room area is~0.98, so the reconstructed | |
| 55 | +value is used directly to fill missing living area. Age is computed as | |
| 56 | +$2026-\text{year built}$, entered together with its square to allow a non-monotone | |
| 57 | +depreciation profile. Bedrooms, bathrooms, parking spaces, and storeys are taken | |
| 58 | +from the attribute block. Missing values are imputed by the within-property-type | |
| 59 | +median, and each imputed field is accompanied by a ``\texttt{has\_}'' indicator so | |
| 60 | +that the regression can distinguish observed from imputed values. | |
| 61 | + | |
| 62 | +Location is captured by \textbf{municipality fixed effects}. City names are | |
| 63 | +normalized (Realtor.ca appends neighborhood qualifiers in parentheses, which we | |
| 64 | +strip), and municipalities with fewer than 20 listings are pooled into a residual | |
| 65 | +category, leaving 195 city effects. Property-type fixed effects and a source | |
| 66 | +indicator complete the locational and platform controls. | |
| 67 | + | |
| 68 | +\subsection{Rule-Based Amenities and Room Detail} | |
| 69 | +\label{subsec:amenities} | |
| 70 | + | |
| 71 | +Seventeen amenity indicators are detected by bilingual (French/English) keyword | |
| 72 | +matching over the concatenated feature list, description, and title: swimming pool, | |
| 73 | +waterfront, garage, fireplace, central air conditioning or heat pump, finished | |
| 74 | +basement, cul-de-sac, proximity to a golf course, ski hill, park, school, highway, | |
| 75 | +and public transit, among others. From the Realtor.ca room block we additionally | |
| 76 | +construct room-detail indicators---presence of a primary bedroom, a basement, an | |
| 77 | +office, a family room, a workshop, a wine cellar, a sunroom, a mezzanine, a library, | |
| 78 | +and a laundry room---together with the room and storey counts. Two listing-quality | |
| 79 | +signals, the number of photographs (capped at~60) and the presence of a virtual | |
| 80 | +tour, proxy for the marketing tier of the listing. | |
| 81 | + | |
| 82 | +\subsection{Language-Model Specifications from Text} | |
| 83 | +\label{subsec:llm} | |
| 84 | + | |
| 85 | +Free-text descriptions contain valuation-relevant information that resists keyword | |
| 86 | +extraction: whether the home is turnkey or a fixer-upper, whether the kitchen or | |
| 87 | +bathroom has been renovated, the flooring and countertop materials, the heating | |
| 88 | +system, the presence of a water view, and income potential. We extract these as a | |
| 89 | +fixed set of structured specifications using a large language model | |
| 90 | +(\texttt{gpt-4o-mini}) with \emph{schema-constrained decoding}: the request specifies | |
| 91 | +a strict JSON schema with closed enumerations and required fields, so the API | |
| 92 | +validates the output at generation time and the model returns exactly the expected | |
| 93 | +fields in the expected value domains. Decoding temperature is set to zero for | |
| 94 | +determinism, and the system instruction constrains the model to report only what the | |
| 95 | +text supports, coding absent information as \texttt{false}/\texttt{unknown}/empty. | |
| 96 | +Each call consumes roughly 600 input and 110 output tokens; the extraction is run in | |
| 97 | +parallel, is resumable through an on-disk cache keyed by the listing identifier, and | |
| 98 | +retries transient errors with exponential backoff, for a total cost of about \$3 | |
| 99 | +across the sample. | |
| 100 | + | |
| 101 | +\subsection{Vision Attributes from the Façade Photograph} | |
| 102 | +\label{subsec:vision} | |
| 103 | + | |
| 104 | +To capture the visual quality that neither structured fields nor text convey, we | |
| 105 | +analyze one photograph per listing---the façade---with a vision--language model | |
| 106 | +(\texttt{claude-haiku-4-5}). For Realtor.ca the first image is the exterior shot; for | |
| 107 | +DuProprio, whose images are captioned, we select the image whose caption is the | |
| 108 | +address rather than a room. The model returns twelve objective attributes under a | |
| 109 | +strict JSON schema: an overall condition score (1--10), a curb-appeal score (1--10), | |
| 110 | +a renovation level, exterior materials, landscaping quality, visible garage, storeys, | |
| 111 | +setting (urban/suburban/rural/waterfront/wooded), a luxury indicator, and photo | |
| 112 | +quality. The system instruction directs the model to ground each score in concrete, | |
| 113 | +reproducible criteria (surface condition, materials, landscaping) rather than | |
| 114 | +subjective taste, and to flag non-exterior images. Images are passed by URL without | |
| 115 | +local download; each call consumes roughly 2{,}470 input and 100 output tokens, for a | |
| 116 | +total cost of about \$57 across the sample. As with the text layer, the pass is | |
| 117 | +parallelized, resumable, and retried. | |
| 118 | + | |
| 119 | +\paragraph{Why schema-constrained decoding matters.} | |
| 120 | +Imposing a typed schema with enumerations, required fields, and | |
| 121 | +\texttt{additionalProperties:false} moves validation into the API layer. The models | |
| 122 | +are forced to emit exactly the expected fields within the expected value domains, so | |
| 123 | +their outputs behave as ordinary tabulated variables---reproducible and free of | |
| 124 | +free-text post-processing. The outputs are then mapped to numeric regressors: | |
| 125 | +booleans to $0/1$, the ordinal condition scale to $0$--$5$, materials to indicator | |
| 126 | +variables, and so on, each with an availability flag. | |
| 127 | + | |
| 128 | +\subsection{Econometric Specification} | |
| 129 | +\label{subsec:econometrics} | |
| 130 | + | |
| 131 | +We adopt a semi-logarithmic functional form, the standard workhorse in the hedonic | |
| 132 | +literature \citep{malpezzi2003hedonic}. Collect the attributes of listing~$i$ into | |
| 133 | +disjoint blocks, | |
| 134 | +% | |
| 135 | +\begin{equation} | |
| 136 | + \resizebox{.94\textwidth}{!}{$\displaystyle | |
| 137 | + \mathbf{X}_i = | |
| 138 | + \big(\underbrace{\ln\text{Area}_i,\ \ln\text{Lot}_i}_{\text{size (logged)}},\ | |
| 139 | + \underbrace{\mathbf{s}_i}_{\text{structural}},\ | |
| 140 | + \underbrace{\mathbf{a}_i}_{\text{amenities}},\ | |
| 141 | + \underbrace{\mathbf{r}_i}_{\text{rooms}},\ | |
| 142 | + \underbrace{\mathbf{m}_i}_{\text{listing}},\ | |
| 143 | + \underbrace{\mathbf{g}_i}_{\text{LLM text}},\ | |
| 144 | + \underbrace{\mathbf{v}_i}_{\text{vision}}\big) | |
| 145 | + $} | |
| 146 | + \label{eq:blocks} | |
| 147 | +\end{equation} | |
| 148 | +% | |
| 149 | +and estimate, by ordinary least squares, | |
| 150 | +% | |
| 151 | +\begin{equation} | |
| 152 | + \ln P_i \;=\; \alpha | |
| 153 | + \;+\; \beta\,\ln\text{Area}_i | |
| 154 | + \;+\; \gamma\,\ln\text{Lot}_i | |
| 155 | + \;+\; \mathbf{X}_i^{\!\top}\bm{\delta} | |
| 156 | + \;+\; \underbrace{\mu_{c(i)}}_{\text{city FE}} | |
| 157 | + \;+\; \underbrace{\tau_{t(i)}}_{\text{type FE}} | |
| 158 | + \;+\; \varepsilon_i , | |
| 159 | + \label{eq:hedonic} | |
| 160 | +\end{equation} | |
| 161 | +% | |
| 162 | +where $c(i)$ and $t(i)$ index the municipality and property type of listing~$i$, and | |
| 163 | +$\mu_c$, $\tau_t$ are the corresponding fixed effects that sweep out | |
| 164 | +location- and type-specific price levels. In matrix form | |
| 165 | +$\mathbf{y}=\mathbf{Z}\bm{\theta}+\bm{\varepsilon}$ with | |
| 166 | +$\widehat{\bm{\theta}}=(\mathbf{Z}^{\!\top}\mathbf{Z})^{-1}\mathbf{Z}^{\!\top}\mathbf{y}$. | |
| 167 | + | |
| 168 | +\paragraph{Interpretation of the coefficients.} | |
| 169 | +Because the size variables enter in logs, $\beta$ and $\gamma$ are \emph{elasticities}: | |
| 170 | +$\partial \ln P/\partial \ln\text{Area}=\beta$. For an indicator regressor $d$ with | |
| 171 | +coefficient $b$, the exact percentage price effect is not $b$ but | |
| 172 | +% | |
| 173 | +\begin{equation} | |
| 174 | + \%\Delta P \;=\; \big(e^{\,b}-1\big)\times 100, | |
| 175 | + \label{eq:halvorsen} | |
| 176 | +\end{equation} | |
| 177 | +% | |
| 178 | +the \citet{halvorsen1980interpretation} correction we apply throughout (for small | |
| 179 | +$b$, $e^{b}-1\approx b$). Under Equation~\eqref{eq:hedonic} the implicit price of a | |
| 180 | +continuous attribute $z_k$ from \eqref{eq:implicit} is $p_k=\delta_k\,P/z_k$, so the | |
| 181 | +semi-log form lets the \emph{dollar} marginal price scale with the price level while | |
| 182 | +keeping the \emph{percentage} effect constant. | |
| 183 | + | |
| 184 | +\paragraph{Two-part coverage design.} | |
| 185 | +Because living area is reported for only about 38\% of Realtor.ca listings, we | |
| 186 | +estimate two nested versions of \eqref{eq:hedonic} and combine their predictions: | |
| 187 | +% | |
| 188 | +\begin{equation} | |
| 189 | + \widehat{\ln P}_i = | |
| 190 | + \begin{cases} | |
| 191 | + \widehat{\ln P}_i^{\,A} & \text{if area is measured (Model A, rich),}\\[2pt] | |
| 192 | + \widehat{\ln P}_i^{\,B} & \text{otherwise (Model B, broad; }\beta\equiv 0). | |
| 193 | + \end{cases} | |
| 194 | + \label{eq:twopart} | |
| 195 | +\end{equation} | |
| 196 | +% | |
| 197 | +Model~A includes $\ln\text{Area}$ and is fit on the measured-area subsample; Model~B | |
| 198 | +drops it and is fit on the full cleaned sample, so every listing can be priced. | |
| 199 | + | |
| 200 | +\paragraph{Inference.} | |
| 201 | +House-price residuals are strongly heteroskedastic, and listings within a | |
| 202 | +municipality share unobserved local shocks that induce within-cluster error | |
| 203 | +correlation. Our preferred standard errors are therefore clustered at the | |
| 204 | +municipality level (195 clusters), the same level as the fixed effects | |
| 205 | +\citep{cameron2015cluster}: | |
| 206 | +% | |
| 207 | +\begin{equation} | |
| 208 | + \widehat{\mathrm{Var}}_{\text{CL}}(\widehat{\bm{\theta}}) | |
| 209 | + = (\mathbf{Z}^{\!\top}\mathbf{Z})^{-1} | |
| 210 | + \Bigg(\sum_{c=1}^{C} | |
| 211 | + \mathbf{Z}_c^{\!\top}\hat{\bm{\varepsilon}}_c | |
| 212 | + \hat{\bm{\varepsilon}}_c^{\!\top}\mathbf{Z}_c\Bigg) | |
| 213 | + (\mathbf{Z}^{\!\top}\mathbf{Z})^{-1}, | |
| 214 | + \label{eq:cluster} | |
| 215 | +\end{equation} | |
| 216 | +% | |
| 217 | +where $\mathbf{Z}_c$ and $\hat{\bm{\varepsilon}}_c$ stack the regressors and | |
| 218 | +residuals of cluster $c$. As a complement we also compute | |
| 219 | +heteroskedasticity-consistent (HC3) standard errors | |
| 220 | +\citep{white1980heteroskedasticity,mackinnon1985some}, which inflate the | |
| 221 | +contribution of high-leverage points; Section~\ref{subsec:robust} shows that the two | |
| 222 | +choices agree on the significance of the property-level AI attributes and differ | |
| 223 | +only for keyword amenities with a strong spatial component. | |
| 224 | + | |
| 225 | +\paragraph{Retransformation.} | |
| 226 | +Predicting the \emph{level} of price requires undoing the log. Because | |
| 227 | +$\mathbb{E}[P\mid\mathbf{Z}]=\exp(\mathbf{Z}\bm{\theta})\,\mathbb{E}[e^{\varepsilon}] | |
| 228 | +\neq \exp(\mathbf{Z}\bm{\theta})$, naive exponentiation is biased downward. Under | |
| 229 | +$\varepsilon\sim\mathcal N(0,\sigma^2)$ we use the closed-form correction, and more | |
| 230 | +generally Duan's nonparametric smearing estimate \citep{duan1983smearing}: | |
| 231 | +% | |
| 232 | +\begin{equation} | |
| 233 | + \widehat{P}_i | |
| 234 | + = \exp\!\big(\widehat{\ln P}_i\big)\cdot | |
| 235 | + \underbrace{\exp\!\big(\tfrac12\hat\sigma^{2}\big)}_{\text{normal}} | |
| 236 | + \qquad\text{or}\qquad | |
| 237 | + \widehat{P}_i | |
| 238 | + = \exp\!\big(\widehat{\ln P}_i\big)\cdot | |
| 239 | + \underbrace{\frac1n\sum_{j=1}^{n} e^{\hat\varepsilon_j}}_{\text{smearing }\hat\phi}. | |
| 240 | + \label{eq:smearing} | |
| 241 | +\end{equation} | |
added
paper/sections/results.tex
+105 −0
@@ -0,0 +1,105 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Results} | |
| 3 | +\label{sec:results} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{Baseline Estimates} | |
| 7 | +\label{subsec:baseline} | |
| 8 | + | |
| 9 | +Table~\ref{tab:regression} reports the two hedonic specifications estimated on the | |
| 10 | +full feature set (structural, amenity, room-detail, listing-signal, LLM text-derived, | |
| 11 | +and vision-derived attributes). Model~A, which includes living area and is estimated | |
| 12 | +on the 9{,}446 listings with a measured area, explains 75.5\% of the variation in log | |
| 13 | +prices ($R^2=0.755$, adjusted $0.748$). Model~B, the area-free specification estimated | |
| 14 | +on all 19{,}123 listings, explains 72.5\% ($R^2=0.725$, adjusted $0.721$). Adding the | |
| 15 | +machine-generated feature layers raises Model~A's $R^2$ from 0.717 (structural, | |
| 16 | +amenity, room, and listing attributes only) to 0.725 with the text layer and 0.755 | |
| 17 | +with the vision layer---a cumulative gain of nearly four points from the two | |
| 18 | +AI-generated feature sets, over and above an already rich hedonic specification. | |
| 19 | + | |
| 20 | +The estimated implicit prices are economically sensible and, with few exceptions, | |
| 21 | +statistically significant with standard errors clustered by municipality. In | |
| 22 | +Model~A, the living-area elasticity is 0.36 (a 10\% larger home commands roughly a | |
| 23 | +3.6\% higher price) and the lot-area elasticity is small but precisely estimated. | |
| 24 | +Each additional \textbf{bathroom} is associated with a 7.6\% premium and each parking | |
| 25 | +space with about 1\%. \textbf{Central air conditioning} carries a 5.5\% premium and | |
| 26 | +\textbf{public-transit} proximity 9.4\%, although the transit coefficient---like | |
| 27 | +other keyword amenities with a strong spatial component---is estimated much less | |
| 28 | +precisely once errors are allowed to correlate within municipalities. | |
| 29 | +\textbf{Waterfront} access is associated with a 6.0--6.4\% premium, significant at | |
| 30 | +conventional levels in the full-sample model. Age enters with a small negative linear | |
| 31 | +term and a positive quadratic, tracing the familiar depreciation-then-heritage | |
| 32 | +profile. | |
| 33 | + | |
| 34 | +\input{tables/regression} | |
| 35 | + | |
| 36 | +\subsection{Text- and Vision-Derived Attributes} | |
| 37 | +\label{subsec:ai_coeffs} | |
| 38 | + | |
| 39 | +The two AI feature layers contribute distinct, largely orthogonal signal. | |
| 40 | + | |
| 41 | +\paragraph{Text (LLM).} Beyond the keyword amenities, the structured specifications | |
| 42 | +carry independent information about finish and condition: a \textbf{renovated | |
| 43 | +kitchen} is associated with a 2.6\% premium, \textbf{hardwood flooring} with 2.4\%, | |
| 44 | +and the ordinal \textbf{condition} scale is positive and significant in both models | |
| 45 | +(1.6\% per step). Listings the model reads as \textbf{income properties} sell about | |
| 46 | +1.9\% below otherwise-comparable houses. The \textbf{new-construction} indicator, | |
| 47 | +by contrast, is indistinguishable from zero once age, the visual renovation level, | |
| 48 | +and the condition scales are controlled for---the premium a naive specification | |
| 49 | +would attribute to the label ``new'' is fully absorbed by the measured quality | |
| 50 | +attributes. | |
| 51 | + | |
| 52 | +\paragraph{Vision (façade).} The image-derived attributes are jointly significant and | |
| 53 | +add the largest single increment to fit. Holding all structural, amenity, and text | |
| 54 | +attributes fixed, a one-point increase in \textbf{curb appeal} (1--10) is associated | |
| 55 | +with a 2.3\% higher price, and a one-step increase in the visual \textbf{renovation | |
| 56 | +level} with 2.4\%. A visible \textbf{luxury} cue adds 5.2\%, a \textbf{visible garage} | |
| 57 | +4.7\%, and each step of \textbf{landscaping} quality 1.5\%. Exterior materials price | |
| 58 | +as expected: \textbf{brick} ($+3.4$\%) and \textbf{stone} ($+3.7$\%) command premia, | |
| 59 | +while \textbf{vinyl} siding carries a 4.4\% discount. The visual \textbf{rural} | |
| 60 | +setting is associated with an 11.5\% discount relative to the omitted category. That | |
| 61 | +these visual measures remain significant after conditioning on the full text and | |
| 62 | +structural set---and with inference clustered at the municipality level---confirms | |
| 63 | +that the façade photograph carries price-relevant information available in no other | |
| 64 | +field. | |
| 65 | + | |
| 66 | +\subsection{Validation of Machine-Generated Features} | |
| 67 | +\label{subsec:validation} | |
| 68 | + | |
| 69 | +Before relying on the AI-generated attributes, we check that they order prices in the | |
| 70 | +expected direction. Figure~\ref{fig:validation} plots the median list price against the | |
| 71 | +vision model's curb-appeal and condition scores among listings with a detected | |
| 72 | +exterior shot. Both relationships are strongly monotone: median price rises steadily | |
| 73 | +from the lowest to the highest visual scores, with no reliance on the regression to | |
| 74 | +impose the pattern. Because these scores are produced by a model that never sees the | |
| 75 | +price, the monotonicity is independent evidence that the vision layer extracts a real, | |
| 76 | +price-relevant quality signal rather than noise. Table~\ref{tab:ai_summary} reports the | |
| 77 | +prevalence of the main text- and vision-derived attributes. | |
| 78 | + | |
| 79 | +\begin{figure}[htbp] | |
| 80 | + \centering | |
| 81 | + \includegraphics[width=\textwidth]{figures/fig_validation.pdf} | |
| 82 | + \caption{Validation of the vision layer. Median list price rises monotonically with | |
| 83 | + the model's curb-appeal (left) and condition (right) scores, among listings with a | |
| 84 | + detected façade shot. The scores are generated without access to price.} | |
| 85 | + \label{fig:validation} | |
| 86 | +\end{figure} | |
| 87 | + | |
| 88 | +\subsection{Location Fixed Effects} | |
| 89 | +\label{subsec:location} | |
| 90 | + | |
| 91 | +The 195 municipality fixed effects absorb the dominant locational component of price. | |
| 92 | +Relative to the reference city, the largest positive premia are estimated for | |
| 93 | +\textbf{Westmount} ($\approx+231$\%), \textbf{Hampstead} ($+173$\%), and | |
| 94 | +\textbf{Mont-Royal} ($+169$\%), followed by outlying municipalities at the low end | |
| 95 | +carrying discounts of 23--30\%. Figure~\ref{fig:premia} plots the extreme effects. The | |
| 96 | +magnitude and ordering align with well-known patterns in the Quebec housing market and | |
| 97 | +lend face validity to the specification. | |
| 98 | + | |
| 99 | +\begin{figure}[htbp] | |
| 100 | + \centering | |
| 101 | + \includegraphics[width=.74\textwidth]{figures/fig_premia.pdf} | |
| 102 | + \caption{Estimated municipality fixed effects (Model~A), the twelve largest and eight | |
| 103 | + smallest, expressed as percentage premia relative to the reference city.} | |
| 104 | + \label{fig:premia} | |
| 105 | +\end{figure} | |
added
paper/sections/robustness.tex
+99 −0
@@ -0,0 +1,99 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Robustness and Limitations} | |
| 3 | +\label{sec:robustness} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{Robustness} | |
| 7 | +\label{subsec:robust} | |
| 8 | + | |
| 9 | +Several features of the design guard against spurious results. The dual | |
| 10 | +specification (Section~\ref{subsec:econometrics}) ensures that estimates are not | |
| 11 | +driven by the non-random availability of living area: coefficients on attributes | |
| 12 | +common to both models retain their sign and rough magnitude whether or not area is | |
| 13 | +conditioned on. The ``\texttt{has\_}'' indicators accompanying each imputed field let | |
| 14 | +the regression separate observed from imputed information, so median imputation does | |
| 15 | +not silently attenuate the size coefficients. The price-per-square-foot trim removes | |
| 16 | +the residual parsing errors that would otherwise dominate the tails, and the | |
| 17 | +municipality fixed effects absorb the first-order spatial confound emphasized by | |
| 18 | +\citet{can1992spatial} and \citet{anselin1988spatial}. | |
| 19 | + | |
| 20 | +\paragraph{Inference.} Because listings within a municipality share unobserved | |
| 21 | +local shocks, our preferred standard errors are clustered at the municipality level | |
| 22 | +(195 clusters; \citealp{cameron2015cluster}), the same level as the fixed effects. | |
| 23 | +Clustering matters exactly where theory predicts: the text- and vision-derived | |
| 24 | +quality attributes---which vary house by house within a municipality---are | |
| 25 | +essentially unaffected (curb appeal, renovation level, luxury, garage, materials, | |
| 26 | +landscaping, and the rural-setting discount all remain significant at the 5\% level | |
| 27 | +or better in Model~A, as do the LLM condition scale, renovated kitchen, and hardwood | |
| 28 | +indicators), whereas keyword amenities with a strong spatial component lose | |
| 29 | +precision (public-transit proximity falls to the 5\% boundary in Model~A and is | |
| 30 | +insignificant in Model~B; central air conditioning is insignificant in Model~B; the | |
| 31 | +curb-appeal score, precisely estimated in Model~A, loses significance in the broader | |
| 32 | +Model~B). | |
| 33 | +Heteroskedasticity-robust (HC3) standard errors | |
| 34 | +\citep{white1980heteroskedasticity,mackinnon1985some} yield the same qualitative | |
| 35 | +pattern with uniformly smaller intervals. This asymmetry is reassuring for our main | |
| 36 | +claim: the AI-generated attributes capture within-market, property-level quality | |
| 37 | +variation rather than repackaged location effects. | |
| 38 | + | |
| 39 | +Table~\ref{tab:robustness} probes the stability of the predictive blend across | |
| 40 | +subsamples. Three patterns stand out. First, accuracy is comparable whether living | |
| 41 | +area is observed (median error 13.5\%) or imputed (14.4\%), confirming that the | |
| 42 | +area-free model prices the incomplete listings almost as well as the rich model. | |
| 43 | +Second, the model fits the broker-listed Realtor.ca inventory better | |
| 44 | +($R^2_{\text{OOS}}=0.80$) than the smaller, noisier for-sale-by-owner DuProprio | |
| 45 | +segment (0.58), though median errors are identical at 14.0\%; the gap is consistent | |
| 46 | +with the compositional and marketing differences between brokered and FSBO | |
| 47 | +platforms documented by \citet{hendel2009mls}. Third, the median error is stable | |
| 48 | +across price terciles (11--16\%), lowest in the mid-market and modestly higher in | |
| 49 | +the tails, where idiosyncratic and luxury attributes are harder to capture. (Within | |
| 50 | +a single tercile $R^2$ is uninformative because the price range is restricted; the | |
| 51 | +median error is the comparable metric.) | |
| 52 | + | |
| 53 | +\input{tables/robustness} | |
| 54 | + | |
| 55 | +\subsection{Limitations} | |
| 56 | +\label{subsec:limits} | |
| 57 | + | |
| 58 | +Five limitations qualify the interpretation of our estimates. | |
| 59 | + | |
| 60 | +\emph{Asking prices.} The dependent variable is the asking price of properties | |
| 61 | +listed for sale, not the transaction price. The list price is a strategic choice | |
| 62 | +that anchors buyer search and bargaining | |
| 63 | +\citep{horowitz1992list,knight2002listing,han2015microstructure}: it is set with | |
| 64 | +sale in mind and is highly correlated with the eventual transaction price, but | |
| 65 | +list-to-sale ratios vary systematically with market conditions and overpricing is | |
| 66 | +more common in thin segments. Two implications follow. First, our implicit prices | |
| 67 | +measure how sellers and their agents \emph{price} attributes into the ask, which | |
| 68 | +converges to buyers' valuation only insofar as the market disciplines the ask; the | |
| 69 | +economically sensible magnitudes and the monotone validation of | |
| 70 | +Section~\ref{subsec:validation} suggest the discipline is substantial. Second, any | |
| 71 | +attribute that shifts the list-to-sale ratio itself (e.g., attributes of hard-to-sell | |
| 72 | +properties) will carry a coefficient that mixes valuation and listing strategy. | |
| 73 | +Replicating the analysis on registry transaction prices is the natural next step. | |
| 74 | + | |
| 75 | +\emph{Non-causality and content endogeneity.} Descriptions and photographs are | |
| 76 | +marketing instruments chosen by sellers and agents. Richer descriptions, more | |
| 77 | +photographs, and higher visual quality correlate with higher-end listing agents and | |
| 78 | +more valuable properties \citep{nowak2017textual}. The coefficients are therefore | |
| 79 | +\emph{predictive} associations, not causal effects; they should not be read as the | |
| 80 | +price change from physically adding a given attribute. We control for two observable | |
| 81 | +dimensions of marketing effort (photo count and virtual tours), but cannot rule out | |
| 82 | +that residual marketing quality loads on the AI-derived attributes. | |
| 83 | + | |
| 84 | +\emph{Machine-generated features.} LLM and vision outputs can contain errors. | |
| 85 | +Zero-temperature decoding, strict schemas, and closed enumerations bound this risk | |
| 86 | +and make it reproducible, and classical measurement error in the extracted scores | |
| 87 | +would bias their coefficients toward zero---making our estimates conservative---but | |
| 88 | +we do not perform systematic human validation, and the visual layer judges each | |
| 89 | +property from a single façade photograph, ignoring interiors. | |
| 90 | + | |
| 91 | +\emph{Selection.} Listed properties are not a random sample of the housing stock, | |
| 92 | +and listings with missing area or room detail are not missing at random (they tend | |
| 93 | +to be less complete listings). The ``\texttt{has\_}'' indicators and the two-part | |
| 94 | +design mitigate, but do not eliminate, the second concern. | |
| 95 | + | |
| 96 | +\emph{Cross-section.} The data are a single-date snapshot, so the model captures no | |
| 97 | +market dynamics and the location premia are fixed as of 3~July~2026. Municipalities | |
| 98 | +with fewer than 20 listings are pooled, reducing spatial resolution where data are | |
| 99 | +sparse. | |
added
paper/sections/titlepage.tex
+77 −0
@@ -0,0 +1,77 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Title Page | |
| 3 | +% ============================================================================ | |
| 4 | +\thispagestyle{empty} | |
| 5 | + | |
| 6 | +\begin{center} | |
| 7 | + | |
| 8 | +% --- Logo --- | |
| 9 | +\includegraphics[width=3.5cm]{uq_logo.jpg} | |
| 10 | + | |
| 11 | +\vspace{0.6cm} | |
| 12 | + | |
| 13 | +{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm] | |
| 14 | +{\footnotesize\textsc{Département des sciences administratives}} | |
| 15 | + | |
| 16 | +\vspace{0.8cm} | |
| 17 | + | |
| 18 | +{\footnotesize\textsc{Working Paper No.~\WPnumber}} | |
| 19 | + | |
| 20 | +\vspace{1.2cm} | |
| 21 | + | |
| 22 | +% --- Title --- | |
| 23 | +{\LARGE\bfseries \WPtitle\par} | |
| 24 | + | |
| 25 | +\ifx\WPsubtitle\empty\else | |
| 26 | + \vspace{0.3cm} | |
| 27 | + {\large\itshape \WPsubtitle\par} | |
| 28 | +\fi | |
| 29 | + | |
| 30 | +\vspace{1.2cm} | |
| 31 | + | |
| 32 | +% --- Author --- | |
| 33 | +{\large \WPauthor}\\[0.3cm] | |
| 34 | +{\normalsize \WPaffiliation}\\[0.15cm] | |
| 35 | +{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm] | |
| 36 | +{\small \WPaddress} | |
| 37 | + | |
| 38 | +\vspace{0.8cm} | |
| 39 | + | |
| 40 | +% --- Date & Version --- | |
| 41 | +{\normalsize \WPdate}\\[0.1cm] | |
| 42 | +{\small Version~\WPversion} | |
| 43 | + | |
| 44 | +\end{center} | |
| 45 | + | |
| 46 | +\vfill | |
| 47 | + | |
| 48 | +\newpage | |
| 49 | + | |
| 50 | +% ============================================================================ | |
| 51 | +% Abstract Page | |
| 52 | +% ============================================================================ | |
| 53 | +\thispagestyle{empty} | |
| 54 | + | |
| 55 | +\vspace*{1cm} | |
| 56 | + | |
| 57 | +\noindent\rule{\textwidth}{0.4pt} | |
| 58 | +\vspace{0.3cm} | |
| 59 | + | |
| 60 | +\noindent\textbf{Abstract} | |
| 61 | + | |
| 62 | +\vspace{0.15cm} | |
| 63 | + | |
| 64 | +\noindent\WPabstract | |
| 65 | + | |
| 66 | +\vspace{0.4cm} | |
| 67 | + | |
| 68 | +\noindent\textbf{Keywords:} \WPkeywords | |
| 69 | + | |
| 70 | +\vspace{0.15cm} | |
| 71 | + | |
| 72 | +\noindent\textbf{JEL Classification:} \WPjel | |
| 73 | + | |
| 74 | +\vspace{0.3cm} | |
| 75 | +\noindent\rule{\textwidth}{0.4pt} | |
| 76 | + | |
| 77 | +\newpage | |
added
paper/tables/ablation.tex
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +% Auto-generated (revision: recomputed on identical folds) | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Feature Ablation: Out-of-Sample $R^2$ by Feature Layer} | |
| 6 | + \label{tab:ablation} | |
| 7 | + \begin{tabular}{@{}lcccc@{}} | |
| 8 | + \toprule | |
| 9 | + Feature set & \# feats & OLS & GBM & Blend \\ | |
| 10 | + \midrule | |
| 11 | + Structural + amenity & 24 & 0.521 & 0.692 & --- \\ | |
| 12 | + \quad + Rooms + listing & 38 & 0.619 & 0.729 & --- \\ | |
| 13 | + \quad + Text (LLM) & 56 & 0.628 & 0.743 & 0.760 \\ | |
| 14 | + \quad + Vision (façade) & 77 & 0.686 & 0.770 & 0.779 \\ | |
| 15 | + \bottomrule | |
| 16 | + \end{tabular} | |
| 17 | + \begin{tablenotes}[flushleft] | |
| 18 | + \footnotesize | |
| 19 | + \item \textit{Notes:} Each row adds a feature layer to the row above (nested). | |
| 20 | + Entries are out-of-sample $R^2$ on $\ln(\text{Price})$ from 5-fold cross-validation | |
| 21 | + with a common seed. ``\# feats'' counts numeric/indicator features before the city, | |
| 22 | + type, and source fixed effects (added in every specification). The blend averages | |
| 23 | + the robust neural network and gradient boosting and is reported where both were | |
| 24 | + trained. GBM handles missing living area natively; OLS uses median imputation. | |
| 25 | + \end{tablenotes} | |
| 26 | + \end{threeparttable} | |
| 27 | +\end{table} | |
added
paper/tables/ai_summary.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% Auto-generated | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Machine-Generated Feature Summary} | |
| 6 | + \label{tab:ai_summary} | |
| 7 | + \begin{tabular}{@{}lc@{\hskip 1.2cm}lc@{}} | |
| 8 | + \toprule | |
| 9 | + Text (LLM) & Share & Vision (façade) & Mean / share \\ | |
| 10 | + \midrule | |
| 11 | + Recently renovated & 26\% & Condition score (1--10) & 7.4 \\ | |
| 12 | + Renovated kitchen & 17\% & Curb appeal (1--10) & 7.2 \\ | |
| 13 | + Hardwood floor & 16\% & Luxury cue & 43\% \\ | |
| 14 | + Quartz/granite counter & 7\% & Garage visible & 46\% \\ | |
| 15 | + New construction & 14\% & Brick façade & 47\% \\ | |
| 16 | + Income property & 12\% & Waterfront setting & 7\% \\ | |
| 17 | + Water view & 24\% & Exterior shot detected & 96\% \\ | |
| 18 | + \bottomrule | |
| 19 | + \end{tabular} | |
| 20 | + \begin{tablenotes}[flushleft] | |
| 21 | + \footnotesize | |
| 22 | + \item \textit{Notes:} Prevalence of selected machine-generated attributes across | |
| 23 | + the 19{,}123 cleaned listings. Text attributes are extracted from the | |
| 24 | + description by \texttt{gpt-4o-mini}; vision attributes from the façade photograph by | |
| 25 | + \texttt{claude-haiku-4-5}. Scores are 1--10 means; the remainder are shares. | |
| 26 | + \end{tablenotes} | |
| 27 | + \end{threeparttable} | |
| 28 | +\end{table} | |
added
paper/tables/models.tex
+39 −0
@@ -0,0 +1,39 @@ | ||
| 1 | +% Auto-generated (revision: full ladder in both panels) | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Predictive Performance: From OLS to Deep Learning} | |
| 6 | + \label{tab:models} | |
| 7 | + \begin{tabular}{@{}lcccc@{}} | |
| 8 | + \toprule | |
| 9 | + Estimator & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% & Within $\pm$25\% \\ | |
| 10 | + \midrule | |
| 11 | + \multicolumn{5}{@{}l}{\textit{Panel A: Structural + amenity + LLM-text features}} \\[3pt] | |
| 12 | + OLS (linear hedonic) & 0.628 & 17.9\% & 43\% & 64\% \\ | |
| 13 | + Random Forest & 0.677 & 17.9\% & 43\% & 64\% \\ | |
| 14 | + Neural network (plain MLP) & 0.711 & 16.1\% & 47\% & 69\% \\ | |
| 15 | + Neural network (robust ResNet) & 0.741 & 15.2\% & 49\% & 70\% \\ | |
| 16 | + Gradient boosting (HistGBM) & 0.743 & 15.4\% & 49\% & 70\% \\ | |
| 17 | + Blend (NN + GBM) & 0.760 & 14.7\% & 51\% & 72\% \\[4pt] | |
| 18 | + \multicolumn{5}{@{}l}{\textit{Panel B: \ + Vision (façade) features}} \\[3pt] | |
| 19 | + OLS (linear hedonic) & 0.686 & 16.9\% & 45\% & 67\% \\ | |
| 20 | + Random Forest & 0.704 & 17.2\% & 45\% & 66\% \\ | |
| 21 | + Neural network (robust ResNet) & 0.747 & 14.8\% & 51\% & 71\% \\ | |
| 22 | + Gradient boosting (HistGBM) & 0.770 & 14.5\% & 51\% & 71\% \\ | |
| 23 | + \textbf{Blend (NN + GBM)} & \textbf{0.779} & \textbf{14.0\%} & \textbf{53\%} & \textbf{73\%} \\ | |
| 24 | + \bottomrule | |
| 25 | + \end{tabular} | |
| 26 | + \begin{tablenotes}[flushleft] | |
| 27 | + \footnotesize | |
| 28 | + \item \textit{Notes:} Out-of-sample performance from 5-fold cross-validation with a | |
| 29 | + common seed (identical folds across rows). $R^2_{\text{OOS}}$ is on | |
| 30 | + $\ln(\text{Price})$; ``Median AE'' is the median absolute percentage error in | |
| 31 | + levels (Equation~\ref{eq:cvmetrics}). Panel~A uses structural, amenity, room-detail, | |
| 32 | + listing-signal, and LLM text-derived features; Panel~B adds the twelve façade | |
| 33 | + vision attributes. City, type, and source enter as one-hot indicators for the | |
| 34 | + linear and tree models and as learned embeddings for the neural networks. The tree | |
| 35 | + and neural models handle missing living area natively; OLS uses within-type median | |
| 36 | + imputation. The best model is the text\,+\,vision blend. | |
| 37 | + \end{tablenotes} | |
| 38 | + \end{threeparttable} | |
| 39 | +\end{table} | |
added
paper/tables/regression.tex
+82 −0
@@ -0,0 +1,82 @@ | ||
| 1 | +% Auto-generated (revision: clustered SEs) | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Hedonic Regression Results (Text + Vision Features)} | |
| 6 | + \label{tab:regression} | |
| 7 | + \begin{tabular}{@{}lcc@{}} | |
| 8 | + \toprule | |
| 9 | + & (1) Model~A & (2) Model~B \\ | |
| 10 | + & \textit{rich} & \textit{broad} \\ | |
| 11 | + \midrule | |
| 12 | + $\ln(\text{Living area})$ & 0.358$^{***}$ & \\ | |
| 13 | + & (0.012) & \\[3pt] | |
| 14 | + $\ln(\text{Lot area})$ & 0.027$^{***}$ & 0.040$^{***}$ \\ | |
| 15 | + & (0.006) & (0.005) \\[3pt] | |
| 16 | + Bedrooms & 0.014$^{***}$ & 0.032$^{***}$ \\ | |
| 17 | + & (0.005) & (0.003) \\[3pt] | |
| 18 | + Bathrooms & 0.073$^{***}$ & 0.113$^{***}$ \\ | |
| 19 | + & (0.007) & (0.007) \\[3pt] | |
| 20 | + Parking & 0.011$^{***}$ & 0.015$^{***}$ \\ | |
| 21 | + & (0.001) & (0.002) \\[3pt] | |
| 22 | + Age & $-$0.004$^{***}$ & $-$0.005$^{***}$ \\ | |
| 23 | + & (0.001) & (0.001) \\[3pt] | |
| 24 | + \multicolumn{3}{@{}l}{\textit{Amenities \& text (LLM)}} \\[2pt] | |
| 25 | + Central A/C & 0.053$^{***}$ & 0.020 \\ | |
| 26 | + & (0.018) & (0.018) \\[3pt] | |
| 27 | + Public transit & 0.090$^{**}$ & 0.058 \\ | |
| 28 | + & (0.046) & (0.039) \\[3pt] | |
| 29 | + Waterfront & 0.058$^{*}$ & 0.062$^{***}$ \\ | |
| 30 | + & (0.033) & (0.019) \\[3pt] | |
| 31 | + Condition (LLM, ord.) & 0.016$^{***}$ & 0.016$^{***}$ \\ | |
| 32 | + & (0.006) & (0.005) \\[3pt] | |
| 33 | + Reno.\ kitchen (LLM) & 0.026$^{***}$ & 0.027$^{***}$ \\ | |
| 34 | + & (0.007) & (0.010) \\[3pt] | |
| 35 | + Hardwood floor (LLM) & 0.024$^{**}$ & 0.030$^{***}$ \\ | |
| 36 | + & (0.009) & (0.011) \\[3pt] | |
| 37 | + New construction (LLM) & $-$0.000 & $-$0.016 \\ | |
| 38 | + & (0.013) & (0.014) \\[3pt] | |
| 39 | + Income property (LLM) & $-$0.019$^{**}$ & $-$0.013$^{*}$ \\ | |
| 40 | + & (0.009) & (0.008) \\[3pt] | |
| 41 | + \multicolumn{3}{@{}l}{\textit{Vision (façade)}} \\[2pt] | |
| 42 | + Curb appeal (1--10) & 0.023$^{***}$ & 0.016 \\ | |
| 43 | + & (0.009) & (0.013) \\[3pt] | |
| 44 | + Renovation level (ord.) & 0.024$^{**}$ & 0.020$^{**}$ \\ | |
| 45 | + & (0.010) & (0.009) \\[3pt] | |
| 46 | + Luxury (visual) & 0.051$^{***}$ & 0.062$^{***}$ \\ | |
| 47 | + & (0.012) & (0.009) \\[3pt] | |
| 48 | + Garage visible & 0.046$^{***}$ & 0.047$^{***}$ \\ | |
| 49 | + & (0.006) & (0.007) \\[3pt] | |
| 50 | + Brick façade & 0.033$^{***}$ & 0.044$^{***}$ \\ | |
| 51 | + & (0.009) & (0.011) \\[3pt] | |
| 52 | + Stone façade & 0.036$^{***}$ & 0.047$^{***}$ \\ | |
| 53 | + & (0.008) & (0.005) \\[3pt] | |
| 54 | + Vinyl façade & $-$0.045$^{***}$ & $-$0.062$^{***}$ \\ | |
| 55 | + & (0.015) & (0.012) \\[3pt] | |
| 56 | + Rural setting & $-$0.122$^{***}$ & $-$0.076$^{***}$ \\ | |
| 57 | + & (0.032) & (0.029) \\[3pt] | |
| 58 | + \midrule | |
| 59 | + City FE / Type FE & Yes & Yes \\ | |
| 60 | + Living area & Yes & No \\ | |
| 61 | + Observations & 9{,}446 & 19{,}123 \\ | |
| 62 | + $R^2$ & 0.755 & 0.725 \\ | |
| 63 | + Adjusted $R^2$ & 0.748 & 0.721 \\ | |
| 64 | + \bottomrule | |
| 65 | + \end{tabular} | |
| 66 | + \begin{tablenotes}[flushleft] | |
| 67 | + \footnotesize | |
| 68 | + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full | |
| 69 | + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample); | |
| 70 | + Model~B omits it (full sample). ``LLM'' variables are extracted from the | |
| 71 | + description text; ``Vision'' variables from the façade photograph. Curb appeal and | |
| 72 | + renovation level enter as scores; luxury, garage, and material/setting are | |
| 73 | + indicators. All specifications include city and property-type fixed effects and the | |
| 74 | + full amenity, room-detail, and listing-signal controls (only selected coefficients | |
| 75 | + shown). Standard errors clustered by municipality (195 clusters) in parentheses. | |
| 76 | + Heteroskedasticity-robust (HC3) standard errors are uniformly smaller and yield | |
| 77 | + the same significance pattern for the text- and vision-derived attributes, except | |
| 78 | + that curb appeal in Model~B is significant under HC3 but not under clustering. | |
| 79 | + $^{***}$, $^{**}$, $^{*}$: significance at 1\%, 5\%, 10\%. | |
| 80 | + \end{tablenotes} | |
| 81 | + \end{threeparttable} | |
| 82 | +\end{table} | |
added
paper/tables/robustness.tex
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +% Auto-generated | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Robustness: Blend Performance Across Subsamples} | |
| 6 | + \label{tab:robustness} | |
| 7 | + \begin{tabular}{@{}lcccc@{}} | |
| 8 | + \toprule | |
| 9 | + Subsample & $N$ & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% \\ | |
| 10 | + \midrule | |
| 11 | + \textit{Full sample} & 19{,}123 & 0.779 & 14.0\% & 53\% \\ | |
| 12 | + \quad Price tercile 1 (low) & 6{,}375 & n/a$^{\dagger}$ & 15.6\% & 49\% \\ | |
| 13 | + \quad Price tercile 2 (mid) & 6{,}412 & n/a$^{\dagger}$ & 11.0\% & 64\% \\ | |
| 14 | + \quad Price tercile 3 (high) & 6{,}336 & n/a$^{\dagger}$ & 16.4\% & 47\% \\ | |
| 15 | + \quad Realtor.ca & 15{,}356 & 0.802 & 14.0\% & 53\% \\ | |
| 16 | + \quad DuProprio & 3{,}767 & 0.579 & 14.0\% & 54\% \\ | |
| 17 | + \quad Measured area & 9{,}446 & 0.773 & 13.5\% & 55\% \\ | |
| 18 | + \quad Imputed area & 9{,}677 & 0.783 & 14.4\% & 52\% \\ | |
| 19 | + \bottomrule | |
| 20 | + \end{tabular} | |
| 21 | + \begin{tablenotes}[flushleft] | |
| 22 | + \footnotesize | |
| 23 | + \item \textit{Notes:} Out-of-sample performance of the text\,+\,vision blend | |
| 24 | + (Table~\ref{tab:models}, Panel~B) computed on subsamples of the same 5-fold | |
| 25 | + cross-validated predictions. Price terciles are defined on the actual list price. | |
| 26 | + ``Measured area'' are listings with a reported or room-reconstructed living area | |
| 27 | + (priced by Model~A); ``Imputed area'' listings are priced by the area-free model. | |
| 28 | + $^{\dagger}$Within a single price tercile the range of $\ln P$ is heavily | |
| 29 | + restricted, so $R^2$---which normalizes by within-group variance---is | |
| 30 | + uninformative; the median absolute error is the comparable accuracy metric across | |
| 31 | + terciles. | |
| 32 | + \end{tablenotes} | |
| 33 | + \end{threeparttable} | |
| 34 | +\end{table} | |
added
paper/tables/summary_stats.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% Auto-generated | |
| 2 | +\begin{table}[htbp] | |
| 3 | + \centering | |
| 4 | + \begin{threeparttable} | |
| 5 | + \caption{Summary Statistics} | |
| 6 | + \label{tab:summary_stats} | |
| 7 | + \begin{tabular}{@{}lcccccc@{}} | |
| 8 | + \toprule | |
| 9 | + Variable & $N$ & Mean & Std.\ Dev. & $p_{25}$ & Median & $p_{75}$ \\ | |
| 10 | + \midrule | |
| 11 | + Price (CAD) & 19{,}123 & 764{{,}}708 & 689{{,}}118 & 439{{,}}000 & 599{{,}}900 & 849{{,}}000 \\ | |
| 12 | + Living area (ft$^2$) & 9{,}446 & 1{{,}}730 & 990 & 1{{,}}128 & 1{{,}}470 & 2{{,}}026 \\ | |
| 13 | + Lot area (ft$^2$) & 19{,}123 & 71{{,}}429 & 309{{,}}209 & 6{{,}}000 & 10{{,}}686 & 34{{,}}978 \\ | |
| 14 | + Bedrooms & 19{,}123 & 3.5 & 1.2 & 3.0 & 3.0 & 4.0 \\ | |
| 15 | + Bathrooms & 19{,}123 & 1.8 & 0.8 & 1.0 & 2.0 & 2.0 \\ | |
| 16 | + Parking spaces & 19{,}123 & 4.0 & 4.4 & 0.0 & 3.0 & 6.0 \\ | |
| 17 | + Storeys & 19{,}123 & 1.6 & 1.6 & 1.0 & 2.0 & 2.0 \\ | |
| 18 | + Age (years) & 19{,}123 & 42.5 & 28.0 & 24.5 & 40.0 & 53.0 \\ | |
| 19 | + Rooms (Realtor) & 19{,}123 & 10.1 & 6.5 & 7.0 & 11.0 & 14.0 \\ | |
| 20 | + Photos & 19{,}123 & 42 & 18 & 30 & 40 & 51 \\ | |
| 21 | + \bottomrule | |
| 22 | + \end{tabular} | |
| 23 | + \begin{tablenotes}[flushleft] | |
| 24 | + \footnotesize | |
| 25 | + \item \textit{Notes:} Descriptive statistics for the cleaned sample of 19{,}123 single-family and townhouse listings (Realtor.ca and DuProprio), scraped on 3~July~2026. 80\% of listings originate from Realtor.ca. Living area is the reported area, or reconstructed from room dimensions where missing. Continuous variables are bounded by the outlier rules described in Section~\ref{sec:data}; price-per-square-foot is trimmed at the 0.5th and 99.5th percentiles. | |
| 26 | + \end{tablenotes} | |
| 27 | + \end{threeparttable} | |
| 28 | +\end{table} | |
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
added
requirements.txt
+6 −0
@@ -0,0 +1,6 @@ | ||
| 1 | +numpy>=1.26 | |
| 2 | +pandas>=2.0 | |
| 3 | +statsmodels>=0.14 | |
| 4 | +scikit-learn>=1.4 | |
| 5 | +matplotlib>=3.8 | |
| 6 | +torch>=2.2 | |
added
results/ablation_extras.json
+62 −0
@@ -0,0 +1,62 @@ | ||
| 1 | +{ | |
| 2 | + "L1": { | |
| 3 | + "nfeat": 24, | |
| 4 | + "ols": { | |
| 5 | + "r2": 0.5208807490583831, | |
| 6 | + "mdae": 19.311838552673066, | |
| 7 | + "w15": 40.46959159127752, | |
| 8 | + "w25": 60.90048632536735 | |
| 9 | + }, | |
| 10 | + "gbm": { | |
| 11 | + "r2": 0.6922092084886344, | |
| 12 | + "mdae": 17.160081786433278, | |
| 13 | + "w15": 44.70532866182084, | |
| 14 | + "w25": 65.6591538984469 | |
| 15 | + } | |
| 16 | + }, | |
| 17 | + "L2": { | |
| 18 | + "nfeat": 38, | |
| 19 | + "ols": { | |
| 20 | + "r2": 0.6193226824225797, | |
| 21 | + "mdae": 18.059509413013565, | |
| 22 | + "w15": 42.81754954766512, | |
| 23 | + "w25": 63.755686869215076 | |
| 24 | + }, | |
| 25 | + "gbm": { | |
| 26 | + "r2": 0.7288688475138769, | |
| 27 | + "mdae": 15.95094002848213, | |
| 28 | + "w15": 47.46640171521205, | |
| 29 | + "w25": 68.38885112168592 | |
| 30 | + } | |
| 31 | + }, | |
| 32 | + "L3": { | |
| 33 | + "nfeat": 56, | |
| 34 | + "ols": { | |
| 35 | + "r2": 0.6277548901386698, | |
| 36 | + "mdae": 17.877027384670708, | |
| 37 | + "w15": 42.92213564817236, | |
| 38 | + "w25": 64.46687235266432 | |
| 39 | + }, | |
| 40 | + "gbm": { | |
| 41 | + "r2": 0.7428875872402715, | |
| 42 | + "mdae": 15.421354827745482, | |
| 43 | + "w15": 48.84170893688229, | |
| 44 | + "w25": 69.91580818909166 | |
| 45 | + } | |
| 46 | + }, | |
| 47 | + "L4": { | |
| 48 | + "nfeat": 77, | |
| 49 | + "ols": { | |
| 50 | + "r2": 0.6862361284610743, | |
| 51 | + "mdae": 16.886250547842508, | |
| 52 | + "w15": 45.25440568948387, | |
| 53 | + "w25": 66.72070281859541 | |
| 54 | + }, | |
| 55 | + "gbm": { | |
| 56 | + "r2": 0.7695283346486282, | |
| 57 | + "mdae": 14.51330744579874, | |
| 58 | + "w15": 51.383151179208284, | |
| 59 | + "w25": 71.29634471578727 | |
| 60 | + } | |
| 61 | + } | |
| 62 | +} | |
| \ No newline at end of file | ||
added
results/hedonic_summary.txt
+536 −0
@@ -0,0 +1,536 @@ | ||
| 1 | +=== MODELE A (avec superficie) === | |
| 2 | + OLS Regression Results | |
| 3 | +============================================================================== | |
| 4 | +Dep. Variable: ln_price R-squared: 0.717 | |
| 5 | +Model: OLS Adj. R-squared: 0.709 | |
| 6 | +Method: Least Squares F-statistic: 93.16 | |
| 7 | +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00 | |
| 8 | +Time: 07:41:43 Log-Likelihood: -1866.7 | |
| 9 | +No. Observations: 9444 AIC: 4215. | |
| 10 | +Df Residuals: 9203 BIC: 5939. | |
| 11 | +Df Model: 240 | |
| 12 | +Covariance Type: HC3 | |
| 13 | +======================================================================================================================== | |
| 14 | + coef std err z P>|z| [0.025 0.975] | |
| 15 | +------------------------------------------------------------------------------------------------------------------------ | |
| 16 | +Intercept 9.7330 0.094 103.249 0.000 9.548 9.918 | |
| 17 | +C(source)[T.realtor] -0.2189 0.022 -10.109 0.000 -0.261 -0.176 | |
| 18 | +C(ptype_fe)[T.Country Home] 0.0045 0.051 0.087 0.930 -0.096 0.105 | |
| 19 | +C(ptype_fe)[T.Home] -0.0071 0.013 -0.525 0.599 -0.034 0.019 | |
| 20 | +C(ptype_fe)[T.House] -0.0307 0.012 -2.558 0.011 -0.054 -0.007 | |
| 21 | +C(ptype_fe)[T.Intergenerational] -0.0200 0.036 -0.558 0.577 -0.090 0.050 | |
| 22 | +C(ptype_fe)[T.Other] 0.2178 0.120 1.822 0.068 -0.016 0.452 | |
| 23 | +C(ptype_fe)[T.Row / Townhouse] -0.1882 0.016 -12.125 0.000 -0.219 -0.158 | |
| 24 | +C(ptype_fe)[T.Semi Detached] -0.0601 0.015 -3.893 0.000 -0.090 -0.030 | |
| 25 | +C(ptype_fe)[T.Townhouse] -0.0038 0.025 -0.151 0.880 -0.053 0.045 | |
| 26 | +C(city_fe)[T.acton vale] -0.0615 0.058 -1.065 0.287 -0.175 0.052 | |
| 27 | +C(city_fe)[T.alma] -0.0170 0.054 -0.317 0.751 -0.122 0.088 | |
| 28 | +C(city_fe)[T.amherst] -0.0746 0.076 -0.980 0.327 -0.224 0.075 | |
| 29 | +C(city_fe)[T.amos] -0.0308 0.063 -0.490 0.624 -0.154 0.092 | |
| 30 | +C(city_fe)[T.austin] 0.2786 0.091 3.061 0.002 0.100 0.457 | |
| 31 | +C(city_fe)[T.baie-comeau] -0.3656 0.083 -4.411 0.000 -0.528 -0.203 | |
| 32 | +C(city_fe)[T.baie-saint-paul] -0.0934 0.053 -1.765 0.078 -0.197 0.010 | |
| 33 | +C(city_fe)[T.beaconsfield] 0.6128 0.052 11.867 0.000 0.512 0.714 | |
| 34 | +C(city_fe)[T.beauport] 0.0424 0.037 1.149 0.250 -0.030 0.115 | |
| 35 | +C(city_fe)[T.beloeil] 0.2255 0.052 4.354 0.000 0.124 0.327 | |
| 36 | +C(city_fe)[T.blainville] 0.3095 0.030 10.218 0.000 0.250 0.369 | |
| 37 | +C(city_fe)[T.blue sea] -0.0272 0.060 -0.450 0.652 -0.146 0.091 | |
| 38 | +C(city_fe)[T.bois-des-filion] 0.1842 0.057 3.221 0.001 0.072 0.296 | |
| 39 | +C(city_fe)[T.boisbriand] 0.2661 0.039 6.854 0.000 0.190 0.342 | |
| 40 | +C(city_fe)[T.boischatel] 0.1368 0.082 1.664 0.096 -0.024 0.298 | |
| 41 | +C(city_fe)[T.boucherville] 0.3473 0.040 8.688 0.000 0.269 0.426 | |
| 42 | +C(city_fe)[T.bromont] 0.3122 0.037 8.391 0.000 0.239 0.385 | |
| 43 | +C(city_fe)[T.brossard] 0.4100 0.030 13.746 0.000 0.352 0.468 | |
| 44 | +C(city_fe)[T.bécancour] -0.0918 0.039 -2.323 0.020 -0.169 -0.014 | |
| 45 | +C(city_fe)[T.cantley] 0.1223 0.033 3.758 0.000 0.059 0.186 | |
| 46 | +C(city_fe)[T.carignan] 0.2488 0.086 2.881 0.004 0.080 0.418 | |
| 47 | +C(city_fe)[T.chandler] -0.2905 0.141 -2.068 0.039 -0.566 -0.015 | |
| 48 | +C(city_fe)[T.chateauguay] 0.2290 0.036 6.421 0.000 0.159 0.299 | |
| 49 | +C(city_fe)[T.chelsea] 0.2572 0.038 6.779 0.000 0.183 0.332 | |
| 50 | +C(city_fe)[T.chertsey] -0.0464 0.064 -0.730 0.466 -0.171 0.078 | |
| 51 | +C(city_fe)[T.chicoutimi] -0.1007 0.046 -2.203 0.028 -0.190 -0.011 | |
| 52 | +C(city_fe)[T.chomedey] 0.3195 0.050 6.430 0.000 0.222 0.417 | |
| 53 | +C(city_fe)[T.châteauguay] 0.2060 0.038 5.390 0.000 0.131 0.281 | |
| 54 | +C(city_fe)[T.contrecoeur] 0.0915 0.057 1.607 0.108 -0.020 0.203 | |
| 55 | +C(city_fe)[T.cowansville] 0.1177 0.050 2.376 0.017 0.021 0.215 | |
| 56 | +C(city_fe)[T.crabtree] 0.0241 0.096 0.251 0.802 -0.164 0.213 | |
| 57 | +C(city_fe)[T.côte-saint-luc] 0.6071 0.050 12.165 0.000 0.509 0.705 | |
| 58 | +C(city_fe)[T.dolbeau-mistassini] -0.2645 0.091 -2.905 0.004 -0.443 -0.086 | |
| 59 | +C(city_fe)[T.dollard-des-ormeaux] 0.4023 0.038 10.535 0.000 0.327 0.477 | |
| 60 | +C(city_fe)[T.donnacona] 0.2465 0.039 6.358 0.000 0.171 0.322 | |
| 61 | +C(city_fe)[T.dorval] 0.5515 0.055 10.101 0.000 0.445 0.659 | |
| 62 | +C(city_fe)[T.drummondville] -0.0196 0.022 -0.895 0.371 -0.062 0.023 | |
| 63 | +C(city_fe)[T.duhamel] 0.1373 0.076 1.815 0.069 -0.011 0.286 | |
| 64 | +C(city_fe)[T.dunham] 0.2726 0.064 4.237 0.000 0.147 0.399 | |
| 65 | +C(city_fe)[T.eastman] 0.3041 0.051 5.919 0.000 0.203 0.405 | |
| 66 | +C(city_fe)[T.entrelacs] 0.2617 0.135 1.945 0.052 -0.002 0.525 | |
| 67 | +C(city_fe)[T.farnham] 0.0545 0.062 0.885 0.376 -0.066 0.175 | |
| 68 | +C(city_fe)[T.fossambault-sur-le-lac] 0.2664 0.079 3.365 0.001 0.111 0.422 | |
| 69 | +C(city_fe)[T.gaspé] -0.2214 0.069 -3.228 0.001 -0.356 -0.087 | |
| 70 | +C(city_fe)[T.gatineau] 0.0901 0.012 7.233 0.000 0.066 0.114 | |
| 71 | +C(city_fe)[T.gore] 0.0786 0.104 0.753 0.451 -0.126 0.283 | |
| 72 | +C(city_fe)[T.granby] 0.1117 0.024 4.632 0.000 0.064 0.159 | |
| 73 | +C(city_fe)[T.grenville-sur-la-rouge] 0.1287 0.078 1.656 0.098 -0.024 0.281 | |
| 74 | +C(city_fe)[T.hampstead] 1.0843 0.057 19.163 0.000 0.973 1.195 | |
| 75 | +C(city_fe)[T.harrington] 0.1429 0.152 0.940 0.347 -0.155 0.441 | |
| 76 | +C(city_fe)[T.hudson] 0.2555 0.199 1.282 0.200 -0.135 0.646 | |
| 77 | +C(city_fe)[T.joliette] -0.0275 0.071 -0.387 0.699 -0.167 0.112 | |
| 78 | +C(city_fe)[T.jonquière] -0.1380 0.045 -3.096 0.002 -0.225 -0.051 | |
| 79 | +C(city_fe)[T.kirkland] 0.3686 0.037 9.967 0.000 0.296 0.441 | |
| 80 | +C(city_fe)[T.l'ange-gardien] 0.0142 0.158 0.090 0.928 -0.295 0.323 | |
| 81 | +C(city_fe)[T.l'assomption] 0.1479 0.060 2.460 0.014 0.030 0.266 | |
| 82 | +C(city_fe)[T.la conception] 0.3351 0.053 6.306 0.000 0.231 0.439 | |
| 83 | +C(city_fe)[T.la malbaie] -0.1347 0.063 -2.142 0.032 -0.258 -0.011 | |
| 84 | +C(city_fe)[T.la minerve] 0.0449 0.069 0.646 0.518 -0.091 0.181 | |
| 85 | +C(city_fe)[T.la prairie] 0.3741 0.035 10.542 0.000 0.305 0.444 | |
| 86 | +C(city_fe)[T.la pêche] 0.0354 0.037 0.968 0.333 -0.036 0.107 | |
| 87 | +C(city_fe)[T.la tuque] -0.3119 0.067 -4.627 0.000 -0.444 -0.180 | |
| 88 | +C(city_fe)[T.labelle] 0.1111 0.113 0.980 0.327 -0.111 0.333 | |
| 89 | +C(city_fe)[T.lac-beauport] 0.3609 0.094 3.852 0.000 0.177 0.545 | |
| 90 | +C(city_fe)[T.lac-brome] 0.5002 0.062 8.091 0.000 0.379 0.621 | |
| 91 | +C(city_fe)[T.lac-des-écorces] -0.0964 0.159 -0.607 0.544 -0.408 0.215 | |
| 92 | +C(city_fe)[T.lac-etchemin] -0.1890 0.135 -1.396 0.163 -0.454 0.076 | |
| 93 | +C(city_fe)[T.lachute] 0.1234 0.075 1.655 0.098 -0.023 0.269 | |
| 94 | +C(city_fe)[T.lanoraie] -0.0824 0.051 -1.623 0.105 -0.182 0.017 | |
| 95 | +C(city_fe)[T.laval] 0.3952 0.020 19.308 0.000 0.355 0.435 | |
| 96 | +C(city_fe)[T.lavaltrie] 0.1283 0.065 1.990 0.047 0.002 0.255 | |
| 97 | +C(city_fe)[T.les éboulements] 0.2007 0.080 2.516 0.012 0.044 0.357 | |
| 98 | +C(city_fe)[T.longueuil] 0.3160 0.020 16.113 0.000 0.278 0.354 | |
| 99 | +C(city_fe)[T.lorraine] 0.1128 0.113 0.995 0.320 -0.109 0.335 | |
| 100 | +C(city_fe)[T.lévis] 0.0663 0.027 2.486 0.013 0.014 0.119 | |
| 101 | +C(city_fe)[T.magog] 0.3570 0.034 10.628 0.000 0.291 0.423 | |
| 102 | +C(city_fe)[T.mandeville] -0.1781 0.080 -2.221 0.026 -0.335 -0.021 | |
| 103 | +C(city_fe)[T.mansfield-et-pontefract] -0.3813 0.079 -4.847 0.000 -0.536 -0.227 | |
| 104 | +C(city_fe)[T.marieville] 0.1717 0.105 1.634 0.102 -0.034 0.377 | |
| 105 | +C(city_fe)[T.mascouche] 0.1768 0.042 4.225 0.000 0.095 0.259 | |
| 106 | +C(city_fe)[T.matane] -0.2714 0.042 -6.488 0.000 -0.353 -0.189 | |
| 107 | +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3175 0.034 9.297 0.000 0.251 0.384 | |
| 108 | +C(city_fe)[T.messines] -0.1638 0.061 -2.701 0.007 -0.283 -0.045 | |
| 109 | +C(city_fe)[T.mille-isles] 0.2388 0.055 4.380 0.000 0.132 0.346 | |
| 110 | +C(city_fe)[T.mirabel] 0.2407 0.027 9.076 0.000 0.189 0.293 | |
| 111 | +C(city_fe)[T.mont-joli] -0.2476 0.177 -1.397 0.162 -0.595 0.100 | |
| 112 | +C(city_fe)[T.mont-laurier] -0.0494 0.038 -1.316 0.188 -0.123 0.024 | |
| 113 | +C(city_fe)[T.mont-royal] 1.0590 0.060 17.641 0.000 0.941 1.177 | |
| 114 | +C(city_fe)[T.mont-saint-hilaire] 0.3721 0.109 3.421 0.001 0.159 0.585 | |
| 115 | +C(city_fe)[T.mont-tremblant] 0.3943 0.040 9.855 0.000 0.316 0.473 | |
| 116 | +C(city_fe)[T.montréal] 0.6721 0.021 31.548 0.000 0.630 0.714 | |
| 117 | +C(city_fe)[T.morin-heights] 0.2972 0.057 5.204 0.000 0.185 0.409 | |
| 118 | +C(city_fe)[T.nominingue] 0.1336 0.105 1.276 0.202 -0.072 0.339 | |
| 119 | +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.2087 0.061 3.406 0.001 0.089 0.329 | |
| 120 | +C(city_fe)[T.notre-dame-de-pontmain] 0.0094 0.061 0.154 0.878 -0.111 0.130 | |
| 121 | +C(city_fe)[T.notre-dame-des-bois] -0.2375 0.074 -3.230 0.001 -0.382 -0.093 | |
| 122 | +C(city_fe)[T.notre-dame-des-prairies] 0.0822 0.118 0.697 0.486 -0.149 0.313 | |
| 123 | +C(city_fe)[T.notre-dame-du-laus] 0.1009 0.053 1.894 0.058 -0.003 0.205 | |
| 124 | +C(city_fe)[T.orford] 0.3230 0.063 5.122 0.000 0.199 0.447 | |
| 125 | +C(city_fe)[T.papineauville] -0.0886 0.124 -0.717 0.473 -0.331 0.153 | |
| 126 | +C(city_fe)[T.petite-rivière-saint-françois] -0.0343 0.084 -0.407 0.684 -0.200 0.131 | |
| 127 | +C(city_fe)[T.piedmont] 0.4643 0.067 6.959 0.000 0.334 0.595 | |
| 128 | +C(city_fe)[T.pincourt] 0.1811 0.055 3.304 0.001 0.074 0.289 | |
| 129 | +C(city_fe)[T.pointe-claire] 0.4251 0.038 11.133 0.000 0.350 0.500 | |
| 130 | +C(city_fe)[T.pont-rouge] 0.0628 0.056 1.115 0.265 -0.048 0.173 | |
| 131 | +C(city_fe)[T.prévost] 0.1183 0.037 3.231 0.001 0.047 0.190 | |
| 132 | +C(city_fe)[T.québec] 0.1687 0.020 8.570 0.000 0.130 0.207 | |
| 133 | +C(city_fe)[T.rawdon] 0.0579 0.047 1.244 0.213 -0.033 0.149 | |
| 134 | +C(city_fe)[T.repentigny] 0.1908 0.032 5.948 0.000 0.128 0.254 | |
| 135 | +C(city_fe)[T.rigaud] 0.2981 0.064 4.632 0.000 0.172 0.424 | |
| 136 | +C(city_fe)[T.rimouski] -0.0613 0.034 -1.824 0.068 -0.127 0.005 | |
| 137 | +C(city_fe)[T.rivière-du-loup] -0.1841 0.094 -1.952 0.051 -0.369 0.001 | |
| 138 | +C(city_fe)[T.rivière-rouge] -0.0045 0.071 -0.063 0.950 -0.143 0.135 | |
| 139 | +C(city_fe)[T.roberval] -0.2600 0.081 -3.206 0.001 -0.419 -0.101 | |
| 140 | +C(city_fe)[T.rosemère] 0.3854 0.089 4.317 0.000 0.210 0.560 | |
| 141 | +C(city_fe)[T.saguenay] -0.0722 0.044 -1.628 0.103 -0.159 0.015 | |
| 142 | +C(city_fe)[T.saint-adolphe-d'howard] 0.3369 0.054 6.269 0.000 0.232 0.442 | |
| 143 | +C(city_fe)[T.saint-amable] 0.3185 0.071 4.461 0.000 0.179 0.458 | |
| 144 | +C(city_fe)[T.saint-anicet] 0.1207 0.072 1.684 0.092 -0.020 0.261 | |
| 145 | +C(city_fe)[T.saint-basile-le-grand] 0.3421 0.093 3.686 0.000 0.160 0.524 | |
| 146 | +C(city_fe)[T.saint-bruno-de-montarville] 0.6408 0.073 8.723 0.000 0.497 0.785 | |
| 147 | +C(city_fe)[T.saint-calixte] 0.1561 0.090 1.730 0.084 -0.021 0.333 | |
| 148 | +C(city_fe)[T.saint-charles-borromée] 0.1371 0.163 0.839 0.401 -0.183 0.457 | |
| 149 | +C(city_fe)[T.saint-colomban] 0.2111 0.070 3.016 0.003 0.074 0.348 | |
| 150 | +C(city_fe)[T.saint-constant] 0.2154 0.052 4.115 0.000 0.113 0.318 | |
| 151 | +C(city_fe)[T.saint-donat] 0.2569 0.067 3.862 0.000 0.127 0.387 | |
| 152 | +C(city_fe)[T.saint-eustache] 0.4482 0.071 6.307 0.000 0.309 0.587 | |
| 153 | +C(city_fe)[T.saint-félicien] -0.2423 0.336 -0.721 0.471 -0.901 0.416 | |
| 154 | +C(city_fe)[T.saint-gabriel-de-brandon] -0.1880 0.080 -2.346 0.019 -0.345 -0.031 | |
| 155 | +C(city_fe)[T.saint-georges] -0.1023 0.063 -1.626 0.104 -0.226 0.021 | |
| 156 | +C(city_fe)[T.saint-hippolyte] 0.3575 0.082 4.378 0.000 0.197 0.517 | |
| 157 | +C(city_fe)[T.saint-honoré] -0.1282 0.072 -1.788 0.074 -0.269 0.012 | |
| 158 | +C(city_fe)[T.saint-hyacinthe] 0.1600 0.061 2.616 0.009 0.040 0.280 | |
| 159 | +C(city_fe)[T.saint-jean-de-matha] 0.1081 0.147 0.734 0.463 -0.181 0.397 | |
| 160 | +C(city_fe)[T.saint-jean-sur-richelieu] 0.1894 0.029 6.448 0.000 0.132 0.247 | |
| 161 | +C(city_fe)[T.saint-jérôme] 0.1558 0.039 4.037 0.000 0.080 0.231 | |
| 162 | +C(city_fe)[T.saint-lambert] 0.6472 0.071 9.070 0.000 0.507 0.787 | |
| 163 | +C(city_fe)[T.saint-lambert-de-lauzon] 0.0527 0.081 0.654 0.513 -0.105 0.210 | |
| 164 | +C(city_fe)[T.saint-lin/laurentides] 0.0940 0.120 0.783 0.433 -0.141 0.329 | |
| 165 | +C(city_fe)[T.saint-marc-des-carrières] -0.2484 0.047 -5.271 0.000 -0.341 -0.156 | |
| 166 | +C(city_fe)[T.saint-michel-des-saints] -0.2325 0.080 -2.906 0.004 -0.389 -0.076 | |
| 167 | +C(city_fe)[T.saint-paul-de-l'île-aux-noix] -0.1222 0.249 -0.491 0.623 -0.610 0.365 | |
| 168 | +C(city_fe)[T.saint-philippe] 0.3232 0.040 8.134 0.000 0.245 0.401 | |
| 169 | +C(city_fe)[T.saint-raymond] 0.1551 0.088 1.768 0.077 -0.017 0.327 | |
| 170 | +C(city_fe)[T.saint-sauveur] 0.2986 0.058 5.155 0.000 0.185 0.412 | |
| 171 | +C(city_fe)[T.saint-zotique] 0.4566 0.074 6.189 0.000 0.312 0.601 | |
| 172 | +C(city_fe)[T.saint-zénon] -0.3094 86.426 -0.004 0.997 -169.702 169.083 | |
| 173 | +C(city_fe)[T.sainte-adèle] 0.3608 0.078 4.614 0.000 0.208 0.514 | |
| 174 | +C(city_fe)[T.sainte-agathe-des-monts] 0.1222 0.092 1.334 0.182 -0.057 0.302 | |
| 175 | +C(city_fe)[T.sainte-anne-des-plaines] 0.8940 0.773 1.156 0.248 -0.622 2.409 | |
| 176 | +C(city_fe)[T.sainte-barbe] 0.1909 0.291 0.655 0.512 -0.380 0.762 | |
| 177 | +C(city_fe)[T.sainte-brigitte-de-laval] 0.1374 0.051 2.680 0.007 0.037 0.238 | |
| 178 | +C(city_fe)[T.sainte-béatrix] 0.0338 0.044 0.764 0.445 -0.053 0.121 | |
| 179 | +C(city_fe)[T.sainte-catherine] 0.3921 0.073 5.382 0.000 0.249 0.535 | |
| 180 | +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.2108 0.068 3.084 0.002 0.077 0.345 | |
| 181 | +C(city_fe)[T.sainte-julie] 0.3731 0.082 4.540 0.000 0.212 0.534 | |
| 182 | +C(city_fe)[T.sainte-julienne] 0.0598 0.076 0.787 0.431 -0.089 0.209 | |
| 183 | +C(city_fe)[T.sainte-lucie-des-laurentides] 0.3382 0.073 4.613 0.000 0.195 0.482 | |
| 184 | +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.1056 0.051 2.065 0.039 0.005 0.206 | |
| 185 | +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.2464 0.035 7.007 0.000 0.177 0.315 | |
| 186 | +C(city_fe)[T.sainte-sophie] 0.0938 0.043 2.179 0.029 0.009 0.178 | |
| 187 | +C(city_fe)[T.sainte-thérèse] 0.2976 0.087 3.428 0.001 0.127 0.468 | |
| 188 | +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.1963 0.064 -3.071 0.002 -0.322 -0.071 | |
| 189 | +C(city_fe)[T.salaberry-de-valleyfield] 0.0587 0.049 1.205 0.228 -0.037 0.154 | |
| 190 | +C(city_fe)[T.sept-îles] -0.1510 0.085 -1.774 0.076 -0.318 0.016 | |
| 191 | +C(city_fe)[T.shawinigan] -0.1856 0.053 -3.475 0.001 -0.290 -0.081 | |
| 192 | +C(city_fe)[T.shefford] 0.1276 0.089 1.437 0.151 -0.046 0.302 | |
| 193 | +C(city_fe)[T.sherbrooke] 0.0976 0.018 5.321 0.000 0.062 0.133 | |
| 194 | +C(city_fe)[T.sorel-tracy] -0.0858 0.046 -1.883 0.060 -0.175 0.003 | |
| 195 | +C(city_fe)[T.st-colomban] 0.2324 0.059 3.962 0.000 0.117 0.347 | |
| 196 | +C(city_fe)[T.st-hippolyte] 0.1484 0.057 2.614 0.009 0.037 0.260 | |
| 197 | +C(city_fe)[T.st-jean-sur-richelieu] 0.1674 0.033 5.005 0.000 0.102 0.233 | |
| 198 | +C(city_fe)[T.st-jérôme] 0.1660 0.032 5.221 0.000 0.104 0.228 | |
| 199 | +C(city_fe)[T.st-sauveur] 0.2742 0.053 5.170 0.000 0.170 0.378 | |
| 200 | +C(city_fe)[T.ste-adèle] 0.1521 0.071 2.155 0.031 0.014 0.290 | |
| 201 | +C(city_fe)[T.ste-julienne] 0.0571 0.043 1.321 0.187 -0.028 0.142 | |
| 202 | +C(city_fe)[T.sutton] 0.3865 0.068 5.668 0.000 0.253 0.520 | |
| 203 | +C(city_fe)[T.terrebonne] 0.2097 0.029 7.341 0.000 0.154 0.266 | |
| 204 | +C(city_fe)[T.thetford mines] -0.3094 0.118 -2.617 0.009 -0.541 -0.078 | |
| 205 | +C(city_fe)[T.trois-rivières] -0.0763 0.023 -3.372 0.001 -0.121 -0.032 | |
| 206 | +C(city_fe)[T.très-saint-rédempteur] 0.4880 0.032 15.400 0.000 0.426 0.550 | |
| 207 | +C(city_fe)[T.témiscouata-sur-le-lac] -0.2810 0.139 -2.018 0.044 -0.554 -0.008 | |
| 208 | +C(city_fe)[T.val-d'or] 0.0673 0.031 2.182 0.029 0.007 0.128 | |
| 209 | +C(city_fe)[T.val-david] 0.0033 0.100 0.033 0.973 -0.193 0.199 | |
| 210 | +C(city_fe)[T.val-des-monts] 0.1145 0.025 4.660 0.000 0.066 0.163 | |
| 211 | +C(city_fe)[T.val-morin] -0.1213 0.092 -1.311 0.190 -0.302 0.060 | |
| 212 | +C(city_fe)[T.vaudreuil-dorion] 0.0676 0.039 1.731 0.083 -0.009 0.144 | |
| 213 | +C(city_fe)[T.venise-en-québec] 0.0774 0.073 1.065 0.287 -0.065 0.220 | |
| 214 | +C(city_fe)[T.victoriaville] -0.0961 0.031 -3.150 0.002 -0.156 -0.036 | |
| 215 | +C(city_fe)[T.waterloo] 0.1009 0.051 1.969 0.049 0.000 0.201 | |
| 216 | +C(city_fe)[T.weedon] -0.1440 0.081 -1.780 0.075 -0.303 0.015 | |
| 217 | +C(city_fe)[T.wentworth-nord] 0.2091 0.088 2.378 0.017 0.037 0.381 | |
| 218 | +C(city_fe)[T.westmount] 1.3088 0.042 31.188 0.000 1.227 1.391 | |
| 219 | +ln_sqft 0.4171 0.013 31.566 0.000 0.391 0.443 | |
| 220 | +ln_lot 0.0257 0.004 6.727 0.000 0.018 0.033 | |
| 221 | +has_lot -0.0242 0.014 -1.683 0.092 -0.052 0.004 | |
| 222 | +beds 0.0022 0.004 0.513 0.608 -0.006 0.011 | |
| 223 | +baths 0.0897 0.007 13.613 0.000 0.077 0.103 | |
| 224 | +parking 0.0109 0.002 7.110 0.000 0.008 0.014 | |
| 225 | +storeys -0.0023 0.007 -0.312 0.755 -0.017 0.012 | |
| 226 | +age -0.0060 0.000 -12.824 0.000 -0.007 -0.005 | |
| 227 | +age2 2.97e-05 3.88e-06 7.650 0.000 2.21e-05 3.73e-05 | |
| 228 | +has_age -0.1804 0.021 -8.685 0.000 -0.221 -0.140 | |
| 229 | +pool 0.0341 0.008 4.444 0.000 0.019 0.049 | |
| 230 | +waterfront 0.0623 0.021 2.965 0.003 0.021 0.104 | |
| 231 | +garage 0.0279 0.007 3.953 0.000 0.014 0.042 | |
| 232 | +fireplace 0.0172 0.008 2.130 0.033 0.001 0.033 | |
| 233 | +central_air 0.1016 0.011 9.291 0.000 0.080 0.123 | |
| 234 | +finished_bsmt -0.0115 0.010 -1.191 0.234 -0.030 0.007 | |
| 235 | +near_water 0.0972 0.023 4.149 0.000 0.051 0.143 | |
| 236 | +cul_de_sac 0.0279 0.012 2.409 0.016 0.005 0.051 | |
| 237 | +golf 0.0225 0.007 3.138 0.002 0.008 0.037 | |
| 238 | +ski 0.0218 0.013 1.649 0.099 -0.004 0.048 | |
| 239 | +near_park 0.0037 0.011 0.328 0.743 -0.019 0.026 | |
| 240 | +near_school -0.0695 0.011 -6.157 0.000 -0.092 -0.047 | |
| 241 | +near_highway 0.0514 0.008 6.362 0.000 0.036 0.067 | |
| 242 | +transit 0.1011 0.009 11.372 0.000 0.084 0.119 | |
| 243 | +no_rear_neighbor -0.0377 0.010 -3.802 0.000 -0.057 -0.018 | |
| 244 | +n_images_c 0.0045 0.000 14.507 0.000 0.004 0.005 | |
| 245 | +virtual_tour 0.0444 0.022 1.998 0.046 0.001 0.088 | |
| 246 | +n_rooms 0.0172 0.002 9.260 0.000 0.014 0.021 | |
| 247 | +n_levels_r -0.0248 0.009 -2.825 0.005 -0.042 -0.008 | |
| 248 | +has_primary_bed 0.0775 0.016 4.958 0.000 0.047 0.108 | |
| 249 | +has_basement -0.0251 0.015 -1.714 0.086 -0.054 0.004 | |
| 250 | +r_office 0.0300 0.011 2.631 0.009 0.008 0.052 | |
| 251 | +r_family 0.0422 0.011 3.973 0.000 0.021 0.063 | |
| 252 | +r_workshop -0.0401 0.014 -2.897 0.004 -0.067 -0.013 | |
| 253 | +r_winecellar 0.1086 0.036 3.034 0.002 0.038 0.179 | |
| 254 | +r_sunroom -0.0042 0.021 -0.201 0.840 -0.046 0.037 | |
| 255 | +r_mezzanine 0.0136 0.026 0.527 0.598 -0.037 0.064 | |
| 256 | +r_library 0.0945 0.084 1.123 0.262 -0.070 0.260 | |
| 257 | +r_laundry 0.0202 0.009 2.222 0.026 0.002 0.038 | |
| 258 | +============================================================================== | |
| 259 | +Omnibus: 403.871 Durbin-Watson: 1.849 | |
| 260 | +Prob(Omnibus): 0.000 Jarque-Bera (JB): 1350.435 | |
| 261 | +Skew: 0.024 Prob(JB): 5.71e-294 | |
| 262 | +Kurtosis: 4.852 Cond. No. 5.36e+16 | |
| 263 | +============================================================================== | |
| 264 | + | |
| 265 | +Notes: | |
| 266 | +[1] Standard Errors are heteroscedasticity robust (HC3) | |
| 267 | +[2] The smallest eigenvalue is 5.8e-23. This might indicate that there are | |
| 268 | +strong multicollinearity problems or that the design matrix is singular. | |
| 269 | + | |
| 270 | +=== MODELE B (sans superficie) === | |
| 271 | + OLS Regression Results | |
| 272 | +============================================================================== | |
| 273 | +Dep. Variable: ln_price R-squared: 0.677 | |
| 274 | +Model: OLS Adj. R-squared: 0.673 | |
| 275 | +Method: Least Squares F-statistic: 149.6 | |
| 276 | +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00 | |
| 277 | +Time: 07:41:43 Log-Likelihood: -6293.3 | |
| 278 | +No. Observations: 19115 AIC: 1.307e+04 | |
| 279 | +Df Residuals: 18875 BIC: 1.495e+04 | |
| 280 | +Df Model: 239 | |
| 281 | +Covariance Type: HC3 | |
| 282 | +======================================================================================================================== | |
| 283 | + coef std err z P>|z| [0.025 0.975] | |
| 284 | +------------------------------------------------------------------------------------------------------------------------ | |
| 285 | +Intercept 12.4073 0.115 108.355 0.000 12.183 12.632 | |
| 286 | +C(source)[T.realtor] -0.3680 0.021 -17.300 0.000 -0.410 -0.326 | |
| 287 | +C(ptype_fe)[T.Country Home] 0.0513 0.067 0.765 0.444 -0.080 0.183 | |
| 288 | +C(ptype_fe)[T.Home] 0.1075 0.083 1.297 0.195 -0.055 0.270 | |
| 289 | +C(ptype_fe)[T.House] -0.0659 0.014 -4.694 0.000 -0.093 -0.038 | |
| 290 | +C(ptype_fe)[T.Intergenerational] 0.1281 0.062 2.051 0.040 0.006 0.250 | |
| 291 | +C(ptype_fe)[T.Other] 0.2713 0.121 2.243 0.025 0.034 0.508 | |
| 292 | +C(ptype_fe)[T.Row / Townhouse] -0.3021 0.014 -21.062 0.000 -0.330 -0.274 | |
| 293 | +C(ptype_fe)[T.Semi Detached] -0.0065 0.057 -0.115 0.909 -0.118 0.105 | |
| 294 | +C(ptype_fe)[T.Townhouse] 0.0963 0.100 0.965 0.335 -0.099 0.292 | |
| 295 | +C(city_fe)[T.acton vale] -0.0167 0.033 -0.499 0.618 -0.082 0.049 | |
| 296 | +C(city_fe)[T.alma] -0.0480 0.060 -0.796 0.426 -0.166 0.070 | |
| 297 | +C(city_fe)[T.amherst] -0.0109 0.050 -0.217 0.828 -0.110 0.088 | |
| 298 | +C(city_fe)[T.amos] -0.1497 0.046 -3.239 0.001 -0.240 -0.059 | |
| 299 | +C(city_fe)[T.austin] 0.4089 0.071 5.776 0.000 0.270 0.548 | |
| 300 | +C(city_fe)[T.baie-comeau] -0.3409 0.071 -4.806 0.000 -0.480 -0.202 | |
| 301 | +C(city_fe)[T.baie-saint-paul] 0.0980 0.074 1.327 0.184 -0.047 0.243 | |
| 302 | +C(city_fe)[T.beaconsfield] 0.7818 0.039 20.221 0.000 0.706 0.858 | |
| 303 | +C(city_fe)[T.beauport] 0.0722 0.041 1.775 0.076 -0.008 0.152 | |
| 304 | +C(city_fe)[T.beloeil] 0.3670 0.049 7.506 0.000 0.271 0.463 | |
| 305 | +C(city_fe)[T.blainville] 0.4382 0.025 17.427 0.000 0.389 0.487 | |
| 306 | +C(city_fe)[T.blue sea] -0.1010 0.064 -1.585 0.113 -0.226 0.024 | |
| 307 | +C(city_fe)[T.bois-des-filion] 0.2889 0.048 5.999 0.000 0.194 0.383 | |
| 308 | +C(city_fe)[T.boisbriand] 0.3584 0.038 9.505 0.000 0.284 0.432 | |
| 309 | +C(city_fe)[T.boischatel] 0.4149 0.076 5.483 0.000 0.267 0.563 | |
| 310 | +C(city_fe)[T.boucherville] 0.4653 0.037 12.616 0.000 0.393 0.538 | |
| 311 | +C(city_fe)[T.bromont] 0.4068 0.034 11.856 0.000 0.340 0.474 | |
| 312 | +C(city_fe)[T.brossard] 0.5145 0.020 25.986 0.000 0.476 0.553 | |
| 313 | +C(city_fe)[T.bécancour] -0.0914 0.033 -2.788 0.005 -0.156 -0.027 | |
| 314 | +C(city_fe)[T.cantley] 0.1594 0.048 3.356 0.001 0.066 0.253 | |
| 315 | +C(city_fe)[T.carignan] 0.3871 0.041 9.472 0.000 0.307 0.467 | |
| 316 | +C(city_fe)[T.chandler] -0.3044 0.081 -3.746 0.000 -0.464 -0.145 | |
| 317 | +C(city_fe)[T.chateauguay] 0.2448 0.040 6.056 0.000 0.166 0.324 | |
| 318 | +C(city_fe)[T.chelsea] 0.2986 0.040 7.393 0.000 0.219 0.378 | |
| 319 | +C(city_fe)[T.chertsey] -0.0194 0.038 -0.508 0.612 -0.094 0.056 | |
| 320 | +C(city_fe)[T.chicoutimi] -0.0770 0.041 -1.880 0.060 -0.157 0.003 | |
| 321 | +C(city_fe)[T.chomedey] 0.4087 0.056 7.260 0.000 0.298 0.519 | |
| 322 | +C(city_fe)[T.châteauguay] 0.2797 0.023 12.337 0.000 0.235 0.324 | |
| 323 | +C(city_fe)[T.contrecoeur] 0.1257 0.034 3.670 0.000 0.059 0.193 | |
| 324 | +C(city_fe)[T.cowansville] 0.1495 0.036 4.133 0.000 0.079 0.220 | |
| 325 | +C(city_fe)[T.crabtree] 0.0639 0.046 1.394 0.163 -0.026 0.154 | |
| 326 | +C(city_fe)[T.côte-saint-luc] 0.7419 0.052 14.316 0.000 0.640 0.843 | |
| 327 | +C(city_fe)[T.dolbeau-mistassini] -0.1920 0.066 -2.895 0.004 -0.322 -0.062 | |
| 328 | +C(city_fe)[T.dollard-des-ormeaux] 0.5263 0.029 18.095 0.000 0.469 0.583 | |
| 329 | +C(city_fe)[T.donnacona] 0.1454 0.035 4.112 0.000 0.076 0.215 | |
| 330 | +C(city_fe)[T.dorval] 0.6752 0.050 13.510 0.000 0.577 0.773 | |
| 331 | +C(city_fe)[T.drummondville] 0.0561 0.019 2.952 0.003 0.019 0.093 | |
| 332 | +C(city_fe)[T.duhamel] 0.0808 0.077 1.049 0.294 -0.070 0.232 | |
| 333 | +C(city_fe)[T.dunham] 0.3488 0.063 5.542 0.000 0.225 0.472 | |
| 334 | +C(city_fe)[T.eastman] 0.3943 0.055 7.157 0.000 0.286 0.502 | |
| 335 | +C(city_fe)[T.entrelacs] 0.2226 0.086 2.601 0.009 0.055 0.390 | |
| 336 | +C(city_fe)[T.farnham] 0.0560 0.038 1.463 0.143 -0.019 0.131 | |
| 337 | +C(city_fe)[T.fossambault-sur-le-lac] 0.3607 0.067 5.395 0.000 0.230 0.492 | |
| 338 | +C(city_fe)[T.gaspé] -0.3110 0.062 -4.989 0.000 -0.433 -0.189 | |
| 339 | +C(city_fe)[T.gatineau] 0.1163 0.013 8.687 0.000 0.090 0.143 | |
| 340 | +C(city_fe)[T.gore] 0.0808 0.058 1.403 0.161 -0.032 0.194 | |
| 341 | +C(city_fe)[T.granby] 0.1488 0.019 7.962 0.000 0.112 0.185 | |
| 342 | +C(city_fe)[T.grenville-sur-la-rouge] 0.1281 0.046 2.801 0.005 0.038 0.218 | |
| 343 | +C(city_fe)[T.hampstead] 1.2858 0.051 25.203 0.000 1.186 1.386 | |
| 344 | +C(city_fe)[T.harrington] -0.0622 0.130 -0.478 0.633 -0.318 0.193 | |
| 345 | +C(city_fe)[T.hudson] 0.4820 0.071 6.816 0.000 0.343 0.621 | |
| 346 | +C(city_fe)[T.joliette] 0.1230 0.046 2.684 0.007 0.033 0.213 | |
| 347 | +C(city_fe)[T.jonquière] -0.1590 0.046 -3.470 0.001 -0.249 -0.069 | |
| 348 | +C(city_fe)[T.kirkland] 0.5347 0.035 15.112 0.000 0.465 0.604 | |
| 349 | +C(city_fe)[T.l'ange-gardien] 0.2734 0.073 3.767 0.000 0.131 0.416 | |
| 350 | +C(city_fe)[T.l'assomption] 0.1985 0.027 7.277 0.000 0.145 0.252 | |
| 351 | +C(city_fe)[T.la conception] 0.2903 0.052 5.600 0.000 0.189 0.392 | |
| 352 | +C(city_fe)[T.la malbaie] -0.1641 0.055 -2.997 0.003 -0.271 -0.057 | |
| 353 | +C(city_fe)[T.la minerve] 0.0741 0.054 1.378 0.168 -0.031 0.180 | |
| 354 | +C(city_fe)[T.la prairie] 0.4728 0.030 15.749 0.000 0.414 0.532 | |
| 355 | +C(city_fe)[T.la pêche] 0.0037 0.039 0.094 0.925 -0.073 0.081 | |
| 356 | +C(city_fe)[T.la tuque] -0.3610 0.049 -7.307 0.000 -0.458 -0.264 | |
| 357 | +C(city_fe)[T.labelle] 0.0955 0.057 1.687 0.092 -0.015 0.206 | |
| 358 | +C(city_fe)[T.lac-beauport] 0.4166 0.089 4.659 0.000 0.241 0.592 | |
| 359 | +C(city_fe)[T.lac-brome] 0.6311 0.057 11.018 0.000 0.519 0.743 | |
| 360 | +C(city_fe)[T.lac-des-écorces] 0.0133 0.068 0.196 0.845 -0.120 0.146 | |
| 361 | +C(city_fe)[T.lac-etchemin] -0.2591 0.057 -4.521 0.000 -0.371 -0.147 | |
| 362 | +C(city_fe)[T.lachute] 0.1603 0.038 4.268 0.000 0.087 0.234 | |
| 363 | +C(city_fe)[T.lanoraie] 0.1578 0.038 4.195 0.000 0.084 0.232 | |
| 364 | +C(city_fe)[T.laval] 0.4618 0.014 33.004 0.000 0.434 0.489 | |
| 365 | +C(city_fe)[T.lavaltrie] 0.1654 0.039 4.251 0.000 0.089 0.242 | |
| 366 | +C(city_fe)[T.les éboulements] 0.1792 0.064 2.798 0.005 0.054 0.305 | |
| 367 | +C(city_fe)[T.longueuil] 0.3736 0.016 23.052 0.000 0.342 0.405 | |
| 368 | +C(city_fe)[T.lorraine] 0.4812 0.056 8.517 0.000 0.370 0.592 | |
| 369 | +C(city_fe)[T.lévis] 0.1144 0.022 5.189 0.000 0.071 0.158 | |
| 370 | +C(city_fe)[T.magog] 0.3435 0.032 10.704 0.000 0.281 0.406 | |
| 371 | +C(city_fe)[T.mandeville] -0.0847 0.041 -2.084 0.037 -0.164 -0.005 | |
| 372 | +C(city_fe)[T.mansfield-et-pontefract] -0.5178 0.067 -7.737 0.000 -0.649 -0.387 | |
| 373 | +C(city_fe)[T.marieville] 0.2018 0.034 5.959 0.000 0.135 0.268 | |
| 374 | +C(city_fe)[T.mascouche] 0.2837 0.021 13.524 0.000 0.243 0.325 | |
| 375 | +C(city_fe)[T.matane] -0.1801 0.047 -3.836 0.000 -0.272 -0.088 | |
| 376 | +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3725 0.047 7.859 0.000 0.280 0.465 | |
| 377 | +C(city_fe)[T.messines] -0.2559 0.053 -4.805 0.000 -0.360 -0.152 | |
| 378 | +C(city_fe)[T.mille-isles] 0.1934 0.052 3.750 0.000 0.092 0.294 | |
| 379 | +C(city_fe)[T.mirabel] 0.2915 0.020 14.237 0.000 0.251 0.332 | |
| 380 | +C(city_fe)[T.mont-joli] -0.2792 0.099 -2.832 0.005 -0.472 -0.086 | |
| 381 | +C(city_fe)[T.mont-laurier] -0.0419 0.040 -1.042 0.298 -0.121 0.037 | |
| 382 | +C(city_fe)[T.mont-royal] 1.2687 0.054 23.406 0.000 1.162 1.375 | |
| 383 | +C(city_fe)[T.mont-saint-hilaire] 0.5053 0.041 12.197 0.000 0.424 0.586 | |
| 384 | +C(city_fe)[T.mont-tremblant] 0.3895 0.036 10.890 0.000 0.319 0.460 | |
| 385 | +C(city_fe)[T.montréal] 0.7168 0.017 42.732 0.000 0.684 0.750 | |
| 386 | +C(city_fe)[T.morin-heights] 0.2959 0.038 7.784 0.000 0.221 0.370 | |
| 387 | +C(city_fe)[T.nominingue] 0.0840 0.085 0.990 0.322 -0.082 0.250 | |
| 388 | +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.4132 0.043 9.698 0.000 0.330 0.497 | |
| 389 | +C(city_fe)[T.notre-dame-de-pontmain] -0.1086 0.060 -1.801 0.072 -0.227 0.010 | |
| 390 | +C(city_fe)[T.notre-dame-des-bois] -0.3513 0.066 -5.303 0.000 -0.481 -0.221 | |
| 391 | +C(city_fe)[T.notre-dame-des-prairies] 0.0815 0.039 2.105 0.035 0.006 0.157 | |
| 392 | +C(city_fe)[T.notre-dame-du-laus] 0.0340 0.057 0.597 0.550 -0.078 0.146 | |
| 393 | +C(city_fe)[T.orford] 0.3442 0.065 5.321 0.000 0.217 0.471 | |
| 394 | +C(city_fe)[T.papineauville] -0.0271 0.113 -0.241 0.810 -0.248 0.194 | |
| 395 | +C(city_fe)[T.petite-rivière-saint-françois] -0.0295 0.054 -0.544 0.587 -0.136 0.077 | |
| 396 | +C(city_fe)[T.piedmont] 0.3505 0.055 6.383 0.000 0.243 0.458 | |
| 397 | +C(city_fe)[T.pincourt] 0.3096 0.038 8.225 0.000 0.236 0.383 | |
| 398 | +C(city_fe)[T.pointe-claire] 0.6052 0.042 14.568 0.000 0.524 0.687 | |
| 399 | +C(city_fe)[T.pont-rouge] -0.0087 0.045 -0.196 0.845 -0.096 0.079 | |
| 400 | +C(city_fe)[T.prévost] 0.1693 0.027 6.376 0.000 0.117 0.221 | |
| 401 | +C(city_fe)[T.québec] 0.1918 0.018 10.706 0.000 0.157 0.227 | |
| 402 | +C(city_fe)[T.rawdon] 0.0432 0.031 1.373 0.170 -0.018 0.105 | |
| 403 | +C(city_fe)[T.repentigny] 0.2854 0.019 14.912 0.000 0.248 0.323 | |
| 404 | +C(city_fe)[T.rigaud] 0.1455 0.046 3.193 0.001 0.056 0.235 | |
| 405 | +C(city_fe)[T.rimouski] -0.0705 0.033 -2.115 0.034 -0.136 -0.005 | |
| 406 | +C(city_fe)[T.rivière-du-loup] -0.0883 0.056 -1.585 0.113 -0.197 0.021 | |
| 407 | +C(city_fe)[T.rivière-rouge] -0.1303 0.058 -2.240 0.025 -0.244 -0.016 | |
| 408 | +C(city_fe)[T.roberval] -0.1680 0.067 -2.506 0.012 -0.299 -0.037 | |
| 409 | +C(city_fe)[T.rosemère] 0.4821 0.039 12.318 0.000 0.405 0.559 | |
| 410 | +C(city_fe)[T.saguenay] -0.0384 0.023 -1.664 0.096 -0.084 0.007 | |
| 411 | +C(city_fe)[T.saint-adolphe-d'howard] 0.1647 0.045 3.665 0.000 0.077 0.253 | |
| 412 | +C(city_fe)[T.saint-amable] 0.2714 0.033 8.280 0.000 0.207 0.336 | |
| 413 | +C(city_fe)[T.saint-anicet] 0.2760 0.070 3.950 0.000 0.139 0.413 | |
| 414 | +C(city_fe)[T.saint-basile-le-grand] 0.3655 0.044 8.222 0.000 0.278 0.453 | |
| 415 | +C(city_fe)[T.saint-bruno-de-montarville] 0.6120 0.039 15.504 0.000 0.535 0.689 | |
| 416 | +C(city_fe)[T.saint-calixte] 0.0123 0.033 0.367 0.714 -0.053 0.078 | |
| 417 | +C(city_fe)[T.saint-charles-borromée] 0.1447 0.050 2.874 0.004 0.046 0.243 | |
| 418 | +C(city_fe)[T.saint-colomban] 0.1238 0.042 2.966 0.003 0.042 0.206 | |
| 419 | +C(city_fe)[T.saint-constant] 0.3327 0.030 11.057 0.000 0.274 0.392 | |
| 420 | +C(city_fe)[T.saint-donat] 0.3373 0.047 7.155 0.000 0.245 0.430 | |
| 421 | +C(city_fe)[T.saint-eustache] 0.3390 0.036 9.470 0.000 0.269 0.409 | |
| 422 | +C(city_fe)[T.saint-félicien] -0.0495 0.059 -0.832 0.405 -0.166 0.067 | |
| 423 | +C(city_fe)[T.saint-gabriel-de-brandon] 0.0228 0.061 0.371 0.711 -0.098 0.143 | |
| 424 | +C(city_fe)[T.saint-georges] -0.1857 0.038 -4.869 0.000 -0.260 -0.111 | |
| 425 | +C(city_fe)[T.saint-hippolyte] 0.2228 0.041 5.377 0.000 0.142 0.304 | |
| 426 | +C(city_fe)[T.saint-honoré] -0.0445 0.061 -0.728 0.466 -0.164 0.075 | |
| 427 | +C(city_fe)[T.saint-hyacinthe] 0.1802 0.034 5.373 0.000 0.114 0.246 | |
| 428 | +C(city_fe)[T.saint-jean-de-matha] 0.0361 0.068 0.530 0.596 -0.097 0.170 | |
| 429 | +C(city_fe)[T.saint-jean-sur-richelieu] 0.2700 0.020 13.718 0.000 0.231 0.309 | |
| 430 | +C(city_fe)[T.saint-jérôme] 0.1966 0.020 9.627 0.000 0.157 0.237 | |
| 431 | +C(city_fe)[T.saint-lambert] 0.8612 0.048 17.881 0.000 0.767 0.956 | |
| 432 | +C(city_fe)[T.saint-lambert-de-lauzon] 0.0811 0.049 1.641 0.101 -0.016 0.178 | |
| 433 | +C(city_fe)[T.saint-lin/laurentides] 0.0843 0.030 2.770 0.006 0.025 0.144 | |
| 434 | +C(city_fe)[T.saint-marc-des-carrières] -0.1463 0.030 -4.804 0.000 -0.206 -0.087 | |
| 435 | +C(city_fe)[T.saint-michel-des-saints] -0.0296 0.067 -0.444 0.657 -0.160 0.101 | |
| 436 | +C(city_fe)[T.saint-paul-de-l'île-aux-noix] 0.0415 0.102 0.407 0.684 -0.159 0.242 | |
| 437 | +C(city_fe)[T.saint-philippe] 0.4171 0.027 15.486 0.000 0.364 0.470 | |
| 438 | +C(city_fe)[T.saint-raymond] 0.0777 0.068 1.140 0.254 -0.056 0.211 | |
| 439 | +C(city_fe)[T.saint-sauveur] 0.2484 0.034 7.368 0.000 0.182 0.314 | |
| 440 | +C(city_fe)[T.saint-zotique] 0.2794 0.039 7.099 0.000 0.202 0.357 | |
| 441 | +C(city_fe)[T.saint-zénon] -0.0054 0.107 -0.050 0.960 -0.215 0.204 | |
| 442 | +C(city_fe)[T.sainte-adèle] 0.2888 0.049 5.952 0.000 0.194 0.384 | |
| 443 | +C(city_fe)[T.sainte-agathe-des-monts] 0.1307 0.052 2.523 0.012 0.029 0.232 | |
| 444 | +C(city_fe)[T.sainte-anne-des-plaines] 0.2630 0.059 4.451 0.000 0.147 0.379 | |
| 445 | +C(city_fe)[T.sainte-barbe] 0.4472 0.064 6.992 0.000 0.322 0.573 | |
| 446 | +C(city_fe)[T.sainte-brigitte-de-laval] 0.1144 0.040 2.844 0.004 0.036 0.193 | |
| 447 | +C(city_fe)[T.sainte-béatrix] -0.0542 0.044 -1.231 0.218 -0.141 0.032 | |
| 448 | +C(city_fe)[T.sainte-catherine] 0.3326 0.033 10.090 0.000 0.268 0.397 | |
| 449 | +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1289 0.061 2.125 0.034 0.010 0.248 | |
| 450 | +C(city_fe)[T.sainte-julie] 0.4505 0.054 8.316 0.000 0.344 0.557 | |
| 451 | +C(city_fe)[T.sainte-julienne] 0.0339 0.035 0.981 0.327 -0.034 0.102 | |
| 452 | +C(city_fe)[T.sainte-lucie-des-laurentides] 0.1581 0.039 4.057 0.000 0.082 0.234 | |
| 453 | +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0159 0.053 0.298 0.766 -0.089 0.121 | |
| 454 | +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.3085 0.023 13.594 0.000 0.264 0.353 | |
| 455 | +C(city_fe)[T.sainte-sophie] 0.0743 0.038 1.969 0.049 0.000 0.148 | |
| 456 | +C(city_fe)[T.sainte-thérèse] 0.3650 0.041 8.871 0.000 0.284 0.446 | |
| 457 | +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2224 0.060 -3.680 0.000 -0.341 -0.104 | |
| 458 | +C(city_fe)[T.salaberry-de-valleyfield] 0.0989 0.045 2.196 0.028 0.011 0.187 | |
| 459 | +C(city_fe)[T.sept-îles] -0.1031 0.037 -2.808 0.005 -0.175 -0.031 | |
| 460 | +C(city_fe)[T.shawinigan] -0.1030 0.026 -4.024 0.000 -0.153 -0.053 | |
| 461 | +C(city_fe)[T.shefford] 0.2575 0.059 4.350 0.000 0.141 0.373 | |
| 462 | +C(city_fe)[T.sherbrooke] 0.0970 0.017 5.655 0.000 0.063 0.131 | |
| 463 | +C(city_fe)[T.sorel-tracy] 0.0076 0.027 0.280 0.780 -0.045 0.061 | |
| 464 | +C(city_fe)[T.st-colomban] 0.2945 0.071 4.138 0.000 0.155 0.434 | |
| 465 | +C(city_fe)[T.st-hippolyte] 0.1163 0.079 1.468 0.142 -0.039 0.272 | |
| 466 | +C(city_fe)[T.st-jean-sur-richelieu] 0.2196 0.035 6.245 0.000 0.151 0.289 | |
| 467 | +C(city_fe)[T.st-jérôme] 0.2480 0.039 6.409 0.000 0.172 0.324 | |
| 468 | +C(city_fe)[T.st-sauveur] 0.4105 0.091 4.505 0.000 0.232 0.589 | |
| 469 | +C(city_fe)[T.ste-adèle] 0.2046 0.066 3.088 0.002 0.075 0.335 | |
| 470 | +C(city_fe)[T.ste-julienne] 0.0905 0.048 1.903 0.057 -0.003 0.184 | |
| 471 | +C(city_fe)[T.sutton] 0.4707 0.053 8.949 0.000 0.368 0.574 | |
| 472 | +C(city_fe)[T.terrebonne] 0.2919 0.018 16.339 0.000 0.257 0.327 | |
| 473 | +C(city_fe)[T.thetford mines] -0.2933 0.058 -5.069 0.000 -0.407 -0.180 | |
| 474 | +C(city_fe)[T.trois-rivières] 0.0072 0.020 0.357 0.721 -0.032 0.047 | |
| 475 | +C(city_fe)[T.très-saint-rédempteur] 0.5979 0.049 12.126 0.000 0.501 0.695 | |
| 476 | +C(city_fe)[T.témiscouata-sur-le-lac] -0.1648 0.074 -2.223 0.026 -0.310 -0.020 | |
| 477 | +C(city_fe)[T.val-d'or] 0.0067 0.030 0.226 0.821 -0.051 0.065 | |
| 478 | +C(city_fe)[T.val-david] 0.0280 0.051 0.551 0.582 -0.072 0.127 | |
| 479 | +C(city_fe)[T.val-des-monts] 0.0953 0.027 3.504 0.000 0.042 0.149 | |
| 480 | +C(city_fe)[T.val-morin] 0.0441 0.060 0.734 0.463 -0.074 0.162 | |
| 481 | +C(city_fe)[T.vaudreuil-dorion] 0.2448 0.026 9.301 0.000 0.193 0.296 | |
| 482 | +C(city_fe)[T.venise-en-québec] 0.1940 0.041 4.773 0.000 0.114 0.274 | |
| 483 | +C(city_fe)[T.victoriaville] -0.0391 0.025 -1.593 0.111 -0.087 0.009 | |
| 484 | +C(city_fe)[T.waterloo] 0.1392 0.045 3.091 0.002 0.051 0.227 | |
| 485 | +C(city_fe)[T.weedon] -0.1707 0.060 -2.823 0.005 -0.289 -0.052 | |
| 486 | +C(city_fe)[T.wentworth-nord] 0.0653 0.073 0.892 0.372 -0.078 0.209 | |
| 487 | +C(city_fe)[T.westmount] 1.6044 0.038 41.845 0.000 1.529 1.680 | |
| 488 | +ln_lot 0.0424 0.003 13.299 0.000 0.036 0.049 | |
| 489 | +has_lot -0.0854 0.014 -5.897 0.000 -0.114 -0.057 | |
| 490 | +beds 0.0237 0.004 5.676 0.000 0.016 0.032 | |
| 491 | +baths 0.1352 0.005 25.581 0.000 0.125 0.146 | |
| 492 | +parking 0.0154 0.002 8.288 0.000 0.012 0.019 | |
| 493 | +storeys 0.0020 0.097 0.021 0.983 -0.188 0.192 | |
| 494 | +age -0.0071 0.000 -18.661 0.000 -0.008 -0.006 | |
| 495 | +age2 3.559e-05 2.82e-06 12.637 0.000 3.01e-05 4.11e-05 | |
| 496 | +has_age -0.1532 0.018 -8.628 0.000 -0.188 -0.118 | |
| 497 | +pool 0.0548 0.007 8.093 0.000 0.042 0.068 | |
| 498 | +waterfront 0.1091 0.018 6.132 0.000 0.074 0.144 | |
| 499 | +garage 0.0533 0.006 9.016 0.000 0.042 0.065 | |
| 500 | +fireplace 0.0116 0.007 1.768 0.077 -0.001 0.024 | |
| 501 | +central_air 0.0651 0.009 7.208 0.000 0.047 0.083 | |
| 502 | +finished_bsmt -0.0169 0.007 -2.438 0.015 -0.031 -0.003 | |
| 503 | +near_water 0.0769 0.021 3.748 0.000 0.037 0.117 | |
| 504 | +cul_de_sac 0.0316 0.009 3.509 0.000 0.014 0.049 | |
| 505 | +golf 0.0197 0.006 3.395 0.001 0.008 0.031 | |
| 506 | +ski 0.0468 0.010 4.760 0.000 0.028 0.066 | |
| 507 | +near_park 0.0043 0.008 0.544 0.587 -0.011 0.020 | |
| 508 | +near_school -0.0571 0.009 -6.578 0.000 -0.074 -0.040 | |
| 509 | +near_highway 0.0499 0.006 7.721 0.000 0.037 0.063 | |
| 510 | +transit 0.0667 0.008 8.828 0.000 0.052 0.082 | |
| 511 | +no_rear_neighbor -0.0478 0.009 -5.132 0.000 -0.066 -0.030 | |
| 512 | +n_images_c 0.0059 0.000 19.406 0.000 0.005 0.006 | |
| 513 | +virtual_tour 0.0277 0.016 1.751 0.080 -0.003 0.059 | |
| 514 | +n_rooms 0.0262 0.001 17.890 0.000 0.023 0.029 | |
| 515 | +n_levels_r 0.0432 0.048 0.897 0.370 -0.051 0.138 | |
| 516 | +has_primary_bed 0.0696 0.010 7.033 0.000 0.050 0.089 | |
| 517 | +has_basement -0.1168 0.044 -2.657 0.008 -0.203 -0.031 | |
| 518 | +r_office 0.0348 0.008 4.562 0.000 0.020 0.050 | |
| 519 | +r_family 0.0372 0.009 4.122 0.000 0.020 0.055 | |
| 520 | +r_workshop -0.0346 0.009 -3.928 0.000 -0.052 -0.017 | |
| 521 | +r_winecellar 0.1413 0.026 5.384 0.000 0.090 0.193 | |
| 522 | +r_sunroom 0.0166 0.014 1.224 0.221 -0.010 0.043 | |
| 523 | +r_mezzanine 0.0202 0.018 1.114 0.265 -0.015 0.056 | |
| 524 | +r_library 0.1466 0.070 2.082 0.037 0.009 0.285 | |
| 525 | +r_laundry 0.0382 0.006 6.301 0.000 0.026 0.050 | |
| 526 | +============================================================================== | |
| 527 | +Omnibus: 2418.650 Durbin-Watson: 1.758 | |
| 528 | +Prob(Omnibus): 0.000 Jarque-Bera (JB): 15665.854 | |
| 529 | +Skew: 0.429 Prob(JB): 0.00 | |
| 530 | +Kurtosis: 7.351 Cond. No. 2.81e+16 | |
| 531 | +============================================================================== | |
| 532 | + | |
| 533 | +Notes: | |
| 534 | +[1] Standard Errors are heteroscedasticity robust (HC3) | |
| 535 | +[2] The smallest eigenvalue is 5.65e-22. This might indicate that there are | |
| 536 | +strong multicollinearity problems or that the design matrix is singular. | |
| \ No newline at end of file | ||
added
results/hedonic_v4_summary.txt
+576 −0
@@ -0,0 +1,576 @@ | ||
| 1 | +=== A+GPT === | |
| 2 | + OLS Regression Results | |
| 3 | +============================================================================== | |
| 4 | +Dep. Variable: ln_price R-squared: 0.725 | |
| 5 | +Model: OLS Adj. R-squared: 0.717 | |
| 6 | +Method: Least Squares F-statistic: 6.503e+04 | |
| 7 | +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00 | |
| 8 | +Time: 08:11:28 Log-Likelihood: -1724.8 | |
| 9 | +No. Observations: 9446 AIC: 3970. | |
| 10 | +Df Residuals: 9186 BIC: 5829. | |
| 11 | +Df Model: 259 | |
| 12 | +Covariance Type: HC3 | |
| 13 | +======================================================================================================================== | |
| 14 | + coef std err z P>|z| [0.025 0.975] | |
| 15 | +------------------------------------------------------------------------------------------------------------------------ | |
| 16 | +Intercept 4.8574 0.047 102.393 0.000 4.764 4.950 | |
| 17 | +C(source)[T.realtor] -0.2485 0.023 -10.988 0.000 -0.293 -0.204 | |
| 18 | +C(ptype_fe)[T.Country Home] 0.0088 0.051 0.174 0.862 -0.090 0.108 | |
| 19 | +C(ptype_fe)[T.Home] -0.0125 0.014 -0.905 0.365 -0.040 0.015 | |
| 20 | +C(ptype_fe)[T.House] -0.0462 0.012 -3.711 0.000 -0.071 -0.022 | |
| 21 | +C(ptype_fe)[T.Intergenerational] 0.0010 0.038 0.026 0.979 -0.073 0.075 | |
| 22 | +C(ptype_fe)[T.Other] 0.2105 0.120 1.749 0.080 -0.025 0.446 | |
| 23 | +C(ptype_fe)[T.Row / Townhouse] -0.2023 0.016 -12.633 0.000 -0.234 -0.171 | |
| 24 | +C(ptype_fe)[T.Semi Detached] -0.0788 0.016 -5.067 0.000 -0.109 -0.048 | |
| 25 | +C(ptype_fe)[T.Townhouse] -0.0203 0.026 -0.770 0.442 -0.072 0.031 | |
| 26 | +C(city_fe)[T.acton vale] -0.0447 0.060 -0.747 0.455 -0.162 0.072 | |
| 27 | +C(city_fe)[T.alma] -0.0355 0.050 -0.704 0.481 -0.134 0.063 | |
| 28 | +C(city_fe)[T.amherst] -0.0742 0.075 -0.983 0.325 -0.222 0.074 | |
| 29 | +C(city_fe)[T.amos] -0.0204 0.062 -0.327 0.744 -0.143 0.102 | |
| 30 | +C(city_fe)[T.austin] 0.2867 0.088 3.250 0.001 0.114 0.460 | |
| 31 | +C(city_fe)[T.baie-comeau] -0.3689 0.079 -4.688 0.000 -0.523 -0.215 | |
| 32 | +C(city_fe)[T.baie-saint-paul] -0.0794 0.062 -1.280 0.200 -0.201 0.042 | |
| 33 | +C(city_fe)[T.beaconsfield] 0.6183 0.050 12.411 0.000 0.521 0.716 | |
| 34 | +C(city_fe)[T.beauport] 0.0450 0.035 1.289 0.198 -0.023 0.113 | |
| 35 | +C(city_fe)[T.beloeil] 0.2256 0.052 4.347 0.000 0.124 0.327 | |
| 36 | +C(city_fe)[T.blainville] 0.3035 0.031 9.859 0.000 0.243 0.364 | |
| 37 | +C(city_fe)[T.blue sea] -0.0483 0.058 -0.837 0.403 -0.161 0.065 | |
| 38 | +C(city_fe)[T.bois-des-filion] 0.1607 0.035 4.566 0.000 0.092 0.230 | |
| 39 | +C(city_fe)[T.boisbriand] 0.2550 0.043 5.892 0.000 0.170 0.340 | |
| 40 | +C(city_fe)[T.boischatel] 0.1434 0.074 1.929 0.054 -0.002 0.289 | |
| 41 | +C(city_fe)[T.boucherville] 0.3369 0.039 8.695 0.000 0.261 0.413 | |
| 42 | +C(city_fe)[T.bromont] 0.3139 0.035 8.946 0.000 0.245 0.383 | |
| 43 | +C(city_fe)[T.brossard] 0.4180 0.030 13.774 0.000 0.359 0.477 | |
| 44 | +C(city_fe)[T.bécancour] -0.1107 0.039 -2.846 0.004 -0.187 -0.034 | |
| 45 | +C(city_fe)[T.cantley] 0.1315 0.033 4.040 0.000 0.068 0.195 | |
| 46 | +C(city_fe)[T.carignan] 0.2473 0.091 2.724 0.006 0.069 0.425 | |
| 47 | +C(city_fe)[T.chandler] -0.2870 0.136 -2.106 0.035 -0.554 -0.020 | |
| 48 | +C(city_fe)[T.chateauguay] 0.2320 0.037 6.320 0.000 0.160 0.304 | |
| 49 | +C(city_fe)[T.chelsea] 0.2610 0.038 6.931 0.000 0.187 0.335 | |
| 50 | +C(city_fe)[T.chertsey] -0.0870 0.060 -1.458 0.145 -0.204 0.030 | |
| 51 | +C(city_fe)[T.chicoutimi] -0.1131 0.042 -2.703 0.007 -0.195 -0.031 | |
| 52 | +C(city_fe)[T.chomedey] 0.3214 0.048 6.746 0.000 0.228 0.415 | |
| 53 | +C(city_fe)[T.châteauguay] 0.2100 0.040 5.244 0.000 0.132 0.289 | |
| 54 | +C(city_fe)[T.contrecoeur] 0.0760 0.059 1.284 0.199 -0.040 0.192 | |
| 55 | +C(city_fe)[T.cowansville] 0.1233 0.043 2.845 0.004 0.038 0.208 | |
| 56 | +C(city_fe)[T.crabtree] 0.0300 0.090 0.334 0.738 -0.146 0.206 | |
| 57 | +C(city_fe)[T.côte-saint-luc] 0.6067 0.051 11.909 0.000 0.507 0.707 | |
| 58 | +C(city_fe)[T.dolbeau-mistassini] -0.2804 0.086 -3.253 0.001 -0.449 -0.111 | |
| 59 | +C(city_fe)[T.dollard-des-ormeaux] 0.4198 0.039 10.848 0.000 0.344 0.496 | |
| 60 | +C(city_fe)[T.donnacona] 0.2326 0.039 5.989 0.000 0.156 0.309 | |
| 61 | +C(city_fe)[T.dorval] 0.5505 0.048 11.354 0.000 0.455 0.645 | |
| 62 | +C(city_fe)[T.drummondville] -0.0231 0.021 -1.108 0.268 -0.064 0.018 | |
| 63 | +C(city_fe)[T.duhamel] 0.1073 0.071 1.511 0.131 -0.032 0.246 | |
| 64 | +C(city_fe)[T.dunham] 0.2692 0.068 3.981 0.000 0.137 0.402 | |
| 65 | +C(city_fe)[T.eastman] 0.2749 0.055 5.019 0.000 0.168 0.382 | |
| 66 | +C(city_fe)[T.entrelacs] 0.2622 0.126 2.080 0.037 0.015 0.509 | |
| 67 | +C(city_fe)[T.farnham] 0.0785 0.058 1.343 0.179 -0.036 0.193 | |
| 68 | +C(city_fe)[T.fossambault-sur-le-lac] 0.2222 0.078 2.844 0.004 0.069 0.375 | |
| 69 | +C(city_fe)[T.gaspé] -0.2403 0.071 -3.372 0.001 -0.380 -0.101 | |
| 70 | +C(city_fe)[T.gatineau] 0.1031 0.013 8.192 0.000 0.078 0.128 | |
| 71 | +C(city_fe)[T.gore] 0.0422 0.099 0.425 0.671 -0.152 0.237 | |
| 72 | +C(city_fe)[T.granby] 0.1272 0.024 5.254 0.000 0.080 0.175 | |
| 73 | +C(city_fe)[T.grenville-sur-la-rouge] 0.1171 0.082 1.433 0.152 -0.043 0.277 | |
| 74 | +C(city_fe)[T.hampstead] 1.0869 0.057 19.051 0.000 0.975 1.199 | |
| 75 | +C(city_fe)[T.harrington] 0.1032 0.145 0.713 0.476 -0.180 0.387 | |
| 76 | +C(city_fe)[T.hudson] 0.2472 0.179 1.383 0.167 -0.103 0.598 | |
| 77 | +C(city_fe)[T.joliette] -0.0242 0.070 -0.346 0.729 -0.161 0.113 | |
| 78 | +C(city_fe)[T.jonquière] -0.1325 0.044 -3.001 0.003 -0.219 -0.046 | |
| 79 | +C(city_fe)[T.kirkland] 0.3831 0.038 10.128 0.000 0.309 0.457 | |
| 80 | +C(city_fe)[T.l'ange-gardien] 0.0259 0.163 0.159 0.874 -0.294 0.346 | |
| 81 | +C(city_fe)[T.l'assomption] 0.1521 0.063 2.426 0.015 0.029 0.275 | |
| 82 | +C(city_fe)[T.l'île-perrot] 0.4022 0.061 6.630 0.000 0.283 0.521 | |
| 83 | +C(city_fe)[T.la conception] 0.3810 0.055 6.871 0.000 0.272 0.490 | |
| 84 | +C(city_fe)[T.la malbaie] -0.1254 0.063 -1.980 0.048 -0.250 -0.001 | |
| 85 | +C(city_fe)[T.la minerve] 0.0403 0.069 0.585 0.559 -0.095 0.175 | |
| 86 | +C(city_fe)[T.la prairie] 0.3665 0.036 10.091 0.000 0.295 0.438 | |
| 87 | +C(city_fe)[T.la pêche] 0.0226 0.036 0.621 0.535 -0.049 0.094 | |
| 88 | +C(city_fe)[T.la tuque] -0.3102 0.065 -4.784 0.000 -0.437 -0.183 | |
| 89 | +C(city_fe)[T.labelle] 0.0982 0.113 0.868 0.386 -0.124 0.320 | |
| 90 | +C(city_fe)[T.lac-beauport] 0.3579 0.088 4.045 0.000 0.185 0.531 | |
| 91 | +C(city_fe)[T.lac-brome] 0.5121 0.063 8.090 0.000 0.388 0.636 | |
| 92 | +C(city_fe)[T.lac-des-écorces] -0.0499 0.154 -0.324 0.746 -0.352 0.252 | |
| 93 | +C(city_fe)[T.lac-etchemin] -0.2092 0.144 -1.457 0.145 -0.491 0.072 | |
| 94 | +C(city_fe)[T.lachute] 0.1368 0.073 1.878 0.060 -0.006 0.280 | |
| 95 | +C(city_fe)[T.lanoraie] -0.1106 0.057 -1.941 0.052 -0.222 0.001 | |
| 96 | +C(city_fe)[T.laval] 0.4064 0.020 19.826 0.000 0.366 0.447 | |
| 97 | +C(city_fe)[T.lavaltrie] 0.1330 0.058 2.301 0.021 0.020 0.246 | |
| 98 | +C(city_fe)[T.les éboulements] 0.1814 0.088 2.065 0.039 0.009 0.354 | |
| 99 | +C(city_fe)[T.longueuil] 0.3166 0.019 16.430 0.000 0.279 0.354 | |
| 100 | +C(city_fe)[T.lorraine] 0.1245 0.106 1.172 0.241 -0.084 0.333 | |
| 101 | +C(city_fe)[T.lévis] 0.0630 0.025 2.508 0.012 0.014 0.112 | |
| 102 | +C(city_fe)[T.magog] 0.3398 0.033 10.422 0.000 0.276 0.404 | |
| 103 | +C(city_fe)[T.mandeville] -0.1594 0.080 -1.980 0.048 -0.317 -0.002 | |
| 104 | +C(city_fe)[T.mansfield-et-pontefract] -0.4148 0.082 -5.049 0.000 -0.576 -0.254 | |
| 105 | +C(city_fe)[T.marieville] 0.1990 0.092 2.158 0.031 0.018 0.380 | |
| 106 | +C(city_fe)[T.mascouche] 0.1885 0.042 4.469 0.000 0.106 0.271 | |
| 107 | +C(city_fe)[T.matane] -0.2595 0.037 -6.937 0.000 -0.333 -0.186 | |
| 108 | +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3424 0.036 9.489 0.000 0.272 0.413 | |
| 109 | +C(city_fe)[T.messines] -0.1621 0.058 -2.784 0.005 -0.276 -0.048 | |
| 110 | +C(city_fe)[T.mille-isles] 0.2411 0.055 4.390 0.000 0.133 0.349 | |
| 111 | +C(city_fe)[T.mirabel] 0.2446 0.027 8.899 0.000 0.191 0.298 | |
| 112 | +C(city_fe)[T.mont-joli] -0.2369 0.183 -1.292 0.196 -0.596 0.123 | |
| 113 | +C(city_fe)[T.mont-laurier] -0.0547 0.037 -1.490 0.136 -0.127 0.017 | |
| 114 | +C(city_fe)[T.mont-royal] 1.0680 0.061 17.561 0.000 0.949 1.187 | |
| 115 | +C(city_fe)[T.mont-saint-hilaire] 0.3694 0.102 3.633 0.000 0.170 0.569 | |
| 116 | +C(city_fe)[T.mont-tremblant] 0.4024 0.040 10.104 0.000 0.324 0.480 | |
| 117 | +C(city_fe)[T.montréal] 0.6861 0.021 32.059 0.000 0.644 0.728 | |
| 118 | +C(city_fe)[T.morin-heights] 0.3081 0.058 5.336 0.000 0.195 0.421 | |
| 119 | +C(city_fe)[T.nominingue] 0.1255 0.095 1.322 0.186 -0.061 0.311 | |
| 120 | +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.2077 0.066 3.151 0.002 0.079 0.337 | |
| 121 | +C(city_fe)[T.notre-dame-de-pontmain] 0.0126 0.061 0.206 0.837 -0.108 0.133 | |
| 122 | +C(city_fe)[T.notre-dame-des-bois] -0.1992 0.075 -2.650 0.008 -0.347 -0.052 | |
| 123 | +C(city_fe)[T.notre-dame-des-prairies] 0.0993 0.116 0.853 0.394 -0.129 0.327 | |
| 124 | +C(city_fe)[T.notre-dame-du-laus] 0.0915 0.053 1.742 0.081 -0.011 0.194 | |
| 125 | +C(city_fe)[T.orford] 0.3190 0.062 5.126 0.000 0.197 0.441 | |
| 126 | +C(city_fe)[T.papineauville] -0.0746 0.124 -0.602 0.547 -0.317 0.168 | |
| 127 | +C(city_fe)[T.petite-rivière-saint-françois] -0.0072 0.088 -0.083 0.934 -0.179 0.165 | |
| 128 | +C(city_fe)[T.piedmont] 0.4510 0.061 7.336 0.000 0.330 0.571 | |
| 129 | +C(city_fe)[T.pincourt] 0.2002 0.052 3.826 0.000 0.098 0.303 | |
| 130 | +C(city_fe)[T.pointe-claire] 0.4417 0.037 11.947 0.000 0.369 0.514 | |
| 131 | +C(city_fe)[T.pont-rouge] 0.1126 0.058 1.925 0.054 -0.002 0.227 | |
| 132 | +C(city_fe)[T.prévost] 0.1344 0.038 3.571 0.000 0.061 0.208 | |
| 133 | +C(city_fe)[T.québec] 0.1808 0.020 9.213 0.000 0.142 0.219 | |
| 134 | +C(city_fe)[T.rawdon] 0.0837 0.045 1.854 0.064 -0.005 0.172 | |
| 135 | +C(city_fe)[T.repentigny] 0.1814 0.033 5.527 0.000 0.117 0.246 | |
| 136 | +C(city_fe)[T.rigaud] 0.2917 0.064 4.548 0.000 0.166 0.417 | |
| 137 | +C(city_fe)[T.rimouski] -0.0524 0.036 -1.452 0.147 -0.123 0.018 | |
| 138 | +C(city_fe)[T.rivière-du-loup] -0.1793 0.089 -2.009 0.044 -0.354 -0.004 | |
| 139 | +C(city_fe)[T.rivière-rouge] -0.0220 0.071 -0.309 0.758 -0.161 0.118 | |
| 140 | +C(city_fe)[T.roberval] -0.2385 0.076 -3.129 0.002 -0.388 -0.089 | |
| 141 | +C(city_fe)[T.rosemère] 0.4051 0.088 4.628 0.000 0.234 0.577 | |
| 142 | +C(city_fe)[T.saguenay] -0.0684 0.044 -1.544 0.123 -0.155 0.018 | |
| 143 | +C(city_fe)[T.saint-adolphe-d'howard] 0.2906 0.054 5.420 0.000 0.186 0.396 | |
| 144 | +C(city_fe)[T.saint-amable] 0.2992 0.084 3.543 0.000 0.134 0.465 | |
| 145 | +C(city_fe)[T.saint-anicet] 0.0324 0.081 0.400 0.689 -0.126 0.191 | |
| 146 | +C(city_fe)[T.saint-basile-le-grand] 0.3769 0.091 4.138 0.000 0.198 0.555 | |
| 147 | +C(city_fe)[T.saint-bruno-de-montarville] 0.6511 0.071 9.114 0.000 0.511 0.791 | |
| 148 | +C(city_fe)[T.saint-calixte] 0.1676 0.088 1.904 0.057 -0.005 0.340 | |
| 149 | +C(city_fe)[T.saint-charles-borromée] 0.1796 0.174 1.034 0.301 -0.161 0.520 | |
| 150 | +C(city_fe)[T.saint-colomban] 0.2115 0.068 3.114 0.002 0.078 0.345 | |
| 151 | +C(city_fe)[T.saint-constant] 0.2084 0.056 3.745 0.000 0.099 0.317 | |
| 152 | +C(city_fe)[T.saint-donat] 0.2478 0.066 3.738 0.000 0.118 0.378 | |
| 153 | +C(city_fe)[T.saint-eustache] 0.4254 0.068 6.255 0.000 0.292 0.559 | |
| 154 | +C(city_fe)[T.saint-félicien] -0.1964 0.320 -0.613 0.540 -0.824 0.431 | |
| 155 | +C(city_fe)[T.saint-gabriel-de-brandon] -0.2016 0.100 -2.014 0.044 -0.398 -0.005 | |
| 156 | +C(city_fe)[T.saint-georges] -0.0958 0.067 -1.426 0.154 -0.228 0.036 | |
| 157 | +C(city_fe)[T.saint-hippolyte] 0.3319 0.079 4.193 0.000 0.177 0.487 | |
| 158 | +C(city_fe)[T.saint-honoré] -0.1112 0.075 -1.478 0.139 -0.259 0.036 | |
| 159 | +C(city_fe)[T.saint-hyacinthe] 0.1609 0.065 2.481 0.013 0.034 0.288 | |
| 160 | +C(city_fe)[T.saint-jean-de-matha] 0.0935 0.157 0.597 0.550 -0.213 0.400 | |
| 161 | +C(city_fe)[T.saint-jean-sur-richelieu] 0.1990 0.031 6.451 0.000 0.139 0.259 | |
| 162 | +C(city_fe)[T.saint-jérôme] 0.1716 0.037 4.677 0.000 0.100 0.243 | |
| 163 | +C(city_fe)[T.saint-lambert] 0.6578 0.069 9.598 0.000 0.523 0.792 | |
| 164 | +C(city_fe)[T.saint-lambert-de-lauzon] 0.0260 0.087 0.298 0.766 -0.145 0.197 | |
| 165 | +C(city_fe)[T.saint-lin/laurentides] 0.1117 0.124 0.898 0.369 -0.132 0.356 | |
| 166 | +C(city_fe)[T.saint-marc-des-carrières] -0.2794 0.048 -5.829 0.000 -0.373 -0.185 | |
| 167 | +C(city_fe)[T.saint-michel-des-saints] -0.2187 0.085 -2.572 0.010 -0.385 -0.052 | |
| 168 | +C(city_fe)[T.saint-paul-de-l'île-aux-noix] -0.1315 0.223 -0.591 0.555 -0.568 0.305 | |
| 169 | +C(city_fe)[T.saint-philippe] 0.3302 0.040 8.276 0.000 0.252 0.408 | |
| 170 | +C(city_fe)[T.saint-raymond] 0.1480 0.085 1.750 0.080 -0.018 0.314 | |
| 171 | +C(city_fe)[T.saint-sauveur] 0.3199 0.059 5.434 0.000 0.205 0.435 | |
| 172 | +C(city_fe)[T.saint-zotique] 0.4330 0.073 5.923 0.000 0.290 0.576 | |
| 173 | +C(city_fe)[T.saint-zénon] -0.2586 76.240 -0.003 0.997 -149.686 149.169 | |
| 174 | +C(city_fe)[T.sainte-adèle] 0.3672 0.077 4.781 0.000 0.217 0.518 | |
| 175 | +C(city_fe)[T.sainte-agathe-des-monts] 0.1184 0.092 1.291 0.197 -0.061 0.298 | |
| 176 | +C(city_fe)[T.sainte-anne-des-plaines] 0.9383 0.778 1.206 0.228 -0.586 2.463 | |
| 177 | +C(city_fe)[T.sainte-barbe] 0.1802 0.339 0.532 0.595 -0.483 0.844 | |
| 178 | +C(city_fe)[T.sainte-brigitte-de-laval] 0.1292 0.052 2.507 0.012 0.028 0.230 | |
| 179 | +C(city_fe)[T.sainte-béatrix] 0.0314 0.045 0.703 0.482 -0.056 0.119 | |
| 180 | +C(city_fe)[T.sainte-catherine] 0.4067 0.072 5.638 0.000 0.265 0.548 | |
| 181 | +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1900 0.066 2.877 0.004 0.061 0.319 | |
| 182 | +C(city_fe)[T.sainte-julie] 0.4254 0.085 4.980 0.000 0.258 0.593 | |
| 183 | +C(city_fe)[T.sainte-julienne] 0.0580 0.073 0.798 0.425 -0.084 0.200 | |
| 184 | +C(city_fe)[T.sainte-lucie-des-laurentides] 0.3427 0.064 5.320 0.000 0.216 0.469 | |
| 185 | +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0785 0.052 1.501 0.133 -0.024 0.181 | |
| 186 | +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.2436 0.036 6.705 0.000 0.172 0.315 | |
| 187 | +C(city_fe)[T.sainte-sophie] 0.1004 0.064 1.561 0.119 -0.026 0.227 | |
| 188 | +C(city_fe)[T.sainte-thérèse] 0.3162 0.085 3.721 0.000 0.150 0.483 | |
| 189 | +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2048 0.075 -2.730 0.006 -0.352 -0.058 | |
| 190 | +C(city_fe)[T.salaberry-de-valleyfield] 0.0382 0.046 0.826 0.409 -0.052 0.129 | |
| 191 | +C(city_fe)[T.sept-îles] -0.1273 0.079 -1.615 0.106 -0.282 0.027 | |
| 192 | +C(city_fe)[T.shawinigan] -0.1806 0.051 -3.520 0.000 -0.281 -0.080 | |
| 193 | +C(city_fe)[T.shefford] 0.1555 0.089 1.741 0.082 -0.020 0.331 | |
| 194 | +C(city_fe)[T.sherbrooke] 0.0978 0.018 5.524 0.000 0.063 0.133 | |
| 195 | +C(city_fe)[T.sorel-tracy] -0.1019 0.044 -2.339 0.019 -0.187 -0.017 | |
| 196 | +C(city_fe)[T.st-colomban] 0.2146 0.057 3.780 0.000 0.103 0.326 | |
| 197 | +C(city_fe)[T.st-hippolyte] 0.1331 0.055 2.435 0.015 0.026 0.240 | |
| 198 | +C(city_fe)[T.st-jean-sur-richelieu] 0.1735 0.032 5.503 0.000 0.112 0.235 | |
| 199 | +C(city_fe)[T.st-jérôme] 0.1551 0.031 5.014 0.000 0.094 0.216 | |
| 200 | +C(city_fe)[T.st-sauveur] 0.2612 0.049 5.298 0.000 0.165 0.358 | |
| 201 | +C(city_fe)[T.ste-adèle] 0.1396 0.069 2.012 0.044 0.004 0.276 | |
| 202 | +C(city_fe)[T.ste-julienne] 0.0377 0.035 1.083 0.279 -0.031 0.106 | |
| 203 | +C(city_fe)[T.sutton] 0.4078 0.066 6.196 0.000 0.279 0.537 | |
| 204 | +C(city_fe)[T.terrebonne] 0.1965 0.028 6.923 0.000 0.141 0.252 | |
| 205 | +C(city_fe)[T.thetford mines] -0.3029 0.117 -2.596 0.009 -0.532 -0.074 | |
| 206 | +C(city_fe)[T.trois-rivières] -0.0810 0.022 -3.706 0.000 -0.124 -0.038 | |
| 207 | +C(city_fe)[T.très-saint-rédempteur] 0.4839 0.032 15.354 0.000 0.422 0.546 | |
| 208 | +C(city_fe)[T.témiscouata-sur-le-lac] -0.3066 0.124 -2.473 0.013 -0.550 -0.064 | |
| 209 | +C(city_fe)[T.val-d'or] 0.0788 0.031 2.543 0.011 0.018 0.139 | |
| 210 | +C(city_fe)[T.val-david] 0.0115 0.117 0.099 0.921 -0.217 0.240 | |
| 211 | +C(city_fe)[T.val-des-monts] 0.1031 0.025 4.142 0.000 0.054 0.152 | |
| 212 | +C(city_fe)[T.val-morin] -0.1267 0.093 -1.357 0.175 -0.310 0.056 | |
| 213 | +C(city_fe)[T.vaudreuil-dorion] 0.0802 0.044 1.807 0.071 -0.007 0.167 | |
| 214 | +C(city_fe)[T.venise-en-québec] 0.0614 0.082 0.744 0.457 -0.100 0.223 | |
| 215 | +C(city_fe)[T.victoriaville] -0.0752 0.029 -2.593 0.010 -0.132 -0.018 | |
| 216 | +C(city_fe)[T.waterloo] 0.0905 0.056 1.608 0.108 -0.020 0.201 | |
| 217 | +C(city_fe)[T.weedon] -0.1348 0.080 -1.675 0.094 -0.293 0.023 | |
| 218 | +C(city_fe)[T.wentworth-nord] 0.1764 0.090 1.951 0.051 -0.001 0.354 | |
| 219 | +C(city_fe)[T.westmount] 1.3152 0.041 31.987 0.000 1.235 1.396 | |
| 220 | +ln_sqft 0.4106 0.013 30.648 0.000 0.384 0.437 | |
| 221 | +ln_lot 0.0253 0.004 6.506 0.000 0.018 0.033 | |
| 222 | +has_lot -0.0201 0.015 -1.369 0.171 -0.049 0.009 | |
| 223 | +beds 0.0057 0.004 1.361 0.173 -0.003 0.014 | |
| 224 | +baths 0.0867 0.007 13.252 0.000 0.074 0.099 | |
| 225 | +parking 0.0112 0.002 7.312 0.000 0.008 0.014 | |
| 226 | +storeys -0.0018 0.009 -0.199 0.843 -0.019 0.016 | |
| 227 | +age -0.0053 0.000 -11.284 0.000 -0.006 -0.004 | |
| 228 | +age2 2.612e-05 3.72e-06 7.024 0.000 1.88e-05 3.34e-05 | |
| 229 | +has_age -0.1623 0.022 -7.477 0.000 -0.205 -0.120 | |
| 230 | +pool 0.0059 0.011 0.526 0.599 -0.016 0.028 | |
| 231 | +waterfront 0.0166 0.022 0.751 0.452 -0.027 0.060 | |
| 232 | +garage 0.0025 0.021 0.119 0.905 -0.038 0.043 | |
| 233 | +fireplace -0.0611 0.016 -3.776 0.000 -0.093 -0.029 | |
| 234 | +central_air 0.0785 0.012 6.669 0.000 0.055 0.102 | |
| 235 | +finished_bsmt -0.0316 0.011 -2.757 0.006 -0.054 -0.009 | |
| 236 | +near_water 0.0910 0.024 3.842 0.000 0.045 0.137 | |
| 237 | +cul_de_sac 0.0268 0.012 2.317 0.020 0.004 0.050 | |
| 238 | +golf 0.0232 0.007 3.268 0.001 0.009 0.037 | |
| 239 | +ski 0.0139 0.013 1.066 0.286 -0.012 0.040 | |
| 240 | +near_park 0.0039 0.011 0.340 0.734 -0.018 0.026 | |
| 241 | +near_school -0.0637 0.011 -5.687 0.000 -0.086 -0.042 | |
| 242 | +near_highway 0.0475 0.008 5.929 0.000 0.032 0.063 | |
| 243 | +transit 0.1036 0.009 11.762 0.000 0.086 0.121 | |
| 244 | +no_rear_neighbor -0.0396 0.010 -4.038 0.000 -0.059 -0.020 | |
| 245 | +n_images_c 0.0044 0.000 14.018 0.000 0.004 0.005 | |
| 246 | +virtual_tour 0.0324 0.022 1.460 0.144 -0.011 0.076 | |
| 247 | +n_rooms 0.0170 0.002 9.265 0.000 0.013 0.021 | |
| 248 | +n_levels_r -0.0226 0.009 -2.526 0.012 -0.040 -0.005 | |
| 249 | +has_primary_bed 0.0753 0.016 4.839 0.000 0.045 0.106 | |
| 250 | +has_basement -0.0169 0.015 -1.148 0.251 -0.046 0.012 | |
| 251 | +r_office 0.0308 0.011 2.735 0.006 0.009 0.053 | |
| 252 | +r_family 0.0359 0.011 3.375 0.001 0.015 0.057 | |
| 253 | +r_workshop -0.0386 0.014 -2.828 0.005 -0.065 -0.012 | |
| 254 | +r_winecellar 0.1108 0.036 3.073 0.002 0.040 0.181 | |
| 255 | +r_sunroom 0.0007 0.021 0.033 0.974 -0.040 0.042 | |
| 256 | +r_mezzanine 0.0175 0.025 0.690 0.490 -0.032 0.067 | |
| 257 | +r_library 0.0971 0.083 1.171 0.242 -0.065 0.260 | |
| 258 | +r_laundry 0.0178 0.009 1.980 0.048 0.000 0.035 | |
| 259 | +g_cond 0.0115 0.004 2.567 0.010 0.003 0.020 | |
| 260 | +g_hardwood 0.0256 0.008 3.211 0.001 0.010 0.041 | |
| 261 | +g_quartz_granite 0.0186 0.011 1.771 0.077 -0.002 0.039 | |
| 262 | +g_view_water 0.0132 0.016 0.845 0.398 -0.017 0.044 | |
| 263 | +has_gpt 4.8574 0.047 102.393 0.000 4.764 4.950 | |
| 264 | +g_turnkey 0.0198 0.008 2.562 0.010 0.005 0.035 | |
| 265 | +g_recently_renovated 0.0050 0.010 0.513 0.608 -0.014 0.024 | |
| 266 | +g_garage 0.0235 0.021 1.131 0.258 -0.017 0.064 | |
| 267 | +g_pool 0.0492 0.012 4.204 0.000 0.026 0.072 | |
| 268 | +g_waterfront 0.0744 0.018 4.135 0.000 0.039 0.110 | |
| 269 | +g_finished_basement 0.0189 0.009 2.203 0.028 0.002 0.036 | |
| 270 | +g_fireplace 0.0907 0.016 5.618 0.000 0.059 0.122 | |
| 271 | +g_central_ac 0.0140 0.010 1.406 0.160 -0.006 0.034 | |
| 272 | +g_new_construction 0.0680 0.014 4.909 0.000 0.041 0.095 | |
| 273 | +g_income_property -0.0331 0.011 -3.083 0.002 -0.054 -0.012 | |
| 274 | +g_renovated_kitchen 0.0298 0.010 2.906 0.004 0.010 0.050 | |
| 275 | +g_renovated_bathroom -0.0060 0.010 -0.614 0.539 -0.025 0.013 | |
| 276 | +g_large_lot -0.0033 0.009 -0.370 0.711 -0.021 0.014 | |
| 277 | +g_quiet_area -0.0028 0.008 -0.369 0.712 -0.018 0.012 | |
| 278 | +============================================================================== | |
| 279 | +Omnibus: 426.401 Durbin-Watson: 1.855 | |
| 280 | +Prob(Omnibus): 0.000 Jarque-Bera (JB): 1443.982 | |
| 281 | +Skew: 0.076 Prob(JB): 0.00 | |
| 282 | +Kurtosis: 4.909 Cond. No. 2.08e+16 | |
| 283 | +============================================================================== | |
| 284 | + | |
| 285 | +Notes: | |
| 286 | +[1] Standard Errors are heteroscedasticity robust (HC3) | |
| 287 | +[2] The smallest eigenvalue is 3.84e-22. This might indicate that there are | |
| 288 | +strong multicollinearity problems or that the design matrix is singular. | |
| 289 | + | |
| 290 | +=== B+GPT === | |
| 291 | + OLS Regression Results | |
| 292 | +============================================================================== | |
| 293 | +Dep. Variable: ln_price R-squared: 0.684 | |
| 294 | +Model: OLS Adj. R-squared: 0.680 | |
| 295 | +Method: Least Squares F-statistic: 1.873e+05 | |
| 296 | +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00 | |
| 297 | +Time: 08:11:28 Log-Likelihood: -6089.2 | |
| 298 | +No. Observations: 19123 AIC: 1.270e+04 | |
| 299 | +Df Residuals: 18864 BIC: 1.473e+04 | |
| 300 | +Df Model: 258 | |
| 301 | +Covariance Type: HC3 | |
| 302 | +======================================================================================================================== | |
| 303 | + coef std err z P>|z| [0.025 0.975] | |
| 304 | +------------------------------------------------------------------------------------------------------------------------ | |
| 305 | +Intercept 6.1815 0.057 108.357 0.000 6.070 6.293 | |
| 306 | +C(source)[T.realtor] -0.3800 0.021 -17.682 0.000 -0.422 -0.338 | |
| 307 | +C(ptype_fe)[T.Country Home] 0.0525 0.067 0.788 0.431 -0.078 0.183 | |
| 308 | +C(ptype_fe)[T.Home] 0.0936 0.081 1.154 0.249 -0.065 0.253 | |
| 309 | +C(ptype_fe)[T.House] -0.0724 0.014 -5.164 0.000 -0.100 -0.045 | |
| 310 | +C(ptype_fe)[T.Intergenerational] 0.1435 0.063 2.291 0.022 0.021 0.266 | |
| 311 | +C(ptype_fe)[T.Other] 0.2526 0.121 2.086 0.037 0.015 0.490 | |
| 312 | +C(ptype_fe)[T.Row / Townhouse] -0.3076 0.014 -21.318 0.000 -0.336 -0.279 | |
| 313 | +C(ptype_fe)[T.Semi Detached] -0.0284 0.057 -0.501 0.616 -0.140 0.083 | |
| 314 | +C(ptype_fe)[T.Townhouse] 0.0700 0.098 0.713 0.476 -0.122 0.262 | |
| 315 | +C(city_fe)[T.acton vale] -0.0098 0.034 -0.284 0.776 -0.077 0.058 | |
| 316 | +C(city_fe)[T.alma] -0.0584 0.054 -1.079 0.281 -0.165 0.048 | |
| 317 | +C(city_fe)[T.amherst] -0.0225 0.049 -0.456 0.648 -0.119 0.074 | |
| 318 | +C(city_fe)[T.amos] -0.1343 0.044 -3.020 0.003 -0.222 -0.047 | |
| 319 | +C(city_fe)[T.austin] 0.4205 0.070 5.972 0.000 0.282 0.559 | |
| 320 | +C(city_fe)[T.baie-comeau] -0.3294 0.069 -4.787 0.000 -0.464 -0.195 | |
| 321 | +C(city_fe)[T.baie-saint-paul] 0.1285 0.077 1.672 0.094 -0.022 0.279 | |
| 322 | +C(city_fe)[T.beaconsfield] 0.7858 0.037 21.069 0.000 0.713 0.859 | |
| 323 | +C(city_fe)[T.beauport] 0.0776 0.037 2.076 0.038 0.004 0.151 | |
| 324 | +C(city_fe)[T.beloeil] 0.3812 0.049 7.840 0.000 0.286 0.476 | |
| 325 | +C(city_fe)[T.blainville] 0.4461 0.025 17.924 0.000 0.397 0.495 | |
| 326 | +C(city_fe)[T.blue sea] -0.1151 0.062 -1.863 0.062 -0.236 0.006 | |
| 327 | +C(city_fe)[T.bois-des-filion] 0.2942 0.048 6.129 0.000 0.200 0.388 | |
| 328 | +C(city_fe)[T.boisbriand] 0.3654 0.038 9.665 0.000 0.291 0.439 | |
| 329 | +C(city_fe)[T.boischatel] 0.4011 0.071 5.616 0.000 0.261 0.541 | |
| 330 | +C(city_fe)[T.boucherville] 0.4671 0.037 12.493 0.000 0.394 0.540 | |
| 331 | +C(city_fe)[T.bromont] 0.4244 0.034 12.535 0.000 0.358 0.491 | |
| 332 | +C(city_fe)[T.brossard] 0.5263 0.020 26.802 0.000 0.488 0.565 | |
| 333 | +C(city_fe)[T.bécancour] -0.0973 0.032 -3.045 0.002 -0.160 -0.035 | |
| 334 | +C(city_fe)[T.cantley] 0.1731 0.046 3.757 0.000 0.083 0.263 | |
| 335 | +C(city_fe)[T.carignan] 0.3790 0.042 8.990 0.000 0.296 0.462 | |
| 336 | +C(city_fe)[T.chandler] -0.2950 0.079 -3.757 0.000 -0.449 -0.141 | |
| 337 | +C(city_fe)[T.chateauguay] 0.2492 0.040 6.176 0.000 0.170 0.328 | |
| 338 | +C(city_fe)[T.chelsea] 0.3027 0.040 7.590 0.000 0.225 0.381 | |
| 339 | +C(city_fe)[T.chertsey] -0.0418 0.038 -1.107 0.268 -0.116 0.032 | |
| 340 | +C(city_fe)[T.chicoutimi] -0.0792 0.040 -1.976 0.048 -0.158 -0.001 | |
| 341 | +C(city_fe)[T.chomedey] 0.4118 0.055 7.515 0.000 0.304 0.519 | |
| 342 | +C(city_fe)[T.châteauguay] 0.2811 0.024 11.697 0.000 0.234 0.328 | |
| 343 | +C(city_fe)[T.contrecoeur] 0.1282 0.035 3.614 0.000 0.059 0.198 | |
| 344 | +C(city_fe)[T.cowansville] 0.1731 0.034 5.103 0.000 0.107 0.240 | |
| 345 | +C(city_fe)[T.crabtree] 0.0682 0.048 1.421 0.155 -0.026 0.162 | |
| 346 | +C(city_fe)[T.côte-saint-luc] 0.7496 0.053 14.252 0.000 0.647 0.853 | |
| 347 | +C(city_fe)[T.dolbeau-mistassini] -0.1900 0.063 -3.033 0.002 -0.313 -0.067 | |
| 348 | +C(city_fe)[T.dollard-des-ormeaux] 0.5396 0.029 18.879 0.000 0.484 0.596 | |
| 349 | +C(city_fe)[T.donnacona] 0.1291 0.035 3.685 0.000 0.060 0.198 | |
| 350 | +C(city_fe)[T.dorval] 0.6824 0.048 14.112 0.000 0.588 0.777 | |
| 351 | +C(city_fe)[T.drummondville] 0.0599 0.018 3.286 0.001 0.024 0.096 | |
| 352 | +C(city_fe)[T.duhamel] 0.0493 0.071 0.697 0.485 -0.089 0.188 | |
| 353 | +C(city_fe)[T.dunham] 0.3393 0.064 5.282 0.000 0.213 0.465 | |
| 354 | +C(city_fe)[T.eastman] 0.3752 0.056 6.645 0.000 0.265 0.486 | |
| 355 | +C(city_fe)[T.entrelacs] 0.2115 0.084 2.512 0.012 0.046 0.377 | |
| 356 | +C(city_fe)[T.farnham] 0.0658 0.035 1.873 0.061 -0.003 0.135 | |
| 357 | +C(city_fe)[T.fossambault-sur-le-lac] 0.3149 0.065 4.826 0.000 0.187 0.443 | |
| 358 | +C(city_fe)[T.gaspé] -0.3176 0.065 -4.918 0.000 -0.444 -0.191 | |
| 359 | +C(city_fe)[T.gatineau] 0.1320 0.013 10.162 0.000 0.107 0.158 | |
| 360 | +C(city_fe)[T.gore] 0.0680 0.055 1.227 0.220 -0.041 0.177 | |
| 361 | +C(city_fe)[T.granby] 0.1692 0.019 9.137 0.000 0.133 0.206 | |
| 362 | +C(city_fe)[T.grenville-sur-la-rouge] 0.1187 0.046 2.558 0.011 0.028 0.210 | |
| 363 | +C(city_fe)[T.hampstead] 1.2817 0.052 24.725 0.000 1.180 1.383 | |
| 364 | +C(city_fe)[T.harrington] -0.0772 0.125 -0.618 0.537 -0.322 0.168 | |
| 365 | +C(city_fe)[T.hudson] 0.4854 0.066 7.303 0.000 0.355 0.616 | |
| 366 | +C(city_fe)[T.joliette] 0.1347 0.045 3.023 0.003 0.047 0.222 | |
| 367 | +C(city_fe)[T.jonquière] -0.1481 0.044 -3.388 0.001 -0.234 -0.062 | |
| 368 | +C(city_fe)[T.kirkland] 0.5562 0.035 15.874 0.000 0.488 0.625 | |
| 369 | +C(city_fe)[T.l'ange-gardien] 0.2606 0.073 3.560 0.000 0.117 0.404 | |
| 370 | +C(city_fe)[T.l'assomption] 0.2026 0.028 7.206 0.000 0.147 0.258 | |
| 371 | +C(city_fe)[T.l'île-perrot] 0.4466 0.049 9.037 0.000 0.350 0.543 | |
| 372 | +C(city_fe)[T.la conception] 0.3260 0.052 6.254 0.000 0.224 0.428 | |
| 373 | +C(city_fe)[T.la malbaie] -0.1569 0.054 -2.898 0.004 -0.263 -0.051 | |
| 374 | +C(city_fe)[T.la minerve] 0.0588 0.052 1.128 0.260 -0.043 0.161 | |
| 375 | +C(city_fe)[T.la prairie] 0.4831 0.030 16.099 0.000 0.424 0.542 | |
| 376 | +C(city_fe)[T.la pêche] -0.0050 0.039 -0.129 0.897 -0.081 0.071 | |
| 377 | +C(city_fe)[T.la tuque] -0.3458 0.048 -7.231 0.000 -0.439 -0.252 | |
| 378 | +C(city_fe)[T.labelle] 0.0865 0.054 1.599 0.110 -0.020 0.193 | |
| 379 | +C(city_fe)[T.lac-beauport] 0.4153 0.085 4.883 0.000 0.249 0.582 | |
| 380 | +C(city_fe)[T.lac-brome] 0.6355 0.056 11.357 0.000 0.526 0.745 | |
| 381 | +C(city_fe)[T.lac-des-écorces] 0.0014 0.067 0.021 0.983 -0.130 0.133 | |
| 382 | +C(city_fe)[T.lac-etchemin] -0.2994 0.057 -5.288 0.000 -0.410 -0.188 | |
| 383 | +C(city_fe)[T.lachute] 0.1773 0.037 4.791 0.000 0.105 0.250 | |
| 384 | +C(city_fe)[T.lanoraie] 0.1419 0.037 3.844 0.000 0.070 0.214 | |
| 385 | +C(city_fe)[T.laval] 0.4740 0.014 34.317 0.000 0.447 0.501 | |
| 386 | +C(city_fe)[T.lavaltrie] 0.1688 0.037 4.538 0.000 0.096 0.242 | |
| 387 | +C(city_fe)[T.les éboulements] 0.1753 0.069 2.547 0.011 0.040 0.310 | |
| 388 | +C(city_fe)[T.longueuil] 0.3851 0.016 24.101 0.000 0.354 0.416 | |
| 389 | +C(city_fe)[T.lorraine] 0.4987 0.056 8.885 0.000 0.389 0.609 | |
| 390 | +C(city_fe)[T.lévis] 0.1158 0.021 5.394 0.000 0.074 0.158 | |
| 391 | +C(city_fe)[T.magog] 0.3283 0.032 10.286 0.000 0.266 0.391 | |
| 392 | +C(city_fe)[T.mandeville] -0.0735 0.040 -1.826 0.068 -0.152 0.005 | |
| 393 | +C(city_fe)[T.mansfield-et-pontefract] -0.5322 0.067 -7.928 0.000 -0.664 -0.401 | |
| 394 | +C(city_fe)[T.marieville] 0.2115 0.031 6.931 0.000 0.152 0.271 | |
| 395 | +C(city_fe)[T.mascouche] 0.3035 0.022 13.981 0.000 0.261 0.346 | |
| 396 | +C(city_fe)[T.matane] -0.1678 0.047 -3.568 0.000 -0.260 -0.076 | |
| 397 | +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.4115 0.048 8.594 0.000 0.318 0.505 | |
| 398 | +C(city_fe)[T.messines] -0.2594 0.051 -5.045 0.000 -0.360 -0.159 | |
| 399 | +C(city_fe)[T.mille-isles] 0.2025 0.051 3.999 0.000 0.103 0.302 | |
| 400 | +C(city_fe)[T.mirabel] 0.3088 0.021 14.650 0.000 0.268 0.350 | |
| 401 | +C(city_fe)[T.mont-joli] -0.2580 0.097 -2.649 0.008 -0.449 -0.067 | |
| 402 | +C(city_fe)[T.mont-laurier] -0.0438 0.041 -1.076 0.282 -0.124 0.036 | |
| 403 | +C(city_fe)[T.mont-royal] 1.2714 0.054 23.624 0.000 1.166 1.377 | |
| 404 | +C(city_fe)[T.mont-saint-hilaire] 0.5104 0.040 12.618 0.000 0.431 0.590 | |
| 405 | +C(city_fe)[T.mont-tremblant] 0.3946 0.035 11.119 0.000 0.325 0.464 | |
| 406 | +C(city_fe)[T.montréal] 0.7272 0.017 43.952 0.000 0.695 0.760 | |
| 407 | +C(city_fe)[T.morin-heights] 0.3128 0.039 7.963 0.000 0.236 0.390 | |
| 408 | +C(city_fe)[T.nominingue] 0.0819 0.081 1.012 0.311 -0.077 0.241 | |
| 409 | +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.3960 0.042 9.361 0.000 0.313 0.479 | |
| 410 | +C(city_fe)[T.notre-dame-de-pontmain] -0.1176 0.060 -1.973 0.049 -0.234 -0.001 | |
| 411 | +C(city_fe)[T.notre-dame-des-bois] -0.3186 0.067 -4.775 0.000 -0.449 -0.188 | |
| 412 | +C(city_fe)[T.notre-dame-des-prairies] 0.0959 0.042 2.279 0.023 0.013 0.178 | |
| 413 | +C(city_fe)[T.notre-dame-du-laus] 0.0251 0.056 0.448 0.654 -0.085 0.135 | |
| 414 | +C(city_fe)[T.orford] 0.3476 0.061 5.701 0.000 0.228 0.467 | |
| 415 | +C(city_fe)[T.papineauville] -0.0121 0.115 -0.105 0.916 -0.238 0.213 | |
| 416 | +C(city_fe)[T.petite-rivière-saint-françois] -0.0216 0.055 -0.391 0.696 -0.130 0.087 | |
| 417 | +C(city_fe)[T.piedmont] 0.3718 0.051 7.301 0.000 0.272 0.472 | |
| 418 | +C(city_fe)[T.pincourt] 0.3208 0.036 8.976 0.000 0.251 0.391 | |
| 419 | +C(city_fe)[T.pointe-claire] 0.6096 0.041 14.960 0.000 0.530 0.689 | |
| 420 | +C(city_fe)[T.pont-rouge] 0.0167 0.044 0.378 0.705 -0.070 0.103 | |
| 421 | +C(city_fe)[T.prévost] 0.1910 0.027 7.046 0.000 0.138 0.244 | |
| 422 | +C(city_fe)[T.québec] 0.2113 0.018 11.997 0.000 0.177 0.246 | |
| 423 | +C(city_fe)[T.rawdon] 0.0548 0.032 1.711 0.087 -0.008 0.118 | |
| 424 | +C(city_fe)[T.repentigny] 0.2910 0.019 15.040 0.000 0.253 0.329 | |
| 425 | +C(city_fe)[T.rigaud] 0.1464 0.045 3.257 0.001 0.058 0.235 | |
| 426 | +C(city_fe)[T.rimouski] -0.0604 0.034 -1.779 0.075 -0.127 0.006 | |
| 427 | +C(city_fe)[T.rivière-du-loup] -0.0744 0.054 -1.380 0.168 -0.180 0.031 | |
| 428 | +C(city_fe)[T.rivière-rouge] -0.1228 0.059 -2.083 0.037 -0.238 -0.007 | |
| 429 | +C(city_fe)[T.roberval] -0.1505 0.064 -2.335 0.020 -0.277 -0.024 | |
| 430 | +C(city_fe)[T.rosemère] 0.4912 0.038 12.813 0.000 0.416 0.566 | |
| 431 | +C(city_fe)[T.saguenay] -0.0362 0.023 -1.577 0.115 -0.081 0.009 | |
| 432 | +C(city_fe)[T.saint-adolphe-d'howard] 0.1426 0.045 3.142 0.002 0.054 0.231 | |
| 433 | +C(city_fe)[T.saint-amable] 0.2897 0.035 8.261 0.000 0.221 0.358 | |
| 434 | +C(city_fe)[T.saint-anicet] 0.2175 0.068 3.178 0.001 0.083 0.352 | |
| 435 | +C(city_fe)[T.saint-basile-le-grand] 0.3978 0.045 8.829 0.000 0.310 0.486 | |
| 436 | +C(city_fe)[T.saint-bruno-de-montarville] 0.6239 0.040 15.791 0.000 0.546 0.701 | |
| 437 | +C(city_fe)[T.saint-calixte] 0.0061 0.034 0.177 0.859 -0.061 0.073 | |
| 438 | +C(city_fe)[T.saint-charles-borromée] 0.1596 0.053 3.015 0.003 0.056 0.263 | |
| 439 | +C(city_fe)[T.saint-colomban] 0.1400 0.041 3.385 0.001 0.059 0.221 | |
| 440 | +C(city_fe)[T.saint-constant] 0.3374 0.031 10.768 0.000 0.276 0.399 | |
| 441 | +C(city_fe)[T.saint-donat] 0.3168 0.047 6.726 0.000 0.224 0.409 | |
| 442 | +C(city_fe)[T.saint-eustache] 0.3452 0.036 9.568 0.000 0.274 0.416 | |
| 443 | +C(city_fe)[T.saint-félicien] -0.0434 0.054 -0.807 0.420 -0.149 0.062 | |
| 444 | +C(city_fe)[T.saint-gabriel-de-brandon] -0.0090 0.063 -0.143 0.887 -0.133 0.115 | |
| 445 | +C(city_fe)[T.saint-georges] -0.1772 0.038 -4.679 0.000 -0.251 -0.103 | |
| 446 | +C(city_fe)[T.saint-hippolyte] 0.2118 0.041 5.225 0.000 0.132 0.291 | |
| 447 | +C(city_fe)[T.saint-honoré] -0.0330 0.062 -0.536 0.592 -0.154 0.088 | |
| 448 | +C(city_fe)[T.saint-hyacinthe] 0.1948 0.033 5.974 0.000 0.131 0.259 | |
| 449 | +C(city_fe)[T.saint-jean-de-matha] 0.0276 0.065 0.422 0.673 -0.101 0.156 | |
| 450 | +C(city_fe)[T.saint-jean-sur-richelieu] 0.2785 0.019 14.288 0.000 0.240 0.317 | |
| 451 | +C(city_fe)[T.saint-jérôme] 0.2141 0.020 10.479 0.000 0.174 0.254 | |
| 452 | +C(city_fe)[T.saint-lambert] 0.8773 0.047 18.632 0.000 0.785 0.970 | |
| 453 | +C(city_fe)[T.saint-lambert-de-lauzon] 0.0684 0.053 1.298 0.194 -0.035 0.172 | |
| 454 | +C(city_fe)[T.saint-lin/laurentides] 0.1066 0.032 3.313 0.001 0.044 0.170 | |
| 455 | +C(city_fe)[T.saint-marc-des-carrières] -0.1398 0.032 -4.307 0.000 -0.203 -0.076 | |
| 456 | +C(city_fe)[T.saint-michel-des-saints] -0.0394 0.067 -0.589 0.556 -0.171 0.092 | |
| 457 | +C(city_fe)[T.saint-paul-de-l'île-aux-noix] 0.0212 0.099 0.214 0.831 -0.173 0.215 | |
| 458 | +C(city_fe)[T.saint-philippe] 0.4244 0.029 14.752 0.000 0.368 0.481 | |
| 459 | +C(city_fe)[T.saint-raymond] 0.0769 0.067 1.154 0.249 -0.054 0.207 | |
| 460 | +C(city_fe)[T.saint-sauveur] 0.2746 0.033 8.211 0.000 0.209 0.340 | |
| 461 | +C(city_fe)[T.saint-zotique] 0.2721 0.037 7.329 0.000 0.199 0.345 | |
| 462 | +C(city_fe)[T.saint-zénon] -0.0122 0.106 -0.115 0.908 -0.221 0.196 | |
| 463 | +C(city_fe)[T.sainte-adèle] 0.3087 0.048 6.408 0.000 0.214 0.403 | |
| 464 | +C(city_fe)[T.sainte-agathe-des-monts] 0.1241 0.050 2.457 0.014 0.025 0.223 | |
| 465 | +C(city_fe)[T.sainte-anne-des-plaines] 0.2834 0.062 4.584 0.000 0.162 0.405 | |
| 466 | +C(city_fe)[T.sainte-barbe] 0.4826 0.071 6.781 0.000 0.343 0.622 | |
| 467 | +C(city_fe)[T.sainte-brigitte-de-laval] 0.1199 0.041 2.899 0.004 0.039 0.201 | |
| 468 | +C(city_fe)[T.sainte-béatrix] -0.0687 0.043 -1.582 0.114 -0.154 0.016 | |
| 469 | +C(city_fe)[T.sainte-catherine] 0.3381 0.034 9.976 0.000 0.272 0.405 | |
| 470 | +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1270 0.059 2.152 0.031 0.011 0.243 | |
| 471 | +C(city_fe)[T.sainte-julie] 0.4826 0.055 8.787 0.000 0.375 0.590 | |
| 472 | +C(city_fe)[T.sainte-julienne] 0.0375 0.035 1.079 0.281 -0.031 0.106 | |
| 473 | +C(city_fe)[T.sainte-lucie-des-laurentides] 0.1493 0.037 3.986 0.000 0.076 0.223 | |
| 474 | +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0068 0.053 0.129 0.897 -0.097 0.110 | |
| 475 | +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.3182 0.023 13.870 0.000 0.273 0.363 | |
| 476 | +C(city_fe)[T.sainte-sophie] 0.0978 0.038 2.566 0.010 0.023 0.173 | |
| 477 | +C(city_fe)[T.sainte-thérèse] 0.3765 0.041 9.286 0.000 0.297 0.456 | |
| 478 | +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2157 0.060 -3.580 0.000 -0.334 -0.098 | |
| 479 | +C(city_fe)[T.salaberry-de-valleyfield] 0.0841 0.043 1.978 0.048 0.001 0.168 | |
| 480 | +C(city_fe)[T.sept-îles] -0.0848 0.036 -2.364 0.018 -0.155 -0.014 | |
| 481 | +C(city_fe)[T.shawinigan] -0.1015 0.025 -4.056 0.000 -0.151 -0.052 | |
| 482 | +C(city_fe)[T.shefford] 0.2935 0.059 4.934 0.000 0.177 0.410 | |
| 483 | +C(city_fe)[T.sherbrooke] 0.1051 0.017 6.332 0.000 0.073 0.138 | |
| 484 | +C(city_fe)[T.sorel-tracy] 0.0059 0.027 0.220 0.826 -0.047 0.059 | |
| 485 | +C(city_fe)[T.st-colomban] 0.2744 0.069 3.978 0.000 0.139 0.410 | |
| 486 | +C(city_fe)[T.st-hippolyte] 0.1024 0.077 1.324 0.185 -0.049 0.254 | |
| 487 | +C(city_fe)[T.st-jean-sur-richelieu] 0.2296 0.034 6.754 0.000 0.163 0.296 | |
| 488 | +C(city_fe)[T.st-jérôme] 0.2389 0.036 6.713 0.000 0.169 0.309 | |
| 489 | +C(city_fe)[T.st-sauveur] 0.4023 0.088 4.549 0.000 0.229 0.576 | |
| 490 | +C(city_fe)[T.ste-adèle] 0.1944 0.064 3.044 0.002 0.069 0.320 | |
| 491 | +C(city_fe)[T.ste-julienne] 0.0665 0.038 1.740 0.082 -0.008 0.141 | |
| 492 | +C(city_fe)[T.sutton] 0.4900 0.051 9.559 0.000 0.390 0.590 | |
| 493 | +C(city_fe)[T.terrebonne] 0.2995 0.018 16.761 0.000 0.264 0.335 | |
| 494 | +C(city_fe)[T.thetford mines] -0.2784 0.058 -4.762 0.000 -0.393 -0.164 | |
| 495 | +C(city_fe)[T.trois-rivières] 0.0129 0.020 0.634 0.526 -0.027 0.053 | |
| 496 | +C(city_fe)[T.très-saint-rédempteur] 0.5938 0.052 11.367 0.000 0.491 0.696 | |
| 497 | +C(city_fe)[T.témiscouata-sur-le-lac] -0.1940 0.074 -2.618 0.009 -0.339 -0.049 | |
| 498 | +C(city_fe)[T.val-d'or] 0.0212 0.029 0.742 0.458 -0.035 0.077 | |
| 499 | +C(city_fe)[T.val-david] 0.0449 0.054 0.828 0.408 -0.061 0.151 | |
| 500 | +C(city_fe)[T.val-des-monts] 0.0856 0.027 3.163 0.002 0.033 0.139 | |
| 501 | +C(city_fe)[T.val-morin] 0.0522 0.057 0.916 0.360 -0.060 0.164 | |
| 502 | +C(city_fe)[T.vaudreuil-dorion] 0.2565 0.029 8.949 0.000 0.200 0.313 | |
| 503 | +C(city_fe)[T.venise-en-québec] 0.1799 0.044 4.085 0.000 0.094 0.266 | |
| 504 | +C(city_fe)[T.victoriaville] -0.0166 0.024 -0.691 0.489 -0.064 0.031 | |
| 505 | +C(city_fe)[T.waterloo] 0.1307 0.049 2.687 0.007 0.035 0.226 | |
| 506 | +C(city_fe)[T.weedon] -0.1577 0.062 -2.550 0.011 -0.279 -0.036 | |
| 507 | +C(city_fe)[T.wentworth-nord] 0.0592 0.072 0.828 0.408 -0.081 0.199 | |
| 508 | +C(city_fe)[T.westmount] 1.6031 0.038 42.458 0.000 1.529 1.677 | |
| 509 | +ln_lot 0.0400 0.003 12.549 0.000 0.034 0.046 | |
| 510 | +has_lot -0.0842 0.015 -5.753 0.000 -0.113 -0.056 | |
| 511 | +beds 0.0281 0.004 6.804 0.000 0.020 0.036 | |
| 512 | +baths 0.1331 0.005 25.473 0.000 0.123 0.143 | |
| 513 | +parking 0.0153 0.002 8.276 0.000 0.012 0.019 | |
| 514 | +storeys 0.0023 0.096 0.024 0.981 -0.186 0.190 | |
| 515 | +age -0.0066 0.000 -16.853 0.000 -0.007 -0.006 | |
| 516 | +age2 3.287e-05 2.79e-06 11.795 0.000 2.74e-05 3.83e-05 | |
| 517 | +has_age -0.1356 0.019 -7.296 0.000 -0.172 -0.099 | |
| 518 | +pool 0.0217 0.011 1.945 0.052 -0.000 0.044 | |
| 519 | +waterfront 0.0497 0.019 2.636 0.008 0.013 0.087 | |
| 520 | +garage 0.0322 0.018 1.748 0.080 -0.004 0.068 | |
| 521 | +fireplace -0.0667 0.016 -4.270 0.000 -0.097 -0.036 | |
| 522 | +central_air 0.0445 0.010 4.577 0.000 0.025 0.064 | |
| 523 | +finished_bsmt -0.0063 0.008 -0.773 0.439 -0.022 0.010 | |
| 524 | +near_water 0.0709 0.020 3.459 0.001 0.031 0.111 | |
| 525 | +cul_de_sac 0.0285 0.009 3.179 0.001 0.011 0.046 | |
| 526 | +golf 0.0180 0.006 3.122 0.002 0.007 0.029 | |
| 527 | +ski 0.0382 0.010 3.899 0.000 0.019 0.057 | |
| 528 | +near_park 0.0074 0.008 0.929 0.353 -0.008 0.023 | |
| 529 | +near_school -0.0520 0.009 -6.027 0.000 -0.069 -0.035 | |
| 530 | +near_highway 0.0475 0.006 7.384 0.000 0.035 0.060 | |
| 531 | +transit 0.0684 0.007 9.180 0.000 0.054 0.083 | |
| 532 | +no_rear_neighbor -0.0453 0.009 -4.890 0.000 -0.063 -0.027 | |
| 533 | +n_images_c 0.0057 0.000 18.731 0.000 0.005 0.006 | |
| 534 | +virtual_tour 0.0207 0.016 1.327 0.184 -0.010 0.051 | |
| 535 | +n_rooms 0.0254 0.001 17.680 0.000 0.023 0.028 | |
| 536 | +n_levels_r 0.0413 0.047 0.871 0.384 -0.052 0.134 | |
| 537 | +has_primary_bed 0.0664 0.010 6.743 0.000 0.047 0.086 | |
| 538 | +has_basement -0.1032 0.043 -2.392 0.017 -0.188 -0.019 | |
| 539 | +r_office 0.0359 0.008 4.728 0.000 0.021 0.051 | |
| 540 | +r_family 0.0359 0.009 4.050 0.000 0.019 0.053 | |
| 541 | +r_workshop -0.0297 0.009 -3.377 0.001 -0.047 -0.012 | |
| 542 | +r_winecellar 0.1393 0.026 5.335 0.000 0.088 0.190 | |
| 543 | +r_sunroom 0.0193 0.013 1.438 0.150 -0.007 0.046 | |
| 544 | +r_mezzanine 0.0208 0.018 1.150 0.250 -0.015 0.056 | |
| 545 | +r_library 0.1475 0.069 2.125 0.034 0.011 0.284 | |
| 546 | +r_laundry 0.0363 0.006 6.025 0.000 0.025 0.048 | |
| 547 | +g_cond 0.0119 0.004 3.005 0.003 0.004 0.020 | |
| 548 | +g_hardwood 0.0315 0.008 3.705 0.000 0.015 0.048 | |
| 549 | +g_quartz_granite 0.0177 0.008 2.091 0.037 0.001 0.034 | |
| 550 | +g_view_water 0.0309 0.014 2.142 0.032 0.003 0.059 | |
| 551 | +has_gpt 6.1815 0.057 108.357 0.000 6.070 6.293 | |
| 552 | +g_turnkey 0.0044 0.006 0.692 0.489 -0.008 0.017 | |
| 553 | +g_recently_renovated 0.0067 0.008 0.868 0.385 -0.008 0.022 | |
| 554 | +g_garage 0.0193 0.018 1.076 0.282 -0.016 0.054 | |
| 555 | +g_pool 0.0484 0.012 4.089 0.000 0.025 0.072 | |
| 556 | +g_waterfront 0.0703 0.016 4.356 0.000 0.039 0.102 | |
| 557 | +g_finished_basement -0.0274 0.007 -4.097 0.000 -0.040 -0.014 | |
| 558 | +g_fireplace 0.0852 0.016 5.392 0.000 0.054 0.116 | |
| 559 | +g_central_ac 0.0312 0.010 3.279 0.001 0.013 0.050 | |
| 560 | +g_new_construction 0.0628 0.013 4.949 0.000 0.038 0.088 | |
| 561 | +g_income_property -0.0340 0.009 -3.966 0.000 -0.051 -0.017 | |
| 562 | +g_renovated_kitchen 0.0338 0.008 4.022 0.000 0.017 0.050 | |
| 563 | +g_renovated_bathroom -0.0109 0.008 -1.389 0.165 -0.026 0.004 | |
| 564 | +g_large_lot 0.0117 0.007 1.781 0.075 -0.001 0.025 | |
| 565 | +g_quiet_area -0.0209 0.006 -3.661 0.000 -0.032 -0.010 | |
| 566 | +============================================================================== | |
| 567 | +Omnibus: 2524.251 Durbin-Watson: 1.763 | |
| 568 | +Prob(Omnibus): 0.000 Jarque-Bera (JB): 16663.299 | |
| 569 | +Skew: 0.454 Prob(JB): 0.00 | |
| 570 | +Kurtosis: 7.482 Cond. No. 9.65e+16 | |
| 571 | +============================================================================== | |
| 572 | + | |
| 573 | +Notes: | |
| 574 | +[1] Standard Errors are heteroscedasticity robust (HC3) | |
| 575 | +[2] The smallest eigenvalue is 4.82e-23. This might indicate that there are | |
| 576 | +strong multicollinearity problems or that the design matrix is singular. | |
| \ No newline at end of file | ||
added
results/revision_extras.json
+486 −0
@@ -0,0 +1,486 @@ | ||
| 1 | +{ | |
| 2 | + "modelA": { | |
| 3 | + "r2": 0.7552103110293896, | |
| 4 | + "adj": 0.7477317389713678, | |
| 5 | + "n": 9446, | |
| 6 | + "coef": { | |
| 7 | + "ln_sqft": { | |
| 8 | + "b": 0.3583611466783186, | |
| 9 | + "se_hc3": 0.013021312719264802, | |
| 10 | + "p_hc3": 9.811436146831853e-167, | |
| 11 | + "se_cl": 0.011546608518322881, | |
| 12 | + "p_cl": 1.7598083257322608e-211 | |
| 13 | + }, | |
| 14 | + "ln_lot": { | |
| 15 | + "b": 0.027321695727434744, | |
| 16 | + "se_hc3": 0.003968643673932088, | |
| 17 | + "p_hc3": 5.803517449373336e-12, | |
| 18 | + "se_cl": 0.0057217322173256846, | |
| 19 | + "p_cl": 1.7964103147193876e-06 | |
| 20 | + }, | |
| 21 | + "beds": { | |
| 22 | + "b": 0.013623293126953095, | |
| 23 | + "se_hc3": 0.0040602600210150885, | |
| 24 | + "p_hc3": 0.0007928582892261605, | |
| 25 | + "se_cl": 0.00457074886728697, | |
| 26 | + "p_cl": 0.002877420342379003 | |
| 27 | + }, | |
| 28 | + "baths": { | |
| 29 | + "b": 0.07348769487973825, | |
| 30 | + "se_hc3": 0.006121072722376004, | |
| 31 | + "p_hc3": 3.3169032296614814e-33, | |
| 32 | + "se_cl": 0.0069532806823969125, | |
| 33 | + "p_cl": 4.1586858986169373e-26 | |
| 34 | + }, | |
| 35 | + "parking": { | |
| 36 | + "b": 0.01149153979340771, | |
| 37 | + "se_hc3": 0.0015211101627679384, | |
| 38 | + "p_hc3": 4.198076313292844e-14, | |
| 39 | + "se_cl": 0.0014144294006288624, | |
| 40 | + "p_cl": 4.491890458706105e-16 | |
| 41 | + }, | |
| 42 | + "age": { | |
| 43 | + "b": -0.003778162499817968, | |
| 44 | + "se_hc3": 0.0004166167508553082, | |
| 45 | + "p_hc3": 1.2047149502398342e-19, | |
| 46 | + "se_cl": 0.0010761773989556892, | |
| 47 | + "p_cl": 0.00044688706454431995 | |
| 48 | + }, | |
| 49 | + "age2": { | |
| 50 | + "b": 2.0412213402030788e-05, | |
| 51 | + "se_hc3": 3.1634803715459156e-06, | |
| 52 | + "p_hc3": 1.1005287533895908e-10, | |
| 53 | + "se_cl": 4.403903629319602e-06, | |
| 54 | + "p_cl": 3.568904338805579e-06 | |
| 55 | + }, | |
| 56 | + "central_air": { | |
| 57 | + "b": 0.053384334694057345, | |
| 58 | + "se_hc3": 0.011164185372283303, | |
| 59 | + "p_hc3": 1.737763866667577e-06, | |
| 60 | + "se_cl": 0.017763303901713427, | |
| 61 | + "p_cl": 0.0026530539182792814 | |
| 62 | + }, | |
| 63 | + "transit": { | |
| 64 | + "b": 0.09000078993798896, | |
| 65 | + "se_hc3": 0.00842353892882497, | |
| 66 | + "p_hc3": 1.2037429378492829e-26, | |
| 67 | + "se_cl": 0.04560367671561142, | |
| 68 | + "p_cl": 0.04843379796570755 | |
| 69 | + }, | |
| 70 | + "waterfront": { | |
| 71 | + "b": 0.05808657079776136, | |
| 72 | + "se_hc3": 0.04276278415929502, | |
| 73 | + "p_hc3": 0.17435455833645552, | |
| 74 | + "se_cl": 0.033496164012264705, | |
| 75 | + "p_cl": 0.08289571923560418 | |
| 76 | + }, | |
| 77 | + "g_cond": { | |
| 78 | + "b": 0.016168128372704954, | |
| 79 | + "se_hc3": 0.004284326158407061, | |
| 80 | + "p_hc3": 0.0001607889544697841, | |
| 81 | + "se_cl": 0.0058314343073510595, | |
| 82 | + "p_cl": 0.005561354758076747 | |
| 83 | + }, | |
| 84 | + "g_renovated_kitchen": { | |
| 85 | + "b": 0.026089835765711222, | |
| 86 | + "se_hc3": 0.009731092534231904, | |
| 87 | + "p_hc3": 0.007338501052577358, | |
| 88 | + "se_cl": 0.006899975364881851, | |
| 89 | + "p_cl": 0.00015610612945056893 | |
| 90 | + }, | |
| 91 | + "g_hardwood": { | |
| 92 | + "b": 0.023654684502188772, | |
| 93 | + "se_hc3": 0.007568398763791719, | |
| 94 | + "p_hc3": 0.0017753064896647113, | |
| 95 | + "se_cl": 0.009325076436933125, | |
| 96 | + "p_cl": 0.011191092284423674 | |
| 97 | + }, | |
| 98 | + "g_new_construction": { | |
| 99 | + "b": -0.00030428071460375516, | |
| 100 | + "se_hc3": 0.013498897144460655, | |
| 101 | + "p_hc3": 0.9820162844667388, | |
| 102 | + "se_cl": 0.013315085313141847, | |
| 103 | + "p_cl": 0.9817680661387342 | |
| 104 | + }, | |
| 105 | + "g_income_property": { | |
| 106 | + "b": -0.01916827298970958, | |
| 107 | + "se_hc3": 0.010125989321693481, | |
| 108 | + "p_hc3": 0.05836082179559654, | |
| 109 | + "se_cl": 0.009252803603413885, | |
| 110 | + "p_cl": 0.03830109511355022 | |
| 111 | + }, | |
| 112 | + "v_curb": { | |
| 113 | + "b": 0.023017903057193298, | |
| 114 | + "se_hc3": 0.007507554436150879, | |
| 115 | + "p_hc3": 0.0021696833221349896, | |
| 116 | + "se_cl": 0.00865392944593514, | |
| 117 | + "p_cl": 0.007818219565218354 | |
| 118 | + }, | |
| 119 | + "v_reno": { | |
| 120 | + "b": 0.023593099837405124, | |
| 121 | + "se_hc3": 0.007459829833637167, | |
| 122 | + "p_hc3": 0.0015632080662522378, | |
| 123 | + "se_cl": 0.009529503385437559, | |
| 124 | + "p_cl": 0.013293976823511309 | |
| 125 | + }, | |
| 126 | + "v_luxury": { | |
| 127 | + "b": 0.05100475597004259, | |
| 128 | + "se_hc3": 0.009917011195885339, | |
| 129 | + "p_hc3": 2.7015815598254944e-07, | |
| 130 | + "se_cl": 0.011932756209389966, | |
| 131 | + "p_cl": 1.916971879355069e-05 | |
| 132 | + }, | |
| 133 | + "v_garage_vis": { | |
| 134 | + "b": 0.04626239493908085, | |
| 135 | + "se_hc3": 0.007095142592775905, | |
| 136 | + "p_hc3": 7.017108386426899e-11, | |
| 137 | + "se_cl": 0.005814520452429543, | |
| 138 | + "p_cl": 1.7718006523061964e-15 | |
| 139 | + }, | |
| 140 | + "v_mat_brique": { | |
| 141 | + "b": 0.033343794442173225, | |
| 142 | + "se_hc3": 0.00827594130032085, | |
| 143 | + "p_hc3": 5.601384271322806e-05, | |
| 144 | + "se_cl": 0.009143815895444862, | |
| 145 | + "p_cl": 0.0002657381339945666 | |
| 146 | + }, | |
| 147 | + "v_mat_pierre": { | |
| 148 | + "b": 0.03615198096489164, | |
| 149 | + "se_hc3": 0.007737924779006411, | |
| 150 | + "p_hc3": 2.982065560443421e-06, | |
| 151 | + "se_cl": 0.008011891510590393, | |
| 152 | + "p_cl": 6.413130561843427e-06 | |
| 153 | + }, | |
| 154 | + "v_mat_vinyle": { | |
| 155 | + "b": -0.04493374989989597, | |
| 156 | + "se_hc3": 0.007597159274294025, | |
| 157 | + "p_hc3": 3.3279171858711188e-09, | |
| 158 | + "se_cl": 0.015100349535056819, | |
| 159 | + "p_cl": 0.0029234340683073927 | |
| 160 | + }, | |
| 161 | + "v_set_rural": { | |
| 162 | + "b": -0.12230705352273197, | |
| 163 | + "se_hc3": 0.04529347936059, | |
| 164 | + "p_hc3": 0.006927205074830596, | |
| 165 | + "se_cl": 0.032184971001596396, | |
| 166 | + "p_cl": 0.00014462056902901985 | |
| 167 | + }, | |
| 168 | + "v_land": { | |
| 169 | + "b": 0.015079214851207945, | |
| 170 | + "se_hc3": 0.005163307230556136, | |
| 171 | + "p_hc3": 0.003495188385196164, | |
| 172 | + "se_cl": 0.0062394992136362536, | |
| 173 | + "p_cl": 0.015660422546723243 | |
| 174 | + }, | |
| 175 | + "v_cond": { | |
| 176 | + "b": 0.011435724114234713, | |
| 177 | + "se_hc3": 0.009935939848613642, | |
| 178 | + "p_hc3": 0.24975470480811712, | |
| 179 | + "se_cl": 0.00793900430747386, | |
| 180 | + "p_cl": 0.1497406477456507 | |
| 181 | + } | |
| 182 | + } | |
| 183 | + }, | |
| 184 | + "modelB": { | |
| 185 | + "r2": 0.7250019979121325, | |
| 186 | + "adj": 0.7209302236414477, | |
| 187 | + "n": 19123, | |
| 188 | + "coef": { | |
| 189 | + "ln_lot": { | |
| 190 | + "b": 0.040421819255842366, | |
| 191 | + "se_hc3": 0.0030793415163729103, | |
| 192 | + "p_hc3": 2.3130287315281885e-39, | |
| 193 | + "se_cl": 0.005130684552116609, | |
| 194 | + "p_cl": 3.314786914788146e-15 | |
| 195 | + }, | |
| 196 | + "beds": { | |
| 197 | + "b": 0.03225644920405396, | |
| 198 | + "se_hc3": 0.0037380829672411845, | |
| 199 | + "p_hc3": 6.181384242360143e-18, | |
| 200 | + "se_cl": 0.0026461656800326736, | |
| 201 | + "p_cl": 3.5192764358646113e-34 | |
| 202 | + }, | |
| 203 | + "baths": { | |
| 204 | + "b": 0.11334162478693238, | |
| 205 | + "se_hc3": 0.004820025879176098, | |
| 206 | + "p_hc3": 2.882946420348713e-122, | |
| 207 | + "se_cl": 0.0065680707475647485, | |
| 208 | + "p_cl": 1.0006690034057575e-66 | |
| 209 | + }, | |
| 210 | + "parking": { | |
| 211 | + "b": 0.014887132710857906, | |
| 212 | + "se_hc3": 0.001722223609697608, | |
| 213 | + "p_hc3": 5.4214611052329344e-18, | |
| 214 | + "se_cl": 0.0019170158855884335, | |
| 215 | + "p_cl": 8.114125760065666e-15 | |
| 216 | + }, | |
| 217 | + "age": { | |
| 218 | + "b": -0.00454728435076453, | |
| 219 | + "se_hc3": 0.00031568867613853834, | |
| 220 | + "p_hc3": 4.859972485857048e-47, | |
| 221 | + "se_cl": 0.0005421521451234677, | |
| 222 | + "p_cl": 4.9671978979020763e-17 | |
| 223 | + }, | |
| 224 | + "age2": { | |
| 225 | + "b": 2.4738741307502545e-05, | |
| 226 | + "se_hc3": 2.2589789920551392e-06, | |
| 227 | + "p_hc3": 6.55056354106643e-28, | |
| 228 | + "se_cl": 1.6280767754927894e-06, | |
| 229 | + "p_cl": 3.8125346654408024e-52 | |
| 230 | + }, | |
| 231 | + "central_air": { | |
| 232 | + "b": 0.020145753639079278, | |
| 233 | + "se_hc3": 0.009178809775081053, | |
| 234 | + "p_hc3": 0.028177150453293415, | |
| 235 | + "se_cl": 0.017821940423307536, | |
| 236 | + "p_cl": 0.25831168008287975 | |
| 237 | + }, | |
| 238 | + "transit": { | |
| 239 | + "b": 0.058387955324457536, | |
| 240 | + "se_hc3": 0.006823518187251239, | |
| 241 | + "p_hc3": 1.1597359094737698e-17, | |
| 242 | + "se_cl": 0.039429380632120564, | |
| 243 | + "p_cl": 0.1386535998125059 | |
| 244 | + }, | |
| 245 | + "waterfront": { | |
| 246 | + "b": 0.061910921726998136, | |
| 247 | + "se_hc3": 0.023452608193077887, | |
| 248 | + "p_hc3": 0.008294738330249088, | |
| 249 | + "se_cl": 0.019005260291841415, | |
| 250 | + "p_cl": 0.0011237145722294154 | |
| 251 | + }, | |
| 252 | + "g_cond": { | |
| 253 | + "b": 0.016317524818277362, | |
| 254 | + "se_hc3": 0.003676322607356966, | |
| 255 | + "p_hc3": 9.056887962882854e-06, | |
| 256 | + "se_cl": 0.0051984051513270825, | |
| 257 | + "p_cl": 0.0016955538138371947 | |
| 258 | + }, | |
| 259 | + "g_renovated_kitchen": { | |
| 260 | + "b": 0.02694519585550771, | |
| 261 | + "se_hc3": 0.00778196222799486, | |
| 262 | + "p_hc3": 0.0005351431624444984, | |
| 263 | + "se_cl": 0.00963562440448091, | |
| 264 | + "p_cl": 0.005167314672277988 | |
| 265 | + }, | |
| 266 | + "g_hardwood": { | |
| 267 | + "b": 0.029508607751202176, | |
| 268 | + "se_hc3": 0.007732639804181234, | |
| 269 | + "p_hc3": 0.000135571823960277, | |
| 270 | + "se_cl": 0.010501088270130441, | |
| 271 | + "p_cl": 0.004953344261501498 | |
| 272 | + }, | |
| 273 | + "g_new_construction": { | |
| 274 | + "b": -0.015913022424995732, | |
| 275 | + "se_hc3": 0.012366399085437902, | |
| 276 | + "p_hc3": 0.19816569619711843, | |
| 277 | + "se_cl": 0.013844198802557594, | |
| 278 | + "p_cl": 0.2503761852618782 | |
| 279 | + }, | |
| 280 | + "g_income_property": { | |
| 281 | + "b": -0.012880740908525923, | |
| 282 | + "se_hc3": 0.008090151720423983, | |
| 283 | + "p_hc3": 0.11135084238149032, | |
| 284 | + "se_cl": 0.007646640747981612, | |
| 285 | + "p_cl": 0.09208572167107143 | |
| 286 | + }, | |
| 287 | + "v_curb": { | |
| 288 | + "b": 0.01595794997392595, | |
| 289 | + "se_hc3": 0.006208945541452705, | |
| 290 | + "p_hc3": 0.010165313745874592, | |
| 291 | + "se_cl": 0.013160921694546399, | |
| 292 | + "p_cl": 0.22531138001532158 | |
| 293 | + }, | |
| 294 | + "v_reno": { | |
| 295 | + "b": 0.01998000405352155, | |
| 296 | + "se_hc3": 0.005938481115424498, | |
| 297 | + "p_hc3": 0.0007668321703145232, | |
| 298 | + "se_cl": 0.00862597465890367, | |
| 299 | + "p_cl": 0.020544051694669916 | |
| 300 | + }, | |
| 301 | + "v_luxury": { | |
| 302 | + "b": 0.061660294830383405, | |
| 303 | + "se_hc3": 0.009917957983108048, | |
| 304 | + "p_hc3": 5.066358372641718e-10, | |
| 305 | + "se_cl": 0.008697717532597092, | |
| 306 | + "p_cl": 1.3484065713602217e-12 | |
| 307 | + }, | |
| 308 | + "v_garage_vis": { | |
| 309 | + "b": 0.04651267305326928, | |
| 310 | + "se_hc3": 0.005473235777759556, | |
| 311 | + "p_hc3": 1.9254685227194248e-17, | |
| 312 | + "se_cl": 0.0067205744806021995, | |
| 313 | + "p_cl": 4.4866578070246505e-12 | |
| 314 | + }, | |
| 315 | + "v_mat_brique": { | |
| 316 | + "b": 0.04403606617642199, | |
| 317 | + "se_hc3": 0.006434026368289839, | |
| 318 | + "p_hc3": 7.687913299168285e-12, | |
| 319 | + "se_cl": 0.011001756372510255, | |
| 320 | + "p_cl": 6.263967386269365e-05 | |
| 321 | + }, | |
| 322 | + "v_mat_pierre": { | |
| 323 | + "b": 0.04658529674361125, | |
| 324 | + "se_hc3": 0.006389001415391058, | |
| 325 | + "p_hc3": 3.0656103177722586e-13, | |
| 326 | + "se_cl": 0.004986368566238395, | |
| 327 | + "p_cl": 9.405939222542233e-21 | |
| 328 | + }, | |
| 329 | + "v_mat_vinyle": { | |
| 330 | + "b": -0.06180816681575538, | |
| 331 | + "se_hc3": 0.0059550197354287316, | |
| 332 | + "p_hc3": 3.0844255288655324e-25, | |
| 333 | + "se_cl": 0.012339743471244337, | |
| 334 | + "p_cl": 5.475063124724954e-07 | |
| 335 | + }, | |
| 336 | + "v_set_rural": { | |
| 337 | + "b": -0.07600759882521751, | |
| 338 | + "se_hc3": 0.03798172502754968, | |
| 339 | + "p_hc3": 0.04537489488193291, | |
| 340 | + "se_cl": 0.02898506811408183, | |
| 341 | + "p_cl": 0.008733802916670029 | |
| 342 | + }, | |
| 343 | + "v_land": { | |
| 344 | + "b": 0.025364605559571263, | |
| 345 | + "se_hc3": 0.004436697037848911, | |
| 346 | + "p_hc3": 1.0841988159731501e-08, | |
| 347 | + "se_cl": 0.004475099360773844, | |
| 348 | + "p_cl": 1.4452250155571873e-08 | |
| 349 | + }, | |
| 350 | + "v_cond": { | |
| 351 | + "b": 0.03275330527845849, | |
| 352 | + "se_hc3": 0.008042409776296826, | |
| 353 | + "p_hc3": 4.649651177320347e-05, | |
| 354 | + "se_cl": 0.00741253911271998, | |
| 355 | + "p_cl": 9.932619742557207e-06 | |
| 356 | + } | |
| 357 | + } | |
| 358 | + }, | |
| 359 | + "ablationA_insample": { | |
| 360 | + "base": 0.7168982211740981, | |
| 361 | + "plus_text": 0.7252251732782795, | |
| 362 | + "plus_text_vision": 0.7552103110293896 | |
| 363 | + }, | |
| 364 | + "city_fe_top": [ | |
| 365 | + [ | |
| 366 | + "westmount", | |
| 367 | + 230.7041938252226 | |
| 368 | + ], | |
| 369 | + [ | |
| 370 | + "hampstead", | |
| 371 | + 173.35777258613155 | |
| 372 | + ], | |
| 373 | + [ | |
| 374 | + "mont-royal", | |
| 375 | + 168.59619659817113 | |
| 376 | + ], | |
| 377 | + [ | |
| 378 | + "sainte-anne-des-plaines", | |
| 379 | + 142.9419234222625 | |
| 380 | + ], | |
| 381 | + [ | |
| 382 | + "montr\u00e9al", | |
| 383 | + 83.5002446059172 | |
| 384 | + ], | |
| 385 | + [ | |
| 386 | + "saint-lambert", | |
| 387 | + 81.58672431832005 | |
| 388 | + ], | |
| 389 | + [ | |
| 390 | + "beaconsfield", | |
| 391 | + 80.76456609139153 | |
| 392 | + ], | |
| 393 | + [ | |
| 394 | + "c\u00f4te-saint-luc", | |
| 395 | + 76.87439413605533 | |
| 396 | + ], | |
| 397 | + [ | |
| 398 | + "saint-bruno-de-montarville", | |
| 399 | + 70.85936417552094 | |
| 400 | + ], | |
| 401 | + [ | |
| 402 | + "dorval", | |
| 403 | + 68.65665951427297 | |
| 404 | + ], | |
| 405 | + [ | |
| 406 | + "lac-brome", | |
| 407 | + 62.14364632969884 | |
| 408 | + ], | |
| 409 | + [ | |
| 410 | + "pointe-claire", | |
| 411 | + 55.677246677163005 | |
| 412 | + ] | |
| 413 | + ], | |
| 414 | + "city_fe_bot": [ | |
| 415 | + [ | |
| 416 | + "mansfield-et-pontefract", | |
| 417 | + -30.06708220027833 | |
| 418 | + ], | |
| 419 | + [ | |
| 420 | + "baie-comeau", | |
| 421 | + -29.663061228503672 | |
| 422 | + ], | |
| 423 | + [ | |
| 424 | + "t\u00e9miscouata-sur-le-lac", | |
| 425 | + -28.11245530496782 | |
| 426 | + ], | |
| 427 | + [ | |
| 428 | + "la tuque", | |
| 429 | + -25.71095774442922 | |
| 430 | + ], | |
| 431 | + [ | |
| 432 | + "saint-marc-des-carri\u00e8res", | |
| 433 | + -25.50454270323863 | |
| 434 | + ], | |
| 435 | + [ | |
| 436 | + "thetford mines", | |
| 437 | + -24.723126467416044 | |
| 438 | + ], | |
| 439 | + [ | |
| 440 | + "chandler", | |
| 441 | + -22.75037961497599 | |
| 442 | + ], | |
| 443 | + [ | |
| 444 | + "dolbeau-mistassini", | |
| 445 | + -22.659205442967977 | |
| 446 | + ] | |
| 447 | + ], | |
| 448 | + "cv": { | |
| 449 | + "ols_v4": { | |
| 450 | + "r2": 0.6277548901390929, | |
| 451 | + "mdae": 17.87702738532425, | |
| 452 | + "w15": 42.92213564817236, | |
| 453 | + "w25": 64.46687235266432 | |
| 454 | + }, | |
| 455 | + "ols_v5": { | |
| 456 | + "r2": 0.6862361284588729, | |
| 457 | + "mdae": 16.88625054752524, | |
| 458 | + "w15": 45.25440568948387, | |
| 459 | + "w25": 66.72070281859541 | |
| 460 | + }, | |
| 461 | + "gbm_v4": { | |
| 462 | + "r2": 0.7417951731584451, | |
| 463 | + "mdae": 15.490609148904856, | |
| 464 | + "w15": 48.826021021806206, | |
| 465 | + "w25": 69.8687444438634 | |
| 466 | + }, | |
| 467 | + "gbm_v5": { | |
| 468 | + "r2": 0.7688886936527248, | |
| 469 | + "mdae": 14.68537838434596, | |
| 470 | + "w15": 50.75040527113946, | |
| 471 | + "w25": 71.32249124091409 | |
| 472 | + }, | |
| 473 | + "rf_v4": { | |
| 474 | + "r2": 0.6765341495717297, | |
| 475 | + "mdae": 17.899891187986572, | |
| 476 | + "w15": 42.84369607279193, | |
| 477 | + "w25": 64.13219683104116 | |
| 478 | + }, | |
| 479 | + "rf_v5": { | |
| 480 | + "r2": 0.7039491915758698, | |
| 481 | + "mdae": 17.201324709196374, | |
| 482 | + "w15": 44.77330962715055, | |
| 483 | + "w25": 65.63823667834545 | |
| 484 | + } | |
| 485 | + } | |
| 486 | +} | |
| \ No newline at end of file | ||
added
scripts/00a_test_openai_extraction.py
+71 −0
@@ -0,0 +1,71 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Test : extraction de specs via GPT-4o-mini (API REST, sans dépendance).""" | |
| 4 | +import json, os, urllib.request, urllib.error | |
| 5 | + | |
| 6 | +# charge la clé depuis .env | |
| 7 | +for line in open(".env", encoding="utf-8"): | |
| 8 | + if line.startswith("OPENAI_API_KEY="): | |
| 9 | + os.environ["OPENAI_API_KEY"] = line.split("=", 1)[1].strip() | |
| 10 | +KEY = os.environ["OPENAI_API_KEY"] | |
| 11 | + | |
| 12 | +SCHEMA = { | |
| 13 | + "type": "object", | |
| 14 | + "additionalProperties": False, | |
| 15 | + "properties": { | |
| 16 | + "condition": {"type": "string", "enum": ["a_renover","original","bon","renove","refait_neuf","neuf","inconnu"]}, | |
| 17 | + "turnkey": {"type": "boolean"}, | |
| 18 | + "recently_renovated": {"type": "boolean"}, | |
| 19 | + "flooring": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"]}}, | |
| 20 | + "countertop": {"type": "string", "enum": ["granit","quartz","marbre","stratifie","bois","autre","inconnu"]}, | |
| 21 | + "heating": {"type": "array", "items": {"type": "string", "enum": ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"]}}, | |
| 22 | + "garage": {"type": "boolean"}, | |
| 23 | + "pool": {"type": "boolean"}, | |
| 24 | + "waterfront": {"type": "boolean"}, | |
| 25 | + "view": {"type": "string", "enum": ["eau","montagne","ville","aucune","inconnu"]}, | |
| 26 | + "finished_basement":{"type": "boolean"}, | |
| 27 | + "fireplace": {"type": "boolean"}, | |
| 28 | + "central_ac": {"type": "boolean"}, | |
| 29 | + "new_construction":{"type": "boolean"}, | |
| 30 | + "income_property": {"type": "boolean"}, | |
| 31 | + "highlights": {"type": "array", "items": {"type": "string"}, "description": "3-6 mots-clés de vente saillants"} | |
| 32 | + }, | |
| 33 | + "required": ["condition","turnkey","recently_renovated","flooring","countertop", | |
| 34 | + "heating","garage","pool","waterfront","view","finished_basement", | |
| 35 | + "fireplace","central_ac","new_construction","income_property","highlights"] | |
| 36 | +} | |
| 37 | + | |
| 38 | +SYS = ("Tu es un extracteur d'attributs immobiliers. À partir de la description " | |
| 39 | + "d'une propriété (français ou anglais), renvoie les specs au format JSON " | |
| 40 | + "strict. Base-toi UNIQUEMENT sur le texte ; si une info est absente, mets " | |
| 41 | + "false / 'inconnu' / [] selon le champ. highlights = 3 à 6 mots-clés.") | |
| 42 | + | |
| 43 | +def extract(desc): | |
| 44 | + body = { | |
| 45 | + "model": "gpt-4o-mini", | |
| 46 | + "temperature": 0, | |
| 47 | + "messages": [{"role": "system", "content": SYS}, | |
| 48 | + {"role": "user", "content": desc[:4000]}], | |
| 49 | + "response_format": {"type": "json_schema", "json_schema": | |
| 50 | + {"name": "specs", "strict": True, "schema": SCHEMA}}, | |
| 51 | + } | |
| 52 | + req = urllib.request.Request( | |
| 53 | + "https://api.openai.com/v1/chat/completions", | |
| 54 | + data=json.dumps(body).encode(), | |
| 55 | + headers={"Authorization": f"Bearer {KEY}", | |
| 56 | + "Content-Type": "application/json"}) | |
| 57 | + with urllib.request.urlopen(req, timeout=60) as r: | |
| 58 | + out = json.load(r) | |
| 59 | + return out["choices"][0]["message"]["content"], out.get("usage") | |
| 60 | + | |
| 61 | +# 3 descriptions du dataset | |
| 62 | +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8")) | |
| 63 | +descs = [d["description"] for d in data if d.get("description")][:3] | |
| 64 | +for i, dsc in enumerate(descs): | |
| 65 | + try: | |
| 66 | + content, usage = extract(dsc) | |
| 67 | + print(f"\n=== Exemple {i+1} (tokens: {usage}) ===") | |
| 68 | + print("DESC:", dsc[:160].replace("\n", " "), "...") | |
| 69 | + print("SPECS:", json.dumps(json.loads(content), ensure_ascii=False)) | |
| 70 | + except urllib.error.HTTPError as e: | |
| 71 | + print("HTTP", e.code, e.read().decode()[:400]); break | |
added
scripts/00b_test_vision_claude.py
+65 −0
@@ -0,0 +1,65 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Test : analyse d'images immobilières via Claude Haiku 4.5 (REST, sans SDK).""" | |
| 4 | +import json, os, urllib.request, urllib.error | |
| 5 | + | |
| 6 | +for line in open(".env", encoding="utf-8"): | |
| 7 | + if line.startswith("ANTHROPIC_API_KEY="): | |
| 8 | + KEY = line.split("=", 1)[1].strip() | |
| 9 | + | |
| 10 | +SCHEMA = { | |
| 11 | + "type": "object", "additionalProperties": False, | |
| 12 | + "properties": { | |
| 13 | + "shot_type": {"type": "string", "enum": ["exterieur_facade","cuisine","salle_bain","salon","chambre","sous_sol","cour_terrain","vue","autre_interieur","autre"]}, | |
| 14 | + "condition_score": {"type": "integer", "description": "1=très délabré, 10=neuf/luxueux"}, | |
| 15 | + "renovation_level": {"type": "string", "enum": ["a_renover","date","standard","renove","haut_de_gamme","inconnu"]}, | |
| 16 | + "natural_light": {"type": "string", "enum": ["sombre","moyen","lumineux","tres_lumineux"]}, | |
| 17 | + "clutter": {"type": "string", "enum": ["vide","epure","normal","encombre"]}, | |
| 18 | + "curb_appeal": {"type": "integer", "description": "attrait extérieur 1-10; 0 si non applicable"}, | |
| 19 | + "materials": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","granit_quartz","stainless","brique","pierre","vinyle","beton","tapis"]}}, | |
| 20 | + "view_quality": {"type": "string", "enum": ["aucune","urbaine","verdure","eau","montagne","inconnu"]}, | |
| 21 | + "staged": {"type": "boolean", "description": "mise en scène pro / home staging"}, | |
| 22 | + "photo_quality": {"type": "integer", "description": "qualité de la photo elle-même 1-10"} | |
| 23 | + }, | |
| 24 | + "required": ["shot_type","condition_score","renovation_level","natural_light","clutter","curb_appeal","materials","view_quality","staged","photo_quality"] | |
| 25 | +} | |
| 26 | +SYS = ("Tu es un évaluateur immobilier objectif. Analyse la photo et renvoie des " | |
| 27 | + "attributs FACTUELS et reproductibles au format JSON. Sois objectif : base " | |
| 28 | + "les scores sur des critères concrets (état des surfaces, finitions, " | |
| 29 | + "luminosité), pas sur un goût personnel.") | |
| 30 | + | |
| 31 | +def analyze(url): | |
| 32 | + body = {"model": "claude-haiku-4-5", "max_tokens": 400, | |
| 33 | + "system": SYS, | |
| 34 | + "messages": [{"role": "user", "content": [ | |
| 35 | + {"type": "image", "source": {"type": "url", "url": url}}, | |
| 36 | + {"type": "text", "text": "Analyse cette photo de propriété."}]}], | |
| 37 | + "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}} | |
| 38 | + req = urllib.request.Request( | |
| 39 | + "https://api.anthropic.com/v1/messages", data=json.dumps(body).encode(), | |
| 40 | + headers={"x-api-key": KEY, "anthropic-version": "2023-06-01", | |
| 41 | + "content-type": "application/json"}) | |
| 42 | + with urllib.request.urlopen(req, timeout=90) as r: | |
| 43 | + out = json.load(r) | |
| 44 | + txt = next(b["text"] for b in out["content"] if b["type"] == "text") | |
| 45 | + return json.loads(txt), out.get("usage") | |
| 46 | + | |
| 47 | +data_r = json.load(open("data/raw/dataset_realtor-property-crawler_2026-07-03_10-36-26-001.json", encoding="utf-8")) | |
| 48 | +data_d = json.load(open("data/raw/dataset_duproprio-property-crawler_2026-07-03_00-37-49-449.json", encoding="utf-8")) | |
| 49 | +# realtor: 1re image = façade ; duproprio: image dont caption ~ Kitchen | |
| 50 | +tests = [] | |
| 51 | +r0 = next(d for d in data_r if d.get("images")) | |
| 52 | +tests.append(("REALTOR 1re photo (façade attendue)", r0["images"][0]["url"])) | |
| 53 | +d0 = next(d for d in data_d if d.get("images")) | |
| 54 | +kit = next((im for im in d0["images"] if "kitchen" in (im.get("caption") or "").lower()), d0["images"][0]) | |
| 55 | +tests.append(("DUPROPRIO (caption=%s)" % (kit.get("caption","?")[:30]), kit["url"])) | |
| 56 | + | |
| 57 | +for label, url in tests: | |
| 58 | + try: | |
| 59 | + specs, usage = analyze(url) | |
| 60 | + print(f"\n=== {label} ===") | |
| 61 | + print("URL:", url[:90]) | |
| 62 | + print("tokens:", usage) | |
| 63 | + print(json.dumps(specs, ensure_ascii=False, indent=1)) | |
| 64 | + except urllib.error.HTTPError as e: | |
| 65 | + print("HTTP", e.code, e.read().decode()[:500]) | |
added
scripts/01_extract_specs_llm.py
+135 −0
@@ -0,0 +1,135 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +""" | |
| 4 | +Extraction des specs de propriété via GPT-4o-mini, à partir des descriptions. | |
| 5 | +Parallèle, résumable (cache specs.jsonl), avec retry/backoff. Aucune dépendance. | |
| 6 | +""" | |
| 7 | +import json, os, time, threading, urllib.request, urllib.error | |
| 8 | +from concurrent.futures import ThreadPoolExecutor, as_completed | |
| 9 | + | |
| 10 | +SRC = "data/processed/maisons_avec_prix.json" | |
| 11 | +OUT = "data/processed/specs.jsonl" | |
| 12 | +WORKERS = int(os.environ.get("WORKERS", "24")) | |
| 13 | +NSHARDS = int(os.environ.get("NSHARDS", "1")) | |
| 14 | +SHARD = int(os.environ.get("SHARD", "0")) | |
| 15 | +MODEL = "gpt-4o-mini" | |
| 16 | + | |
| 17 | +# clé | |
| 18 | +for line in open(".env", encoding="utf-8"): | |
| 19 | + if line.startswith("OPENAI_API_KEY="): | |
| 20 | + os.environ["OPENAI_API_KEY"] = line.split("=", 1)[1].strip() | |
| 21 | +KEY = os.environ["OPENAI_API_KEY"] | |
| 22 | + | |
| 23 | +SCHEMA = { | |
| 24 | + "type": "object", "additionalProperties": False, | |
| 25 | + "properties": { | |
| 26 | + "condition": {"type": "string", "enum": ["a_renover","original","bon","renove","refait_neuf","neuf","inconnu"]}, | |
| 27 | + "turnkey": {"type": "boolean"}, | |
| 28 | + "recently_renovated": {"type": "boolean"}, | |
| 29 | + "flooring": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"]}}, | |
| 30 | + "countertop": {"type": "string", "enum": ["granit","quartz","marbre","stratifie","bois","autre","inconnu"]}, | |
| 31 | + "heating": {"type": "array", "items": {"type": "string", "enum": ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"]}}, | |
| 32 | + "garage": {"type": "boolean"}, | |
| 33 | + "pool": {"type": "boolean"}, | |
| 34 | + "waterfront": {"type": "boolean"}, | |
| 35 | + "view": {"type": "string", "enum": ["eau","montagne","ville","aucune","inconnu"]}, | |
| 36 | + "finished_basement": {"type": "boolean"}, | |
| 37 | + "fireplace": {"type": "boolean"}, | |
| 38 | + "central_ac": {"type": "boolean"}, | |
| 39 | + "new_construction": {"type": "boolean"}, | |
| 40 | + "income_property": {"type": "boolean"}, | |
| 41 | + "renovated_kitchen": {"type": "boolean"}, | |
| 42 | + "renovated_bathroom": {"type": "boolean"}, | |
| 43 | + "large_lot": {"type": "boolean"}, | |
| 44 | + "quiet_area": {"type": "boolean"}, | |
| 45 | + "highlights": {"type": "array", "items": {"type": "string"}} | |
| 46 | + }, | |
| 47 | + "required": ["condition","turnkey","recently_renovated","flooring","countertop", | |
| 48 | + "heating","garage","pool","waterfront","view","finished_basement", | |
| 49 | + "fireplace","central_ac","new_construction","income_property", | |
| 50 | + "renovated_kitchen","renovated_bathroom","large_lot","quiet_area", | |
| 51 | + "highlights"] | |
| 52 | +} | |
| 53 | +SYS = ("Tu es un extracteur d'attributs immobiliers. À partir de la description " | |
| 54 | + "d'une propriété (français ou anglais), renvoie les specs au format JSON " | |
| 55 | + "strict. Base-toi UNIQUEMENT sur le texte ; si une info est absente, mets " | |
| 56 | + "false / 'inconnu' / [] selon le champ. highlights = 3 à 6 mots-clés de vente.") | |
| 57 | + | |
| 58 | +def call(desc, tries=5): | |
| 59 | + body = {"model": MODEL, "temperature": 0, | |
| 60 | + "messages": [{"role": "system", "content": SYS}, | |
| 61 | + {"role": "user", "content": desc[:4000]}], | |
| 62 | + "response_format": {"type": "json_schema", "json_schema": | |
| 63 | + {"name": "specs", "strict": True, "schema": SCHEMA}}} | |
| 64 | + data = json.dumps(body).encode() | |
| 65 | + for t in range(tries): | |
| 66 | + try: | |
| 67 | + req = urllib.request.Request( | |
| 68 | + "https://api.openai.com/v1/chat/completions", data=data, | |
| 69 | + headers={"Authorization": f"Bearer {KEY}", | |
| 70 | + "Content-Type": "application/json"}) | |
| 71 | + with urllib.request.urlopen(req, timeout=90) as r: | |
| 72 | + out = json.load(r) | |
| 73 | + return json.loads(out["choices"][0]["message"]["content"]) | |
| 74 | + except urllib.error.HTTPError as e: | |
| 75 | + if e.code in (429, 500, 502, 503, 504) and t < tries - 1: | |
| 76 | + time.sleep(2 ** t + 0.5); continue | |
| 77 | + raise | |
| 78 | + except (urllib.error.URLError, TimeoutError): | |
| 79 | + if t < tries - 1: | |
| 80 | + time.sleep(2 ** t + 0.5); continue | |
| 81 | + raise | |
| 82 | + return None | |
| 83 | + | |
| 84 | +# charge données + cache | |
| 85 | +data = json.load(open(SRC, encoding="utf-8")) | |
| 86 | +done = set() | |
| 87 | +if os.path.exists(OUT): | |
| 88 | + for line in open(OUT, encoding="utf-8"): | |
| 89 | + try: | |
| 90 | + done.add(json.loads(line)["uid"]) | |
| 91 | + except Exception: | |
| 92 | + pass | |
| 93 | + | |
| 94 | +todo = [] | |
| 95 | +for d in data: | |
| 96 | + uid = f"{d.get('source')}:{d.get('listingId')}" | |
| 97 | + desc = d.get("description") | |
| 98 | + if uid in done or not desc: | |
| 99 | + continue | |
| 100 | + todo.append((uid, desc)) | |
| 101 | + | |
| 102 | +# sharding : partition disjointe et complète par index (déterministe) | |
| 103 | +if NSHARDS > 1: | |
| 104 | + todo = [t for i, t in enumerate(todo) if i % NSHARDS == SHARD] | |
| 105 | + | |
| 106 | +print(f"[shard {SHARD}/{NSHARDS}, workers={WORKERS}] Total={len(data)} " | |
| 107 | + f"déjà fait={len(done)} à traiter (ce shard)={len(todo)}", flush=True) | |
| 108 | + | |
| 109 | +lock = threading.Lock() | |
| 110 | +counter = {"n": 0, "err": 0} | |
| 111 | +fh = open(OUT, "a", encoding="utf-8") | |
| 112 | + | |
| 113 | +def work(item): | |
| 114 | + uid, desc = item | |
| 115 | + try: | |
| 116 | + specs = call(desc) | |
| 117 | + rec = {"uid": uid, "specs": specs} | |
| 118 | + except Exception as e: | |
| 119 | + rec = {"uid": uid, "error": str(e)[:200]} | |
| 120 | + with lock: | |
| 121 | + fh.write(json.dumps(rec, ensure_ascii=False) + "\n") | |
| 122 | + fh.flush() | |
| 123 | + counter["n"] += 1 | |
| 124 | + if "error" in rec: | |
| 125 | + counter["err"] += 1 | |
| 126 | + if counter["n"] % 250 == 0: | |
| 127 | + print(f" {counter['n']}/{len(todo)} (err={counter['err']})", | |
| 128 | + flush=True) | |
| 129 | + | |
| 130 | +t0 = time.time() | |
| 131 | +with ThreadPoolExecutor(max_workers=WORKERS) as ex: | |
| 132 | + list(ex.map(work, todo)) | |
| 133 | +fh.close() | |
| 134 | +print(f"Terminé : {counter['n']} traités, {counter['err']} erreurs, " | |
| 135 | + f"{time.time()-t0:.0f}s", flush=True) | |
added
scripts/02_facade_vision.py
+135 −0
@@ -0,0 +1,135 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +""" | |
| 4 | +Analyse de la photo de façade (1/annonce) via Claude Haiku 4.5. | |
| 5 | +Attributs visuels OBJECTIFS. Parallèle, résumable (facade_specs.jsonl), retry. | |
| 6 | +""" | |
| 7 | +import json, os, time, threading, urllib.request, urllib.error | |
| 8 | +from concurrent.futures import ThreadPoolExecutor | |
| 9 | + | |
| 10 | +SRC = "data/processed/maisons_avec_prix.json" | |
| 11 | +OUT = "data/processed/facade_specs.jsonl" | |
| 12 | +WORKERS = int(os.environ.get("WORKERS", "16")) | |
| 13 | +NSHARDS = int(os.environ.get("NSHARDS", "1")) | |
| 14 | +SHARD = int(os.environ.get("SHARD", "0")) | |
| 15 | +MODEL = "claude-haiku-4-5" | |
| 16 | + | |
| 17 | +for line in open(".env", encoding="utf-8"): | |
| 18 | + if line.startswith("ANTHROPIC_API_KEY="): | |
| 19 | + KEY = line.split("=", 1)[1].strip() | |
| 20 | + | |
| 21 | +SCHEMA = { | |
| 22 | + "type": "object", "additionalProperties": False, | |
| 23 | + "properties": { | |
| 24 | + "is_exterior": {"type": "boolean", "description": "la photo montre-t-elle l'extérieur/façade"}, | |
| 25 | + "building_type_visual": {"type": "string", "enum": ["maison_detachee","jumele","rangee","condo_immeuble","chalet","terrain","autre"]}, | |
| 26 | + "condition_score": {"type": "integer", "description": "état visuel 1=délabré à 10=neuf/luxueux"}, | |
| 27 | + "curb_appeal": {"type": "integer", "description": "attrait extérieur 1-10"}, | |
| 28 | + "renovation_level": {"type": "string", "enum": ["a_renover","date","standard","renove","haut_de_gamme","inconnu"]}, | |
| 29 | + "exterior_materials": {"type": "array", "items": {"type": "string", "enum": ["brique","pierre","vinyle","bois","aluminium","stucco","crepi","beton","autre"]}}, | |
| 30 | + "landscaping": {"type": "string", "enum": ["aucun","minimal","entretenu","amenage_soigne"]}, | |
| 31 | + "has_garage_visible": {"type": "boolean"}, | |
| 32 | + "storeys_visual": {"type": "string", "enum": ["1","1.5","2","3+","inconnu"]}, | |
| 33 | + "setting": {"type": "string", "enum": ["urbain_dense","banlieue","rural","bord_eau","boise","inconnu"]}, | |
| 34 | + "photo_quality": {"type": "integer", "description": "qualité de la photo 1-10"}, | |
| 35 | + "curb_luxury": {"type": "boolean", "description": "signes de luxe (grande taille, matériaux nobles, prestige)"} | |
| 36 | + }, | |
| 37 | + "required": ["is_exterior","building_type_visual","condition_score","curb_appeal", | |
| 38 | + "renovation_level","exterior_materials","landscaping","has_garage_visible", | |
| 39 | + "storeys_visual","setting","photo_quality","curb_luxury"] | |
| 40 | +} | |
| 41 | +SYS = ("Tu es un évaluateur immobilier objectif. Analyse la photo (censée être la " | |
| 42 | + "façade d'une propriété) et renvoie des attributs FACTUELS et reproductibles " | |
| 43 | + "au format JSON. Fonde les scores sur des critères concrets (état des " | |
| 44 | + "surfaces, matériaux, aménagement), pas sur un goût personnel. Si la photo " | |
| 45 | + "n'est pas un extérieur, mets is_exterior=false et remplis au mieux.") | |
| 46 | + | |
| 47 | +ROOM_KW = ("kitchen","cuisine","bathroom","salle de bain","bedroom","chambre", | |
| 48 | + "living","salon","basement","sous-sol","dining","salle à manger", | |
| 49 | + "garage","office","bureau","laundry","closet","stair") | |
| 50 | + | |
| 51 | +def pick_facade(d): | |
| 52 | + imgs = d.get("images") or [] | |
| 53 | + if not imgs: | |
| 54 | + return None | |
| 55 | + if d.get("source") == "realtor": | |
| 56 | + return imgs[0].get("url") # 1re photo = extérieur | |
| 57 | + # duproprio : caption = adresse (sans mot-clé de pièce) = photo extérieure | |
| 58 | + for im in imgs: | |
| 59 | + cap = (im.get("caption") or "").lower() | |
| 60 | + if cap and not any(k in cap for k in ROOM_KW): | |
| 61 | + return im.get("url") | |
| 62 | + return imgs[0].get("url") | |
| 63 | + | |
| 64 | +def call(url, tries=5): | |
| 65 | + body = {"model": MODEL, "max_tokens": 400, "system": SYS, | |
| 66 | + "messages": [{"role": "user", "content": [ | |
| 67 | + {"type": "image", "source": {"type": "url", "url": url}}, | |
| 68 | + {"type": "text", "text": "Analyse cette photo de propriété."}]}], | |
| 69 | + "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}} | |
| 70 | + data = json.dumps(body).encode() | |
| 71 | + for t in range(tries): | |
| 72 | + try: | |
| 73 | + req = urllib.request.Request( | |
| 74 | + "https://api.anthropic.com/v1/messages", data=data, | |
| 75 | + headers={"x-api-key": KEY, "anthropic-version": "2023-06-01", | |
| 76 | + "content-type": "application/json"}) | |
| 77 | + with urllib.request.urlopen(req, timeout=120) as r: | |
| 78 | + out = json.load(r) | |
| 79 | + txt = next(b["text"] for b in out["content"] if b["type"] == "text") | |
| 80 | + return json.loads(txt) | |
| 81 | + except urllib.error.HTTPError as e: | |
| 82 | + # 400 inclus : fetch d'URL image transitoire côté Anthropic sous charge | |
| 83 | + if e.code in (400, 429, 500, 502, 503, 504, 529) and t < tries - 1: | |
| 84 | + time.sleep(2 ** t + 1); continue | |
| 85 | + raise | |
| 86 | + except (urllib.error.URLError, TimeoutError, ValueError): | |
| 87 | + if t < tries - 1: | |
| 88 | + time.sleep(2 ** t + 1); continue | |
| 89 | + raise | |
| 90 | + return None | |
| 91 | + | |
| 92 | +data = json.load(open(SRC, encoding="utf-8")) | |
| 93 | +done = set() | |
| 94 | +if os.path.exists(OUT): | |
| 95 | + for line in open(OUT, encoding="utf-8"): | |
| 96 | + try: done.add(json.loads(line)["uid"]) | |
| 97 | + except Exception: pass | |
| 98 | + | |
| 99 | +todo = [] | |
| 100 | +for d in data: | |
| 101 | + uid = f"{d.get('source')}:{d.get('listingId')}" | |
| 102 | + if uid in done: | |
| 103 | + continue | |
| 104 | + url = pick_facade(d) | |
| 105 | + if url: | |
| 106 | + todo.append((uid, url)) | |
| 107 | + | |
| 108 | +if NSHARDS > 1: | |
| 109 | + todo = [t for i, t in enumerate(todo) if i % NSHARDS == SHARD] | |
| 110 | + | |
| 111 | +print(f"[shard {SHARD}/{NSHARDS}, workers={WORKERS}] Total={len(data)} " | |
| 112 | + f"déjà fait={len(done)} à traiter (ce shard)={len(todo)}", flush=True) | |
| 113 | + | |
| 114 | +lock = threading.Lock() | |
| 115 | +c = {"n": 0, "err": 0} | |
| 116 | +fh = open(OUT, "a", encoding="utf-8") | |
| 117 | + | |
| 118 | +def work(item): | |
| 119 | + uid, url = item | |
| 120 | + try: | |
| 121 | + rec = {"uid": uid, "facade": call(url)} | |
| 122 | + except Exception as e: | |
| 123 | + rec = {"uid": uid, "error": str(e)[:200]} | |
| 124 | + with lock: | |
| 125 | + fh.write(json.dumps(rec, ensure_ascii=False) + "\n"); fh.flush() | |
| 126 | + c["n"] += 1 | |
| 127 | + if "error" in rec: c["err"] += 1 | |
| 128 | + if c["n"] % 250 == 0: | |
| 129 | + print(f" {c['n']}/{len(todo)} (err={c['err']})", flush=True) | |
| 130 | + | |
| 131 | +t0 = time.time() | |
| 132 | +with ThreadPoolExecutor(max_workers=WORKERS) as ex: | |
| 133 | + list(ex.map(work, todo)) | |
| 134 | +fh.close() | |
| 135 | +print(f"Terminé : {c['n']} traités, {c['err']} erreurs, {time.time()-t0:.0f}s", flush=True) | |
added
scripts/03_merge_llm_features.py
+84 −0
@@ -0,0 +1,84 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Fusionne les features GPT (specs.jsonl) aux données maison (maisons_avec_prix.json).""" | |
| 4 | +import json, csv | |
| 5 | + | |
| 6 | +# 1) charger specs GPT | |
| 7 | +specs = {} | |
| 8 | +for line in open("data/processed/specs.jsonl", encoding="utf-8"): | |
| 9 | + r = json.loads(line) | |
| 10 | + if r.get("specs"): | |
| 11 | + specs[r["uid"]] = r["specs"] | |
| 12 | +print(f"Specs GPT chargées : {len(specs)}") | |
| 13 | + | |
| 14 | +# 2) charger maisons | |
| 15 | +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8")) | |
| 16 | + | |
| 17 | +# 3) JSON fusionné : record complet + gptSpecs imbriqué | |
| 18 | +merged = [] | |
| 19 | +n_match = 0 | |
| 20 | +for d in data: | |
| 21 | + uid = f"{d.get('source')}:{d.get('listingId')}" | |
| 22 | + d2 = dict(d) | |
| 23 | + d2["uid"] = uid | |
| 24 | + s = specs.get(uid) | |
| 25 | + d2["gptSpecs"] = s # None si absent | |
| 26 | + if s: | |
| 27 | + n_match += 1 | |
| 28 | + merged.append(d2) | |
| 29 | +json.dump(merged, open("data/processed/maisons_specs.json", "w", encoding="utf-8"), | |
| 30 | + ensure_ascii=False) | |
| 31 | +print(f"JSON fusionné -> maisons_specs.json ({len(merged)} records, " | |
| 32 | + f"{n_match} avec specs)") | |
| 33 | + | |
| 34 | +# 4) CSV aplati pour analyse | |
| 35 | +BOOL = ["turnkey","recently_renovated","garage","pool","waterfront", | |
| 36 | + "finished_basement","fireplace","central_ac","new_construction", | |
| 37 | + "income_property","renovated_kitchen","renovated_bathroom", | |
| 38 | + "large_lot","quiet_area"] | |
| 39 | +FLOOR = ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"] | |
| 40 | +HEAT = ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"] | |
| 41 | + | |
| 42 | +cols = (["uid","source","listingId","price","currency","city","province", | |
| 43 | + "ptype","sqft_raw","lot_raw","beds","baths","yearBuilt","parking", | |
| 44 | + "has_gpt","g_condition","g_countertop","g_view"] | |
| 45 | + + ["g_"+b for b in BOOL] | |
| 46 | + + ["fl_"+f for f in FLOOR] | |
| 47 | + + ["heat_"+h for h in HEAT] | |
| 48 | + + ["g_highlights"]) | |
| 49 | + | |
| 50 | +with open("data/processed/maisons_specs.csv", "w", newline="", encoding="utf-8") as fh: | |
| 51 | + w = csv.writer(fh) | |
| 52 | + w.writerow(cols) | |
| 53 | + for d in data: | |
| 54 | + uid = f"{d.get('source')}:{d.get('listingId')}" | |
| 55 | + p = d.get("property") or {} | |
| 56 | + ad = d.get("address") or {} | |
| 57 | + s = specs.get(uid) | |
| 58 | + row = [uid, d.get("source"), d.get("listingId"), d.get("priceNumber"), | |
| 59 | + d.get("currency"), ad.get("city"), ad.get("province"), | |
| 60 | + (p.get("buildingType") or p.get("type")), | |
| 61 | + p.get("squareFeet"), p.get("lotSize"), | |
| 62 | + p.get("bedrooms"), p.get("bathrooms"), p.get("yearBuilt"), | |
| 63 | + p.get("parkingSpaces"), | |
| 64 | + 1 if s else 0, | |
| 65 | + (s or {}).get("condition",""), | |
| 66 | + (s or {}).get("countertop",""), | |
| 67 | + (s or {}).get("view","")] | |
| 68 | + row += [int(bool((s or {}).get(b))) if s else "" for b in BOOL] | |
| 69 | + fl = set((s or {}).get("flooring") or []) | |
| 70 | + row += [int(f in fl) if s else "" for f in FLOOR] | |
| 71 | + ht = set((s or {}).get("heating") or []) | |
| 72 | + row += [int(h in ht) if s else "" for h in HEAT] | |
| 73 | + row += ["; ".join((s or {}).get("highlights") or [])] | |
| 74 | + w.writerow(row) | |
| 75 | +print(f"CSV aplati -> maisons_specs.csv ({len(cols)} colonnes)") | |
| 76 | + | |
| 77 | +# 5) petit récap de distribution | |
| 78 | +import collections | |
| 79 | +cond = collections.Counter(s.get("condition") for s in specs.values()) | |
| 80 | +print("\nRépartition 'condition' :", dict(cond)) | |
| 81 | +for b in ["turnkey","recently_renovated","central_ac","pool","waterfront", | |
| 82 | + "renovated_kitchen","income_property"]: | |
| 83 | + n = sum(1 for s in specs.values() if s.get(b)) | |
| 84 | + print(f" {b:20s}: {n:5d} ({n/len(specs)*100:.1f}%)") | |
added
scripts/04_hedonic_ols_base.py
+338 −0
@@ -0,0 +1,338 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +""" | |
| 4 | +Nettoyage des aberrations + modèle hédonique de prix des maisons | |
| 5 | +(Realtor + DuProprio combinés). Log-linéaire, effets fixes ville, SE robustes. | |
| 6 | +""" | |
| 7 | +import json, re, warnings | |
| 8 | +import numpy as np | |
| 9 | +import pandas as pd | |
| 10 | +import statsmodels.formula.api as smf | |
| 11 | + | |
| 12 | +warnings.filterwarnings("ignore") | |
| 13 | +SRC = "data/processed/maisons_avec_prix.json" | |
| 14 | +CUR_YEAR = 2026 | |
| 15 | + | |
| 16 | +# ---------- Chargement ---------- | |
| 17 | +with open(SRC, encoding="utf-8") as f: | |
| 18 | + data = json.load(f) | |
| 19 | +print(f"Chargé : {len(data)} enregistrements") | |
| 20 | + | |
| 21 | +# ---------- Parsers ---------- | |
| 22 | +def parse_area(s): | |
| 23 | + """Retourne une superficie en pieds carrés (ft²).""" | |
| 24 | + if not s or not isinstance(s, str): | |
| 25 | + return np.nan | |
| 26 | + txt = s.lower() | |
| 27 | + is_m2 = ("m2" in txt) or ("m²" in txt) | |
| 28 | + # garder chiffres, points, virgules, espaces ; retirer unités | |
| 29 | + num = re.sub(r"[^0-9.,\s]", "", txt).strip() | |
| 30 | + num = num.replace(" ", "").replace(",", "") # séparateurs de milliers | |
| 31 | + try: | |
| 32 | + v = float(num) | |
| 33 | + except ValueError: | |
| 34 | + return np.nan | |
| 35 | + if v <= 0: | |
| 36 | + return np.nan | |
| 37 | + return v * 10.7639 if is_m2 else v | |
| 38 | + | |
| 39 | +def clean_city(city, source): | |
| 40 | + if not city or not isinstance(city, str): | |
| 41 | + return None | |
| 42 | + c = city.split("(")[0].strip() # retire quartiers realtor | |
| 43 | + c = re.sub(r"\s+", " ", c).lower().strip() | |
| 44 | + return c or None | |
| 45 | + | |
| 46 | +def to_int(x): | |
| 47 | + try: | |
| 48 | + return int(x) | |
| 49 | + except (TypeError, ValueError): | |
| 50 | + return np.nan | |
| 51 | + | |
| 52 | +# amenity keywords -> recherche dans features + description + title (bilingue) | |
| 53 | +AMENITIES = { | |
| 54 | + "pool": ["swimming pool", "piscine", "inground pool", "pool"], | |
| 55 | + "waterfront": ["waterfront", "bord de l'eau", "navigable", "water access", | |
| 56 | + "lakefront", "riverfront", "au bord de", "waterfront"], | |
| 57 | + "garage": ["garage"], | |
| 58 | + "fireplace": ["fireplace", "foyer", "wood stove", "poêle"], | |
| 59 | + "central_air": ["central air", "air conditionné", "climatisation", "a/c", | |
| 60 | + "heat-pump", "thermopompe", "air exchanger"], | |
| 61 | + "finished_bsmt":["totally finished", "finished basement", "sous-sol fini", | |
| 62 | + "basement"], | |
| 63 | + "near_water": ["near water", "waterfront", "navigable"], | |
| 64 | + "cul_de_sac": ["cul-de-sac", "cul de sac"], | |
| 65 | + "golf": ["golf"], | |
| 66 | + "ski": ["ski"], | |
| 67 | + "near_park": ["park", "parc"], | |
| 68 | + "near_school": ["school", "école", "cegep", "university", "université", | |
| 69 | + "elementary", "high school"], | |
| 70 | + "near_highway":["highway", "autoroute"], | |
| 71 | + "transit": ["public transit", "public transportation", "transport"], | |
| 72 | + "no_rear_neighbor":["no backyard neighbors", "no rear neighbor", | |
| 73 | + "sans voisin arrière"], | |
| 74 | +} | |
| 75 | + | |
| 76 | +# ---------- Parsing des pièces (Realtor) ---------- | |
| 77 | +def _side_ft(side): | |
| 78 | + s = side.lower() | |
| 79 | + if "ft" in s: | |
| 80 | + ft = re.search(r"([\d.]+)\s*ft", s) | |
| 81 | + inch = re.search(r"([\d.]+)\s*in", s) | |
| 82 | + return (float(ft.group(1)) if ft else 0) + \ | |
| 83 | + (float(inch.group(1)) / 12 if inch else 0) | |
| 84 | + m = re.search(r"([\d.,]+)\s*m", s) | |
| 85 | + if m: | |
| 86 | + try: | |
| 87 | + return float(m.group(1).replace(",", ".")) * 3.28084 | |
| 88 | + except ValueError: | |
| 89 | + return None | |
| 90 | + return None | |
| 91 | + | |
| 92 | +def _room_area(dim): | |
| 93 | + if not dim or "x" not in dim: | |
| 94 | + return None | |
| 95 | + a, _, b = dim.partition("x") | |
| 96 | + la, lb = _side_ft(a), _side_ft(b) | |
| 97 | + if la and lb and 1 < la < 200 and 1 < lb < 200: | |
| 98 | + return la * lb | |
| 99 | + return None | |
| 100 | + | |
| 101 | +ROOM_FLAGS = { # pièces à valeur potentielle | |
| 102 | + "r_office": ["office"], | |
| 103 | + "r_family": ["family room"], | |
| 104 | + "r_workshop": ["workshop"], | |
| 105 | + "r_winecellar": ["wine cellar"], | |
| 106 | + "r_sunroom": ["sunroom", "solarium"], | |
| 107 | + "r_mezzanine": ["mezzanine"], | |
| 108 | + "r_library": ["library"], | |
| 109 | + "r_laundry": ["laundry"], | |
| 110 | +} | |
| 111 | + | |
| 112 | +def parse_rooms(rooms): | |
| 113 | + out = {"room_area": np.nan, "n_rooms": 0, "n_bed_r": 0, "n_bath_r": 0, | |
| 114 | + "has_primary_bed": 0, "has_basement": 0, "n_levels_r": 0} | |
| 115 | + for k in ROOM_FLAGS: | |
| 116 | + out[k] = 0 | |
| 117 | + if not rooms: | |
| 118 | + return out | |
| 119 | + area, levels = 0.0, set() | |
| 120 | + for r in rooms: | |
| 121 | + nm = (r.get("name") or "").lower() | |
| 122 | + lv = (r.get("level") or "").lower() | |
| 123 | + out["n_rooms"] += 1 | |
| 124 | + if lv: | |
| 125 | + levels.add(lv) | |
| 126 | + if "basement" in lv or "sous-sol" in lv: | |
| 127 | + out["has_basement"] = 1 | |
| 128 | + if "primary bedroom" in nm: | |
| 129 | + out["has_primary_bed"] = 1 | |
| 130 | + out["n_bed_r"] += 1 | |
| 131 | + elif "bedroom" in nm: | |
| 132 | + out["n_bed_r"] += 1 | |
| 133 | + if "bathroom" in nm: | |
| 134 | + out["n_bath_r"] += 1 | |
| 135 | + for k, kws in ROOM_FLAGS.items(): | |
| 136 | + if any(w in nm for w in kws): | |
| 137 | + out[k] = 1 | |
| 138 | + a = _room_area(r.get("dimensions")) | |
| 139 | + if a: | |
| 140 | + area += a | |
| 141 | + out["room_area"] = area if area > 0 else np.nan | |
| 142 | + out["n_levels_r"] = len(levels) | |
| 143 | + return out | |
| 144 | + | |
| 145 | +rows = [] | |
| 146 | +for d in data: | |
| 147 | + prop = d.get("property") or {} | |
| 148 | + addr = d.get("address") or {} | |
| 149 | + feats = d.get("features") or [] | |
| 150 | + blob = (" ".join(feats) + " " + (d.get("description") or "") + " " + | |
| 151 | + (d.get("title") or "")).lower() | |
| 152 | + | |
| 153 | + yb = to_int(prop.get("yearBuilt")) | |
| 154 | + age = (CUR_YEAR - yb) if (not pd.isna(yb) and 1700 <= yb <= CUR_YEAR) else np.nan | |
| 155 | + | |
| 156 | + ptype = prop.get("buildingType") or prop.get("type") # realtor/duproprio | |
| 157 | + storeys = prop.get("storeys") or prop.get("levels") | |
| 158 | + try: | |
| 159 | + storeys = float(str(storeys).replace(",", ".")) | |
| 160 | + except (TypeError, ValueError): | |
| 161 | + storeys = np.nan | |
| 162 | + | |
| 163 | + rec = { | |
| 164 | + "price": d.get("priceNumber"), | |
| 165 | + "source": d.get("source"), | |
| 166 | + "sqft": parse_area(prop.get("squareFeet")), | |
| 167 | + "lot_sqft": parse_area(prop.get("lotSize")), | |
| 168 | + "beds": to_int(prop.get("bedrooms")), | |
| 169 | + "baths": to_int(prop.get("bathrooms")), | |
| 170 | + "parking": to_int(prop.get("parkingSpaces")), # realtor seulement | |
| 171 | + "storeys": storeys, | |
| 172 | + "age": age, | |
| 173 | + "ptype": (ptype or "Unknown").strip(), | |
| 174 | + "city": clean_city(addr.get("city"), d.get("source")), | |
| 175 | + "region": (addr.get("region") or "").strip().lower() or None, | |
| 176 | + } | |
| 177 | + for name, kws in AMENITIES.items(): | |
| 178 | + rec[name] = int(any(k in blob for k in kws)) | |
| 179 | + # signaux "autre chose" : photos + visite virtuelle | |
| 180 | + rec["n_images"] = len(d.get("images") or []) | |
| 181 | + rec["virtual_tour"] = int(bool(d.get("virtualTourUrl"))) | |
| 182 | + # détail des pièces | |
| 183 | + rec.update(parse_rooms(d.get("rooms"))) | |
| 184 | + rows.append(rec) | |
| 185 | + | |
| 186 | +df = pd.DataFrame(rows) | |
| 187 | +print(f"DataFrame brut : {df.shape}") | |
| 188 | + | |
| 189 | +# ---------- Nettoyage des aberrations ---------- | |
| 190 | +n0 = len(df) | |
| 191 | +df = df[df["price"].notna()] | |
| 192 | +# prix plausibles pour une maison au Québec | |
| 193 | +df = df[(df["price"] >= 50_000) & (df["price"] <= 15_000_000)] | |
| 194 | +# superficie habitable : borner si présente, garder les NA (imputés + flag) | |
| 195 | +df.loc[(df["sqft"] < 300) | (df["sqft"] > 20_000), "sqft"] = np.nan | |
| 196 | +# terrain plausible (garder NA autorisé -> mais on impute plus bas) | |
| 197 | +df.loc[(df["lot_sqft"] < 200) | (df["lot_sqft"] > 5_000_000), "lot_sqft"] = np.nan | |
| 198 | +# chambres / sdb plausibles | |
| 199 | +df = df[(df["beds"].between(0, 15)) | (df["beds"].isna())] | |
| 200 | +df = df[(df["baths"].between(0, 12)) | (df["baths"].isna())] | |
| 201 | +df.loc[df["beds"] > 15, "beds"] = np.nan | |
| 202 | +df.loc[df["baths"] > 12, "baths"] = np.nan | |
| 203 | +# prix au pied carré aberrant (sur les lignes qui ont une superficie réelle) | |
| 204 | +df["ppsf"] = df["price"] / df["sqft"] | |
| 205 | +lo, hi = df["ppsf"].quantile([0.005, 0.995]) | |
| 206 | +bad = df["ppsf"].notna() & ((df["ppsf"] < lo) | (df["ppsf"] > hi)) | |
| 207 | +df = df[~bad] | |
| 208 | +print(f"Après nettoyage : {len(df)} (retirés : {n0 - len(df)}) " | |
| 209 | + f"[filtre ppsf {lo:.0f}–{hi:.0f} $/ft² sur superficies réelles]") | |
| 210 | + | |
| 211 | +# ---------- Superficie : réelle > reconstruite via pièces > imputée ---------- | |
| 212 | +# calibration : ratio médian (superficie réelle / somme des pièces) sur recouvrement | |
| 213 | +ov = df[df["sqft"].notna() & df["room_area"].notna() & | |
| 214 | + (df["room_area"] > 200)] | |
| 215 | +ratio = (ov["sqft"] / ov["room_area"]).median() | |
| 216 | +print(f"Calibration surface pièces : ratio réel/pièces = {ratio:.3f} " | |
| 217 | + f"(n={len(ov)})") | |
| 218 | +room_est = df["room_area"] * ratio | |
| 219 | +room_ok = df["room_area"].notna() & (room_est.between(300, 20_000)) | |
| 220 | + | |
| 221 | +df["sqft_final"] = df["sqft"] | |
| 222 | +fill = df["sqft_final"].isna() & room_ok | |
| 223 | +df.loc[fill, "sqft_final"] = room_est[fill] | |
| 224 | +df["sqft_measured"] = df["sqft_final"].notna().astype(int) # réelle ou pièces | |
| 225 | +n_room_fill = int(fill.sum()) | |
| 226 | +print(f"Superficies : réelles={int(df['sqft'].notna().sum())}, " | |
| 227 | + f"complétées via pièces=+{n_room_fill}, " | |
| 228 | + f"mesurées au total={int(df['sqft_measured'].sum())}/{len(df)}") | |
| 229 | + | |
| 230 | +# imputation du reste = médiane par type | |
| 231 | +df["has_sqft"] = df["sqft_measured"] | |
| 232 | +df["sqft_imp"] = df.groupby("ptype")["sqft_final"].transform( | |
| 233 | + lambda s: s.fillna(s.median())) | |
| 234 | +df["sqft_imp"] = df["sqft_imp"].fillna(df["sqft_final"].median()) | |
| 235 | + | |
| 236 | +df["has_lot"] = df["lot_sqft"].notna().astype(int) | |
| 237 | +df["lot_sqft"] = df["lot_sqft"].fillna(df["lot_sqft"].median()) | |
| 238 | +df["has_age"] = df["age"].notna().astype(int) | |
| 239 | +df["age"] = df["age"].fillna(df["age"].median()).clip(lower=0) | |
| 240 | +df["beds"] = df["beds"].fillna(df["beds"].median()) | |
| 241 | +df["baths"] = df["baths"].fillna(df["baths"].median()) | |
| 242 | +df["parking"] = df["parking"].fillna(0) | |
| 243 | +df["storeys"] = df["storeys"].fillna(df["storeys"].median()) | |
| 244 | + | |
| 245 | +df["ln_price"] = np.log(df["price"]) | |
| 246 | +df["ln_sqft"] = np.log(df["sqft_imp"]) | |
| 247 | +df["ln_lot"] = np.log(df["lot_sqft"]) | |
| 248 | +df["age2"] = df["age"] ** 2 | |
| 249 | + | |
| 250 | +# Effets fixes ville : regrouper les villes rares | |
| 251 | +vc = df["city"].value_counts() | |
| 252 | +keep = set(vc[vc >= 20].index) | |
| 253 | +df["city_fe"] = df["city"].where(df["city"].isin(keep), other="__other__") | |
| 254 | +df["city_fe"] = df["city_fe"].fillna("__other__") | |
| 255 | +print(f"Villes en effets fixes : {df['city_fe'].nunique()} " | |
| 256 | + f"(seuil ≥20 listings ; {len(keep)} villes nommées)") | |
| 257 | + | |
| 258 | +# Type de propriété harmonisé (regrouper rares) | |
| 259 | +tc = df["ptype"].value_counts() | |
| 260 | +keept = set(tc[tc >= 30].index) | |
| 261 | +df["ptype_fe"] = df["ptype"].where(df["ptype"].isin(keept), other="Other") | |
| 262 | + | |
| 263 | +# ---------- Modèles hédoniques ---------- | |
| 264 | +amen = list(AMENITIES.keys()) | |
| 265 | +def pct(b): | |
| 266 | + return (np.exp(b) - 1) * 100 | |
| 267 | + | |
| 268 | +# variables "détail des pièces" + autres signaux (Realtor surtout) | |
| 269 | +df["n_images_c"] = df["n_images"].clip(upper=60) # écrête les extrêmes | |
| 270 | +room_vars = ["n_rooms", "n_levels_r", "has_primary_bed", "has_basement", | |
| 271 | + "r_office", "r_family", "r_workshop", "r_winecellar", | |
| 272 | + "r_sunroom", "r_mezzanine", "r_library", "r_laundry"] | |
| 273 | +extra = ["n_images_c", "virtual_tour"] + room_vars | |
| 274 | + | |
| 275 | +base = ("ln_lot + has_lot + beds + baths + parking + storeys " | |
| 276 | + "+ age + age2 + has_age + C(source) + C(ptype_fe) + " | |
| 277 | + + " + ".join(amen) + " + " + " + ".join(extra) | |
| 278 | + + " + C(city_fe)") | |
| 279 | +fA = "ln_price ~ ln_sqft + " + base # riche (superficie réelle) | |
| 280 | +fB = "ln_price ~ " + base # large (sans superficie) | |
| 281 | + | |
| 282 | +dfA = df[df["has_sqft"] == 1].copy() | |
| 283 | +print(f"\nModèle A (avec superficie réelle) : N={len(dfA)}") | |
| 284 | +mA = smf.ols(fA, data=dfA).fit(cov_type="HC3") | |
| 285 | +print(f"Modèle B (sans superficie, couverture large) : N={len(df)}") | |
| 286 | +mB = smf.ols(fB, data=df).fit(cov_type="HC3") | |
| 287 | + | |
| 288 | +sdA, sdB = np.std(mA.resid), np.std(mB.resid) | |
| 289 | + | |
| 290 | +def report(m, sd, name): | |
| 291 | + print("\n" + "=" * 70) | |
| 292 | + print(f"{name} N={int(m.nobs)} R²={m.rsquared:.4f} " | |
| 293 | + f"R²_adj={m.rsquared_adj:.4f} " | |
| 294 | + f"erreur typique ±{(np.exp(sd)-1)*100:.1f}%") | |
| 295 | + print("=" * 70) | |
| 296 | + pr, pv = m.params, m.pvalues | |
| 297 | + show = [p for p in pr.index if not p.startswith("C(city_fe)")] | |
| 298 | + print(f"{'variable':<30}{'coef':>10}{'~%impact':>11}{'p':>9}") | |
| 299 | + for p in show: | |
| 300 | + st = "***" if pv[p]<.01 else "**" if pv[p]<.05 else "*" if pv[p]<.1 else "" | |
| 301 | + pc = f"{pct(pr[p]):+.1f}%" if abs(pr[p]) < 3 else "" | |
| 302 | + print(f"{p:<30}{pr[p]:>10.4f}{pc:>11}{pv[p]:>9.3g} {st}") | |
| 303 | + | |
| 304 | +report(mA, sdA, "MODÈLE A — hédonique riche (superficie incluse)") | |
| 305 | +report(mB, sdB, "MODÈLE B — couverture large (sans superficie)") | |
| 306 | + | |
| 307 | +# ---------- Prédictions combinées ---------- | |
| 308 | +# A pour les biens à superficie réelle, B sinon | |
| 309 | +predA = np.exp(mA.predict(df)) * np.exp(sdA**2 / 2) | |
| 310 | +predB = np.exp(mB.predict(df)) * np.exp(sdB**2 / 2) | |
| 311 | +df["pred_price"] = np.where(df["has_sqft"] == 1, predA, predB) | |
| 312 | +df["model_used"] = np.where(df["has_sqft"] == 1, "A", "B") | |
| 313 | +df["resid_pct"] = (df["price"] / df["pred_price"] - 1) * 100 | |
| 314 | + | |
| 315 | +out_cols = (["source", "city", "ptype", "price", "pred_price", "resid_pct", | |
| 316 | + "model_used", "sqft", "room_area", "sqft_final", "sqft_measured", | |
| 317 | + "lot_sqft", "beds", "baths", "parking", "storeys", "age", | |
| 318 | + "n_images", "virtual_tour"] + amen + room_vars) | |
| 319 | +df[out_cols].to_csv("data/processed/maisons_hedonic.csv", index=False) | |
| 320 | +print(f"\nDataset final + prédictions -> maisons_hedonic.csv ({len(df)} lignes)") | |
| 321 | + | |
| 322 | +# top villes (effets fixes du modèle A) | |
| 323 | +cityfe = {k.split("T.")[1].rstrip("]"): v for k, v in mA.params.items() | |
| 324 | + if k.startswith("C(city_fe)")} | |
| 325 | +cf = pd.Series(cityfe).sort_values(ascending=False) | |
| 326 | +print("\nTop 12 villes — prime de localisation (Modèle A, vs référence):") | |
| 327 | +for c, b in cf.head(12).items(): | |
| 328 | + print(f" {c:<30}{pct(b):+.1f}%") | |
| 329 | +print("Bottom 6:") | |
| 330 | +for c, b in cf.tail(6).items(): | |
| 331 | + print(f" {c:<30}{pct(b):+.1f}%") | |
| 332 | + | |
| 333 | +with open("results/hedonic_summary.txt", "w", encoding="utf-8") as fh: | |
| 334 | + fh.write("=== MODELE A (avec superficie) ===\n") | |
| 335 | + fh.write(str(mA.summary())) | |
| 336 | + fh.write("\n\n=== MODELE B (sans superficie) ===\n") | |
| 337 | + fh.write(str(mB.summary())) | |
| 338 | +print("Résumés complets -> hedonic_summary.txt") | |
added
scripts/05_hedonic_ols_v4_text.py
+249 −0
@@ -0,0 +1,249 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +""" | |
| 4 | +Modèle hédonique v4 : ajoute les specs extraites par GPT-4o-mini (specs.jsonl) | |
| 5 | +au modèle v3. Réutilise la même logique de nettoyage/features + effets fixes. | |
| 6 | +""" | |
| 7 | +import json, re, warnings | |
| 8 | +import numpy as np | |
| 9 | +import pandas as pd | |
| 10 | +import statsmodels.formula.api as smf | |
| 11 | + | |
| 12 | +warnings.filterwarnings("ignore") | |
| 13 | +CUR_YEAR = 2026 | |
| 14 | + | |
| 15 | +# ---------- charge specs GPT ---------- | |
| 16 | +specs = {} | |
| 17 | +for line in open("data/processed/specs.jsonl", encoding="utf-8"): | |
| 18 | + try: | |
| 19 | + r = json.loads(line) | |
| 20 | + except Exception: | |
| 21 | + continue | |
| 22 | + if "specs" in r and r.get("specs"): | |
| 23 | + specs[r["uid"]] = r["specs"] | |
| 24 | +print(f"Specs GPT chargées : {len(specs)}") | |
| 25 | + | |
| 26 | +# specs -> features numériques (dummies + catégorielles réduites) | |
| 27 | +COND_SCORE = {"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4, | |
| 28 | + "neuf":5,"inconnu":2} | |
| 29 | +BOOL_SPECS = ["turnkey","recently_renovated","garage","pool","waterfront", | |
| 30 | + "finished_basement","fireplace","central_ac","new_construction", | |
| 31 | + "income_property","renovated_kitchen","renovated_bathroom", | |
| 32 | + "large_lot","quiet_area"] | |
| 33 | + | |
| 34 | +def spec_feats(uid): | |
| 35 | + s = specs.get(uid) | |
| 36 | + out = {"g_" + b: 0 for b in BOOL_SPECS} | |
| 37 | + out["g_cond"] = 2 | |
| 38 | + out["g_hardwood"] = 0; out["g_quartz_granite"] = 0; out["g_view_water"] = 0 | |
| 39 | + out["has_gpt"] = 0 | |
| 40 | + if not s: | |
| 41 | + return out | |
| 42 | + out["has_gpt"] = 1 | |
| 43 | + for b in BOOL_SPECS: | |
| 44 | + out["g_" + b] = int(bool(s.get(b))) | |
| 45 | + out["g_cond"] = COND_SCORE.get(s.get("condition", "inconnu"), 2) | |
| 46 | + fl = s.get("flooring") or [] | |
| 47 | + out["g_hardwood"] = int("bois_franc" in fl) | |
| 48 | + out["g_quartz_granite"] = int(s.get("countertop") in ("quartz", "granit", "marbre")) | |
| 49 | + out["g_view_water"] = int(s.get("view") == "eau") | |
| 50 | + return out | |
| 51 | + | |
| 52 | +# ===================== pipeline v3 (repris) ===================== | |
| 53 | +SRC = "data/processed/maisons_avec_prix.json" | |
| 54 | +data = json.load(open(SRC, encoding="utf-8")) | |
| 55 | + | |
| 56 | +def parse_area(s): | |
| 57 | + if not s or not isinstance(s, str): return np.nan | |
| 58 | + txt = s.lower(); is_m2 = ("m2" in txt) or ("m²" in txt) | |
| 59 | + num = re.sub(r"[^0-9.,\s]", "", txt).strip().replace(" ", "").replace(",", "") | |
| 60 | + try: v = float(num) | |
| 61 | + except ValueError: return np.nan | |
| 62 | + if v <= 0: return np.nan | |
| 63 | + return v * 10.7639 if is_m2 else v | |
| 64 | + | |
| 65 | +def clean_city(city, source): | |
| 66 | + if not city or not isinstance(city, str): return None | |
| 67 | + c = city.split("(")[0].strip(); c = re.sub(r"\s+", " ", c).lower().strip() | |
| 68 | + return c or None | |
| 69 | + | |
| 70 | +def to_int(x): | |
| 71 | + try: return int(x) | |
| 72 | + except (TypeError, ValueError): return np.nan | |
| 73 | + | |
| 74 | +AMENITIES = { | |
| 75 | + "pool":["swimming pool","piscine","inground pool","pool"], | |
| 76 | + "waterfront":["waterfront","bord de l'eau","navigable","water access","lakefront","riverfront","au bord de"], | |
| 77 | + "garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"], | |
| 78 | + "central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"], | |
| 79 | + "finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"], | |
| 80 | + "near_water":["near water","waterfront","navigable"], | |
| 81 | + "cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"], | |
| 82 | + "near_park":["park","parc"], | |
| 83 | + "near_school":["school","école","cegep","university","université","elementary","high school"], | |
| 84 | + "near_highway":["highway","autoroute"], | |
| 85 | + "transit":["public transit","public transportation","transport"], | |
| 86 | + "no_rear_neighbor":["no backyard neighbors","no rear neighbor","sans voisin arrière"], | |
| 87 | +} | |
| 88 | + | |
| 89 | +def _side_ft(side): | |
| 90 | + s = side.lower() | |
| 91 | + if "ft" in s: | |
| 92 | + ft = re.search(r"([\d.]+)\s*ft", s); inch = re.search(r"([\d.]+)\s*in", s) | |
| 93 | + return (float(ft.group(1)) if ft else 0)+(float(inch.group(1))/12 if inch else 0) | |
| 94 | + m = re.search(r"([\d.,]+)\s*m", s) | |
| 95 | + if m: | |
| 96 | + try: return float(m.group(1).replace(",", ".")) * 3.28084 | |
| 97 | + except ValueError: return None | |
| 98 | + return None | |
| 99 | + | |
| 100 | +def _room_area(dim): | |
| 101 | + if not dim or "x" not in dim: return None | |
| 102 | + a, _, b = dim.partition("x"); la, lb = _side_ft(a), _side_ft(b) | |
| 103 | + if la and lb and 1 < la < 200 and 1 < lb < 200: return la * lb | |
| 104 | + return None | |
| 105 | + | |
| 106 | +ROOM_FLAGS = {"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"], | |
| 107 | + "r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"], | |
| 108 | + "r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 109 | + | |
| 110 | +def parse_rooms(rooms): | |
| 111 | + out = {"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 112 | + for k in ROOM_FLAGS: out[k] = 0 | |
| 113 | + if not rooms: return out | |
| 114 | + area, levels = 0.0, set() | |
| 115 | + for r in rooms: | |
| 116 | + nm = (r.get("name") or "").lower(); lv = (r.get("level") or "").lower() | |
| 117 | + out["n_rooms"] += 1 | |
| 118 | + if lv: levels.add(lv) | |
| 119 | + if "basement" in lv or "sous-sol" in lv: out["has_basement"] = 1 | |
| 120 | + if "primary bedroom" in nm: out["has_primary_bed"] = 1 | |
| 121 | + for k, kws in ROOM_FLAGS.items(): | |
| 122 | + if any(w in nm for w in kws): out[k] = 1 | |
| 123 | + a = _room_area(r.get("dimensions")) | |
| 124 | + if a: area += a | |
| 125 | + out["room_area"] = area if area > 0 else np.nan | |
| 126 | + out["n_levels_r"] = len(levels) | |
| 127 | + return out | |
| 128 | + | |
| 129 | +rows = [] | |
| 130 | +for d in data: | |
| 131 | + prop = d.get("property") or {}; addr = d.get("address") or {} | |
| 132 | + feats = d.get("features") or [] | |
| 133 | + blob = (" ".join(feats)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 134 | + yb = to_int(prop.get("yearBuilt")) | |
| 135 | + age = (CUR_YEAR-yb) if (not pd.isna(yb) and 1700 <= yb <= CUR_YEAR) else np.nan | |
| 136 | + ptype = prop.get("buildingType") or prop.get("type") | |
| 137 | + storeys = prop.get("storeys") or prop.get("levels") | |
| 138 | + try: storeys = float(str(storeys).replace(",", ".")) | |
| 139 | + except (TypeError, ValueError): storeys = np.nan | |
| 140 | + uid = f"{d.get('source')}:{d.get('listingId')}" | |
| 141 | + rec = {"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"), | |
| 142 | + "sqft":parse_area(prop.get("squareFeet")),"lot_sqft":parse_area(prop.get("lotSize")), | |
| 143 | + "beds":to_int(prop.get("bedrooms")),"baths":to_int(prop.get("bathrooms")), | |
| 144 | + "parking":to_int(prop.get("parkingSpaces")),"storeys":storeys,"age":age, | |
| 145 | + "ptype":(ptype or "Unknown").strip(),"city":clean_city(addr.get("city"),d.get("source"))} | |
| 146 | + for name, kws in AMENITIES.items(): | |
| 147 | + rec[name] = int(any(k in blob for k in kws)) | |
| 148 | + rec["n_images"] = len(d.get("images") or []) | |
| 149 | + rec["virtual_tour"] = int(bool(d.get("virtualTourUrl"))) | |
| 150 | + rec.update(parse_rooms(d.get("rooms"))) | |
| 151 | + rec.update(spec_feats(uid)) | |
| 152 | + rows.append(rec) | |
| 153 | + | |
| 154 | +df = pd.DataFrame(rows) | |
| 155 | + | |
| 156 | +# nettoyage identique v3 | |
| 157 | +n0 = len(df) | |
| 158 | +df = df[df["price"].notna()] | |
| 159 | +df = df[(df["price"] >= 50_000) & (df["price"] <= 15_000_000)] | |
| 160 | +df.loc[(df["sqft"] < 300) | (df["sqft"] > 20_000), "sqft"] = np.nan | |
| 161 | +df.loc[(df["lot_sqft"] < 200) | (df["lot_sqft"] > 5_000_000), "lot_sqft"] = np.nan | |
| 162 | +df.loc[df["beds"] > 15, "beds"] = np.nan | |
| 163 | +df.loc[df["baths"] > 12, "baths"] = np.nan | |
| 164 | +df["ppsf"] = df["price"] / df["sqft"] | |
| 165 | +lo, hi = df["ppsf"].quantile([0.005, 0.995]) | |
| 166 | +df = df[~(df["ppsf"].notna() & ((df["ppsf"] < lo) | (df["ppsf"] > hi)))] | |
| 167 | + | |
| 168 | +# surface finale | |
| 169 | +ov = df[df["sqft"].notna() & df["room_area"].notna() & (df["room_area"] > 200)] | |
| 170 | +ratio = (ov["sqft"] / ov["room_area"]).median() | |
| 171 | +room_est = df["room_area"] * ratio | |
| 172 | +room_ok = df["room_area"].notna() & (room_est.between(300, 20_000)) | |
| 173 | +df["sqft_final"] = df["sqft"] | |
| 174 | +fill = df["sqft_final"].isna() & room_ok | |
| 175 | +df.loc[fill, "sqft_final"] = room_est[fill] | |
| 176 | +df["has_sqft"] = df["sqft_final"].notna().astype(int) | |
| 177 | +df["sqft_imp"] = df.groupby("ptype")["sqft_final"].transform(lambda s: s.fillna(s.median())) | |
| 178 | +df["sqft_imp"] = df["sqft_imp"].fillna(df["sqft_final"].median()) | |
| 179 | + | |
| 180 | +df["has_lot"] = df["lot_sqft"].notna().astype(int) | |
| 181 | +df["lot_sqft"] = df["lot_sqft"].fillna(df["lot_sqft"].median()) | |
| 182 | +df["has_age"] = df["age"].notna().astype(int) | |
| 183 | +df["age"] = df["age"].fillna(df["age"].median()).clip(lower=0) | |
| 184 | +for c in ["beds","baths","storeys"]: | |
| 185 | + df[c] = df[c].fillna(df[c].median()) | |
| 186 | +df["parking"] = df["parking"].fillna(0) | |
| 187 | +df["ln_price"] = np.log(df["price"]); df["ln_sqft"] = np.log(df["sqft_imp"]) | |
| 188 | +df["ln_lot"] = np.log(df["lot_sqft"]); df["age2"] = df["age"]**2 | |
| 189 | +df["n_images_c"] = df["n_images"].clip(upper=60) | |
| 190 | + | |
| 191 | +vc = df["city"].value_counts(); keep = set(vc[vc >= 20].index) | |
| 192 | +df["city_fe"] = df["city"].where(df["city"].isin(keep), "__other__").fillna("__other__") | |
| 193 | +tc = df["ptype"].value_counts(); keept = set(tc[tc >= 30].index) | |
| 194 | +df["ptype_fe"] = df["ptype"].where(df["ptype"].isin(keept), "Other") | |
| 195 | +print(f"N nettoyé={len(df)} villes FE={df['city_fe'].nunique()} " | |
| 196 | + f"avec specs GPT={int(df['has_gpt'].sum())}") | |
| 197 | + | |
| 198 | +# ---------- modèles ---------- | |
| 199 | +amen = list(AMENITIES.keys()) | |
| 200 | +room_vars = ["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office", | |
| 201 | + "r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine", | |
| 202 | + "r_library","r_laundry"] | |
| 203 | +gpt_vars = ["g_cond","g_hardwood","g_quartz_granite","g_view_water","has_gpt"] + \ | |
| 204 | + ["g_" + b for b in BOOL_SPECS] | |
| 205 | +extra = ["n_images_c","virtual_tour"] + room_vars + gpt_vars | |
| 206 | +base = ("ln_lot + has_lot + beds + baths + parking + storeys + age + age2 + has_age " | |
| 207 | + "+ C(source) + C(ptype_fe) + " + " + ".join(amen) + " + " + " + ".join(extra) | |
| 208 | + + " + C(city_fe)") | |
| 209 | +fA = "ln_price ~ ln_sqft + " + base | |
| 210 | +fB = "ln_price ~ " + base | |
| 211 | + | |
| 212 | +def pct(b): return (np.exp(b)-1)*100 | |
| 213 | +dfA = df[df["has_sqft"] == 1].copy() | |
| 214 | +mA = smf.ols(fA, data=dfA).fit(cov_type="HC3") | |
| 215 | +mB = smf.ols(fB, data=df).fit(cov_type="HC3") | |
| 216 | +sdA, sdB = np.std(mA.resid), np.std(mB.resid) | |
| 217 | + | |
| 218 | +def report(m, sd, name): | |
| 219 | + print("\n" + "=" * 70) | |
| 220 | + print(f"{name} N={int(m.nobs)} R²={m.rsquared:.4f} R²adj={m.rsquared_adj:.4f} " | |
| 221 | + f"±{(np.exp(sd)-1)*100:.1f}%") | |
| 222 | + print("=" * 70) | |
| 223 | + pr, pv = m.params, m.pvalues | |
| 224 | + print("-- specs GPT (impact %) --") | |
| 225 | + for p in pr.index: | |
| 226 | + if p.startswith("g_") or p == "has_gpt": | |
| 227 | + st = "***" if pv[p]<.01 else "**" if pv[p]<.05 else "*" if pv[p]<.1 else "" | |
| 228 | + print(f" {p:<22}{pr[p]:>9.4f}{pct(pr[p]):>+8.1f}% p={pv[p]:.2g} {st}") | |
| 229 | + | |
| 230 | +report(mA, sdA, "MODÈLE A+GPT (avec superficie)") | |
| 231 | +report(mB, sdB, "MODÈLE B+GPT (couverture large)") | |
| 232 | + | |
| 233 | +predA = np.exp(mA.predict(df)) * np.exp(sdA**2/2) | |
| 234 | +predB = np.exp(mB.predict(df)) * np.exp(sdB**2/2) | |
| 235 | +df["pred_price"] = np.where(df["has_sqft"] == 1, predA, predB) | |
| 236 | +df["model_used"] = np.where(df["has_sqft"] == 1, "A", "B") | |
| 237 | +df["resid_pct"] = (df["price"]/df["pred_price"] - 1) * 100 | |
| 238 | +print(f"\nErreur abs médiane: {df.resid_pct.abs().median():.1f}% " | |
| 239 | + f"| ±15%: {(df.resid_pct.abs()<=15).mean()*100:.0f}% " | |
| 240 | + f"±25%: {(df.resid_pct.abs()<=25).mean()*100:.0f}%") | |
| 241 | + | |
| 242 | +out = (["uid","source","city","ptype","price","pred_price","resid_pct","model_used", | |
| 243 | + "sqft_final","has_sqft","lot_sqft","beds","baths","parking","age","n_images", | |
| 244 | + "virtual_tour"] + amen + room_vars + gpt_vars) | |
| 245 | +df[out].to_csv("data/processed/maisons_hedonic_v4.csv", index=False) | |
| 246 | +print(f"-> maisons_hedonic_v4.csv ({len(df)} lignes)") | |
| 247 | +with open("results/hedonic_v4_summary.txt", "w", encoding="utf-8") as fh: | |
| 248 | + fh.write("=== A+GPT ===\n"+str(mA.summary())+"\n\n=== B+GPT ===\n"+str(mB.summary())) | |
| 249 | +print("-> hedonic_v4_summary.txt") | |
added
scripts/06_hedonic_ml_cv.py
+171 −0
@@ -0,0 +1,171 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Comparaison OLS vs modèles ML (RF, Gradient Boosting) en validation croisée 5-fold. | |
| 4 | +Même jeu de variables que la v4 (structurel + aménités + pièces + listing + GPT + FE).""" | |
| 5 | +import json, re, warnings | |
| 6 | +import numpy as np, pandas as pd | |
| 7 | +from sklearn.linear_model import LinearRegression | |
| 8 | +from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor | |
| 9 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 10 | +from sklearn.metrics import r2_score | |
| 11 | +warnings.filterwarnings("ignore") | |
| 12 | +CUR_YEAR = 2026 | |
| 13 | + | |
| 14 | +# ---- specs GPT ---- | |
| 15 | +specs = {} | |
| 16 | +for l in open("data/processed/specs.jsonl", encoding="utf-8"): | |
| 17 | + r = json.loads(l) | |
| 18 | + if r.get("specs"): specs[r["uid"]] = r["specs"] | |
| 19 | +COND = {"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2} | |
| 20 | +BOOL = ["turnkey","recently_renovated","garage","pool","waterfront","finished_basement", | |
| 21 | + "fireplace","central_ac","new_construction","income_property","renovated_kitchen", | |
| 22 | + "renovated_bathroom","large_lot","quiet_area"] | |
| 23 | +def gfeat(uid): | |
| 24 | + s = specs.get(uid); o = {"g_"+b:0 for b in BOOL} | |
| 25 | + o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0}) | |
| 26 | + if not s: return o | |
| 27 | + for b in BOOL: o["g_"+b] = int(bool(s.get(b))) | |
| 28 | + o["g_cond"] = COND.get(s.get("condition","inconnu"),2) | |
| 29 | + fl = s.get("flooring") or []; o["g_hardwood"] = int("bois_franc" in fl) | |
| 30 | + o["g_quartz_granite"] = int(s.get("countertop") in ("quartz","granit","marbre")) | |
| 31 | + o["g_view_water"] = int(s.get("view")=="eau"); return o | |
| 32 | + | |
| 33 | +# ---- parsing (repris) ---- | |
| 34 | +def parse_area(s): | |
| 35 | + if not s or not isinstance(s,str): return np.nan | |
| 36 | + t=s.lower(); m=("m2" in t) or ("m²" in t) | |
| 37 | + n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 38 | + try: v=float(n) | |
| 39 | + except: return np.nan | |
| 40 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 41 | +def cc(c): | |
| 42 | + if not c or not isinstance(c,str): return None | |
| 43 | + c=c.split("(")[0].strip(); return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 44 | +def ti(x): | |
| 45 | + try: return int(x) | |
| 46 | + except: return np.nan | |
| 47 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"], | |
| 48 | +"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"], | |
| 49 | +"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"], | |
| 50 | +"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"], | |
| 51 | +"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"], | |
| 52 | +"golf":["golf"],"ski":["ski"],"near_park":["park","parc"], | |
| 53 | +"near_school":["school","école","cegep","university","université","elementary","high school"], | |
| 54 | +"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"], | |
| 55 | +"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 56 | +def sd(s): | |
| 57 | + s=s.lower() | |
| 58 | + if "ft" in s: | |
| 59 | + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s) | |
| 60 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 61 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 62 | + if m: | |
| 63 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 64 | + except: return None | |
| 65 | + return None | |
| 66 | +def ra(d): | |
| 67 | + if not d or "x" not in d: return None | |
| 68 | + a,_,b=d.partition("x");la,lb=sd(a),sd(b) | |
| 69 | + return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 70 | +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"], | |
| 71 | +"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 72 | +def pr(rooms): | |
| 73 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 74 | + for k in RF: o[k]=0 | |
| 75 | + if not rooms: return o | |
| 76 | + ar=0.0;lv=set() | |
| 77 | + for r in rooms: | |
| 78 | + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1 | |
| 79 | + if l: lv.add(l) | |
| 80 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 81 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 82 | + for k,kw in RF.items(): | |
| 83 | + if any(w in nm for w in kw): o[k]=1 | |
| 84 | + a=ra(r.get("dimensions")) | |
| 85 | + if a: ar+=a | |
| 86 | + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o | |
| 87 | + | |
| 88 | +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8")) | |
| 89 | +rows=[] | |
| 90 | +for d in data: | |
| 91 | + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or [] | |
| 92 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 93 | + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 94 | + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels") | |
| 95 | + try: st=float(str(st).replace(",",".")) | |
| 96 | + except: st=np.nan | |
| 97 | + uid=f"{d.get('source')}:{d.get('listingId')}" | |
| 98 | + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")), | |
| 99 | + "lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")), | |
| 100 | + "parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))} | |
| 101 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 102 | + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 103 | + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec) | |
| 104 | +df=pd.DataFrame(rows) | |
| 105 | + | |
| 106 | +# nettoyage (identique) | |
| 107 | +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 108 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 109 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 110 | +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan | |
| 111 | +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 112 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 113 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)] | |
| 114 | +ratio=(ov["sqft"]/ov["room_area"]).median();re_=df["room_area"]*ratio | |
| 115 | +ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 116 | +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill] | |
| 117 | +df["has_sqft"]=df["sqft_final"].notna().astype(int) | |
| 118 | +# pour ML : garder NaN (les arbres gèrent) ; pour OLS on impute une copie | |
| 119 | +df["ln_price"]=np.log(df["price"]);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 120 | +vc=df["city"].value_counts();keep=set(vc[vc>=20].index) | |
| 121 | +df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 122 | +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 123 | +print(f"N={len(df)}") | |
| 124 | + | |
| 125 | +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 126 | +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL] | |
| 127 | +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv | |
| 128 | +cat=["city_fe","ptype_fe","source"] | |
| 129 | + | |
| 130 | +y=df["ln_price"].values | |
| 131 | +price=df["price"].values | |
| 132 | +Xcat=pd.get_dummies(df[cat],prefix=cat,dtype=float) | |
| 133 | +# --- matrice ML (NaN autorisés pour arbres) --- | |
| 134 | +Xml=pd.concat([df[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1) | |
| 135 | +# --- matrice OLS (imputée médiane par type) --- | |
| 136 | +dfi=df.copy() | |
| 137 | +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]: | |
| 138 | + dfi[c]=dfi.groupby("ptype")[c].transform(lambda s:s.fillna(s.median())) | |
| 139 | + dfi[c]=dfi[c].fillna(dfi[c].median()) | |
| 140 | +dfi["parking"]=dfi["parking"].fillna(0) | |
| 141 | +Xols=pd.concat([dfi[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1) | |
| 142 | + | |
| 143 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 144 | +def evalcv(model,X): | |
| 145 | + pred=cross_val_predict(model,X,y,cv=kf,n_jobs=-1) | |
| 146 | + r2=r2_score(y,pred) | |
| 147 | + mape=np.median(np.abs(np.exp(pred-y)-1))*100 | |
| 148 | + w15=np.mean(np.abs(np.exp(pred-y)-1)<=.15)*100 | |
| 149 | + w25=np.mean(np.abs(np.exp(pred-y)-1)<=.25)*100 | |
| 150 | + return r2,mape,w15,w25 | |
| 151 | + | |
| 152 | +print("\nValidation croisée 5-fold (out-of-sample) — cible = ln(prix)\n") | |
| 153 | +print(f"{'Modèle':<34}{'R² OOS':>9}{'MAE méd.':>10}{'±15%':>7}{'±25%':>7}") | |
| 154 | +res={} | |
| 155 | +res["OLS (linéaire, imputé)"]=evalcv(LinearRegression(),Xols) | |
| 156 | +res["Random Forest (400)"]=evalcv(RandomForestRegressor(n_estimators=400,max_features=0.5,min_samples_leaf=3,n_jobs=-1,random_state=42),Xml) | |
| 157 | +res["HistGradientBoosting"]=evalcv(HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42),Xml) | |
| 158 | +res["HistGB (tuned depth)"]=evalcv(HistGradientBoostingRegressor(max_iter=900,learning_rate=0.03,max_leaf_nodes=127,min_samples_leaf=20,l2_regularization=2.0,random_state=42),Xml) | |
| 159 | +for k,(r2,mape,w15,w25) in res.items(): | |
| 160 | + print(f"{k:<34}{r2:>9.4f}{mape:>9.1f}%{w15:>6.0f}%{w25:>6.0f}%") | |
| 161 | + | |
| 162 | +# importance GBM (permutation rapide via feature_importances_ du HistGB n'existe pas → refit + importances via gain n/a) | |
| 163 | +gb=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42).fit(Xml,y) | |
| 164 | +from sklearn.inspection import permutation_importance | |
| 165 | +import numpy as _np | |
| 166 | +idx=_np.random.RandomState(0).choice(len(Xml),4000,replace=False) | |
| 167 | +pi=permutation_importance(gb,Xml.iloc[idx],y[idx],n_repeats=3,random_state=0,n_jobs=-1) | |
| 168 | +imp=pd.Series(pi.importances_mean,index=Xml.columns).sort_values(ascending=False) | |
| 169 | +print("\nTop 20 variables (importance par permutation, GBM) :") | |
| 170 | +for name,v in imp.head(20).items(): print(f" {name:<26}{v:.4f}") | |
| 171 | +print("\nImportance cumulée des variables GPT (g_*) :","%.4f"%imp[[c for c in Xml.columns if c.startswith('g_')]].sum()) | |
added
scripts/07_torch_nn.py
+170 −0
@@ -0,0 +1,170 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Réseau de neurones tabulaire (embeddings ville/type + MLP) — 5-fold CV. | |
| 4 | +Comparable aux modèles de hedonic_ml.py (même variables, même KFold seed=42).""" | |
| 5 | +import json, re, warnings | |
| 6 | +import numpy as np, pandas as pd | |
| 7 | +import torch, torch.nn as nn | |
| 8 | +from sklearn.model_selection import KFold | |
| 9 | +from sklearn.metrics import r2_score | |
| 10 | +warnings.filterwarnings("ignore") | |
| 11 | +torch.manual_seed(42); np.random.seed(42) | |
| 12 | +CUR_YEAR=2026 | |
| 13 | +dev="mps" if torch.backends.mps.is_available() else "cpu" | |
| 14 | +print("device:",dev) | |
| 15 | + | |
| 16 | +# ---- specs GPT ---- | |
| 17 | +specs={} | |
| 18 | +for l in open("data/processed/specs.jsonl",encoding="utf-8"): | |
| 19 | + r=json.loads(l) | |
| 20 | + if r.get("specs"): specs[r["uid"]]=r["specs"] | |
| 21 | +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2} | |
| 22 | +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"] | |
| 23 | +def gfeat(uid): | |
| 24 | + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0}) | |
| 25 | + if not s: return o | |
| 26 | + for b in BOOL: o["g_"+b]=int(bool(s.get(b))) | |
| 27 | + o["g_cond"]=COND.get(s.get("condition","inconnu"),2) | |
| 28 | + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl) | |
| 29 | + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o | |
| 30 | +def parse_area(s): | |
| 31 | + if not s or not isinstance(s,str): return np.nan | |
| 32 | + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 33 | + try: v=float(n) | |
| 34 | + except: return np.nan | |
| 35 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 36 | +def cc(c): | |
| 37 | + if not c or not isinstance(c,str): return None | |
| 38 | + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 39 | +def ti(x): | |
| 40 | + try: return int(x) | |
| 41 | + except: return np.nan | |
| 42 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 43 | +def sd(s): | |
| 44 | + s=s.lower() | |
| 45 | + if "ft" in s: | |
| 46 | + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s) | |
| 47 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 48 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 49 | + if m: | |
| 50 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 51 | + except: return None | |
| 52 | + return None | |
| 53 | +def ra(d): | |
| 54 | + if not d or "x" not in d: return None | |
| 55 | + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 56 | +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 57 | +def pr(rooms): | |
| 58 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 59 | + for k in RF: o[k]=0 | |
| 60 | + if not rooms: return o | |
| 61 | + ar=0.0;lv=set() | |
| 62 | + for r in rooms: | |
| 63 | + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1 | |
| 64 | + if l: lv.add(l) | |
| 65 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 66 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 67 | + for k,kw in RF.items(): | |
| 68 | + if any(w in nm for w in kw): o[k]=1 | |
| 69 | + a=ra(r.get("dimensions")) | |
| 70 | + if a: ar+=a | |
| 71 | + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o | |
| 72 | +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[] | |
| 73 | +for d in data: | |
| 74 | + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or [] | |
| 75 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 76 | + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 77 | + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels") | |
| 78 | + try: st=float(str(st).replace(",",".")) | |
| 79 | + except: st=np.nan | |
| 80 | + uid=f"{d.get('source')}:{d.get('listingId')}" | |
| 81 | + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))} | |
| 82 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 83 | + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 84 | + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec) | |
| 85 | +df=pd.DataFrame(rows) | |
| 86 | +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 87 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 88 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 89 | +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan | |
| 90 | +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 91 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 92 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median() | |
| 93 | +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 94 | +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill] | |
| 95 | +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 96 | +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 97 | +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 98 | +df=df.reset_index(drop=True);print("N=",len(df)) | |
| 99 | + | |
| 100 | +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 101 | +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL] | |
| 102 | +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv | |
| 103 | +y=df["ln_price"]=np.log(df["price"]);y=y.values.astype(np.float32) | |
| 104 | +# encodage catégoriel -> indices | |
| 105 | +cats=["city_fe","ptype_fe","source"];maps={};card={} | |
| 106 | +for c in cats: | |
| 107 | + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};maps[c]=m;card[c]=len(u) | |
| 108 | + df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64) | |
| 109 | +Xnum=df[num].astype(np.float32).values | |
| 110 | +Xcat=df[[c+"_ix" for c in cats]].values.astype(np.int64) | |
| 111 | + | |
| 112 | +class Net(nn.Module): | |
| 113 | + def __init__(self,ncard,nnum): | |
| 114 | + super().__init__() | |
| 115 | + self.embs=nn.ModuleList() | |
| 116 | + tot=0 | |
| 117 | + for c in ncard: | |
| 118 | + d=min(50,(c+1)//2); d=max(d,2); self.embs.append(nn.Embedding(c,d)); tot+=d | |
| 119 | + self.bn=nn.BatchNorm1d(nnum) | |
| 120 | + self.net=nn.Sequential( | |
| 121 | + nn.Linear(tot+nnum,256),nn.ReLU(),nn.BatchNorm1d(256),nn.Dropout(0.3), | |
| 122 | + nn.Linear(256,128),nn.ReLU(),nn.BatchNorm1d(128),nn.Dropout(0.2), | |
| 123 | + nn.Linear(128,64),nn.ReLU(),nn.Dropout(0.1), | |
| 124 | + nn.Linear(64,1)) | |
| 125 | + def forward(self,xc,xn): | |
| 126 | + e=[emb(xc[:,i]) for i,emb in enumerate(self.embs)] | |
| 127 | + x=torch.cat(e+[self.bn(xn)],1);return self.net(x).squeeze(1) | |
| 128 | + | |
| 129 | +def train_predict(tr,te): | |
| 130 | + # imputation médiane (train) + standardisation | |
| 131 | + med=np.nanmedian(Xnum[tr],axis=0);med=np.where(np.isnan(med),0,med) | |
| 132 | + Xn=Xnum.copy() | |
| 133 | + for j in range(Xn.shape[1]): | |
| 134 | + col=Xn[:,j];mask=np.isnan(col);col[mask]=med[j];Xn[:,j]=col | |
| 135 | + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg | |
| 136 | + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys | |
| 137 | + # split interne train/val pour early stopping | |
| 138 | + ntr=len(tr);perm=np.random.permutation(ntr);vcut=int(ntr*0.9) | |
| 139 | + tri=tr[perm[:vcut]];vai=tr[perm[vcut:]] | |
| 140 | + def T(a,dt): return torch.tensor(a,device=dev,dtype=dt) | |
| 141 | + model=Net([card[c] for c in cats],Xn.shape[1]).to(dev) | |
| 142 | + opt=torch.optim.Adam(model.parameters(),lr=1e-3,weight_decay=1e-5) | |
| 143 | + lossf=nn.MSELoss() | |
| 144 | + Xc_t=T(Xcat,torch.long);Xn_t=T(Xn,torch.float32);y_t=T(yz,torch.float32) | |
| 145 | + best=1e9;bstate=None;patience=0 | |
| 146 | + for ep in range(150): | |
| 147 | + model.train();idx=np.random.permutation(tri) | |
| 148 | + for i in range(0,len(idx),512): | |
| 149 | + b=idx[i:i+512] | |
| 150 | + opt.zero_grad();p=model(Xc_t[b],Xn_t[b]);l=lossf(p,y_t[b]);l.backward();opt.step() | |
| 151 | + model.eval() | |
| 152 | + with torch.no_grad(): | |
| 153 | + vp=model(Xc_t[vai],Xn_t[vai]);vl=lossf(vp,y_t[vai]).item() | |
| 154 | + if vl<best-1e-4: best=vl;bstate={k:v.clone() for k,v in model.state_dict().items()};patience=0 | |
| 155 | + else: | |
| 156 | + patience+=1 | |
| 157 | + if patience>=12: break | |
| 158 | + model.load_state_dict(bstate);model.eval() | |
| 159 | + with torch.no_grad(): | |
| 160 | + pz=model(Xc_t[te],Xn_t[te]).cpu().numpy() | |
| 161 | + return pz*ys+ym | |
| 162 | + | |
| 163 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 164 | +pred=np.zeros_like(y) | |
| 165 | +for f,(tr,te) in enumerate(kf.split(Xnum)): | |
| 166 | + pred[te]=train_predict(tr,te);print(f" fold {f+1} fait") | |
| 167 | +r2=r2_score(y,pred);err=np.abs(np.exp(pred-y)-1) | |
| 168 | +print("\n=== Réseau de neurones (embeddings + MLP) — 5-fold CV OOS ===") | |
| 169 | +print(f"R² OOS = {r2:.4f} | erreur médiane = {np.median(err)*100:.1f}% | ±15% = {np.mean(err<=.15)*100:.0f}% ±25% = {np.mean(err<=.25)*100:.0f}%") | |
| 170 | +print("\nRappel (mêmes folds) : OLS 0.628 | RF 0.678 | HistGBM 0.743") | |
added
scripts/08_torch_nn_robust.py
+192 −0
@@ -0,0 +1,192 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Réseau de neurones tabulaire ROBUSTE : embeddings + ResNet-MLP, | |
| 4 | +ensemble multi-seed, Huber loss, AdamW + cosine, grad-clip, early stopping. | |
| 5 | +5-fold CV out-of-sample, comparable aux autres modèles (seed=42).""" | |
| 6 | +import json, re, warnings | |
| 7 | +import numpy as np, pandas as pd | |
| 8 | +import torch, torch.nn as nn | |
| 9 | +from sklearn.model_selection import KFold | |
| 10 | +from sklearn.metrics import r2_score | |
| 11 | +warnings.filterwarnings("ignore") | |
| 12 | +CUR_YEAR=2026 | |
| 13 | +dev="mps" if torch.backends.mps.is_available() else "cpu" | |
| 14 | +print("device:",dev) | |
| 15 | +SEEDS=[42,1,7] # ensemble par fold | |
| 16 | +D_MAIN=256; N_BLOCKS=3; D_HIDDEN=512 | |
| 17 | + | |
| 18 | +# ======== reconstruction du dataset (identique v4) ======== | |
| 19 | +specs={} | |
| 20 | +for l in open("data/processed/specs.jsonl",encoding="utf-8"): | |
| 21 | + r=json.loads(l) | |
| 22 | + if r.get("specs"): specs[r["uid"]]=r["specs"] | |
| 23 | +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2} | |
| 24 | +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"] | |
| 25 | +def gfeat(uid): | |
| 26 | + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0}) | |
| 27 | + if not s: return o | |
| 28 | + for b in BOOL: o["g_"+b]=int(bool(s.get(b))) | |
| 29 | + o["g_cond"]=COND.get(s.get("condition","inconnu"),2) | |
| 30 | + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl) | |
| 31 | + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o | |
| 32 | +def parse_area(s): | |
| 33 | + if not s or not isinstance(s,str): return np.nan | |
| 34 | + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 35 | + try: v=float(n) | |
| 36 | + except: return np.nan | |
| 37 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 38 | +def cc(c): | |
| 39 | + if not c or not isinstance(c,str): return None | |
| 40 | + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 41 | +def ti(x): | |
| 42 | + try: return int(x) | |
| 43 | + except: return np.nan | |
| 44 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 45 | +def sd(s): | |
| 46 | + s=s.lower() | |
| 47 | + if "ft" in s: | |
| 48 | + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s) | |
| 49 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 50 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 51 | + if m: | |
| 52 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 53 | + except: return None | |
| 54 | + return None | |
| 55 | +def ra(d): | |
| 56 | + if not d or "x" not in d: return None | |
| 57 | + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 58 | +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 59 | +def pr(rooms): | |
| 60 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 61 | + for k in RF: o[k]=0 | |
| 62 | + if not rooms: return o | |
| 63 | + ar=0.0;lv=set() | |
| 64 | + for r in rooms: | |
| 65 | + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1 | |
| 66 | + if l: lv.add(l) | |
| 67 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 68 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 69 | + for k,kw in RF.items(): | |
| 70 | + if any(w in nm for w in kw): o[k]=1 | |
| 71 | + a=ra(r.get("dimensions")) | |
| 72 | + if a: ar+=a | |
| 73 | + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o | |
| 74 | +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[] | |
| 75 | +for d in data: | |
| 76 | + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or [] | |
| 77 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 78 | + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 79 | + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels") | |
| 80 | + try: st=float(str(st).replace(",",".")) | |
| 81 | + except: st=np.nan | |
| 82 | + uid=f"{d.get('source')}:{d.get('listingId')}" | |
| 83 | + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))} | |
| 84 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 85 | + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 86 | + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec) | |
| 87 | +df=pd.DataFrame(rows) | |
| 88 | +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 89 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 90 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 91 | +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan | |
| 92 | +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 93 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 94 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median() | |
| 95 | +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 96 | +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill] | |
| 97 | +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 98 | +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 99 | +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 100 | +df=df.reset_index(drop=True);print("N=",len(df)) | |
| 101 | + | |
| 102 | +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 103 | +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL] | |
| 104 | +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv | |
| 105 | +# indicateurs de manquant (robustesse) pour les colonnes à NaN | |
| 106 | +missnum=["sqft_final","lot_sqft","beds","baths","storeys","age","age2"] | |
| 107 | +for c in missnum: df["miss_"+c]=df[c].isna().astype(np.float32) | |
| 108 | +num=num+["miss_"+c for c in missnum] | |
| 109 | +y=np.log(df["price"].values).astype(np.float32) | |
| 110 | +cats=["city_fe","ptype_fe","source"];card={} | |
| 111 | +for c in cats: | |
| 112 | + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};card[c]=len(u) | |
| 113 | + df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64) | |
| 114 | +Xnum=df[num].astype(np.float32).values | |
| 115 | +Xcat=df[[c+"_ix" for c in cats]].values.astype(np.int64) | |
| 116 | + | |
| 117 | +class ResBlock(nn.Module): | |
| 118 | + def __init__(self,d,dh,p): | |
| 119 | + super().__init__() | |
| 120 | + self.bn=nn.BatchNorm1d(d);self.l1=nn.Linear(d,dh);self.act=nn.GELU() | |
| 121 | + self.dp1=nn.Dropout(p);self.l2=nn.Linear(dh,d);self.dp2=nn.Dropout(p) | |
| 122 | + def forward(self,x): | |
| 123 | + z=self.bn(x);z=self.act(self.l1(z));z=self.dp1(z);z=self.l2(z);z=self.dp2(z);return x+z | |
| 124 | + | |
| 125 | +class ResNetTab(nn.Module): | |
| 126 | + def __init__(self,ncard,nnum): | |
| 127 | + super().__init__() | |
| 128 | + self.embs=nn.ModuleList();tot=0 | |
| 129 | + for c in ncard: | |
| 130 | + d=max(2,min(64,round(1.6*c**0.56)));self.embs.append(nn.Embedding(c,d));tot+=d | |
| 131 | + self.embdp=nn.Dropout(0.1) | |
| 132 | + self.bn0=nn.BatchNorm1d(nnum) | |
| 133 | + self.proj=nn.Linear(tot+nnum,D_MAIN) | |
| 134 | + self.blocks=nn.ModuleList([ResBlock(D_MAIN,D_HIDDEN,0.2) for _ in range(N_BLOCKS)]) | |
| 135 | + self.head=nn.Sequential(nn.BatchNorm1d(D_MAIN),nn.GELU(),nn.Linear(D_MAIN,1)) | |
| 136 | + def forward(self,xc,xn): | |
| 137 | + e=[emb(xc[:,i]) for i,emb in enumerate(self.embs)] | |
| 138 | + x=torch.cat([self.embdp(torch.cat(e,1)),self.bn0(xn)],1) | |
| 139 | + x=self.proj(x) | |
| 140 | + for b in self.blocks: x=b(x) | |
| 141 | + return self.head(x).squeeze(1) | |
| 142 | + | |
| 143 | +def train_one(tr,te,seed): | |
| 144 | + torch.manual_seed(seed);np.random.seed(seed) | |
| 145 | + med=np.nanmedian(Xnum[tr],axis=0);med=np.where(np.isnan(med),0,med) | |
| 146 | + Xn=Xnum.copy() | |
| 147 | + for j in range(Xn.shape[1]): | |
| 148 | + col=Xn[:,j];col[np.isnan(col)]=med[j];Xn[:,j]=col | |
| 149 | + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg | |
| 150 | + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys | |
| 151 | + ntr=len(tr);perm=np.random.permutation(ntr);vc_=int(ntr*0.9) | |
| 152 | + tri=tr[perm[:vc_]];vai=tr[perm[vc_:]] | |
| 153 | + def T(a,dt): return torch.tensor(a,device=dev,dtype=dt) | |
| 154 | + model=ResNetTab([card[c] for c in cats],Xn.shape[1]).to(dev) | |
| 155 | + opt=torch.optim.AdamW(model.parameters(),lr=2e-3,weight_decay=1e-4) | |
| 156 | + EPOCHS=200 | |
| 157 | + sched=torch.optim.lr_scheduler.CosineAnnealingLR(opt,T_max=EPOCHS) | |
| 158 | + lossf=nn.SmoothL1Loss(beta=1.0) # Huber : robuste aux queues | |
| 159 | + Xc_t=T(Xcat,torch.long);Xn_t=T(Xn,torch.float32);y_t=T(yz,torch.float32) | |
| 160 | + best=1e9;bstate=None;pat=0 | |
| 161 | + for ep in range(EPOCHS): | |
| 162 | + model.train();idx=np.random.permutation(tri) | |
| 163 | + for i in range(0,len(idx),512): | |
| 164 | + b=idx[i:i+512] | |
| 165 | + opt.zero_grad();p=model(Xc_t[b],Xn_t[b]);l=lossf(p,y_t[b]) | |
| 166 | + l.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5.0);opt.step() | |
| 167 | + sched.step() | |
| 168 | + model.eval() | |
| 169 | + with torch.no_grad(): | |
| 170 | + vp=model(Xc_t[vai],Xn_t[vai]);vl=((vp-y_t[vai])**2).mean().item() | |
| 171 | + if vl<best-1e-4: best=vl;bstate={k:v.clone() for k,v in model.state_dict().items()};pat=0 | |
| 172 | + else: | |
| 173 | + pat+=1 | |
| 174 | + if pat>=20: break | |
| 175 | + model.load_state_dict(bstate);model.eval() | |
| 176 | + with torch.no_grad(): | |
| 177 | + pz=model(Xc_t[te],Xn_t[te]).cpu().numpy() | |
| 178 | + return pz*ys+ym | |
| 179 | + | |
| 180 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 181 | +pred=np.zeros_like(y) | |
| 182 | +for f,(tr,te) in enumerate(kf.split(Xnum)): | |
| 183 | + ps=np.zeros(len(te),dtype=np.float64) | |
| 184 | + for sd_ in SEEDS: | |
| 185 | + ps+=train_one(tr,te,sd_) | |
| 186 | + pred[te]=ps/len(SEEDS) # moyenne d'ensemble | |
| 187 | + r2f=r2_score(y[te],pred[te]);print(f" fold {f+1}: R²={r2f:.4f} (ensemble de {len(SEEDS)})") | |
| 188 | +r2=r2_score(y,pred);err=np.abs(np.exp(pred-y)-1) | |
| 189 | +print("\n=== NN ROBUSTE (ResNet-tab, ensemble multi-seed) — 5-fold CV OOS ===") | |
| 190 | +print(f"R² OOS = {r2:.4f} | erreur médiane = {np.median(err)*100:.1f}% | ±15% = {np.mean(err<=.15)*100:.0f}% | ±25% = {np.mean(err<=.25)*100:.0f}%") | |
| 191 | +print("\nRappel (mêmes folds): OLS 0.628 | RF 0.678 | NN simple 0.711 | HistGBM 0.743") | |
| 192 | +np.save("data/processed/nn_robust_oos.npy",pred) | |
added
scripts/09_v5_ml_text_vision.py
+213 −0
@@ -0,0 +1,213 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""v5 : ablation texte (v4) vs texte+vision (v5). GBM + NN robuste + blend, 5-fold CV.""" | |
| 4 | +import json, re, warnings | |
| 5 | +import numpy as np, pandas as pd | |
| 6 | +import torch, torch.nn as nn | |
| 7 | +from sklearn.ensemble import HistGradientBoostingRegressor | |
| 8 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 9 | +from sklearn.metrics import r2_score | |
| 10 | +warnings.filterwarnings("ignore") | |
| 11 | +torch.manual_seed(42); np.random.seed(42) | |
| 12 | +CUR_YEAR=2026 | |
| 13 | +dev="mps" if torch.backends.mps.is_available() else "cpu"; print("device:",dev) | |
| 14 | + | |
| 15 | +# ---- GPT specs ---- | |
| 16 | +specs={} | |
| 17 | +for l in open("data/processed/specs.jsonl",encoding="utf-8"): | |
| 18 | + r=json.loads(l) | |
| 19 | + if r.get("specs"): specs[r["uid"]]=r["specs"] | |
| 20 | +# ---- vision ---- | |
| 21 | +facade={} | |
| 22 | +for l in open("data/processed/facade_specs.jsonl",encoding="utf-8"): | |
| 23 | + r=json.loads(l) | |
| 24 | + if r.get("facade"): facade[r["uid"]]=r["facade"] | |
| 25 | +print("specs GPT:",len(specs),"| façades:",len(facade)) | |
| 26 | + | |
| 27 | +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2} | |
| 28 | +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"] | |
| 29 | +def gfeat(uid): | |
| 30 | + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0}) | |
| 31 | + if not s: return o | |
| 32 | + for b in BOOL: o["g_"+b]=int(bool(s.get(b))) | |
| 33 | + o["g_cond"]=COND.get(s.get("condition","inconnu"),2) | |
| 34 | + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl) | |
| 35 | + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o | |
| 36 | +RENO={"a_renover":0,"date":1,"standard":2,"renove":3,"haut_de_gamme":4,"inconnu":2} | |
| 37 | +LAND={"aucun":0,"minimal":1,"entretenu":2,"amenage_soigne":3} | |
| 38 | +SETS=["urbain_dense","banlieue","rural","bord_eau","boise"] | |
| 39 | +MATS=["brique","pierre","vinyle","bois","aluminium","stucco","crepi","beton"] | |
| 40 | +def vfeat(uid): | |
| 41 | + f=facade.get(uid) | |
| 42 | + o={"v_cond":5,"v_curb":5,"v_photoq":5,"v_reno":2,"v_land":1,"v_luxury":0,"v_garage_vis":0,"v_is_ext":0} | |
| 43 | + for s in SETS: o["v_set_"+s]=0 | |
| 44 | + for m in MATS: o["v_mat_"+m]=0 | |
| 45 | + if not f: return o | |
| 46 | + o["v_cond"]=f.get("condition_score") or 5;o["v_curb"]=f.get("curb_appeal") or 5;o["v_photoq"]=f.get("photo_quality") or 5 | |
| 47 | + o["v_reno"]=RENO.get(f.get("renovation_level","inconnu"),2);o["v_land"]=LAND.get(f.get("landscaping","minimal"),1) | |
| 48 | + o["v_luxury"]=int(bool(f.get("curb_luxury")));o["v_garage_vis"]=int(bool(f.get("has_garage_visible")));o["v_is_ext"]=int(bool(f.get("is_exterior"))) | |
| 49 | + st=f.get("setting") | |
| 50 | + if st in SETS: o["v_set_"+st]=1 | |
| 51 | + for m in (f.get("exterior_materials") or []): | |
| 52 | + if m in MATS: o["v_mat_"+m]=1 | |
| 53 | + return o | |
| 54 | +# ---- parsing structurel ---- | |
| 55 | +def pa(s): | |
| 56 | + if not s or not isinstance(s,str): return np.nan | |
| 57 | + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 58 | + try: v=float(n) | |
| 59 | + except: return np.nan | |
| 60 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 61 | +def cc(c): | |
| 62 | + if not c or not isinstance(c,str): return None | |
| 63 | + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 64 | +def ti(x): | |
| 65 | + try: return int(x) | |
| 66 | + except: return np.nan | |
| 67 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 68 | +def sd(s): | |
| 69 | + s=s.lower() | |
| 70 | + if "ft" in s: | |
| 71 | + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s) | |
| 72 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 73 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 74 | + if m: | |
| 75 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 76 | + except: return None | |
| 77 | + return None | |
| 78 | +def ra(d): | |
| 79 | + if not d or "x" not in d: return None | |
| 80 | + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 81 | +RFR={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 82 | +def pr(rooms): | |
| 83 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 84 | + for k in RFR: o[k]=0 | |
| 85 | + if not rooms: return o | |
| 86 | + ar=0.0;lv=set() | |
| 87 | + for r in rooms: | |
| 88 | + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1 | |
| 89 | + if l: lv.add(l) | |
| 90 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 91 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 92 | + for k,kw in RFR.items(): | |
| 93 | + if any(w in nm for w in kw): o[k]=1 | |
| 94 | + a=ra(r.get("dimensions")) | |
| 95 | + if a: ar+=a | |
| 96 | + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o | |
| 97 | + | |
| 98 | +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[] | |
| 99 | +for d in data: | |
| 100 | + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or [] | |
| 101 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 102 | + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 103 | + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels") | |
| 104 | + try: st=float(str(st).replace(",",".")) | |
| 105 | + except: st=np.nan | |
| 106 | + uid=f"{d.get('source')}:{d.get('listingId')}" | |
| 107 | + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":pa(p.get("squareFeet")),"lot_sqft":pa(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))} | |
| 108 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 109 | + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 110 | + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rec.update(vfeat(uid));rows.append(rec) | |
| 111 | +df=pd.DataFrame(rows) | |
| 112 | +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 113 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 114 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 115 | +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan | |
| 116 | +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 117 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 118 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median() | |
| 119 | +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 120 | +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill] | |
| 121 | +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 122 | +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 123 | +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 124 | +df=df.reset_index(drop=True);print("N=",len(df)) | |
| 125 | +y=np.log(df["price"].values).astype(np.float32) | |
| 126 | + | |
| 127 | +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 128 | +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL] | |
| 129 | +vv=["v_cond","v_curb","v_photoq","v_reno","v_land","v_luxury","v_garage_vis","v_is_ext"]+["v_set_"+s for s in SETS]+["v_mat_"+m for m in MATS] | |
| 130 | +base=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv | |
| 131 | +NUM_v4=base+gv | |
| 132 | +NUM_v5=base+gv+vv | |
| 133 | +missnum=["sqft_final","lot_sqft","beds","baths","storeys","age","age2"] | |
| 134 | +for c in missnum: df["miss_"+c]=df[c].isna().astype(np.float32) | |
| 135 | +MISS=["miss_"+c for c in missnum] | |
| 136 | + | |
| 137 | +cats=["city_fe","ptype_fe","source"];card={} | |
| 138 | +for c in cats: | |
| 139 | + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};card[c]=len(u);df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64) | |
| 140 | +Xcat_ix=df[[c+"_ix" for c in cats]].values.astype(np.int64) | |
| 141 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 142 | + | |
| 143 | +def gbm_cv(numcols): | |
| 144 | + X=pd.concat([df[numcols].reset_index(drop=True),pd.get_dummies(df[cats],dtype=float).reset_index(drop=True)],axis=1) | |
| 145 | + m=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42) | |
| 146 | + return cross_val_predict(m,X,y,cv=kf,n_jobs=-1) | |
| 147 | + | |
| 148 | +# ---- NN robuste ---- | |
| 149 | +D_MAIN=256;D_HID=512;NB=3;SEEDS=[42,1,7] | |
| 150 | +class RB(nn.Module): | |
| 151 | + def __init__(s,d,dh,p): | |
| 152 | + super().__init__();s.bn=nn.BatchNorm1d(d);s.l1=nn.Linear(d,dh);s.act=nn.GELU();s.d1=nn.Dropout(p);s.l2=nn.Linear(dh,d);s.d2=nn.Dropout(p) | |
| 153 | + def forward(s,x): z=s.d1(s.act(s.l1(s.bn(x))));return x+s.d2(s.l2(z)) | |
| 154 | +class Net(nn.Module): | |
| 155 | + def __init__(s,nc,nn_): | |
| 156 | + super().__init__();s.embs=nn.ModuleList();tot=0 | |
| 157 | + for c in nc: d=max(2,min(64,round(1.6*c**0.56)));s.embs.append(nn.Embedding(c,d));tot+=d | |
| 158 | + s.ed=nn.Dropout(0.1);s.bn0=nn.BatchNorm1d(nn_);s.proj=nn.Linear(tot+nn_,D_MAIN) | |
| 159 | + s.bl=nn.ModuleList([RB(D_MAIN,D_HID,0.2) for _ in range(NB)]) | |
| 160 | + s.head=nn.Sequential(nn.BatchNorm1d(D_MAIN),nn.GELU(),nn.Linear(D_MAIN,1)) | |
| 161 | + def forward(s,xc,xn): | |
| 162 | + e=[emb(xc[:,i]) for i,emb in enumerate(s.embs)] | |
| 163 | + x=s.proj(torch.cat([s.ed(torch.cat(e,1)),s.bn0(xn)],1)) | |
| 164 | + for b in s.bl: x=b(x) | |
| 165 | + return s.head(x).squeeze(1) | |
| 166 | +def nn_fold(Xnum,tr,te,seed): | |
| 167 | + torch.manual_seed(seed);np.random.seed(seed) | |
| 168 | + med=np.nanmedian(Xnum[tr],0);med=np.where(np.isnan(med),0,med);Xn=Xnum.copy() | |
| 169 | + for j in range(Xn.shape[1]): col=Xn[:,j];col[np.isnan(col)]=med[j];Xn[:,j]=col | |
| 170 | + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg | |
| 171 | + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys | |
| 172 | + perm=np.random.permutation(len(tr));cut=int(len(tr)*0.9);tri=tr[perm[:cut]];vai=tr[perm[cut:]] | |
| 173 | + T=lambda a,dt: torch.tensor(a,device=dev,dtype=dt) | |
| 174 | + model=Net([card[c] for c in cats],Xn.shape[1]).to(dev) | |
| 175 | + opt=torch.optim.AdamW(model.parameters(),lr=2e-3,weight_decay=1e-4) | |
| 176 | + sch=torch.optim.lr_scheduler.CosineAnnealingLR(opt,T_max=200);lf=nn.SmoothL1Loss(beta=1.0) | |
| 177 | + Xc=T(Xcat_ix,torch.long);Xnt=T(Xn,torch.float32);yt=T(yz,torch.float32) | |
| 178 | + best=1e9;bs=None;pat=0 | |
| 179 | + for ep in range(200): | |
| 180 | + model.train();idx=np.random.permutation(tri) | |
| 181 | + for i in range(0,len(idx),512): | |
| 182 | + b=idx[i:i+512];opt.zero_grad();l=lf(model(Xc[b],Xnt[b]),yt[b]);l.backward() | |
| 183 | + torch.nn.utils.clip_grad_norm_(model.parameters(),5.0);opt.step() | |
| 184 | + sch.step();model.eval() | |
| 185 | + with torch.no_grad(): vl=((model(Xc[vai],Xnt[vai])-yt[vai])**2).mean().item() | |
| 186 | + if vl<best-1e-4: best=vl;bs={k:v.clone() for k,v in model.state_dict().items()};pat=0 | |
| 187 | + else: | |
| 188 | + pat+=1 | |
| 189 | + if pat>=20: break | |
| 190 | + model.load_state_dict(bs);model.eval() | |
| 191 | + with torch.no_grad(): return model(Xc[te],Xnt[te]).cpu().numpy()*ys+ym | |
| 192 | +def nn_cv(numcols): | |
| 193 | + Xnum=df[numcols+MISS].astype(np.float32).values;pred=np.zeros_like(y) | |
| 194 | + for tr,te in kf.split(Xnum): | |
| 195 | + ps=np.zeros(len(te)) | |
| 196 | + for s in SEEDS: ps+=nn_fold(Xnum,tr,te,s) | |
| 197 | + pred[te]=ps/len(SEEDS) | |
| 198 | + return pred | |
| 199 | +def stat(p): | |
| 200 | + e=np.abs(np.exp(p-y)-1);return r2_score(y,p),np.median(e)*100,np.mean(e<=.15)*100,np.mean(e<=.25)*100 | |
| 201 | + | |
| 202 | +print("\n>>> GBM ablation") | |
| 203 | +g4=gbm_cv(NUM_v4);g5=gbm_cv(NUM_v5) | |
| 204 | +print("\n>>> NN v5 (peut prendre ~2 min)") | |
| 205 | +n5=nn_cv(NUM_v5) | |
| 206 | +b5=0.5*g5+0.5*n5 | |
| 207 | + | |
| 208 | +print("\n================ RÉSULTATS v5 (5-fold CV OOS) ================") | |
| 209 | +print(f"{'Modèle':<34}{'R²OOS':>8}{'MAEméd':>9}{'±15%':>7}{'±25%':>7}") | |
| 210 | +for nm,p in [("GBM v4 (texte)",g4),("GBM v5 (texte+vision)",g5),("NN v5 (texte+vision)",n5),("BLEND v5 (NN+GBM)",b5)]: | |
| 211 | + r2,m,a,b=stat(p);print(f"{nm:<34}{r2:>8.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%") | |
| 212 | +print("\nRappel v4 blend: 0.760 | GBM 0.743 | NN 0.741") | |
| 213 | +np.save("data/processed/v5_blend_oos.npy",b5) | |
added
scripts/10_final_blend.py
+117 −0
@@ -0,0 +1,117 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Blend (stacking) NN robuste + HistGradientBoosting, mêmes folds (seed=42).""" | |
| 4 | +import json, re, warnings | |
| 5 | +import numpy as np, pandas as pd | |
| 6 | +from sklearn.ensemble import HistGradientBoostingRegressor | |
| 7 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 8 | +from sklearn.metrics import r2_score | |
| 9 | +warnings.filterwarnings("ignore") | |
| 10 | +CUR_YEAR=2026 | |
| 11 | +# --- rebuild df exactement comme torch_nn_robust (même ordre) --- | |
| 12 | +specs={} | |
| 13 | +for l in open("data/processed/specs.jsonl",encoding="utf-8"): | |
| 14 | + r=json.loads(l) | |
| 15 | + if r.get("specs"): specs[r["uid"]]=r["specs"] | |
| 16 | +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2} | |
| 17 | +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"] | |
| 18 | +def gfeat(uid): | |
| 19 | + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0}) | |
| 20 | + if not s: return o | |
| 21 | + for b in BOOL: o["g_"+b]=int(bool(s.get(b))) | |
| 22 | + o["g_cond"]=COND.get(s.get("condition","inconnu"),2) | |
| 23 | + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl) | |
| 24 | + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o | |
| 25 | +def parse_area(s): | |
| 26 | + if not s or not isinstance(s,str): return np.nan | |
| 27 | + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 28 | + try: v=float(n) | |
| 29 | + except: return np.nan | |
| 30 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 31 | +def cc(c): | |
| 32 | + if not c or not isinstance(c,str): return None | |
| 33 | + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 34 | +def ti(x): | |
| 35 | + try: return int(x) | |
| 36 | + except: return np.nan | |
| 37 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 38 | +def sd(s): | |
| 39 | + s=s.lower() | |
| 40 | + if "ft" in s: | |
| 41 | + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s) | |
| 42 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 43 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 44 | + if m: | |
| 45 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 46 | + except: return None | |
| 47 | + return None | |
| 48 | +def ra(d): | |
| 49 | + if not d or "x" not in d: return None | |
| 50 | + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 51 | +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]} | |
| 52 | +def pr(rooms): | |
| 53 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 54 | + for k in RF: o[k]=0 | |
| 55 | + if not rooms: return o | |
| 56 | + ar=0.0;lv=set() | |
| 57 | + for r in rooms: | |
| 58 | + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1 | |
| 59 | + if l: lv.add(l) | |
| 60 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 61 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 62 | + for k,kw in RF.items(): | |
| 63 | + if any(w in nm for w in kw): o[k]=1 | |
| 64 | + a=ra(r.get("dimensions")) | |
| 65 | + if a: ar+=a | |
| 66 | + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o | |
| 67 | +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[] | |
| 68 | +for d in data: | |
| 69 | + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or [] | |
| 70 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 71 | + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 72 | + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels") | |
| 73 | + try: st=float(str(st).replace(",",".")) | |
| 74 | + except: st=np.nan | |
| 75 | + uid=f"{d.get('source')}:{d.get('listingId')}" | |
| 76 | + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))} | |
| 77 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 78 | + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 79 | + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec) | |
| 80 | +df=pd.DataFrame(rows) | |
| 81 | +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 82 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 83 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 84 | +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan | |
| 85 | +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 86 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 87 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median() | |
| 88 | +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 89 | +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill] | |
| 90 | +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 91 | +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 92 | +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 93 | +df=df.reset_index(drop=True) | |
| 94 | +y=np.log(df["price"].values) | |
| 95 | +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 96 | +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL] | |
| 97 | +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv | |
| 98 | +Xcat=pd.get_dummies(df[["city_fe","ptype_fe","source"]],dtype=float) | |
| 99 | +X=pd.concat([df[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1) | |
| 100 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 101 | +gbm=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42) | |
| 102 | +pgbm=cross_val_predict(gbm,X,y,cv=kf,n_jobs=-1) | |
| 103 | +pnn=np.load("data/processed/nn_robust_oos.npy") | |
| 104 | +assert len(pnn)==len(y),f"désalignement {len(pnn)} vs {len(y)}" | |
| 105 | +def stat(p): | |
| 106 | + e=np.abs(np.exp(p-y)-1) | |
| 107 | + return r2_score(y,p),np.median(e)*100,np.mean(e<=.15)*100,np.mean(e<=.25)*100 | |
| 108 | +print(f"{'Modèle':<30}{'R² OOS':>9}{'MAE méd':>9}{'±15%':>7}{'±25%':>7}") | |
| 109 | +for nm,p in [("HistGBM",pgbm),("NN robuste",pnn)]: | |
| 110 | + r2,m,a,b=stat(p);print(f"{nm:<30}{r2:>9.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%") | |
| 111 | +best=None | |
| 112 | +for w in [0.3,0.4,0.5,0.6,0.7]: | |
| 113 | + p=w*pgbm+(1-w)*pnn;r2,m,a,b=stat(p) | |
| 114 | + tag=f"Blend {w:.1f}·GBM+{1-w:.1f}·NN" | |
| 115 | + print(f"{tag:<30}{r2:>9.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%") | |
| 116 | + if best is None or r2>best[0]: best=(r2,w) | |
| 117 | +print(f"\nMeilleur blend: w_GBM={best[1]:.1f} R²={best[0]:.4f}") | |
added
scripts/11_extras_v5_robustness.py
+134 −0
@@ -0,0 +1,134 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Robustesse (par segment) + figures primes de localisation & validation IA.""" | |
| 4 | +import numpy as np, pandas as pd, warnings, statsmodels.formula.api as smf | |
| 5 | +import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt | |
| 6 | +warnings.filterwarnings("ignore") | |
| 7 | +plt.rcParams.update({"font.family":"serif","font.size":10,"axes.grid":True, | |
| 8 | + "grid.alpha":0.25,"axes.axisbelow":True,"figure.dpi":150,"savefig.bbox":"tight"}) | |
| 9 | +C1,C2,C3="#3B6EA5","#C4462E","#4A8C5B" | |
| 10 | +exec(open("v5_ml.py").read().split("kf=KFold")[0]) | |
| 11 | +amen_=list(AMEN.keys()) | |
| 12 | +bl=np.load("data/processed/v5_blend_oos.npy"); price=df["price"].values; y=np.log(price) | |
| 13 | +from sklearn.metrics import r2_score | |
| 14 | +def stat(mask): | |
| 15 | + yy=y[mask];pp=bl[mask];e=np.abs(np.exp(pp-yy)-1) | |
| 16 | + return len(yy),r2_score(yy,pp),np.median(e)*100,np.mean(e<=.15)*100 | |
| 17 | + | |
| 18 | +# ---- robustesse par segment ---- | |
| 19 | +seg=[] | |
| 20 | +seg.append(("\\textit{Full sample}",np.ones(len(y),bool))) | |
| 21 | +q1,q2=np.quantile(price,[1/3,2/3]) | |
| 22 | +seg.append(("\\quad Price tercile 1 (low)",price<=q1)) | |
| 23 | +seg.append(("\\quad Price tercile 2 (mid)",(price>q1)&(price<=q2))) | |
| 24 | +seg.append(("\\quad Price tercile 3 (high)",price>q2)) | |
| 25 | +seg.append(("\\quad Realtor.ca",df["source"].values=="realtor")) | |
| 26 | +seg.append(("\\quad DuProprio",df["source"].values=="duproprio")) | |
| 27 | +seg.append(("\\quad Measured area",df["has_sqft"].values==1)) | |
| 28 | +seg.append(("\\quad Imputed area",df["has_sqft"].values==0)) | |
| 29 | +rows=[] | |
| 30 | +for nm,m in seg: | |
| 31 | + n,r2,md,w=stat(m) | |
| 32 | + # R^2 non informatif sur une plage de prix restreinte (terciles) -> n/a | |
| 33 | + r2c = "n/a$^{\\dagger}$" if "tercile" in nm else f"{r2:.3f}" | |
| 34 | + rows.append(f" {nm} & {n:,} & {r2c} & {md:.1f}\\% & {w:.0f}\\% \\\\".replace(",","{,}")) | |
| 35 | +tab=r"""% Auto-generated | |
| 36 | +\begin{table}[htbp] | |
| 37 | + \centering | |
| 38 | + \begin{threeparttable} | |
| 39 | + \caption{Robustness: Blend Performance Across Subsamples} | |
| 40 | + \label{tab:robustness} | |
| 41 | + \begin{tabular}{@{}lcccc@{}} | |
| 42 | + \toprule | |
| 43 | + Subsample & $N$ & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% \\ | |
| 44 | + \midrule | |
| 45 | +"""+"\n".join(rows)+r""" | |
| 46 | + \bottomrule | |
| 47 | + \end{tabular} | |
| 48 | + \begin{tablenotes}[flushleft] | |
| 49 | + \footnotesize | |
| 50 | + \item \textit{Notes:} Out-of-sample performance of the text\,+\,vision blend | |
| 51 | + (Table~\ref{tab:models}, Panel~B) computed on subsamples of the same 5-fold | |
| 52 | + cross-validated predictions. Price terciles are defined on the actual list price. | |
| 53 | + ``Measured area'' are listings with a reported or room-reconstructed living area | |
| 54 | + (priced by Model~A); ``Imputed area'' listings are priced by the area-free model. | |
| 55 | + $^{\dagger}$Within a single price tercile the range of $\ln P$ is heavily | |
| 56 | + restricted, so $R^2$---which normalizes by within-group variance---is | |
| 57 | + uninformative; the median absolute error is the comparable accuracy metric across | |
| 58 | + terciles. | |
| 59 | + \end{tablenotes} | |
| 60 | + \end{threeparttable} | |
| 61 | +\end{table} | |
| 62 | +""" | |
| 63 | +open("paper/tables/robustness.tex","w").write(tab) | |
| 64 | +print("-> tables/robustness.tex"); [print(f" {nm[:28]:<30} n={stat(m)[0]:5d} R2={stat(m)[1]:.3f} md={stat(m)[2]:.1f}%") for nm,m in seg] | |
| 65 | + | |
| 66 | +# ---- OLS v5 pour primes de localisation ---- | |
| 67 | +df["ln_price"]=y | |
| 68 | +df["ln_sqft"]=np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median())) | |
| 69 | +df["ln_lot"]=np.log(df["lot_sqft"].fillna(df["lot_sqft"].median()));df["has_lot"]=df["lot_sqft"].notna().astype(int) | |
| 70 | +df["has_age"]=df["age"].notna().astype(int) | |
| 71 | +for c in ["beds","baths","storeys","age","age2"]: | |
| 72 | + df[c]=df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median()) | |
| 73 | +df["parking"]=df["parking"].fillna(0) | |
| 74 | +form=("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+"+"+".join(amen_)+"+n_images_c+virtual_tour+"+"+".join(rv)+"+"+"+".join(gv)+"+"+"+".join(vv)+"+C(city_fe)") | |
| 75 | +mA=smf.ols(form,data=df[df["has_sqft"]==1]).fit(cov_type="HC3") | |
| 76 | +pct=lambda b:(np.exp(b)-1)*100 | |
| 77 | +cf={k.split("T.")[1].rstrip("]"):pct(v) for k,v in mA.params.items() if k.startswith("C(city_fe)")} | |
| 78 | +s=pd.Series(cf).sort_values() | |
| 79 | +top=s.tail(12);bot=s.head(8) | |
| 80 | +fig,ax=plt.subplots(figsize=(6.6,4.6)) | |
| 81 | +sel=pd.concat([bot,top]) | |
| 82 | +cols=[C2 if v<0 else C1 for v in sel.values] | |
| 83 | +ax.barh(range(len(sel)),sel.values,color=cols,edgecolor="white") | |
| 84 | +ax.set_yticks(range(len(sel)));ax.set_yticklabels([c.title() for c in sel.index],fontsize=8) | |
| 85 | +ax.axvline(0,color="black",lw=0.8) | |
| 86 | +ax.set_xlabel("Location premium vs reference city (\\%)") | |
| 87 | +ax.set_title("Municipality fixed effects (largest and smallest)",fontsize=11) | |
| 88 | +fig.savefig("paper/figures/fig_premia.pdf");plt.close(fig);print("-> fig_premia.pdf") | |
| 89 | + | |
| 90 | +# ---- validation IA : prix moyen par score visuel ---- | |
| 91 | +ext=df[df["v_is_ext"]==1] | |
| 92 | +fig,axs=plt.subplots(1,2,figsize=(7.6,3.5)) | |
| 93 | +for ax,col,lab,c in [(axs[0],"v_curb","Vision curb appeal (1--10)",C3),(axs[1],"v_cond","Vision condition score (1--10)",C1)]: | |
| 94 | + g=ext.groupby(col)["price"].median()/1e3 | |
| 95 | + g=g[g.index>=3] | |
| 96 | + ax.plot(g.index,g.values,"-o",color=c,lw=2) | |
| 97 | + ax.set_xlabel(lab);ax.set_ylabel("Median price (\\$000)") | |
| 98 | +axs[0].set_title("Price rises with visual curb appeal",fontsize=10) | |
| 99 | +axs[1].set_title("Price rises with visual condition",fontsize=10) | |
| 100 | +fig.savefig("paper/figures/fig_validation.pdf");plt.close(fig);print("-> fig_validation.pdf") | |
| 101 | + | |
| 102 | +# ---- table résumé features IA ---- | |
| 103 | +def sh(col): return df[col].mean()*100 | |
| 104 | +aitab=r"""% Auto-generated | |
| 105 | +\begin{table}[htbp] | |
| 106 | + \centering | |
| 107 | + \begin{threeparttable} | |
| 108 | + \caption{Machine-Generated Feature Summary} | |
| 109 | + \label{tab:ai_summary} | |
| 110 | + \begin{tabular}{@{}lc@{\hskip 1.2cm}lc@{}} | |
| 111 | + \toprule | |
| 112 | + Text (LLM) & Share & Vision (façade) & Mean / share \\ | |
| 113 | + \midrule | |
| 114 | +"""+f""" Recently renovated & {sh('g_recently_renovated'):.0f}\\% & Condition score (1--10) & {df['v_cond'].mean():.1f} \\\\ | |
| 115 | + Renovated kitchen & {sh('g_renovated_kitchen'):.0f}\\% & Curb appeal (1--10) & {df['v_curb'].mean():.1f} \\\\ | |
| 116 | + Hardwood floor & {sh('g_hardwood'):.0f}\\% & Luxury cue & {sh('v_luxury'):.0f}\\% \\\\ | |
| 117 | + Quartz/granite counter & {sh('g_quartz_granite'):.0f}\\% & Garage visible & {sh('v_garage_vis'):.0f}\\% \\\\ | |
| 118 | + New construction & {sh('g_new_construction'):.0f}\\% & Brick façade & {sh('v_mat_brique'):.0f}\\% \\\\ | |
| 119 | + Income property & {sh('g_income_property'):.0f}\\% & Waterfront setting & {sh('v_set_bord_eau'):.0f}\\% \\\\ | |
| 120 | + Water view & {sh('g_view_water'):.0f}\\% & Exterior shot detected & {sh('v_is_ext'):.0f}\\% \\\\ | |
| 121 | +""" + r""" \bottomrule | |
| 122 | + \end{tabular} | |
| 123 | + \begin{tablenotes}[flushleft] | |
| 124 | + \footnotesize | |
| 125 | + \item \textit{Notes:} Prevalence of selected machine-generated attributes across | |
| 126 | + the """ + f"{len(df):,}".replace(",","{,}") + r""" cleaned listings. Text attributes are extracted from the | |
| 127 | + description by \texttt{gpt-4o-mini}; vision attributes from the façade photograph by | |
| 128 | + \texttt{claude-haiku-4-5}. Scores are 1--10 means; the remainder are shares. | |
| 129 | + \end{tablenotes} | |
| 130 | + \end{threeparttable} | |
| 131 | +\end{table} | |
| 132 | +""" | |
| 133 | +open("paper/tables/ai_summary.tex","w").write(aitab) | |
| 134 | +print("-> tables/ai_summary.tex") | |
added
scripts/12_ablation_extras.py
+47 −0
@@ -0,0 +1,47 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Ablation CV complète (4 couches x OLS/GBM) pour cohérence du tableau ablation.tex.""" | |
| 4 | +import json, warnings | |
| 5 | +import numpy as np, pandas as pd | |
| 6 | +from sklearn.linear_model import LinearRegression | |
| 7 | +from sklearn.ensemble import HistGradientBoostingRegressor | |
| 8 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 9 | +from sklearn.metrics import r2_score | |
| 10 | +warnings.filterwarnings("ignore") | |
| 11 | + | |
| 12 | +exec(open("v5_ml.py").read().split("kf=KFold")[0]) | |
| 13 | +kf = KFold(n_splits=5, shuffle=True, random_state=42) | |
| 14 | +amen_ = list(AMEN.keys()) | |
| 15 | + | |
| 16 | +L1 = ["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2"] + amen_ | |
| 17 | +L2 = L1 + ["n_images_c","virtual_tour"] + rv | |
| 18 | +L3 = L2 + gv | |
| 19 | +L4 = L3 + vv | |
| 20 | +print("feats:", len(L1), len(L2), len(L3), len(L4)) | |
| 21 | + | |
| 22 | +Xcat = pd.get_dummies(df[cats], prefix=cats, dtype=float) | |
| 23 | +dfi = df.copy() | |
| 24 | +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]: | |
| 25 | + dfi[c] = dfi.groupby("ptype")[c].transform(lambda s: s.fillna(s.median())) | |
| 26 | + dfi[c] = dfi[c].fillna(dfi[c].median()) | |
| 27 | +dfi["parking"] = dfi["parking"].fillna(0) | |
| 28 | + | |
| 29 | +def stats(pred): | |
| 30 | + e = np.abs(np.exp(pred - y) - 1) | |
| 31 | + return {"r2": float(r2_score(y, pred)), "mdae": float(np.median(e) * 100), | |
| 32 | + "w15": float(np.mean(e <= .15) * 100), "w25": float(np.mean(e <= .25) * 100)} | |
| 33 | +def cv_ols(numcols): | |
| 34 | + X = pd.concat([dfi[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1) | |
| 35 | + X = X.fillna(X.median()) | |
| 36 | + return stats(cross_val_predict(LinearRegression(), X, y, cv=kf, n_jobs=-1)) | |
| 37 | +def cv_gbm(numcols): | |
| 38 | + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1) | |
| 39 | + m = HistGradientBoostingRegressor(max_iter=600, learning_rate=0.05, max_leaf_nodes=63, l2_regularization=1.0, random_state=42) | |
| 40 | + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1)) | |
| 41 | + | |
| 42 | +out = {} | |
| 43 | +for name, cols in [("L1", L1), ("L2", L2), ("L3", L3), ("L4", L4)]: | |
| 44 | + out[name] = {"nfeat": len(cols), "ols": cv_ols(cols), "gbm": cv_gbm(cols)} | |
| 45 | + print(name, out[name]) | |
| 46 | +json.dump(out, open("results/ablation_extras.json", "w"), indent=1) | |
| 47 | +print("-> ablation_extras.json") | |
added
scripts/13_revision_extras.py
+112 −0
@@ -0,0 +1,112 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Estimations complémentaires pour la révision JHE : | |
| 4 | + - OLS A/B avec SE HC3 ET clusterisées par municipalité | |
| 5 | + - ablation in-sample du modèle A (base / +texte / +texte+vision) | |
| 6 | + - effets fixes de ville extrêmes | |
| 7 | + - CV 5-fold OLS et RF pour les panels A (texte) et B (texte+vision) | |
| 8 | + Résultats -> revision_extras.json | |
| 9 | +""" | |
| 10 | +import json, warnings | |
| 11 | +import numpy as np, pandas as pd | |
| 12 | +import statsmodels.formula.api as smf | |
| 13 | +from sklearn.linear_model import LinearRegression | |
| 14 | +from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor | |
| 15 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 16 | +from sklearn.metrics import r2_score | |
| 17 | +warnings.filterwarnings("ignore") | |
| 18 | + | |
| 19 | +# pipeline v5 jusqu'avant la CV (construit df, AMEN, rv, gv, vv, NUM_v4, NUM_v5, MISS, cats, y, kf) | |
| 20 | +exec(open("v5_ml.py").read().split("kf=KFold")[0]) | |
| 21 | +amen_ = list(AMEN.keys()) | |
| 22 | +out = {} | |
| 23 | + | |
| 24 | +# ---------- OLS A/B : HC3 vs cluster ---------- | |
| 25 | +df["ln_price"] = np.log(df["price"]) | |
| 26 | +df["ln_sqft"] = np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median())) | |
| 27 | +df["ln_lot"] = np.log(df["lot_sqft"].fillna(df["lot_sqft"].median())); df["has_lot"] = df["lot_sqft"].notna().astype(int) | |
| 28 | +df["has_age"] = df["age"].notna().astype(int) | |
| 29 | +for c in ["beds","baths","storeys","age","age2"]: | |
| 30 | + df[c] = df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median()) | |
| 31 | +df["parking"] = df["parking"].fillna(0) | |
| 32 | + | |
| 33 | +form = ("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+" | |
| 34 | + + "+".join(amen_) + "+n_images_c+virtual_tour+" + "+".join(rv) + "+" + "+".join(gv) + "+" + "+".join(vv) + "+C(city_fe)") | |
| 35 | +dfA = df[df["has_sqft"] == 1] | |
| 36 | +mA = smf.ols(form, data=dfA).fit(cov_type="HC3") | |
| 37 | +mB = smf.ols(form.replace("ln_sqft+",""), data=df).fit(cov_type="HC3") | |
| 38 | +mAc = smf.ols(form, data=dfA).fit(cov_type="cluster", cov_kwds={"groups": dfA["city_fe"]}) | |
| 39 | +mBc = smf.ols(form.replace("ln_sqft+",""), data=df).fit(cov_type="cluster", cov_kwds={"groups": df["city_fe"]}) | |
| 40 | + | |
| 41 | +keys = ["ln_sqft","ln_lot","beds","baths","parking","age","age2", | |
| 42 | + "central_air","transit","waterfront", | |
| 43 | + "g_cond","g_renovated_kitchen","g_hardwood","g_new_construction","g_income_property", | |
| 44 | + "v_curb","v_reno","v_luxury","v_garage_vis", | |
| 45 | + "v_mat_brique","v_mat_pierre","v_mat_vinyle","v_set_rural","v_land","v_cond"] | |
| 46 | +def grab(m, mc): | |
| 47 | + o = {} | |
| 48 | + for k in keys: | |
| 49 | + if k in m.params.index: | |
| 50 | + o[k] = {"b": float(m.params[k]), "se_hc3": float(m.bse[k]), "p_hc3": float(m.pvalues[k]), | |
| 51 | + "se_cl": float(mc.bse[k]), "p_cl": float(mc.pvalues[k])} | |
| 52 | + return o | |
| 53 | +out["modelA"] = {"r2": mA.rsquared, "adj": mA.rsquared_adj, "n": int(mA.nobs), "coef": grab(mA, mAc)} | |
| 54 | +out["modelB"] = {"r2": mB.rsquared, "adj": mB.rsquared_adj, "n": int(mB.nobs), "coef": grab(mB, mBc)} | |
| 55 | +print("A R2=%.4f N=%d | B R2=%.4f N=%d" % (mA.rsquared, mA.nobs, mB.rsquared, mB.nobs)) | |
| 56 | + | |
| 57 | +# ---------- ablation in-sample modèle A ---------- | |
| 58 | +base_form = ("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+" | |
| 59 | + + "+".join(amen_) + "+n_images_c+virtual_tour+" + "+".join(rv) + "+C(city_fe)") | |
| 60 | +mA0 = smf.ols(base_form, data=dfA).fit() | |
| 61 | +mA1 = smf.ols(base_form + "+" + "+".join(gv), data=dfA).fit() | |
| 62 | +mA2 = smf.ols(base_form + "+" + "+".join(gv) + "+" + "+".join(vv), data=dfA).fit() | |
| 63 | +out["ablationA_insample"] = {"base": mA0.rsquared, "plus_text": mA1.rsquared, "plus_text_vision": mA2.rsquared} | |
| 64 | +print("ablation A in-sample:", out["ablationA_insample"]) | |
| 65 | + | |
| 66 | +# ---------- effets fixes ville extrêmes ---------- | |
| 67 | +fes = {k.split("T.")[1].rstrip("]"): float(v) for k, v in mA.params.items() if k.startswith("C(city_fe)")} | |
| 68 | +top = sorted(fes.items(), key=lambda kv: -kv[1])[:12] | |
| 69 | +bot = sorted(fes.items(), key=lambda kv: kv[1])[:8] | |
| 70 | +out["city_fe_top"] = [(k, (np.exp(v) - 1) * 100) for k, v in top] | |
| 71 | +out["city_fe_bot"] = [(k, (np.exp(v) - 1) * 100) for k, v in bot] | |
| 72 | +print("top villes:", out["city_fe_top"][:4]) | |
| 73 | + | |
| 74 | +# ---------- CV : OLS et RF, panels texte / texte+vision ---------- | |
| 75 | +kf = KFold(n_splits=5, shuffle=True, random_state=42) | |
| 76 | +Xcat = pd.get_dummies(df[cats], prefix=cats, dtype=float) | |
| 77 | +dfi = df.copy() | |
| 78 | +def stats(pred): | |
| 79 | + e = np.abs(np.exp(pred - y) - 1) | |
| 80 | + return {"r2": float(r2_score(y, pred)), "mdae": float(np.median(e) * 100), | |
| 81 | + "w15": float(np.mean(e <= .15) * 100), "w25": float(np.mean(e <= .25) * 100)} | |
| 82 | +def cv_ols(numcols): | |
| 83 | + X = pd.concat([dfi[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1) | |
| 84 | + X = X.fillna(X.median()) | |
| 85 | + return stats(cross_val_predict(LinearRegression(), X, y, cv=kf, n_jobs=-1)) | |
| 86 | +def cv_rf(numcols): | |
| 87 | + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1) | |
| 88 | + m = RandomForestRegressor(n_estimators=400, max_features=0.5, min_samples_leaf=3, n_jobs=-1, random_state=42) | |
| 89 | + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1)) | |
| 90 | +def cv_gbm(numcols): | |
| 91 | + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1) | |
| 92 | + m = HistGradientBoostingRegressor(max_iter=600, learning_rate=0.05, max_leaf_nodes=63, l2_regularization=1.0, random_state=42) | |
| 93 | + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1)) | |
| 94 | + | |
| 95 | +# matrice OLS imputée médiane par type (comme hedonic_ml) | |
| 96 | +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]: | |
| 97 | + dfi[c] = dfi.groupby("ptype")[c].transform(lambda s: s.fillna(s.median())) | |
| 98 | + dfi[c] = dfi[c].fillna(dfi[c].median()) | |
| 99 | +dfi["parking"] = dfi["parking"].fillna(0) | |
| 100 | + | |
| 101 | +print(">>> CV OLS v4/v5") | |
| 102 | +out["cv"] = {"ols_v4": cv_ols(NUM_v4), "ols_v5": cv_ols(NUM_v5)} | |
| 103 | +print("OLS:", out["cv"]["ols_v4"], out["cv"]["ols_v5"]) | |
| 104 | +print(">>> CV GBM v4/v5 (contrôle)") | |
| 105 | +out["cv"]["gbm_v4"] = cv_gbm(NUM_v4); out["cv"]["gbm_v5"] = cv_gbm(NUM_v5) | |
| 106 | +print("GBM:", out["cv"]["gbm_v4"], out["cv"]["gbm_v5"]) | |
| 107 | +print(">>> CV RF v4/v5 (plus long)") | |
| 108 | +out["cv"]["rf_v4"] = cv_rf(NUM_v4); out["cv"]["rf_v5"] = cv_rf(NUM_v5) | |
| 109 | +print("RF:", out["cv"]["rf_v4"], out["cv"]["rf_v5"]) | |
| 110 | + | |
| 111 | +json.dump(out, open("results/revision_extras.json", "w"), indent=1, default=float) | |
| 112 | +print("-> revision_extras.json") | |
added
scripts/14_figures_v5.py
+152 −0
@@ -0,0 +1,152 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Figures + tableau d'ablation détaillé pour le papier v5.""" | |
| 4 | +import numpy as np, pandas as pd, warnings, json | |
| 5 | +import matplotlib | |
| 6 | +matplotlib.use("Agg") | |
| 7 | +import matplotlib.pyplot as plt | |
| 8 | +from sklearn.linear_model import LinearRegression | |
| 9 | +from sklearn.ensemble import HistGradientBoostingRegressor | |
| 10 | +from sklearn.model_selection import KFold, cross_val_predict | |
| 11 | +from sklearn.metrics import r2_score | |
| 12 | +from sklearn.inspection import permutation_importance | |
| 13 | +warnings.filterwarnings("ignore") | |
| 14 | +plt.rcParams.update({"font.family":"serif","font.size":10,"axes.grid":True, | |
| 15 | + "grid.alpha":0.25,"axes.axisbelow":True,"figure.dpi":150,"savefig.bbox":"tight"}) | |
| 16 | +C_OLS,C_GBM,C_BL="#3B6EA5","#C4462E","#4A8C5B" | |
| 17 | + | |
| 18 | +# build df via v5_ml prefix | |
| 19 | +exec(open("v5_ml.py").read().split("kf=KFold")[0]) | |
| 20 | +kf=KFold(n_splits=5,shuffle=True,random_state=42) | |
| 21 | +onehot=pd.get_dummies(df[cats],dtype=float).reset_index(drop=True) | |
| 22 | + | |
| 23 | +struct=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2"] | |
| 24 | +amen_=list(AMEN.keys()) | |
| 25 | +S0=struct+amen_ | |
| 26 | +S1=S0+rv+["n_images_c","virtual_tour"] | |
| 27 | +S2=S1+gv | |
| 28 | +S3=S2+vv | |
| 29 | +sets=[("Structural\n+ amenity",S0),("+ Rooms\n+ listing",S1),("+ Text\n(LLM)",S2),("+ Vision\n(façade)",S3)] | |
| 30 | + | |
| 31 | +def ols_r2(cols): | |
| 32 | + Xi=df[cols].copy() | |
| 33 | + for c in cols: Xi[c]=Xi[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median()) | |
| 34 | + X=pd.concat([Xi.reset_index(drop=True),onehot],axis=1) | |
| 35 | + p=cross_val_predict(LinearRegression(),X,y,cv=kf,n_jobs=-1);return r2_score(y,p),p | |
| 36 | +def gbm_r2(cols): | |
| 37 | + X=pd.concat([df[cols].reset_index(drop=True),onehot],axis=1) | |
| 38 | + m=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42) | |
| 39 | + p=cross_val_predict(m,X,y,cv=kf,n_jobs=-1);return r2_score(y,p),p | |
| 40 | + | |
| 41 | +ols=[];gbm=[];gpred=None | |
| 42 | +for nm,cols in sets: | |
| 43 | + ro,_=ols_r2(cols);rg,pg=gbm_r2(cols);ols.append(ro);gbm.append(rg) | |
| 44 | + if nm.startswith("+ Vision"): gpred=pg | |
| 45 | + print(f"{nm.replace(chr(10),' '):<22} OLS={ro:.4f} GBM={rg:.4f}") | |
| 46 | + | |
| 47 | +# blend numbers (déjà calculés) : S2 texte, S3 texte+vision | |
| 48 | +blend={2:0.760,3:0.779} | |
| 49 | + | |
| 50 | +# ===== Figure 1 : ablation ===== | |
| 51 | +x=np.arange(len(sets)) | |
| 52 | +fig,ax=plt.subplots(figsize=(6.4,3.9)) | |
| 53 | +ax.plot(x,ols,"-o",color=C_OLS,lw=2,label="OLS (linear)") | |
| 54 | +ax.plot(x,gbm,"-s",color=C_GBM,lw=2,label="Gradient boosting") | |
| 55 | +ax.plot([2,3],[blend[2],blend[3]],"-D",color=C_BL,lw=2,label="Blend (NN+GBM)") | |
| 56 | +for xi,(ro,rg) in enumerate(zip(ols,gbm)): | |
| 57 | + ax.annotate(f"{rg:.3f}",(xi,rg),textcoords="offset points",xytext=(0,7),ha="center",fontsize=8,color=C_GBM) | |
| 58 | + ax.annotate(f"{ro:.3f}",(xi,ro),textcoords="offset points",xytext=(0,-13),ha="center",fontsize=8,color=C_OLS) | |
| 59 | +ax.annotate(f"{blend[3]:.3f}",(3,blend[3]),textcoords="offset points",xytext=(0,7),ha="center",fontsize=8,color=C_BL) | |
| 60 | +ax.set_xticks(x);ax.set_xticklabels([s[0] for s in sets]) | |
| 61 | +ax.set_ylabel(r"Out-of-sample $R^2$ (5-fold CV)");ax.set_ylim(0.55,0.80) | |
| 62 | +ax.legend(frameon=False,loc="lower right",fontsize=9) | |
| 63 | +ax.set_title("Incremental gain from each feature layer",fontsize=11) | |
| 64 | +fig.savefig("paper/figures/fig_ablation.pdf");plt.close(fig) | |
| 65 | +print("-> fig_ablation.pdf") | |
| 66 | + | |
| 67 | +# ===== Figure 2 : calibration + résidus (blend v5) ===== | |
| 68 | +try: bl=np.load("data/processed/v5_blend_oos.npy") | |
| 69 | +except Exception: bl=gpred | |
| 70 | +yp=np.exp(bl);ya=np.exp(y) | |
| 71 | +fig,axs=plt.subplots(1,2,figsize=(7.6,3.6)) | |
| 72 | +ax=axs[0] | |
| 73 | +hb=ax.hexbin(ya/1e3,yp/1e3,gridsize=45,bins="log",cmap="Blues",xscale="log",yscale="log",mincnt=1) | |
| 74 | +lim=[60,6000];ax.plot(lim,lim,"--",color="black",lw=1) | |
| 75 | +ax.set_xlim(lim);ax.set_ylim(lim);ax.set_xlabel("Actual price (\\$000)");ax.set_ylabel("Predicted price (\\$000)") | |
| 76 | +ax.set_title(f"Calibration (blend, $R^2$={r2_score(y,bl):.3f})",fontsize=10) | |
| 77 | +r=(yp/ya-1)*100 | |
| 78 | +ax2=axs[1] | |
| 79 | +ax2.hist(np.clip(r,-60,60),bins=50,color=C_BL,alpha=0.85,edgecolor="white",lw=0.3) | |
| 80 | +ax2.axvline(0,color="black",lw=1,ls="--") | |
| 81 | +ax2.set_xlabel("Prediction error (\\%)");ax2.set_ylabel("Listings") | |
| 82 | +ax2.set_title(f"Errors (median |.|={np.median(np.abs(r)):.1f}\\%)",fontsize=10) | |
| 83 | +fig.savefig("paper/figures/fig_calibration.pdf");plt.close(fig) | |
| 84 | +print("-> fig_calibration.pdf") | |
| 85 | + | |
| 86 | +# ===== Figure 3 : importance par bloc (GBM v5) ===== | |
| 87 | +Xfull=pd.concat([df[S3].reset_index(drop=True),onehot],axis=1) | |
| 88 | +gb=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42).fit(Xfull,y) | |
| 89 | +idx=np.random.RandomState(0).choice(len(Xfull),4000,replace=False) | |
| 90 | +pi=permutation_importance(gb,Xfull.iloc[idx],y[idx],n_repeats=3,random_state=0,n_jobs=-1) | |
| 91 | +imp=pd.Series(pi.importances_mean,index=Xfull.columns).clip(lower=0) | |
| 92 | +def block(c): | |
| 93 | + if c.startswith("city_fe") or c.startswith("ptype_fe") or c.startswith("source"): return "Location/type FE" | |
| 94 | + if c in ("sqft_final","has_sqft","lot_sqft"): return "Size" | |
| 95 | + if c in ("beds","baths","parking","storeys","age","age2"): return "Structural" | |
| 96 | + if c in amen_: return "Amenities (rules)" | |
| 97 | + if c in rv or c in ("n_images_c","virtual_tour"): return "Rooms / listing" | |
| 98 | + if c in gv: return "Text (LLM)" | |
| 99 | + if c in vv: return "Vision (façade)" | |
| 100 | + return "Other" | |
| 101 | +bl_imp=imp.groupby(imp.index.map(block)).sum().sort_values() | |
| 102 | +bl_imp=bl_imp/bl_imp.sum()*100 | |
| 103 | +fig,ax=plt.subplots(figsize=(6.4,3.6)) | |
| 104 | +colors=["#8899AA"]*len(bl_imp) | |
| 105 | +for i,n in enumerate(bl_imp.index): | |
| 106 | + if n=="Vision (façade)": colors[i]=C_GBM | |
| 107 | + if n=="Text (LLM)": colors[i]=C_BL | |
| 108 | +ax.barh(bl_imp.index,bl_imp.values,color=colors,edgecolor="white") | |
| 109 | +for i,v in enumerate(bl_imp.values): ax.text(v+0.4,i,f"{v:.0f}\\%",va="center",fontsize=8) | |
| 110 | +ax.set_xlabel("Share of permutation importance (\\%)") | |
| 111 | +ax.set_title("Feature-block importance (gradient boosting, v5)",fontsize=11) | |
| 112 | +ax.set_xlim(0,max(bl_imp.values)*1.15) | |
| 113 | +fig.savefig("paper/figures/fig_importance.pdf");plt.close(fig) | |
| 114 | +print("-> fig_importance.pdf") | |
| 115 | +print("blocs:",{k:round(v,1) for k,v in bl_imp.sort_values(ascending=False).items()}) | |
| 116 | + | |
| 117 | +# ===== Table ablation détaillée ===== | |
| 118 | +def med(p): return np.median(np.abs(np.exp(p-y)-1))*100 | |
| 119 | +rowsL=[] | |
| 120 | +labels=["Structural + amenity","\\quad + Rooms + listing","\\quad + Text (LLM)","\\quad + Vision (façade)"] | |
| 121 | +for i,(nm,cols) in enumerate(sets): | |
| 122 | + ro,po=ols_r2(cols);rg,pg=gbm_r2(cols) | |
| 123 | + bcell = f"{blend[i]:.3f}" if i in blend else "---" | |
| 124 | + rowsL.append(f" {labels[i]} & {len(cols)} & {ro:.3f} & {rg:.3f} & {bcell} \\\\") | |
| 125 | +tab=r"""% Auto-generated v5 | |
| 126 | +\begin{table}[htbp] | |
| 127 | + \centering | |
| 128 | + \begin{threeparttable} | |
| 129 | + \caption{Feature Ablation: Out-of-Sample $R^2$ by Feature Layer} | |
| 130 | + \label{tab:ablation} | |
| 131 | + \begin{tabular}{@{}lcccc@{}} | |
| 132 | + \toprule | |
| 133 | + Feature set & \# feats & OLS & GBM & Blend \\ | |
| 134 | + \midrule | |
| 135 | +"""+"\n".join(rowsL)+r""" | |
| 136 | + \bottomrule | |
| 137 | + \end{tabular} | |
| 138 | + \begin{tablenotes}[flushleft] | |
| 139 | + \footnotesize | |
| 140 | + \item \textit{Notes:} Each row adds a feature layer to the row above (nested). | |
| 141 | + Entries are out-of-sample $R^2$ on $\ln(\text{Price})$ from 5-fold cross-validation | |
| 142 | + with a common seed. ``\# feats'' counts numeric/indicator features before the city, | |
| 143 | + type, and source fixed effects (added in every specification). The blend averages | |
| 144 | + the robust neural network and gradient boosting and is reported where both were | |
| 145 | + trained. GBM and the neural network handle missing living area natively; OLS uses | |
| 146 | + within-type median imputation. | |
| 147 | + \end{tablenotes} | |
| 148 | + \end{threeparttable} | |
| 149 | +\end{table} | |
| 150 | +""" | |
| 151 | +open("paper/tables/ablation.tex","w").write(tab) | |
| 152 | +print("-> paper/tables/ablation.tex") | |
added
scripts/15_make_tables.py
+209 −0
@@ -0,0 +1,209 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Génère les tableaux LaTeX (summary stats + régression) avec vrais chiffres.""" | |
| 4 | +import json, re, warnings | |
| 5 | +import numpy as np, pandas as pd | |
| 6 | +import statsmodels.formula.api as smf | |
| 7 | +warnings.filterwarnings("ignore") | |
| 8 | +CUR_YEAR = 2026 | |
| 9 | +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8")) | |
| 10 | + | |
| 11 | +def parse_area(s): | |
| 12 | + if not s or not isinstance(s, str): return np.nan | |
| 13 | + t=s.lower(); m=("m2" in t) or ("m²" in t) | |
| 14 | + n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","") | |
| 15 | + try: v=float(n) | |
| 16 | + except: return np.nan | |
| 17 | + return (v*10.7639 if m else v) if v>0 else np.nan | |
| 18 | +def clean_city(c): | |
| 19 | + if not c or not isinstance(c,str): return None | |
| 20 | + c=c.split("(")[0].strip(); return re.sub(r"\s+"," ",c).lower().strip() or None | |
| 21 | +def to_int(x): | |
| 22 | + try: return int(x) | |
| 23 | + except: return np.nan | |
| 24 | +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"], | |
| 25 | +"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"], | |
| 26 | +"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"], | |
| 27 | +"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"], | |
| 28 | +"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"], | |
| 29 | +"golf":["golf"],"ski":["ski"],"near_park":["park","parc"], | |
| 30 | +"near_school":["school","école","cegep","university","université","elementary","high school"], | |
| 31 | +"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"], | |
| 32 | +"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]} | |
| 33 | +def side(s): | |
| 34 | + s=s.lower() | |
| 35 | + if "ft" in s: | |
| 36 | + f=re.search(r"([\d.]+)\s*ft",s); i=re.search(r"([\d.]+)\s*in",s) | |
| 37 | + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0) | |
| 38 | + m=re.search(r"([\d.,]+)\s*m",s) | |
| 39 | + if m: | |
| 40 | + try: return float(m.group(1).replace(",","."))*3.28084 | |
| 41 | + except: return None | |
| 42 | + return None | |
| 43 | +def rarea(d): | |
| 44 | + if not d or "x" not in d: return None | |
| 45 | + a,_,b=d.partition("x"); la,lb=side(a),side(b) | |
| 46 | + return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None | |
| 47 | +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"], | |
| 48 | +"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"], | |
| 49 | +"r_library":["library"],"r_laundry":["laundry"]} | |
| 50 | +def prooms(rooms): | |
| 51 | + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0} | |
| 52 | + for k in RF: o[k]=0 | |
| 53 | + if not rooms: return o | |
| 54 | + ar=0.0; lv=set() | |
| 55 | + for r in rooms: | |
| 56 | + nm=(r.get("name") or "").lower(); l=(r.get("level") or "").lower(); o["n_rooms"]+=1 | |
| 57 | + if l: lv.add(l) | |
| 58 | + if "basement" in l or "sous-sol" in l: o["has_basement"]=1 | |
| 59 | + if "primary bedroom" in nm: o["has_primary_bed"]=1 | |
| 60 | + for k,kw in RF.items(): | |
| 61 | + if any(w in nm for w in kw): o[k]=1 | |
| 62 | + a=rarea(r.get("dimensions")) | |
| 63 | + if a: ar+=a | |
| 64 | + o["room_area"]=ar if ar>0 else np.nan; o["n_levels_r"]=len(lv); return o | |
| 65 | +rows=[] | |
| 66 | +for d in data: | |
| 67 | + p=d.get("property") or {}; ad=d.get("address") or {}; ft=d.get("features") or [] | |
| 68 | + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower() | |
| 69 | + yb=to_int(p.get("yearBuilt")); age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan | |
| 70 | + pt=p.get("buildingType") or p.get("type"); st=p.get("storeys") or p.get("levels") | |
| 71 | + try: st=float(str(st).replace(",",".")) | |
| 72 | + except: st=np.nan | |
| 73 | + rec={"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")), | |
| 74 | + "lot_sqft":parse_area(p.get("lotSize")),"beds":to_int(p.get("bedrooms")),"baths":to_int(p.get("bathrooms")), | |
| 75 | + "parking":to_int(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(), | |
| 76 | + "city":clean_city(ad.get("city"))} | |
| 77 | + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw)) | |
| 78 | + rec["n_images"]=len(d.get("images") or []); rec["virtual_tour"]=int(bool(d.get("virtualTourUrl"))) | |
| 79 | + rec.update(prooms(d.get("rooms"))); rows.append(rec) | |
| 80 | +df=pd.DataFrame(rows) | |
| 81 | +df=df[df["price"].notna()]; df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)] | |
| 82 | +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan | |
| 83 | +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan | |
| 84 | +df.loc[df["beds"]>15,"beds"]=np.nan; df.loc[df["baths"]>12,"baths"]=np.nan | |
| 85 | +df["ppsf"]=df["price"]/df["sqft"]; lo,hi=df["ppsf"].quantile([0.005,0.995]) | |
| 86 | +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))] | |
| 87 | +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)] | |
| 88 | +ratio=(ov["sqft"]/ov["room_area"]).median(); re_=df["room_area"]*ratio | |
| 89 | +ok=df["room_area"].notna()&(re_.between(300,20000)) | |
| 90 | +df["sqft_final"]=df["sqft"]; fill=df["sqft_final"].isna()&ok; df.loc[fill,"sqft_final"]=re_[fill] | |
| 91 | +df["has_sqft"]=df["sqft_final"].notna().astype(int) | |
| 92 | +df["sqft_imp"]=df.groupby("ptype")["sqft_final"].transform(lambda s:s.fillna(s.median())).fillna(df["sqft_final"].median()) | |
| 93 | +df["has_lot"]=df["lot_sqft"].notna().astype(int); df["lot_sqft"]=df["lot_sqft"].fillna(df["lot_sqft"].median()) | |
| 94 | +df["has_age"]=df["age"].notna().astype(int); df["age"]=df["age"].fillna(df["age"].median()).clip(lower=0) | |
| 95 | +for c in ["beds","baths","storeys"]: df[c]=df[c].fillna(df[c].median()) | |
| 96 | +df["parking"]=df["parking"].fillna(0) | |
| 97 | +df["ln_price"]=np.log(df["price"]); df["ln_sqft"]=np.log(df["sqft_imp"]) | |
| 98 | +df["ln_lot"]=np.log(df["lot_sqft"]); df["age2"]=df["age"]**2; df["n_images_c"]=df["n_images"].clip(upper=60) | |
| 99 | +vc=df["city"].value_counts(); keep=set(vc[vc>=20].index) | |
| 100 | +df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__") | |
| 101 | +tc=df["ptype"].value_counts(); df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other") | |
| 102 | + | |
| 103 | +amen=list(AMEN.keys()); rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"] | |
| 104 | +extra=["n_images_c","virtual_tour"]+rv | |
| 105 | +base="ln_lot + has_lot + beds + baths + parking + storeys + age + age2 + has_age + C(source) + C(ptype_fe) + "+" + ".join(amen)+" + "+" + ".join(extra)+" + C(city_fe)" | |
| 106 | +fA="ln_price ~ ln_sqft + "+base; fB="ln_price ~ "+base | |
| 107 | +dfA=df[df["has_sqft"]==1].copy() | |
| 108 | +mA=smf.ols(fA,data=dfA).fit(cov_type="HC3"); mB=smf.ols(fB,data=df).fit(cov_type="HC3") | |
| 109 | +sdA,sdB=np.std(mA.resid),np.std(mB.resid) | |
| 110 | + | |
| 111 | +# ---------- summary stats table ---------- | |
| 112 | +def q(s,p): return s.quantile(p) | |
| 113 | +def fmt(x,d=0): | |
| 114 | + if d==0: return f"{x:,.0f}".replace(",","{,}") | |
| 115 | + return f"{x:,.{d}f}".replace(",","{,}") | |
| 116 | +svars=[("Price (CAD)","price",0),("Living area (ft$^2$)","sqft_final",0), | |
| 117 | +("Lot area (ft$^2$)","lot_sqft",0),("Bedrooms","beds",1),("Bathrooms","baths",1), | |
| 118 | +("Parking spaces","parking",1),("Storeys","storeys",1),("Age (years)","age",1), | |
| 119 | +("Rooms (Realtor)","n_rooms",1),("Photos","n_images",0)] | |
| 120 | +lines=[] | |
| 121 | +for lab,c,d in svars: | |
| 122 | + s=df[c].dropna(); | |
| 123 | + lines.append(f"{lab} & {len(s):,} & {fmt(s.mean(),d)} & {fmt(s.std(),d)} & {fmt(q(s,.25),d)} & {fmt(q(s,.5),d)} & {fmt(q(s,.75),d)} \\\\".replace(",","{,}")) | |
| 124 | +share_r=(df.source=="realtor").mean()*100 | |
| 125 | +tbl_sum=r"""% Auto-generated | |
| 126 | +\begin{table}[htbp] | |
| 127 | + \centering | |
| 128 | + \begin{threeparttable} | |
| 129 | + \caption{Summary Statistics} | |
| 130 | + \label{tab:summary_stats} | |
| 131 | + \begin{tabular}{@{}lcccccc@{}} | |
| 132 | + \toprule | |
| 133 | + Variable & $N$ & Mean & Std.\ Dev. & $p_{25}$ & Median & $p_{75}$ \\ | |
| 134 | + \midrule | |
| 135 | +"""+"\n".join(" "+l for l in lines)+r""" | |
| 136 | + \bottomrule | |
| 137 | + \end{tabular} | |
| 138 | + \begin{tablenotes}[flushleft] | |
| 139 | + \footnotesize | |
| 140 | + \item \textit{Notes:} Descriptive statistics for the cleaned sample of """+f"{len(df):,}".replace(",","{,}")+r""" single-family and townhouse listings (Realtor.ca and DuProprio), scraped on 3~July~2026. """+f"{share_r:.0f}"+r"""\% of listings originate from Realtor.ca. Living area is the reported area, or reconstructed from room dimensions where missing. Continuous variables are bounded by the outlier rules described in Section~\ref{sec:data}; price-per-square-foot is trimmed at the 0.5th and 99.5th percentiles. | |
| 141 | + \end{tablenotes} | |
| 142 | + \end{threeparttable} | |
| 143 | +\end{table} | |
| 144 | +""" | |
| 145 | +open("paper/tables/summary_stats.tex","w").write(tbl_sum) | |
| 146 | + | |
| 147 | +# ---------- regression table ---------- | |
| 148 | +def star(p): return "^{***}" if p<.01 else "^{**}" if p<.05 else "^{*}" if p<.1 else "" | |
| 149 | +disp=[("ln_sqft",r"$\ln(\text{Living area})$"),("ln_lot",r"$\ln(\text{Lot area})$"), | |
| 150 | +("beds","Bedrooms"),("baths","Bathrooms"),("parking","Parking"),("age","Age"), | |
| 151 | +("central_air","Central A/C"),("transit","Public transit"),("waterfront","Waterfront"), | |
| 152 | +("pool","Pool"),("garage","Garage"),("fireplace","Fireplace"),("near_school","Near school"), | |
| 153 | +("has_primary_bed","Primary bedroom"),("r_winecellar","Wine cellar"), | |
| 154 | +("n_images_c","Photos (count)"),("virtual_tour","Virtual tour"), | |
| 155 | +("C(source)[T.realtor]","Realtor.ca (vs DuProprio)")] | |
| 156 | +def cell(m,key): | |
| 157 | + if key not in m.params.index: return "","" | |
| 158 | + b=m.params[key]; se=m.bse[key]; p=m.pvalues[key] | |
| 159 | + sign="$-$" if b<0 else "" | |
| 160 | + return f"{sign}{abs(b):.3f}${star(p)}$", f"({se:.3f})" | |
| 161 | +rowsL=[] | |
| 162 | +for key,lab in disp: | |
| 163 | + a1,a2=cell(mA,key); b1,b2=cell(mB,key) | |
| 164 | + rowsL.append(f" {lab} & {a1} & {b1} \\\\") | |
| 165 | + rowsL.append(f" & {a2} & {b2} \\\\[3pt]") | |
| 166 | +tbl_reg=r"""% Auto-generated | |
| 167 | +\begin{table}[htbp] | |
| 168 | + \centering | |
| 169 | + \begin{threeparttable} | |
| 170 | + \caption{Hedonic Regression Results} | |
| 171 | + \label{tab:regression} | |
| 172 | + \begin{tabular}{@{}lcc@{}} | |
| 173 | + \toprule | |
| 174 | + & (1) Model~A & (2) Model~B \\ | |
| 175 | + & \textit{rich} & \textit{broad} \\ | |
| 176 | + \midrule | |
| 177 | +"""+"\n".join(rowsL)+r""" | |
| 178 | + \midrule | |
| 179 | + City FE & Yes & Yes \\ | |
| 180 | + Property-type FE & Yes & Yes \\ | |
| 181 | + Living area & Yes & No \\ | |
| 182 | + Observations & """+f"{int(mA.nobs):,}".replace(",","{,}")+r""" & """+f"{int(mB.nobs):,}".replace(",","{,}")+r""" \\ | |
| 183 | + $R^2$ & """+f"{mA.rsquared:.3f}"+r""" & """+f"{mB.rsquared:.3f}"+r""" \\ | |
| 184 | + Adjusted $R^2$ & """+f"{mA.rsquared_adj:.3f}"+r""" & """+f"{mB.rsquared_adj:.3f}"+r""" \\ | |
| 185 | + \bottomrule | |
| 186 | + \end{tabular} | |
| 187 | + \begin{tablenotes}[flushleft] | |
| 188 | + \footnotesize | |
| 189 | + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic}. The dependent variable is $\ln(\text{Price})$. Model~A includes $\ln(\text{Living area})$ and is estimated on listings with a measured (reported or room-reconstructed) area; Model~B omits living area and is estimated on the full cleaned sample. All specifications include city and property-type fixed effects and the full set of amenity, room-detail, and listing-signal controls (Section~\ref{sec:methodology}); only selected coefficients are reported. Heteroskedasticity-robust (HC3) standard errors in parentheses. $^{***}$, $^{**}$, $^{*}$ denote significance at the 1\%, 5\%, and 10\% levels. | |
| 190 | + \end{tablenotes} | |
| 191 | + \end{threeparttable} | |
| 192 | +\end{table} | |
| 193 | +""" | |
| 194 | +open("paper/tables/regression.tex","w").write(tbl_reg) | |
| 195 | + | |
| 196 | +# key numbers for the text | |
| 197 | +predA=np.exp(mA.predict(df))*np.exp(sdA**2/2); predB=np.exp(mB.predict(df))*np.exp(sdB**2/2) | |
| 198 | +df["pred"]=np.where(df["has_sqft"]==1,predA,predB); df["rp"]=(df["price"]/df["pred"]-1)*100 | |
| 199 | +print("N_clean",len(df),"N_A",int(mA.nobs),"N_B",int(mB.nobs)) | |
| 200 | +print("R2_A %.4f adj %.4f errpct %.1f"%(mA.rsquared,mA.rsquared_adj,(np.exp(sdA)-1)*100)) | |
| 201 | +print("R2_B %.4f adj %.4f errpct %.1f"%(mB.rsquared,mB.rsquared_adj,(np.exp(sdB)-1)*100)) | |
| 202 | +print("median|err| %.1f within15 %.0f within25 %.0f"%(df.rp.abs().median(),(df.rp.abs()<=15).mean()*100,(df.rp.abs()<=25).mean()*100)) | |
| 203 | +print("nb_cities_fe",df.city_fe.nunique(),"median_price",int(df.price.median())) | |
| 204 | +cf={k.split("T.")[1].rstrip("]"):v for k,v in mA.params.items() if k.startswith("C(city_fe)")} | |
| 205 | +top=pd.Series(cf).sort_values(ascending=False) | |
| 206 | +print("TOP:",[(c,round((np.exp(b)-1)*100)) for c,b in top.head(5).items()]) | |
| 207 | +print("BOT:",[(c,round((np.exp(b)-1)*100)) for c,b in top.tail(3).items()]) | |
| 208 | +print("share_realtor %.1f"%((df.source=='realtor').mean()*100)) | |
| 209 | +print("Tables written to paper/tables/") | |
added
scripts/16_make_tables_v5.py
+124 −0
@@ -0,0 +1,124 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Régénère les tableaux LaTeX du papier en v5 (texte GPT + vision).""" | |
| 4 | +import numpy as np, pandas as pd, statsmodels.formula.api as smf, warnings | |
| 5 | +warnings.filterwarnings("ignore") | |
| 6 | +# construire df via le pipeline de v5_ml (jusqu'avant la CV) | |
| 7 | +exec(open("v5_ml.py").read().split("kf=KFold")[0]) | |
| 8 | +amen_=list(AMEN.keys()) | |
| 9 | +df["ln_price"]=np.log(df["price"]) | |
| 10 | +df["ln_sqft"]=np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median())) | |
| 11 | +df["ln_lot"]=np.log(df["lot_sqft"].fillna(df["lot_sqft"].median()));df["has_lot"]=df["lot_sqft"].notna().astype(int) | |
| 12 | +df["has_age"]=df["age"].notna().astype(int) | |
| 13 | +for c in ["beds","baths","storeys","age","age2"]: | |
| 14 | + df[c]=df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median()) | |
| 15 | +df["parking"]=df["parking"].fillna(0) | |
| 16 | +form=("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+" | |
| 17 | + +"+".join(amen_)+"+n_images_c+virtual_tour+"+"+".join(rv)+"+"+"+".join(gv)+"+"+"+".join(vv)+"+C(city_fe)") | |
| 18 | +mA=smf.ols(form,data=df[df["has_sqft"]==1]).fit(cov_type="HC3") | |
| 19 | +mB=smf.ols(form.replace("ln_sqft+",""),data=df).fit(cov_type="HC3") | |
| 20 | +print("A R2=%.4f adj=%.4f N=%d | B R2=%.4f adj=%.4f N=%d"%(mA.rsquared,mA.rsquared_adj,mA.nobs,mB.rsquared,mB.rsquared_adj,mB.nobs)) | |
| 21 | + | |
| 22 | +def star(p): return "^{***}" if p<.01 else "^{**}" if p<.05 else "^{*}" if p<.1 else "" | |
| 23 | +def cell(m,k): | |
| 24 | + if k not in m.params.index: return "","" | |
| 25 | + b=m.params[k];se=m.bse[k];p=m.pvalues[k];s="$-$" if b<0 else "" | |
| 26 | + return f"{s}{abs(b):.3f}${star(p)}$",f"({se:.3f})" | |
| 27 | +disp=[("ln_sqft",r"$\ln(\text{Living area})$"),("ln_lot",r"$\ln(\text{Lot area})$"), | |
| 28 | +("beds","Bedrooms"),("baths","Bathrooms"),("parking","Parking"),("age","Age"), | |
| 29 | +("__A__","\\textit{Amenities \\& text (LLM)}"), | |
| 30 | +("central_air","Central A/C"),("transit","Public transit"),("waterfront","Waterfront"), | |
| 31 | +("g_cond","Condition (LLM, ord.)"),("g_renovated_kitchen","Reno.\\ kitchen (LLM)"), | |
| 32 | +("g_hardwood","Hardwood floor (LLM)"),("g_new_construction","New construction (LLM)"), | |
| 33 | +("g_income_property","Income property (LLM)"), | |
| 34 | +("__V__","\\textit{Vision (façade)}"), | |
| 35 | +("v_curb","Curb appeal (1--10)"),("v_reno","Renovation level (ord.)"), | |
| 36 | +("v_luxury","Luxury (visual)"),("v_garage_vis","Garage visible"), | |
| 37 | +("v_mat_brique","Brick façade"),("v_mat_pierre","Stone façade"), | |
| 38 | +("v_mat_vinyle","Vinyl façade"),("v_set_rural","Rural setting")] | |
| 39 | +L=[] | |
| 40 | +for k,lab in disp: | |
| 41 | + if k in ("__A__","__V__"): | |
| 42 | + L.append(f" \\multicolumn{{3}}{{@{{}}l}}{{{lab}}} \\\\[2pt]");continue | |
| 43 | + a1,a2=cell(mA,k);b1,b2=cell(mB,k) | |
| 44 | + L.append(f" {lab} & {a1} & {b1} \\\\") | |
| 45 | + L.append(f" & {a2} & {b2} \\\\[3pt]") | |
| 46 | +reg=r"""% Auto-generated v5 | |
| 47 | +\begin{table}[htbp] | |
| 48 | + \centering | |
| 49 | + \begin{threeparttable} | |
| 50 | + \caption{Hedonic Regression Results (Text + Vision Features)} | |
| 51 | + \label{tab:regression} | |
| 52 | + \begin{tabular}{@{}lcc@{}} | |
| 53 | + \toprule | |
| 54 | + & (1) Model~A & (2) Model~B \\ | |
| 55 | + & \textit{rich} & \textit{broad} \\ | |
| 56 | + \midrule | |
| 57 | +"""+"\n".join(L)+r""" | |
| 58 | + \midrule | |
| 59 | + City FE / Type FE & Yes & Yes \\ | |
| 60 | + Living area & Yes & No \\ | |
| 61 | + Observations & """+f"{int(mA.nobs):,}".replace(",","{,}")+r""" & """+f"{int(mB.nobs):,}".replace(",","{,}")+r""" \\ | |
| 62 | + $R^2$ & """+f"{mA.rsquared:.3f}"+r""" & """+f"{mB.rsquared:.3f}"+r""" \\ | |
| 63 | + Adjusted $R^2$ & """+f"{mA.rsquared_adj:.3f}"+r""" & """+f"{mB.rsquared_adj:.3f}"+r""" \\ | |
| 64 | + \bottomrule | |
| 65 | + \end{tabular} | |
| 66 | + \begin{tablenotes}[flushleft] | |
| 67 | + \footnotesize | |
| 68 | + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full | |
| 69 | + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample); | |
| 70 | + Model~B omits it (full sample). ``LLM'' variables are extracted from the | |
| 71 | + description text; ``Vision'' variables from the façade photograph. Curb appeal and | |
| 72 | + renovation level enter as scores; luxury, garage, and material/setting are | |
| 73 | + indicators. All specifications include city and property-type fixed effects and the | |
| 74 | + full amenity, room-detail, and listing-signal controls (only selected coefficients | |
| 75 | + shown). HC3 standard errors in parentheses. $^{***}$, $^{**}$, $^{*}$: significance | |
| 76 | + at 1\%, 5\%, 10\%. | |
| 77 | + \end{tablenotes} | |
| 78 | + \end{threeparttable} | |
| 79 | +\end{table} | |
| 80 | +""" | |
| 81 | +open("paper/tables/regression.tex","w").write(reg) | |
| 82 | +print("-> paper/tables/regression.tex (v5)") | |
| 83 | + | |
| 84 | +# ---- models.tex : deux panels (texte / +vision) avec chiffres CV ---- | |
| 85 | +models=r"""% Auto-generated v5 | |
| 86 | +\begin{table}[htbp] | |
| 87 | + \centering | |
| 88 | + \begin{threeparttable} | |
| 89 | + \caption{Predictive Performance: From OLS to Deep Learning} | |
| 90 | + \label{tab:models} | |
| 91 | + \begin{tabular}{@{}lcccc@{}} | |
| 92 | + \toprule | |
| 93 | + Estimator & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% & Within $\pm$25\% \\ | |
| 94 | + \midrule | |
| 95 | + \multicolumn{5}{@{}l}{\textit{Panel A: Structural + amenity + LLM-text features}} \\[3pt] | |
| 96 | + OLS (linear hedonic) & 0.628 & 17.9\% & 43\% & 64\% \\ | |
| 97 | + Random Forest & 0.678 & 18.0\% & 43\% & 64\% \\ | |
| 98 | + Neural network (plain MLP) & 0.711 & 16.1\% & 47\% & 69\% \\ | |
| 99 | + Neural network (robust ResNet) & 0.741 & 15.2\% & 49\% & 70\% \\ | |
| 100 | + Gradient boosting (HistGBM) & 0.743 & 15.4\% & 49\% & 70\% \\ | |
| 101 | + Blend (NN + GBM) & 0.760 & 14.7\% & 51\% & 72\% \\[4pt] | |
| 102 | + \multicolumn{5}{@{}l}{\textit{Panel B: \ + Vision (façade) features}} \\[3pt] | |
| 103 | + Gradient boosting (HistGBM) & 0.770 & 14.5\% & 51\% & 71\% \\ | |
| 104 | + Neural network (robust ResNet) & 0.747 & 14.8\% & 51\% & 71\% \\ | |
| 105 | + \textbf{Blend (NN + GBM)} & \textbf{0.779} & \textbf{14.0\%} & \textbf{53\%} & \textbf{73\%} \\ | |
| 106 | + \bottomrule | |
| 107 | + \end{tabular} | |
| 108 | + \begin{tablenotes}[flushleft] | |
| 109 | + \footnotesize | |
| 110 | + \item \textit{Notes:} Out-of-sample performance from 5-fold cross-validation with a | |
| 111 | + common seed (identical folds across rows). $R^2_{\text{OOS}}$ is on | |
| 112 | + $\ln(\text{Price})$; ``Median AE'' is the median absolute percentage error in | |
| 113 | + levels (Equation~\ref{eq:cvmetrics}). Panel~A uses structural, amenity, room-detail, | |
| 114 | + listing-signal, and LLM text-derived features; Panel~B adds the twelve façade | |
| 115 | + vision attributes. City, type, and source enter as one-hot indicators for the | |
| 116 | + linear and tree models and as learned embeddings for the neural networks. The tree | |
| 117 | + and neural models handle missing living area natively. The best model is the | |
| 118 | + text\,+\,vision blend. | |
| 119 | + \end{tablenotes} | |
| 120 | + \end{threeparttable} | |
| 121 | +\end{table} | |
| 122 | +""" | |
| 123 | +open("paper/tables/models.tex","w").write(models) | |
| 124 | +print("-> paper/tables/models.tex (v5, 2 panels)") | |
added
scripts/17_gen_regression_table.py
+85 −0
@@ -0,0 +1,85 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# -*- coding: utf-8 -*- | |
| 3 | +"""Régénère paper/tables/regression.tex à partir de revision_extras.json | |
| 4 | + avec SE clusterisées par municipalité (et stars associées).""" | |
| 5 | +import json | |
| 6 | + | |
| 7 | +R = json.load(open("results/revision_extras.json")) | |
| 8 | + | |
| 9 | +def star(p): | |
| 10 | + return "^{***}" if p < .01 else "^{**}" if p < .05 else "^{*}" if p < .1 else "" | |
| 11 | + | |
| 12 | +def cell(model, k): | |
| 13 | + c = R[model]["coef"].get(k) | |
| 14 | + if not c: | |
| 15 | + return "", "" | |
| 16 | + b, se, p = c["b"], c["se_cl"], c["p_cl"] | |
| 17 | + s = "$-$" if b < 0 else "" | |
| 18 | + st = star(p) | |
| 19 | + coef = f"{s}{abs(b):.3f}" + (f"${st}$" if st else "") | |
| 20 | + return coef, f"({se:.3f})" | |
| 21 | + | |
| 22 | +disp = [("ln_sqft", r"$\ln(\text{Living area})$"), ("ln_lot", r"$\ln(\text{Lot area})$"), | |
| 23 | + ("beds", "Bedrooms"), ("baths", "Bathrooms"), ("parking", "Parking"), ("age", "Age"), | |
| 24 | + ("__A__", r"\textit{Amenities \& text (LLM)}"), | |
| 25 | + ("central_air", "Central A/C"), ("transit", "Public transit"), ("waterfront", "Waterfront"), | |
| 26 | + ("g_cond", "Condition (LLM, ord.)"), ("g_renovated_kitchen", r"Reno.\ kitchen (LLM)"), | |
| 27 | + ("g_hardwood", "Hardwood floor (LLM)"), ("g_new_construction", "New construction (LLM)"), | |
| 28 | + ("g_income_property", "Income property (LLM)"), | |
| 29 | + ("__V__", r"\textit{Vision (façade)}"), | |
| 30 | + ("v_curb", "Curb appeal (1--10)"), ("v_reno", "Renovation level (ord.)"), | |
| 31 | + ("v_luxury", "Luxury (visual)"), ("v_garage_vis", "Garage visible"), | |
| 32 | + ("v_mat_brique", "Brick façade"), ("v_mat_pierre", "Stone façade"), | |
| 33 | + ("v_mat_vinyle", "Vinyl façade"), ("v_set_rural", "Rural setting")] | |
| 34 | + | |
| 35 | +L = [] | |
| 36 | +for k, lab in disp: | |
| 37 | + if k in ("__A__", "__V__"): | |
| 38 | + L.append(f" \\multicolumn{{3}}{{@{{}}l}}{{{lab}}} \\\\[2pt]") | |
| 39 | + continue | |
| 40 | + a1, a2 = cell("modelA", k) | |
| 41 | + b1, b2 = cell("modelB", k) | |
| 42 | + L.append(f" {lab} & {a1} & {b1} \\\\") | |
| 43 | + L.append(f" & {a2} & {b2} \\\\[3pt]") | |
| 44 | + | |
| 45 | +nA, nB = R["modelA"]["n"], R["modelB"]["n"] | |
| 46 | +tex = r"""% Auto-generated (revision: clustered SEs) | |
| 47 | +\begin{table}[htbp] | |
| 48 | + \centering | |
| 49 | + \begin{threeparttable} | |
| 50 | + \caption{Hedonic Regression Results (Text + Vision Features)} | |
| 51 | + \label{tab:regression} | |
| 52 | + \begin{tabular}{@{}lcc@{}} | |
| 53 | + \toprule | |
| 54 | + & (1) Model~A & (2) Model~B \\ | |
| 55 | + & \textit{rich} & \textit{broad} \\ | |
| 56 | + \midrule | |
| 57 | +""" + "\n".join(L) + f""" | |
| 58 | + \\midrule | |
| 59 | + City FE / Type FE & Yes & Yes \\\\ | |
| 60 | + Living area & Yes & No \\\\ | |
| 61 | + Observations & {nA:,} & {nB:,} \\\\ | |
| 62 | + $R^2$ & {R['modelA']['r2']:.3f} & {R['modelB']['r2']:.3f} \\\\ | |
| 63 | + Adjusted $R^2$ & {R['modelA']['adj']:.3f} & {R['modelB']['adj']:.3f} \\\\ | |
| 64 | + \\bottomrule | |
| 65 | + \\end{{tabular}} | |
| 66 | +""".replace(f"{nA:,}", f"{nA:,}".replace(",", "{,}")).replace(f"{nB:,}", f"{nB:,}".replace(",", "{,}")) + r""" \begin{tablenotes}[flushleft] | |
| 67 | + \footnotesize | |
| 68 | + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full | |
| 69 | + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample); | |
| 70 | + Model~B omits it (full sample). ``LLM'' variables are extracted from the | |
| 71 | + description text; ``Vision'' variables from the façade photograph. Curb appeal and | |
| 72 | + renovation level enter as scores; luxury, garage, and material/setting are | |
| 73 | + indicators. All specifications include city and property-type fixed effects and the | |
| 74 | + full amenity, room-detail, and listing-signal controls (only selected coefficients | |
| 75 | + shown). Standard errors clustered by municipality (195 clusters) in parentheses. | |
| 76 | + Heteroskedasticity-robust (HC3) standard errors are uniformly smaller and yield | |
| 77 | + the same significance pattern for the text- and vision-derived attributes, except | |
| 78 | + that curb appeal in Model~B is significant under HC3 but not under clustering. | |
| 79 | + $^{***}$, $^{**}$, $^{*}$: significance at 1\%, 5\%, 10\%. | |
| 80 | + \end{tablenotes} | |
| 81 | + \end{threeparttable} | |
| 82 | +\end{table} | |
| 83 | +""" | |
| 84 | +open("paper/tables/regression.tex", "w").write(tex) | |
| 85 | +print("-> paper/tables/regression.tex (clustered SEs)") | |
| 86 | ||