SPB Git forge

spb/wp14_uqo

Public
2commits 1branches 1releases
909.0 KBsize
maindefault branch
21 min agolast push
Python 57% TeX 42.9%

harmonisation standard UQO WP (2026-10-17)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 1 h ago (Oct 17, 2026)

61 changed files +7,212 −0

added .env.example +2 −0
@@ -0,0 +1,2 @@
1 +OPENAI_API_KEY=
2 +ANTHROPIC_API_KEY=
added .gitignore +42 −0
@@ -0,0 +1,42 @@
1 +# Raw & processed data — never committed (see data/raw/README.md for provenance)
2 +data/raw/*
3 +!data/raw/README.md
4 +data/processed/*
5 +!data/processed/.gitkeep
6 +
7 +# Python
8 +__pycache__/
9 +*.pyc
10 +.pytest_cache/
11 +.venv/
12 +*.egg-info/
13 +
14 +# LaTeX build artifacts (paper/main.pdf IS committed)
15 +paper/*.aux
16 +paper/*.log
17 +paper/*.out
18 +paper/*.toc
19 +paper/*.lof
20 +paper/*.lot
21 +paper/*.fls
22 +paper/*.fdb_latexmk
23 +paper/*.bbl
24 +paper/*.blg
25 +paper/*.spl
26 +paper/*.synctex.gz
27 +paper/*.run.xml
28 +paper/*.bcf
29 +
30 +# Logs
31 +logs/
32 +*.log
33 +
34 +# macOS
35 +.DS_Store
36 +
37 +# Project-specific
38 +# OPENAI_API_KEY / ANTHROPIC_API_KEY — never commit
39 +.env
40 +# LLM/vision caches (specs.jsonl, facade_specs.jsonl), merged datasets, OOS predictions
41 +data/processed/*
42 +!data/processed/README.md
added README.md +164 −0
@@ -0,0 +1,164 @@
1 +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
2 +# 🏠 Hedonic House Pricing with Language-Model and Vision Features
3 +
4 +**UQO Working Paper No. 14** — *Evidence from Quebec Real-Estate Listings*
5 +
6 +[![Paper](https://img.shields.io/badge/paper-PDF%20(33%20p.)-b31b1b?logo=latex&logoColor=white)](paper/main.pdf)
7 +[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)](requirements.txt)
8 +[![Data](https://img.shields.io/badge/data-19%2C123%20house%20listings%20·%20Realtor.ca%20%2B%20DuProprio-4051b5)](data/raw/README.md)
9 +[![Reproducible](https://img.shields.io/badge/reproducible-end--to--end%20pipeline-2e7d32)](scripts/)
10 +[![Institution](https://img.shields.io/badge/UQO-D%C3%A9pt.%20des%20sciences%20administratives-16365c)](https://uqo.ca)
11 +[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-16365c)](mailto:simon-pierre.boucher@uqo.ca)
12 +
13 +> **TL;DR** — Much of a home's value is recorded only in the free text and the photographs of its listing. We use a
14 +> large language model to turn each description into structured specifications and a vision–language model to turn the
15 +> façade photo into objective visual attributes, both under strict JSON schemas. On 19,123 Quebec house listings the two
16 +> AI layers lift the R² of an already rich log-linear hedonic model from 0.717 to 0.755; the vision layer is the largest
17 +> out-of-sample gain of any feature block for every estimator (OLS, gradient boosting, neural nets); the best ensemble
18 +> reaches out-of-sample R² 0.779 with a 14.0% median absolute error. Implicit prices are economically sensible (+2.3% per
19 +> point of curb appeal, +3–4% for brick or stone) and survive municipality-clustered inference.
20 +
21 +---
22 +
23 +## 📌 Status
24 +
25 +| Item | Value |
26 +|---|---|
27 +| Version | 2.0 (July 2026) — revised draft after an internal review round (`scripts/13_revision_extras.py`) |
28 +| SSRN | not yet submitted |
29 +| Journal | Elsevier-format highlights prepared (`docs/highlights_elsevier.tex`); target to be decided |
30 +| Code & data | reproducible from `scripts/` run from the repository root; raw crawls and LLM caches not committed |
31 +
32 +## 🎯 What this paper does
33 +
34 +Hedonic models are bounded by the characteristics an analyst can measure. We treat general-purpose AI models as a
35 +**feature-engineering technology**: each listing description is parsed by an LLM into a fixed schema of property
36 +specifications (renovations, finishes, systems, extras), and each façade photograph is parsed by a vision–language model
37 +into objective attributes (cladding, storeys, garage, curb appeal). The outputs are ordinary econometric variables and
38 +enter a log-linear hedonic regression with city fixed effects:
39 +
40 +```
41 +ln P_i = α_city(i) + X_i'β + S_i'γ (LLM text layer) + V_i'δ (vision layer) + ε_i
42 +```
43 +
44 +We then measure, under identical 5-fold splits, how much each layer adds for OLS, histogram gradient boosting, a
45 +residual MLP with city/type embeddings, and a stacked blend.
46 +
47 +## 🏆 Headline results
48 +
49 +| Object | Estimate |
50 +|---|---|
51 +| Base hedonic OLS (structural + location), in-sample R² | 0.717 |
52 +| + LLM text layer + vision layer | 0.755 |
53 +| Out-of-sample R², OLS: base → +text → +text+vision (5-fold) | 0.52 → 0.62 → … (see `paper/tables/ablation.tex`) |
54 +| Out-of-sample R², GBM: base → +text | 0.69 → 0.73 |
55 +| Best ensemble (robust NN + GBM blend) | R²_OOS = 0.779, median absolute error 14.0% |
56 +| Curb-appeal score (1–5) | +2.3% per point |
57 +| Brick / stone façade | +3–4% |
58 +| Inference | HC3 and municipality-clustered standard errors agree |
59 +
60 +## 🗂 Repository layout
61 +
62 +```
63 +hedonic-llm-vision-features/
64 +├── data/
65 +│ ├── raw/ # two Apify crawls (Realtor.ca, DuProprio), 2026-07-03 — not committed
66 +│ └── processed/ # LLM/vision caches, merged datasets, OOS predictions — not committed
67 +├── scripts/ # run from the repository root, in order
68 +│ ├── 00a_test_openai_extraction.py / 00b_test_vision_claude.py # API smoke tests
69 +│ ├── 01_extract_specs_llm.py # description → JSON specs (parallel, resumable)
70 +│ ├── 02_facade_vision.py # façade photo → visual attributes
71 +│ ├── 03_merge_llm_features.py
72 +│ ├── 04_hedonic_ols_base.py / 05_hedonic_ols_v4_text.py
73 +│ ├── 06_hedonic_ml_cv.py / 07_torch_nn.py / 08_torch_nn_robust.py
74 +│ ├── 09_v5_ml_text_vision.py / 10_final_blend.py
75 +│ ├── 11_extras_v5_robustness.py / 12_ablation_extras.py / 13_revision_extras.py
76 +│ └── 14_figures_v5.py / 15_make_tables.py / 16_make_tables_v5.py / 17_gen_regression_table.py
77 +├── results/ # hedonic summaries (txt), ablation_extras.json, revision_extras.json
78 +├── docs/ # METHODOLOGIE.md + methodologie.tex/pdf (French data & method note), Elsevier highlights
79 +└── paper/ # main.tex + sections/ + tables/ + figures/ + references.bib → main.pdf
80 +```
81 +
82 +## 🚀 Quick start
83 +
84 +```bash
85 +python3 -m pip install -r requirements.txt
86 +cp .env.example .env # OPENAI_API_KEY=…, ANTHROPIC_API_KEY=… (only steps 00–02 call the APIs)
87 +# place the two Apify crawls in data/raw/ and build data/processed/maisons_avec_prix.json (docs/METHODOLOGIE.md §2–4), then:
88 +WORKERS=24 python3 scripts/01_extract_specs_llm.py
89 +python3 scripts/02_facade_vision.py
90 +python3 scripts/03_merge_llm_features.py
91 +python3 scripts/05_hedonic_ols_v4_text.py
92 +python3 scripts/08_torch_nn_robust.py && python3 scripts/09_v5_ml_text_vision.py && python3 scripts/10_final_blend.py
93 +python3 scripts/11_extras_v5_robustness.py && python3 scripts/12_ablation_extras.py && python3 scripts/13_revision_extras.py
94 +python3 scripts/14_figures_v5.py && python3 scripts/16_make_tables_v5.py && python3 scripts/17_gen_regression_table.py
95 +cd paper && make # latexmk → main.pdf
96 +```
97 +
98 +LLM extraction ≈ 600 tokens in / 110 out per listing (24 threads); vision ≈ 2,470 tokens in per image (16 threads);
99 +both are resumable. Econometrics and ML ≈ 20 minutes on an Apple-silicon laptop (PyTorch on MPS).
100 +
101 +## 🗃 Data
102 +
103 +27,302 Realtor.ca and 7,693 DuProprio listings crawled on 2026-07-03; houses (single-family and townhouse) with a valid
104 +price retained, outliers removed by the rules in `docs/METHODOLOGIE.md` §3, giving 19,123 observations. Structural
105 +variables (living area, lot, rooms, year built, type, city), text-derived specifications and façade attributes. See
106 +`data/raw/README.md` and `data/processed/README.md`.
107 +
108 +## 🔬 Methodology
109 +
110 +1. **AI feature layers** with strict JSON schemas and retry/backoff — `scripts/01_…`, `scripts/02_…`.
111 +2. **Log-linear hedonic OLS** with city fixed effects, HC3 and municipality-clustered SEs — `scripts/05_…`, `scripts/13_…`.
112 +3. **Ablation by feature block × estimator** under identical folds (OLS, GBM, NN) — `scripts/09_…`, `scripts/12_…`.
113 +4. **Stacked blend** of the robust NN and HistGradientBoosting — `scripts/10_final_blend.py`.
114 +5. **Robustness** by segment, extreme city effects, calibration and importance plots — `scripts/11_…`, `scripts/14_…`.
115 +
116 +## 🖼 Figures & tables inventory
117 +
118 +| Figure | Content |
119 +|---|---|
120 +| `fig_ablation` | out-of-sample gain by feature block and estimator |
121 +| `fig_calibration` | predicted vs actual, blend |
122 +| `fig_importance` | feature importance |
123 +| `fig_premia` | location premia |
124 +| `fig_validation` | validation of AI-extracted attributes |
125 +| `nn_diagram` (TikZ) | architecture of the robust NN |
126 +
127 +Tables (`paper/tables/*.tex`): summary stats, regression (A/B panels), models, ablation, robustness, AI summary.
128 +
129 +## ⚠️ Limitations
130 +
131 +- Asking prices from listings, single cross-section (July 2026).
132 +- Realtor.ca listings lack coordinates; location enters through city fixed effects and postal codes.
133 +- LLM/vision outputs are model-dependent; schemas and prompts are versioned in the scripts.
134 +- API costs make the extraction layers the expensive part of replication.
135 +
136 +## 📝 Citation
137 +
138 +```bibtex
139 +@techreport{boucher2026llmvisionhedonic,
140 + author = {Boucher, Simon-Pierre},
141 + title = {Hedonic House Pricing with Language-Model and Vision Features: Evidence from Quebec Real-Estate Listings},
142 + institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives},
143 + type = {Working Paper},
144 + number = {14},
145 + year = {2026},
146 + month = {July}
147 +}
148 +```
149 +
150 +## 👤 Author & contact
151 +
152 +**Simon-Pierre Boucher** — Département des sciences administratives,
153 +Université du Québec en Outaouais (UQO), Gatineau, QC.
154 +📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca)
155 +
156 +## 🗒 Notes
157 +
158 +- Numbering: this paper was first issued as "Working Paper No. 11" (July 2026); number 11 belongs to *Half a Million
159 + Prices, Twenty Models*. Renumbered **14** on 2026-10-17; cover date fixed to July 2026 (was `\today`).
160 +- Harmonised 2026-10-17: flat scripts moved to `scripts/` (numbered), data to `data/`, logs removed, summaries to
161 + `results/`, methodology note to `docs/`; file paths inside the scripts now point to `data/…` and `results/…`
162 + (run them from the repository root). Bibliography switched from `plainnat` to `aer`.
163 +- The script `13_revision_extras.py` was written for a Journal of Housing Economics revision round (clustered SEs,
164 + in-sample ablation, extreme city effects).
added data/processed/README.md +10 −0
@@ -0,0 +1,10 @@
1 +# Processed data (not committed)
2 +
3 +| File | Produced by | Content |
4 +|---|---|---|
5 +| `maisons_avec_prix.json` | house filter + cleaning described in `docs/METHODOLOGIE.md` §2–4 (combined Realtor + DuProprio houses with price) | 19,123 house listings |
6 +| `specs.jsonl` | `scripts/01_extract_specs_llm.py` (GPT-4o-mini, strict JSON schema, resumable cache keyed by uid) | structured specifications extracted from descriptions |
7 +| `facade_specs.jsonl` | `scripts/02_facade_vision.py` (vision–language model on the façade photo) | objective visual attributes (curb appeal, cladding, …) |
8 +| `maisons_specs.json` / `.csv` | `scripts/03_merge_llm_features.py` | listings merged with LLM features |
9 +| `maisons_hedonic.csv`, `maisons_hedonic_v4.csv` | `scripts/04_…`, `scripts/05_…` | final datasets with fitted price and `resid_pct` |
10 +| `nn_robust_oos.npy`, `v5_blend_oos.npy` | `scripts/08_…`, `scripts/09_…` | out-of-sample predictions (5-fold, seed 42) reused by steps 10–14 |
added data/raw/README.md +14 −0
@@ -0,0 +1,14 @@
1 +# Raw data — Realtor.ca and DuProprio listings (Apify crawlers, 2026-07-03)
2 +
3 +| File | Source | Records | Size |
4 +|---|---|---|---|
5 +| `dataset_realtor-property-crawler_2026-07-03_10-36-26-001.json` | Realtor.ca (MLS) via Apify `realtor-property-crawler` | 27,302 listings | 161 MB |
6 +| `dataset_duproprio-property-crawler_2026-07-03_00-37-49-449.json` | DuProprio via Apify `duproprio-property-crawler` | 7,693 listings | 87 MB |
7 +
8 +Cross-sectional snapshot extracted on 2026-07-03 (no time dimension). Common fields: `source`, `url`, `listingId`,
9 +`price`, `address{…}`, `property{…}`, `features[]`, `description`, `images[]`. Realtor adds `rooms[]` (room dimensions)
10 +but no coordinates; DuProprio adds `coordinates{lat,lon}` and captioned images. Unique key: `uid = source:listingId`.
11 +Houses (single-family + townhouse) with a valid price: 19,123 listings used in the paper.
12 +
13 +Not committed (size and crawler terms). Re-run the two Apify actors or request the snapshot from the author.
14 +Full data dictionary and cleaning rules: `docs/METHODOLOGIE.md` (French).
added docs/METHODOLOGIE.md +172 −0
@@ -0,0 +1,172 @@
1 +# Méthodologie — Modèle hédonique de prix des maisons (Québec)
2 +
3 +*Pipeline : Realtor.ca + DuProprio → nettoyage → feature engineering (règles + LLM + vision) → régression hédonique log-linéaire à effets fixes.*
4 +
5 +---
6 +
7 +## 1. Origine des données
8 +
9 +| Source | Fichier | Nb enreg. | Nature |
10 +|---|---|---|---|
11 +| **Realtor.ca** | `dataset_realtor-property-crawler_...json` (161 Mo) | 27 302 | Annonces MLS via crawler Realtor (Apify) |
12 +| **DuProprio** | `dataset_duproprio-property-crawler_...json` (87 Mo) | 7 693 | Annonces vente sans intermédiaire (Apify) |
13 +
14 +- **Collecte** : scrapers Apify, extraction le **2026-07-03**. Chaque enregistrement est un objet JSON structuré.
15 +- **Champs communs** : `source`, `url`, `listingId`, `title`, `price`/`priceNumber`, `currency`, `address{full,street,city,province,postalCode,...}`, `property{...}`, `features[]`, `description`, `images[]`, `scrapedAt`.
16 +- **Spécificités** :
17 + - Realtor : `property.buildingType`, `storeys`, `yearBuilt`, `parkingSpaces`, **`rooms[]`** (dimensions par pièce), pas de coordonnées GPS, images sans caption utile.
18 + - DuProprio : `property.type`, `levels`, **`coordinates{lat,lon}`** (7 691 présentes), images **avec captions** (`Kitchen`, etc.), pas de `rooms`.
19 +- **Instantané transversal** (cross-section) : un seul point dans le temps → pas de dimension temporelle, donc pas d'effet fixe temporel nécessaire.
20 +
21 +**Identifiant unique** construit : `uid = source + ":" + listingId` (évite toute collision entre les deux espaces d'ID).
22 +
23 +---
24 +
25 +## 2. Construction du dataset combiné « maisons »
26 +
27 +1. **Fusion** des deux tableaux JSON (schéma harmonisé).
28 +2. **Définition de « maison »** (choix utilisateur : unifamiliales + townhouses) :
29 + - Realtor : `buildingType ∈ {House, Row / Townhouse}`
30 + - DuProprio : `type ∈ {Home, Bungalow, Semi Detached, Intergenerational, Country Home, Cottage, Villa, Townhouse}`
31 + - Exclus : condos, plex (duplex→quintuplex), terrains/lots, fermes, commercial, mobile homes.
32 +3. **Filtre prix présent** : `priceNumber` non nul et > 0.
33 +
34 +→ **19 273 maisons avec prix** (15 470 Realtor + 3 803 DuProprio).
35 +
36 +---
37 +
38 +## 3. Nettoyage des aberrations
39 +
40 +Les prix scrappés contiennent des erreurs de parsing (min 899 $, max 368 G$). Règles appliquées :
41 +
42 +| Variable | Règle | Justification |
43 +|---|---|---|
44 +| Prix | garder **50 k$ – 15 M$** | plage plausible d'une maison au QC ; retire loyers mal parsés et concaténations |
45 +| Superficie habitable | 300 – 20 000 ft² (sinon → manquant) | valeurs physiquement possibles |
46 +| Terrain | 200 – 5 000 000 ft² (sinon → manquant) | idem |
47 +| Chambres / sdb | 0–15 / 0–12 (sinon → manquant) | plafonds réalistes |
48 +| **Prix/pied²** | garder **P0,5 – P99,5** (≈ 73–1 276 $/ft²) sur les biens à surface réelle | retire les erreurs résiduelles de parsing prix ou surface |
49 +
50 +→ **19 115 maisons** conservées (158 retirées).
51 +
52 +**Principe** : on borne plutôt que de supprimer quand une seule variable est aberrante (on met la variable à *manquant* et on l'impute), pour ne pas perdre l'observation entière.
53 +
54 +---
55 +
56 +## 4. Construction des variables
57 +
58 +### 4.1 Variable cible
59 +`ln_price = ln(priceNumber)` — transformation logarithmique (rend la distribution symétrique, linéarise les effets multiplicatifs, coefficients ≈ **% d'impact sur le prix**).
60 +
61 +### 4.2 Variables structurelles (règles déterministes)
62 +- **Surface & terrain** : parsing d'unités hétérogènes → tout en **pieds carrés**.
63 + - `"1535 sqft"`, `"1,333.2 ft²"`, `"11 561.52 ft²"` (séparateurs de milliers variés), `"961.6 m2"` (m² × 10,7639).
64 +- **Reconstruction de surface via les pièces (Realtor)** : dimensions `"23 ft ,4 in x 13 ft ,8 in"` → pieds-décimaux → aire ; somme sur toutes les pièces. Calibration sur le recouvrement (ratio médian surface réelle / somme pièces ≈ **0,98**) → sert à combler `squareFeet` manquant.
65 + - Constat : les annonces à pièces détaillées ≈ celles qui ont déjà `squareFeet` → gain de couverture faible, mais les **comptes/types de pièces** restent informatifs.
66 +- **Autres** : `beds`, `baths`, `parking` (Realtor), `storeys`/`levels`, `age = 2026 − yearBuilt` (+ `age²` pour la courbure), drapeaux `has_*` marquant les valeurs imputées.
67 +
68 +### 4.3 Aménités (règles par mots-clés bilingues)
69 +17 dummies détectés dans `features[] + description + title` (FR/EN) : piscine, bord de l'eau, garage, foyer, climatisation/thermopompe, sous-sol fini, cul-de-sac, golf, ski, proximité parc/école/autoroute/transport, absence de voisin arrière.
70 +
71 +### 4.4 Détail des pièces (Realtor, `rooms[]`)
72 +Nombre de pièces, nombre de niveaux, chambre principale, sous-sol, bureau, salle familiale, atelier, **cellier à vin**, solarium, mezzanine, bibliothèque, buanderie.
73 +
74 +### 4.5 Signaux de listing
75 +Nombre de photos (`n_images`, écrêté à 60), présence de visite virtuelle — proxies de gamme/qualité de la mise en marché.
76 +
77 +### 4.6 Variables géographiques (effets fixes)
78 +- **Ville** nettoyée (Realtor : on retire le quartier entre parenthèses ; minuscules ; normalisation). Villes avec **≥ 20 annonces** → dummy propre ; les rares → `__other__`. ≈ 194 villes en effets fixes.
79 +- **Type de propriété** harmonisé, regroupé si < 30 annonces.
80 +- **Source** (Realtor/DuProprio) → dummy captant les différences systématiques entre plateformes.
81 +
82 +### 4.7 Imputation
83 +Valeurs manquantes après nettoyage : imputées par **médiane par type de propriété** (puis médiane globale), **avec un drapeau `has_*`** pour que le modèle distingue « valeur observée » de « valeur imputée ».
84 +
85 +---
86 +
87 +## 5. Mécanismes IA pour le feature engineering
88 +
89 +Deux couches IA extraient des variables **impossibles à obtenir par règles** — l'une du texte, l'autre des images.
90 +
91 +### 5.1 Extraction de specs depuis la description — **GPT-4o-mini** (OpenAI)
92 +- **But** : transformer la description libre en 20 attributs structurés (état : à rénover→neuf, clé en main, cuisine/sdb rénovée, revêtements de sol, comptoirs quartz/granit, chauffage, vue sur l'eau, revenus locatifs, grand terrain, secteur tranquille, + mots-clés de vente).
93 +- **Mécanisme** : appel Chat Completions avec **`response_format = json_schema` strict** (schéma imposé, énumérations fermées) → sortie JSON valide garantie, pas de parsing fragile. `temperature = 0` (déterminisme). Système : « extracteur factuel, uniquement d'après le texte ; absence → false/inconnu/[] ».
94 +- **Ingénierie** : ~600 tokens in / ~110 out par annonce ; **parallèle** (24 threads), **résumable** (cache `specs.jsonl` clé = uid), **retry** exponentiel sur 429/5xx.
95 +- **Coût/temps** : ~$3 / ~19 k annonces.
96 +
97 +### 5.2 Attributs visuels de façade — **Claude Haiku 4.5** (Anthropic, vision)
98 +- **But** : lire ce que le texte ne dit pas — **état visuel réel, attrait extérieur (curb appeal), niveau de rénovation, matériaux, aménagement paysager, cadre (urbain/banlieue/rural/bord d'eau/boisé), indice de luxe, qualité de la photo**.
99 +- **Sélection d'image** : 1 photo/annonce = la façade. Realtor → 1re photo. DuProprio → photo dont la caption est l'adresse (pas une pièce), via exclusion de mots-clés de pièces.
100 +- **Mécanisme** : Messages API, image passée par **URL** (`source.type = url`, pas de téléchargement), **`output_config.format = json_schema`** (12 attributs, énumérations fermées). Système : « évaluateur objectif ; scores fondés sur critères concrets (surfaces, matériaux, aménagement), pas sur un goût personnel ; si non-extérieur → is_exterior=false ».
101 +- **Ingénierie** : ~2 470 tokens in / ~100 out par image ; **parallèle** (16 threads), **résumable** (cache `facade_specs.jsonl`), **retry** sur 429/5xx/529.
102 +- **Coût/temps** : ~$57 / ~19 k images.
103 +- **Pourquoi Haiku** : modèle vision le moins cher et rapide, largement suffisant pour de la classification d'attributs concrets.
104 +
105 +### 5.3 Pourquoi le JSON structuré strict est central
106 +Le schéma imposé (types + `enum` + champs requis + `additionalProperties:false`) fait valider la sortie **au niveau de l'API** : le modèle est forcé de renvoyer exactement les champs attendus, dans le bon domaine de valeurs. On obtient des variables **directement tabulables**, reproductibles, sans post-traitement de texte libre.
107 +
108 +### 5.4 Intégration au dataset
109 +Les sorties LLM (clé `uid`) sont converties en variables numériques : booléens → 0/1, `condition` → score ordinal 0-5, revêtements → dummies (bois franc, quartz/granit), vue → dummy eau, etc., **avec un drapeau `has_gpt` / `is_exterior`** pour marquer la disponibilité.
110 +
111 +---
112 +
113 +## 6. Modèle économétrique
114 +
115 +### 6.1 Spécification — hédonique log-linéaire à effets fixes
116 +$$\ln(P_i) = \alpha + \beta\,\ln(\text{surface}_i) + \gamma\,\ln(\text{terrain}_i) + \sum_k \delta_k X_{ki} + \sum_c \lambda_c \mathbb{1}[\text{ville}_i=c] + \varepsilon_i$$
117 +
118 +- **Log-log** sur surface et terrain → coefficients = **élasticités** (% prix par % surface).
119 +- **Log-niveau** sur le reste → un coefficient $b$ ≈ **$(e^b-1)\times100$ % d'impact** sur le prix.
120 +- **Effets fixes ville** ($\lambda_c$) : absorbent tout le différentiel de localisation (le plus gros déterminant du prix). Chaque coefficient = prime/décote de la ville vs la ville de référence.
121 +- **Effets fixes type de propriété** et **source**.
122 +
123 +### 6.2 Double spécification (couverture vs richesse)
124 +Realtor ne rapporte `squareFeet` que pour ~38 % des annonces → deux modèles, prédictions combinées :
125 +- **Modèle A (riche)** : inclut `ln(surface)`, estimé sur les biens à surface **mesurée** (réelle ou reconstruite via pièces). Meilleur ajustement.
126 +- **Modèle B (large)** : sans surface, estimé sur **toutes** les maisons → sert à pricer celles sans surface (via beds/baths/terrain/pièces + specs).
127 +- Prédiction finale : A si surface mesurée, sinon B.
128 +
129 +### 6.3 Estimation & inférence
130 +- **MCO (OLS)** sur `ln_price`.
131 +- **Erreurs-types robustes à l'hétéroscédasticité HC3** (les résidus immobiliers sont hétéroscédastiques ; HC3 est conservateur en petit/moyen échantillon).
132 +- **Correction du biais de rétrotransformation** : $\hat{P} = e^{\hat{\ln P}} \times e^{\hat\sigma^2/2}$ (correction normale de Duan) — sinon $e^{\hat{\ln P}}$ sous-estime systématiquement le niveau.
133 +
134 +### 6.4 Interprétation
135 +Coefficients rapportés en **% d'impact prix** + significativité (p-values sur SE HC3). Effets fixes ville présentés comme primes de localisation.
136 +
137 +---
138 +
139 +## 7. Statistiques & diagnostics
140 +
141 +| Indicateur | Signification |
142 +|---|---|
143 +| **R² / R²-ajusté** | part de variance de `ln_price` expliquée |
144 +| **Écart-type des résidus (log)** | → **erreur typique en %** = $(e^{\sigma}-1)\times100$ |
145 +| **Erreur absolue médiane** | \|prix réel / prix prédit − 1\| médian sur toutes les prédictions |
146 +| **% à ±15 % / ±25 %** | part des prédictions dans la fourchette |
147 +| **`resid_pct`** par annonce | surprix/sous-prix vs modèle → repère aubaines & surévaluations |
148 +
149 +Progression observée : v2 (structurel) R²≈0,70 → v3 (+ pièces/photos) R²≈0,72 → v4 (+ specs GPT) → v5 (+ vision) attendu.
150 +
151 +---
152 +
153 +## 8. Limites & biais (transparence)
154 +
155 +- **Endogénéité / non-causalité** : de belles photos ou des specs riches corrèlent avec des courtiers haut de gamme et des biens chers → coefficients **prédictifs**, pas causaux. À ne pas lire comme « ajouter une piscine ⇒ +X % ».
156 +- **Biais de sélection** : uniquement des biens **en vente** (prix demandé, pas prix de transaction) ; les biens sans surface/pièces sont non aléatoires (annonces moins complètes).
157 +- **Imputation** : les médianes par type réduisent la variance des variables imputées (atténuation possible des coefficients) — d'où les drapeaux `has_*`.
158 +- **Extraction LLM** : erreurs/hallucinations possibles ; `temperature=0` + schéma strict + énumérations fermées les limitent, mais pas de validation humaine systématique. La vision juge sur **une seule** photo.
159 +- **Instantané unique** : pas de dynamique de marché ; primes de localisation figées au 2026-07-03.
160 +- **Couverture géographique** : villes < 20 annonces regroupées en `__other__` (moins de finesse spatiale là où les données sont rares).
161 +
162 +---
163 +
164 +## 9. Fichiers produits
165 +
166 +| Fichier | Contenu |
167 +|---|---|
168 +| `hedonic.py`, `hedonic_v4.py` | pipelines reproductibles (nettoyage + modèles) |
169 +| `extract_specs.py`, `facade_vision.py` | extracteurs LLM (texte / vision), parallèles & résumables |
170 +| `specs.jsonl`, `facade_specs.jsonl` | sorties LLM brutes (clé uid) |
171 +| `maisons_hedonic*.csv` | dataset final + prix prédit + `resid_pct` |
172 +| `hedonic_*_summary.txt` | sorties statistiques complètes |
added docs/highlights_elsevier.tex +17 −0
@@ -0,0 +1,17 @@
1 +% ============================================================================
2 +% Highlights — separate file required at Elsevier submission
3 +% (3-5 bullets, max 85 characters each, results-oriented)
4 +% ============================================================================
5 +\documentclass[12pt]{article}
6 +\usepackage[utf8]{inputenc}
7 +\usepackage[T1]{fontenc}
8 +\begin{document}
9 +\section*{Highlights}
10 +\begin{itemize}
11 + \item LLM and vision models turn listing text and photos into hedonic regressors
12 + \item Schema-constrained decoding makes AI outputs reproducible econometric variables
13 + \item Two AI feature layers add four points of $R^2$ to a rich hedonic model
14 + \item Façade-vision features give the largest out-of-sample gain of any block
15 + \item Curb appeal, materials, and luxury cues carry sensible implicit prices
16 +\end{itemize}
17 +\end{document}
added docs/methodologie.pdf +0 −0

Binary file not shown.

added docs/methodologie.tex +372 −0
@@ -0,0 +1,372 @@
1 +\documentclass[11pt,a4paper]{article}
2 +
3 +\usepackage[utf8]{inputenc}
4 +\usepackage[T1]{fontenc}
5 +\usepackage[french]{babel}
6 +\usepackage{lmodern}
7 +\usepackage{amsmath,amssymb}
8 +\usepackage{booktabs}
9 +\usepackage{longtable}
10 +\usepackage{array}
11 +\usepackage{geometry}
12 +\usepackage{xcolor}
13 +\usepackage{enumitem}
14 +\usepackage{hyperref}
15 +\usepackage{fancyhdr}
16 +
17 +\geometry{margin=2.4cm}
18 +\hypersetup{colorlinks=true, linkcolor=blue!55!black, urlcolor=blue!55!black,
19 + citecolor=blue!55!black}
20 +
21 +\newcommand{\code}[1]{\texttt{\small #1}}
22 +\renewcommand{\arraystretch}{1.25}
23 +
24 +\pagestyle{fancy}
25 +\fancyhf{}
26 +\lhead{\small Modèle hédonique de prix — Québec}
27 +\rhead{\small\thepage}
28 +\renewcommand{\headrulewidth}{0.4pt}
29 +
30 +\title{\textbf{Méthodologie}\\[2mm]
31 +\large Modèle hédonique de prix des maisons (Québec)\\[1mm]
32 +\normalsize Realtor.ca $+$ DuProprio $\rightarrow$ nettoyage $\rightarrow$ ingénierie
33 +de variables (règles $+$ LLM $+$ vision) $\rightarrow$ régression hédonique
34 +log-linéaire à effets fixes}
35 +\author{}
36 +\date{Données extraites le 3 juillet 2026}
37 +
38 +\begin{document}
39 +\maketitle
40 +\thispagestyle{fancy}
41 +
42 +\begin{abstract}
43 +\noindent Ce document décrit la méthodologie complète de construction d'un modèle
44 +hédonique de prix pour les maisons unifamiliales et maisons en rangée du Québec.
45 +Le pipeline combine deux jeux de données d'annonces immobilières, un nettoyage des
46 +valeurs aberrantes fondé sur des règles, une ingénierie de variables à quatre
47 +origines (règles déterministes, extraction de texte par grand modèle de langage,
48 +analyse d'images par modèle de vision, effets fixes géographiques) et une
49 +estimation par moindres carrés ordinaires du logarithme du prix avec erreurs-types
50 +robustes à l'hétéroscédasticité.
51 +\end{abstract}
52 +
53 +\tableofcontents
54 +\bigskip
55 +
56 +%======================================================================
57 +\section{Origine des données}
58 +
59 +\begin{longtable}{@{}p{2.4cm}p{5.2cm}rp{4.6cm}@{}}
60 +\toprule
61 +\textbf{Source} & \textbf{Fichier} & \textbf{Nb} & \textbf{Nature}\\
62 +\midrule
63 +\endhead
64 +Realtor.ca & \code{dataset\_realtor-\dots.json} (161 Mo) & 27\,302 &
65 +Annonces MLS via crawler Apify\\
66 +DuProprio & \code{dataset\_duproprio-\dots.json} (87 Mo) & 7\,693 &
67 +Annonces de vente sans intermédiaire (Apify)\\
68 +\bottomrule
69 +\end{longtable}
70 +
71 +\begin{itemize}[leftmargin=1.2em]
72 + \item \textbf{Collecte} : scrapers Apify, extraction le \textbf{2026-07-03}. Chaque
73 + enregistrement est un objet JSON structuré.
74 + \item \textbf{Champs communs} : \code{source}, \code{url}, \code{listingId},
75 + \code{title}, \code{price}/\code{priceNumber}, \code{currency},
76 + \code{address\{...\}}, \code{property\{...\}}, \code{features[]},
77 + \code{description}, \code{images[]}, \code{scrapedAt}.
78 + \item \textbf{Spécificités} :
79 + \begin{itemize}[leftmargin=1.2em]
80 + \item Realtor : \code{buildingType}, \code{storeys}, \code{yearBuilt},
81 + \code{parkingSpaces}, \textbf{\code{rooms[]}} (dimensions par pièce) ;
82 + pas de coordonnées GPS ; images sans légende utile.
83 + \item DuProprio : \code{type}, \code{levels},
84 + \textbf{\code{coordinates}} (7\,691 présentes) ; images \textbf{avec légendes}
85 + (\emph{Kitchen}, etc.) ; pas de \code{rooms}.
86 + \end{itemize}
87 + \item \textbf{Instantané transversal} (une seule date) : pas de dimension
88 + temporelle, donc pas d'effet fixe temporel requis.
89 +\end{itemize}
90 +
91 +\noindent\textbf{Identifiant unique} : \code{uid = source + ":" + listingId}
92 +(évite toute collision entre les deux espaces d'identifiants).
93 +
94 +%======================================================================
95 +\section{Construction du dataset «~maisons~»}
96 +
97 +\begin{enumerate}[leftmargin=1.4em]
98 + \item \textbf{Fusion} des deux tableaux JSON (schéma harmonisé).
99 + \item \textbf{Définition de «~maison~»} (unifamiliales $+$ maisons en rangée) :
100 + \begin{itemize}[leftmargin=1.2em]
101 + \item Realtor : \code{buildingType} $\in \{$House, Row/Townhouse$\}$ ;
102 + \item DuProprio : \code{type} $\in \{$Home, Bungalow, Semi Detached,
103 + Intergenerational, Country Home, Cottage, Villa, Townhouse$\}$ ;
104 + \item \emph{Exclus} : condos, plex (duplex$\rightarrow$quintuplex),
105 + terrains/lots, fermes, commercial, maisons mobiles.
106 + \end{itemize}
107 + \item \textbf{Filtre prix présent} : \code{priceNumber} non nul et $>0$.
108 +\end{enumerate}
109 +
110 +\noindent$\rightarrow$ \textbf{19\,273 maisons avec prix} (15\,470 Realtor
111 +$+$ 3\,803 DuProprio).
112 +
113 +%======================================================================
114 +\section{Nettoyage des valeurs aberrantes}
115 +
116 +Les prix extraits contiennent des erreurs d'analyse syntaxique (minimum
117 +observé 899~\$, maximum 368~G\$). Règles appliquées :
118 +
119 +\begin{longtable}{@{}p{3.0cm}p{6.2cm}p{5.2cm}@{}}
120 +\toprule
121 +\textbf{Variable} & \textbf{Règle} & \textbf{Justification}\\
122 +\midrule
123 +\endhead
124 +Prix & garder \textbf{50\,k\$ -- 15\,M\$} & plage plausible au QC ; retire loyers
125 +mal analysés et concaténations\\
126 +Surface habitable & 300 -- 20\,000 ft$^2$ (sinon $\rightarrow$ manquant) &
127 +valeurs physiquement possibles\\
128 +Terrain & 200 -- 5\,000\,000 ft$^2$ (sinon $\rightarrow$ manquant) & idem\\
129 +Chambres / sdb & 0--15 / 0--12 (sinon $\rightarrow$ manquant) & plafonds réalistes\\
130 +Prix/pied$^2$ & garder \textbf{P0{,}5 -- P99{,}5} ($\approx 73$--$1276~\$$/ft$^2$) &
131 +retire les erreurs résiduelles de prix ou surface\\
132 +\bottomrule
133 +\end{longtable}
134 +
135 +\noindent$\rightarrow$ \textbf{19\,115 maisons} conservées (158 retirées).
136 +
137 +\paragraph{Principe.} On \emph{borne} plutôt que de supprimer quand une seule
138 +variable est aberrante : la variable est mise à \emph{manquant} puis imputée, pour
139 +ne pas perdre l'observation entière.
140 +
141 +%======================================================================
142 +\section{Construction des variables}
143 +
144 +\subsection{Variable cible}
145 +\[
146 +\code{ln\_price} = \ln(\code{priceNumber})
147 +\]
148 +La transformation logarithmique rend la distribution symétrique, linéarise les
149 +effets multiplicatifs et donne des coefficients interprétables en pourcentage
150 +d'impact sur le prix.
151 +
152 +\subsection{Variables structurelles (règles déterministes)}
153 +\begin{itemize}[leftmargin=1.2em]
154 + \item \textbf{Surface \& terrain} : analyse d'unités hétérogènes, tout converti en
155 + \textbf{pieds carrés} : \code{"1535 sqft"}, \code{"1,333.2 ft²"},
156 + \code{"11 561.52 ft²"} (séparateurs de milliers variés),
157 + \code{"961.6 m2"} ($\times 10{,}7639$).
158 + \item \textbf{Reconstruction de surface via les pièces (Realtor)} : dimensions
159 + \code{"23 ft ,4 in x 13 ft ,8 in"} $\rightarrow$ pieds décimaux $\rightarrow$ aire,
160 + sommées sur toutes les pièces. Calibration sur le recouvrement (ratio médian
161 + surface réelle / somme des pièces $\approx 0{,}98$) pour combler
162 + \code{squareFeet} manquant.
163 + \item \textbf{Autres} : \code{beds}, \code{baths}, \code{parking},
164 + \code{storeys}/\code{levels}, \code{age} $= 2026 - \code{yearBuilt}$ (avec
165 + \code{age}$^2$ pour la courbure), drapeaux \code{has\_*} marquant les valeurs
166 + imputées.
167 +\end{itemize}
168 +
169 +\subsection{Aménités (règles par mots-clés bilingues)}
170 +17 variables indicatrices détectées dans \code{features[]} $+$ \code{description}
171 +$+$ \code{title} (FR/EN) : piscine, bord de l'eau, garage, foyer, climatisation/
172 +thermopompe, sous-sol fini, cul-de-sac, golf, ski, proximité parc/école/autoroute/
173 +transport, absence de voisin arrière.
174 +
175 +\subsection{Détail des pièces (Realtor)}
176 +Nombre de pièces, nombre de niveaux, chambre principale, sous-sol, bureau, salle
177 +familiale, atelier, cellier à vin, solarium, mezzanine, bibliothèque, buanderie.
178 +
179 +\subsection{Signaux de mise en marché}
180 +Nombre de photos (\code{n\_images}, écrêté à 60), présence de visite virtuelle —
181 +indicateurs indirects de gamme et de qualité de l'annonce.
182 +
183 +\subsection{Variables géographiques (effets fixes)}
184 +\begin{itemize}[leftmargin=1.2em]
185 + \item \textbf{Ville} nettoyée (Realtor : retrait du quartier entre parenthèses ;
186 + minuscules ; normalisation). Villes avec $\ge 20$ annonces $\rightarrow$
187 + indicatrice propre, les rares $\rightarrow$ \code{\_\_other\_\_}
188 + ($\approx 194$ villes en effets fixes).
189 + \item \textbf{Type de propriété} harmonisé, regroupé si $<30$ annonces.
190 + \item \textbf{Source} (Realtor/DuProprio) $\rightarrow$ indicatrice captant les
191 + différences systématiques entre plateformes.
192 +\end{itemize}
193 +
194 +\subsection{Imputation}
195 +Valeurs manquantes après nettoyage : imputées par \textbf{médiane par type de
196 +propriété} (puis médiane globale), \textbf{avec un drapeau \code{has\_*}} pour que
197 +le modèle distingue valeur observée et valeur imputée.
198 +
199 +%======================================================================
200 +\section{Mécanismes d'IA pour l'ingénierie de variables}
201 +
202 +Deux couches d'IA extraient des variables impossibles à obtenir par règles~: l'une
203 +depuis le texte, l'autre depuis les images.
204 +
205 +\subsection{Extraction de specs depuis la description --- GPT-4o-mini}
206 +\begin{itemize}[leftmargin=1.2em]
207 + \item \textbf{But} : transformer la description libre en 20 attributs structurés
208 + (état~: à rénover$\rightarrow$neuf ; clé en main ; cuisine/sdb rénovée ;
209 + revêtements de sol ; comptoirs quartz/granit ; chauffage ; vue sur l'eau ;
210 + revenus locatifs ; grand terrain ; secteur tranquille ; mots-clés de vente).
211 + \item \textbf{Mécanisme} : appel \emph{Chat Completions} avec
212 + \code{response\_format = json\_schema} \textbf{strict} (schéma imposé,
213 + énumérations fermées) $\rightarrow$ sortie JSON valide garantie, sans analyse
214 + fragile. \code{temperature = 0} (déterminisme). Instruction système : extracteur
215 + factuel, uniquement d'après le texte ; absence $\rightarrow$ false/inconnu/[].
216 + \item \textbf{Ingénierie} : $\sim$600 tokens en entrée / $\sim$110 en sortie par
217 + annonce ; \textbf{parallèle} (24 fils), \textbf{résumable} (cache
218 + \code{specs.jsonl}, clé \code{uid}), \textbf{réessais} exponentiels sur 429/5xx.
219 + \item \textbf{Coût / temps} : $\sim$\$3 pour $\sim$19\,k annonces.
220 +\end{itemize}
221 +
222 +\subsection{Attributs visuels de façade --- Claude Haiku 4.5 (vision)}
223 +\begin{itemize}[leftmargin=1.2em]
224 + \item \textbf{But} : lire ce que le texte ne dit pas --- état visuel réel,
225 + attrait extérieur (\emph{curb appeal}), niveau de rénovation, matériaux,
226 + aménagement paysager, cadre (urbain/banlieue/rural/bord d'eau/boisé), indice de
227 + luxe, qualité de la photo.
228 + \item \textbf{Sélection d'image} : une photo/annonce $=$ la façade. Realtor
229 + $\rightarrow$ 1\up{re} photo ; DuProprio $\rightarrow$ photo dont la légende est
230 + l'adresse (pas une pièce), par exclusion de mots-clés de pièces.
231 + \item \textbf{Mécanisme} : \emph{Messages API}, image passée par \textbf{URL}
232 + (\code{source.type = url}, sans téléchargement),
233 + \code{output\_config.format = json\_schema} (12 attributs, énumérations fermées).
234 + Instruction système : évaluateur objectif ; scores fondés sur des critères
235 + concrets (surfaces, matériaux, aménagement), pas sur un goût personnel.
236 + \item \textbf{Ingénierie} : $\sim$2\,470 tokens en entrée / $\sim$100 en sortie
237 + par image ; \textbf{parallèle} (16 fils), \textbf{résumable} (cache
238 + \code{facade\_specs.jsonl}), \textbf{réessais} sur 429/5xx/529.
239 + \item \textbf{Coût / temps} : $\sim$\$57 pour $\sim$19\,k images.
240 + \item \textbf{Choix de Haiku} : modèle de vision le moins cher et rapide,
241 + suffisant pour de la classification d'attributs concrets.
242 +\end{itemize}
243 +
244 +\subsection{Rôle central du JSON structuré strict}
245 +Le schéma imposé (types $+$ \code{enum} $+$ champs requis $+$
246 +\code{additionalProperties:false}) fait valider la sortie \textbf{au niveau de
247 +l'API}~: le modèle renvoie exactement les champs attendus, dans le bon domaine de
248 +valeurs. On obtient des variables directement tabulables et reproductibles, sans
249 +post-traitement de texte libre.
250 +
251 +\subsection{Intégration au dataset}
252 +Les sorties des modèles (clé \code{uid}) sont converties en variables numériques~:
253 +booléens $\rightarrow$ 0/1, état $\rightarrow$ score ordinal 0--5, revêtements
254 +$\rightarrow$ indicatrices (bois franc, quartz/granit), vue $\rightarrow$
255 +indicatrice eau, etc., avec un drapeau \code{has\_gpt} / \code{is\_exterior}
256 +marquant la disponibilité.
257 +
258 +%======================================================================
259 +\section{Modèle économétrique}
260 +
261 +\subsection{Spécification --- hédonique log-linéaire à effets fixes}
262 +\begin{equation}
263 +\ln(P_i) = \alpha + \beta\,\ln(\text{surface}_i) + \gamma\,\ln(\text{terrain}_i)
264 ++ \sum_k \delta_k X_{ki}
265 ++ \sum_c \lambda_c\,\mathbb{1}[\text{ville}_i=c]
266 ++ \varepsilon_i
267 +\end{equation}
268 +
269 +\begin{itemize}[leftmargin=1.2em]
270 + \item \textbf{Log--log} sur surface et terrain $\rightarrow$ coefficients $=$
271 + \textbf{élasticités} (\% de prix par \% de surface).
272 + \item \textbf{Log--niveau} sur le reste $\rightarrow$ un coefficient $b$ vaut
273 + $\approx (e^{b}-1)\times 100$~\% d'impact sur le prix.
274 + \item \textbf{Effets fixes ville} ($\lambda_c$) : absorbent le différentiel de
275 + localisation (premier déterminant du prix) ; chaque coefficient $=$ prime/décote
276 + de la ville vs la référence.
277 + \item \textbf{Effets fixes} type de propriété et source.
278 +\end{itemize}
279 +
280 +\subsection{Double spécification (couverture vs richesse)}
281 +Realtor ne rapporte \code{squareFeet} que pour $\sim$38~\% des annonces, d'où deux
282 +modèles et des prédictions combinées~:
283 +\begin{itemize}[leftmargin=1.2em]
284 + \item \textbf{Modèle A (riche)} : inclut $\ln(\text{surface})$, estimé sur les
285 + biens à surface \emph{mesurée} (réelle ou reconstruite via les pièces). Meilleur
286 + ajustement.
287 + \item \textbf{Modèle B (large)} : sans surface, estimé sur \emph{toutes} les
288 + maisons $\rightarrow$ price celles sans surface (beds/baths/terrain/pièces $+$
289 + specs).
290 + \item \textbf{Prédiction finale} : A si surface mesurée, sinon B.
291 +\end{itemize}
292 +
293 +\subsection{Estimation et inférence}
294 +\begin{itemize}[leftmargin=1.2em]
295 + \item \textbf{Moindres carrés ordinaires} sur \code{ln\_price}.
296 + \item \textbf{Erreurs-types robustes à l'hétéroscédasticité HC3} (les résidus
297 + immobiliers sont hétéroscédastiques ; HC3 est conservateur en échantillon
298 + moyen).
299 + \item \textbf{Correction du biais de rétrotransformation} :
300 + \[
301 + \hat{P} = e^{\widehat{\ln P}} \times e^{\hat\sigma^2/2}
302 + \]
303 + (correction normale de Duan) --- sinon $e^{\widehat{\ln P}}$ sous-estime
304 + systématiquement le niveau de prix.
305 +\end{itemize}
306 +
307 +\subsection{Interprétation}
308 +Coefficients rapportés en \textbf{\% d'impact prix} $+$ significativité (p-values
309 +sur SE HC3). Effets fixes ville présentés comme primes de localisation.
310 +
311 +%======================================================================
312 +\section{Statistiques et diagnostics}
313 +
314 +\begin{longtable}{@{}p{5.2cm}p{9.4cm}@{}}
315 +\toprule
316 +\textbf{Indicateur} & \textbf{Signification}\\
317 +\midrule
318 +\endhead
319 +$R^2$ / $R^2$ ajusté & part de variance de \code{ln\_price} expliquée\\
320 +Écart-type des résidus (log) & $\rightarrow$ erreur typique en \% $=(e^{\sigma}-1)\times100$\\
321 +Erreur absolue médiane & $|\,$prix réel $/$ prix prédit $-1\,|$ médian\\
322 +\% à $\pm15$~\% / $\pm25$~\% & part des prédictions dans la fourchette\\
323 +\code{resid\_pct} par annonce & surprix/sous-prix vs modèle
324 +$\rightarrow$ repère aubaines et surévaluations\\
325 +\bottomrule
326 +\end{longtable}
327 +
328 +\noindent Progression observée : v2 (structurel) $R^2\approx0{,}70$
329 +$\rightarrow$ v3 ($+$ pièces/photos) $R^2\approx0{,}72$
330 +$\rightarrow$ v4 ($+$ specs GPT) $\rightarrow$ v5 ($+$ vision) attendu.
331 +
332 +%======================================================================
333 +\section{Limites et biais}
334 +
335 +\begin{itemize}[leftmargin=1.2em]
336 + \item \textbf{Endogénéité / non-causalité} : de belles photos ou des specs riches
337 + corrèlent avec des courtiers haut de gamme et des biens chers $\rightarrow$
338 + coefficients \emph{prédictifs}, pas causaux.
339 + \item \textbf{Biais de sélection} : uniquement des biens \emph{en vente} (prix
340 + demandé, pas prix de transaction) ; les biens sans surface/pièces sont non
341 + aléatoires (annonces moins complètes).
342 + \item \textbf{Imputation} : les médianes par type réduisent la variance des
343 + variables imputées (atténuation possible des coefficients), d'où les drapeaux
344 + \code{has\_*}.
345 + \item \textbf{Extraction LLM} : erreurs/hallucinations possibles ;
346 + \code{temperature=0} $+$ schéma strict $+$ énumérations fermées les limitent,
347 + sans validation humaine systématique. La vision juge sur \emph{une seule} photo.
348 + \item \textbf{Instantané unique} : pas de dynamique de marché ; primes de
349 + localisation figées au 2026-07-03.
350 + \item \textbf{Couverture géographique} : villes $<20$ annonces regroupées en
351 + \code{\_\_other\_\_}.
352 +\end{itemize}
353 +
354 +%======================================================================
355 +\section{Fichiers produits}
356 +
357 +\begin{longtable}{@{}p{5.4cm}p{9.2cm}@{}}
358 +\toprule
359 +\textbf{Fichier} & \textbf{Contenu}\\
360 +\midrule
361 +\endhead
362 +\code{hedonic.py}, \code{hedonic\_v4.py} & pipelines reproductibles (nettoyage $+$
363 +modèles)\\
364 +\code{extract\_specs.py}, \code{facade\_vision.py} & extracteurs LLM (texte /
365 +vision), parallèles et résumables\\
366 +\code{specs.jsonl}, \code{facade\_specs.jsonl} & sorties LLM brutes (clé \code{uid})\\
367 +\code{maisons\_hedonic*.csv} & dataset final $+$ prix prédit $+$ \code{resid\_pct}\\
368 +\code{hedonic\_*\_summary.txt} & sorties statistiques complètes\\
369 +\bottomrule
370 +\end{longtable}
371 +
372 +\end{document}
added paper/.latexmkrc +5 −0
@@ -0,0 +1,5 @@
1 +$pdf_mode = 1;
2 +$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S';
3 +$bibtex_use = 2;
4 +$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg';
5 +@default_files = ('main.tex');
added paper/Makefile +11 −0
@@ -0,0 +1,11 @@
1 +.PHONY: all clean distclean
2 +
3 +all:
4 + latexmk
5 +
6 +clean:
7 + latexmk -c
8 +
9 +distclean:
10 + latexmk -C
11 + rm -f *.synctex.gz *.run.xml *.bbl *.bcf
added paper/appendix/appendix.tex +92 −0
@@ -0,0 +1,92 @@
1 +% ============================================================================
2 +% Appendix
3 +% ============================================================================
4 +\newpage
5 +\begin{appendices}
6 +
7 +\renewcommand{\thetable}{A\arabic{table}}
8 +\renewcommand{\thefigure}{A\arabic{figure}}
9 +\setcounter{table}{0}
10 +\setcounter{figure}{0}
11 +
12 +\section{Machine-Generated Feature Definitions}
13 +\label{app:features}
14 +
15 +Table~\ref{tab:app_features} lists the attributes extracted by the language and
16 +vision models under their respective strict JSON schemas. All fields use closed
17 +enumerations or booleans; absent information is coded as \texttt{false},
18 +\texttt{unknown}, or an empty list.
19 +
20 +\begin{table}[htbp]
21 + \centering
22 + \begin{threeparttable}
23 + \caption{Attributes Extracted by the AI Models}
24 + \label{tab:app_features}
25 + \small
26 + \begin{tabular}{@{}p{2.9cm}p{4.4cm}p{6.3cm}@{}}
27 + \toprule
28 + Layer & Field & Domain \\
29 + \midrule
30 + \multicolumn{3}{@{}l}{\textit{Panel A: Text (LLM, \texttt{gpt-4o-mini})}} \\[3pt]
31 + Condition & \texttt{condition} & to renovate / original / good / renovated / like-new / new \\
32 + Turnkey & \texttt{turnkey} & boolean \\
33 + Renovated kitchen & \texttt{renovated\_kitchen} & boolean \\
34 + Renovated bathroom & \texttt{renovated\_bathroom} & boolean \\
35 + Flooring & \texttt{flooring} & hardwood / ceramic / laminate / carpet / vinyl / concrete \\
36 + Countertop & \texttt{countertop} & granite / quartz / marble / laminate / wood \\
37 + Heating & \texttt{heating} & electric / heat pump / wood / gas / oil / radiant / geothermal \\
38 + Water view & \texttt{view} & water / mountain / city / none \\
39 + Income property & \texttt{income\_property} & boolean \\
40 + Large lot; quiet area & \texttt{large\_lot}, \texttt{quiet\_area} & boolean \\[6pt]
41 + \multicolumn{3}{@{}l}{\textit{Panel B: Vision (\texttt{claude-haiku-4-5}, façade)}} \\[3pt]
42 + Building type (visual) & \texttt{building\_type\_visual} & detached / semi / row / condo / cottage / land \\
43 + Condition score & \texttt{condition\_score} & integer 1--10 \\
44 + Curb appeal & \texttt{curb\_appeal} & integer 1--10 \\
45 + Renovation level & \texttt{renovation\_level} & to renovate / dated / standard / renovated / high-end \\
46 + Exterior materials & \texttt{exterior\_materials} & brick / stone / vinyl / wood / aluminum / stucco / concrete \\
47 + Landscaping & \texttt{landscaping} & none / minimal / maintained / manicured \\
48 + Setting & \texttt{setting} & dense urban / suburban / rural / waterfront / wooded \\
49 + Luxury indicator & \texttt{curb\_luxury} & boolean \\
50 + Photo quality & \texttt{photo\_quality} & integer 1--10 \\
51 + \bottomrule
52 + \end{tabular}
53 + \begin{tablenotes}[flushleft]
54 + \footnotesize
55 + \item \textit{Notes:} Both extractions use schema-constrained decoding with
56 + closed enumerations, required fields, and \texttt{additionalProperties:false};
57 + decoding temperature is zero. The text layer reads the free-text description;
58 + the vision layer analyzes one façade photograph per listing. Fields are mapped
59 + to numeric regressors (booleans to $0/1$, ordinal scales to integers, categories
60 + to indicators), each with an availability flag.
61 + \end{tablenotes}
62 + \end{threeparttable}
63 +\end{table}
64 +
65 +\section{Sample Construction Flow}
66 +\label{app:flow}
67 +
68 +\begin{table}[htbp]
69 + \centering
70 + \begin{threeparttable}
71 + \caption{From Raw Records to Estimation Sample}
72 + \label{tab:app_flow}
73 + \begin{tabular}{@{}lr@{}}
74 + \toprule
75 + Step & Records \\
76 + \midrule
77 + Realtor.ca raw & 27{,}302 \\
78 + DuProprio raw & 7{,}693 \\
79 + Houses (single-family + townhouse) with price & 19{,}273 \\
80 + After outlier cleaning & 19{,}123 \\
81 + \quad of which with measured living area (Model~A) & 9{,}446 \\
82 + \bottomrule
83 + \end{tabular}
84 + \begin{tablenotes}[flushleft]
85 + \footnotesize
86 + \item \textit{Notes:} See Section~\ref{sec:data}. ``Measured living area'' means
87 + reported area or area reconstructed from room-level dimensions.
88 + \end{tablenotes}
89 + \end{threeparttable}
90 +\end{table}
91 +
92 +\end{appendices}
added paper/figures/fig_ablation.pdf +0 −0

Binary file not shown.

added paper/figures/fig_calibration.pdf +0 −0

Binary file not shown.

added paper/figures/fig_importance.pdf +0 −0

Binary file not shown.

added paper/figures/fig_premia.pdf +0 −0

Binary file not shown.

added paper/figures/fig_validation.pdf +0 −0

Binary file not shown.

added paper/figures/nn_diagram.tex +80 −0
@@ -0,0 +1,80 @@
1 +% ============================================================================
2 +% TikZ: Robust tabular neural network architecture
3 +% ============================================================================
4 +\begin{tikzpicture}[
5 + font=\small,
6 + >=Latex,
7 + node distance=6mm and 6mm,
8 + box/.style={rounded corners=2pt, draw=black!55, align=center,
9 + minimum height=8mm, inner sep=3pt, text width=24mm},
10 + cat/.style={box, fill=NavyBlue!8},
11 + num/.style={box, fill=OliveGreen!10, text width=30mm},
12 + emb/.style={box, fill=BurntOrange!14, text width=22mm},
13 + op/.style={box, fill=black!5},
14 + spine/.style={box, fill=black!6, text width=52mm},
15 + res/.style={box, fill=Plum!10, text width=52mm, minimum height=11mm},
16 + outb/.style={box, fill=NavyBlue!16, text width=34mm, font=\bfseries},
17 + detail/.style={rounded corners=2pt, draw=black!40, dashed, fill=Plum!5,
18 + align=left, text width=40mm, inner sep=4pt, font=\footnotesize},
19 + ar/.style={->, thick, black!65}
20 +]
21 +
22 +% ---- inputs ----
23 +\node[cat] (city) {City index\\{\footnotesize (195 cats)}};
24 +\node[cat, right=of city] (type) {Type index};
25 +\node[cat, right=of type] (src) {Source index};
26 +\node[num, right=10mm of src] (numin) {Numeric features\\{\footnotesize $\approx$45: area, lot, beds,\\baths, age, amenities,\\rooms, GPT specs}};
27 +
28 +% ---- embeddings / bn ----
29 +\node[emb, below=of city] (ecity) {Embedding\\{\footnotesize dim 29}};
30 +\node[emb, below=of type] (etype) {Embedding\\{\footnotesize dim 4}};
31 +\node[emb, below=of src] (esrc) {Embedding\\{\footnotesize dim 2}};
32 +\node[op, below=of numin, text width=30mm] (bn) {BatchNorm\\{\footnotesize + missingness flags}};
33 +
34 +\draw[ar] (city)--(ecity); \draw[ar] (type)--(etype);
35 +\draw[ar] (src)--(esrc); \draw[ar] (numin)--(bn);
36 +
37 +% ---- concat ----
38 +\node[spine, below=12mm of etype.south, xshift=8mm] (concat)
39 + {Concatenate \;$\rightarrow$\; $\mathbb{R}^{\,\text{emb}+\text{num}}$};
40 +\draw[ar] (ecity) |- (concat.west);
41 +\draw[ar] (etype) -- (concat);
42 +\draw[ar] (esrc) -- (concat);
43 +\draw[ar] (bn) -| (concat.east);
44 +
45 +% ---- projection ----
46 +\node[spine, below=of concat] (proj) {Linear $\rightarrow$ 256\;\; (trunk)};
47 +\draw[ar] (concat)--(proj);
48 +
49 +% ---- residual blocks ----
50 +\node[res, below=of proj] (resb)
51 + {Residual block \;$\times\,3$\\{\footnotesize BN $\to$ GELU $\to$ Lin$_{512}$ $\to$ Dropout $\to$ Lin$_{256}$ \;$\oplus$\; skip}};
52 +\draw[ar] (proj)--(resb);
53 +
54 +% ---- head ----
55 +\node[spine, below=of resb] (head)
56 + {Head:\; BatchNorm $\to$ GELU $\to$ Linear$_{1}$};
57 +\draw[ar] (resb)--(head);
58 +
59 +% ---- output ----
60 +\node[outb, below=of head] (out) {$\widehat{\ln(\text{Price})}$};
61 +\draw[ar] (head)--(out);
62 +
63 +% ---- residual detail callout ----
64 +\node[detail, right=10mm of resb] (dtl)
65 + {\textbf{Res block:} $x \mapsto x + f(x)$\\
66 + $f=$ Lin$_{256}\!\circ$Drop$\circ$Lin$_{512}\!\circ$GELU$\circ$BN};
67 +\draw[->, dashed, black!45] (resb.east)--(dtl.west);
68 +
69 +% ---- training / ensemble annotation ----
70 +\node[detail, right=10mm of head, text width=42mm, fill=OliveGreen!6, draw=black!40] (train)
71 + {\textbf{Training (robust):}\\
72 + Huber loss $\cdot$ AdamW $\cdot$ cosine LR\\
73 + grad-clip 5 $\cdot$ dropout $\cdot$ early stop};
74 +
75 +% ---- ensemble + blend ----
76 +\node[op, below=8mm of out, text width=64mm, fill=NavyBlue!7] (ens)
77 + {Ensemble of 3 seeds (average) \;$\Rightarrow$\; blend 50/50 with gradient boosting};
78 +\draw[ar] (out)--(ens);
79 +
80 +\end{tikzpicture}
added paper/main.pdf +0 −0

Binary file not shown.

added paper/main.tex +150 −0
@@ -0,0 +1,150 @@
1 +% ============================================================================
2 +% UQO Working Paper — Hedonic House Pricing with LLM & Vision Features
3 +% ============================================================================
4 +\documentclass[12pt,letterpaper]{article}
5 +
6 +% --- Encoding & Language ---
7 +\usepackage[utf8]{inputenc}
8 +\usepackage[T1]{fontenc}
9 +\usepackage[english]{babel}
10 +
11 +% --- Page Layout ---
12 +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
13 +
14 +% --- Typography ---
15 +\usepackage{mathptmx}
16 +\usepackage{microtype}
17 +\usepackage{setspace}
18 +\onehalfspacing
19 +
20 +% --- Math ---
21 +\usepackage{amsmath,amssymb,amsthm}
22 +\usepackage{bm}
23 +
24 +% --- Tables ---
25 +\usepackage{booktabs}
26 +\usepackage{tabularx}
27 +\usepackage{multirow}
28 +\usepackage{threeparttable}
29 +
30 +% --- Figures ---
31 +\usepackage{graphicx}
32 +\usepackage{subcaption}
33 +\usepackage{float}
34 +\usepackage{tikz}
35 +\usetikzlibrary{arrows.meta,positioning,fit,backgrounds,calc,shapes.geometric}
36 +
37 +% --- Colors & Links ---
38 +\usepackage[dvipsnames]{xcolor}
39 +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true,pdftitle={Hedonic House Pricing with Language-Model and Vision Features: Evidence from Quebec Real-Estate Listings},pdfauthor={Simon-Pierre Boucher}]{hyperref}
40 +
41 +% --- Bibliography ---
42 +\usepackage[authoryear,round,semicolon]{natbib}
43 +\bibliographystyle{aer}
44 +
45 +% --- Captions ---
46 +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
47 +
48 +% --- Headers & Footers ---
49 +\usepackage{fancyhdr}
50 +\pagestyle{fancy}
51 +\fancyhf{}
52 +\renewcommand{\headrulewidth}{0pt}
53 +\fancyfoot[C]{\thepage}
54 +
55 +% --- Section Formatting ---
56 +\usepackage{titlesec}
57 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
58 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
59 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
60 +
61 +% --- Appendix Support ---
62 +\usepackage[toc,page]{appendix}
63 +
64 +% --- Misc ---
65 +\usepackage{enumitem}
66 +\setlist{nosep,leftmargin=*}
67 +
68 +% ============================================================================
69 +% METADATA
70 +% ============================================================================
71 +\newcommand{\WPnumber}{14}
72 +\newcommand{\WPtitle}{Hedonic House Pricing with Language-Model and Vision Features}
73 +\newcommand{\WPsubtitle}{Evidence from Quebec Real-Estate Listings}
74 +\newcommand{\WPdate}{July 2026}
75 +\newcommand{\WPversion}{2.0}
76 +\newcommand{\WPabstract}{%
77 + Hedonic house-price models are bounded by the characteristics an analyst can
78 + measure, yet much of a home's value is recorded only in the free-text description
79 + and photographs of its listing. We treat general-purpose AI models as a
80 + feature-engineering technology for hedonic analysis: a large language model
81 + converts each description into structured property specifications, and a
82 + vision--language model converts the façade photograph into objective visual
83 + attributes, both under strict JSON schemas that render the outputs reproducible
84 + econometric variables. Using 19{,}123 Quebec house listings, the two AI layers
85 + raise the $R^2$ of an already rich log--linear hedonic specification from 0.717 to
86 + 0.755, and the vision layer delivers the largest out-of-sample gain of any feature
87 + block for every estimator, from OLS to gradient boosting and neural networks (best
88 + ensemble: out-of-sample $R^2$ of 0.779, median absolute error 14.0\%). The
89 + extracted attributes carry economically sensible implicit prices---2.3\% per point
90 + of curb appeal, 3--4\% for brick or stone façades---that survive
91 + municipality-clustered inference.
92 +}
93 +\newcommand{\WPkeywords}{hedonic pricing, house prices, large language models, computer vision, feature engineering, Quebec}
94 +\newcommand{\WPjel}{R31, C21, C55, R21}
95 +
96 +% --- Author ---
97 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
98 +\newcommand{\WPaffiliation}{%
99 + Département des sciences administratives\\
100 + Université du Québec en Outaouais}
101 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
102 +\newcommand{\WPaddress}{%
103 + Gatineau -- Pavillon Alexandre-Taché\\
104 + 283, boulevard Alexandre-Taché\\
105 + Gatineau, Québec, Canada J9A 1L8}
106 +
107 +% ============================================================================
108 +\begin{document}
109 +
110 +\input{sections/titlepage}
111 +
112 +\input{sections/introduction}
113 +\input{sections/literature}
114 +\input{sections/data}
115 +\input{sections/methodology}
116 +\input{sections/results}
117 +\input{sections/benchmark}
118 +\input{sections/robustness}
119 +\input{sections/conclusion}
120 +
121 +% ============================================================================
122 +% Declarations (Elsevier requirements)
123 +% ============================================================================
124 +\section*{Declarations}
125 +
126 +\paragraph{Funding.} This research did not receive any specific grant from funding
127 +agencies in the public, commercial, or not-for-profit sectors.
128 +
129 +\paragraph{Declaration of competing interest.} The author declares that he has no
130 +known competing financial interests or personal relationships that could have
131 +appeared to influence the work reported in this paper.
132 +
133 +\paragraph{Data availability.} The listing data were collected from publicly
134 +accessible web platforms (Realtor.ca and DuProprio) on 3~July~2026. The full
135 +data-construction code, the extraction schemas, and the derived analysis dataset
136 +(excluding copyrighted listing text and images) are available from the author upon
137 +reasonable request.
138 +
139 +\paragraph{Use of AI models.} Large language and vision--language models
140 +(\texttt{gpt-4o-mini} and \texttt{claude-haiku-4-5}) are the object of study of this
141 +paper and were used exclusively as documented in Section~\ref{sec:methodology} to
142 +extract structured features from listing text and photographs. They were not used to
143 +generate the analysis or the conclusions of the paper.
144 +
145 +\newpage
146 +\bibliography{references}
147 +
148 +\input{appendix/appendix}
149 +
150 +\end{document}
added paper/references.bib +316 −0
@@ -0,0 +1,316 @@
1 +@article{rosen1974hedonic,
2 + title={Hedonic prices and implicit markets: product differentiation in pure competition},
3 + author={Rosen, Sherwin},
4 + journal={Journal of Political Economy},
5 + volume={82},
6 + number={1},
7 + pages={34--55},
8 + year={1974}
9 +}
10 +
11 +@article{lancaster1966new,
12 + title={A new approach to consumer theory},
13 + author={Lancaster, Kelvin J.},
14 + journal={Journal of Political Economy},
15 + volume={74},
16 + number={2},
17 + pages={132--157},
18 + year={1966}
19 +}
20 +
21 +@article{sirmans2005composition,
22 + title={The composition of hedonic pricing models},
23 + author={Sirmans, Stacy G. and Macpherson, David A. and Zietz, Emily N.},
24 + journal={Journal of Real Estate Literature},
25 + volume={13},
26 + number={1},
27 + pages={1--44},
28 + year={2005}
29 +}
30 +
31 +@incollection{malpezzi2003hedonic,
32 + title={Hedonic pricing models: a selective and applied review},
33 + author={Malpezzi, Stephen},
34 + booktitle={Housing Economics and Public Policy},
35 + editor={O'Sullivan, Tony and Gibb, Kenneth},
36 + pages={67--89},
37 + publisher={Blackwell Science},
38 + year={2003}
39 +}
40 +
41 +@article{duan1983smearing,
42 + title={Smearing estimate: a nonparametric retransformation method},
43 + author={Duan, Naihua},
44 + journal={Journal of the American Statistical Association},
45 + volume={78},
46 + number={383},
47 + pages={605--610},
48 + year={1983}
49 +}
50 +
51 +@article{white1980heteroskedasticity,
52 + title={A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity},
53 + author={White, Halbert},
54 + journal={Econometrica},
55 + volume={48},
56 + number={4},
57 + pages={817--838},
58 + year={1980}
59 +}
60 +
61 +@article{mackinnon1985some,
62 + title={Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties},
63 + author={MacKinnon, James G. and White, Halbert},
64 + journal={Journal of Econometrics},
65 + volume={29},
66 + number={3},
67 + pages={305--325},
68 + year={1985}
69 +}
70 +
71 +@article{can1992spatial,
72 + title={Specification and estimation of hedonic housing price models},
73 + author={Can, Ayse},
74 + journal={Regional Science and Urban Economics},
75 + volume={22},
76 + number={3},
77 + pages={453--474},
78 + year={1992}
79 +}
80 +
81 +@book{anselin1988spatial,
82 + title={Spatial Econometrics: Methods and Models},
83 + author={Anselin, Luc},
84 + publisher={Kluwer Academic Publishers},
85 + address={Dordrecht},
86 + year={1988}
87 +}
88 +
89 +@article{gentzkow2019text,
90 + title={Text as data},
91 + author={Gentzkow, Matthew and Kelly, Bryan and Taddy, Matt},
92 + journal={Journal of Economic Literature},
93 + volume={57},
94 + number={3},
95 + pages={535--574},
96 + year={2019}
97 +}
98 +
99 +@article{poursaeed2018vision,
100 + title={Vision-based real estate price estimation},
101 + author={Poursaeed, Omid and Matera, Tomas and Belongie, Serge},
102 + journal={Machine Vision and Applications},
103 + volume={29},
104 + number={4},
105 + pages={667--676},
106 + year={2018}
107 +}
108 +
109 +@inproceedings{ahmed2016house,
110 + title={House price estimation from visual and textual features},
111 + author={Ahmed, Eman H. and Moustafa, Mohamed},
112 + booktitle={Proceedings of the 8th International Joint Conference on Computational Intelligence (NCTA)},
113 + pages={62--68},
114 + year={2016}
115 +}
116 +
117 +@article{bin2004prediction,
118 + title={A prediction comparison of housing sales prices by parametric versus semi-parametric regressions},
119 + author={Bin, Okmyung},
120 + journal={Journal of Housing Economics},
121 + volume={13},
122 + number={1},
123 + pages={68--84},
124 + year={2004}
125 +}
126 +
127 +@article{mullainathan2017machine,
128 + title={Machine learning: an applied econometric approach},
129 + author={Mullainathan, Sendhil and Spiess, Jann},
130 + journal={Journal of Economic Perspectives},
131 + volume={31},
132 + number={2},
133 + pages={87--106},
134 + year={2017}
135 +}
136 +
137 +@article{halvorsen1980interpretation,
138 + title={The interpretation of dummy variables in semilogarithmic equations},
139 + author={Halvorsen, Robert and Palmquist, Raymond},
140 + journal={American Economic Review},
141 + volume={70},
142 + number={3},
143 + pages={474--475},
144 + year={1980}
145 +}
146 +
147 +@article{breiman2001random,
148 + title={Random forests},
149 + author={Breiman, Leo},
150 + journal={Machine Learning},
151 + volume={45},
152 + number={1},
153 + pages={5--32},
154 + year={2001}
155 +}
156 +
157 +@article{friedman2001greedy,
158 + title={Greedy function approximation: a gradient boosting machine},
159 + author={Friedman, Jerome H.},
160 + journal={Annals of Statistics},
161 + volume={29},
162 + number={5},
163 + pages={1189--1232},
164 + year={2001}
165 +}
166 +
167 +@article{guo2016entity,
168 + title={Entity embeddings of categorical variables},
169 + author={Guo, Cheng and Berkhahn, Felix},
170 + journal={arXiv preprint arXiv:1604.06737},
171 + year={2016}
172 +}
173 +
174 +@inproceedings{loshchilov2019decoupled,
175 + title={Decoupled weight decay regularization},
176 + author={Loshchilov, Ilya and Hutter, Frank},
177 + booktitle={International Conference on Learning Representations (ICLR)},
178 + year={2019}
179 +}
180 +
181 +@inproceedings{gorishniy2021revisiting,
182 + title={Revisiting deep learning models for tabular data},
183 + author={Gorishniy, Yury and Rubachev, Ivan and Khrulkov, Valentin and Babenko, Artem},
184 + booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
185 + year={2021}
186 +}
187 +
188 +@inproceedings{grinsztajn2022why,
189 + title={Why do tree-based models still outperform deep learning on tabular data?},
190 + author={Grinsztajn, L{\'e}o and Oyallon, Edouard and Varoquaux, Ga{\"e}l},
191 + booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
192 + year={2022}
193 +}
194 +
195 +@inproceedings{he2016deep,
196 + title={Deep residual learning for image recognition},
197 + author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
198 + booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
199 + pages={770--778},
200 + year={2016}
201 +}
202 +
203 +@article{nowak2017textual,
204 + title={Textual analysis in real estate},
205 + author={Nowak, Adam and Smith, Patrick S.},
206 + journal={Journal of Applied Econometrics},
207 + volume={32},
208 + number={4},
209 + pages={896--918},
210 + year={2017}
211 +}
212 +
213 +@article{shen2021information,
214 + title={Information value of property description: A machine learning approach},
215 + author={Shen, Lily and Ross, Stephen L.},
216 + journal={Journal of Urban Economics},
217 + volume={121},
218 + pages={103299},
219 + year={2021}
220 +}
221 +
222 +@article{johnson2020curb,
223 + title={Valuing curb appeal},
224 + author={Johnson, Erik B. and Tidwell, Alan and Villupuram, Sriram V.},
225 + journal={Journal of Real Estate Finance and Economics},
226 + volume={60},
227 + number={1-2},
228 + pages={111--133},
229 + year={2020}
230 +}
231 +
232 +@article{lindenthal2021architecture,
233 + title={Machine learning, architectural styles and property values},
234 + author={Lindenthal, Thies and Johnson, Erik B.},
235 + journal={Journal of Real Estate Finance and Economics},
236 + year={2021}
237 +}
238 +
239 +@techreport{glaeser2018vision,
240 + title={Computer vision and real estate: Do looks matter and do incentives determine looks},
241 + author={Glaeser, Edward L. and Kincaid, Michael Scott and Naik, Nikhil},
242 + institution={National Bureau of Economic Research},
243 + type={Working Paper},
244 + number={25174},
245 + year={2018}
246 +}
247 +
248 +@article{kok2017bigdata,
249 + title={Big data in real estate? {F}rom manual appraisal to automated valuation},
250 + author={Kok, Nils and Koponen, Eija-Leena and Mart{\'i}nez-Barbosa, Carmen Adriana},
251 + journal={Journal of Portfolio Management},
252 + volume={43},
253 + number={6},
254 + pages={202--211},
255 + year={2017}
256 +}
257 +
258 +@article{hendel2009mls,
259 + title={The relative performance of real estate marketing platforms: {MLS} versus {FSBOMadison.com}},
260 + author={Hendel, Igal and Nevo, Aviv and Ortalo-Magn{\'e}, Fran{\c{c}}ois},
261 + journal={American Economic Review},
262 + volume={99},
263 + number={5},
264 + pages={1878--1898},
265 + year={2009}
266 +}
267 +
268 +@article{horowitz1992list,
269 + title={The role of the list price in housing markets: Theory and an econometric model},
270 + author={Horowitz, Joel L.},
271 + journal={Journal of Applied Econometrics},
272 + volume={7},
273 + number={2},
274 + pages={115--129},
275 + year={1992}
276 +}
277 +
278 +@article{knight2002listing,
279 + title={Listing price, time on market, and ultimate selling price: Causes and effects of listing price changes},
280 + author={Knight, John R.},
281 + journal={Real Estate Economics},
282 + volume={30},
283 + number={2},
284 + pages={213--237},
285 + year={2002}
286 +}
287 +
288 +@incollection{han2015microstructure,
289 + title={The microstructure of housing markets: Search, bargaining, and brokerage},
290 + author={Han, Lu and Strange, William C.},
291 + booktitle={Handbook of Regional and Urban Economics},
292 + volume={5},
293 + pages={813--886},
294 + publisher={Elsevier},
295 + year={2015}
296 +}
297 +
298 +@article{hill2013hedonic,
299 + title={Hedonic price indexes for residential housing: A survey, evaluation and taxonomy},
300 + author={Hill, Robert J.},
301 + journal={Journal of Economic Surveys},
302 + volume={27},
303 + number={5},
304 + pages={879--914},
305 + year={2013}
306 +}
307 +
308 +@article{cameron2015cluster,
309 + title={A practitioner's guide to cluster-robust inference},
310 + author={Cameron, A. Colin and Miller, Douglas L.},
311 + journal={Journal of Human Resources},
312 + volume={50},
313 + number={2},
314 + pages={317--372},
315 + year={2015}
316 +}
added paper/sections/benchmark.tex +260 −0
@@ -0,0 +1,260 @@
1 +% ============================================================================
2 +\section{Predictive Benchmark: From OLS to Deep Learning}
3 +\label{sec:benchmark}
4 +% ============================================================================
5 +
6 +The hedonic OLS of Section~\ref{sec:results} is designed for interpretation: its
7 +coefficients are implicit prices. It is not, however, the most accurate predictor of
8 +price. To map the accuracy frontier---and to gauge how much curvature and interaction
9 +the linear model leaves on the table---we benchmark a ladder of estimators of
10 +increasing flexibility on the \emph{identical} feature set, under a common
11 +5-fold cross-validation protocol. All rows of Table~\ref{tab:models} share the same
12 +folds and random seed, so differences reflect the estimator, not the split.
13 +
14 +\subsection{The Estimator Ladder}
15 +\label{subsec:ladder}
16 +
17 +\paragraph{(1) Ordinary least squares.} The linear hedonic of
18 +Equation~\eqref{eq:hedonic}, with one-hot city, property-type, and source indicators
19 +and within-property-type median imputation of missing continuous attributes. It is
20 +the interpretable baseline and, mechanically, the least flexible: it imposes constant
21 +marginal effects and additive separability.
22 +
23 +\paragraph{(2) Random forest.} A bagged ensemble of regression trees
24 +\citep{mullainathan2017machine}. Trees capture non-linearities and low-order
25 +interactions automatically and handle missing values natively, but averaging
26 +deep decorrelated trees tends to over-smooth sharp price thresholds.
27 +
28 +\paragraph{(3) Gradient boosting.} A histogram-based gradient-boosted tree ensemble
29 +(600 boosting iterations, learning rate 0.05, up to 63 leaves, $L_2$ regularization).
30 +Boosting fits trees sequentially to residuals, which lets it model both smooth
31 +curvature (age, area) and abrupt thresholds; like the forest it ingests missing
32 +values directly. It is the strongest single model in our comparison.
33 +
34 +\paragraph{(4) Neural networks.} A plain multilayer perceptron and a robust
35 +residual network with entity embeddings, described in
36 +Section~\ref{subsec:nn}. Neural networks excel when high-cardinality categoricals
37 +(here, 195 municipalities) can be represented as dense learned vectors rather than
38 +sparse indicators.
39 +
40 +\paragraph{(5) Blend.} A simple 50/50 average of the robust neural network and the
41 +gradient-boosting predictions. Because the two models make partly decorrelated
42 +errors---the network smooths through embeddings while the boosted trees cut sharp
43 +thresholds---their average cancels idiosyncratic noise, a standard stacking result.
44 +
45 +\subsection{Formal Definitions}
46 +\label{subsec:formal}
47 +
48 +Let $y_i=\ln P_i$ and let $\mathbf{x}_i$ denote the feature vector of
49 +Equation~\eqref{eq:blocks}. Each estimator approximates the conditional mean
50 +$f(\mathbf{x})=\mathbb{E}[y\mid\mathbf{x}]$.
51 +
52 +\paragraph{Linear (OLS).} $f(\mathbf{x})=\mathbf{x}^{\!\top}\bm{\theta}$ with the
53 +closed form of Section~\ref{subsec:econometrics}; categoricals enter as one-hot
54 +indicators.
55 +
56 +\paragraph{Random forest \citep{breiman2001random}.} An average of $B$ regression
57 +trees, each grown on a bootstrap resample with a random feature subset at every split:
58 +%
59 +\begin{equation}
60 + \hat f_{\text{RF}}(\mathbf{x})=\frac1B\sum_{b=1}^{B} T_b(\mathbf{x};\Theta_b),
61 + \label{eq:rf}
62 +\end{equation}
63 +%
64 +where $\Theta_b$ denotes the randomness (resample and split subsets) of tree~$b$.
65 +
66 +\paragraph{Gradient boosting \citep{friedman2001greedy}.} A forward stagewise additive
67 +expansion built by fitting each new tree to the negative gradient of the loss:
68 +%
69 +\begin{equation}
70 + \resizebox{.94\textwidth}{!}{$\displaystyle
71 + F_M(\mathbf{x}) = F_0 + \nu\sum_{m=1}^{M} f_m(\mathbf{x}),
72 + \quad
73 + f_m = \arg\min_{f}\sum_{i=1}^{n}
74 + \big(\!-g_{mi}-f(\mathbf{x}_i)\big)^2,
75 + \quad
76 + g_{mi}=\frac{\partial L\big(y_i,F_{m-1}(\mathbf{x}_i)\big)}{\partial F_{m-1}(\mathbf{x}_i)}
77 + $}
78 + \label{eq:gbm}
79 +\end{equation}
80 +%
81 +with learning rate $\nu$ and $L_2$-regularized histogram trees $f_m$; missing values
82 +are routed to a learned default direction at each split.
83 +
84 +\paragraph{Neural network (robust).} High-cardinality categoricals are mapped to
85 +learned embeddings \citep{guo2016entity}, concatenated with the batch-normalized
86 +numeric block and projected to a $d$-dimensional trunk:
87 +%
88 +\begin{equation}
89 + \mathbf{h}_0 = \mathbf{W}_0
90 + \Big[\, \textstyle\bigoplus_{c\in\{\text{city},\text{type},\text{src}\}}
91 + \mathbf{E}_c(\text{cat}_{ci})
92 + \;\Vert\; \mathrm{BN}(\mathbf{x}^{\text{num}}_i)\,\Big].
93 + \label{eq:nn_in}
94 +\end{equation}
95 +%
96 +The trunk passes through $L=3$ pre-activation residual blocks
97 +\citep{he2016deep,gorishniy2021revisiting},
98 +%
99 +\begin{equation}
100 + \mathbf{h}_{\ell}
101 + = \mathbf{h}_{\ell-1}
102 + + \mathrm{Drop}\!\Big(\mathbf{W}_2^{(\ell)}\,
103 + \mathrm{Drop}\!\big(\phi(\mathbf{W}_1^{(\ell)}\,
104 + \mathrm{BN}(\mathbf{h}_{\ell-1}))\big)\Big),
105 + \qquad \phi=\text{GELU},
106 + \label{eq:resblock}
107 +\end{equation}
108 +%
109 +and a normalization head yields $\hat y_i = \mathbf{w}^{\!\top}\phi(\mathrm{BN}(\mathbf{h}_L))+b$.
110 +Parameters minimize the Huber (smooth-$L_1$) loss, robust to price outliers,
111 +%
112 +\begin{equation}
113 + L_\delta(r)=
114 + \begin{cases}
115 + \tfrac12 r^2, & |r|\le \delta,\\[2pt]
116 + \delta\big(|r|-\tfrac12\delta\big), & |r|>\delta,
117 + \end{cases}
118 + \qquad r_i=\hat y_i-y_i,\;\delta=1,
119 + \label{eq:huber}
120 +\end{equation}
121 +%
122 +optimized with AdamW (decoupled weight decay) \citep{loshchilov2019decoupled} under a
123 +cosine-annealed learning rate $\eta_t=\tfrac12\eta_0\big(1+\cos(\pi t/T)\big)$,
124 +gradient-norm clipping, and early stopping. Three seeds are averaged per fold.
125 +
126 +\paragraph{Blend.} A convex combination of the two strongest models in log space,
127 +%
128 +\begin{equation}
129 + \widehat{\ln P}^{\,\text{blend}}
130 + = w\,\widehat{\ln P}^{\,\text{GBM}}
131 + + (1-w)\,\widehat{\ln P}^{\,\text{NN}},
132 + \qquad w=0.5.
133 + \label{eq:blend}
134 +\end{equation}
135 +
136 +\paragraph{Evaluation.} With $\hat y_{-k(i)}$ the prediction for $i$ from the model
137 +trained on the folds excluding $i$'s fold, we report out-of-sample fit and level
138 +accuracy,
139 +%
140 +\begin{equation}
141 + R^2_{\text{OOS}} = 1-\frac{\sum_i\big(y_i-\hat y_{-k(i)}\big)^2}
142 + {\sum_i\big(y_i-\bar y\big)^2},
143 + \qquad
144 + \text{MdAPE}=\operatorname*{median}_i\Big|e^{\,\hat y_{-k(i)}-y_i}-1\Big|.
145 + \label{eq:cvmetrics}
146 +\end{equation}
147 +
148 +\subsection{Results}
149 +\label{subsec:bench_results}
150 +
151 +\input{tables/models}
152 +
153 +Table~\ref{tab:models} reports the ladder in two panels. In Panel~A (structural,
154 +amenity, and text features) out-of-sample $R^2$ rises monotonically with flexibility:
155 +from 0.628 for OLS, to 0.677 for the random forest, 0.711 for the plain network, and a
156 +near-tie at the top between the robust network (0.741) and gradient boosting (0.743);
157 +their blend attains $R^2_{\text{OOS}}=0.760$. Panel~B adds the twelve façade vision
158 +attributes. Vision lifts \emph{every} estimator---OLS gains 5.8 points
159 +(0.628 $\to$ 0.686), the random forest 2.7 (0.677 $\to$ 0.704), and gradient boosting
160 +2.7 (0.743 $\to$ 0.770), the single largest increment from any one feature layer at
161 +the frontier---and the \textbf{text\,+\,vision blend attains
162 +$R^2_{\text{OOS}}=0.779$ with a median absolute error of 14.0\%}, the best model in
163 +the study.
164 +
165 +Three lessons follow. First, on the full feature set roughly nine points of
166 +out-of-sample $R^2$ separate the interpretable linear model (0.686) from the
167 +predictive frontier (0.779)---the price of interpretability, and a measure of the
168 +curvature and interactions the hedonic form omits. Notably, the vision layer buys the
169 +linear model more accuracy (+5.8 points) than switching from OLS to a random forest
170 +does (+4.9), so better measurement rivals better estimation. Second, on this tabular
171 +problem gradient boosting and a carefully regularized neural network are
172 +statistically comparable, consistent with the broader evidence that tree ensembles
173 +remain strong baselines on tabular data \citep{grinsztajn2022why}; the gains come
174 +from \emph{combining} them. Third, the façade-vision layer is not redundant with the
175 +text and structural fields: it moves the frontier more than any other block we add,
176 +confirming that image-derived quality is genuinely orthogonal information.
177 +
178 +\begin{figure}[htbp]
179 + \centering
180 + \includegraphics[width=.72\textwidth]{figures/fig_ablation.pdf}
181 + \caption{Out-of-sample $R^2$ as feature layers are added (nested, left to right).
182 + Each layer improves both the linear and boosted models; the façade-vision layer
183 + delivers the largest single jump---about $+5.8$ points for OLS and $+2.7$ for
184 + gradient boosting.}
185 + \label{fig:ablation}
186 +\end{figure}
187 +
188 +\subsection{Feature Ablation and Importance}
189 +\label{subsec:ablation}
190 +
191 +Table~\ref{tab:ablation} and Figure~\ref{fig:ablation} decompose the accuracy gains
192 +layer by layer. Reading down the table, room detail and listing signals add the first
193 +large increment, the LLM text layer a smaller one, and the vision layer the single
194 +largest jump for every estimator---raising OLS from 0.628 to 0.686 and gradient
195 +boosting from 0.743 to 0.770 out of sample. The pattern is monotone: no layer degrades
196 +fit, and the two AI layers together account for a substantial share of the distance
197 +between the structural baseline and the frontier.
198 +
199 +\input{tables/ablation}
200 +
201 +Figure~\ref{fig:importance} aggregates the gradient-boosting permutation importances
202 +into feature blocks. Structural attributes and location fixed effects dominate, as
203 +expected, but the \textbf{vision block alone accounts for 13\% of total importance}---
204 +more than the rule-based amenities and the LLM text block combined---reinforcing that
205 +the façade photograph is a first-order, not marginal, source of price information.
206 +Figure~\ref{fig:calibration} shows the blend is well calibrated across the price range,
207 +with an approximately symmetric, tight error distribution centered near zero.
208 +
209 +\begin{figure}[htbp]
210 + \centering
211 + \includegraphics[width=.72\textwidth]{figures/fig_importance.pdf}
212 + \caption{Share of gradient-boosting permutation importance by feature block
213 + (text\,+\,vision model). The vision block accounts for 13\% of total importance,
214 + exceeding the rule-based amenity and LLM-text blocks.}
215 + \label{fig:importance}
216 +\end{figure}
217 +
218 +\begin{figure}[htbp]
219 + \centering
220 + \includegraphics[width=\textwidth]{figures/fig_calibration.pdf}
221 + \caption{Out-of-sample calibration of the text\,+\,vision blend. Left: predicted
222 + versus actual price (log--log; dashed line is the 45$^\circ$ identity). Right:
223 + distribution of percentage prediction errors, with median absolute error of 14.0\%.}
224 + \label{fig:calibration}
225 +\end{figure}
226 +
227 +\subsection{A Robust Tabular Neural Network}
228 +\label{subsec:nn}
229 +
230 +Figure~\ref{fig:nn} details the robust network. High-cardinality categoricals enter
231 +as learned \textbf{entity embeddings}: each municipality, property type, and source is
232 +mapped to a dense vector (dimension $\approx 1.6\,C^{0.56}$ in cardinality $C$, so
233 +about 29 dimensions for the 195 cities). The embeddings are concatenated with the
234 +batch-normalized numeric block and projected to a 256-dimensional trunk, which passes
235 +through three \textbf{residual blocks}
236 +(BatchNorm $\to$ GELU $\to$ Linear$_{512}$ $\to$ Dropout $\to$ Linear$_{256}$, with a
237 +skip connection), followed by a normalization head that outputs $\ln(\text{Price})$.
238 +
239 +Robustness comes from the training recipe rather than raw capacity:
240 +a \textbf{Huber (smooth-$L_1$) loss} that damps the influence of price outliers;
241 +\textbf{AdamW} with decoupled weight decay and \textbf{cosine-annealed} learning rate;
242 +\textbf{gradient-norm clipping} at 5; \textbf{dropout} and batch normalization
243 +throughout; \textbf{learned missingness indicators} appended to the numeric block so
244 +the network can exploit the non-random pattern of missing fields; and
245 +\textbf{early stopping} on a held-out 10\% of each training fold. Finally, three
246 +networks with different random seeds are trained per fold and \textbf{averaged}, which
247 +reduces prediction variance. The result (0.741) is competitive with gradient boosting
248 +and, once blended with it, produces the best model in the study.
249 +
250 +\begin{figure}[htbp]
251 + \centering
252 + \input{figures/nn_diagram}
253 + \caption{Architecture of the robust tabular neural network. High-cardinality
254 + categoricals enter as learned embeddings; numeric features are batch-normalized and
255 + augmented with missingness indicators. The concatenated representation is projected
256 + to a 256-dimensional trunk, refined by three residual blocks, and mapped to
257 + $\ln(\text{Price})$. Three seeds are ensembled per fold; the final model blends this
258 + network 50/50 with gradient boosting.}
259 + \label{fig:nn}
260 +\end{figure}
added paper/sections/conclusion.tex +41 −0
@@ -0,0 +1,41 @@
1 +% ============================================================================
2 +\section{Conclusion}
3 +\label{sec:conclusion}
4 +% ============================================================================
5 +
6 +We have estimated a hedonic price model for Quebec houses that augments the standard
7 +set of structural, locational, and amenity attributes with two machine-generated
8 +feature layers: structured specifications extracted from free-text descriptions by a
9 +large language model, and objective visual attributes extracted from façade
10 +photographs by a vision--language model. Schema-constrained decoding renders these
11 +outputs as ordinary, reproducible econometric variables, obtained at a marginal cost
12 +of a fraction of a cent per listing. The central message is methodological:
13 +general-purpose AI models, disciplined by strict output schemas, function as a
14 +low-cost feature-engineering technology that measurably expands the characteristics
15 +set available to hedonic analysis.
16 +
17 +The gains are economically meaningful. The two AI layers add nearly four points of
18 +in-sample $R^2$ to an already rich log--linear specification (0.717 to 0.755 with
19 +living area observed), and they raise the out-of-sample accuracy of every estimator
20 +in our ladder, from OLS to gradient boosting and an embedding-based residual neural
21 +network; the best blend attains an out-of-sample $R^2$ of 0.779 with a median
22 +absolute prediction error of 14.0\%. The façade-vision layer alone contributes the
23 +largest increment of any feature block (2.7 points for gradient boosting, 5.8 for
24 +OLS), confirming that image-derived quality is genuinely orthogonal to text and
25 +structured fields. Crucially for hedonic practice, the extracted attributes carry
26 +interpretable implicit prices---curb appeal, renovation level, luxury cues, and
27 +exterior materials all price sensibly and survive municipality-clustered
28 +inference---so the technology augments the classical regression rather than
29 +replacing it with a black box.
30 +
31 +Three extensions are natural. First, the same schema-constrained approach scales
32 +from a single façade image to a small panel of interior photographs (kitchen,
33 +bathroom, living space), which our design already accommodates. Second, the
34 +DuProprio coordinates permit continuous spatial features---distance to the urban
35 +core, to water, or to amenities---that would refine the geography beyond discrete
36 +city effects. Third, and most important for external validity, linking listings to
37 +registry transaction prices would establish how much of the AI-measured quality
38 +premium survives bargaining. For appraisal, automated valuation, and property-tax
39 +assessment---applications that consume hedonic models at scale---the practical
40 +implication is immediate: the unstructured majority of a listing's information
41 +content is no longer out of reach.
added paper/sections/data.tex +74 −0
@@ -0,0 +1,74 @@
1 +% ============================================================================
2 +\section{Data}
3 +\label{sec:data}
4 +% ============================================================================
5 +
6 +\subsection{Data Sources}
7 +\label{subsec:data_sources}
8 +
9 +We assemble a cross-section of residential listings from two complementary Quebec
10 +platforms, both retrieved on 3~July~2026 via web-scraping actors. The first,
11 +\textbf{Realtor.ca}, is the national Multiple Listing Service (MLS) portal and
12 +provides broker-listed inventory (27{,}302 raw records). The second,
13 +\textbf{DuProprio}, is the dominant for-sale-by-owner platform in Quebec (7{,}693
14 +raw records). Pooling the two sources broadens coverage across price segments and
15 +sale channels---brokered and FSBO listings differ systematically in seller
16 +composition and marketing \citep{hendel2009mls}---and a source indicator absorbs any
17 +systematic level difference between them. The price we observe is the \emph{asking}
18 +price; Section~\ref{subsec:limits} discusses the interpretation of hedonic
19 +coefficients estimated on list rather than transaction prices
20 +\citep{horowitz1992list,knight2002listing}.
21 +
22 +Each record is a structured JSON object containing a price, an address, a
23 +property-attribute block, a free-text description, a list of features, and an array
24 +of image URLs. The two sources differ in their auxiliary fields: Realtor.ca records
25 +include room-by-room dimensions but no geographic coordinates, whereas DuProprio
26 +records include coordinates and captioned images but no room block. We construct a
27 +unique key by concatenating the source and the listing identifier. Because the data
28 +form a single-date cross-section, no time dimension is modeled.
29 +
30 +\subsection{Sample Construction}
31 +\label{subsec:sample}
32 +
33 +We restrict attention to \emph{houses}, defined as single-family dwellings and
34 +townhouses. On Realtor.ca this corresponds to building types \textit{House} and
35 +\textit{Row/Townhouse}; on DuProprio it corresponds to the property types
36 +\textit{Home}, \textit{Bungalow}, \textit{Semi Detached}, \textit{Intergenerational},
37 +\textit{Country Home}, \textit{Cottage}, \textit{Villa}, and \textit{Townhouse}.
38 +Condominiums, income properties (duplex through quintuplex), vacant land, farms,
39 +commercial properties, and mobile homes are excluded. After requiring a non-missing,
40 +positive price, the pooled sample contains 19{,}273 houses.
41 +
42 +\subsection{Outlier Cleaning}
43 +\label{subsec:cleaning}
44 +
45 +Scraped prices contain parsing artifacts spanning several orders of magnitude (the
46 +raw minimum and maximum are \$899 and \$368~billion). We apply conservative,
47 +transparent bounds rather than model-based trimming: prices are retained in the
48 +\$50{,}000--\$15{,}000{,}000 range; living area in 300--20{,}000~ft$^2$; lot area in
49 +200--5{,}000{,}000~ft$^2$; bedrooms in 0--15 and bathrooms in 0--12. When a single
50 +attribute violates its bound it is set to missing (and later imputed) rather than
51 +deleting the observation, preserving sample size. Finally, we trim the
52 +price-per-square-foot distribution at its 0.5th and 99.5th percentiles
53 +(approximately \$73--\$1{,}276/ft$^2$) among listings with a measured area, removing
54 +residual price- or area-parsing errors. The cleaned sample contains
55 +\textbf{19{,}123 houses}, of which 80.3\% originate from Realtor.ca.
56 +
57 +\subsection{Summary Statistics}
58 +\label{subsec:summary_stats}
59 +
60 +Table~\ref{tab:summary_stats} reports descriptive statistics for the key variables.
61 +The median list price is \$599{,}900. The distribution of living area is right-skewed,
62 +motivating the logarithmic transformation adopted in Section~\ref{sec:methodology}.
63 +Living area is reported for only a minority of Realtor.ca listings; where it is
64 +missing we reconstruct it from room-level dimensions (Section~\ref{subsec:structural}),
65 +and where neither is available we rely on the area-free specification.
66 +
67 +\input{tables/summary_stats}
68 +
69 +Table~\ref{tab:ai_summary} previews the machine-generated attributes described in
70 +Section~\ref{sec:methodology}: roughly a quarter of listings are described as recently
71 +renovated, the mean visual condition and curb-appeal scores are 7.4 and 7.2 on a
72 +10-point scale, and 43\% of façades display a visual luxury cue.
73 +
74 +\input{tables/ai_summary}
added paper/sections/introduction.tex +87 −0
@@ -0,0 +1,87 @@
1 +% ============================================================================
2 +\section{Introduction}
3 +\label{sec:introduction}
4 +% ============================================================================
5 +
6 +The hedonic price framework decomposes the market value of a heterogeneous good
7 +into the implicit prices of its characteristics \citep{lancaster1966new,
8 +rosen1974hedonic}. In housing markets it remains the workhorse tool for appraisal,
9 +index construction, property-tax assessment, and policy analysis. Yet the empirical
10 +performance of hedonic models is bounded by the richness of the characteristics an
11 +analyst can measure. A large share of what determines a home's value---its state of
12 +repair, the quality of its finishes, its curb appeal, the character of its
13 +street---is recorded not in structured fields but in the free-text description and
14 +the photographs that accompany each listing. These modalities have historically been
15 +discarded because they were costly to encode at scale, leaving hedonic estimates
16 +exposed to omitted-quality bias and hedonic predictions short of their potential
17 +accuracy.
18 +
19 +This paper treats general-purpose artificial-intelligence models as a
20 +\emph{feature-engineering technology} for hedonic analysis, and quantifies the
21 +explanatory power that technology adds. We assemble a cross-section of 19{,}123
22 +single-family houses and townhouses listed for sale in Quebec, pooling the
23 +brokerage-listed MLS inventory from Realtor.ca with the for-sale-by-owner inventory
24 +from DuProprio. On top of standard structural, locational, and amenity attributes,
25 +we build two machine-generated feature layers. First, a large language model (LLM)
26 +reads each free-text description and returns a fixed set of property
27 +specifications---state of repair, renovation of the kitchen and bathroom, flooring
28 +and countertop materials, heating system, view, and income potential---under a
29 +strict JSON schema that guarantees valid, reproducible output. Second, a
30 +vision--language model inspects the façade photograph of each property and returns
31 +objective visual attributes: overall condition, curb appeal, renovation level,
32 +exterior materials, landscaping, setting, and a luxury indicator. Because both
33 +extractions are schema-constrained, deterministic, and blind to the listing price,
34 +their outputs behave as ordinary tabulated regressors rather than as noisy text.
35 +
36 +Our contribution is threefold. First, we show that AI-based feature engineering
37 +delivers economically meaningful gains in explanatory power over an already rich
38 +hedonic specification: the two machine-generated layers raise in-sample $R^2$ from
39 +0.717 to 0.755 (nearly four points) in our preferred specification, and raise the
40 +out-of-sample $R^2$ of every estimator we consider, with the façade-vision layer
41 +alone contributing the largest single increment of any feature block (2.7 points for
42 +gradient boosting, 5.8 points for OLS). Second, we show that the extracted
43 +attributes carry interpretable implicit prices---a one-point increase in visual curb
44 +appeal is associated with a 2.3\% higher asking price, a visible luxury cue with
45 +5.2\%, brick or stone façades with premia of 3--4\% relative to vinyl siding, and a
46 +renovated kitchen (read from the text) with 2.6\%---so the approach augments rather
47 +than replaces the classical hedonic regression. Third, we document a complete,
48 +reproducible pipeline from raw scraped listings to an analysis-ready dataset,
49 +including rules-based outlier cleaning, unit harmonization that reconstructs living
50 +area from room-level dimensions, and the schema-constrained extraction protocol,
51 +with a total AI cost of about \$60 for the full sample---roughly a third of a cent
52 +per listing.
53 +
54 +Our main findings can be summarized as follows. A log--linear specification with 195
55 +municipality fixed effects, property-type fixed effects, and
56 +municipality-clustered standard errors explains 75.5\% of the variation in log
57 +asking prices when living area is observed, and 72.5\% on the full sample when it is
58 +not. The estimated implicit prices are economically sensible: the living-area
59 +elasticity is 0.36, each additional bathroom is associated with a 7.6\% premium, and
60 +the age profile traces the familiar depreciation-then-heritage pattern. The
61 +municipality effects span a wide range---from roughly $+231$\% in Westmount to about
62 +$-30$\% in remote municipalities---and order the province's markets as expected.
63 +Benchmarking a ladder of estimators on the identical feature set, an ensemble blend
64 +of a robust embedding-based neural network and gradient boosting attains an
65 +out-of-sample $R^2$ of 0.779 with a median absolute prediction error of 14.0\%. We
66 +interpret all estimates as \emph{predictive} associations rather than causal
67 +effects, since listing content is chosen by sellers and correlates with unobserved
68 +determinants of value, and we discuss the asking-price nature of the dependent
69 +variable at length in Section~\ref{sec:robustness}.
70 +
71 +Beyond the specific Quebec application, the exercise speaks to a general measurement
72 +problem in housing economics. Text- and image-derived quality measures have until
73 +now required bespoke supervised models trained on labeled data
74 +\citep{nowak2017textual,shen2021information,johnson2020curb}. General-purpose
75 +foundation models with schema-constrained decoding lower the fixed cost of that
76 +measurement step to nearly zero, produce attributes that are human-readable rather
77 +than latent, and can be audited field by field. Our results suggest that this
78 +technology moves the effective characteristics frontier of hedonic analysis by
79 +enough to matter for appraisal, automated valuation, and assessment applications.
80 +
81 +The remainder of the paper is organized as follows. Section~\ref{sec:literature}
82 +reviews the related literature. Section~\ref{sec:data} describes the data sources
83 +and cleaning. Section~\ref{sec:methodology} presents the feature-engineering
84 +pipeline and the econometric specification. Section~\ref{sec:results} reports the
85 +main results. Section~\ref{sec:benchmark} maps the predictive frontier across
86 +estimators. Section~\ref{sec:robustness} discusses robustness and limitations.
87 +Section~\ref{sec:conclusion} concludes.
added paper/sections/literature.tex +74 −0
@@ -0,0 +1,74 @@
1 +% ============================================================================
2 +\section{Related Literature}
3 +\label{sec:literature}
4 +% ============================================================================
5 +
6 +This paper connects four strands of research.
7 +
8 +The first is the classical \textbf{hedonic pricing} literature. Building on
9 +\citet{lancaster1966new}, \citet{rosen1974hedonic} formalized the market for
10 +differentiated goods as a market for their underlying characteristics, providing the
11 +theoretical foundation for interpreting regression coefficients as implicit prices.
12 +The empirical housing literature that followed is vast; \citet{sirmans2005composition}
13 +survey the characteristics most commonly included in hedonic house-price models and
14 +document substantial heterogeneity in specification, \citet{malpezzi2003hedonic}
15 +provides a critical review of functional-form and identification choices, and
16 +\citet{hill2013hedonic} surveys the use of hedonic methods for house-price
17 +measurement. A recurring theme in this literature is omitted-variable bias from
18 +unmeasured quality: the state of repair, the standard of finishes, and the visual
19 +quality of a dwelling are priced by the market but rarely recorded in the structured
20 +fields available to the econometrician. Our contribution speaks directly to this
21 +measurement problem. Our log--linear specification, with logged continuous attributes
22 +and a dependent variable in logs, follows the mainstream of this literature.
23 +
24 +The second strand concerns \textbf{spatial structure, functional form, and
25 +prediction}. \citet{can1992spatial} and \citet{anselin1988spatial} emphasize that
26 +unobserved locational amenities induce spatial dependence that, if ignored, biases
27 +coefficient estimates; we absorb the first-order component of this problem through a
28 +rich set of municipality fixed effects and cluster inference at the same level.
29 +On the prediction side, \citet{bin2004prediction} compares parametric and
30 +semi-parametric hedonic predictors, \citet{mullainathan2017machine} frame
31 +machine-learning methods as estimators of the conditional expectation
32 +$\mathbb{E}[P\mid\mathbf{z}]$ well suited to prediction tasks, and
33 +\citet{kok2017bigdata} document the rise of automated valuation models in industry.
34 +This literature generally finds that flexible learners beat linear hedonics on
35 +accuracy at some cost in interpretability; our estimator ladder
36 +(Section~\ref{sec:benchmark}) reproduces that pattern on a common feature set, and
37 +our focus is on how much of the gap is closed not by a better \emph{estimator} but by
38 +better \emph{measurement}.
39 +
40 +The third strand is the use of \textbf{unstructured listing content}---text and
41 +images---in real-estate valuation. \citet{gentzkow2019text} survey text-as-data
42 +methods in economics. In real estate specifically, \citet{nowak2017textual} show that
43 +words in MLS remarks predict transaction prices and reduce hedonic pricing errors,
44 +and \citet{shen2021information} extract quality signals from property descriptions
45 +with supervised machine learning, finding that the text carries information about
46 +unobserved quality that is capitalized into prices. On the image side,
47 +\citet{ahmed2016house} combine visual and textual features for price estimation;
48 +\citet{poursaeed2018vision} show that learned measures of interior and exterior
49 +quality carry independent predictive power; \citet{glaeser2018vision} use street-level
50 +imagery of Boston houses to measure exterior quality and study its interaction with
51 +incentives; \citet{johnson2020curb} construct a curb-appeal index from Google Street
52 +View images and estimate its capitalization into sale prices; and
53 +\citet{lindenthal2021architecture} classify architectural styles from photographs and
54 +document style premia. Relative to this work, our approach differs in the
55 +\emph{technology of measurement}: rather than training bespoke supervised models on
56 +labeled data---which requires labels, task-specific engineering, and yields opaque
57 +scores---we use general-purpose large language and vision--language models with
58 +schema-constrained decoding to produce a fixed set of interpretable, human-readable
59 +attributes (e.g., ``renovated kitchen,'' ``brick façade,'' a 1--10 curb-appeal score)
60 +at a marginal cost of a fraction of a cent per listing, and we embed those attributes
61 +in a transparent hedonic regression whose coefficients can be read as implicit
62 +prices. The models never observe the price, so the extracted attributes cannot
63 +mechanically encode the outcome.
64 +
65 +The fourth strand is the \textbf{microstructure of listing markets}, which frames two
66 +institutional features of our data. Our sample pools a brokered MLS platform
67 +(Realtor.ca) with a for-sale-by-owner platform (DuProprio); \citet{hendel2009mls}
68 +compare exactly these two market forms and document differences in seller
69 +composition, marketing, and outcomes, motivating our platform indicator and
70 +platform-specific robustness checks. And because we observe asking rather than
71 +transaction prices, the interpretation of our dependent variable rests on the
72 +list-price literature: the asking price is a strategic choice that anchors search and
73 +bargaining \citep{horowitz1992list,knight2002listing,han2015microstructure}. We
74 +return to this limitation in Section~\ref{sec:robustness}.
added paper/sections/methodology.tex +241 −0
@@ -0,0 +1,241 @@
1 +% ============================================================================
2 +\section{Methodology}
3 +\label{sec:methodology}
4 +% ============================================================================
5 +
6 +Our empirical strategy has two parts: a feature-engineering pipeline that turns raw
7 +listings into an analysis-ready design matrix (Sections~\ref{subsec:structural}--%
8 +\ref{subsec:vision}), and a hedonic regression that maps those features to price
9 +(Section~\ref{subsec:econometrics}).
10 +
11 +\subsection{Theoretical Framework}
12 +\label{subsec:theory}
13 +
14 +Following \citet{rosen1974hedonic}, a house is a bundle of $K$ characteristics
15 +$\mathbf{z}=(z_1,\dots,z_K)$, and the market clears through a hedonic price function
16 +$P(\mathbf{z})$ that equates the marginal bid of buyers to the marginal offer of
17 +sellers. A buyer with income $y$ and preferences $U(\mathbf{x},\mathbf{z})$ over a
18 +numéraire good $\mathbf{x}$ and housing attributes chooses $\mathbf{z}$ to solve
19 +%
20 +\begin{equation}
21 + \max_{\mathbf{x},\,\mathbf{z}}\; U(\mathbf{x},\mathbf{z})
22 + \quad\text{s.t.}\quad y = \mathbf{x} + P(\mathbf{z}),
23 + \label{eq:consumer}
24 +\end{equation}
25 +%
26 +which yields the first-order condition that, at the optimum, the implicit
27 +(\emph{marginal}) price of each attribute equals the buyer's marginal willingness to
28 +pay:
29 +%
30 +\begin{equation}
31 + p_k(\mathbf{z}) \;\equiv\; \frac{\partial P(\mathbf{z})}{\partial z_k}
32 + \;=\; \frac{\partial U/\partial z_k}{\partial U/\partial x},
33 + \qquad k = 1,\dots,K.
34 + \label{eq:implicit}
35 +\end{equation}
36 +%
37 +Equation~\eqref{eq:implicit} is the object of interest: the slope of the hedonic
38 +surface in the direction of attribute~$z_k$. The empirical task is to recover
39 +$P(\mathbf{z})$---and hence its gradient---from observed prices and characteristics.
40 +The linear model of Section~\ref{subsec:econometrics} imposes a parametric form on
41 +$P(\cdot)$ and reads the implicit prices off its coefficients; the machine-learning
42 +estimators of Section~\ref{sec:benchmark} approximate $P(\cdot)$ nonparametrically,
43 +trading the closed-form gradient for predictive flexibility.
44 +
45 +\subsection{Structural and Locational Attributes}
46 +\label{subsec:structural}
47 +
48 +Structural attributes are extracted by deterministic rules. Living area and lot area
49 +are reported in heterogeneous units (\texttt{"1535 sqft"}, \texttt{"1,333.2 ft²"},
50 +\texttt{"11 561.52 ft²"}, \texttt{"961.6 m2"}); we parse each string and convert all
51 +areas to square feet. For Realtor.ca listings that report room-by-room dimensions
52 +(e.g.\ \texttt{"23 ft ,4 in x 13 ft ,8 in"}) but no aggregate living area, we sum the
53 +parsed room areas and calibrate the total against listings where both are observed;
54 +the median ratio of reported area to summed-room area is~0.98, so the reconstructed
55 +value is used directly to fill missing living area. Age is computed as
56 +$2026-\text{year built}$, entered together with its square to allow a non-monotone
57 +depreciation profile. Bedrooms, bathrooms, parking spaces, and storeys are taken
58 +from the attribute block. Missing values are imputed by the within-property-type
59 +median, and each imputed field is accompanied by a ``\texttt{has\_}'' indicator so
60 +that the regression can distinguish observed from imputed values.
61 +
62 +Location is captured by \textbf{municipality fixed effects}. City names are
63 +normalized (Realtor.ca appends neighborhood qualifiers in parentheses, which we
64 +strip), and municipalities with fewer than 20 listings are pooled into a residual
65 +category, leaving 195 city effects. Property-type fixed effects and a source
66 +indicator complete the locational and platform controls.
67 +
68 +\subsection{Rule-Based Amenities and Room Detail}
69 +\label{subsec:amenities}
70 +
71 +Seventeen amenity indicators are detected by bilingual (French/English) keyword
72 +matching over the concatenated feature list, description, and title: swimming pool,
73 +waterfront, garage, fireplace, central air conditioning or heat pump, finished
74 +basement, cul-de-sac, proximity to a golf course, ski hill, park, school, highway,
75 +and public transit, among others. From the Realtor.ca room block we additionally
76 +construct room-detail indicators---presence of a primary bedroom, a basement, an
77 +office, a family room, a workshop, a wine cellar, a sunroom, a mezzanine, a library,
78 +and a laundry room---together with the room and storey counts. Two listing-quality
79 +signals, the number of photographs (capped at~60) and the presence of a virtual
80 +tour, proxy for the marketing tier of the listing.
81 +
82 +\subsection{Language-Model Specifications from Text}
83 +\label{subsec:llm}
84 +
85 +Free-text descriptions contain valuation-relevant information that resists keyword
86 +extraction: whether the home is turnkey or a fixer-upper, whether the kitchen or
87 +bathroom has been renovated, the flooring and countertop materials, the heating
88 +system, the presence of a water view, and income potential. We extract these as a
89 +fixed set of structured specifications using a large language model
90 +(\texttt{gpt-4o-mini}) with \emph{schema-constrained decoding}: the request specifies
91 +a strict JSON schema with closed enumerations and required fields, so the API
92 +validates the output at generation time and the model returns exactly the expected
93 +fields in the expected value domains. Decoding temperature is set to zero for
94 +determinism, and the system instruction constrains the model to report only what the
95 +text supports, coding absent information as \texttt{false}/\texttt{unknown}/empty.
96 +Each call consumes roughly 600 input and 110 output tokens; the extraction is run in
97 +parallel, is resumable through an on-disk cache keyed by the listing identifier, and
98 +retries transient errors with exponential backoff, for a total cost of about \$3
99 +across the sample.
100 +
101 +\subsection{Vision Attributes from the Façade Photograph}
102 +\label{subsec:vision}
103 +
104 +To capture the visual quality that neither structured fields nor text convey, we
105 +analyze one photograph per listing---the façade---with a vision--language model
106 +(\texttt{claude-haiku-4-5}). For Realtor.ca the first image is the exterior shot; for
107 +DuProprio, whose images are captioned, we select the image whose caption is the
108 +address rather than a room. The model returns twelve objective attributes under a
109 +strict JSON schema: an overall condition score (1--10), a curb-appeal score (1--10),
110 +a renovation level, exterior materials, landscaping quality, visible garage, storeys,
111 +setting (urban/suburban/rural/waterfront/wooded), a luxury indicator, and photo
112 +quality. The system instruction directs the model to ground each score in concrete,
113 +reproducible criteria (surface condition, materials, landscaping) rather than
114 +subjective taste, and to flag non-exterior images. Images are passed by URL without
115 +local download; each call consumes roughly 2{,}470 input and 100 output tokens, for a
116 +total cost of about \$57 across the sample. As with the text layer, the pass is
117 +parallelized, resumable, and retried.
118 +
119 +\paragraph{Why schema-constrained decoding matters.}
120 +Imposing a typed schema with enumerations, required fields, and
121 +\texttt{additionalProperties:false} moves validation into the API layer. The models
122 +are forced to emit exactly the expected fields within the expected value domains, so
123 +their outputs behave as ordinary tabulated variables---reproducible and free of
124 +free-text post-processing. The outputs are then mapped to numeric regressors:
125 +booleans to $0/1$, the ordinal condition scale to $0$--$5$, materials to indicator
126 +variables, and so on, each with an availability flag.
127 +
128 +\subsection{Econometric Specification}
129 +\label{subsec:econometrics}
130 +
131 +We adopt a semi-logarithmic functional form, the standard workhorse in the hedonic
132 +literature \citep{malpezzi2003hedonic}. Collect the attributes of listing~$i$ into
133 +disjoint blocks,
134 +%
135 +\begin{equation}
136 + \resizebox{.94\textwidth}{!}{$\displaystyle
137 + \mathbf{X}_i =
138 + \big(\underbrace{\ln\text{Area}_i,\ \ln\text{Lot}_i}_{\text{size (logged)}},\
139 + \underbrace{\mathbf{s}_i}_{\text{structural}},\
140 + \underbrace{\mathbf{a}_i}_{\text{amenities}},\
141 + \underbrace{\mathbf{r}_i}_{\text{rooms}},\
142 + \underbrace{\mathbf{m}_i}_{\text{listing}},\
143 + \underbrace{\mathbf{g}_i}_{\text{LLM text}},\
144 + \underbrace{\mathbf{v}_i}_{\text{vision}}\big)
145 + $}
146 + \label{eq:blocks}
147 +\end{equation}
148 +%
149 +and estimate, by ordinary least squares,
150 +%
151 +\begin{equation}
152 + \ln P_i \;=\; \alpha
153 + \;+\; \beta\,\ln\text{Area}_i
154 + \;+\; \gamma\,\ln\text{Lot}_i
155 + \;+\; \mathbf{X}_i^{\!\top}\bm{\delta}
156 + \;+\; \underbrace{\mu_{c(i)}}_{\text{city FE}}
157 + \;+\; \underbrace{\tau_{t(i)}}_{\text{type FE}}
158 + \;+\; \varepsilon_i ,
159 + \label{eq:hedonic}
160 +\end{equation}
161 +%
162 +where $c(i)$ and $t(i)$ index the municipality and property type of listing~$i$, and
163 +$\mu_c$, $\tau_t$ are the corresponding fixed effects that sweep out
164 +location- and type-specific price levels. In matrix form
165 +$\mathbf{y}=\mathbf{Z}\bm{\theta}+\bm{\varepsilon}$ with
166 +$\widehat{\bm{\theta}}=(\mathbf{Z}^{\!\top}\mathbf{Z})^{-1}\mathbf{Z}^{\!\top}\mathbf{y}$.
167 +
168 +\paragraph{Interpretation of the coefficients.}
169 +Because the size variables enter in logs, $\beta$ and $\gamma$ are \emph{elasticities}:
170 +$\partial \ln P/\partial \ln\text{Area}=\beta$. For an indicator regressor $d$ with
171 +coefficient $b$, the exact percentage price effect is not $b$ but
172 +%
173 +\begin{equation}
174 + \%\Delta P \;=\; \big(e^{\,b}-1\big)\times 100,
175 + \label{eq:halvorsen}
176 +\end{equation}
177 +%
178 +the \citet{halvorsen1980interpretation} correction we apply throughout (for small
179 +$b$, $e^{b}-1\approx b$). Under Equation~\eqref{eq:hedonic} the implicit price of a
180 +continuous attribute $z_k$ from \eqref{eq:implicit} is $p_k=\delta_k\,P/z_k$, so the
181 +semi-log form lets the \emph{dollar} marginal price scale with the price level while
182 +keeping the \emph{percentage} effect constant.
183 +
184 +\paragraph{Two-part coverage design.}
185 +Because living area is reported for only about 38\% of Realtor.ca listings, we
186 +estimate two nested versions of \eqref{eq:hedonic} and combine their predictions:
187 +%
188 +\begin{equation}
189 + \widehat{\ln P}_i =
190 + \begin{cases}
191 + \widehat{\ln P}_i^{\,A} & \text{if area is measured (Model A, rich),}\\[2pt]
192 + \widehat{\ln P}_i^{\,B} & \text{otherwise (Model B, broad; }\beta\equiv 0).
193 + \end{cases}
194 + \label{eq:twopart}
195 +\end{equation}
196 +%
197 +Model~A includes $\ln\text{Area}$ and is fit on the measured-area subsample; Model~B
198 +drops it and is fit on the full cleaned sample, so every listing can be priced.
199 +
200 +\paragraph{Inference.}
201 +House-price residuals are strongly heteroskedastic, and listings within a
202 +municipality share unobserved local shocks that induce within-cluster error
203 +correlation. Our preferred standard errors are therefore clustered at the
204 +municipality level (195 clusters), the same level as the fixed effects
205 +\citep{cameron2015cluster}:
206 +%
207 +\begin{equation}
208 + \widehat{\mathrm{Var}}_{\text{CL}}(\widehat{\bm{\theta}})
209 + = (\mathbf{Z}^{\!\top}\mathbf{Z})^{-1}
210 + \Bigg(\sum_{c=1}^{C}
211 + \mathbf{Z}_c^{\!\top}\hat{\bm{\varepsilon}}_c
212 + \hat{\bm{\varepsilon}}_c^{\!\top}\mathbf{Z}_c\Bigg)
213 + (\mathbf{Z}^{\!\top}\mathbf{Z})^{-1},
214 + \label{eq:cluster}
215 +\end{equation}
216 +%
217 +where $\mathbf{Z}_c$ and $\hat{\bm{\varepsilon}}_c$ stack the regressors and
218 +residuals of cluster $c$. As a complement we also compute
219 +heteroskedasticity-consistent (HC3) standard errors
220 +\citep{white1980heteroskedasticity,mackinnon1985some}, which inflate the
221 +contribution of high-leverage points; Section~\ref{subsec:robust} shows that the two
222 +choices agree on the significance of the property-level AI attributes and differ
223 +only for keyword amenities with a strong spatial component.
224 +
225 +\paragraph{Retransformation.}
226 +Predicting the \emph{level} of price requires undoing the log. Because
227 +$\mathbb{E}[P\mid\mathbf{Z}]=\exp(\mathbf{Z}\bm{\theta})\,\mathbb{E}[e^{\varepsilon}]
228 +\neq \exp(\mathbf{Z}\bm{\theta})$, naive exponentiation is biased downward. Under
229 +$\varepsilon\sim\mathcal N(0,\sigma^2)$ we use the closed-form correction, and more
230 +generally Duan's nonparametric smearing estimate \citep{duan1983smearing}:
231 +%
232 +\begin{equation}
233 + \widehat{P}_i
234 + = \exp\!\big(\widehat{\ln P}_i\big)\cdot
235 + \underbrace{\exp\!\big(\tfrac12\hat\sigma^{2}\big)}_{\text{normal}}
236 + \qquad\text{or}\qquad
237 + \widehat{P}_i
238 + = \exp\!\big(\widehat{\ln P}_i\big)\cdot
239 + \underbrace{\frac1n\sum_{j=1}^{n} e^{\hat\varepsilon_j}}_{\text{smearing }\hat\phi}.
240 + \label{eq:smearing}
241 +\end{equation}
added paper/sections/results.tex +105 −0
@@ -0,0 +1,105 @@
1 +% ============================================================================
2 +\section{Results}
3 +\label{sec:results}
4 +% ============================================================================
5 +
6 +\subsection{Baseline Estimates}
7 +\label{subsec:baseline}
8 +
9 +Table~\ref{tab:regression} reports the two hedonic specifications estimated on the
10 +full feature set (structural, amenity, room-detail, listing-signal, LLM text-derived,
11 +and vision-derived attributes). Model~A, which includes living area and is estimated
12 +on the 9{,}446 listings with a measured area, explains 75.5\% of the variation in log
13 +prices ($R^2=0.755$, adjusted $0.748$). Model~B, the area-free specification estimated
14 +on all 19{,}123 listings, explains 72.5\% ($R^2=0.725$, adjusted $0.721$). Adding the
15 +machine-generated feature layers raises Model~A's $R^2$ from 0.717 (structural,
16 +amenity, room, and listing attributes only) to 0.725 with the text layer and 0.755
17 +with the vision layer---a cumulative gain of nearly four points from the two
18 +AI-generated feature sets, over and above an already rich hedonic specification.
19 +
20 +The estimated implicit prices are economically sensible and, with few exceptions,
21 +statistically significant with standard errors clustered by municipality. In
22 +Model~A, the living-area elasticity is 0.36 (a 10\% larger home commands roughly a
23 +3.6\% higher price) and the lot-area elasticity is small but precisely estimated.
24 +Each additional \textbf{bathroom} is associated with a 7.6\% premium and each parking
25 +space with about 1\%. \textbf{Central air conditioning} carries a 5.5\% premium and
26 +\textbf{public-transit} proximity 9.4\%, although the transit coefficient---like
27 +other keyword amenities with a strong spatial component---is estimated much less
28 +precisely once errors are allowed to correlate within municipalities.
29 +\textbf{Waterfront} access is associated with a 6.0--6.4\% premium, significant at
30 +conventional levels in the full-sample model. Age enters with a small negative linear
31 +term and a positive quadratic, tracing the familiar depreciation-then-heritage
32 +profile.
33 +
34 +\input{tables/regression}
35 +
36 +\subsection{Text- and Vision-Derived Attributes}
37 +\label{subsec:ai_coeffs}
38 +
39 +The two AI feature layers contribute distinct, largely orthogonal signal.
40 +
41 +\paragraph{Text (LLM).} Beyond the keyword amenities, the structured specifications
42 +carry independent information about finish and condition: a \textbf{renovated
43 +kitchen} is associated with a 2.6\% premium, \textbf{hardwood flooring} with 2.4\%,
44 +and the ordinal \textbf{condition} scale is positive and significant in both models
45 +(1.6\% per step). Listings the model reads as \textbf{income properties} sell about
46 +1.9\% below otherwise-comparable houses. The \textbf{new-construction} indicator,
47 +by contrast, is indistinguishable from zero once age, the visual renovation level,
48 +and the condition scales are controlled for---the premium a naive specification
49 +would attribute to the label ``new'' is fully absorbed by the measured quality
50 +attributes.
51 +
52 +\paragraph{Vision (façade).} The image-derived attributes are jointly significant and
53 +add the largest single increment to fit. Holding all structural, amenity, and text
54 +attributes fixed, a one-point increase in \textbf{curb appeal} (1--10) is associated
55 +with a 2.3\% higher price, and a one-step increase in the visual \textbf{renovation
56 +level} with 2.4\%. A visible \textbf{luxury} cue adds 5.2\%, a \textbf{visible garage}
57 +4.7\%, and each step of \textbf{landscaping} quality 1.5\%. Exterior materials price
58 +as expected: \textbf{brick} ($+3.4$\%) and \textbf{stone} ($+3.7$\%) command premia,
59 +while \textbf{vinyl} siding carries a 4.4\% discount. The visual \textbf{rural}
60 +setting is associated with an 11.5\% discount relative to the omitted category. That
61 +these visual measures remain significant after conditioning on the full text and
62 +structural set---and with inference clustered at the municipality level---confirms
63 +that the façade photograph carries price-relevant information available in no other
64 +field.
65 +
66 +\subsection{Validation of Machine-Generated Features}
67 +\label{subsec:validation}
68 +
69 +Before relying on the AI-generated attributes, we check that they order prices in the
70 +expected direction. Figure~\ref{fig:validation} plots the median list price against the
71 +vision model's curb-appeal and condition scores among listings with a detected
72 +exterior shot. Both relationships are strongly monotone: median price rises steadily
73 +from the lowest to the highest visual scores, with no reliance on the regression to
74 +impose the pattern. Because these scores are produced by a model that never sees the
75 +price, the monotonicity is independent evidence that the vision layer extracts a real,
76 +price-relevant quality signal rather than noise. Table~\ref{tab:ai_summary} reports the
77 +prevalence of the main text- and vision-derived attributes.
78 +
79 +\begin{figure}[htbp]
80 + \centering
81 + \includegraphics[width=\textwidth]{figures/fig_validation.pdf}
82 + \caption{Validation of the vision layer. Median list price rises monotonically with
83 + the model's curb-appeal (left) and condition (right) scores, among listings with a
84 + detected façade shot. The scores are generated without access to price.}
85 + \label{fig:validation}
86 +\end{figure}
87 +
88 +\subsection{Location Fixed Effects}
89 +\label{subsec:location}
90 +
91 +The 195 municipality fixed effects absorb the dominant locational component of price.
92 +Relative to the reference city, the largest positive premia are estimated for
93 +\textbf{Westmount} ($\approx+231$\%), \textbf{Hampstead} ($+173$\%), and
94 +\textbf{Mont-Royal} ($+169$\%), followed by outlying municipalities at the low end
95 +carrying discounts of 23--30\%. Figure~\ref{fig:premia} plots the extreme effects. The
96 +magnitude and ordering align with well-known patterns in the Quebec housing market and
97 +lend face validity to the specification.
98 +
99 +\begin{figure}[htbp]
100 + \centering
101 + \includegraphics[width=.74\textwidth]{figures/fig_premia.pdf}
102 + \caption{Estimated municipality fixed effects (Model~A), the twelve largest and eight
103 + smallest, expressed as percentage premia relative to the reference city.}
104 + \label{fig:premia}
105 +\end{figure}
added paper/sections/robustness.tex +99 −0
@@ -0,0 +1,99 @@
1 +% ============================================================================
2 +\section{Robustness and Limitations}
3 +\label{sec:robustness}
4 +% ============================================================================
5 +
6 +\subsection{Robustness}
7 +\label{subsec:robust}
8 +
9 +Several features of the design guard against spurious results. The dual
10 +specification (Section~\ref{subsec:econometrics}) ensures that estimates are not
11 +driven by the non-random availability of living area: coefficients on attributes
12 +common to both models retain their sign and rough magnitude whether or not area is
13 +conditioned on. The ``\texttt{has\_}'' indicators accompanying each imputed field let
14 +the regression separate observed from imputed information, so median imputation does
15 +not silently attenuate the size coefficients. The price-per-square-foot trim removes
16 +the residual parsing errors that would otherwise dominate the tails, and the
17 +municipality fixed effects absorb the first-order spatial confound emphasized by
18 +\citet{can1992spatial} and \citet{anselin1988spatial}.
19 +
20 +\paragraph{Inference.} Because listings within a municipality share unobserved
21 +local shocks, our preferred standard errors are clustered at the municipality level
22 +(195 clusters; \citealp{cameron2015cluster}), the same level as the fixed effects.
23 +Clustering matters exactly where theory predicts: the text- and vision-derived
24 +quality attributes---which vary house by house within a municipality---are
25 +essentially unaffected (curb appeal, renovation level, luxury, garage, materials,
26 +landscaping, and the rural-setting discount all remain significant at the 5\% level
27 +or better in Model~A, as do the LLM condition scale, renovated kitchen, and hardwood
28 +indicators), whereas keyword amenities with a strong spatial component lose
29 +precision (public-transit proximity falls to the 5\% boundary in Model~A and is
30 +insignificant in Model~B; central air conditioning is insignificant in Model~B; the
31 +curb-appeal score, precisely estimated in Model~A, loses significance in the broader
32 +Model~B).
33 +Heteroskedasticity-robust (HC3) standard errors
34 +\citep{white1980heteroskedasticity,mackinnon1985some} yield the same qualitative
35 +pattern with uniformly smaller intervals. This asymmetry is reassuring for our main
36 +claim: the AI-generated attributes capture within-market, property-level quality
37 +variation rather than repackaged location effects.
38 +
39 +Table~\ref{tab:robustness} probes the stability of the predictive blend across
40 +subsamples. Three patterns stand out. First, accuracy is comparable whether living
41 +area is observed (median error 13.5\%) or imputed (14.4\%), confirming that the
42 +area-free model prices the incomplete listings almost as well as the rich model.
43 +Second, the model fits the broker-listed Realtor.ca inventory better
44 +($R^2_{\text{OOS}}=0.80$) than the smaller, noisier for-sale-by-owner DuProprio
45 +segment (0.58), though median errors are identical at 14.0\%; the gap is consistent
46 +with the compositional and marketing differences between brokered and FSBO
47 +platforms documented by \citet{hendel2009mls}. Third, the median error is stable
48 +across price terciles (11--16\%), lowest in the mid-market and modestly higher in
49 +the tails, where idiosyncratic and luxury attributes are harder to capture. (Within
50 +a single tercile $R^2$ is uninformative because the price range is restricted; the
51 +median error is the comparable metric.)
52 +
53 +\input{tables/robustness}
54 +
55 +\subsection{Limitations}
56 +\label{subsec:limits}
57 +
58 +Five limitations qualify the interpretation of our estimates.
59 +
60 +\emph{Asking prices.} The dependent variable is the asking price of properties
61 +listed for sale, not the transaction price. The list price is a strategic choice
62 +that anchors buyer search and bargaining
63 +\citep{horowitz1992list,knight2002listing,han2015microstructure}: it is set with
64 +sale in mind and is highly correlated with the eventual transaction price, but
65 +list-to-sale ratios vary systematically with market conditions and overpricing is
66 +more common in thin segments. Two implications follow. First, our implicit prices
67 +measure how sellers and their agents \emph{price} attributes into the ask, which
68 +converges to buyers' valuation only insofar as the market disciplines the ask; the
69 +economically sensible magnitudes and the monotone validation of
70 +Section~\ref{subsec:validation} suggest the discipline is substantial. Second, any
71 +attribute that shifts the list-to-sale ratio itself (e.g., attributes of hard-to-sell
72 +properties) will carry a coefficient that mixes valuation and listing strategy.
73 +Replicating the analysis on registry transaction prices is the natural next step.
74 +
75 +\emph{Non-causality and content endogeneity.} Descriptions and photographs are
76 +marketing instruments chosen by sellers and agents. Richer descriptions, more
77 +photographs, and higher visual quality correlate with higher-end listing agents and
78 +more valuable properties \citep{nowak2017textual}. The coefficients are therefore
79 +\emph{predictive} associations, not causal effects; they should not be read as the
80 +price change from physically adding a given attribute. We control for two observable
81 +dimensions of marketing effort (photo count and virtual tours), but cannot rule out
82 +that residual marketing quality loads on the AI-derived attributes.
83 +
84 +\emph{Machine-generated features.} LLM and vision outputs can contain errors.
85 +Zero-temperature decoding, strict schemas, and closed enumerations bound this risk
86 +and make it reproducible, and classical measurement error in the extracted scores
87 +would bias their coefficients toward zero---making our estimates conservative---but
88 +we do not perform systematic human validation, and the visual layer judges each
89 +property from a single façade photograph, ignoring interiors.
90 +
91 +\emph{Selection.} Listed properties are not a random sample of the housing stock,
92 +and listings with missing area or room detail are not missing at random (they tend
93 +to be less complete listings). The ``\texttt{has\_}'' indicators and the two-part
94 +design mitigate, but do not eliminate, the second concern.
95 +
96 +\emph{Cross-section.} The data are a single-date snapshot, so the model captures no
97 +market dynamics and the location premia are fixed as of 3~July~2026. Municipalities
98 +with fewer than 20 listings are pooled, reducing spatial resolution where data are
99 +sparse.
added paper/sections/titlepage.tex +77 −0
@@ -0,0 +1,77 @@
1 +% ============================================================================
2 +% Title Page
3 +% ============================================================================
4 +\thispagestyle{empty}
5 +
6 +\begin{center}
7 +
8 +% --- Logo ---
9 +\includegraphics[width=3.5cm]{uq_logo.jpg}
10 +
11 +\vspace{0.6cm}
12 +
13 +{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm]
14 +{\footnotesize\textsc{Département des sciences administratives}}
15 +
16 +\vspace{0.8cm}
17 +
18 +{\footnotesize\textsc{Working Paper No.~\WPnumber}}
19 +
20 +\vspace{1.2cm}
21 +
22 +% --- Title ---
23 +{\LARGE\bfseries \WPtitle\par}
24 +
25 +\ifx\WPsubtitle\empty\else
26 + \vspace{0.3cm}
27 + {\large\itshape \WPsubtitle\par}
28 +\fi
29 +
30 +\vspace{1.2cm}
31 +
32 +% --- Author ---
33 +{\large \WPauthor}\\[0.3cm]
34 +{\normalsize \WPaffiliation}\\[0.15cm]
35 +{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm]
36 +{\small \WPaddress}
37 +
38 +\vspace{0.8cm}
39 +
40 +% --- Date & Version ---
41 +{\normalsize \WPdate}\\[0.1cm]
42 +{\small Version~\WPversion}
43 +
44 +\end{center}
45 +
46 +\vfill
47 +
48 +\newpage
49 +
50 +% ============================================================================
51 +% Abstract Page
52 +% ============================================================================
53 +\thispagestyle{empty}
54 +
55 +\vspace*{1cm}
56 +
57 +\noindent\rule{\textwidth}{0.4pt}
58 +\vspace{0.3cm}
59 +
60 +\noindent\textbf{Abstract}
61 +
62 +\vspace{0.15cm}
63 +
64 +\noindent\WPabstract
65 +
66 +\vspace{0.4cm}
67 +
68 +\noindent\textbf{Keywords:} \WPkeywords
69 +
70 +\vspace{0.15cm}
71 +
72 +\noindent\textbf{JEL Classification:} \WPjel
73 +
74 +\vspace{0.3cm}
75 +\noindent\rule{\textwidth}{0.4pt}
76 +
77 +\newpage
added paper/tables/ablation.tex +27 −0
@@ -0,0 +1,27 @@
1 +% Auto-generated (revision: recomputed on identical folds)
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Feature Ablation: Out-of-Sample $R^2$ by Feature Layer}
6 + \label{tab:ablation}
7 + \begin{tabular}{@{}lcccc@{}}
8 + \toprule
9 + Feature set & \# feats & OLS & GBM & Blend \\
10 + \midrule
11 + Structural + amenity & 24 & 0.521 & 0.692 & --- \\
12 + \quad + Rooms + listing & 38 & 0.619 & 0.729 & --- \\
13 + \quad + Text (LLM) & 56 & 0.628 & 0.743 & 0.760 \\
14 + \quad + Vision (façade) & 77 & 0.686 & 0.770 & 0.779 \\
15 + \bottomrule
16 + \end{tabular}
17 + \begin{tablenotes}[flushleft]
18 + \footnotesize
19 + \item \textit{Notes:} Each row adds a feature layer to the row above (nested).
20 + Entries are out-of-sample $R^2$ on $\ln(\text{Price})$ from 5-fold cross-validation
21 + with a common seed. ``\# feats'' counts numeric/indicator features before the city,
22 + type, and source fixed effects (added in every specification). The blend averages
23 + the robust neural network and gradient boosting and is reported where both were
24 + trained. GBM handles missing living area natively; OLS uses median imputation.
25 + \end{tablenotes}
26 + \end{threeparttable}
27 +\end{table}
added paper/tables/ai_summary.tex +28 −0
@@ -0,0 +1,28 @@
1 +% Auto-generated
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Machine-Generated Feature Summary}
6 + \label{tab:ai_summary}
7 + \begin{tabular}{@{}lc@{\hskip 1.2cm}lc@{}}
8 + \toprule
9 + Text (LLM) & Share & Vision (façade) & Mean / share \\
10 + \midrule
11 + Recently renovated & 26\% & Condition score (1--10) & 7.4 \\
12 + Renovated kitchen & 17\% & Curb appeal (1--10) & 7.2 \\
13 + Hardwood floor & 16\% & Luxury cue & 43\% \\
14 + Quartz/granite counter & 7\% & Garage visible & 46\% \\
15 + New construction & 14\% & Brick façade & 47\% \\
16 + Income property & 12\% & Waterfront setting & 7\% \\
17 + Water view & 24\% & Exterior shot detected & 96\% \\
18 + \bottomrule
19 + \end{tabular}
20 + \begin{tablenotes}[flushleft]
21 + \footnotesize
22 + \item \textit{Notes:} Prevalence of selected machine-generated attributes across
23 + the 19{,}123 cleaned listings. Text attributes are extracted from the
24 + description by \texttt{gpt-4o-mini}; vision attributes from the façade photograph by
25 + \texttt{claude-haiku-4-5}. Scores are 1--10 means; the remainder are shares.
26 + \end{tablenotes}
27 + \end{threeparttable}
28 +\end{table}
added paper/tables/models.tex +39 −0
@@ -0,0 +1,39 @@
1 +% Auto-generated (revision: full ladder in both panels)
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Predictive Performance: From OLS to Deep Learning}
6 + \label{tab:models}
7 + \begin{tabular}{@{}lcccc@{}}
8 + \toprule
9 + Estimator & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% & Within $\pm$25\% \\
10 + \midrule
11 + \multicolumn{5}{@{}l}{\textit{Panel A: Structural + amenity + LLM-text features}} \\[3pt]
12 + OLS (linear hedonic) & 0.628 & 17.9\% & 43\% & 64\% \\
13 + Random Forest & 0.677 & 17.9\% & 43\% & 64\% \\
14 + Neural network (plain MLP) & 0.711 & 16.1\% & 47\% & 69\% \\
15 + Neural network (robust ResNet) & 0.741 & 15.2\% & 49\% & 70\% \\
16 + Gradient boosting (HistGBM) & 0.743 & 15.4\% & 49\% & 70\% \\
17 + Blend (NN + GBM) & 0.760 & 14.7\% & 51\% & 72\% \\[4pt]
18 + \multicolumn{5}{@{}l}{\textit{Panel B: \ + Vision (façade) features}} \\[3pt]
19 + OLS (linear hedonic) & 0.686 & 16.9\% & 45\% & 67\% \\
20 + Random Forest & 0.704 & 17.2\% & 45\% & 66\% \\
21 + Neural network (robust ResNet) & 0.747 & 14.8\% & 51\% & 71\% \\
22 + Gradient boosting (HistGBM) & 0.770 & 14.5\% & 51\% & 71\% \\
23 + \textbf{Blend (NN + GBM)} & \textbf{0.779} & \textbf{14.0\%} & \textbf{53\%} & \textbf{73\%} \\
24 + \bottomrule
25 + \end{tabular}
26 + \begin{tablenotes}[flushleft]
27 + \footnotesize
28 + \item \textit{Notes:} Out-of-sample performance from 5-fold cross-validation with a
29 + common seed (identical folds across rows). $R^2_{\text{OOS}}$ is on
30 + $\ln(\text{Price})$; ``Median AE'' is the median absolute percentage error in
31 + levels (Equation~\ref{eq:cvmetrics}). Panel~A uses structural, amenity, room-detail,
32 + listing-signal, and LLM text-derived features; Panel~B adds the twelve façade
33 + vision attributes. City, type, and source enter as one-hot indicators for the
34 + linear and tree models and as learned embeddings for the neural networks. The tree
35 + and neural models handle missing living area natively; OLS uses within-type median
36 + imputation. The best model is the text\,+\,vision blend.
37 + \end{tablenotes}
38 + \end{threeparttable}
39 +\end{table}
added paper/tables/regression.tex +82 −0
@@ -0,0 +1,82 @@
1 +% Auto-generated (revision: clustered SEs)
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Hedonic Regression Results (Text + Vision Features)}
6 + \label{tab:regression}
7 + \begin{tabular}{@{}lcc@{}}
8 + \toprule
9 + & (1) Model~A & (2) Model~B \\
10 + & \textit{rich} & \textit{broad} \\
11 + \midrule
12 + $\ln(\text{Living area})$ & 0.358$^{***}$ & \\
13 + & (0.012) & \\[3pt]
14 + $\ln(\text{Lot area})$ & 0.027$^{***}$ & 0.040$^{***}$ \\
15 + & (0.006) & (0.005) \\[3pt]
16 + Bedrooms & 0.014$^{***}$ & 0.032$^{***}$ \\
17 + & (0.005) & (0.003) \\[3pt]
18 + Bathrooms & 0.073$^{***}$ & 0.113$^{***}$ \\
19 + & (0.007) & (0.007) \\[3pt]
20 + Parking & 0.011$^{***}$ & 0.015$^{***}$ \\
21 + & (0.001) & (0.002) \\[3pt]
22 + Age & $-$0.004$^{***}$ & $-$0.005$^{***}$ \\
23 + & (0.001) & (0.001) \\[3pt]
24 + \multicolumn{3}{@{}l}{\textit{Amenities \& text (LLM)}} \\[2pt]
25 + Central A/C & 0.053$^{***}$ & 0.020 \\
26 + & (0.018) & (0.018) \\[3pt]
27 + Public transit & 0.090$^{**}$ & 0.058 \\
28 + & (0.046) & (0.039) \\[3pt]
29 + Waterfront & 0.058$^{*}$ & 0.062$^{***}$ \\
30 + & (0.033) & (0.019) \\[3pt]
31 + Condition (LLM, ord.) & 0.016$^{***}$ & 0.016$^{***}$ \\
32 + & (0.006) & (0.005) \\[3pt]
33 + Reno.\ kitchen (LLM) & 0.026$^{***}$ & 0.027$^{***}$ \\
34 + & (0.007) & (0.010) \\[3pt]
35 + Hardwood floor (LLM) & 0.024$^{**}$ & 0.030$^{***}$ \\
36 + & (0.009) & (0.011) \\[3pt]
37 + New construction (LLM) & $-$0.000 & $-$0.016 \\
38 + & (0.013) & (0.014) \\[3pt]
39 + Income property (LLM) & $-$0.019$^{**}$ & $-$0.013$^{*}$ \\
40 + & (0.009) & (0.008) \\[3pt]
41 + \multicolumn{3}{@{}l}{\textit{Vision (façade)}} \\[2pt]
42 + Curb appeal (1--10) & 0.023$^{***}$ & 0.016 \\
43 + & (0.009) & (0.013) \\[3pt]
44 + Renovation level (ord.) & 0.024$^{**}$ & 0.020$^{**}$ \\
45 + & (0.010) & (0.009) \\[3pt]
46 + Luxury (visual) & 0.051$^{***}$ & 0.062$^{***}$ \\
47 + & (0.012) & (0.009) \\[3pt]
48 + Garage visible & 0.046$^{***}$ & 0.047$^{***}$ \\
49 + & (0.006) & (0.007) \\[3pt]
50 + Brick façade & 0.033$^{***}$ & 0.044$^{***}$ \\
51 + & (0.009) & (0.011) \\[3pt]
52 + Stone façade & 0.036$^{***}$ & 0.047$^{***}$ \\
53 + & (0.008) & (0.005) \\[3pt]
54 + Vinyl façade & $-$0.045$^{***}$ & $-$0.062$^{***}$ \\
55 + & (0.015) & (0.012) \\[3pt]
56 + Rural setting & $-$0.122$^{***}$ & $-$0.076$^{***}$ \\
57 + & (0.032) & (0.029) \\[3pt]
58 + \midrule
59 + City FE / Type FE & Yes & Yes \\
60 + Living area & Yes & No \\
61 + Observations & 9{,}446 & 19{,}123 \\
62 + $R^2$ & 0.755 & 0.725 \\
63 + Adjusted $R^2$ & 0.748 & 0.721 \\
64 + \bottomrule
65 + \end{tabular}
66 + \begin{tablenotes}[flushleft]
67 + \footnotesize
68 + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full
69 + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample);
70 + Model~B omits it (full sample). ``LLM'' variables are extracted from the
71 + description text; ``Vision'' variables from the façade photograph. Curb appeal and
72 + renovation level enter as scores; luxury, garage, and material/setting are
73 + indicators. All specifications include city and property-type fixed effects and the
74 + full amenity, room-detail, and listing-signal controls (only selected coefficients
75 + shown). Standard errors clustered by municipality (195 clusters) in parentheses.
76 + Heteroskedasticity-robust (HC3) standard errors are uniformly smaller and yield
77 + the same significance pattern for the text- and vision-derived attributes, except
78 + that curb appeal in Model~B is significant under HC3 but not under clustering.
79 + $^{***}$, $^{**}$, $^{*}$: significance at 1\%, 5\%, 10\%.
80 + \end{tablenotes}
81 + \end{threeparttable}
82 +\end{table}
added paper/tables/robustness.tex +34 −0
@@ -0,0 +1,34 @@
1 +% Auto-generated
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Robustness: Blend Performance Across Subsamples}
6 + \label{tab:robustness}
7 + \begin{tabular}{@{}lcccc@{}}
8 + \toprule
9 + Subsample & $N$ & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% \\
10 + \midrule
11 + \textit{Full sample} & 19{,}123 & 0.779 & 14.0\% & 53\% \\
12 + \quad Price tercile 1 (low) & 6{,}375 & n/a$^{\dagger}$ & 15.6\% & 49\% \\
13 + \quad Price tercile 2 (mid) & 6{,}412 & n/a$^{\dagger}$ & 11.0\% & 64\% \\
14 + \quad Price tercile 3 (high) & 6{,}336 & n/a$^{\dagger}$ & 16.4\% & 47\% \\
15 + \quad Realtor.ca & 15{,}356 & 0.802 & 14.0\% & 53\% \\
16 + \quad DuProprio & 3{,}767 & 0.579 & 14.0\% & 54\% \\
17 + \quad Measured area & 9{,}446 & 0.773 & 13.5\% & 55\% \\
18 + \quad Imputed area & 9{,}677 & 0.783 & 14.4\% & 52\% \\
19 + \bottomrule
20 + \end{tabular}
21 + \begin{tablenotes}[flushleft]
22 + \footnotesize
23 + \item \textit{Notes:} Out-of-sample performance of the text\,+\,vision blend
24 + (Table~\ref{tab:models}, Panel~B) computed on subsamples of the same 5-fold
25 + cross-validated predictions. Price terciles are defined on the actual list price.
26 + ``Measured area'' are listings with a reported or room-reconstructed living area
27 + (priced by Model~A); ``Imputed area'' listings are priced by the area-free model.
28 + $^{\dagger}$Within a single price tercile the range of $\ln P$ is heavily
29 + restricted, so $R^2$---which normalizes by within-group variance---is
30 + uninformative; the median absolute error is the comparable accuracy metric across
31 + terciles.
32 + \end{tablenotes}
33 + \end{threeparttable}
34 +\end{table}
added paper/tables/summary_stats.tex +28 −0
@@ -0,0 +1,28 @@
1 +% Auto-generated
2 +\begin{table}[htbp]
3 + \centering
4 + \begin{threeparttable}
5 + \caption{Summary Statistics}
6 + \label{tab:summary_stats}
7 + \begin{tabular}{@{}lcccccc@{}}
8 + \toprule
9 + Variable & $N$ & Mean & Std.\ Dev. & $p_{25}$ & Median & $p_{75}$ \\
10 + \midrule
11 + Price (CAD) & 19{,}123 & 764{{,}}708 & 689{{,}}118 & 439{{,}}000 & 599{{,}}900 & 849{{,}}000 \\
12 + Living area (ft$^2$) & 9{,}446 & 1{{,}}730 & 990 & 1{{,}}128 & 1{{,}}470 & 2{{,}}026 \\
13 + Lot area (ft$^2$) & 19{,}123 & 71{{,}}429 & 309{{,}}209 & 6{{,}}000 & 10{{,}}686 & 34{{,}}978 \\
14 + Bedrooms & 19{,}123 & 3.5 & 1.2 & 3.0 & 3.0 & 4.0 \\
15 + Bathrooms & 19{,}123 & 1.8 & 0.8 & 1.0 & 2.0 & 2.0 \\
16 + Parking spaces & 19{,}123 & 4.0 & 4.4 & 0.0 & 3.0 & 6.0 \\
17 + Storeys & 19{,}123 & 1.6 & 1.6 & 1.0 & 2.0 & 2.0 \\
18 + Age (years) & 19{,}123 & 42.5 & 28.0 & 24.5 & 40.0 & 53.0 \\
19 + Rooms (Realtor) & 19{,}123 & 10.1 & 6.5 & 7.0 & 11.0 & 14.0 \\
20 + Photos & 19{,}123 & 42 & 18 & 30 & 40 & 51 \\
21 + \bottomrule
22 + \end{tabular}
23 + \begin{tablenotes}[flushleft]
24 + \footnotesize
25 + \item \textit{Notes:} Descriptive statistics for the cleaned sample of 19{,}123 single-family and townhouse listings (Realtor.ca and DuProprio), scraped on 3~July~2026. 80\% of listings originate from Realtor.ca. Living area is the reported area, or reconstructed from room dimensions where missing. Continuous variables are bounded by the outlier rules described in Section~\ref{sec:data}; price-per-square-foot is trimmed at the 0.5th and 99.5th percentiles.
26 + \end{tablenotes}
27 + \end{threeparttable}
28 +\end{table}
added paper/uq_logo.jpg +0 −0

Binary file not shown.

added requirements.txt +6 −0
@@ -0,0 +1,6 @@
1 +numpy>=1.26
2 +pandas>=2.0
3 +statsmodels>=0.14
4 +scikit-learn>=1.4
5 +matplotlib>=3.8
6 +torch>=2.2
added results/ablation_extras.json +62 −0
@@ -0,0 +1,62 @@
1 +{
2 + "L1": {
3 + "nfeat": 24,
4 + "ols": {
5 + "r2": 0.5208807490583831,
6 + "mdae": 19.311838552673066,
7 + "w15": 40.46959159127752,
8 + "w25": 60.90048632536735
9 + },
10 + "gbm": {
11 + "r2": 0.6922092084886344,
12 + "mdae": 17.160081786433278,
13 + "w15": 44.70532866182084,
14 + "w25": 65.6591538984469
15 + }
16 + },
17 + "L2": {
18 + "nfeat": 38,
19 + "ols": {
20 + "r2": 0.6193226824225797,
21 + "mdae": 18.059509413013565,
22 + "w15": 42.81754954766512,
23 + "w25": 63.755686869215076
24 + },
25 + "gbm": {
26 + "r2": 0.7288688475138769,
27 + "mdae": 15.95094002848213,
28 + "w15": 47.46640171521205,
29 + "w25": 68.38885112168592
30 + }
31 + },
32 + "L3": {
33 + "nfeat": 56,
34 + "ols": {
35 + "r2": 0.6277548901386698,
36 + "mdae": 17.877027384670708,
37 + "w15": 42.92213564817236,
38 + "w25": 64.46687235266432
39 + },
40 + "gbm": {
41 + "r2": 0.7428875872402715,
42 + "mdae": 15.421354827745482,
43 + "w15": 48.84170893688229,
44 + "w25": 69.91580818909166
45 + }
46 + },
47 + "L4": {
48 + "nfeat": 77,
49 + "ols": {
50 + "r2": 0.6862361284610743,
51 + "mdae": 16.886250547842508,
52 + "w15": 45.25440568948387,
53 + "w25": 66.72070281859541
54 + },
55 + "gbm": {
56 + "r2": 0.7695283346486282,
57 + "mdae": 14.51330744579874,
58 + "w15": 51.383151179208284,
59 + "w25": 71.29634471578727
60 + }
61 + }
62 +}
\ No newline at end of file
added results/hedonic_summary.txt +536 −0
@@ -0,0 +1,536 @@
1 +=== MODELE A (avec superficie) ===
2 + OLS Regression Results
3 +==============================================================================
4 +Dep. Variable: ln_price R-squared: 0.717
5 +Model: OLS Adj. R-squared: 0.709
6 +Method: Least Squares F-statistic: 93.16
7 +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00
8 +Time: 07:41:43 Log-Likelihood: -1866.7
9 +No. Observations: 9444 AIC: 4215.
10 +Df Residuals: 9203 BIC: 5939.
11 +Df Model: 240
12 +Covariance Type: HC3
13 +========================================================================================================================
14 + coef std err z P>|z| [0.025 0.975]
15 +------------------------------------------------------------------------------------------------------------------------
16 +Intercept 9.7330 0.094 103.249 0.000 9.548 9.918
17 +C(source)[T.realtor] -0.2189 0.022 -10.109 0.000 -0.261 -0.176
18 +C(ptype_fe)[T.Country Home] 0.0045 0.051 0.087 0.930 -0.096 0.105
19 +C(ptype_fe)[T.Home] -0.0071 0.013 -0.525 0.599 -0.034 0.019
20 +C(ptype_fe)[T.House] -0.0307 0.012 -2.558 0.011 -0.054 -0.007
21 +C(ptype_fe)[T.Intergenerational] -0.0200 0.036 -0.558 0.577 -0.090 0.050
22 +C(ptype_fe)[T.Other] 0.2178 0.120 1.822 0.068 -0.016 0.452
23 +C(ptype_fe)[T.Row / Townhouse] -0.1882 0.016 -12.125 0.000 -0.219 -0.158
24 +C(ptype_fe)[T.Semi Detached] -0.0601 0.015 -3.893 0.000 -0.090 -0.030
25 +C(ptype_fe)[T.Townhouse] -0.0038 0.025 -0.151 0.880 -0.053 0.045
26 +C(city_fe)[T.acton vale] -0.0615 0.058 -1.065 0.287 -0.175 0.052
27 +C(city_fe)[T.alma] -0.0170 0.054 -0.317 0.751 -0.122 0.088
28 +C(city_fe)[T.amherst] -0.0746 0.076 -0.980 0.327 -0.224 0.075
29 +C(city_fe)[T.amos] -0.0308 0.063 -0.490 0.624 -0.154 0.092
30 +C(city_fe)[T.austin] 0.2786 0.091 3.061 0.002 0.100 0.457
31 +C(city_fe)[T.baie-comeau] -0.3656 0.083 -4.411 0.000 -0.528 -0.203
32 +C(city_fe)[T.baie-saint-paul] -0.0934 0.053 -1.765 0.078 -0.197 0.010
33 +C(city_fe)[T.beaconsfield] 0.6128 0.052 11.867 0.000 0.512 0.714
34 +C(city_fe)[T.beauport] 0.0424 0.037 1.149 0.250 -0.030 0.115
35 +C(city_fe)[T.beloeil] 0.2255 0.052 4.354 0.000 0.124 0.327
36 +C(city_fe)[T.blainville] 0.3095 0.030 10.218 0.000 0.250 0.369
37 +C(city_fe)[T.blue sea] -0.0272 0.060 -0.450 0.652 -0.146 0.091
38 +C(city_fe)[T.bois-des-filion] 0.1842 0.057 3.221 0.001 0.072 0.296
39 +C(city_fe)[T.boisbriand] 0.2661 0.039 6.854 0.000 0.190 0.342
40 +C(city_fe)[T.boischatel] 0.1368 0.082 1.664 0.096 -0.024 0.298
41 +C(city_fe)[T.boucherville] 0.3473 0.040 8.688 0.000 0.269 0.426
42 +C(city_fe)[T.bromont] 0.3122 0.037 8.391 0.000 0.239 0.385
43 +C(city_fe)[T.brossard] 0.4100 0.030 13.746 0.000 0.352 0.468
44 +C(city_fe)[T.bécancour] -0.0918 0.039 -2.323 0.020 -0.169 -0.014
45 +C(city_fe)[T.cantley] 0.1223 0.033 3.758 0.000 0.059 0.186
46 +C(city_fe)[T.carignan] 0.2488 0.086 2.881 0.004 0.080 0.418
47 +C(city_fe)[T.chandler] -0.2905 0.141 -2.068 0.039 -0.566 -0.015
48 +C(city_fe)[T.chateauguay] 0.2290 0.036 6.421 0.000 0.159 0.299
49 +C(city_fe)[T.chelsea] 0.2572 0.038 6.779 0.000 0.183 0.332
50 +C(city_fe)[T.chertsey] -0.0464 0.064 -0.730 0.466 -0.171 0.078
51 +C(city_fe)[T.chicoutimi] -0.1007 0.046 -2.203 0.028 -0.190 -0.011
52 +C(city_fe)[T.chomedey] 0.3195 0.050 6.430 0.000 0.222 0.417
53 +C(city_fe)[T.châteauguay] 0.2060 0.038 5.390 0.000 0.131 0.281
54 +C(city_fe)[T.contrecoeur] 0.0915 0.057 1.607 0.108 -0.020 0.203
55 +C(city_fe)[T.cowansville] 0.1177 0.050 2.376 0.017 0.021 0.215
56 +C(city_fe)[T.crabtree] 0.0241 0.096 0.251 0.802 -0.164 0.213
57 +C(city_fe)[T.côte-saint-luc] 0.6071 0.050 12.165 0.000 0.509 0.705
58 +C(city_fe)[T.dolbeau-mistassini] -0.2645 0.091 -2.905 0.004 -0.443 -0.086
59 +C(city_fe)[T.dollard-des-ormeaux] 0.4023 0.038 10.535 0.000 0.327 0.477
60 +C(city_fe)[T.donnacona] 0.2465 0.039 6.358 0.000 0.171 0.322
61 +C(city_fe)[T.dorval] 0.5515 0.055 10.101 0.000 0.445 0.659
62 +C(city_fe)[T.drummondville] -0.0196 0.022 -0.895 0.371 -0.062 0.023
63 +C(city_fe)[T.duhamel] 0.1373 0.076 1.815 0.069 -0.011 0.286
64 +C(city_fe)[T.dunham] 0.2726 0.064 4.237 0.000 0.147 0.399
65 +C(city_fe)[T.eastman] 0.3041 0.051 5.919 0.000 0.203 0.405
66 +C(city_fe)[T.entrelacs] 0.2617 0.135 1.945 0.052 -0.002 0.525
67 +C(city_fe)[T.farnham] 0.0545 0.062 0.885 0.376 -0.066 0.175
68 +C(city_fe)[T.fossambault-sur-le-lac] 0.2664 0.079 3.365 0.001 0.111 0.422
69 +C(city_fe)[T.gaspé] -0.2214 0.069 -3.228 0.001 -0.356 -0.087
70 +C(city_fe)[T.gatineau] 0.0901 0.012 7.233 0.000 0.066 0.114
71 +C(city_fe)[T.gore] 0.0786 0.104 0.753 0.451 -0.126 0.283
72 +C(city_fe)[T.granby] 0.1117 0.024 4.632 0.000 0.064 0.159
73 +C(city_fe)[T.grenville-sur-la-rouge] 0.1287 0.078 1.656 0.098 -0.024 0.281
74 +C(city_fe)[T.hampstead] 1.0843 0.057 19.163 0.000 0.973 1.195
75 +C(city_fe)[T.harrington] 0.1429 0.152 0.940 0.347 -0.155 0.441
76 +C(city_fe)[T.hudson] 0.2555 0.199 1.282 0.200 -0.135 0.646
77 +C(city_fe)[T.joliette] -0.0275 0.071 -0.387 0.699 -0.167 0.112
78 +C(city_fe)[T.jonquière] -0.1380 0.045 -3.096 0.002 -0.225 -0.051
79 +C(city_fe)[T.kirkland] 0.3686 0.037 9.967 0.000 0.296 0.441
80 +C(city_fe)[T.l'ange-gardien] 0.0142 0.158 0.090 0.928 -0.295 0.323
81 +C(city_fe)[T.l'assomption] 0.1479 0.060 2.460 0.014 0.030 0.266
82 +C(city_fe)[T.la conception] 0.3351 0.053 6.306 0.000 0.231 0.439
83 +C(city_fe)[T.la malbaie] -0.1347 0.063 -2.142 0.032 -0.258 -0.011
84 +C(city_fe)[T.la minerve] 0.0449 0.069 0.646 0.518 -0.091 0.181
85 +C(city_fe)[T.la prairie] 0.3741 0.035 10.542 0.000 0.305 0.444
86 +C(city_fe)[T.la pêche] 0.0354 0.037 0.968 0.333 -0.036 0.107
87 +C(city_fe)[T.la tuque] -0.3119 0.067 -4.627 0.000 -0.444 -0.180
88 +C(city_fe)[T.labelle] 0.1111 0.113 0.980 0.327 -0.111 0.333
89 +C(city_fe)[T.lac-beauport] 0.3609 0.094 3.852 0.000 0.177 0.545
90 +C(city_fe)[T.lac-brome] 0.5002 0.062 8.091 0.000 0.379 0.621
91 +C(city_fe)[T.lac-des-écorces] -0.0964 0.159 -0.607 0.544 -0.408 0.215
92 +C(city_fe)[T.lac-etchemin] -0.1890 0.135 -1.396 0.163 -0.454 0.076
93 +C(city_fe)[T.lachute] 0.1234 0.075 1.655 0.098 -0.023 0.269
94 +C(city_fe)[T.lanoraie] -0.0824 0.051 -1.623 0.105 -0.182 0.017
95 +C(city_fe)[T.laval] 0.3952 0.020 19.308 0.000 0.355 0.435
96 +C(city_fe)[T.lavaltrie] 0.1283 0.065 1.990 0.047 0.002 0.255
97 +C(city_fe)[T.les éboulements] 0.2007 0.080 2.516 0.012 0.044 0.357
98 +C(city_fe)[T.longueuil] 0.3160 0.020 16.113 0.000 0.278 0.354
99 +C(city_fe)[T.lorraine] 0.1128 0.113 0.995 0.320 -0.109 0.335
100 +C(city_fe)[T.lévis] 0.0663 0.027 2.486 0.013 0.014 0.119
101 +C(city_fe)[T.magog] 0.3570 0.034 10.628 0.000 0.291 0.423
102 +C(city_fe)[T.mandeville] -0.1781 0.080 -2.221 0.026 -0.335 -0.021
103 +C(city_fe)[T.mansfield-et-pontefract] -0.3813 0.079 -4.847 0.000 -0.536 -0.227
104 +C(city_fe)[T.marieville] 0.1717 0.105 1.634 0.102 -0.034 0.377
105 +C(city_fe)[T.mascouche] 0.1768 0.042 4.225 0.000 0.095 0.259
106 +C(city_fe)[T.matane] -0.2714 0.042 -6.488 0.000 -0.353 -0.189
107 +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3175 0.034 9.297 0.000 0.251 0.384
108 +C(city_fe)[T.messines] -0.1638 0.061 -2.701 0.007 -0.283 -0.045
109 +C(city_fe)[T.mille-isles] 0.2388 0.055 4.380 0.000 0.132 0.346
110 +C(city_fe)[T.mirabel] 0.2407 0.027 9.076 0.000 0.189 0.293
111 +C(city_fe)[T.mont-joli] -0.2476 0.177 -1.397 0.162 -0.595 0.100
112 +C(city_fe)[T.mont-laurier] -0.0494 0.038 -1.316 0.188 -0.123 0.024
113 +C(city_fe)[T.mont-royal] 1.0590 0.060 17.641 0.000 0.941 1.177
114 +C(city_fe)[T.mont-saint-hilaire] 0.3721 0.109 3.421 0.001 0.159 0.585
115 +C(city_fe)[T.mont-tremblant] 0.3943 0.040 9.855 0.000 0.316 0.473
116 +C(city_fe)[T.montréal] 0.6721 0.021 31.548 0.000 0.630 0.714
117 +C(city_fe)[T.morin-heights] 0.2972 0.057 5.204 0.000 0.185 0.409
118 +C(city_fe)[T.nominingue] 0.1336 0.105 1.276 0.202 -0.072 0.339
119 +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.2087 0.061 3.406 0.001 0.089 0.329
120 +C(city_fe)[T.notre-dame-de-pontmain] 0.0094 0.061 0.154 0.878 -0.111 0.130
121 +C(city_fe)[T.notre-dame-des-bois] -0.2375 0.074 -3.230 0.001 -0.382 -0.093
122 +C(city_fe)[T.notre-dame-des-prairies] 0.0822 0.118 0.697 0.486 -0.149 0.313
123 +C(city_fe)[T.notre-dame-du-laus] 0.1009 0.053 1.894 0.058 -0.003 0.205
124 +C(city_fe)[T.orford] 0.3230 0.063 5.122 0.000 0.199 0.447
125 +C(city_fe)[T.papineauville] -0.0886 0.124 -0.717 0.473 -0.331 0.153
126 +C(city_fe)[T.petite-rivière-saint-françois] -0.0343 0.084 -0.407 0.684 -0.200 0.131
127 +C(city_fe)[T.piedmont] 0.4643 0.067 6.959 0.000 0.334 0.595
128 +C(city_fe)[T.pincourt] 0.1811 0.055 3.304 0.001 0.074 0.289
129 +C(city_fe)[T.pointe-claire] 0.4251 0.038 11.133 0.000 0.350 0.500
130 +C(city_fe)[T.pont-rouge] 0.0628 0.056 1.115 0.265 -0.048 0.173
131 +C(city_fe)[T.prévost] 0.1183 0.037 3.231 0.001 0.047 0.190
132 +C(city_fe)[T.québec] 0.1687 0.020 8.570 0.000 0.130 0.207
133 +C(city_fe)[T.rawdon] 0.0579 0.047 1.244 0.213 -0.033 0.149
134 +C(city_fe)[T.repentigny] 0.1908 0.032 5.948 0.000 0.128 0.254
135 +C(city_fe)[T.rigaud] 0.2981 0.064 4.632 0.000 0.172 0.424
136 +C(city_fe)[T.rimouski] -0.0613 0.034 -1.824 0.068 -0.127 0.005
137 +C(city_fe)[T.rivière-du-loup] -0.1841 0.094 -1.952 0.051 -0.369 0.001
138 +C(city_fe)[T.rivière-rouge] -0.0045 0.071 -0.063 0.950 -0.143 0.135
139 +C(city_fe)[T.roberval] -0.2600 0.081 -3.206 0.001 -0.419 -0.101
140 +C(city_fe)[T.rosemère] 0.3854 0.089 4.317 0.000 0.210 0.560
141 +C(city_fe)[T.saguenay] -0.0722 0.044 -1.628 0.103 -0.159 0.015
142 +C(city_fe)[T.saint-adolphe-d'howard] 0.3369 0.054 6.269 0.000 0.232 0.442
143 +C(city_fe)[T.saint-amable] 0.3185 0.071 4.461 0.000 0.179 0.458
144 +C(city_fe)[T.saint-anicet] 0.1207 0.072 1.684 0.092 -0.020 0.261
145 +C(city_fe)[T.saint-basile-le-grand] 0.3421 0.093 3.686 0.000 0.160 0.524
146 +C(city_fe)[T.saint-bruno-de-montarville] 0.6408 0.073 8.723 0.000 0.497 0.785
147 +C(city_fe)[T.saint-calixte] 0.1561 0.090 1.730 0.084 -0.021 0.333
148 +C(city_fe)[T.saint-charles-borromée] 0.1371 0.163 0.839 0.401 -0.183 0.457
149 +C(city_fe)[T.saint-colomban] 0.2111 0.070 3.016 0.003 0.074 0.348
150 +C(city_fe)[T.saint-constant] 0.2154 0.052 4.115 0.000 0.113 0.318
151 +C(city_fe)[T.saint-donat] 0.2569 0.067 3.862 0.000 0.127 0.387
152 +C(city_fe)[T.saint-eustache] 0.4482 0.071 6.307 0.000 0.309 0.587
153 +C(city_fe)[T.saint-félicien] -0.2423 0.336 -0.721 0.471 -0.901 0.416
154 +C(city_fe)[T.saint-gabriel-de-brandon] -0.1880 0.080 -2.346 0.019 -0.345 -0.031
155 +C(city_fe)[T.saint-georges] -0.1023 0.063 -1.626 0.104 -0.226 0.021
156 +C(city_fe)[T.saint-hippolyte] 0.3575 0.082 4.378 0.000 0.197 0.517
157 +C(city_fe)[T.saint-honoré] -0.1282 0.072 -1.788 0.074 -0.269 0.012
158 +C(city_fe)[T.saint-hyacinthe] 0.1600 0.061 2.616 0.009 0.040 0.280
159 +C(city_fe)[T.saint-jean-de-matha] 0.1081 0.147 0.734 0.463 -0.181 0.397
160 +C(city_fe)[T.saint-jean-sur-richelieu] 0.1894 0.029 6.448 0.000 0.132 0.247
161 +C(city_fe)[T.saint-jérôme] 0.1558 0.039 4.037 0.000 0.080 0.231
162 +C(city_fe)[T.saint-lambert] 0.6472 0.071 9.070 0.000 0.507 0.787
163 +C(city_fe)[T.saint-lambert-de-lauzon] 0.0527 0.081 0.654 0.513 -0.105 0.210
164 +C(city_fe)[T.saint-lin/laurentides] 0.0940 0.120 0.783 0.433 -0.141 0.329
165 +C(city_fe)[T.saint-marc-des-carrières] -0.2484 0.047 -5.271 0.000 -0.341 -0.156
166 +C(city_fe)[T.saint-michel-des-saints] -0.2325 0.080 -2.906 0.004 -0.389 -0.076
167 +C(city_fe)[T.saint-paul-de-l'île-aux-noix] -0.1222 0.249 -0.491 0.623 -0.610 0.365
168 +C(city_fe)[T.saint-philippe] 0.3232 0.040 8.134 0.000 0.245 0.401
169 +C(city_fe)[T.saint-raymond] 0.1551 0.088 1.768 0.077 -0.017 0.327
170 +C(city_fe)[T.saint-sauveur] 0.2986 0.058 5.155 0.000 0.185 0.412
171 +C(city_fe)[T.saint-zotique] 0.4566 0.074 6.189 0.000 0.312 0.601
172 +C(city_fe)[T.saint-zénon] -0.3094 86.426 -0.004 0.997 -169.702 169.083
173 +C(city_fe)[T.sainte-adèle] 0.3608 0.078 4.614 0.000 0.208 0.514
174 +C(city_fe)[T.sainte-agathe-des-monts] 0.1222 0.092 1.334 0.182 -0.057 0.302
175 +C(city_fe)[T.sainte-anne-des-plaines] 0.8940 0.773 1.156 0.248 -0.622 2.409
176 +C(city_fe)[T.sainte-barbe] 0.1909 0.291 0.655 0.512 -0.380 0.762
177 +C(city_fe)[T.sainte-brigitte-de-laval] 0.1374 0.051 2.680 0.007 0.037 0.238
178 +C(city_fe)[T.sainte-béatrix] 0.0338 0.044 0.764 0.445 -0.053 0.121
179 +C(city_fe)[T.sainte-catherine] 0.3921 0.073 5.382 0.000 0.249 0.535
180 +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.2108 0.068 3.084 0.002 0.077 0.345
181 +C(city_fe)[T.sainte-julie] 0.3731 0.082 4.540 0.000 0.212 0.534
182 +C(city_fe)[T.sainte-julienne] 0.0598 0.076 0.787 0.431 -0.089 0.209
183 +C(city_fe)[T.sainte-lucie-des-laurentides] 0.3382 0.073 4.613 0.000 0.195 0.482
184 +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.1056 0.051 2.065 0.039 0.005 0.206
185 +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.2464 0.035 7.007 0.000 0.177 0.315
186 +C(city_fe)[T.sainte-sophie] 0.0938 0.043 2.179 0.029 0.009 0.178
187 +C(city_fe)[T.sainte-thérèse] 0.2976 0.087 3.428 0.001 0.127 0.468
188 +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.1963 0.064 -3.071 0.002 -0.322 -0.071
189 +C(city_fe)[T.salaberry-de-valleyfield] 0.0587 0.049 1.205 0.228 -0.037 0.154
190 +C(city_fe)[T.sept-îles] -0.1510 0.085 -1.774 0.076 -0.318 0.016
191 +C(city_fe)[T.shawinigan] -0.1856 0.053 -3.475 0.001 -0.290 -0.081
192 +C(city_fe)[T.shefford] 0.1276 0.089 1.437 0.151 -0.046 0.302
193 +C(city_fe)[T.sherbrooke] 0.0976 0.018 5.321 0.000 0.062 0.133
194 +C(city_fe)[T.sorel-tracy] -0.0858 0.046 -1.883 0.060 -0.175 0.003
195 +C(city_fe)[T.st-colomban] 0.2324 0.059 3.962 0.000 0.117 0.347
196 +C(city_fe)[T.st-hippolyte] 0.1484 0.057 2.614 0.009 0.037 0.260
197 +C(city_fe)[T.st-jean-sur-richelieu] 0.1674 0.033 5.005 0.000 0.102 0.233
198 +C(city_fe)[T.st-jérôme] 0.1660 0.032 5.221 0.000 0.104 0.228
199 +C(city_fe)[T.st-sauveur] 0.2742 0.053 5.170 0.000 0.170 0.378
200 +C(city_fe)[T.ste-adèle] 0.1521 0.071 2.155 0.031 0.014 0.290
201 +C(city_fe)[T.ste-julienne] 0.0571 0.043 1.321 0.187 -0.028 0.142
202 +C(city_fe)[T.sutton] 0.3865 0.068 5.668 0.000 0.253 0.520
203 +C(city_fe)[T.terrebonne] 0.2097 0.029 7.341 0.000 0.154 0.266
204 +C(city_fe)[T.thetford mines] -0.3094 0.118 -2.617 0.009 -0.541 -0.078
205 +C(city_fe)[T.trois-rivières] -0.0763 0.023 -3.372 0.001 -0.121 -0.032
206 +C(city_fe)[T.très-saint-rédempteur] 0.4880 0.032 15.400 0.000 0.426 0.550
207 +C(city_fe)[T.témiscouata-sur-le-lac] -0.2810 0.139 -2.018 0.044 -0.554 -0.008
208 +C(city_fe)[T.val-d'or] 0.0673 0.031 2.182 0.029 0.007 0.128
209 +C(city_fe)[T.val-david] 0.0033 0.100 0.033 0.973 -0.193 0.199
210 +C(city_fe)[T.val-des-monts] 0.1145 0.025 4.660 0.000 0.066 0.163
211 +C(city_fe)[T.val-morin] -0.1213 0.092 -1.311 0.190 -0.302 0.060
212 +C(city_fe)[T.vaudreuil-dorion] 0.0676 0.039 1.731 0.083 -0.009 0.144
213 +C(city_fe)[T.venise-en-québec] 0.0774 0.073 1.065 0.287 -0.065 0.220
214 +C(city_fe)[T.victoriaville] -0.0961 0.031 -3.150 0.002 -0.156 -0.036
215 +C(city_fe)[T.waterloo] 0.1009 0.051 1.969 0.049 0.000 0.201
216 +C(city_fe)[T.weedon] -0.1440 0.081 -1.780 0.075 -0.303 0.015
217 +C(city_fe)[T.wentworth-nord] 0.2091 0.088 2.378 0.017 0.037 0.381
218 +C(city_fe)[T.westmount] 1.3088 0.042 31.188 0.000 1.227 1.391
219 +ln_sqft 0.4171 0.013 31.566 0.000 0.391 0.443
220 +ln_lot 0.0257 0.004 6.727 0.000 0.018 0.033
221 +has_lot -0.0242 0.014 -1.683 0.092 -0.052 0.004
222 +beds 0.0022 0.004 0.513 0.608 -0.006 0.011
223 +baths 0.0897 0.007 13.613 0.000 0.077 0.103
224 +parking 0.0109 0.002 7.110 0.000 0.008 0.014
225 +storeys -0.0023 0.007 -0.312 0.755 -0.017 0.012
226 +age -0.0060 0.000 -12.824 0.000 -0.007 -0.005
227 +age2 2.97e-05 3.88e-06 7.650 0.000 2.21e-05 3.73e-05
228 +has_age -0.1804 0.021 -8.685 0.000 -0.221 -0.140
229 +pool 0.0341 0.008 4.444 0.000 0.019 0.049
230 +waterfront 0.0623 0.021 2.965 0.003 0.021 0.104
231 +garage 0.0279 0.007 3.953 0.000 0.014 0.042
232 +fireplace 0.0172 0.008 2.130 0.033 0.001 0.033
233 +central_air 0.1016 0.011 9.291 0.000 0.080 0.123
234 +finished_bsmt -0.0115 0.010 -1.191 0.234 -0.030 0.007
235 +near_water 0.0972 0.023 4.149 0.000 0.051 0.143
236 +cul_de_sac 0.0279 0.012 2.409 0.016 0.005 0.051
237 +golf 0.0225 0.007 3.138 0.002 0.008 0.037
238 +ski 0.0218 0.013 1.649 0.099 -0.004 0.048
239 +near_park 0.0037 0.011 0.328 0.743 -0.019 0.026
240 +near_school -0.0695 0.011 -6.157 0.000 -0.092 -0.047
241 +near_highway 0.0514 0.008 6.362 0.000 0.036 0.067
242 +transit 0.1011 0.009 11.372 0.000 0.084 0.119
243 +no_rear_neighbor -0.0377 0.010 -3.802 0.000 -0.057 -0.018
244 +n_images_c 0.0045 0.000 14.507 0.000 0.004 0.005
245 +virtual_tour 0.0444 0.022 1.998 0.046 0.001 0.088
246 +n_rooms 0.0172 0.002 9.260 0.000 0.014 0.021
247 +n_levels_r -0.0248 0.009 -2.825 0.005 -0.042 -0.008
248 +has_primary_bed 0.0775 0.016 4.958 0.000 0.047 0.108
249 +has_basement -0.0251 0.015 -1.714 0.086 -0.054 0.004
250 +r_office 0.0300 0.011 2.631 0.009 0.008 0.052
251 +r_family 0.0422 0.011 3.973 0.000 0.021 0.063
252 +r_workshop -0.0401 0.014 -2.897 0.004 -0.067 -0.013
253 +r_winecellar 0.1086 0.036 3.034 0.002 0.038 0.179
254 +r_sunroom -0.0042 0.021 -0.201 0.840 -0.046 0.037
255 +r_mezzanine 0.0136 0.026 0.527 0.598 -0.037 0.064
256 +r_library 0.0945 0.084 1.123 0.262 -0.070 0.260
257 +r_laundry 0.0202 0.009 2.222 0.026 0.002 0.038
258 +==============================================================================
259 +Omnibus: 403.871 Durbin-Watson: 1.849
260 +Prob(Omnibus): 0.000 Jarque-Bera (JB): 1350.435
261 +Skew: 0.024 Prob(JB): 5.71e-294
262 +Kurtosis: 4.852 Cond. No. 5.36e+16
263 +==============================================================================
264 +
265 +Notes:
266 +[1] Standard Errors are heteroscedasticity robust (HC3)
267 +[2] The smallest eigenvalue is 5.8e-23. This might indicate that there are
268 +strong multicollinearity problems or that the design matrix is singular.
269 +
270 +=== MODELE B (sans superficie) ===
271 + OLS Regression Results
272 +==============================================================================
273 +Dep. Variable: ln_price R-squared: 0.677
274 +Model: OLS Adj. R-squared: 0.673
275 +Method: Least Squares F-statistic: 149.6
276 +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00
277 +Time: 07:41:43 Log-Likelihood: -6293.3
278 +No. Observations: 19115 AIC: 1.307e+04
279 +Df Residuals: 18875 BIC: 1.495e+04
280 +Df Model: 239
281 +Covariance Type: HC3
282 +========================================================================================================================
283 + coef std err z P>|z| [0.025 0.975]
284 +------------------------------------------------------------------------------------------------------------------------
285 +Intercept 12.4073 0.115 108.355 0.000 12.183 12.632
286 +C(source)[T.realtor] -0.3680 0.021 -17.300 0.000 -0.410 -0.326
287 +C(ptype_fe)[T.Country Home] 0.0513 0.067 0.765 0.444 -0.080 0.183
288 +C(ptype_fe)[T.Home] 0.1075 0.083 1.297 0.195 -0.055 0.270
289 +C(ptype_fe)[T.House] -0.0659 0.014 -4.694 0.000 -0.093 -0.038
290 +C(ptype_fe)[T.Intergenerational] 0.1281 0.062 2.051 0.040 0.006 0.250
291 +C(ptype_fe)[T.Other] 0.2713 0.121 2.243 0.025 0.034 0.508
292 +C(ptype_fe)[T.Row / Townhouse] -0.3021 0.014 -21.062 0.000 -0.330 -0.274
293 +C(ptype_fe)[T.Semi Detached] -0.0065 0.057 -0.115 0.909 -0.118 0.105
294 +C(ptype_fe)[T.Townhouse] 0.0963 0.100 0.965 0.335 -0.099 0.292
295 +C(city_fe)[T.acton vale] -0.0167 0.033 -0.499 0.618 -0.082 0.049
296 +C(city_fe)[T.alma] -0.0480 0.060 -0.796 0.426 -0.166 0.070
297 +C(city_fe)[T.amherst] -0.0109 0.050 -0.217 0.828 -0.110 0.088
298 +C(city_fe)[T.amos] -0.1497 0.046 -3.239 0.001 -0.240 -0.059
299 +C(city_fe)[T.austin] 0.4089 0.071 5.776 0.000 0.270 0.548
300 +C(city_fe)[T.baie-comeau] -0.3409 0.071 -4.806 0.000 -0.480 -0.202
301 +C(city_fe)[T.baie-saint-paul] 0.0980 0.074 1.327 0.184 -0.047 0.243
302 +C(city_fe)[T.beaconsfield] 0.7818 0.039 20.221 0.000 0.706 0.858
303 +C(city_fe)[T.beauport] 0.0722 0.041 1.775 0.076 -0.008 0.152
304 +C(city_fe)[T.beloeil] 0.3670 0.049 7.506 0.000 0.271 0.463
305 +C(city_fe)[T.blainville] 0.4382 0.025 17.427 0.000 0.389 0.487
306 +C(city_fe)[T.blue sea] -0.1010 0.064 -1.585 0.113 -0.226 0.024
307 +C(city_fe)[T.bois-des-filion] 0.2889 0.048 5.999 0.000 0.194 0.383
308 +C(city_fe)[T.boisbriand] 0.3584 0.038 9.505 0.000 0.284 0.432
309 +C(city_fe)[T.boischatel] 0.4149 0.076 5.483 0.000 0.267 0.563
310 +C(city_fe)[T.boucherville] 0.4653 0.037 12.616 0.000 0.393 0.538
311 +C(city_fe)[T.bromont] 0.4068 0.034 11.856 0.000 0.340 0.474
312 +C(city_fe)[T.brossard] 0.5145 0.020 25.986 0.000 0.476 0.553
313 +C(city_fe)[T.bécancour] -0.0914 0.033 -2.788 0.005 -0.156 -0.027
314 +C(city_fe)[T.cantley] 0.1594 0.048 3.356 0.001 0.066 0.253
315 +C(city_fe)[T.carignan] 0.3871 0.041 9.472 0.000 0.307 0.467
316 +C(city_fe)[T.chandler] -0.3044 0.081 -3.746 0.000 -0.464 -0.145
317 +C(city_fe)[T.chateauguay] 0.2448 0.040 6.056 0.000 0.166 0.324
318 +C(city_fe)[T.chelsea] 0.2986 0.040 7.393 0.000 0.219 0.378
319 +C(city_fe)[T.chertsey] -0.0194 0.038 -0.508 0.612 -0.094 0.056
320 +C(city_fe)[T.chicoutimi] -0.0770 0.041 -1.880 0.060 -0.157 0.003
321 +C(city_fe)[T.chomedey] 0.4087 0.056 7.260 0.000 0.298 0.519
322 +C(city_fe)[T.châteauguay] 0.2797 0.023 12.337 0.000 0.235 0.324
323 +C(city_fe)[T.contrecoeur] 0.1257 0.034 3.670 0.000 0.059 0.193
324 +C(city_fe)[T.cowansville] 0.1495 0.036 4.133 0.000 0.079 0.220
325 +C(city_fe)[T.crabtree] 0.0639 0.046 1.394 0.163 -0.026 0.154
326 +C(city_fe)[T.côte-saint-luc] 0.7419 0.052 14.316 0.000 0.640 0.843
327 +C(city_fe)[T.dolbeau-mistassini] -0.1920 0.066 -2.895 0.004 -0.322 -0.062
328 +C(city_fe)[T.dollard-des-ormeaux] 0.5263 0.029 18.095 0.000 0.469 0.583
329 +C(city_fe)[T.donnacona] 0.1454 0.035 4.112 0.000 0.076 0.215
330 +C(city_fe)[T.dorval] 0.6752 0.050 13.510 0.000 0.577 0.773
331 +C(city_fe)[T.drummondville] 0.0561 0.019 2.952 0.003 0.019 0.093
332 +C(city_fe)[T.duhamel] 0.0808 0.077 1.049 0.294 -0.070 0.232
333 +C(city_fe)[T.dunham] 0.3488 0.063 5.542 0.000 0.225 0.472
334 +C(city_fe)[T.eastman] 0.3943 0.055 7.157 0.000 0.286 0.502
335 +C(city_fe)[T.entrelacs] 0.2226 0.086 2.601 0.009 0.055 0.390
336 +C(city_fe)[T.farnham] 0.0560 0.038 1.463 0.143 -0.019 0.131
337 +C(city_fe)[T.fossambault-sur-le-lac] 0.3607 0.067 5.395 0.000 0.230 0.492
338 +C(city_fe)[T.gaspé] -0.3110 0.062 -4.989 0.000 -0.433 -0.189
339 +C(city_fe)[T.gatineau] 0.1163 0.013 8.687 0.000 0.090 0.143
340 +C(city_fe)[T.gore] 0.0808 0.058 1.403 0.161 -0.032 0.194
341 +C(city_fe)[T.granby] 0.1488 0.019 7.962 0.000 0.112 0.185
342 +C(city_fe)[T.grenville-sur-la-rouge] 0.1281 0.046 2.801 0.005 0.038 0.218
343 +C(city_fe)[T.hampstead] 1.2858 0.051 25.203 0.000 1.186 1.386
344 +C(city_fe)[T.harrington] -0.0622 0.130 -0.478 0.633 -0.318 0.193
345 +C(city_fe)[T.hudson] 0.4820 0.071 6.816 0.000 0.343 0.621
346 +C(city_fe)[T.joliette] 0.1230 0.046 2.684 0.007 0.033 0.213
347 +C(city_fe)[T.jonquière] -0.1590 0.046 -3.470 0.001 -0.249 -0.069
348 +C(city_fe)[T.kirkland] 0.5347 0.035 15.112 0.000 0.465 0.604
349 +C(city_fe)[T.l'ange-gardien] 0.2734 0.073 3.767 0.000 0.131 0.416
350 +C(city_fe)[T.l'assomption] 0.1985 0.027 7.277 0.000 0.145 0.252
351 +C(city_fe)[T.la conception] 0.2903 0.052 5.600 0.000 0.189 0.392
352 +C(city_fe)[T.la malbaie] -0.1641 0.055 -2.997 0.003 -0.271 -0.057
353 +C(city_fe)[T.la minerve] 0.0741 0.054 1.378 0.168 -0.031 0.180
354 +C(city_fe)[T.la prairie] 0.4728 0.030 15.749 0.000 0.414 0.532
355 +C(city_fe)[T.la pêche] 0.0037 0.039 0.094 0.925 -0.073 0.081
356 +C(city_fe)[T.la tuque] -0.3610 0.049 -7.307 0.000 -0.458 -0.264
357 +C(city_fe)[T.labelle] 0.0955 0.057 1.687 0.092 -0.015 0.206
358 +C(city_fe)[T.lac-beauport] 0.4166 0.089 4.659 0.000 0.241 0.592
359 +C(city_fe)[T.lac-brome] 0.6311 0.057 11.018 0.000 0.519 0.743
360 +C(city_fe)[T.lac-des-écorces] 0.0133 0.068 0.196 0.845 -0.120 0.146
361 +C(city_fe)[T.lac-etchemin] -0.2591 0.057 -4.521 0.000 -0.371 -0.147
362 +C(city_fe)[T.lachute] 0.1603 0.038 4.268 0.000 0.087 0.234
363 +C(city_fe)[T.lanoraie] 0.1578 0.038 4.195 0.000 0.084 0.232
364 +C(city_fe)[T.laval] 0.4618 0.014 33.004 0.000 0.434 0.489
365 +C(city_fe)[T.lavaltrie] 0.1654 0.039 4.251 0.000 0.089 0.242
366 +C(city_fe)[T.les éboulements] 0.1792 0.064 2.798 0.005 0.054 0.305
367 +C(city_fe)[T.longueuil] 0.3736 0.016 23.052 0.000 0.342 0.405
368 +C(city_fe)[T.lorraine] 0.4812 0.056 8.517 0.000 0.370 0.592
369 +C(city_fe)[T.lévis] 0.1144 0.022 5.189 0.000 0.071 0.158
370 +C(city_fe)[T.magog] 0.3435 0.032 10.704 0.000 0.281 0.406
371 +C(city_fe)[T.mandeville] -0.0847 0.041 -2.084 0.037 -0.164 -0.005
372 +C(city_fe)[T.mansfield-et-pontefract] -0.5178 0.067 -7.737 0.000 -0.649 -0.387
373 +C(city_fe)[T.marieville] 0.2018 0.034 5.959 0.000 0.135 0.268
374 +C(city_fe)[T.mascouche] 0.2837 0.021 13.524 0.000 0.243 0.325
375 +C(city_fe)[T.matane] -0.1801 0.047 -3.836 0.000 -0.272 -0.088
376 +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3725 0.047 7.859 0.000 0.280 0.465
377 +C(city_fe)[T.messines] -0.2559 0.053 -4.805 0.000 -0.360 -0.152
378 +C(city_fe)[T.mille-isles] 0.1934 0.052 3.750 0.000 0.092 0.294
379 +C(city_fe)[T.mirabel] 0.2915 0.020 14.237 0.000 0.251 0.332
380 +C(city_fe)[T.mont-joli] -0.2792 0.099 -2.832 0.005 -0.472 -0.086
381 +C(city_fe)[T.mont-laurier] -0.0419 0.040 -1.042 0.298 -0.121 0.037
382 +C(city_fe)[T.mont-royal] 1.2687 0.054 23.406 0.000 1.162 1.375
383 +C(city_fe)[T.mont-saint-hilaire] 0.5053 0.041 12.197 0.000 0.424 0.586
384 +C(city_fe)[T.mont-tremblant] 0.3895 0.036 10.890 0.000 0.319 0.460
385 +C(city_fe)[T.montréal] 0.7168 0.017 42.732 0.000 0.684 0.750
386 +C(city_fe)[T.morin-heights] 0.2959 0.038 7.784 0.000 0.221 0.370
387 +C(city_fe)[T.nominingue] 0.0840 0.085 0.990 0.322 -0.082 0.250
388 +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.4132 0.043 9.698 0.000 0.330 0.497
389 +C(city_fe)[T.notre-dame-de-pontmain] -0.1086 0.060 -1.801 0.072 -0.227 0.010
390 +C(city_fe)[T.notre-dame-des-bois] -0.3513 0.066 -5.303 0.000 -0.481 -0.221
391 +C(city_fe)[T.notre-dame-des-prairies] 0.0815 0.039 2.105 0.035 0.006 0.157
392 +C(city_fe)[T.notre-dame-du-laus] 0.0340 0.057 0.597 0.550 -0.078 0.146
393 +C(city_fe)[T.orford] 0.3442 0.065 5.321 0.000 0.217 0.471
394 +C(city_fe)[T.papineauville] -0.0271 0.113 -0.241 0.810 -0.248 0.194
395 +C(city_fe)[T.petite-rivière-saint-françois] -0.0295 0.054 -0.544 0.587 -0.136 0.077
396 +C(city_fe)[T.piedmont] 0.3505 0.055 6.383 0.000 0.243 0.458
397 +C(city_fe)[T.pincourt] 0.3096 0.038 8.225 0.000 0.236 0.383
398 +C(city_fe)[T.pointe-claire] 0.6052 0.042 14.568 0.000 0.524 0.687
399 +C(city_fe)[T.pont-rouge] -0.0087 0.045 -0.196 0.845 -0.096 0.079
400 +C(city_fe)[T.prévost] 0.1693 0.027 6.376 0.000 0.117 0.221
401 +C(city_fe)[T.québec] 0.1918 0.018 10.706 0.000 0.157 0.227
402 +C(city_fe)[T.rawdon] 0.0432 0.031 1.373 0.170 -0.018 0.105
403 +C(city_fe)[T.repentigny] 0.2854 0.019 14.912 0.000 0.248 0.323
404 +C(city_fe)[T.rigaud] 0.1455 0.046 3.193 0.001 0.056 0.235
405 +C(city_fe)[T.rimouski] -0.0705 0.033 -2.115 0.034 -0.136 -0.005
406 +C(city_fe)[T.rivière-du-loup] -0.0883 0.056 -1.585 0.113 -0.197 0.021
407 +C(city_fe)[T.rivière-rouge] -0.1303 0.058 -2.240 0.025 -0.244 -0.016
408 +C(city_fe)[T.roberval] -0.1680 0.067 -2.506 0.012 -0.299 -0.037
409 +C(city_fe)[T.rosemère] 0.4821 0.039 12.318 0.000 0.405 0.559
410 +C(city_fe)[T.saguenay] -0.0384 0.023 -1.664 0.096 -0.084 0.007
411 +C(city_fe)[T.saint-adolphe-d'howard] 0.1647 0.045 3.665 0.000 0.077 0.253
412 +C(city_fe)[T.saint-amable] 0.2714 0.033 8.280 0.000 0.207 0.336
413 +C(city_fe)[T.saint-anicet] 0.2760 0.070 3.950 0.000 0.139 0.413
414 +C(city_fe)[T.saint-basile-le-grand] 0.3655 0.044 8.222 0.000 0.278 0.453
415 +C(city_fe)[T.saint-bruno-de-montarville] 0.6120 0.039 15.504 0.000 0.535 0.689
416 +C(city_fe)[T.saint-calixte] 0.0123 0.033 0.367 0.714 -0.053 0.078
417 +C(city_fe)[T.saint-charles-borromée] 0.1447 0.050 2.874 0.004 0.046 0.243
418 +C(city_fe)[T.saint-colomban] 0.1238 0.042 2.966 0.003 0.042 0.206
419 +C(city_fe)[T.saint-constant] 0.3327 0.030 11.057 0.000 0.274 0.392
420 +C(city_fe)[T.saint-donat] 0.3373 0.047 7.155 0.000 0.245 0.430
421 +C(city_fe)[T.saint-eustache] 0.3390 0.036 9.470 0.000 0.269 0.409
422 +C(city_fe)[T.saint-félicien] -0.0495 0.059 -0.832 0.405 -0.166 0.067
423 +C(city_fe)[T.saint-gabriel-de-brandon] 0.0228 0.061 0.371 0.711 -0.098 0.143
424 +C(city_fe)[T.saint-georges] -0.1857 0.038 -4.869 0.000 -0.260 -0.111
425 +C(city_fe)[T.saint-hippolyte] 0.2228 0.041 5.377 0.000 0.142 0.304
426 +C(city_fe)[T.saint-honoré] -0.0445 0.061 -0.728 0.466 -0.164 0.075
427 +C(city_fe)[T.saint-hyacinthe] 0.1802 0.034 5.373 0.000 0.114 0.246
428 +C(city_fe)[T.saint-jean-de-matha] 0.0361 0.068 0.530 0.596 -0.097 0.170
429 +C(city_fe)[T.saint-jean-sur-richelieu] 0.2700 0.020 13.718 0.000 0.231 0.309
430 +C(city_fe)[T.saint-jérôme] 0.1966 0.020 9.627 0.000 0.157 0.237
431 +C(city_fe)[T.saint-lambert] 0.8612 0.048 17.881 0.000 0.767 0.956
432 +C(city_fe)[T.saint-lambert-de-lauzon] 0.0811 0.049 1.641 0.101 -0.016 0.178
433 +C(city_fe)[T.saint-lin/laurentides] 0.0843 0.030 2.770 0.006 0.025 0.144
434 +C(city_fe)[T.saint-marc-des-carrières] -0.1463 0.030 -4.804 0.000 -0.206 -0.087
435 +C(city_fe)[T.saint-michel-des-saints] -0.0296 0.067 -0.444 0.657 -0.160 0.101
436 +C(city_fe)[T.saint-paul-de-l'île-aux-noix] 0.0415 0.102 0.407 0.684 -0.159 0.242
437 +C(city_fe)[T.saint-philippe] 0.4171 0.027 15.486 0.000 0.364 0.470
438 +C(city_fe)[T.saint-raymond] 0.0777 0.068 1.140 0.254 -0.056 0.211
439 +C(city_fe)[T.saint-sauveur] 0.2484 0.034 7.368 0.000 0.182 0.314
440 +C(city_fe)[T.saint-zotique] 0.2794 0.039 7.099 0.000 0.202 0.357
441 +C(city_fe)[T.saint-zénon] -0.0054 0.107 -0.050 0.960 -0.215 0.204
442 +C(city_fe)[T.sainte-adèle] 0.2888 0.049 5.952 0.000 0.194 0.384
443 +C(city_fe)[T.sainte-agathe-des-monts] 0.1307 0.052 2.523 0.012 0.029 0.232
444 +C(city_fe)[T.sainte-anne-des-plaines] 0.2630 0.059 4.451 0.000 0.147 0.379
445 +C(city_fe)[T.sainte-barbe] 0.4472 0.064 6.992 0.000 0.322 0.573
446 +C(city_fe)[T.sainte-brigitte-de-laval] 0.1144 0.040 2.844 0.004 0.036 0.193
447 +C(city_fe)[T.sainte-béatrix] -0.0542 0.044 -1.231 0.218 -0.141 0.032
448 +C(city_fe)[T.sainte-catherine] 0.3326 0.033 10.090 0.000 0.268 0.397
449 +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1289 0.061 2.125 0.034 0.010 0.248
450 +C(city_fe)[T.sainte-julie] 0.4505 0.054 8.316 0.000 0.344 0.557
451 +C(city_fe)[T.sainte-julienne] 0.0339 0.035 0.981 0.327 -0.034 0.102
452 +C(city_fe)[T.sainte-lucie-des-laurentides] 0.1581 0.039 4.057 0.000 0.082 0.234
453 +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0159 0.053 0.298 0.766 -0.089 0.121
454 +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.3085 0.023 13.594 0.000 0.264 0.353
455 +C(city_fe)[T.sainte-sophie] 0.0743 0.038 1.969 0.049 0.000 0.148
456 +C(city_fe)[T.sainte-thérèse] 0.3650 0.041 8.871 0.000 0.284 0.446
457 +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2224 0.060 -3.680 0.000 -0.341 -0.104
458 +C(city_fe)[T.salaberry-de-valleyfield] 0.0989 0.045 2.196 0.028 0.011 0.187
459 +C(city_fe)[T.sept-îles] -0.1031 0.037 -2.808 0.005 -0.175 -0.031
460 +C(city_fe)[T.shawinigan] -0.1030 0.026 -4.024 0.000 -0.153 -0.053
461 +C(city_fe)[T.shefford] 0.2575 0.059 4.350 0.000 0.141 0.373
462 +C(city_fe)[T.sherbrooke] 0.0970 0.017 5.655 0.000 0.063 0.131
463 +C(city_fe)[T.sorel-tracy] 0.0076 0.027 0.280 0.780 -0.045 0.061
464 +C(city_fe)[T.st-colomban] 0.2945 0.071 4.138 0.000 0.155 0.434
465 +C(city_fe)[T.st-hippolyte] 0.1163 0.079 1.468 0.142 -0.039 0.272
466 +C(city_fe)[T.st-jean-sur-richelieu] 0.2196 0.035 6.245 0.000 0.151 0.289
467 +C(city_fe)[T.st-jérôme] 0.2480 0.039 6.409 0.000 0.172 0.324
468 +C(city_fe)[T.st-sauveur] 0.4105 0.091 4.505 0.000 0.232 0.589
469 +C(city_fe)[T.ste-adèle] 0.2046 0.066 3.088 0.002 0.075 0.335
470 +C(city_fe)[T.ste-julienne] 0.0905 0.048 1.903 0.057 -0.003 0.184
471 +C(city_fe)[T.sutton] 0.4707 0.053 8.949 0.000 0.368 0.574
472 +C(city_fe)[T.terrebonne] 0.2919 0.018 16.339 0.000 0.257 0.327
473 +C(city_fe)[T.thetford mines] -0.2933 0.058 -5.069 0.000 -0.407 -0.180
474 +C(city_fe)[T.trois-rivières] 0.0072 0.020 0.357 0.721 -0.032 0.047
475 +C(city_fe)[T.très-saint-rédempteur] 0.5979 0.049 12.126 0.000 0.501 0.695
476 +C(city_fe)[T.témiscouata-sur-le-lac] -0.1648 0.074 -2.223 0.026 -0.310 -0.020
477 +C(city_fe)[T.val-d'or] 0.0067 0.030 0.226 0.821 -0.051 0.065
478 +C(city_fe)[T.val-david] 0.0280 0.051 0.551 0.582 -0.072 0.127
479 +C(city_fe)[T.val-des-monts] 0.0953 0.027 3.504 0.000 0.042 0.149
480 +C(city_fe)[T.val-morin] 0.0441 0.060 0.734 0.463 -0.074 0.162
481 +C(city_fe)[T.vaudreuil-dorion] 0.2448 0.026 9.301 0.000 0.193 0.296
482 +C(city_fe)[T.venise-en-québec] 0.1940 0.041 4.773 0.000 0.114 0.274
483 +C(city_fe)[T.victoriaville] -0.0391 0.025 -1.593 0.111 -0.087 0.009
484 +C(city_fe)[T.waterloo] 0.1392 0.045 3.091 0.002 0.051 0.227
485 +C(city_fe)[T.weedon] -0.1707 0.060 -2.823 0.005 -0.289 -0.052
486 +C(city_fe)[T.wentworth-nord] 0.0653 0.073 0.892 0.372 -0.078 0.209
487 +C(city_fe)[T.westmount] 1.6044 0.038 41.845 0.000 1.529 1.680
488 +ln_lot 0.0424 0.003 13.299 0.000 0.036 0.049
489 +has_lot -0.0854 0.014 -5.897 0.000 -0.114 -0.057
490 +beds 0.0237 0.004 5.676 0.000 0.016 0.032
491 +baths 0.1352 0.005 25.581 0.000 0.125 0.146
492 +parking 0.0154 0.002 8.288 0.000 0.012 0.019
493 +storeys 0.0020 0.097 0.021 0.983 -0.188 0.192
494 +age -0.0071 0.000 -18.661 0.000 -0.008 -0.006
495 +age2 3.559e-05 2.82e-06 12.637 0.000 3.01e-05 4.11e-05
496 +has_age -0.1532 0.018 -8.628 0.000 -0.188 -0.118
497 +pool 0.0548 0.007 8.093 0.000 0.042 0.068
498 +waterfront 0.1091 0.018 6.132 0.000 0.074 0.144
499 +garage 0.0533 0.006 9.016 0.000 0.042 0.065
500 +fireplace 0.0116 0.007 1.768 0.077 -0.001 0.024
501 +central_air 0.0651 0.009 7.208 0.000 0.047 0.083
502 +finished_bsmt -0.0169 0.007 -2.438 0.015 -0.031 -0.003
503 +near_water 0.0769 0.021 3.748 0.000 0.037 0.117
504 +cul_de_sac 0.0316 0.009 3.509 0.000 0.014 0.049
505 +golf 0.0197 0.006 3.395 0.001 0.008 0.031
506 +ski 0.0468 0.010 4.760 0.000 0.028 0.066
507 +near_park 0.0043 0.008 0.544 0.587 -0.011 0.020
508 +near_school -0.0571 0.009 -6.578 0.000 -0.074 -0.040
509 +near_highway 0.0499 0.006 7.721 0.000 0.037 0.063
510 +transit 0.0667 0.008 8.828 0.000 0.052 0.082
511 +no_rear_neighbor -0.0478 0.009 -5.132 0.000 -0.066 -0.030
512 +n_images_c 0.0059 0.000 19.406 0.000 0.005 0.006
513 +virtual_tour 0.0277 0.016 1.751 0.080 -0.003 0.059
514 +n_rooms 0.0262 0.001 17.890 0.000 0.023 0.029
515 +n_levels_r 0.0432 0.048 0.897 0.370 -0.051 0.138
516 +has_primary_bed 0.0696 0.010 7.033 0.000 0.050 0.089
517 +has_basement -0.1168 0.044 -2.657 0.008 -0.203 -0.031
518 +r_office 0.0348 0.008 4.562 0.000 0.020 0.050
519 +r_family 0.0372 0.009 4.122 0.000 0.020 0.055
520 +r_workshop -0.0346 0.009 -3.928 0.000 -0.052 -0.017
521 +r_winecellar 0.1413 0.026 5.384 0.000 0.090 0.193
522 +r_sunroom 0.0166 0.014 1.224 0.221 -0.010 0.043
523 +r_mezzanine 0.0202 0.018 1.114 0.265 -0.015 0.056
524 +r_library 0.1466 0.070 2.082 0.037 0.009 0.285
525 +r_laundry 0.0382 0.006 6.301 0.000 0.026 0.050
526 +==============================================================================
527 +Omnibus: 2418.650 Durbin-Watson: 1.758
528 +Prob(Omnibus): 0.000 Jarque-Bera (JB): 15665.854
529 +Skew: 0.429 Prob(JB): 0.00
530 +Kurtosis: 7.351 Cond. No. 2.81e+16
531 +==============================================================================
532 +
533 +Notes:
534 +[1] Standard Errors are heteroscedasticity robust (HC3)
535 +[2] The smallest eigenvalue is 5.65e-22. This might indicate that there are
536 +strong multicollinearity problems or that the design matrix is singular.
\ No newline at end of file
added results/hedonic_v4_summary.txt +576 −0
@@ -0,0 +1,576 @@
1 +=== A+GPT ===
2 + OLS Regression Results
3 +==============================================================================
4 +Dep. Variable: ln_price R-squared: 0.725
5 +Model: OLS Adj. R-squared: 0.717
6 +Method: Least Squares F-statistic: 6.503e+04
7 +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00
8 +Time: 08:11:28 Log-Likelihood: -1724.8
9 +No. Observations: 9446 AIC: 3970.
10 +Df Residuals: 9186 BIC: 5829.
11 +Df Model: 259
12 +Covariance Type: HC3
13 +========================================================================================================================
14 + coef std err z P>|z| [0.025 0.975]
15 +------------------------------------------------------------------------------------------------------------------------
16 +Intercept 4.8574 0.047 102.393 0.000 4.764 4.950
17 +C(source)[T.realtor] -0.2485 0.023 -10.988 0.000 -0.293 -0.204
18 +C(ptype_fe)[T.Country Home] 0.0088 0.051 0.174 0.862 -0.090 0.108
19 +C(ptype_fe)[T.Home] -0.0125 0.014 -0.905 0.365 -0.040 0.015
20 +C(ptype_fe)[T.House] -0.0462 0.012 -3.711 0.000 -0.071 -0.022
21 +C(ptype_fe)[T.Intergenerational] 0.0010 0.038 0.026 0.979 -0.073 0.075
22 +C(ptype_fe)[T.Other] 0.2105 0.120 1.749 0.080 -0.025 0.446
23 +C(ptype_fe)[T.Row / Townhouse] -0.2023 0.016 -12.633 0.000 -0.234 -0.171
24 +C(ptype_fe)[T.Semi Detached] -0.0788 0.016 -5.067 0.000 -0.109 -0.048
25 +C(ptype_fe)[T.Townhouse] -0.0203 0.026 -0.770 0.442 -0.072 0.031
26 +C(city_fe)[T.acton vale] -0.0447 0.060 -0.747 0.455 -0.162 0.072
27 +C(city_fe)[T.alma] -0.0355 0.050 -0.704 0.481 -0.134 0.063
28 +C(city_fe)[T.amherst] -0.0742 0.075 -0.983 0.325 -0.222 0.074
29 +C(city_fe)[T.amos] -0.0204 0.062 -0.327 0.744 -0.143 0.102
30 +C(city_fe)[T.austin] 0.2867 0.088 3.250 0.001 0.114 0.460
31 +C(city_fe)[T.baie-comeau] -0.3689 0.079 -4.688 0.000 -0.523 -0.215
32 +C(city_fe)[T.baie-saint-paul] -0.0794 0.062 -1.280 0.200 -0.201 0.042
33 +C(city_fe)[T.beaconsfield] 0.6183 0.050 12.411 0.000 0.521 0.716
34 +C(city_fe)[T.beauport] 0.0450 0.035 1.289 0.198 -0.023 0.113
35 +C(city_fe)[T.beloeil] 0.2256 0.052 4.347 0.000 0.124 0.327
36 +C(city_fe)[T.blainville] 0.3035 0.031 9.859 0.000 0.243 0.364
37 +C(city_fe)[T.blue sea] -0.0483 0.058 -0.837 0.403 -0.161 0.065
38 +C(city_fe)[T.bois-des-filion] 0.1607 0.035 4.566 0.000 0.092 0.230
39 +C(city_fe)[T.boisbriand] 0.2550 0.043 5.892 0.000 0.170 0.340
40 +C(city_fe)[T.boischatel] 0.1434 0.074 1.929 0.054 -0.002 0.289
41 +C(city_fe)[T.boucherville] 0.3369 0.039 8.695 0.000 0.261 0.413
42 +C(city_fe)[T.bromont] 0.3139 0.035 8.946 0.000 0.245 0.383
43 +C(city_fe)[T.brossard] 0.4180 0.030 13.774 0.000 0.359 0.477
44 +C(city_fe)[T.bécancour] -0.1107 0.039 -2.846 0.004 -0.187 -0.034
45 +C(city_fe)[T.cantley] 0.1315 0.033 4.040 0.000 0.068 0.195
46 +C(city_fe)[T.carignan] 0.2473 0.091 2.724 0.006 0.069 0.425
47 +C(city_fe)[T.chandler] -0.2870 0.136 -2.106 0.035 -0.554 -0.020
48 +C(city_fe)[T.chateauguay] 0.2320 0.037 6.320 0.000 0.160 0.304
49 +C(city_fe)[T.chelsea] 0.2610 0.038 6.931 0.000 0.187 0.335
50 +C(city_fe)[T.chertsey] -0.0870 0.060 -1.458 0.145 -0.204 0.030
51 +C(city_fe)[T.chicoutimi] -0.1131 0.042 -2.703 0.007 -0.195 -0.031
52 +C(city_fe)[T.chomedey] 0.3214 0.048 6.746 0.000 0.228 0.415
53 +C(city_fe)[T.châteauguay] 0.2100 0.040 5.244 0.000 0.132 0.289
54 +C(city_fe)[T.contrecoeur] 0.0760 0.059 1.284 0.199 -0.040 0.192
55 +C(city_fe)[T.cowansville] 0.1233 0.043 2.845 0.004 0.038 0.208
56 +C(city_fe)[T.crabtree] 0.0300 0.090 0.334 0.738 -0.146 0.206
57 +C(city_fe)[T.côte-saint-luc] 0.6067 0.051 11.909 0.000 0.507 0.707
58 +C(city_fe)[T.dolbeau-mistassini] -0.2804 0.086 -3.253 0.001 -0.449 -0.111
59 +C(city_fe)[T.dollard-des-ormeaux] 0.4198 0.039 10.848 0.000 0.344 0.496
60 +C(city_fe)[T.donnacona] 0.2326 0.039 5.989 0.000 0.156 0.309
61 +C(city_fe)[T.dorval] 0.5505 0.048 11.354 0.000 0.455 0.645
62 +C(city_fe)[T.drummondville] -0.0231 0.021 -1.108 0.268 -0.064 0.018
63 +C(city_fe)[T.duhamel] 0.1073 0.071 1.511 0.131 -0.032 0.246
64 +C(city_fe)[T.dunham] 0.2692 0.068 3.981 0.000 0.137 0.402
65 +C(city_fe)[T.eastman] 0.2749 0.055 5.019 0.000 0.168 0.382
66 +C(city_fe)[T.entrelacs] 0.2622 0.126 2.080 0.037 0.015 0.509
67 +C(city_fe)[T.farnham] 0.0785 0.058 1.343 0.179 -0.036 0.193
68 +C(city_fe)[T.fossambault-sur-le-lac] 0.2222 0.078 2.844 0.004 0.069 0.375
69 +C(city_fe)[T.gaspé] -0.2403 0.071 -3.372 0.001 -0.380 -0.101
70 +C(city_fe)[T.gatineau] 0.1031 0.013 8.192 0.000 0.078 0.128
71 +C(city_fe)[T.gore] 0.0422 0.099 0.425 0.671 -0.152 0.237
72 +C(city_fe)[T.granby] 0.1272 0.024 5.254 0.000 0.080 0.175
73 +C(city_fe)[T.grenville-sur-la-rouge] 0.1171 0.082 1.433 0.152 -0.043 0.277
74 +C(city_fe)[T.hampstead] 1.0869 0.057 19.051 0.000 0.975 1.199
75 +C(city_fe)[T.harrington] 0.1032 0.145 0.713 0.476 -0.180 0.387
76 +C(city_fe)[T.hudson] 0.2472 0.179 1.383 0.167 -0.103 0.598
77 +C(city_fe)[T.joliette] -0.0242 0.070 -0.346 0.729 -0.161 0.113
78 +C(city_fe)[T.jonquière] -0.1325 0.044 -3.001 0.003 -0.219 -0.046
79 +C(city_fe)[T.kirkland] 0.3831 0.038 10.128 0.000 0.309 0.457
80 +C(city_fe)[T.l'ange-gardien] 0.0259 0.163 0.159 0.874 -0.294 0.346
81 +C(city_fe)[T.l'assomption] 0.1521 0.063 2.426 0.015 0.029 0.275
82 +C(city_fe)[T.l'île-perrot] 0.4022 0.061 6.630 0.000 0.283 0.521
83 +C(city_fe)[T.la conception] 0.3810 0.055 6.871 0.000 0.272 0.490
84 +C(city_fe)[T.la malbaie] -0.1254 0.063 -1.980 0.048 -0.250 -0.001
85 +C(city_fe)[T.la minerve] 0.0403 0.069 0.585 0.559 -0.095 0.175
86 +C(city_fe)[T.la prairie] 0.3665 0.036 10.091 0.000 0.295 0.438
87 +C(city_fe)[T.la pêche] 0.0226 0.036 0.621 0.535 -0.049 0.094
88 +C(city_fe)[T.la tuque] -0.3102 0.065 -4.784 0.000 -0.437 -0.183
89 +C(city_fe)[T.labelle] 0.0982 0.113 0.868 0.386 -0.124 0.320
90 +C(city_fe)[T.lac-beauport] 0.3579 0.088 4.045 0.000 0.185 0.531
91 +C(city_fe)[T.lac-brome] 0.5121 0.063 8.090 0.000 0.388 0.636
92 +C(city_fe)[T.lac-des-écorces] -0.0499 0.154 -0.324 0.746 -0.352 0.252
93 +C(city_fe)[T.lac-etchemin] -0.2092 0.144 -1.457 0.145 -0.491 0.072
94 +C(city_fe)[T.lachute] 0.1368 0.073 1.878 0.060 -0.006 0.280
95 +C(city_fe)[T.lanoraie] -0.1106 0.057 -1.941 0.052 -0.222 0.001
96 +C(city_fe)[T.laval] 0.4064 0.020 19.826 0.000 0.366 0.447
97 +C(city_fe)[T.lavaltrie] 0.1330 0.058 2.301 0.021 0.020 0.246
98 +C(city_fe)[T.les éboulements] 0.1814 0.088 2.065 0.039 0.009 0.354
99 +C(city_fe)[T.longueuil] 0.3166 0.019 16.430 0.000 0.279 0.354
100 +C(city_fe)[T.lorraine] 0.1245 0.106 1.172 0.241 -0.084 0.333
101 +C(city_fe)[T.lévis] 0.0630 0.025 2.508 0.012 0.014 0.112
102 +C(city_fe)[T.magog] 0.3398 0.033 10.422 0.000 0.276 0.404
103 +C(city_fe)[T.mandeville] -0.1594 0.080 -1.980 0.048 -0.317 -0.002
104 +C(city_fe)[T.mansfield-et-pontefract] -0.4148 0.082 -5.049 0.000 -0.576 -0.254
105 +C(city_fe)[T.marieville] 0.1990 0.092 2.158 0.031 0.018 0.380
106 +C(city_fe)[T.mascouche] 0.1885 0.042 4.469 0.000 0.106 0.271
107 +C(city_fe)[T.matane] -0.2595 0.037 -6.937 0.000 -0.333 -0.186
108 +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.3424 0.036 9.489 0.000 0.272 0.413
109 +C(city_fe)[T.messines] -0.1621 0.058 -2.784 0.005 -0.276 -0.048
110 +C(city_fe)[T.mille-isles] 0.2411 0.055 4.390 0.000 0.133 0.349
111 +C(city_fe)[T.mirabel] 0.2446 0.027 8.899 0.000 0.191 0.298
112 +C(city_fe)[T.mont-joli] -0.2369 0.183 -1.292 0.196 -0.596 0.123
113 +C(city_fe)[T.mont-laurier] -0.0547 0.037 -1.490 0.136 -0.127 0.017
114 +C(city_fe)[T.mont-royal] 1.0680 0.061 17.561 0.000 0.949 1.187
115 +C(city_fe)[T.mont-saint-hilaire] 0.3694 0.102 3.633 0.000 0.170 0.569
116 +C(city_fe)[T.mont-tremblant] 0.4024 0.040 10.104 0.000 0.324 0.480
117 +C(city_fe)[T.montréal] 0.6861 0.021 32.059 0.000 0.644 0.728
118 +C(city_fe)[T.morin-heights] 0.3081 0.058 5.336 0.000 0.195 0.421
119 +C(city_fe)[T.nominingue] 0.1255 0.095 1.322 0.186 -0.061 0.311
120 +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.2077 0.066 3.151 0.002 0.079 0.337
121 +C(city_fe)[T.notre-dame-de-pontmain] 0.0126 0.061 0.206 0.837 -0.108 0.133
122 +C(city_fe)[T.notre-dame-des-bois] -0.1992 0.075 -2.650 0.008 -0.347 -0.052
123 +C(city_fe)[T.notre-dame-des-prairies] 0.0993 0.116 0.853 0.394 -0.129 0.327
124 +C(city_fe)[T.notre-dame-du-laus] 0.0915 0.053 1.742 0.081 -0.011 0.194
125 +C(city_fe)[T.orford] 0.3190 0.062 5.126 0.000 0.197 0.441
126 +C(city_fe)[T.papineauville] -0.0746 0.124 -0.602 0.547 -0.317 0.168
127 +C(city_fe)[T.petite-rivière-saint-françois] -0.0072 0.088 -0.083 0.934 -0.179 0.165
128 +C(city_fe)[T.piedmont] 0.4510 0.061 7.336 0.000 0.330 0.571
129 +C(city_fe)[T.pincourt] 0.2002 0.052 3.826 0.000 0.098 0.303
130 +C(city_fe)[T.pointe-claire] 0.4417 0.037 11.947 0.000 0.369 0.514
131 +C(city_fe)[T.pont-rouge] 0.1126 0.058 1.925 0.054 -0.002 0.227
132 +C(city_fe)[T.prévost] 0.1344 0.038 3.571 0.000 0.061 0.208
133 +C(city_fe)[T.québec] 0.1808 0.020 9.213 0.000 0.142 0.219
134 +C(city_fe)[T.rawdon] 0.0837 0.045 1.854 0.064 -0.005 0.172
135 +C(city_fe)[T.repentigny] 0.1814 0.033 5.527 0.000 0.117 0.246
136 +C(city_fe)[T.rigaud] 0.2917 0.064 4.548 0.000 0.166 0.417
137 +C(city_fe)[T.rimouski] -0.0524 0.036 -1.452 0.147 -0.123 0.018
138 +C(city_fe)[T.rivière-du-loup] -0.1793 0.089 -2.009 0.044 -0.354 -0.004
139 +C(city_fe)[T.rivière-rouge] -0.0220 0.071 -0.309 0.758 -0.161 0.118
140 +C(city_fe)[T.roberval] -0.2385 0.076 -3.129 0.002 -0.388 -0.089
141 +C(city_fe)[T.rosemère] 0.4051 0.088 4.628 0.000 0.234 0.577
142 +C(city_fe)[T.saguenay] -0.0684 0.044 -1.544 0.123 -0.155 0.018
143 +C(city_fe)[T.saint-adolphe-d'howard] 0.2906 0.054 5.420 0.000 0.186 0.396
144 +C(city_fe)[T.saint-amable] 0.2992 0.084 3.543 0.000 0.134 0.465
145 +C(city_fe)[T.saint-anicet] 0.0324 0.081 0.400 0.689 -0.126 0.191
146 +C(city_fe)[T.saint-basile-le-grand] 0.3769 0.091 4.138 0.000 0.198 0.555
147 +C(city_fe)[T.saint-bruno-de-montarville] 0.6511 0.071 9.114 0.000 0.511 0.791
148 +C(city_fe)[T.saint-calixte] 0.1676 0.088 1.904 0.057 -0.005 0.340
149 +C(city_fe)[T.saint-charles-borromée] 0.1796 0.174 1.034 0.301 -0.161 0.520
150 +C(city_fe)[T.saint-colomban] 0.2115 0.068 3.114 0.002 0.078 0.345
151 +C(city_fe)[T.saint-constant] 0.2084 0.056 3.745 0.000 0.099 0.317
152 +C(city_fe)[T.saint-donat] 0.2478 0.066 3.738 0.000 0.118 0.378
153 +C(city_fe)[T.saint-eustache] 0.4254 0.068 6.255 0.000 0.292 0.559
154 +C(city_fe)[T.saint-félicien] -0.1964 0.320 -0.613 0.540 -0.824 0.431
155 +C(city_fe)[T.saint-gabriel-de-brandon] -0.2016 0.100 -2.014 0.044 -0.398 -0.005
156 +C(city_fe)[T.saint-georges] -0.0958 0.067 -1.426 0.154 -0.228 0.036
157 +C(city_fe)[T.saint-hippolyte] 0.3319 0.079 4.193 0.000 0.177 0.487
158 +C(city_fe)[T.saint-honoré] -0.1112 0.075 -1.478 0.139 -0.259 0.036
159 +C(city_fe)[T.saint-hyacinthe] 0.1609 0.065 2.481 0.013 0.034 0.288
160 +C(city_fe)[T.saint-jean-de-matha] 0.0935 0.157 0.597 0.550 -0.213 0.400
161 +C(city_fe)[T.saint-jean-sur-richelieu] 0.1990 0.031 6.451 0.000 0.139 0.259
162 +C(city_fe)[T.saint-jérôme] 0.1716 0.037 4.677 0.000 0.100 0.243
163 +C(city_fe)[T.saint-lambert] 0.6578 0.069 9.598 0.000 0.523 0.792
164 +C(city_fe)[T.saint-lambert-de-lauzon] 0.0260 0.087 0.298 0.766 -0.145 0.197
165 +C(city_fe)[T.saint-lin/laurentides] 0.1117 0.124 0.898 0.369 -0.132 0.356
166 +C(city_fe)[T.saint-marc-des-carrières] -0.2794 0.048 -5.829 0.000 -0.373 -0.185
167 +C(city_fe)[T.saint-michel-des-saints] -0.2187 0.085 -2.572 0.010 -0.385 -0.052
168 +C(city_fe)[T.saint-paul-de-l'île-aux-noix] -0.1315 0.223 -0.591 0.555 -0.568 0.305
169 +C(city_fe)[T.saint-philippe] 0.3302 0.040 8.276 0.000 0.252 0.408
170 +C(city_fe)[T.saint-raymond] 0.1480 0.085 1.750 0.080 -0.018 0.314
171 +C(city_fe)[T.saint-sauveur] 0.3199 0.059 5.434 0.000 0.205 0.435
172 +C(city_fe)[T.saint-zotique] 0.4330 0.073 5.923 0.000 0.290 0.576
173 +C(city_fe)[T.saint-zénon] -0.2586 76.240 -0.003 0.997 -149.686 149.169
174 +C(city_fe)[T.sainte-adèle] 0.3672 0.077 4.781 0.000 0.217 0.518
175 +C(city_fe)[T.sainte-agathe-des-monts] 0.1184 0.092 1.291 0.197 -0.061 0.298
176 +C(city_fe)[T.sainte-anne-des-plaines] 0.9383 0.778 1.206 0.228 -0.586 2.463
177 +C(city_fe)[T.sainte-barbe] 0.1802 0.339 0.532 0.595 -0.483 0.844
178 +C(city_fe)[T.sainte-brigitte-de-laval] 0.1292 0.052 2.507 0.012 0.028 0.230
179 +C(city_fe)[T.sainte-béatrix] 0.0314 0.045 0.703 0.482 -0.056 0.119
180 +C(city_fe)[T.sainte-catherine] 0.4067 0.072 5.638 0.000 0.265 0.548
181 +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1900 0.066 2.877 0.004 0.061 0.319
182 +C(city_fe)[T.sainte-julie] 0.4254 0.085 4.980 0.000 0.258 0.593
183 +C(city_fe)[T.sainte-julienne] 0.0580 0.073 0.798 0.425 -0.084 0.200
184 +C(city_fe)[T.sainte-lucie-des-laurentides] 0.3427 0.064 5.320 0.000 0.216 0.469
185 +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0785 0.052 1.501 0.133 -0.024 0.181
186 +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.2436 0.036 6.705 0.000 0.172 0.315
187 +C(city_fe)[T.sainte-sophie] 0.1004 0.064 1.561 0.119 -0.026 0.227
188 +C(city_fe)[T.sainte-thérèse] 0.3162 0.085 3.721 0.000 0.150 0.483
189 +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2048 0.075 -2.730 0.006 -0.352 -0.058
190 +C(city_fe)[T.salaberry-de-valleyfield] 0.0382 0.046 0.826 0.409 -0.052 0.129
191 +C(city_fe)[T.sept-îles] -0.1273 0.079 -1.615 0.106 -0.282 0.027
192 +C(city_fe)[T.shawinigan] -0.1806 0.051 -3.520 0.000 -0.281 -0.080
193 +C(city_fe)[T.shefford] 0.1555 0.089 1.741 0.082 -0.020 0.331
194 +C(city_fe)[T.sherbrooke] 0.0978 0.018 5.524 0.000 0.063 0.133
195 +C(city_fe)[T.sorel-tracy] -0.1019 0.044 -2.339 0.019 -0.187 -0.017
196 +C(city_fe)[T.st-colomban] 0.2146 0.057 3.780 0.000 0.103 0.326
197 +C(city_fe)[T.st-hippolyte] 0.1331 0.055 2.435 0.015 0.026 0.240
198 +C(city_fe)[T.st-jean-sur-richelieu] 0.1735 0.032 5.503 0.000 0.112 0.235
199 +C(city_fe)[T.st-jérôme] 0.1551 0.031 5.014 0.000 0.094 0.216
200 +C(city_fe)[T.st-sauveur] 0.2612 0.049 5.298 0.000 0.165 0.358
201 +C(city_fe)[T.ste-adèle] 0.1396 0.069 2.012 0.044 0.004 0.276
202 +C(city_fe)[T.ste-julienne] 0.0377 0.035 1.083 0.279 -0.031 0.106
203 +C(city_fe)[T.sutton] 0.4078 0.066 6.196 0.000 0.279 0.537
204 +C(city_fe)[T.terrebonne] 0.1965 0.028 6.923 0.000 0.141 0.252
205 +C(city_fe)[T.thetford mines] -0.3029 0.117 -2.596 0.009 -0.532 -0.074
206 +C(city_fe)[T.trois-rivières] -0.0810 0.022 -3.706 0.000 -0.124 -0.038
207 +C(city_fe)[T.très-saint-rédempteur] 0.4839 0.032 15.354 0.000 0.422 0.546
208 +C(city_fe)[T.témiscouata-sur-le-lac] -0.3066 0.124 -2.473 0.013 -0.550 -0.064
209 +C(city_fe)[T.val-d'or] 0.0788 0.031 2.543 0.011 0.018 0.139
210 +C(city_fe)[T.val-david] 0.0115 0.117 0.099 0.921 -0.217 0.240
211 +C(city_fe)[T.val-des-monts] 0.1031 0.025 4.142 0.000 0.054 0.152
212 +C(city_fe)[T.val-morin] -0.1267 0.093 -1.357 0.175 -0.310 0.056
213 +C(city_fe)[T.vaudreuil-dorion] 0.0802 0.044 1.807 0.071 -0.007 0.167
214 +C(city_fe)[T.venise-en-québec] 0.0614 0.082 0.744 0.457 -0.100 0.223
215 +C(city_fe)[T.victoriaville] -0.0752 0.029 -2.593 0.010 -0.132 -0.018
216 +C(city_fe)[T.waterloo] 0.0905 0.056 1.608 0.108 -0.020 0.201
217 +C(city_fe)[T.weedon] -0.1348 0.080 -1.675 0.094 -0.293 0.023
218 +C(city_fe)[T.wentworth-nord] 0.1764 0.090 1.951 0.051 -0.001 0.354
219 +C(city_fe)[T.westmount] 1.3152 0.041 31.987 0.000 1.235 1.396
220 +ln_sqft 0.4106 0.013 30.648 0.000 0.384 0.437
221 +ln_lot 0.0253 0.004 6.506 0.000 0.018 0.033
222 +has_lot -0.0201 0.015 -1.369 0.171 -0.049 0.009
223 +beds 0.0057 0.004 1.361 0.173 -0.003 0.014
224 +baths 0.0867 0.007 13.252 0.000 0.074 0.099
225 +parking 0.0112 0.002 7.312 0.000 0.008 0.014
226 +storeys -0.0018 0.009 -0.199 0.843 -0.019 0.016
227 +age -0.0053 0.000 -11.284 0.000 -0.006 -0.004
228 +age2 2.612e-05 3.72e-06 7.024 0.000 1.88e-05 3.34e-05
229 +has_age -0.1623 0.022 -7.477 0.000 -0.205 -0.120
230 +pool 0.0059 0.011 0.526 0.599 -0.016 0.028
231 +waterfront 0.0166 0.022 0.751 0.452 -0.027 0.060
232 +garage 0.0025 0.021 0.119 0.905 -0.038 0.043
233 +fireplace -0.0611 0.016 -3.776 0.000 -0.093 -0.029
234 +central_air 0.0785 0.012 6.669 0.000 0.055 0.102
235 +finished_bsmt -0.0316 0.011 -2.757 0.006 -0.054 -0.009
236 +near_water 0.0910 0.024 3.842 0.000 0.045 0.137
237 +cul_de_sac 0.0268 0.012 2.317 0.020 0.004 0.050
238 +golf 0.0232 0.007 3.268 0.001 0.009 0.037
239 +ski 0.0139 0.013 1.066 0.286 -0.012 0.040
240 +near_park 0.0039 0.011 0.340 0.734 -0.018 0.026
241 +near_school -0.0637 0.011 -5.687 0.000 -0.086 -0.042
242 +near_highway 0.0475 0.008 5.929 0.000 0.032 0.063
243 +transit 0.1036 0.009 11.762 0.000 0.086 0.121
244 +no_rear_neighbor -0.0396 0.010 -4.038 0.000 -0.059 -0.020
245 +n_images_c 0.0044 0.000 14.018 0.000 0.004 0.005
246 +virtual_tour 0.0324 0.022 1.460 0.144 -0.011 0.076
247 +n_rooms 0.0170 0.002 9.265 0.000 0.013 0.021
248 +n_levels_r -0.0226 0.009 -2.526 0.012 -0.040 -0.005
249 +has_primary_bed 0.0753 0.016 4.839 0.000 0.045 0.106
250 +has_basement -0.0169 0.015 -1.148 0.251 -0.046 0.012
251 +r_office 0.0308 0.011 2.735 0.006 0.009 0.053
252 +r_family 0.0359 0.011 3.375 0.001 0.015 0.057
253 +r_workshop -0.0386 0.014 -2.828 0.005 -0.065 -0.012
254 +r_winecellar 0.1108 0.036 3.073 0.002 0.040 0.181
255 +r_sunroom 0.0007 0.021 0.033 0.974 -0.040 0.042
256 +r_mezzanine 0.0175 0.025 0.690 0.490 -0.032 0.067
257 +r_library 0.0971 0.083 1.171 0.242 -0.065 0.260
258 +r_laundry 0.0178 0.009 1.980 0.048 0.000 0.035
259 +g_cond 0.0115 0.004 2.567 0.010 0.003 0.020
260 +g_hardwood 0.0256 0.008 3.211 0.001 0.010 0.041
261 +g_quartz_granite 0.0186 0.011 1.771 0.077 -0.002 0.039
262 +g_view_water 0.0132 0.016 0.845 0.398 -0.017 0.044
263 +has_gpt 4.8574 0.047 102.393 0.000 4.764 4.950
264 +g_turnkey 0.0198 0.008 2.562 0.010 0.005 0.035
265 +g_recently_renovated 0.0050 0.010 0.513 0.608 -0.014 0.024
266 +g_garage 0.0235 0.021 1.131 0.258 -0.017 0.064
267 +g_pool 0.0492 0.012 4.204 0.000 0.026 0.072
268 +g_waterfront 0.0744 0.018 4.135 0.000 0.039 0.110
269 +g_finished_basement 0.0189 0.009 2.203 0.028 0.002 0.036
270 +g_fireplace 0.0907 0.016 5.618 0.000 0.059 0.122
271 +g_central_ac 0.0140 0.010 1.406 0.160 -0.006 0.034
272 +g_new_construction 0.0680 0.014 4.909 0.000 0.041 0.095
273 +g_income_property -0.0331 0.011 -3.083 0.002 -0.054 -0.012
274 +g_renovated_kitchen 0.0298 0.010 2.906 0.004 0.010 0.050
275 +g_renovated_bathroom -0.0060 0.010 -0.614 0.539 -0.025 0.013
276 +g_large_lot -0.0033 0.009 -0.370 0.711 -0.021 0.014
277 +g_quiet_area -0.0028 0.008 -0.369 0.712 -0.018 0.012
278 +==============================================================================
279 +Omnibus: 426.401 Durbin-Watson: 1.855
280 +Prob(Omnibus): 0.000 Jarque-Bera (JB): 1443.982
281 +Skew: 0.076 Prob(JB): 0.00
282 +Kurtosis: 4.909 Cond. No. 2.08e+16
283 +==============================================================================
284 +
285 +Notes:
286 +[1] Standard Errors are heteroscedasticity robust (HC3)
287 +[2] The smallest eigenvalue is 3.84e-22. This might indicate that there are
288 +strong multicollinearity problems or that the design matrix is singular.
289 +
290 +=== B+GPT ===
291 + OLS Regression Results
292 +==============================================================================
293 +Dep. Variable: ln_price R-squared: 0.684
294 +Model: OLS Adj. R-squared: 0.680
295 +Method: Least Squares F-statistic: 1.873e+05
296 +Date: Fri, 03 Jul 2026 Prob (F-statistic): 0.00
297 +Time: 08:11:28 Log-Likelihood: -6089.2
298 +No. Observations: 19123 AIC: 1.270e+04
299 +Df Residuals: 18864 BIC: 1.473e+04
300 +Df Model: 258
301 +Covariance Type: HC3
302 +========================================================================================================================
303 + coef std err z P>|z| [0.025 0.975]
304 +------------------------------------------------------------------------------------------------------------------------
305 +Intercept 6.1815 0.057 108.357 0.000 6.070 6.293
306 +C(source)[T.realtor] -0.3800 0.021 -17.682 0.000 -0.422 -0.338
307 +C(ptype_fe)[T.Country Home] 0.0525 0.067 0.788 0.431 -0.078 0.183
308 +C(ptype_fe)[T.Home] 0.0936 0.081 1.154 0.249 -0.065 0.253
309 +C(ptype_fe)[T.House] -0.0724 0.014 -5.164 0.000 -0.100 -0.045
310 +C(ptype_fe)[T.Intergenerational] 0.1435 0.063 2.291 0.022 0.021 0.266
311 +C(ptype_fe)[T.Other] 0.2526 0.121 2.086 0.037 0.015 0.490
312 +C(ptype_fe)[T.Row / Townhouse] -0.3076 0.014 -21.318 0.000 -0.336 -0.279
313 +C(ptype_fe)[T.Semi Detached] -0.0284 0.057 -0.501 0.616 -0.140 0.083
314 +C(ptype_fe)[T.Townhouse] 0.0700 0.098 0.713 0.476 -0.122 0.262
315 +C(city_fe)[T.acton vale] -0.0098 0.034 -0.284 0.776 -0.077 0.058
316 +C(city_fe)[T.alma] -0.0584 0.054 -1.079 0.281 -0.165 0.048
317 +C(city_fe)[T.amherst] -0.0225 0.049 -0.456 0.648 -0.119 0.074
318 +C(city_fe)[T.amos] -0.1343 0.044 -3.020 0.003 -0.222 -0.047
319 +C(city_fe)[T.austin] 0.4205 0.070 5.972 0.000 0.282 0.559
320 +C(city_fe)[T.baie-comeau] -0.3294 0.069 -4.787 0.000 -0.464 -0.195
321 +C(city_fe)[T.baie-saint-paul] 0.1285 0.077 1.672 0.094 -0.022 0.279
322 +C(city_fe)[T.beaconsfield] 0.7858 0.037 21.069 0.000 0.713 0.859
323 +C(city_fe)[T.beauport] 0.0776 0.037 2.076 0.038 0.004 0.151
324 +C(city_fe)[T.beloeil] 0.3812 0.049 7.840 0.000 0.286 0.476
325 +C(city_fe)[T.blainville] 0.4461 0.025 17.924 0.000 0.397 0.495
326 +C(city_fe)[T.blue sea] -0.1151 0.062 -1.863 0.062 -0.236 0.006
327 +C(city_fe)[T.bois-des-filion] 0.2942 0.048 6.129 0.000 0.200 0.388
328 +C(city_fe)[T.boisbriand] 0.3654 0.038 9.665 0.000 0.291 0.439
329 +C(city_fe)[T.boischatel] 0.4011 0.071 5.616 0.000 0.261 0.541
330 +C(city_fe)[T.boucherville] 0.4671 0.037 12.493 0.000 0.394 0.540
331 +C(city_fe)[T.bromont] 0.4244 0.034 12.535 0.000 0.358 0.491
332 +C(city_fe)[T.brossard] 0.5263 0.020 26.802 0.000 0.488 0.565
333 +C(city_fe)[T.bécancour] -0.0973 0.032 -3.045 0.002 -0.160 -0.035
334 +C(city_fe)[T.cantley] 0.1731 0.046 3.757 0.000 0.083 0.263
335 +C(city_fe)[T.carignan] 0.3790 0.042 8.990 0.000 0.296 0.462
336 +C(city_fe)[T.chandler] -0.2950 0.079 -3.757 0.000 -0.449 -0.141
337 +C(city_fe)[T.chateauguay] 0.2492 0.040 6.176 0.000 0.170 0.328
338 +C(city_fe)[T.chelsea] 0.3027 0.040 7.590 0.000 0.225 0.381
339 +C(city_fe)[T.chertsey] -0.0418 0.038 -1.107 0.268 -0.116 0.032
340 +C(city_fe)[T.chicoutimi] -0.0792 0.040 -1.976 0.048 -0.158 -0.001
341 +C(city_fe)[T.chomedey] 0.4118 0.055 7.515 0.000 0.304 0.519
342 +C(city_fe)[T.châteauguay] 0.2811 0.024 11.697 0.000 0.234 0.328
343 +C(city_fe)[T.contrecoeur] 0.1282 0.035 3.614 0.000 0.059 0.198
344 +C(city_fe)[T.cowansville] 0.1731 0.034 5.103 0.000 0.107 0.240
345 +C(city_fe)[T.crabtree] 0.0682 0.048 1.421 0.155 -0.026 0.162
346 +C(city_fe)[T.côte-saint-luc] 0.7496 0.053 14.252 0.000 0.647 0.853
347 +C(city_fe)[T.dolbeau-mistassini] -0.1900 0.063 -3.033 0.002 -0.313 -0.067
348 +C(city_fe)[T.dollard-des-ormeaux] 0.5396 0.029 18.879 0.000 0.484 0.596
349 +C(city_fe)[T.donnacona] 0.1291 0.035 3.685 0.000 0.060 0.198
350 +C(city_fe)[T.dorval] 0.6824 0.048 14.112 0.000 0.588 0.777
351 +C(city_fe)[T.drummondville] 0.0599 0.018 3.286 0.001 0.024 0.096
352 +C(city_fe)[T.duhamel] 0.0493 0.071 0.697 0.485 -0.089 0.188
353 +C(city_fe)[T.dunham] 0.3393 0.064 5.282 0.000 0.213 0.465
354 +C(city_fe)[T.eastman] 0.3752 0.056 6.645 0.000 0.265 0.486
355 +C(city_fe)[T.entrelacs] 0.2115 0.084 2.512 0.012 0.046 0.377
356 +C(city_fe)[T.farnham] 0.0658 0.035 1.873 0.061 -0.003 0.135
357 +C(city_fe)[T.fossambault-sur-le-lac] 0.3149 0.065 4.826 0.000 0.187 0.443
358 +C(city_fe)[T.gaspé] -0.3176 0.065 -4.918 0.000 -0.444 -0.191
359 +C(city_fe)[T.gatineau] 0.1320 0.013 10.162 0.000 0.107 0.158
360 +C(city_fe)[T.gore] 0.0680 0.055 1.227 0.220 -0.041 0.177
361 +C(city_fe)[T.granby] 0.1692 0.019 9.137 0.000 0.133 0.206
362 +C(city_fe)[T.grenville-sur-la-rouge] 0.1187 0.046 2.558 0.011 0.028 0.210
363 +C(city_fe)[T.hampstead] 1.2817 0.052 24.725 0.000 1.180 1.383
364 +C(city_fe)[T.harrington] -0.0772 0.125 -0.618 0.537 -0.322 0.168
365 +C(city_fe)[T.hudson] 0.4854 0.066 7.303 0.000 0.355 0.616
366 +C(city_fe)[T.joliette] 0.1347 0.045 3.023 0.003 0.047 0.222
367 +C(city_fe)[T.jonquière] -0.1481 0.044 -3.388 0.001 -0.234 -0.062
368 +C(city_fe)[T.kirkland] 0.5562 0.035 15.874 0.000 0.488 0.625
369 +C(city_fe)[T.l'ange-gardien] 0.2606 0.073 3.560 0.000 0.117 0.404
370 +C(city_fe)[T.l'assomption] 0.2026 0.028 7.206 0.000 0.147 0.258
371 +C(city_fe)[T.l'île-perrot] 0.4466 0.049 9.037 0.000 0.350 0.543
372 +C(city_fe)[T.la conception] 0.3260 0.052 6.254 0.000 0.224 0.428
373 +C(city_fe)[T.la malbaie] -0.1569 0.054 -2.898 0.004 -0.263 -0.051
374 +C(city_fe)[T.la minerve] 0.0588 0.052 1.128 0.260 -0.043 0.161
375 +C(city_fe)[T.la prairie] 0.4831 0.030 16.099 0.000 0.424 0.542
376 +C(city_fe)[T.la pêche] -0.0050 0.039 -0.129 0.897 -0.081 0.071
377 +C(city_fe)[T.la tuque] -0.3458 0.048 -7.231 0.000 -0.439 -0.252
378 +C(city_fe)[T.labelle] 0.0865 0.054 1.599 0.110 -0.020 0.193
379 +C(city_fe)[T.lac-beauport] 0.4153 0.085 4.883 0.000 0.249 0.582
380 +C(city_fe)[T.lac-brome] 0.6355 0.056 11.357 0.000 0.526 0.745
381 +C(city_fe)[T.lac-des-écorces] 0.0014 0.067 0.021 0.983 -0.130 0.133
382 +C(city_fe)[T.lac-etchemin] -0.2994 0.057 -5.288 0.000 -0.410 -0.188
383 +C(city_fe)[T.lachute] 0.1773 0.037 4.791 0.000 0.105 0.250
384 +C(city_fe)[T.lanoraie] 0.1419 0.037 3.844 0.000 0.070 0.214
385 +C(city_fe)[T.laval] 0.4740 0.014 34.317 0.000 0.447 0.501
386 +C(city_fe)[T.lavaltrie] 0.1688 0.037 4.538 0.000 0.096 0.242
387 +C(city_fe)[T.les éboulements] 0.1753 0.069 2.547 0.011 0.040 0.310
388 +C(city_fe)[T.longueuil] 0.3851 0.016 24.101 0.000 0.354 0.416
389 +C(city_fe)[T.lorraine] 0.4987 0.056 8.885 0.000 0.389 0.609
390 +C(city_fe)[T.lévis] 0.1158 0.021 5.394 0.000 0.074 0.158
391 +C(city_fe)[T.magog] 0.3283 0.032 10.286 0.000 0.266 0.391
392 +C(city_fe)[T.mandeville] -0.0735 0.040 -1.826 0.068 -0.152 0.005
393 +C(city_fe)[T.mansfield-et-pontefract] -0.5322 0.067 -7.928 0.000 -0.664 -0.401
394 +C(city_fe)[T.marieville] 0.2115 0.031 6.931 0.000 0.152 0.271
395 +C(city_fe)[T.mascouche] 0.3035 0.022 13.981 0.000 0.261 0.346
396 +C(city_fe)[T.matane] -0.1678 0.047 -3.568 0.000 -0.260 -0.076
397 +C(city_fe)[T.mercier / hochelaga / maisonneuve] 0.4115 0.048 8.594 0.000 0.318 0.505
398 +C(city_fe)[T.messines] -0.2594 0.051 -5.045 0.000 -0.360 -0.159
399 +C(city_fe)[T.mille-isles] 0.2025 0.051 3.999 0.000 0.103 0.302
400 +C(city_fe)[T.mirabel] 0.3088 0.021 14.650 0.000 0.268 0.350
401 +C(city_fe)[T.mont-joli] -0.2580 0.097 -2.649 0.008 -0.449 -0.067
402 +C(city_fe)[T.mont-laurier] -0.0438 0.041 -1.076 0.282 -0.124 0.036
403 +C(city_fe)[T.mont-royal] 1.2714 0.054 23.624 0.000 1.166 1.377
404 +C(city_fe)[T.mont-saint-hilaire] 0.5104 0.040 12.618 0.000 0.431 0.590
405 +C(city_fe)[T.mont-tremblant] 0.3946 0.035 11.119 0.000 0.325 0.464
406 +C(city_fe)[T.montréal] 0.7272 0.017 43.952 0.000 0.695 0.760
407 +C(city_fe)[T.morin-heights] 0.3128 0.039 7.963 0.000 0.236 0.390
408 +C(city_fe)[T.nominingue] 0.0819 0.081 1.012 0.311 -0.077 0.241
409 +C(city_fe)[T.notre-dame-de-l'île-perrot] 0.3960 0.042 9.361 0.000 0.313 0.479
410 +C(city_fe)[T.notre-dame-de-pontmain] -0.1176 0.060 -1.973 0.049 -0.234 -0.001
411 +C(city_fe)[T.notre-dame-des-bois] -0.3186 0.067 -4.775 0.000 -0.449 -0.188
412 +C(city_fe)[T.notre-dame-des-prairies] 0.0959 0.042 2.279 0.023 0.013 0.178
413 +C(city_fe)[T.notre-dame-du-laus] 0.0251 0.056 0.448 0.654 -0.085 0.135
414 +C(city_fe)[T.orford] 0.3476 0.061 5.701 0.000 0.228 0.467
415 +C(city_fe)[T.papineauville] -0.0121 0.115 -0.105 0.916 -0.238 0.213
416 +C(city_fe)[T.petite-rivière-saint-françois] -0.0216 0.055 -0.391 0.696 -0.130 0.087
417 +C(city_fe)[T.piedmont] 0.3718 0.051 7.301 0.000 0.272 0.472
418 +C(city_fe)[T.pincourt] 0.3208 0.036 8.976 0.000 0.251 0.391
419 +C(city_fe)[T.pointe-claire] 0.6096 0.041 14.960 0.000 0.530 0.689
420 +C(city_fe)[T.pont-rouge] 0.0167 0.044 0.378 0.705 -0.070 0.103
421 +C(city_fe)[T.prévost] 0.1910 0.027 7.046 0.000 0.138 0.244
422 +C(city_fe)[T.québec] 0.2113 0.018 11.997 0.000 0.177 0.246
423 +C(city_fe)[T.rawdon] 0.0548 0.032 1.711 0.087 -0.008 0.118
424 +C(city_fe)[T.repentigny] 0.2910 0.019 15.040 0.000 0.253 0.329
425 +C(city_fe)[T.rigaud] 0.1464 0.045 3.257 0.001 0.058 0.235
426 +C(city_fe)[T.rimouski] -0.0604 0.034 -1.779 0.075 -0.127 0.006
427 +C(city_fe)[T.rivière-du-loup] -0.0744 0.054 -1.380 0.168 -0.180 0.031
428 +C(city_fe)[T.rivière-rouge] -0.1228 0.059 -2.083 0.037 -0.238 -0.007
429 +C(city_fe)[T.roberval] -0.1505 0.064 -2.335 0.020 -0.277 -0.024
430 +C(city_fe)[T.rosemère] 0.4912 0.038 12.813 0.000 0.416 0.566
431 +C(city_fe)[T.saguenay] -0.0362 0.023 -1.577 0.115 -0.081 0.009
432 +C(city_fe)[T.saint-adolphe-d'howard] 0.1426 0.045 3.142 0.002 0.054 0.231
433 +C(city_fe)[T.saint-amable] 0.2897 0.035 8.261 0.000 0.221 0.358
434 +C(city_fe)[T.saint-anicet] 0.2175 0.068 3.178 0.001 0.083 0.352
435 +C(city_fe)[T.saint-basile-le-grand] 0.3978 0.045 8.829 0.000 0.310 0.486
436 +C(city_fe)[T.saint-bruno-de-montarville] 0.6239 0.040 15.791 0.000 0.546 0.701
437 +C(city_fe)[T.saint-calixte] 0.0061 0.034 0.177 0.859 -0.061 0.073
438 +C(city_fe)[T.saint-charles-borromée] 0.1596 0.053 3.015 0.003 0.056 0.263
439 +C(city_fe)[T.saint-colomban] 0.1400 0.041 3.385 0.001 0.059 0.221
440 +C(city_fe)[T.saint-constant] 0.3374 0.031 10.768 0.000 0.276 0.399
441 +C(city_fe)[T.saint-donat] 0.3168 0.047 6.726 0.000 0.224 0.409
442 +C(city_fe)[T.saint-eustache] 0.3452 0.036 9.568 0.000 0.274 0.416
443 +C(city_fe)[T.saint-félicien] -0.0434 0.054 -0.807 0.420 -0.149 0.062
444 +C(city_fe)[T.saint-gabriel-de-brandon] -0.0090 0.063 -0.143 0.887 -0.133 0.115
445 +C(city_fe)[T.saint-georges] -0.1772 0.038 -4.679 0.000 -0.251 -0.103
446 +C(city_fe)[T.saint-hippolyte] 0.2118 0.041 5.225 0.000 0.132 0.291
447 +C(city_fe)[T.saint-honoré] -0.0330 0.062 -0.536 0.592 -0.154 0.088
448 +C(city_fe)[T.saint-hyacinthe] 0.1948 0.033 5.974 0.000 0.131 0.259
449 +C(city_fe)[T.saint-jean-de-matha] 0.0276 0.065 0.422 0.673 -0.101 0.156
450 +C(city_fe)[T.saint-jean-sur-richelieu] 0.2785 0.019 14.288 0.000 0.240 0.317
451 +C(city_fe)[T.saint-jérôme] 0.2141 0.020 10.479 0.000 0.174 0.254
452 +C(city_fe)[T.saint-lambert] 0.8773 0.047 18.632 0.000 0.785 0.970
453 +C(city_fe)[T.saint-lambert-de-lauzon] 0.0684 0.053 1.298 0.194 -0.035 0.172
454 +C(city_fe)[T.saint-lin/laurentides] 0.1066 0.032 3.313 0.001 0.044 0.170
455 +C(city_fe)[T.saint-marc-des-carrières] -0.1398 0.032 -4.307 0.000 -0.203 -0.076
456 +C(city_fe)[T.saint-michel-des-saints] -0.0394 0.067 -0.589 0.556 -0.171 0.092
457 +C(city_fe)[T.saint-paul-de-l'île-aux-noix] 0.0212 0.099 0.214 0.831 -0.173 0.215
458 +C(city_fe)[T.saint-philippe] 0.4244 0.029 14.752 0.000 0.368 0.481
459 +C(city_fe)[T.saint-raymond] 0.0769 0.067 1.154 0.249 -0.054 0.207
460 +C(city_fe)[T.saint-sauveur] 0.2746 0.033 8.211 0.000 0.209 0.340
461 +C(city_fe)[T.saint-zotique] 0.2721 0.037 7.329 0.000 0.199 0.345
462 +C(city_fe)[T.saint-zénon] -0.0122 0.106 -0.115 0.908 -0.221 0.196
463 +C(city_fe)[T.sainte-adèle] 0.3087 0.048 6.408 0.000 0.214 0.403
464 +C(city_fe)[T.sainte-agathe-des-monts] 0.1241 0.050 2.457 0.014 0.025 0.223
465 +C(city_fe)[T.sainte-anne-des-plaines] 0.2834 0.062 4.584 0.000 0.162 0.405
466 +C(city_fe)[T.sainte-barbe] 0.4826 0.071 6.781 0.000 0.343 0.622
467 +C(city_fe)[T.sainte-brigitte-de-laval] 0.1199 0.041 2.899 0.004 0.039 0.201
468 +C(city_fe)[T.sainte-béatrix] -0.0687 0.043 -1.582 0.114 -0.154 0.016
469 +C(city_fe)[T.sainte-catherine] 0.3381 0.034 9.976 0.000 0.272 0.405
470 +C(city_fe)[T.sainte-catherine-de-la-jacques-cartier] 0.1270 0.059 2.152 0.031 0.011 0.243
471 +C(city_fe)[T.sainte-julie] 0.4826 0.055 8.787 0.000 0.375 0.590
472 +C(city_fe)[T.sainte-julienne] 0.0375 0.035 1.079 0.281 -0.031 0.106
473 +C(city_fe)[T.sainte-lucie-des-laurentides] 0.1493 0.037 3.986 0.000 0.076 0.223
474 +C(city_fe)[T.sainte-marguerite-du-lac-masson] 0.0068 0.053 0.129 0.897 -0.097 0.110
475 +C(city_fe)[T.sainte-marthe-sur-le-lac] 0.3182 0.023 13.870 0.000 0.273 0.363
476 +C(city_fe)[T.sainte-sophie] 0.0978 0.038 2.566 0.010 0.023 0.173
477 +C(city_fe)[T.sainte-thérèse] 0.3765 0.041 9.286 0.000 0.297 0.456
478 +C(city_fe)[T.sainte-émélie-de-l'énergie] -0.2157 0.060 -3.580 0.000 -0.334 -0.098
479 +C(city_fe)[T.salaberry-de-valleyfield] 0.0841 0.043 1.978 0.048 0.001 0.168
480 +C(city_fe)[T.sept-îles] -0.0848 0.036 -2.364 0.018 -0.155 -0.014
481 +C(city_fe)[T.shawinigan] -0.1015 0.025 -4.056 0.000 -0.151 -0.052
482 +C(city_fe)[T.shefford] 0.2935 0.059 4.934 0.000 0.177 0.410
483 +C(city_fe)[T.sherbrooke] 0.1051 0.017 6.332 0.000 0.073 0.138
484 +C(city_fe)[T.sorel-tracy] 0.0059 0.027 0.220 0.826 -0.047 0.059
485 +C(city_fe)[T.st-colomban] 0.2744 0.069 3.978 0.000 0.139 0.410
486 +C(city_fe)[T.st-hippolyte] 0.1024 0.077 1.324 0.185 -0.049 0.254
487 +C(city_fe)[T.st-jean-sur-richelieu] 0.2296 0.034 6.754 0.000 0.163 0.296
488 +C(city_fe)[T.st-jérôme] 0.2389 0.036 6.713 0.000 0.169 0.309
489 +C(city_fe)[T.st-sauveur] 0.4023 0.088 4.549 0.000 0.229 0.576
490 +C(city_fe)[T.ste-adèle] 0.1944 0.064 3.044 0.002 0.069 0.320
491 +C(city_fe)[T.ste-julienne] 0.0665 0.038 1.740 0.082 -0.008 0.141
492 +C(city_fe)[T.sutton] 0.4900 0.051 9.559 0.000 0.390 0.590
493 +C(city_fe)[T.terrebonne] 0.2995 0.018 16.761 0.000 0.264 0.335
494 +C(city_fe)[T.thetford mines] -0.2784 0.058 -4.762 0.000 -0.393 -0.164
495 +C(city_fe)[T.trois-rivières] 0.0129 0.020 0.634 0.526 -0.027 0.053
496 +C(city_fe)[T.très-saint-rédempteur] 0.5938 0.052 11.367 0.000 0.491 0.696
497 +C(city_fe)[T.témiscouata-sur-le-lac] -0.1940 0.074 -2.618 0.009 -0.339 -0.049
498 +C(city_fe)[T.val-d'or] 0.0212 0.029 0.742 0.458 -0.035 0.077
499 +C(city_fe)[T.val-david] 0.0449 0.054 0.828 0.408 -0.061 0.151
500 +C(city_fe)[T.val-des-monts] 0.0856 0.027 3.163 0.002 0.033 0.139
501 +C(city_fe)[T.val-morin] 0.0522 0.057 0.916 0.360 -0.060 0.164
502 +C(city_fe)[T.vaudreuil-dorion] 0.2565 0.029 8.949 0.000 0.200 0.313
503 +C(city_fe)[T.venise-en-québec] 0.1799 0.044 4.085 0.000 0.094 0.266
504 +C(city_fe)[T.victoriaville] -0.0166 0.024 -0.691 0.489 -0.064 0.031
505 +C(city_fe)[T.waterloo] 0.1307 0.049 2.687 0.007 0.035 0.226
506 +C(city_fe)[T.weedon] -0.1577 0.062 -2.550 0.011 -0.279 -0.036
507 +C(city_fe)[T.wentworth-nord] 0.0592 0.072 0.828 0.408 -0.081 0.199
508 +C(city_fe)[T.westmount] 1.6031 0.038 42.458 0.000 1.529 1.677
509 +ln_lot 0.0400 0.003 12.549 0.000 0.034 0.046
510 +has_lot -0.0842 0.015 -5.753 0.000 -0.113 -0.056
511 +beds 0.0281 0.004 6.804 0.000 0.020 0.036
512 +baths 0.1331 0.005 25.473 0.000 0.123 0.143
513 +parking 0.0153 0.002 8.276 0.000 0.012 0.019
514 +storeys 0.0023 0.096 0.024 0.981 -0.186 0.190
515 +age -0.0066 0.000 -16.853 0.000 -0.007 -0.006
516 +age2 3.287e-05 2.79e-06 11.795 0.000 2.74e-05 3.83e-05
517 +has_age -0.1356 0.019 -7.296 0.000 -0.172 -0.099
518 +pool 0.0217 0.011 1.945 0.052 -0.000 0.044
519 +waterfront 0.0497 0.019 2.636 0.008 0.013 0.087
520 +garage 0.0322 0.018 1.748 0.080 -0.004 0.068
521 +fireplace -0.0667 0.016 -4.270 0.000 -0.097 -0.036
522 +central_air 0.0445 0.010 4.577 0.000 0.025 0.064
523 +finished_bsmt -0.0063 0.008 -0.773 0.439 -0.022 0.010
524 +near_water 0.0709 0.020 3.459 0.001 0.031 0.111
525 +cul_de_sac 0.0285 0.009 3.179 0.001 0.011 0.046
526 +golf 0.0180 0.006 3.122 0.002 0.007 0.029
527 +ski 0.0382 0.010 3.899 0.000 0.019 0.057
528 +near_park 0.0074 0.008 0.929 0.353 -0.008 0.023
529 +near_school -0.0520 0.009 -6.027 0.000 -0.069 -0.035
530 +near_highway 0.0475 0.006 7.384 0.000 0.035 0.060
531 +transit 0.0684 0.007 9.180 0.000 0.054 0.083
532 +no_rear_neighbor -0.0453 0.009 -4.890 0.000 -0.063 -0.027
533 +n_images_c 0.0057 0.000 18.731 0.000 0.005 0.006
534 +virtual_tour 0.0207 0.016 1.327 0.184 -0.010 0.051
535 +n_rooms 0.0254 0.001 17.680 0.000 0.023 0.028
536 +n_levels_r 0.0413 0.047 0.871 0.384 -0.052 0.134
537 +has_primary_bed 0.0664 0.010 6.743 0.000 0.047 0.086
538 +has_basement -0.1032 0.043 -2.392 0.017 -0.188 -0.019
539 +r_office 0.0359 0.008 4.728 0.000 0.021 0.051
540 +r_family 0.0359 0.009 4.050 0.000 0.019 0.053
541 +r_workshop -0.0297 0.009 -3.377 0.001 -0.047 -0.012
542 +r_winecellar 0.1393 0.026 5.335 0.000 0.088 0.190
543 +r_sunroom 0.0193 0.013 1.438 0.150 -0.007 0.046
544 +r_mezzanine 0.0208 0.018 1.150 0.250 -0.015 0.056
545 +r_library 0.1475 0.069 2.125 0.034 0.011 0.284
546 +r_laundry 0.0363 0.006 6.025 0.000 0.025 0.048
547 +g_cond 0.0119 0.004 3.005 0.003 0.004 0.020
548 +g_hardwood 0.0315 0.008 3.705 0.000 0.015 0.048
549 +g_quartz_granite 0.0177 0.008 2.091 0.037 0.001 0.034
550 +g_view_water 0.0309 0.014 2.142 0.032 0.003 0.059
551 +has_gpt 6.1815 0.057 108.357 0.000 6.070 6.293
552 +g_turnkey 0.0044 0.006 0.692 0.489 -0.008 0.017
553 +g_recently_renovated 0.0067 0.008 0.868 0.385 -0.008 0.022
554 +g_garage 0.0193 0.018 1.076 0.282 -0.016 0.054
555 +g_pool 0.0484 0.012 4.089 0.000 0.025 0.072
556 +g_waterfront 0.0703 0.016 4.356 0.000 0.039 0.102
557 +g_finished_basement -0.0274 0.007 -4.097 0.000 -0.040 -0.014
558 +g_fireplace 0.0852 0.016 5.392 0.000 0.054 0.116
559 +g_central_ac 0.0312 0.010 3.279 0.001 0.013 0.050
560 +g_new_construction 0.0628 0.013 4.949 0.000 0.038 0.088
561 +g_income_property -0.0340 0.009 -3.966 0.000 -0.051 -0.017
562 +g_renovated_kitchen 0.0338 0.008 4.022 0.000 0.017 0.050
563 +g_renovated_bathroom -0.0109 0.008 -1.389 0.165 -0.026 0.004
564 +g_large_lot 0.0117 0.007 1.781 0.075 -0.001 0.025
565 +g_quiet_area -0.0209 0.006 -3.661 0.000 -0.032 -0.010
566 +==============================================================================
567 +Omnibus: 2524.251 Durbin-Watson: 1.763
568 +Prob(Omnibus): 0.000 Jarque-Bera (JB): 16663.299
569 +Skew: 0.454 Prob(JB): 0.00
570 +Kurtosis: 7.482 Cond. No. 9.65e+16
571 +==============================================================================
572 +
573 +Notes:
574 +[1] Standard Errors are heteroscedasticity robust (HC3)
575 +[2] The smallest eigenvalue is 4.82e-23. This might indicate that there are
576 +strong multicollinearity problems or that the design matrix is singular.
\ No newline at end of file
added results/revision_extras.json +486 −0
@@ -0,0 +1,486 @@
1 +{
2 + "modelA": {
3 + "r2": 0.7552103110293896,
4 + "adj": 0.7477317389713678,
5 + "n": 9446,
6 + "coef": {
7 + "ln_sqft": {
8 + "b": 0.3583611466783186,
9 + "se_hc3": 0.013021312719264802,
10 + "p_hc3": 9.811436146831853e-167,
11 + "se_cl": 0.011546608518322881,
12 + "p_cl": 1.7598083257322608e-211
13 + },
14 + "ln_lot": {
15 + "b": 0.027321695727434744,
16 + "se_hc3": 0.003968643673932088,
17 + "p_hc3": 5.803517449373336e-12,
18 + "se_cl": 0.0057217322173256846,
19 + "p_cl": 1.7964103147193876e-06
20 + },
21 + "beds": {
22 + "b": 0.013623293126953095,
23 + "se_hc3": 0.0040602600210150885,
24 + "p_hc3": 0.0007928582892261605,
25 + "se_cl": 0.00457074886728697,
26 + "p_cl": 0.002877420342379003
27 + },
28 + "baths": {
29 + "b": 0.07348769487973825,
30 + "se_hc3": 0.006121072722376004,
31 + "p_hc3": 3.3169032296614814e-33,
32 + "se_cl": 0.0069532806823969125,
33 + "p_cl": 4.1586858986169373e-26
34 + },
35 + "parking": {
36 + "b": 0.01149153979340771,
37 + "se_hc3": 0.0015211101627679384,
38 + "p_hc3": 4.198076313292844e-14,
39 + "se_cl": 0.0014144294006288624,
40 + "p_cl": 4.491890458706105e-16
41 + },
42 + "age": {
43 + "b": -0.003778162499817968,
44 + "se_hc3": 0.0004166167508553082,
45 + "p_hc3": 1.2047149502398342e-19,
46 + "se_cl": 0.0010761773989556892,
47 + "p_cl": 0.00044688706454431995
48 + },
49 + "age2": {
50 + "b": 2.0412213402030788e-05,
51 + "se_hc3": 3.1634803715459156e-06,
52 + "p_hc3": 1.1005287533895908e-10,
53 + "se_cl": 4.403903629319602e-06,
54 + "p_cl": 3.568904338805579e-06
55 + },
56 + "central_air": {
57 + "b": 0.053384334694057345,
58 + "se_hc3": 0.011164185372283303,
59 + "p_hc3": 1.737763866667577e-06,
60 + "se_cl": 0.017763303901713427,
61 + "p_cl": 0.0026530539182792814
62 + },
63 + "transit": {
64 + "b": 0.09000078993798896,
65 + "se_hc3": 0.00842353892882497,
66 + "p_hc3": 1.2037429378492829e-26,
67 + "se_cl": 0.04560367671561142,
68 + "p_cl": 0.04843379796570755
69 + },
70 + "waterfront": {
71 + "b": 0.05808657079776136,
72 + "se_hc3": 0.04276278415929502,
73 + "p_hc3": 0.17435455833645552,
74 + "se_cl": 0.033496164012264705,
75 + "p_cl": 0.08289571923560418
76 + },
77 + "g_cond": {
78 + "b": 0.016168128372704954,
79 + "se_hc3": 0.004284326158407061,
80 + "p_hc3": 0.0001607889544697841,
81 + "se_cl": 0.0058314343073510595,
82 + "p_cl": 0.005561354758076747
83 + },
84 + "g_renovated_kitchen": {
85 + "b": 0.026089835765711222,
86 + "se_hc3": 0.009731092534231904,
87 + "p_hc3": 0.007338501052577358,
88 + "se_cl": 0.006899975364881851,
89 + "p_cl": 0.00015610612945056893
90 + },
91 + "g_hardwood": {
92 + "b": 0.023654684502188772,
93 + "se_hc3": 0.007568398763791719,
94 + "p_hc3": 0.0017753064896647113,
95 + "se_cl": 0.009325076436933125,
96 + "p_cl": 0.011191092284423674
97 + },
98 + "g_new_construction": {
99 + "b": -0.00030428071460375516,
100 + "se_hc3": 0.013498897144460655,
101 + "p_hc3": 0.9820162844667388,
102 + "se_cl": 0.013315085313141847,
103 + "p_cl": 0.9817680661387342
104 + },
105 + "g_income_property": {
106 + "b": -0.01916827298970958,
107 + "se_hc3": 0.010125989321693481,
108 + "p_hc3": 0.05836082179559654,
109 + "se_cl": 0.009252803603413885,
110 + "p_cl": 0.03830109511355022
111 + },
112 + "v_curb": {
113 + "b": 0.023017903057193298,
114 + "se_hc3": 0.007507554436150879,
115 + "p_hc3": 0.0021696833221349896,
116 + "se_cl": 0.00865392944593514,
117 + "p_cl": 0.007818219565218354
118 + },
119 + "v_reno": {
120 + "b": 0.023593099837405124,
121 + "se_hc3": 0.007459829833637167,
122 + "p_hc3": 0.0015632080662522378,
123 + "se_cl": 0.009529503385437559,
124 + "p_cl": 0.013293976823511309
125 + },
126 + "v_luxury": {
127 + "b": 0.05100475597004259,
128 + "se_hc3": 0.009917011195885339,
129 + "p_hc3": 2.7015815598254944e-07,
130 + "se_cl": 0.011932756209389966,
131 + "p_cl": 1.916971879355069e-05
132 + },
133 + "v_garage_vis": {
134 + "b": 0.04626239493908085,
135 + "se_hc3": 0.007095142592775905,
136 + "p_hc3": 7.017108386426899e-11,
137 + "se_cl": 0.005814520452429543,
138 + "p_cl": 1.7718006523061964e-15
139 + },
140 + "v_mat_brique": {
141 + "b": 0.033343794442173225,
142 + "se_hc3": 0.00827594130032085,
143 + "p_hc3": 5.601384271322806e-05,
144 + "se_cl": 0.009143815895444862,
145 + "p_cl": 0.0002657381339945666
146 + },
147 + "v_mat_pierre": {
148 + "b": 0.03615198096489164,
149 + "se_hc3": 0.007737924779006411,
150 + "p_hc3": 2.982065560443421e-06,
151 + "se_cl": 0.008011891510590393,
152 + "p_cl": 6.413130561843427e-06
153 + },
154 + "v_mat_vinyle": {
155 + "b": -0.04493374989989597,
156 + "se_hc3": 0.007597159274294025,
157 + "p_hc3": 3.3279171858711188e-09,
158 + "se_cl": 0.015100349535056819,
159 + "p_cl": 0.0029234340683073927
160 + },
161 + "v_set_rural": {
162 + "b": -0.12230705352273197,
163 + "se_hc3": 0.04529347936059,
164 + "p_hc3": 0.006927205074830596,
165 + "se_cl": 0.032184971001596396,
166 + "p_cl": 0.00014462056902901985
167 + },
168 + "v_land": {
169 + "b": 0.015079214851207945,
170 + "se_hc3": 0.005163307230556136,
171 + "p_hc3": 0.003495188385196164,
172 + "se_cl": 0.0062394992136362536,
173 + "p_cl": 0.015660422546723243
174 + },
175 + "v_cond": {
176 + "b": 0.011435724114234713,
177 + "se_hc3": 0.009935939848613642,
178 + "p_hc3": 0.24975470480811712,
179 + "se_cl": 0.00793900430747386,
180 + "p_cl": 0.1497406477456507
181 + }
182 + }
183 + },
184 + "modelB": {
185 + "r2": 0.7250019979121325,
186 + "adj": 0.7209302236414477,
187 + "n": 19123,
188 + "coef": {
189 + "ln_lot": {
190 + "b": 0.040421819255842366,
191 + "se_hc3": 0.0030793415163729103,
192 + "p_hc3": 2.3130287315281885e-39,
193 + "se_cl": 0.005130684552116609,
194 + "p_cl": 3.314786914788146e-15
195 + },
196 + "beds": {
197 + "b": 0.03225644920405396,
198 + "se_hc3": 0.0037380829672411845,
199 + "p_hc3": 6.181384242360143e-18,
200 + "se_cl": 0.0026461656800326736,
201 + "p_cl": 3.5192764358646113e-34
202 + },
203 + "baths": {
204 + "b": 0.11334162478693238,
205 + "se_hc3": 0.004820025879176098,
206 + "p_hc3": 2.882946420348713e-122,
207 + "se_cl": 0.0065680707475647485,
208 + "p_cl": 1.0006690034057575e-66
209 + },
210 + "parking": {
211 + "b": 0.014887132710857906,
212 + "se_hc3": 0.001722223609697608,
213 + "p_hc3": 5.4214611052329344e-18,
214 + "se_cl": 0.0019170158855884335,
215 + "p_cl": 8.114125760065666e-15
216 + },
217 + "age": {
218 + "b": -0.00454728435076453,
219 + "se_hc3": 0.00031568867613853834,
220 + "p_hc3": 4.859972485857048e-47,
221 + "se_cl": 0.0005421521451234677,
222 + "p_cl": 4.9671978979020763e-17
223 + },
224 + "age2": {
225 + "b": 2.4738741307502545e-05,
226 + "se_hc3": 2.2589789920551392e-06,
227 + "p_hc3": 6.55056354106643e-28,
228 + "se_cl": 1.6280767754927894e-06,
229 + "p_cl": 3.8125346654408024e-52
230 + },
231 + "central_air": {
232 + "b": 0.020145753639079278,
233 + "se_hc3": 0.009178809775081053,
234 + "p_hc3": 0.028177150453293415,
235 + "se_cl": 0.017821940423307536,
236 + "p_cl": 0.25831168008287975
237 + },
238 + "transit": {
239 + "b": 0.058387955324457536,
240 + "se_hc3": 0.006823518187251239,
241 + "p_hc3": 1.1597359094737698e-17,
242 + "se_cl": 0.039429380632120564,
243 + "p_cl": 0.1386535998125059
244 + },
245 + "waterfront": {
246 + "b": 0.061910921726998136,
247 + "se_hc3": 0.023452608193077887,
248 + "p_hc3": 0.008294738330249088,
249 + "se_cl": 0.019005260291841415,
250 + "p_cl": 0.0011237145722294154
251 + },
252 + "g_cond": {
253 + "b": 0.016317524818277362,
254 + "se_hc3": 0.003676322607356966,
255 + "p_hc3": 9.056887962882854e-06,
256 + "se_cl": 0.0051984051513270825,
257 + "p_cl": 0.0016955538138371947
258 + },
259 + "g_renovated_kitchen": {
260 + "b": 0.02694519585550771,
261 + "se_hc3": 0.00778196222799486,
262 + "p_hc3": 0.0005351431624444984,
263 + "se_cl": 0.00963562440448091,
264 + "p_cl": 0.005167314672277988
265 + },
266 + "g_hardwood": {
267 + "b": 0.029508607751202176,
268 + "se_hc3": 0.007732639804181234,
269 + "p_hc3": 0.000135571823960277,
270 + "se_cl": 0.010501088270130441,
271 + "p_cl": 0.004953344261501498
272 + },
273 + "g_new_construction": {
274 + "b": -0.015913022424995732,
275 + "se_hc3": 0.012366399085437902,
276 + "p_hc3": 0.19816569619711843,
277 + "se_cl": 0.013844198802557594,
278 + "p_cl": 0.2503761852618782
279 + },
280 + "g_income_property": {
281 + "b": -0.012880740908525923,
282 + "se_hc3": 0.008090151720423983,
283 + "p_hc3": 0.11135084238149032,
284 + "se_cl": 0.007646640747981612,
285 + "p_cl": 0.09208572167107143
286 + },
287 + "v_curb": {
288 + "b": 0.01595794997392595,
289 + "se_hc3": 0.006208945541452705,
290 + "p_hc3": 0.010165313745874592,
291 + "se_cl": 0.013160921694546399,
292 + "p_cl": 0.22531138001532158
293 + },
294 + "v_reno": {
295 + "b": 0.01998000405352155,
296 + "se_hc3": 0.005938481115424498,
297 + "p_hc3": 0.0007668321703145232,
298 + "se_cl": 0.00862597465890367,
299 + "p_cl": 0.020544051694669916
300 + },
301 + "v_luxury": {
302 + "b": 0.061660294830383405,
303 + "se_hc3": 0.009917957983108048,
304 + "p_hc3": 5.066358372641718e-10,
305 + "se_cl": 0.008697717532597092,
306 + "p_cl": 1.3484065713602217e-12
307 + },
308 + "v_garage_vis": {
309 + "b": 0.04651267305326928,
310 + "se_hc3": 0.005473235777759556,
311 + "p_hc3": 1.9254685227194248e-17,
312 + "se_cl": 0.0067205744806021995,
313 + "p_cl": 4.4866578070246505e-12
314 + },
315 + "v_mat_brique": {
316 + "b": 0.04403606617642199,
317 + "se_hc3": 0.006434026368289839,
318 + "p_hc3": 7.687913299168285e-12,
319 + "se_cl": 0.011001756372510255,
320 + "p_cl": 6.263967386269365e-05
321 + },
322 + "v_mat_pierre": {
323 + "b": 0.04658529674361125,
324 + "se_hc3": 0.006389001415391058,
325 + "p_hc3": 3.0656103177722586e-13,
326 + "se_cl": 0.004986368566238395,
327 + "p_cl": 9.405939222542233e-21
328 + },
329 + "v_mat_vinyle": {
330 + "b": -0.06180816681575538,
331 + "se_hc3": 0.0059550197354287316,
332 + "p_hc3": 3.0844255288655324e-25,
333 + "se_cl": 0.012339743471244337,
334 + "p_cl": 5.475063124724954e-07
335 + },
336 + "v_set_rural": {
337 + "b": -0.07600759882521751,
338 + "se_hc3": 0.03798172502754968,
339 + "p_hc3": 0.04537489488193291,
340 + "se_cl": 0.02898506811408183,
341 + "p_cl": 0.008733802916670029
342 + },
343 + "v_land": {
344 + "b": 0.025364605559571263,
345 + "se_hc3": 0.004436697037848911,
346 + "p_hc3": 1.0841988159731501e-08,
347 + "se_cl": 0.004475099360773844,
348 + "p_cl": 1.4452250155571873e-08
349 + },
350 + "v_cond": {
351 + "b": 0.03275330527845849,
352 + "se_hc3": 0.008042409776296826,
353 + "p_hc3": 4.649651177320347e-05,
354 + "se_cl": 0.00741253911271998,
355 + "p_cl": 9.932619742557207e-06
356 + }
357 + }
358 + },
359 + "ablationA_insample": {
360 + "base": 0.7168982211740981,
361 + "plus_text": 0.7252251732782795,
362 + "plus_text_vision": 0.7552103110293896
363 + },
364 + "city_fe_top": [
365 + [
366 + "westmount",
367 + 230.7041938252226
368 + ],
369 + [
370 + "hampstead",
371 + 173.35777258613155
372 + ],
373 + [
374 + "mont-royal",
375 + 168.59619659817113
376 + ],
377 + [
378 + "sainte-anne-des-plaines",
379 + 142.9419234222625
380 + ],
381 + [
382 + "montr\u00e9al",
383 + 83.5002446059172
384 + ],
385 + [
386 + "saint-lambert",
387 + 81.58672431832005
388 + ],
389 + [
390 + "beaconsfield",
391 + 80.76456609139153
392 + ],
393 + [
394 + "c\u00f4te-saint-luc",
395 + 76.87439413605533
396 + ],
397 + [
398 + "saint-bruno-de-montarville",
399 + 70.85936417552094
400 + ],
401 + [
402 + "dorval",
403 + 68.65665951427297
404 + ],
405 + [
406 + "lac-brome",
407 + 62.14364632969884
408 + ],
409 + [
410 + "pointe-claire",
411 + 55.677246677163005
412 + ]
413 + ],
414 + "city_fe_bot": [
415 + [
416 + "mansfield-et-pontefract",
417 + -30.06708220027833
418 + ],
419 + [
420 + "baie-comeau",
421 + -29.663061228503672
422 + ],
423 + [
424 + "t\u00e9miscouata-sur-le-lac",
425 + -28.11245530496782
426 + ],
427 + [
428 + "la tuque",
429 + -25.71095774442922
430 + ],
431 + [
432 + "saint-marc-des-carri\u00e8res",
433 + -25.50454270323863
434 + ],
435 + [
436 + "thetford mines",
437 + -24.723126467416044
438 + ],
439 + [
440 + "chandler",
441 + -22.75037961497599
442 + ],
443 + [
444 + "dolbeau-mistassini",
445 + -22.659205442967977
446 + ]
447 + ],
448 + "cv": {
449 + "ols_v4": {
450 + "r2": 0.6277548901390929,
451 + "mdae": 17.87702738532425,
452 + "w15": 42.92213564817236,
453 + "w25": 64.46687235266432
454 + },
455 + "ols_v5": {
456 + "r2": 0.6862361284588729,
457 + "mdae": 16.88625054752524,
458 + "w15": 45.25440568948387,
459 + "w25": 66.72070281859541
460 + },
461 + "gbm_v4": {
462 + "r2": 0.7417951731584451,
463 + "mdae": 15.490609148904856,
464 + "w15": 48.826021021806206,
465 + "w25": 69.8687444438634
466 + },
467 + "gbm_v5": {
468 + "r2": 0.7688886936527248,
469 + "mdae": 14.68537838434596,
470 + "w15": 50.75040527113946,
471 + "w25": 71.32249124091409
472 + },
473 + "rf_v4": {
474 + "r2": 0.6765341495717297,
475 + "mdae": 17.899891187986572,
476 + "w15": 42.84369607279193,
477 + "w25": 64.13219683104116
478 + },
479 + "rf_v5": {
480 + "r2": 0.7039491915758698,
481 + "mdae": 17.201324709196374,
482 + "w15": 44.77330962715055,
483 + "w25": 65.63823667834545
484 + }
485 + }
486 +}
\ No newline at end of file
added scripts/00a_test_openai_extraction.py +71 −0
@@ -0,0 +1,71 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Test : extraction de specs via GPT-4o-mini (API REST, sans dépendance)."""
4 +import json, os, urllib.request, urllib.error
5 +
6 +# charge la clé depuis .env
7 +for line in open(".env", encoding="utf-8"):
8 + if line.startswith("OPENAI_API_KEY="):
9 + os.environ["OPENAI_API_KEY"] = line.split("=", 1)[1].strip()
10 +KEY = os.environ["OPENAI_API_KEY"]
11 +
12 +SCHEMA = {
13 + "type": "object",
14 + "additionalProperties": False,
15 + "properties": {
16 + "condition": {"type": "string", "enum": ["a_renover","original","bon","renove","refait_neuf","neuf","inconnu"]},
17 + "turnkey": {"type": "boolean"},
18 + "recently_renovated": {"type": "boolean"},
19 + "flooring": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"]}},
20 + "countertop": {"type": "string", "enum": ["granit","quartz","marbre","stratifie","bois","autre","inconnu"]},
21 + "heating": {"type": "array", "items": {"type": "string", "enum": ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"]}},
22 + "garage": {"type": "boolean"},
23 + "pool": {"type": "boolean"},
24 + "waterfront": {"type": "boolean"},
25 + "view": {"type": "string", "enum": ["eau","montagne","ville","aucune","inconnu"]},
26 + "finished_basement":{"type": "boolean"},
27 + "fireplace": {"type": "boolean"},
28 + "central_ac": {"type": "boolean"},
29 + "new_construction":{"type": "boolean"},
30 + "income_property": {"type": "boolean"},
31 + "highlights": {"type": "array", "items": {"type": "string"}, "description": "3-6 mots-clés de vente saillants"}
32 + },
33 + "required": ["condition","turnkey","recently_renovated","flooring","countertop",
34 + "heating","garage","pool","waterfront","view","finished_basement",
35 + "fireplace","central_ac","new_construction","income_property","highlights"]
36 +}
37 +
38 +SYS = ("Tu es un extracteur d'attributs immobiliers. À partir de la description "
39 + "d'une propriété (français ou anglais), renvoie les specs au format JSON "
40 + "strict. Base-toi UNIQUEMENT sur le texte ; si une info est absente, mets "
41 + "false / 'inconnu' / [] selon le champ. highlights = 3 à 6 mots-clés.")
42 +
43 +def extract(desc):
44 + body = {
45 + "model": "gpt-4o-mini",
46 + "temperature": 0,
47 + "messages": [{"role": "system", "content": SYS},
48 + {"role": "user", "content": desc[:4000]}],
49 + "response_format": {"type": "json_schema", "json_schema":
50 + {"name": "specs", "strict": True, "schema": SCHEMA}},
51 + }
52 + req = urllib.request.Request(
53 + "https://api.openai.com/v1/chat/completions",
54 + data=json.dumps(body).encode(),
55 + headers={"Authorization": f"Bearer {KEY}",
56 + "Content-Type": "application/json"})
57 + with urllib.request.urlopen(req, timeout=60) as r:
58 + out = json.load(r)
59 + return out["choices"][0]["message"]["content"], out.get("usage")
60 +
61 +# 3 descriptions du dataset
62 +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8"))
63 +descs = [d["description"] for d in data if d.get("description")][:3]
64 +for i, dsc in enumerate(descs):
65 + try:
66 + content, usage = extract(dsc)
67 + print(f"\n=== Exemple {i+1} (tokens: {usage}) ===")
68 + print("DESC:", dsc[:160].replace("\n", " "), "...")
69 + print("SPECS:", json.dumps(json.loads(content), ensure_ascii=False))
70 + except urllib.error.HTTPError as e:
71 + print("HTTP", e.code, e.read().decode()[:400]); break
added scripts/00b_test_vision_claude.py +65 −0
@@ -0,0 +1,65 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Test : analyse d'images immobilières via Claude Haiku 4.5 (REST, sans SDK)."""
4 +import json, os, urllib.request, urllib.error
5 +
6 +for line in open(".env", encoding="utf-8"):
7 + if line.startswith("ANTHROPIC_API_KEY="):
8 + KEY = line.split("=", 1)[1].strip()
9 +
10 +SCHEMA = {
11 + "type": "object", "additionalProperties": False,
12 + "properties": {
13 + "shot_type": {"type": "string", "enum": ["exterieur_facade","cuisine","salle_bain","salon","chambre","sous_sol","cour_terrain","vue","autre_interieur","autre"]},
14 + "condition_score": {"type": "integer", "description": "1=très délabré, 10=neuf/luxueux"},
15 + "renovation_level": {"type": "string", "enum": ["a_renover","date","standard","renove","haut_de_gamme","inconnu"]},
16 + "natural_light": {"type": "string", "enum": ["sombre","moyen","lumineux","tres_lumineux"]},
17 + "clutter": {"type": "string", "enum": ["vide","epure","normal","encombre"]},
18 + "curb_appeal": {"type": "integer", "description": "attrait extérieur 1-10; 0 si non applicable"},
19 + "materials": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","granit_quartz","stainless","brique","pierre","vinyle","beton","tapis"]}},
20 + "view_quality": {"type": "string", "enum": ["aucune","urbaine","verdure","eau","montagne","inconnu"]},
21 + "staged": {"type": "boolean", "description": "mise en scène pro / home staging"},
22 + "photo_quality": {"type": "integer", "description": "qualité de la photo elle-même 1-10"}
23 + },
24 + "required": ["shot_type","condition_score","renovation_level","natural_light","clutter","curb_appeal","materials","view_quality","staged","photo_quality"]
25 +}
26 +SYS = ("Tu es un évaluateur immobilier objectif. Analyse la photo et renvoie des "
27 + "attributs FACTUELS et reproductibles au format JSON. Sois objectif : base "
28 + "les scores sur des critères concrets (état des surfaces, finitions, "
29 + "luminosité), pas sur un goût personnel.")
30 +
31 +def analyze(url):
32 + body = {"model": "claude-haiku-4-5", "max_tokens": 400,
33 + "system": SYS,
34 + "messages": [{"role": "user", "content": [
35 + {"type": "image", "source": {"type": "url", "url": url}},
36 + {"type": "text", "text": "Analyse cette photo de propriété."}]}],
37 + "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}}
38 + req = urllib.request.Request(
39 + "https://api.anthropic.com/v1/messages", data=json.dumps(body).encode(),
40 + headers={"x-api-key": KEY, "anthropic-version": "2023-06-01",
41 + "content-type": "application/json"})
42 + with urllib.request.urlopen(req, timeout=90) as r:
43 + out = json.load(r)
44 + txt = next(b["text"] for b in out["content"] if b["type"] == "text")
45 + return json.loads(txt), out.get("usage")
46 +
47 +data_r = json.load(open("data/raw/dataset_realtor-property-crawler_2026-07-03_10-36-26-001.json", encoding="utf-8"))
48 +data_d = json.load(open("data/raw/dataset_duproprio-property-crawler_2026-07-03_00-37-49-449.json", encoding="utf-8"))
49 +# realtor: 1re image = façade ; duproprio: image dont caption ~ Kitchen
50 +tests = []
51 +r0 = next(d for d in data_r if d.get("images"))
52 +tests.append(("REALTOR 1re photo (façade attendue)", r0["images"][0]["url"]))
53 +d0 = next(d for d in data_d if d.get("images"))
54 +kit = next((im for im in d0["images"] if "kitchen" in (im.get("caption") or "").lower()), d0["images"][0])
55 +tests.append(("DUPROPRIO (caption=%s)" % (kit.get("caption","?")[:30]), kit["url"]))
56 +
57 +for label, url in tests:
58 + try:
59 + specs, usage = analyze(url)
60 + print(f"\n=== {label} ===")
61 + print("URL:", url[:90])
62 + print("tokens:", usage)
63 + print(json.dumps(specs, ensure_ascii=False, indent=1))
64 + except urllib.error.HTTPError as e:
65 + print("HTTP", e.code, e.read().decode()[:500])
added scripts/01_extract_specs_llm.py +135 −0
@@ -0,0 +1,135 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""
4 +Extraction des specs de propriété via GPT-4o-mini, à partir des descriptions.
5 +Parallèle, résumable (cache specs.jsonl), avec retry/backoff. Aucune dépendance.
6 +"""
7 +import json, os, time, threading, urllib.request, urllib.error
8 +from concurrent.futures import ThreadPoolExecutor, as_completed
9 +
10 +SRC = "data/processed/maisons_avec_prix.json"
11 +OUT = "data/processed/specs.jsonl"
12 +WORKERS = int(os.environ.get("WORKERS", "24"))
13 +NSHARDS = int(os.environ.get("NSHARDS", "1"))
14 +SHARD = int(os.environ.get("SHARD", "0"))
15 +MODEL = "gpt-4o-mini"
16 +
17 +# clé
18 +for line in open(".env", encoding="utf-8"):
19 + if line.startswith("OPENAI_API_KEY="):
20 + os.environ["OPENAI_API_KEY"] = line.split("=", 1)[1].strip()
21 +KEY = os.environ["OPENAI_API_KEY"]
22 +
23 +SCHEMA = {
24 + "type": "object", "additionalProperties": False,
25 + "properties": {
26 + "condition": {"type": "string", "enum": ["a_renover","original","bon","renove","refait_neuf","neuf","inconnu"]},
27 + "turnkey": {"type": "boolean"},
28 + "recently_renovated": {"type": "boolean"},
29 + "flooring": {"type": "array", "items": {"type": "string", "enum": ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"]}},
30 + "countertop": {"type": "string", "enum": ["granit","quartz","marbre","stratifie","bois","autre","inconnu"]},
31 + "heating": {"type": "array", "items": {"type": "string", "enum": ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"]}},
32 + "garage": {"type": "boolean"},
33 + "pool": {"type": "boolean"},
34 + "waterfront": {"type": "boolean"},
35 + "view": {"type": "string", "enum": ["eau","montagne","ville","aucune","inconnu"]},
36 + "finished_basement": {"type": "boolean"},
37 + "fireplace": {"type": "boolean"},
38 + "central_ac": {"type": "boolean"},
39 + "new_construction": {"type": "boolean"},
40 + "income_property": {"type": "boolean"},
41 + "renovated_kitchen": {"type": "boolean"},
42 + "renovated_bathroom": {"type": "boolean"},
43 + "large_lot": {"type": "boolean"},
44 + "quiet_area": {"type": "boolean"},
45 + "highlights": {"type": "array", "items": {"type": "string"}}
46 + },
47 + "required": ["condition","turnkey","recently_renovated","flooring","countertop",
48 + "heating","garage","pool","waterfront","view","finished_basement",
49 + "fireplace","central_ac","new_construction","income_property",
50 + "renovated_kitchen","renovated_bathroom","large_lot","quiet_area",
51 + "highlights"]
52 +}
53 +SYS = ("Tu es un extracteur d'attributs immobiliers. À partir de la description "
54 + "d'une propriété (français ou anglais), renvoie les specs au format JSON "
55 + "strict. Base-toi UNIQUEMENT sur le texte ; si une info est absente, mets "
56 + "false / 'inconnu' / [] selon le champ. highlights = 3 à 6 mots-clés de vente.")
57 +
58 +def call(desc, tries=5):
59 + body = {"model": MODEL, "temperature": 0,
60 + "messages": [{"role": "system", "content": SYS},
61 + {"role": "user", "content": desc[:4000]}],
62 + "response_format": {"type": "json_schema", "json_schema":
63 + {"name": "specs", "strict": True, "schema": SCHEMA}}}
64 + data = json.dumps(body).encode()
65 + for t in range(tries):
66 + try:
67 + req = urllib.request.Request(
68 + "https://api.openai.com/v1/chat/completions", data=data,
69 + headers={"Authorization": f"Bearer {KEY}",
70 + "Content-Type": "application/json"})
71 + with urllib.request.urlopen(req, timeout=90) as r:
72 + out = json.load(r)
73 + return json.loads(out["choices"][0]["message"]["content"])
74 + except urllib.error.HTTPError as e:
75 + if e.code in (429, 500, 502, 503, 504) and t < tries - 1:
76 + time.sleep(2 ** t + 0.5); continue
77 + raise
78 + except (urllib.error.URLError, TimeoutError):
79 + if t < tries - 1:
80 + time.sleep(2 ** t + 0.5); continue
81 + raise
82 + return None
83 +
84 +# charge données + cache
85 +data = json.load(open(SRC, encoding="utf-8"))
86 +done = set()
87 +if os.path.exists(OUT):
88 + for line in open(OUT, encoding="utf-8"):
89 + try:
90 + done.add(json.loads(line)["uid"])
91 + except Exception:
92 + pass
93 +
94 +todo = []
95 +for d in data:
96 + uid = f"{d.get('source')}:{d.get('listingId')}"
97 + desc = d.get("description")
98 + if uid in done or not desc:
99 + continue
100 + todo.append((uid, desc))
101 +
102 +# sharding : partition disjointe et complète par index (déterministe)
103 +if NSHARDS > 1:
104 + todo = [t for i, t in enumerate(todo) if i % NSHARDS == SHARD]
105 +
106 +print(f"[shard {SHARD}/{NSHARDS}, workers={WORKERS}] Total={len(data)} "
107 + f"déjà fait={len(done)} à traiter (ce shard)={len(todo)}", flush=True)
108 +
109 +lock = threading.Lock()
110 +counter = {"n": 0, "err": 0}
111 +fh = open(OUT, "a", encoding="utf-8")
112 +
113 +def work(item):
114 + uid, desc = item
115 + try:
116 + specs = call(desc)
117 + rec = {"uid": uid, "specs": specs}
118 + except Exception as e:
119 + rec = {"uid": uid, "error": str(e)[:200]}
120 + with lock:
121 + fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
122 + fh.flush()
123 + counter["n"] += 1
124 + if "error" in rec:
125 + counter["err"] += 1
126 + if counter["n"] % 250 == 0:
127 + print(f" {counter['n']}/{len(todo)} (err={counter['err']})",
128 + flush=True)
129 +
130 +t0 = time.time()
131 +with ThreadPoolExecutor(max_workers=WORKERS) as ex:
132 + list(ex.map(work, todo))
133 +fh.close()
134 +print(f"Terminé : {counter['n']} traités, {counter['err']} erreurs, "
135 + f"{time.time()-t0:.0f}s", flush=True)
added scripts/02_facade_vision.py +135 −0
@@ -0,0 +1,135 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""
4 +Analyse de la photo de façade (1/annonce) via Claude Haiku 4.5.
5 +Attributs visuels OBJECTIFS. Parallèle, résumable (facade_specs.jsonl), retry.
6 +"""
7 +import json, os, time, threading, urllib.request, urllib.error
8 +from concurrent.futures import ThreadPoolExecutor
9 +
10 +SRC = "data/processed/maisons_avec_prix.json"
11 +OUT = "data/processed/facade_specs.jsonl"
12 +WORKERS = int(os.environ.get("WORKERS", "16"))
13 +NSHARDS = int(os.environ.get("NSHARDS", "1"))
14 +SHARD = int(os.environ.get("SHARD", "0"))
15 +MODEL = "claude-haiku-4-5"
16 +
17 +for line in open(".env", encoding="utf-8"):
18 + if line.startswith("ANTHROPIC_API_KEY="):
19 + KEY = line.split("=", 1)[1].strip()
20 +
21 +SCHEMA = {
22 + "type": "object", "additionalProperties": False,
23 + "properties": {
24 + "is_exterior": {"type": "boolean", "description": "la photo montre-t-elle l'extérieur/façade"},
25 + "building_type_visual": {"type": "string", "enum": ["maison_detachee","jumele","rangee","condo_immeuble","chalet","terrain","autre"]},
26 + "condition_score": {"type": "integer", "description": "état visuel 1=délabré à 10=neuf/luxueux"},
27 + "curb_appeal": {"type": "integer", "description": "attrait extérieur 1-10"},
28 + "renovation_level": {"type": "string", "enum": ["a_renover","date","standard","renove","haut_de_gamme","inconnu"]},
29 + "exterior_materials": {"type": "array", "items": {"type": "string", "enum": ["brique","pierre","vinyle","bois","aluminium","stucco","crepi","beton","autre"]}},
30 + "landscaping": {"type": "string", "enum": ["aucun","minimal","entretenu","amenage_soigne"]},
31 + "has_garage_visible": {"type": "boolean"},
32 + "storeys_visual": {"type": "string", "enum": ["1","1.5","2","3+","inconnu"]},
33 + "setting": {"type": "string", "enum": ["urbain_dense","banlieue","rural","bord_eau","boise","inconnu"]},
34 + "photo_quality": {"type": "integer", "description": "qualité de la photo 1-10"},
35 + "curb_luxury": {"type": "boolean", "description": "signes de luxe (grande taille, matériaux nobles, prestige)"}
36 + },
37 + "required": ["is_exterior","building_type_visual","condition_score","curb_appeal",
38 + "renovation_level","exterior_materials","landscaping","has_garage_visible",
39 + "storeys_visual","setting","photo_quality","curb_luxury"]
40 +}
41 +SYS = ("Tu es un évaluateur immobilier objectif. Analyse la photo (censée être la "
42 + "façade d'une propriété) et renvoie des attributs FACTUELS et reproductibles "
43 + "au format JSON. Fonde les scores sur des critères concrets (état des "
44 + "surfaces, matériaux, aménagement), pas sur un goût personnel. Si la photo "
45 + "n'est pas un extérieur, mets is_exterior=false et remplis au mieux.")
46 +
47 +ROOM_KW = ("kitchen","cuisine","bathroom","salle de bain","bedroom","chambre",
48 + "living","salon","basement","sous-sol","dining","salle à manger",
49 + "garage","office","bureau","laundry","closet","stair")
50 +
51 +def pick_facade(d):
52 + imgs = d.get("images") or []
53 + if not imgs:
54 + return None
55 + if d.get("source") == "realtor":
56 + return imgs[0].get("url") # 1re photo = extérieur
57 + # duproprio : caption = adresse (sans mot-clé de pièce) = photo extérieure
58 + for im in imgs:
59 + cap = (im.get("caption") or "").lower()
60 + if cap and not any(k in cap for k in ROOM_KW):
61 + return im.get("url")
62 + return imgs[0].get("url")
63 +
64 +def call(url, tries=5):
65 + body = {"model": MODEL, "max_tokens": 400, "system": SYS,
66 + "messages": [{"role": "user", "content": [
67 + {"type": "image", "source": {"type": "url", "url": url}},
68 + {"type": "text", "text": "Analyse cette photo de propriété."}]}],
69 + "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}}
70 + data = json.dumps(body).encode()
71 + for t in range(tries):
72 + try:
73 + req = urllib.request.Request(
74 + "https://api.anthropic.com/v1/messages", data=data,
75 + headers={"x-api-key": KEY, "anthropic-version": "2023-06-01",
76 + "content-type": "application/json"})
77 + with urllib.request.urlopen(req, timeout=120) as r:
78 + out = json.load(r)
79 + txt = next(b["text"] for b in out["content"] if b["type"] == "text")
80 + return json.loads(txt)
81 + except urllib.error.HTTPError as e:
82 + # 400 inclus : fetch d'URL image transitoire côté Anthropic sous charge
83 + if e.code in (400, 429, 500, 502, 503, 504, 529) and t < tries - 1:
84 + time.sleep(2 ** t + 1); continue
85 + raise
86 + except (urllib.error.URLError, TimeoutError, ValueError):
87 + if t < tries - 1:
88 + time.sleep(2 ** t + 1); continue
89 + raise
90 + return None
91 +
92 +data = json.load(open(SRC, encoding="utf-8"))
93 +done = set()
94 +if os.path.exists(OUT):
95 + for line in open(OUT, encoding="utf-8"):
96 + try: done.add(json.loads(line)["uid"])
97 + except Exception: pass
98 +
99 +todo = []
100 +for d in data:
101 + uid = f"{d.get('source')}:{d.get('listingId')}"
102 + if uid in done:
103 + continue
104 + url = pick_facade(d)
105 + if url:
106 + todo.append((uid, url))
107 +
108 +if NSHARDS > 1:
109 + todo = [t for i, t in enumerate(todo) if i % NSHARDS == SHARD]
110 +
111 +print(f"[shard {SHARD}/{NSHARDS}, workers={WORKERS}] Total={len(data)} "
112 + f"déjà fait={len(done)} à traiter (ce shard)={len(todo)}", flush=True)
113 +
114 +lock = threading.Lock()
115 +c = {"n": 0, "err": 0}
116 +fh = open(OUT, "a", encoding="utf-8")
117 +
118 +def work(item):
119 + uid, url = item
120 + try:
121 + rec = {"uid": uid, "facade": call(url)}
122 + except Exception as e:
123 + rec = {"uid": uid, "error": str(e)[:200]}
124 + with lock:
125 + fh.write(json.dumps(rec, ensure_ascii=False) + "\n"); fh.flush()
126 + c["n"] += 1
127 + if "error" in rec: c["err"] += 1
128 + if c["n"] % 250 == 0:
129 + print(f" {c['n']}/{len(todo)} (err={c['err']})", flush=True)
130 +
131 +t0 = time.time()
132 +with ThreadPoolExecutor(max_workers=WORKERS) as ex:
133 + list(ex.map(work, todo))
134 +fh.close()
135 +print(f"Terminé : {c['n']} traités, {c['err']} erreurs, {time.time()-t0:.0f}s", flush=True)
added scripts/03_merge_llm_features.py +84 −0
@@ -0,0 +1,84 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Fusionne les features GPT (specs.jsonl) aux données maison (maisons_avec_prix.json)."""
4 +import json, csv
5 +
6 +# 1) charger specs GPT
7 +specs = {}
8 +for line in open("data/processed/specs.jsonl", encoding="utf-8"):
9 + r = json.loads(line)
10 + if r.get("specs"):
11 + specs[r["uid"]] = r["specs"]
12 +print(f"Specs GPT chargées : {len(specs)}")
13 +
14 +# 2) charger maisons
15 +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8"))
16 +
17 +# 3) JSON fusionné : record complet + gptSpecs imbriqué
18 +merged = []
19 +n_match = 0
20 +for d in data:
21 + uid = f"{d.get('source')}:{d.get('listingId')}"
22 + d2 = dict(d)
23 + d2["uid"] = uid
24 + s = specs.get(uid)
25 + d2["gptSpecs"] = s # None si absent
26 + if s:
27 + n_match += 1
28 + merged.append(d2)
29 +json.dump(merged, open("data/processed/maisons_specs.json", "w", encoding="utf-8"),
30 + ensure_ascii=False)
31 +print(f"JSON fusionné -> maisons_specs.json ({len(merged)} records, "
32 + f"{n_match} avec specs)")
33 +
34 +# 4) CSV aplati pour analyse
35 +BOOL = ["turnkey","recently_renovated","garage","pool","waterfront",
36 + "finished_basement","fireplace","central_ac","new_construction",
37 + "income_property","renovated_kitchen","renovated_bathroom",
38 + "large_lot","quiet_area"]
39 +FLOOR = ["bois_franc","ceramique","plancher_flottant","tapis","vinyle","beton","autre"]
40 +HEAT = ["electrique","plinthes","thermopompe","bois","gaz","mazout","radiant","geothermie","autre"]
41 +
42 +cols = (["uid","source","listingId","price","currency","city","province",
43 + "ptype","sqft_raw","lot_raw","beds","baths","yearBuilt","parking",
44 + "has_gpt","g_condition","g_countertop","g_view"]
45 + + ["g_"+b for b in BOOL]
46 + + ["fl_"+f for f in FLOOR]
47 + + ["heat_"+h for h in HEAT]
48 + + ["g_highlights"])
49 +
50 +with open("data/processed/maisons_specs.csv", "w", newline="", encoding="utf-8") as fh:
51 + w = csv.writer(fh)
52 + w.writerow(cols)
53 + for d in data:
54 + uid = f"{d.get('source')}:{d.get('listingId')}"
55 + p = d.get("property") or {}
56 + ad = d.get("address") or {}
57 + s = specs.get(uid)
58 + row = [uid, d.get("source"), d.get("listingId"), d.get("priceNumber"),
59 + d.get("currency"), ad.get("city"), ad.get("province"),
60 + (p.get("buildingType") or p.get("type")),
61 + p.get("squareFeet"), p.get("lotSize"),
62 + p.get("bedrooms"), p.get("bathrooms"), p.get("yearBuilt"),
63 + p.get("parkingSpaces"),
64 + 1 if s else 0,
65 + (s or {}).get("condition",""),
66 + (s or {}).get("countertop",""),
67 + (s or {}).get("view","")]
68 + row += [int(bool((s or {}).get(b))) if s else "" for b in BOOL]
69 + fl = set((s or {}).get("flooring") or [])
70 + row += [int(f in fl) if s else "" for f in FLOOR]
71 + ht = set((s or {}).get("heating") or [])
72 + row += [int(h in ht) if s else "" for h in HEAT]
73 + row += ["; ".join((s or {}).get("highlights") or [])]
74 + w.writerow(row)
75 +print(f"CSV aplati -> maisons_specs.csv ({len(cols)} colonnes)")
76 +
77 +# 5) petit récap de distribution
78 +import collections
79 +cond = collections.Counter(s.get("condition") for s in specs.values())
80 +print("\nRépartition 'condition' :", dict(cond))
81 +for b in ["turnkey","recently_renovated","central_ac","pool","waterfront",
82 + "renovated_kitchen","income_property"]:
83 + n = sum(1 for s in specs.values() if s.get(b))
84 + print(f" {b:20s}: {n:5d} ({n/len(specs)*100:.1f}%)")
added scripts/04_hedonic_ols_base.py +338 −0
@@ -0,0 +1,338 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""
4 +Nettoyage des aberrations + modèle hédonique de prix des maisons
5 +(Realtor + DuProprio combinés). Log-linéaire, effets fixes ville, SE robustes.
6 +"""
7 +import json, re, warnings
8 +import numpy as np
9 +import pandas as pd
10 +import statsmodels.formula.api as smf
11 +
12 +warnings.filterwarnings("ignore")
13 +SRC = "data/processed/maisons_avec_prix.json"
14 +CUR_YEAR = 2026
15 +
16 +# ---------- Chargement ----------
17 +with open(SRC, encoding="utf-8") as f:
18 + data = json.load(f)
19 +print(f"Chargé : {len(data)} enregistrements")
20 +
21 +# ---------- Parsers ----------
22 +def parse_area(s):
23 + """Retourne une superficie en pieds carrés (ft²)."""
24 + if not s or not isinstance(s, str):
25 + return np.nan
26 + txt = s.lower()
27 + is_m2 = ("m2" in txt) or ("m²" in txt)
28 + # garder chiffres, points, virgules, espaces ; retirer unités
29 + num = re.sub(r"[^0-9.,\s]", "", txt).strip()
30 + num = num.replace(" ", "").replace(",", "") # séparateurs de milliers
31 + try:
32 + v = float(num)
33 + except ValueError:
34 + return np.nan
35 + if v <= 0:
36 + return np.nan
37 + return v * 10.7639 if is_m2 else v
38 +
39 +def clean_city(city, source):
40 + if not city or not isinstance(city, str):
41 + return None
42 + c = city.split("(")[0].strip() # retire quartiers realtor
43 + c = re.sub(r"\s+", " ", c).lower().strip()
44 + return c or None
45 +
46 +def to_int(x):
47 + try:
48 + return int(x)
49 + except (TypeError, ValueError):
50 + return np.nan
51 +
52 +# amenity keywords -> recherche dans features + description + title (bilingue)
53 +AMENITIES = {
54 + "pool": ["swimming pool", "piscine", "inground pool", "pool"],
55 + "waterfront": ["waterfront", "bord de l'eau", "navigable", "water access",
56 + "lakefront", "riverfront", "au bord de", "waterfront"],
57 + "garage": ["garage"],
58 + "fireplace": ["fireplace", "foyer", "wood stove", "poêle"],
59 + "central_air": ["central air", "air conditionné", "climatisation", "a/c",
60 + "heat-pump", "thermopompe", "air exchanger"],
61 + "finished_bsmt":["totally finished", "finished basement", "sous-sol fini",
62 + "basement"],
63 + "near_water": ["near water", "waterfront", "navigable"],
64 + "cul_de_sac": ["cul-de-sac", "cul de sac"],
65 + "golf": ["golf"],
66 + "ski": ["ski"],
67 + "near_park": ["park", "parc"],
68 + "near_school": ["school", "école", "cegep", "university", "université",
69 + "elementary", "high school"],
70 + "near_highway":["highway", "autoroute"],
71 + "transit": ["public transit", "public transportation", "transport"],
72 + "no_rear_neighbor":["no backyard neighbors", "no rear neighbor",
73 + "sans voisin arrière"],
74 +}
75 +
76 +# ---------- Parsing des pièces (Realtor) ----------
77 +def _side_ft(side):
78 + s = side.lower()
79 + if "ft" in s:
80 + ft = re.search(r"([\d.]+)\s*ft", s)
81 + inch = re.search(r"([\d.]+)\s*in", s)
82 + return (float(ft.group(1)) if ft else 0) + \
83 + (float(inch.group(1)) / 12 if inch else 0)
84 + m = re.search(r"([\d.,]+)\s*m", s)
85 + if m:
86 + try:
87 + return float(m.group(1).replace(",", ".")) * 3.28084
88 + except ValueError:
89 + return None
90 + return None
91 +
92 +def _room_area(dim):
93 + if not dim or "x" not in dim:
94 + return None
95 + a, _, b = dim.partition("x")
96 + la, lb = _side_ft(a), _side_ft(b)
97 + if la and lb and 1 < la < 200 and 1 < lb < 200:
98 + return la * lb
99 + return None
100 +
101 +ROOM_FLAGS = { # pièces à valeur potentielle
102 + "r_office": ["office"],
103 + "r_family": ["family room"],
104 + "r_workshop": ["workshop"],
105 + "r_winecellar": ["wine cellar"],
106 + "r_sunroom": ["sunroom", "solarium"],
107 + "r_mezzanine": ["mezzanine"],
108 + "r_library": ["library"],
109 + "r_laundry": ["laundry"],
110 +}
111 +
112 +def parse_rooms(rooms):
113 + out = {"room_area": np.nan, "n_rooms": 0, "n_bed_r": 0, "n_bath_r": 0,
114 + "has_primary_bed": 0, "has_basement": 0, "n_levels_r": 0}
115 + for k in ROOM_FLAGS:
116 + out[k] = 0
117 + if not rooms:
118 + return out
119 + area, levels = 0.0, set()
120 + for r in rooms:
121 + nm = (r.get("name") or "").lower()
122 + lv = (r.get("level") or "").lower()
123 + out["n_rooms"] += 1
124 + if lv:
125 + levels.add(lv)
126 + if "basement" in lv or "sous-sol" in lv:
127 + out["has_basement"] = 1
128 + if "primary bedroom" in nm:
129 + out["has_primary_bed"] = 1
130 + out["n_bed_r"] += 1
131 + elif "bedroom" in nm:
132 + out["n_bed_r"] += 1
133 + if "bathroom" in nm:
134 + out["n_bath_r"] += 1
135 + for k, kws in ROOM_FLAGS.items():
136 + if any(w in nm for w in kws):
137 + out[k] = 1
138 + a = _room_area(r.get("dimensions"))
139 + if a:
140 + area += a
141 + out["room_area"] = area if area > 0 else np.nan
142 + out["n_levels_r"] = len(levels)
143 + return out
144 +
145 +rows = []
146 +for d in data:
147 + prop = d.get("property") or {}
148 + addr = d.get("address") or {}
149 + feats = d.get("features") or []
150 + blob = (" ".join(feats) + " " + (d.get("description") or "") + " " +
151 + (d.get("title") or "")).lower()
152 +
153 + yb = to_int(prop.get("yearBuilt"))
154 + age = (CUR_YEAR - yb) if (not pd.isna(yb) and 1700 <= yb <= CUR_YEAR) else np.nan
155 +
156 + ptype = prop.get("buildingType") or prop.get("type") # realtor/duproprio
157 + storeys = prop.get("storeys") or prop.get("levels")
158 + try:
159 + storeys = float(str(storeys).replace(",", "."))
160 + except (TypeError, ValueError):
161 + storeys = np.nan
162 +
163 + rec = {
164 + "price": d.get("priceNumber"),
165 + "source": d.get("source"),
166 + "sqft": parse_area(prop.get("squareFeet")),
167 + "lot_sqft": parse_area(prop.get("lotSize")),
168 + "beds": to_int(prop.get("bedrooms")),
169 + "baths": to_int(prop.get("bathrooms")),
170 + "parking": to_int(prop.get("parkingSpaces")), # realtor seulement
171 + "storeys": storeys,
172 + "age": age,
173 + "ptype": (ptype or "Unknown").strip(),
174 + "city": clean_city(addr.get("city"), d.get("source")),
175 + "region": (addr.get("region") or "").strip().lower() or None,
176 + }
177 + for name, kws in AMENITIES.items():
178 + rec[name] = int(any(k in blob for k in kws))
179 + # signaux "autre chose" : photos + visite virtuelle
180 + rec["n_images"] = len(d.get("images") or [])
181 + rec["virtual_tour"] = int(bool(d.get("virtualTourUrl")))
182 + # détail des pièces
183 + rec.update(parse_rooms(d.get("rooms")))
184 + rows.append(rec)
185 +
186 +df = pd.DataFrame(rows)
187 +print(f"DataFrame brut : {df.shape}")
188 +
189 +# ---------- Nettoyage des aberrations ----------
190 +n0 = len(df)
191 +df = df[df["price"].notna()]
192 +# prix plausibles pour une maison au Québec
193 +df = df[(df["price"] >= 50_000) & (df["price"] <= 15_000_000)]
194 +# superficie habitable : borner si présente, garder les NA (imputés + flag)
195 +df.loc[(df["sqft"] < 300) | (df["sqft"] > 20_000), "sqft"] = np.nan
196 +# terrain plausible (garder NA autorisé -> mais on impute plus bas)
197 +df.loc[(df["lot_sqft"] < 200) | (df["lot_sqft"] > 5_000_000), "lot_sqft"] = np.nan
198 +# chambres / sdb plausibles
199 +df = df[(df["beds"].between(0, 15)) | (df["beds"].isna())]
200 +df = df[(df["baths"].between(0, 12)) | (df["baths"].isna())]
201 +df.loc[df["beds"] > 15, "beds"] = np.nan
202 +df.loc[df["baths"] > 12, "baths"] = np.nan
203 +# prix au pied carré aberrant (sur les lignes qui ont une superficie réelle)
204 +df["ppsf"] = df["price"] / df["sqft"]
205 +lo, hi = df["ppsf"].quantile([0.005, 0.995])
206 +bad = df["ppsf"].notna() & ((df["ppsf"] < lo) | (df["ppsf"] > hi))
207 +df = df[~bad]
208 +print(f"Après nettoyage : {len(df)} (retirés : {n0 - len(df)}) "
209 + f"[filtre ppsf {lo:.0f}–{hi:.0f} $/ft² sur superficies réelles]")
210 +
211 +# ---------- Superficie : réelle > reconstruite via pièces > imputée ----------
212 +# calibration : ratio médian (superficie réelle / somme des pièces) sur recouvrement
213 +ov = df[df["sqft"].notna() & df["room_area"].notna() &
214 + (df["room_area"] > 200)]
215 +ratio = (ov["sqft"] / ov["room_area"]).median()
216 +print(f"Calibration surface pièces : ratio réel/pièces = {ratio:.3f} "
217 + f"(n={len(ov)})")
218 +room_est = df["room_area"] * ratio
219 +room_ok = df["room_area"].notna() & (room_est.between(300, 20_000))
220 +
221 +df["sqft_final"] = df["sqft"]
222 +fill = df["sqft_final"].isna() & room_ok
223 +df.loc[fill, "sqft_final"] = room_est[fill]
224 +df["sqft_measured"] = df["sqft_final"].notna().astype(int) # réelle ou pièces
225 +n_room_fill = int(fill.sum())
226 +print(f"Superficies : réelles={int(df['sqft'].notna().sum())}, "
227 + f"complétées via pièces=+{n_room_fill}, "
228 + f"mesurées au total={int(df['sqft_measured'].sum())}/{len(df)}")
229 +
230 +# imputation du reste = médiane par type
231 +df["has_sqft"] = df["sqft_measured"]
232 +df["sqft_imp"] = df.groupby("ptype")["sqft_final"].transform(
233 + lambda s: s.fillna(s.median()))
234 +df["sqft_imp"] = df["sqft_imp"].fillna(df["sqft_final"].median())
235 +
236 +df["has_lot"] = df["lot_sqft"].notna().astype(int)
237 +df["lot_sqft"] = df["lot_sqft"].fillna(df["lot_sqft"].median())
238 +df["has_age"] = df["age"].notna().astype(int)
239 +df["age"] = df["age"].fillna(df["age"].median()).clip(lower=0)
240 +df["beds"] = df["beds"].fillna(df["beds"].median())
241 +df["baths"] = df["baths"].fillna(df["baths"].median())
242 +df["parking"] = df["parking"].fillna(0)
243 +df["storeys"] = df["storeys"].fillna(df["storeys"].median())
244 +
245 +df["ln_price"] = np.log(df["price"])
246 +df["ln_sqft"] = np.log(df["sqft_imp"])
247 +df["ln_lot"] = np.log(df["lot_sqft"])
248 +df["age2"] = df["age"] ** 2
249 +
250 +# Effets fixes ville : regrouper les villes rares
251 +vc = df["city"].value_counts()
252 +keep = set(vc[vc >= 20].index)
253 +df["city_fe"] = df["city"].where(df["city"].isin(keep), other="__other__")
254 +df["city_fe"] = df["city_fe"].fillna("__other__")
255 +print(f"Villes en effets fixes : {df['city_fe'].nunique()} "
256 + f"(seuil ≥20 listings ; {len(keep)} villes nommées)")
257 +
258 +# Type de propriété harmonisé (regrouper rares)
259 +tc = df["ptype"].value_counts()
260 +keept = set(tc[tc >= 30].index)
261 +df["ptype_fe"] = df["ptype"].where(df["ptype"].isin(keept), other="Other")
262 +
263 +# ---------- Modèles hédoniques ----------
264 +amen = list(AMENITIES.keys())
265 +def pct(b):
266 + return (np.exp(b) - 1) * 100
267 +
268 +# variables "détail des pièces" + autres signaux (Realtor surtout)
269 +df["n_images_c"] = df["n_images"].clip(upper=60) # écrête les extrêmes
270 +room_vars = ["n_rooms", "n_levels_r", "has_primary_bed", "has_basement",
271 + "r_office", "r_family", "r_workshop", "r_winecellar",
272 + "r_sunroom", "r_mezzanine", "r_library", "r_laundry"]
273 +extra = ["n_images_c", "virtual_tour"] + room_vars
274 +
275 +base = ("ln_lot + has_lot + beds + baths + parking + storeys "
276 + "+ age + age2 + has_age + C(source) + C(ptype_fe) + "
277 + + " + ".join(amen) + " + " + " + ".join(extra)
278 + + " + C(city_fe)")
279 +fA = "ln_price ~ ln_sqft + " + base # riche (superficie réelle)
280 +fB = "ln_price ~ " + base # large (sans superficie)
281 +
282 +dfA = df[df["has_sqft"] == 1].copy()
283 +print(f"\nModèle A (avec superficie réelle) : N={len(dfA)}")
284 +mA = smf.ols(fA, data=dfA).fit(cov_type="HC3")
285 +print(f"Modèle B (sans superficie, couverture large) : N={len(df)}")
286 +mB = smf.ols(fB, data=df).fit(cov_type="HC3")
287 +
288 +sdA, sdB = np.std(mA.resid), np.std(mB.resid)
289 +
290 +def report(m, sd, name):
291 + print("\n" + "=" * 70)
292 + print(f"{name} N={int(m.nobs)} R²={m.rsquared:.4f} "
293 + f"R²_adj={m.rsquared_adj:.4f} "
294 + f"erreur typique ±{(np.exp(sd)-1)*100:.1f}%")
295 + print("=" * 70)
296 + pr, pv = m.params, m.pvalues
297 + show = [p for p in pr.index if not p.startswith("C(city_fe)")]
298 + print(f"{'variable':<30}{'coef':>10}{'~%impact':>11}{'p':>9}")
299 + for p in show:
300 + st = "***" if pv[p]<.01 else "**" if pv[p]<.05 else "*" if pv[p]<.1 else ""
301 + pc = f"{pct(pr[p]):+.1f}%" if abs(pr[p]) < 3 else ""
302 + print(f"{p:<30}{pr[p]:>10.4f}{pc:>11}{pv[p]:>9.3g} {st}")
303 +
304 +report(mA, sdA, "MODÈLE A — hédonique riche (superficie incluse)")
305 +report(mB, sdB, "MODÈLE B — couverture large (sans superficie)")
306 +
307 +# ---------- Prédictions combinées ----------
308 +# A pour les biens à superficie réelle, B sinon
309 +predA = np.exp(mA.predict(df)) * np.exp(sdA**2 / 2)
310 +predB = np.exp(mB.predict(df)) * np.exp(sdB**2 / 2)
311 +df["pred_price"] = np.where(df["has_sqft"] == 1, predA, predB)
312 +df["model_used"] = np.where(df["has_sqft"] == 1, "A", "B")
313 +df["resid_pct"] = (df["price"] / df["pred_price"] - 1) * 100
314 +
315 +out_cols = (["source", "city", "ptype", "price", "pred_price", "resid_pct",
316 + "model_used", "sqft", "room_area", "sqft_final", "sqft_measured",
317 + "lot_sqft", "beds", "baths", "parking", "storeys", "age",
318 + "n_images", "virtual_tour"] + amen + room_vars)
319 +df[out_cols].to_csv("data/processed/maisons_hedonic.csv", index=False)
320 +print(f"\nDataset final + prédictions -> maisons_hedonic.csv ({len(df)} lignes)")
321 +
322 +# top villes (effets fixes du modèle A)
323 +cityfe = {k.split("T.")[1].rstrip("]"): v for k, v in mA.params.items()
324 + if k.startswith("C(city_fe)")}
325 +cf = pd.Series(cityfe).sort_values(ascending=False)
326 +print("\nTop 12 villes — prime de localisation (Modèle A, vs référence):")
327 +for c, b in cf.head(12).items():
328 + print(f" {c:<30}{pct(b):+.1f}%")
329 +print("Bottom 6:")
330 +for c, b in cf.tail(6).items():
331 + print(f" {c:<30}{pct(b):+.1f}%")
332 +
333 +with open("results/hedonic_summary.txt", "w", encoding="utf-8") as fh:
334 + fh.write("=== MODELE A (avec superficie) ===\n")
335 + fh.write(str(mA.summary()))
336 + fh.write("\n\n=== MODELE B (sans superficie) ===\n")
337 + fh.write(str(mB.summary()))
338 +print("Résumés complets -> hedonic_summary.txt")
added scripts/05_hedonic_ols_v4_text.py +249 −0
@@ -0,0 +1,249 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""
4 +Modèle hédonique v4 : ajoute les specs extraites par GPT-4o-mini (specs.jsonl)
5 +au modèle v3. Réutilise la même logique de nettoyage/features + effets fixes.
6 +"""
7 +import json, re, warnings
8 +import numpy as np
9 +import pandas as pd
10 +import statsmodels.formula.api as smf
11 +
12 +warnings.filterwarnings("ignore")
13 +CUR_YEAR = 2026
14 +
15 +# ---------- charge specs GPT ----------
16 +specs = {}
17 +for line in open("data/processed/specs.jsonl", encoding="utf-8"):
18 + try:
19 + r = json.loads(line)
20 + except Exception:
21 + continue
22 + if "specs" in r and r.get("specs"):
23 + specs[r["uid"]] = r["specs"]
24 +print(f"Specs GPT chargées : {len(specs)}")
25 +
26 +# specs -> features numériques (dummies + catégorielles réduites)
27 +COND_SCORE = {"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,
28 + "neuf":5,"inconnu":2}
29 +BOOL_SPECS = ["turnkey","recently_renovated","garage","pool","waterfront",
30 + "finished_basement","fireplace","central_ac","new_construction",
31 + "income_property","renovated_kitchen","renovated_bathroom",
32 + "large_lot","quiet_area"]
33 +
34 +def spec_feats(uid):
35 + s = specs.get(uid)
36 + out = {"g_" + b: 0 for b in BOOL_SPECS}
37 + out["g_cond"] = 2
38 + out["g_hardwood"] = 0; out["g_quartz_granite"] = 0; out["g_view_water"] = 0
39 + out["has_gpt"] = 0
40 + if not s:
41 + return out
42 + out["has_gpt"] = 1
43 + for b in BOOL_SPECS:
44 + out["g_" + b] = int(bool(s.get(b)))
45 + out["g_cond"] = COND_SCORE.get(s.get("condition", "inconnu"), 2)
46 + fl = s.get("flooring") or []
47 + out["g_hardwood"] = int("bois_franc" in fl)
48 + out["g_quartz_granite"] = int(s.get("countertop") in ("quartz", "granit", "marbre"))
49 + out["g_view_water"] = int(s.get("view") == "eau")
50 + return out
51 +
52 +# ===================== pipeline v3 (repris) =====================
53 +SRC = "data/processed/maisons_avec_prix.json"
54 +data = json.load(open(SRC, encoding="utf-8"))
55 +
56 +def parse_area(s):
57 + if not s or not isinstance(s, str): return np.nan
58 + txt = s.lower(); is_m2 = ("m2" in txt) or ("m²" in txt)
59 + num = re.sub(r"[^0-9.,\s]", "", txt).strip().replace(" ", "").replace(",", "")
60 + try: v = float(num)
61 + except ValueError: return np.nan
62 + if v <= 0: return np.nan
63 + return v * 10.7639 if is_m2 else v
64 +
65 +def clean_city(city, source):
66 + if not city or not isinstance(city, str): return None
67 + c = city.split("(")[0].strip(); c = re.sub(r"\s+", " ", c).lower().strip()
68 + return c or None
69 +
70 +def to_int(x):
71 + try: return int(x)
72 + except (TypeError, ValueError): return np.nan
73 +
74 +AMENITIES = {
75 + "pool":["swimming pool","piscine","inground pool","pool"],
76 + "waterfront":["waterfront","bord de l'eau","navigable","water access","lakefront","riverfront","au bord de"],
77 + "garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],
78 + "central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],
79 + "finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],
80 + "near_water":["near water","waterfront","navigable"],
81 + "cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],
82 + "near_park":["park","parc"],
83 + "near_school":["school","école","cegep","university","université","elementary","high school"],
84 + "near_highway":["highway","autoroute"],
85 + "transit":["public transit","public transportation","transport"],
86 + "no_rear_neighbor":["no backyard neighbors","no rear neighbor","sans voisin arrière"],
87 +}
88 +
89 +def _side_ft(side):
90 + s = side.lower()
91 + if "ft" in s:
92 + ft = re.search(r"([\d.]+)\s*ft", s); inch = re.search(r"([\d.]+)\s*in", s)
93 + return (float(ft.group(1)) if ft else 0)+(float(inch.group(1))/12 if inch else 0)
94 + m = re.search(r"([\d.,]+)\s*m", s)
95 + if m:
96 + try: return float(m.group(1).replace(",", ".")) * 3.28084
97 + except ValueError: return None
98 + return None
99 +
100 +def _room_area(dim):
101 + if not dim or "x" not in dim: return None
102 + a, _, b = dim.partition("x"); la, lb = _side_ft(a), _side_ft(b)
103 + if la and lb and 1 < la < 200 and 1 < lb < 200: return la * lb
104 + return None
105 +
106 +ROOM_FLAGS = {"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],
107 + "r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],
108 + "r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
109 +
110 +def parse_rooms(rooms):
111 + out = {"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
112 + for k in ROOM_FLAGS: out[k] = 0
113 + if not rooms: return out
114 + area, levels = 0.0, set()
115 + for r in rooms:
116 + nm = (r.get("name") or "").lower(); lv = (r.get("level") or "").lower()
117 + out["n_rooms"] += 1
118 + if lv: levels.add(lv)
119 + if "basement" in lv or "sous-sol" in lv: out["has_basement"] = 1
120 + if "primary bedroom" in nm: out["has_primary_bed"] = 1
121 + for k, kws in ROOM_FLAGS.items():
122 + if any(w in nm for w in kws): out[k] = 1
123 + a = _room_area(r.get("dimensions"))
124 + if a: area += a
125 + out["room_area"] = area if area > 0 else np.nan
126 + out["n_levels_r"] = len(levels)
127 + return out
128 +
129 +rows = []
130 +for d in data:
131 + prop = d.get("property") or {}; addr = d.get("address") or {}
132 + feats = d.get("features") or []
133 + blob = (" ".join(feats)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
134 + yb = to_int(prop.get("yearBuilt"))
135 + age = (CUR_YEAR-yb) if (not pd.isna(yb) and 1700 <= yb <= CUR_YEAR) else np.nan
136 + ptype = prop.get("buildingType") or prop.get("type")
137 + storeys = prop.get("storeys") or prop.get("levels")
138 + try: storeys = float(str(storeys).replace(",", "."))
139 + except (TypeError, ValueError): storeys = np.nan
140 + uid = f"{d.get('source')}:{d.get('listingId')}"
141 + rec = {"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),
142 + "sqft":parse_area(prop.get("squareFeet")),"lot_sqft":parse_area(prop.get("lotSize")),
143 + "beds":to_int(prop.get("bedrooms")),"baths":to_int(prop.get("bathrooms")),
144 + "parking":to_int(prop.get("parkingSpaces")),"storeys":storeys,"age":age,
145 + "ptype":(ptype or "Unknown").strip(),"city":clean_city(addr.get("city"),d.get("source"))}
146 + for name, kws in AMENITIES.items():
147 + rec[name] = int(any(k in blob for k in kws))
148 + rec["n_images"] = len(d.get("images") or [])
149 + rec["virtual_tour"] = int(bool(d.get("virtualTourUrl")))
150 + rec.update(parse_rooms(d.get("rooms")))
151 + rec.update(spec_feats(uid))
152 + rows.append(rec)
153 +
154 +df = pd.DataFrame(rows)
155 +
156 +# nettoyage identique v3
157 +n0 = len(df)
158 +df = df[df["price"].notna()]
159 +df = df[(df["price"] >= 50_000) & (df["price"] <= 15_000_000)]
160 +df.loc[(df["sqft"] < 300) | (df["sqft"] > 20_000), "sqft"] = np.nan
161 +df.loc[(df["lot_sqft"] < 200) | (df["lot_sqft"] > 5_000_000), "lot_sqft"] = np.nan
162 +df.loc[df["beds"] > 15, "beds"] = np.nan
163 +df.loc[df["baths"] > 12, "baths"] = np.nan
164 +df["ppsf"] = df["price"] / df["sqft"]
165 +lo, hi = df["ppsf"].quantile([0.005, 0.995])
166 +df = df[~(df["ppsf"].notna() & ((df["ppsf"] < lo) | (df["ppsf"] > hi)))]
167 +
168 +# surface finale
169 +ov = df[df["sqft"].notna() & df["room_area"].notna() & (df["room_area"] > 200)]
170 +ratio = (ov["sqft"] / ov["room_area"]).median()
171 +room_est = df["room_area"] * ratio
172 +room_ok = df["room_area"].notna() & (room_est.between(300, 20_000))
173 +df["sqft_final"] = df["sqft"]
174 +fill = df["sqft_final"].isna() & room_ok
175 +df.loc[fill, "sqft_final"] = room_est[fill]
176 +df["has_sqft"] = df["sqft_final"].notna().astype(int)
177 +df["sqft_imp"] = df.groupby("ptype")["sqft_final"].transform(lambda s: s.fillna(s.median()))
178 +df["sqft_imp"] = df["sqft_imp"].fillna(df["sqft_final"].median())
179 +
180 +df["has_lot"] = df["lot_sqft"].notna().astype(int)
181 +df["lot_sqft"] = df["lot_sqft"].fillna(df["lot_sqft"].median())
182 +df["has_age"] = df["age"].notna().astype(int)
183 +df["age"] = df["age"].fillna(df["age"].median()).clip(lower=0)
184 +for c in ["beds","baths","storeys"]:
185 + df[c] = df[c].fillna(df[c].median())
186 +df["parking"] = df["parking"].fillna(0)
187 +df["ln_price"] = np.log(df["price"]); df["ln_sqft"] = np.log(df["sqft_imp"])
188 +df["ln_lot"] = np.log(df["lot_sqft"]); df["age2"] = df["age"]**2
189 +df["n_images_c"] = df["n_images"].clip(upper=60)
190 +
191 +vc = df["city"].value_counts(); keep = set(vc[vc >= 20].index)
192 +df["city_fe"] = df["city"].where(df["city"].isin(keep), "__other__").fillna("__other__")
193 +tc = df["ptype"].value_counts(); keept = set(tc[tc >= 30].index)
194 +df["ptype_fe"] = df["ptype"].where(df["ptype"].isin(keept), "Other")
195 +print(f"N nettoyé={len(df)} villes FE={df['city_fe'].nunique()} "
196 + f"avec specs GPT={int(df['has_gpt'].sum())}")
197 +
198 +# ---------- modèles ----------
199 +amen = list(AMENITIES.keys())
200 +room_vars = ["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office",
201 + "r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine",
202 + "r_library","r_laundry"]
203 +gpt_vars = ["g_cond","g_hardwood","g_quartz_granite","g_view_water","has_gpt"] + \
204 + ["g_" + b for b in BOOL_SPECS]
205 +extra = ["n_images_c","virtual_tour"] + room_vars + gpt_vars
206 +base = ("ln_lot + has_lot + beds + baths + parking + storeys + age + age2 + has_age "
207 + "+ C(source) + C(ptype_fe) + " + " + ".join(amen) + " + " + " + ".join(extra)
208 + + " + C(city_fe)")
209 +fA = "ln_price ~ ln_sqft + " + base
210 +fB = "ln_price ~ " + base
211 +
212 +def pct(b): return (np.exp(b)-1)*100
213 +dfA = df[df["has_sqft"] == 1].copy()
214 +mA = smf.ols(fA, data=dfA).fit(cov_type="HC3")
215 +mB = smf.ols(fB, data=df).fit(cov_type="HC3")
216 +sdA, sdB = np.std(mA.resid), np.std(mB.resid)
217 +
218 +def report(m, sd, name):
219 + print("\n" + "=" * 70)
220 + print(f"{name} N={int(m.nobs)} R²={m.rsquared:.4f} R²adj={m.rsquared_adj:.4f} "
221 + f"±{(np.exp(sd)-1)*100:.1f}%")
222 + print("=" * 70)
223 + pr, pv = m.params, m.pvalues
224 + print("-- specs GPT (impact %) --")
225 + for p in pr.index:
226 + if p.startswith("g_") or p == "has_gpt":
227 + st = "***" if pv[p]<.01 else "**" if pv[p]<.05 else "*" if pv[p]<.1 else ""
228 + print(f" {p:<22}{pr[p]:>9.4f}{pct(pr[p]):>+8.1f}% p={pv[p]:.2g} {st}")
229 +
230 +report(mA, sdA, "MODÈLE A+GPT (avec superficie)")
231 +report(mB, sdB, "MODÈLE B+GPT (couverture large)")
232 +
233 +predA = np.exp(mA.predict(df)) * np.exp(sdA**2/2)
234 +predB = np.exp(mB.predict(df)) * np.exp(sdB**2/2)
235 +df["pred_price"] = np.where(df["has_sqft"] == 1, predA, predB)
236 +df["model_used"] = np.where(df["has_sqft"] == 1, "A", "B")
237 +df["resid_pct"] = (df["price"]/df["pred_price"] - 1) * 100
238 +print(f"\nErreur abs médiane: {df.resid_pct.abs().median():.1f}% "
239 + f"| ±15%: {(df.resid_pct.abs()<=15).mean()*100:.0f}% "
240 + f"±25%: {(df.resid_pct.abs()<=25).mean()*100:.0f}%")
241 +
242 +out = (["uid","source","city","ptype","price","pred_price","resid_pct","model_used",
243 + "sqft_final","has_sqft","lot_sqft","beds","baths","parking","age","n_images",
244 + "virtual_tour"] + amen + room_vars + gpt_vars)
245 +df[out].to_csv("data/processed/maisons_hedonic_v4.csv", index=False)
246 +print(f"-> maisons_hedonic_v4.csv ({len(df)} lignes)")
247 +with open("results/hedonic_v4_summary.txt", "w", encoding="utf-8") as fh:
248 + fh.write("=== A+GPT ===\n"+str(mA.summary())+"\n\n=== B+GPT ===\n"+str(mB.summary()))
249 +print("-> hedonic_v4_summary.txt")
added scripts/06_hedonic_ml_cv.py +171 −0
@@ -0,0 +1,171 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Comparaison OLS vs modèles ML (RF, Gradient Boosting) en validation croisée 5-fold.
4 +Même jeu de variables que la v4 (structurel + aménités + pièces + listing + GPT + FE)."""
5 +import json, re, warnings
6 +import numpy as np, pandas as pd
7 +from sklearn.linear_model import LinearRegression
8 +from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
9 +from sklearn.model_selection import KFold, cross_val_predict
10 +from sklearn.metrics import r2_score
11 +warnings.filterwarnings("ignore")
12 +CUR_YEAR = 2026
13 +
14 +# ---- specs GPT ----
15 +specs = {}
16 +for l in open("data/processed/specs.jsonl", encoding="utf-8"):
17 + r = json.loads(l)
18 + if r.get("specs"): specs[r["uid"]] = r["specs"]
19 +COND = {"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2}
20 +BOOL = ["turnkey","recently_renovated","garage","pool","waterfront","finished_basement",
21 + "fireplace","central_ac","new_construction","income_property","renovated_kitchen",
22 + "renovated_bathroom","large_lot","quiet_area"]
23 +def gfeat(uid):
24 + s = specs.get(uid); o = {"g_"+b:0 for b in BOOL}
25 + o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0})
26 + if not s: return o
27 + for b in BOOL: o["g_"+b] = int(bool(s.get(b)))
28 + o["g_cond"] = COND.get(s.get("condition","inconnu"),2)
29 + fl = s.get("flooring") or []; o["g_hardwood"] = int("bois_franc" in fl)
30 + o["g_quartz_granite"] = int(s.get("countertop") in ("quartz","granit","marbre"))
31 + o["g_view_water"] = int(s.get("view")=="eau"); return o
32 +
33 +# ---- parsing (repris) ----
34 +def parse_area(s):
35 + if not s or not isinstance(s,str): return np.nan
36 + t=s.lower(); m=("m2" in t) or ("m²" in t)
37 + n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
38 + try: v=float(n)
39 + except: return np.nan
40 + return (v*10.7639 if m else v) if v>0 else np.nan
41 +def cc(c):
42 + if not c or not isinstance(c,str): return None
43 + c=c.split("(")[0].strip(); return re.sub(r"\s+"," ",c).lower().strip() or None
44 +def ti(x):
45 + try: return int(x)
46 + except: return np.nan
47 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],
48 +"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],
49 +"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],
50 +"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],
51 +"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],
52 +"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],
53 +"near_school":["school","école","cegep","university","université","elementary","high school"],
54 +"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],
55 +"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
56 +def sd(s):
57 + s=s.lower()
58 + if "ft" in s:
59 + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s)
60 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
61 + m=re.search(r"([\d.,]+)\s*m",s)
62 + if m:
63 + try: return float(m.group(1).replace(",","."))*3.28084
64 + except: return None
65 + return None
66 +def ra(d):
67 + if not d or "x" not in d: return None
68 + a,_,b=d.partition("x");la,lb=sd(a),sd(b)
69 + return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
70 +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],
71 +"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
72 +def pr(rooms):
73 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
74 + for k in RF: o[k]=0
75 + if not rooms: return o
76 + ar=0.0;lv=set()
77 + for r in rooms:
78 + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1
79 + if l: lv.add(l)
80 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
81 + if "primary bedroom" in nm: o["has_primary_bed"]=1
82 + for k,kw in RF.items():
83 + if any(w in nm for w in kw): o[k]=1
84 + a=ra(r.get("dimensions"))
85 + if a: ar+=a
86 + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o
87 +
88 +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"))
89 +rows=[]
90 +for d in data:
91 + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or []
92 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
93 + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
94 + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels")
95 + try: st=float(str(st).replace(",","."))
96 + except: st=np.nan
97 + uid=f"{d.get('source')}:{d.get('listingId')}"
98 + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),
99 + "lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),
100 + "parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))}
101 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
102 + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
103 + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec)
104 +df=pd.DataFrame(rows)
105 +
106 +# nettoyage (identique)
107 +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
108 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
109 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
110 +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan
111 +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995])
112 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
113 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)]
114 +ratio=(ov["sqft"]/ov["room_area"]).median();re_=df["room_area"]*ratio
115 +ok=df["room_area"].notna()&(re_.between(300,20000))
116 +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill]
117 +df["has_sqft"]=df["sqft_final"].notna().astype(int)
118 +# pour ML : garder NaN (les arbres gèrent) ; pour OLS on impute une copie
119 +df["ln_price"]=np.log(df["price"]);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60)
120 +vc=df["city"].value_counts();keep=set(vc[vc>=20].index)
121 +df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
122 +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
123 +print(f"N={len(df)}")
124 +
125 +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
126 +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL]
127 +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv
128 +cat=["city_fe","ptype_fe","source"]
129 +
130 +y=df["ln_price"].values
131 +price=df["price"].values
132 +Xcat=pd.get_dummies(df[cat],prefix=cat,dtype=float)
133 +# --- matrice ML (NaN autorisés pour arbres) ---
134 +Xml=pd.concat([df[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1)
135 +# --- matrice OLS (imputée médiane par type) ---
136 +dfi=df.copy()
137 +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]:
138 + dfi[c]=dfi.groupby("ptype")[c].transform(lambda s:s.fillna(s.median()))
139 + dfi[c]=dfi[c].fillna(dfi[c].median())
140 +dfi["parking"]=dfi["parking"].fillna(0)
141 +Xols=pd.concat([dfi[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1)
142 +
143 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
144 +def evalcv(model,X):
145 + pred=cross_val_predict(model,X,y,cv=kf,n_jobs=-1)
146 + r2=r2_score(y,pred)
147 + mape=np.median(np.abs(np.exp(pred-y)-1))*100
148 + w15=np.mean(np.abs(np.exp(pred-y)-1)<=.15)*100
149 + w25=np.mean(np.abs(np.exp(pred-y)-1)<=.25)*100
150 + return r2,mape,w15,w25
151 +
152 +print("\nValidation croisée 5-fold (out-of-sample) — cible = ln(prix)\n")
153 +print(f"{'Modèle':<34}{'R² OOS':>9}{'MAE méd.':>10}{'±15%':>7}{'±25%':>7}")
154 +res={}
155 +res["OLS (linéaire, imputé)"]=evalcv(LinearRegression(),Xols)
156 +res["Random Forest (400)"]=evalcv(RandomForestRegressor(n_estimators=400,max_features=0.5,min_samples_leaf=3,n_jobs=-1,random_state=42),Xml)
157 +res["HistGradientBoosting"]=evalcv(HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42),Xml)
158 +res["HistGB (tuned depth)"]=evalcv(HistGradientBoostingRegressor(max_iter=900,learning_rate=0.03,max_leaf_nodes=127,min_samples_leaf=20,l2_regularization=2.0,random_state=42),Xml)
159 +for k,(r2,mape,w15,w25) in res.items():
160 + print(f"{k:<34}{r2:>9.4f}{mape:>9.1f}%{w15:>6.0f}%{w25:>6.0f}%")
161 +
162 +# importance GBM (permutation rapide via feature_importances_ du HistGB n'existe pas → refit + importances via gain n/a)
163 +gb=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42).fit(Xml,y)
164 +from sklearn.inspection import permutation_importance
165 +import numpy as _np
166 +idx=_np.random.RandomState(0).choice(len(Xml),4000,replace=False)
167 +pi=permutation_importance(gb,Xml.iloc[idx],y[idx],n_repeats=3,random_state=0,n_jobs=-1)
168 +imp=pd.Series(pi.importances_mean,index=Xml.columns).sort_values(ascending=False)
169 +print("\nTop 20 variables (importance par permutation, GBM) :")
170 +for name,v in imp.head(20).items(): print(f" {name:<26}{v:.4f}")
171 +print("\nImportance cumulée des variables GPT (g_*) :","%.4f"%imp[[c for c in Xml.columns if c.startswith('g_')]].sum())
added scripts/07_torch_nn.py +170 −0
@@ -0,0 +1,170 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Réseau de neurones tabulaire (embeddings ville/type + MLP) — 5-fold CV.
4 +Comparable aux modèles de hedonic_ml.py (même variables, même KFold seed=42)."""
5 +import json, re, warnings
6 +import numpy as np, pandas as pd
7 +import torch, torch.nn as nn
8 +from sklearn.model_selection import KFold
9 +from sklearn.metrics import r2_score
10 +warnings.filterwarnings("ignore")
11 +torch.manual_seed(42); np.random.seed(42)
12 +CUR_YEAR=2026
13 +dev="mps" if torch.backends.mps.is_available() else "cpu"
14 +print("device:",dev)
15 +
16 +# ---- specs GPT ----
17 +specs={}
18 +for l in open("data/processed/specs.jsonl",encoding="utf-8"):
19 + r=json.loads(l)
20 + if r.get("specs"): specs[r["uid"]]=r["specs"]
21 +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2}
22 +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"]
23 +def gfeat(uid):
24 + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0})
25 + if not s: return o
26 + for b in BOOL: o["g_"+b]=int(bool(s.get(b)))
27 + o["g_cond"]=COND.get(s.get("condition","inconnu"),2)
28 + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl)
29 + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o
30 +def parse_area(s):
31 + if not s or not isinstance(s,str): return np.nan
32 + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
33 + try: v=float(n)
34 + except: return np.nan
35 + return (v*10.7639 if m else v) if v>0 else np.nan
36 +def cc(c):
37 + if not c or not isinstance(c,str): return None
38 + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None
39 +def ti(x):
40 + try: return int(x)
41 + except: return np.nan
42 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
43 +def sd(s):
44 + s=s.lower()
45 + if "ft" in s:
46 + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s)
47 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
48 + m=re.search(r"([\d.,]+)\s*m",s)
49 + if m:
50 + try: return float(m.group(1).replace(",","."))*3.28084
51 + except: return None
52 + return None
53 +def ra(d):
54 + if not d or "x" not in d: return None
55 + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
56 +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
57 +def pr(rooms):
58 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
59 + for k in RF: o[k]=0
60 + if not rooms: return o
61 + ar=0.0;lv=set()
62 + for r in rooms:
63 + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1
64 + if l: lv.add(l)
65 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
66 + if "primary bedroom" in nm: o["has_primary_bed"]=1
67 + for k,kw in RF.items():
68 + if any(w in nm for w in kw): o[k]=1
69 + a=ra(r.get("dimensions"))
70 + if a: ar+=a
71 + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o
72 +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[]
73 +for d in data:
74 + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or []
75 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
76 + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
77 + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels")
78 + try: st=float(str(st).replace(",","."))
79 + except: st=np.nan
80 + uid=f"{d.get('source')}:{d.get('listingId')}"
81 + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))}
82 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
83 + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
84 + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec)
85 +df=pd.DataFrame(rows)
86 +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
87 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
88 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
89 +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan
90 +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995])
91 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
92 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median()
93 +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000))
94 +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill]
95 +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60)
96 +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
97 +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
98 +df=df.reset_index(drop=True);print("N=",len(df))
99 +
100 +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
101 +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL]
102 +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv
103 +y=df["ln_price"]=np.log(df["price"]);y=y.values.astype(np.float32)
104 +# encodage catégoriel -> indices
105 +cats=["city_fe","ptype_fe","source"];maps={};card={}
106 +for c in cats:
107 + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};maps[c]=m;card[c]=len(u)
108 + df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64)
109 +Xnum=df[num].astype(np.float32).values
110 +Xcat=df[[c+"_ix" for c in cats]].values.astype(np.int64)
111 +
112 +class Net(nn.Module):
113 + def __init__(self,ncard,nnum):
114 + super().__init__()
115 + self.embs=nn.ModuleList()
116 + tot=0
117 + for c in ncard:
118 + d=min(50,(c+1)//2); d=max(d,2); self.embs.append(nn.Embedding(c,d)); tot+=d
119 + self.bn=nn.BatchNorm1d(nnum)
120 + self.net=nn.Sequential(
121 + nn.Linear(tot+nnum,256),nn.ReLU(),nn.BatchNorm1d(256),nn.Dropout(0.3),
122 + nn.Linear(256,128),nn.ReLU(),nn.BatchNorm1d(128),nn.Dropout(0.2),
123 + nn.Linear(128,64),nn.ReLU(),nn.Dropout(0.1),
124 + nn.Linear(64,1))
125 + def forward(self,xc,xn):
126 + e=[emb(xc[:,i]) for i,emb in enumerate(self.embs)]
127 + x=torch.cat(e+[self.bn(xn)],1);return self.net(x).squeeze(1)
128 +
129 +def train_predict(tr,te):
130 + # imputation médiane (train) + standardisation
131 + med=np.nanmedian(Xnum[tr],axis=0);med=np.where(np.isnan(med),0,med)
132 + Xn=Xnum.copy()
133 + for j in range(Xn.shape[1]):
134 + col=Xn[:,j];mask=np.isnan(col);col[mask]=med[j];Xn[:,j]=col
135 + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg
136 + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys
137 + # split interne train/val pour early stopping
138 + ntr=len(tr);perm=np.random.permutation(ntr);vcut=int(ntr*0.9)
139 + tri=tr[perm[:vcut]];vai=tr[perm[vcut:]]
140 + def T(a,dt): return torch.tensor(a,device=dev,dtype=dt)
141 + model=Net([card[c] for c in cats],Xn.shape[1]).to(dev)
142 + opt=torch.optim.Adam(model.parameters(),lr=1e-3,weight_decay=1e-5)
143 + lossf=nn.MSELoss()
144 + Xc_t=T(Xcat,torch.long);Xn_t=T(Xn,torch.float32);y_t=T(yz,torch.float32)
145 + best=1e9;bstate=None;patience=0
146 + for ep in range(150):
147 + model.train();idx=np.random.permutation(tri)
148 + for i in range(0,len(idx),512):
149 + b=idx[i:i+512]
150 + opt.zero_grad();p=model(Xc_t[b],Xn_t[b]);l=lossf(p,y_t[b]);l.backward();opt.step()
151 + model.eval()
152 + with torch.no_grad():
153 + vp=model(Xc_t[vai],Xn_t[vai]);vl=lossf(vp,y_t[vai]).item()
154 + if vl<best-1e-4: best=vl;bstate={k:v.clone() for k,v in model.state_dict().items()};patience=0
155 + else:
156 + patience+=1
157 + if patience>=12: break
158 + model.load_state_dict(bstate);model.eval()
159 + with torch.no_grad():
160 + pz=model(Xc_t[te],Xn_t[te]).cpu().numpy()
161 + return pz*ys+ym
162 +
163 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
164 +pred=np.zeros_like(y)
165 +for f,(tr,te) in enumerate(kf.split(Xnum)):
166 + pred[te]=train_predict(tr,te);print(f" fold {f+1} fait")
167 +r2=r2_score(y,pred);err=np.abs(np.exp(pred-y)-1)
168 +print("\n=== Réseau de neurones (embeddings + MLP) — 5-fold CV OOS ===")
169 +print(f"R² OOS = {r2:.4f} | erreur médiane = {np.median(err)*100:.1f}% | ±15% = {np.mean(err<=.15)*100:.0f}% ±25% = {np.mean(err<=.25)*100:.0f}%")
170 +print("\nRappel (mêmes folds) : OLS 0.628 | RF 0.678 | HistGBM 0.743")
added scripts/08_torch_nn_robust.py +192 −0
@@ -0,0 +1,192 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Réseau de neurones tabulaire ROBUSTE : embeddings + ResNet-MLP,
4 +ensemble multi-seed, Huber loss, AdamW + cosine, grad-clip, early stopping.
5 +5-fold CV out-of-sample, comparable aux autres modèles (seed=42)."""
6 +import json, re, warnings
7 +import numpy as np, pandas as pd
8 +import torch, torch.nn as nn
9 +from sklearn.model_selection import KFold
10 +from sklearn.metrics import r2_score
11 +warnings.filterwarnings("ignore")
12 +CUR_YEAR=2026
13 +dev="mps" if torch.backends.mps.is_available() else "cpu"
14 +print("device:",dev)
15 +SEEDS=[42,1,7] # ensemble par fold
16 +D_MAIN=256; N_BLOCKS=3; D_HIDDEN=512
17 +
18 +# ======== reconstruction du dataset (identique v4) ========
19 +specs={}
20 +for l in open("data/processed/specs.jsonl",encoding="utf-8"):
21 + r=json.loads(l)
22 + if r.get("specs"): specs[r["uid"]]=r["specs"]
23 +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2}
24 +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"]
25 +def gfeat(uid):
26 + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0})
27 + if not s: return o
28 + for b in BOOL: o["g_"+b]=int(bool(s.get(b)))
29 + o["g_cond"]=COND.get(s.get("condition","inconnu"),2)
30 + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl)
31 + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o
32 +def parse_area(s):
33 + if not s or not isinstance(s,str): return np.nan
34 + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
35 + try: v=float(n)
36 + except: return np.nan
37 + return (v*10.7639 if m else v) if v>0 else np.nan
38 +def cc(c):
39 + if not c or not isinstance(c,str): return None
40 + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None
41 +def ti(x):
42 + try: return int(x)
43 + except: return np.nan
44 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
45 +def sd(s):
46 + s=s.lower()
47 + if "ft" in s:
48 + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s)
49 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
50 + m=re.search(r"([\d.,]+)\s*m",s)
51 + if m:
52 + try: return float(m.group(1).replace(",","."))*3.28084
53 + except: return None
54 + return None
55 +def ra(d):
56 + if not d or "x" not in d: return None
57 + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
58 +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
59 +def pr(rooms):
60 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
61 + for k in RF: o[k]=0
62 + if not rooms: return o
63 + ar=0.0;lv=set()
64 + for r in rooms:
65 + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1
66 + if l: lv.add(l)
67 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
68 + if "primary bedroom" in nm: o["has_primary_bed"]=1
69 + for k,kw in RF.items():
70 + if any(w in nm for w in kw): o[k]=1
71 + a=ra(r.get("dimensions"))
72 + if a: ar+=a
73 + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o
74 +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[]
75 +for d in data:
76 + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or []
77 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
78 + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
79 + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels")
80 + try: st=float(str(st).replace(",","."))
81 + except: st=np.nan
82 + uid=f"{d.get('source')}:{d.get('listingId')}"
83 + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))}
84 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
85 + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
86 + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec)
87 +df=pd.DataFrame(rows)
88 +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
89 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
90 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
91 +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan
92 +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995])
93 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
94 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median()
95 +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000))
96 +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill]
97 +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60)
98 +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
99 +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
100 +df=df.reset_index(drop=True);print("N=",len(df))
101 +
102 +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
103 +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL]
104 +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv
105 +# indicateurs de manquant (robustesse) pour les colonnes à NaN
106 +missnum=["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]
107 +for c in missnum: df["miss_"+c]=df[c].isna().astype(np.float32)
108 +num=num+["miss_"+c for c in missnum]
109 +y=np.log(df["price"].values).astype(np.float32)
110 +cats=["city_fe","ptype_fe","source"];card={}
111 +for c in cats:
112 + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};card[c]=len(u)
113 + df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64)
114 +Xnum=df[num].astype(np.float32).values
115 +Xcat=df[[c+"_ix" for c in cats]].values.astype(np.int64)
116 +
117 +class ResBlock(nn.Module):
118 + def __init__(self,d,dh,p):
119 + super().__init__()
120 + self.bn=nn.BatchNorm1d(d);self.l1=nn.Linear(d,dh);self.act=nn.GELU()
121 + self.dp1=nn.Dropout(p);self.l2=nn.Linear(dh,d);self.dp2=nn.Dropout(p)
122 + def forward(self,x):
123 + z=self.bn(x);z=self.act(self.l1(z));z=self.dp1(z);z=self.l2(z);z=self.dp2(z);return x+z
124 +
125 +class ResNetTab(nn.Module):
126 + def __init__(self,ncard,nnum):
127 + super().__init__()
128 + self.embs=nn.ModuleList();tot=0
129 + for c in ncard:
130 + d=max(2,min(64,round(1.6*c**0.56)));self.embs.append(nn.Embedding(c,d));tot+=d
131 + self.embdp=nn.Dropout(0.1)
132 + self.bn0=nn.BatchNorm1d(nnum)
133 + self.proj=nn.Linear(tot+nnum,D_MAIN)
134 + self.blocks=nn.ModuleList([ResBlock(D_MAIN,D_HIDDEN,0.2) for _ in range(N_BLOCKS)])
135 + self.head=nn.Sequential(nn.BatchNorm1d(D_MAIN),nn.GELU(),nn.Linear(D_MAIN,1))
136 + def forward(self,xc,xn):
137 + e=[emb(xc[:,i]) for i,emb in enumerate(self.embs)]
138 + x=torch.cat([self.embdp(torch.cat(e,1)),self.bn0(xn)],1)
139 + x=self.proj(x)
140 + for b in self.blocks: x=b(x)
141 + return self.head(x).squeeze(1)
142 +
143 +def train_one(tr,te,seed):
144 + torch.manual_seed(seed);np.random.seed(seed)
145 + med=np.nanmedian(Xnum[tr],axis=0);med=np.where(np.isnan(med),0,med)
146 + Xn=Xnum.copy()
147 + for j in range(Xn.shape[1]):
148 + col=Xn[:,j];col[np.isnan(col)]=med[j];Xn[:,j]=col
149 + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg
150 + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys
151 + ntr=len(tr);perm=np.random.permutation(ntr);vc_=int(ntr*0.9)
152 + tri=tr[perm[:vc_]];vai=tr[perm[vc_:]]
153 + def T(a,dt): return torch.tensor(a,device=dev,dtype=dt)
154 + model=ResNetTab([card[c] for c in cats],Xn.shape[1]).to(dev)
155 + opt=torch.optim.AdamW(model.parameters(),lr=2e-3,weight_decay=1e-4)
156 + EPOCHS=200
157 + sched=torch.optim.lr_scheduler.CosineAnnealingLR(opt,T_max=EPOCHS)
158 + lossf=nn.SmoothL1Loss(beta=1.0) # Huber : robuste aux queues
159 + Xc_t=T(Xcat,torch.long);Xn_t=T(Xn,torch.float32);y_t=T(yz,torch.float32)
160 + best=1e9;bstate=None;pat=0
161 + for ep in range(EPOCHS):
162 + model.train();idx=np.random.permutation(tri)
163 + for i in range(0,len(idx),512):
164 + b=idx[i:i+512]
165 + opt.zero_grad();p=model(Xc_t[b],Xn_t[b]);l=lossf(p,y_t[b])
166 + l.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),5.0);opt.step()
167 + sched.step()
168 + model.eval()
169 + with torch.no_grad():
170 + vp=model(Xc_t[vai],Xn_t[vai]);vl=((vp-y_t[vai])**2).mean().item()
171 + if vl<best-1e-4: best=vl;bstate={k:v.clone() for k,v in model.state_dict().items()};pat=0
172 + else:
173 + pat+=1
174 + if pat>=20: break
175 + model.load_state_dict(bstate);model.eval()
176 + with torch.no_grad():
177 + pz=model(Xc_t[te],Xn_t[te]).cpu().numpy()
178 + return pz*ys+ym
179 +
180 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
181 +pred=np.zeros_like(y)
182 +for f,(tr,te) in enumerate(kf.split(Xnum)):
183 + ps=np.zeros(len(te),dtype=np.float64)
184 + for sd_ in SEEDS:
185 + ps+=train_one(tr,te,sd_)
186 + pred[te]=ps/len(SEEDS) # moyenne d'ensemble
187 + r2f=r2_score(y[te],pred[te]);print(f" fold {f+1}: R²={r2f:.4f} (ensemble de {len(SEEDS)})")
188 +r2=r2_score(y,pred);err=np.abs(np.exp(pred-y)-1)
189 +print("\n=== NN ROBUSTE (ResNet-tab, ensemble multi-seed) — 5-fold CV OOS ===")
190 +print(f"R² OOS = {r2:.4f} | erreur médiane = {np.median(err)*100:.1f}% | ±15% = {np.mean(err<=.15)*100:.0f}% | ±25% = {np.mean(err<=.25)*100:.0f}%")
191 +print("\nRappel (mêmes folds): OLS 0.628 | RF 0.678 | NN simple 0.711 | HistGBM 0.743")
192 +np.save("data/processed/nn_robust_oos.npy",pred)
added scripts/09_v5_ml_text_vision.py +213 −0
@@ -0,0 +1,213 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""v5 : ablation texte (v4) vs texte+vision (v5). GBM + NN robuste + blend, 5-fold CV."""
4 +import json, re, warnings
5 +import numpy as np, pandas as pd
6 +import torch, torch.nn as nn
7 +from sklearn.ensemble import HistGradientBoostingRegressor
8 +from sklearn.model_selection import KFold, cross_val_predict
9 +from sklearn.metrics import r2_score
10 +warnings.filterwarnings("ignore")
11 +torch.manual_seed(42); np.random.seed(42)
12 +CUR_YEAR=2026
13 +dev="mps" if torch.backends.mps.is_available() else "cpu"; print("device:",dev)
14 +
15 +# ---- GPT specs ----
16 +specs={}
17 +for l in open("data/processed/specs.jsonl",encoding="utf-8"):
18 + r=json.loads(l)
19 + if r.get("specs"): specs[r["uid"]]=r["specs"]
20 +# ---- vision ----
21 +facade={}
22 +for l in open("data/processed/facade_specs.jsonl",encoding="utf-8"):
23 + r=json.loads(l)
24 + if r.get("facade"): facade[r["uid"]]=r["facade"]
25 +print("specs GPT:",len(specs),"| façades:",len(facade))
26 +
27 +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2}
28 +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"]
29 +def gfeat(uid):
30 + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0})
31 + if not s: return o
32 + for b in BOOL: o["g_"+b]=int(bool(s.get(b)))
33 + o["g_cond"]=COND.get(s.get("condition","inconnu"),2)
34 + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl)
35 + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o
36 +RENO={"a_renover":0,"date":1,"standard":2,"renove":3,"haut_de_gamme":4,"inconnu":2}
37 +LAND={"aucun":0,"minimal":1,"entretenu":2,"amenage_soigne":3}
38 +SETS=["urbain_dense","banlieue","rural","bord_eau","boise"]
39 +MATS=["brique","pierre","vinyle","bois","aluminium","stucco","crepi","beton"]
40 +def vfeat(uid):
41 + f=facade.get(uid)
42 + o={"v_cond":5,"v_curb":5,"v_photoq":5,"v_reno":2,"v_land":1,"v_luxury":0,"v_garage_vis":0,"v_is_ext":0}
43 + for s in SETS: o["v_set_"+s]=0
44 + for m in MATS: o["v_mat_"+m]=0
45 + if not f: return o
46 + o["v_cond"]=f.get("condition_score") or 5;o["v_curb"]=f.get("curb_appeal") or 5;o["v_photoq"]=f.get("photo_quality") or 5
47 + o["v_reno"]=RENO.get(f.get("renovation_level","inconnu"),2);o["v_land"]=LAND.get(f.get("landscaping","minimal"),1)
48 + o["v_luxury"]=int(bool(f.get("curb_luxury")));o["v_garage_vis"]=int(bool(f.get("has_garage_visible")));o["v_is_ext"]=int(bool(f.get("is_exterior")))
49 + st=f.get("setting")
50 + if st in SETS: o["v_set_"+st]=1
51 + for m in (f.get("exterior_materials") or []):
52 + if m in MATS: o["v_mat_"+m]=1
53 + return o
54 +# ---- parsing structurel ----
55 +def pa(s):
56 + if not s or not isinstance(s,str): return np.nan
57 + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
58 + try: v=float(n)
59 + except: return np.nan
60 + return (v*10.7639 if m else v) if v>0 else np.nan
61 +def cc(c):
62 + if not c or not isinstance(c,str): return None
63 + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None
64 +def ti(x):
65 + try: return int(x)
66 + except: return np.nan
67 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
68 +def sd(s):
69 + s=s.lower()
70 + if "ft" in s:
71 + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s)
72 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
73 + m=re.search(r"([\d.,]+)\s*m",s)
74 + if m:
75 + try: return float(m.group(1).replace(",","."))*3.28084
76 + except: return None
77 + return None
78 +def ra(d):
79 + if not d or "x" not in d: return None
80 + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
81 +RFR={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
82 +def pr(rooms):
83 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
84 + for k in RFR: o[k]=0
85 + if not rooms: return o
86 + ar=0.0;lv=set()
87 + for r in rooms:
88 + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1
89 + if l: lv.add(l)
90 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
91 + if "primary bedroom" in nm: o["has_primary_bed"]=1
92 + for k,kw in RFR.items():
93 + if any(w in nm for w in kw): o[k]=1
94 + a=ra(r.get("dimensions"))
95 + if a: ar+=a
96 + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o
97 +
98 +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[]
99 +for d in data:
100 + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or []
101 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
102 + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
103 + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels")
104 + try: st=float(str(st).replace(",","."))
105 + except: st=np.nan
106 + uid=f"{d.get('source')}:{d.get('listingId')}"
107 + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":pa(p.get("squareFeet")),"lot_sqft":pa(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))}
108 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
109 + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
110 + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rec.update(vfeat(uid));rows.append(rec)
111 +df=pd.DataFrame(rows)
112 +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
113 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
114 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
115 +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan
116 +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995])
117 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
118 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median()
119 +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000))
120 +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill]
121 +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60)
122 +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
123 +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
124 +df=df.reset_index(drop=True);print("N=",len(df))
125 +y=np.log(df["price"].values).astype(np.float32)
126 +
127 +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
128 +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL]
129 +vv=["v_cond","v_curb","v_photoq","v_reno","v_land","v_luxury","v_garage_vis","v_is_ext"]+["v_set_"+s for s in SETS]+["v_mat_"+m for m in MATS]
130 +base=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv
131 +NUM_v4=base+gv
132 +NUM_v5=base+gv+vv
133 +missnum=["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]
134 +for c in missnum: df["miss_"+c]=df[c].isna().astype(np.float32)
135 +MISS=["miss_"+c for c in missnum]
136 +
137 +cats=["city_fe","ptype_fe","source"];card={}
138 +for c in cats:
139 + u=sorted(df[c].astype(str).unique());m={v:i for i,v in enumerate(u)};card[c]=len(u);df[c+"_ix"]=df[c].astype(str).map(m).astype(np.int64)
140 +Xcat_ix=df[[c+"_ix" for c in cats]].values.astype(np.int64)
141 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
142 +
143 +def gbm_cv(numcols):
144 + X=pd.concat([df[numcols].reset_index(drop=True),pd.get_dummies(df[cats],dtype=float).reset_index(drop=True)],axis=1)
145 + m=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42)
146 + return cross_val_predict(m,X,y,cv=kf,n_jobs=-1)
147 +
148 +# ---- NN robuste ----
149 +D_MAIN=256;D_HID=512;NB=3;SEEDS=[42,1,7]
150 +class RB(nn.Module):
151 + def __init__(s,d,dh,p):
152 + super().__init__();s.bn=nn.BatchNorm1d(d);s.l1=nn.Linear(d,dh);s.act=nn.GELU();s.d1=nn.Dropout(p);s.l2=nn.Linear(dh,d);s.d2=nn.Dropout(p)
153 + def forward(s,x): z=s.d1(s.act(s.l1(s.bn(x))));return x+s.d2(s.l2(z))
154 +class Net(nn.Module):
155 + def __init__(s,nc,nn_):
156 + super().__init__();s.embs=nn.ModuleList();tot=0
157 + for c in nc: d=max(2,min(64,round(1.6*c**0.56)));s.embs.append(nn.Embedding(c,d));tot+=d
158 + s.ed=nn.Dropout(0.1);s.bn0=nn.BatchNorm1d(nn_);s.proj=nn.Linear(tot+nn_,D_MAIN)
159 + s.bl=nn.ModuleList([RB(D_MAIN,D_HID,0.2) for _ in range(NB)])
160 + s.head=nn.Sequential(nn.BatchNorm1d(D_MAIN),nn.GELU(),nn.Linear(D_MAIN,1))
161 + def forward(s,xc,xn):
162 + e=[emb(xc[:,i]) for i,emb in enumerate(s.embs)]
163 + x=s.proj(torch.cat([s.ed(torch.cat(e,1)),s.bn0(xn)],1))
164 + for b in s.bl: x=b(x)
165 + return s.head(x).squeeze(1)
166 +def nn_fold(Xnum,tr,te,seed):
167 + torch.manual_seed(seed);np.random.seed(seed)
168 + med=np.nanmedian(Xnum[tr],0);med=np.where(np.isnan(med),0,med);Xn=Xnum.copy()
169 + for j in range(Xn.shape[1]): col=Xn[:,j];col[np.isnan(col)]=med[j];Xn[:,j]=col
170 + mu=Xn[tr].mean(0);sg=Xn[tr].std(0)+1e-6;Xn=(Xn-mu)/sg
171 + ym=y[tr].mean();ys=y[tr].std()+1e-6;yz=(y-ym)/ys
172 + perm=np.random.permutation(len(tr));cut=int(len(tr)*0.9);tri=tr[perm[:cut]];vai=tr[perm[cut:]]
173 + T=lambda a,dt: torch.tensor(a,device=dev,dtype=dt)
174 + model=Net([card[c] for c in cats],Xn.shape[1]).to(dev)
175 + opt=torch.optim.AdamW(model.parameters(),lr=2e-3,weight_decay=1e-4)
176 + sch=torch.optim.lr_scheduler.CosineAnnealingLR(opt,T_max=200);lf=nn.SmoothL1Loss(beta=1.0)
177 + Xc=T(Xcat_ix,torch.long);Xnt=T(Xn,torch.float32);yt=T(yz,torch.float32)
178 + best=1e9;bs=None;pat=0
179 + for ep in range(200):
180 + model.train();idx=np.random.permutation(tri)
181 + for i in range(0,len(idx),512):
182 + b=idx[i:i+512];opt.zero_grad();l=lf(model(Xc[b],Xnt[b]),yt[b]);l.backward()
183 + torch.nn.utils.clip_grad_norm_(model.parameters(),5.0);opt.step()
184 + sch.step();model.eval()
185 + with torch.no_grad(): vl=((model(Xc[vai],Xnt[vai])-yt[vai])**2).mean().item()
186 + if vl<best-1e-4: best=vl;bs={k:v.clone() for k,v in model.state_dict().items()};pat=0
187 + else:
188 + pat+=1
189 + if pat>=20: break
190 + model.load_state_dict(bs);model.eval()
191 + with torch.no_grad(): return model(Xc[te],Xnt[te]).cpu().numpy()*ys+ym
192 +def nn_cv(numcols):
193 + Xnum=df[numcols+MISS].astype(np.float32).values;pred=np.zeros_like(y)
194 + for tr,te in kf.split(Xnum):
195 + ps=np.zeros(len(te))
196 + for s in SEEDS: ps+=nn_fold(Xnum,tr,te,s)
197 + pred[te]=ps/len(SEEDS)
198 + return pred
199 +def stat(p):
200 + e=np.abs(np.exp(p-y)-1);return r2_score(y,p),np.median(e)*100,np.mean(e<=.15)*100,np.mean(e<=.25)*100
201 +
202 +print("\n>>> GBM ablation")
203 +g4=gbm_cv(NUM_v4);g5=gbm_cv(NUM_v5)
204 +print("\n>>> NN v5 (peut prendre ~2 min)")
205 +n5=nn_cv(NUM_v5)
206 +b5=0.5*g5+0.5*n5
207 +
208 +print("\n================ RÉSULTATS v5 (5-fold CV OOS) ================")
209 +print(f"{'Modèle':<34}{'R²OOS':>8}{'MAEméd':>9}{'±15%':>7}{'±25%':>7}")
210 +for nm,p in [("GBM v4 (texte)",g4),("GBM v5 (texte+vision)",g5),("NN v5 (texte+vision)",n5),("BLEND v5 (NN+GBM)",b5)]:
211 + r2,m,a,b=stat(p);print(f"{nm:<34}{r2:>8.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%")
212 +print("\nRappel v4 blend: 0.760 | GBM 0.743 | NN 0.741")
213 +np.save("data/processed/v5_blend_oos.npy",b5)
added scripts/10_final_blend.py +117 −0
@@ -0,0 +1,117 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Blend (stacking) NN robuste + HistGradientBoosting, mêmes folds (seed=42)."""
4 +import json, re, warnings
5 +import numpy as np, pandas as pd
6 +from sklearn.ensemble import HistGradientBoostingRegressor
7 +from sklearn.model_selection import KFold, cross_val_predict
8 +from sklearn.metrics import r2_score
9 +warnings.filterwarnings("ignore")
10 +CUR_YEAR=2026
11 +# --- rebuild df exactement comme torch_nn_robust (même ordre) ---
12 +specs={}
13 +for l in open("data/processed/specs.jsonl",encoding="utf-8"):
14 + r=json.loads(l)
15 + if r.get("specs"): specs[r["uid"]]=r["specs"]
16 +COND={"a_renover":0,"original":1,"bon":2,"renove":3,"refait_neuf":4,"neuf":5,"inconnu":2}
17 +BOOL=["turnkey","recently_renovated","garage","pool","waterfront","finished_basement","fireplace","central_ac","new_construction","income_property","renovated_kitchen","renovated_bathroom","large_lot","quiet_area"]
18 +def gfeat(uid):
19 + s=specs.get(uid);o={"g_"+b:0 for b in BOOL};o.update({"g_cond":2,"g_hardwood":0,"g_quartz_granite":0,"g_view_water":0})
20 + if not s: return o
21 + for b in BOOL: o["g_"+b]=int(bool(s.get(b)))
22 + o["g_cond"]=COND.get(s.get("condition","inconnu"),2)
23 + fl=s.get("flooring") or [];o["g_hardwood"]=int("bois_franc" in fl)
24 + o["g_quartz_granite"]=int(s.get("countertop") in ("quartz","granit","marbre"));o["g_view_water"]=int(s.get("view")=="eau");return o
25 +def parse_area(s):
26 + if not s or not isinstance(s,str): return np.nan
27 + t=s.lower();m=("m2" in t) or ("m²" in t);n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
28 + try: v=float(n)
29 + except: return np.nan
30 + return (v*10.7639 if m else v) if v>0 else np.nan
31 +def cc(c):
32 + if not c or not isinstance(c,str): return None
33 + c=c.split("(")[0].strip();return re.sub(r"\s+"," ",c).lower().strip() or None
34 +def ti(x):
35 + try: return int(x)
36 + except: return np.nan
37 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],"near_school":["school","école","cegep","university","université","elementary","high school"],"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
38 +def sd(s):
39 + s=s.lower()
40 + if "ft" in s:
41 + f=re.search(r"([\d.]+)\s*ft",s);i=re.search(r"([\d.]+)\s*in",s)
42 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
43 + m=re.search(r"([\d.,]+)\s*m",s)
44 + if m:
45 + try: return float(m.group(1).replace(",","."))*3.28084
46 + except: return None
47 + return None
48 +def ra(d):
49 + if not d or "x" not in d: return None
50 + a,_,b=d.partition("x");la,lb=sd(a),sd(b);return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
51 +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],"r_library":["library"],"r_laundry":["laundry"]}
52 +def pr(rooms):
53 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
54 + for k in RF: o[k]=0
55 + if not rooms: return o
56 + ar=0.0;lv=set()
57 + for r in rooms:
58 + nm=(r.get("name") or "").lower();l=(r.get("level") or "").lower();o["n_rooms"]+=1
59 + if l: lv.add(l)
60 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
61 + if "primary bedroom" in nm: o["has_primary_bed"]=1
62 + for k,kw in RF.items():
63 + if any(w in nm for w in kw): o[k]=1
64 + a=ra(r.get("dimensions"))
65 + if a: ar+=a
66 + o["room_area"]=ar if ar>0 else np.nan;o["n_levels_r"]=len(lv);return o
67 +data=json.load(open("data/processed/maisons_avec_prix.json",encoding="utf-8"));rows=[]
68 +for d in data:
69 + p=d.get("property") or {};ad=d.get("address") or {};ft=d.get("features") or []
70 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
71 + yb=ti(p.get("yearBuilt"));age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
72 + pt=p.get("buildingType") or p.get("type");st=p.get("storeys") or p.get("levels")
73 + try: st=float(str(st).replace(",","."))
74 + except: st=np.nan
75 + uid=f"{d.get('source')}:{d.get('listingId')}"
76 + rec={"uid":uid,"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),"lot_sqft":parse_area(p.get("lotSize")),"beds":ti(p.get("bedrooms")),"baths":ti(p.get("bathrooms")),"parking":ti(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),"city":cc(ad.get("city"))}
77 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
78 + rec["n_images"]=len(d.get("images") or []);rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
79 + rec.update(pr(d.get("rooms")));rec.update(gfeat(uid));rows.append(rec)
80 +df=pd.DataFrame(rows)
81 +df=df[df["price"].notna()];df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
82 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
83 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
84 +df.loc[df["beds"]>15,"beds"]=np.nan;df.loc[df["baths"]>12,"baths"]=np.nan
85 +df["ppsf"]=df["price"]/df["sqft"];lo,hi=df["ppsf"].quantile([0.005,0.995])
86 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
87 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)];ratio=(ov["sqft"]/ov["room_area"]).median()
88 +re_=df["room_area"]*ratio;ok=df["room_area"].notna()&(re_.between(300,20000))
89 +df["sqft_final"]=df["sqft"];fill=df["sqft_final"].isna()&ok;df.loc[fill,"sqft_final"]=re_[fill]
90 +df["has_sqft"]=df["sqft_final"].notna().astype(int);df["age2"]=df["age"]**2;df["n_images_c"]=df["n_images"].clip(upper=60)
91 +vc=df["city"].value_counts();keep=set(vc[vc>=20].index);df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
92 +tc=df["ptype"].value_counts();df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
93 +df=df.reset_index(drop=True)
94 +y=np.log(df["price"].values)
95 +amen=list(AMEN.keys());rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
96 +gv=["g_cond","g_hardwood","g_quartz_granite","g_view_water"]+["g_"+b for b in BOOL]
97 +num=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2","n_images_c","virtual_tour"]+amen+rv+gv
98 +Xcat=pd.get_dummies(df[["city_fe","ptype_fe","source"]],dtype=float)
99 +X=pd.concat([df[num].reset_index(drop=True),Xcat.reset_index(drop=True)],axis=1)
100 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
101 +gbm=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42)
102 +pgbm=cross_val_predict(gbm,X,y,cv=kf,n_jobs=-1)
103 +pnn=np.load("data/processed/nn_robust_oos.npy")
104 +assert len(pnn)==len(y),f"désalignement {len(pnn)} vs {len(y)}"
105 +def stat(p):
106 + e=np.abs(np.exp(p-y)-1)
107 + return r2_score(y,p),np.median(e)*100,np.mean(e<=.15)*100,np.mean(e<=.25)*100
108 +print(f"{'Modèle':<30}{'R² OOS':>9}{'MAE méd':>9}{'±15%':>7}{'±25%':>7}")
109 +for nm,p in [("HistGBM",pgbm),("NN robuste",pnn)]:
110 + r2,m,a,b=stat(p);print(f"{nm:<30}{r2:>9.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%")
111 +best=None
112 +for w in [0.3,0.4,0.5,0.6,0.7]:
113 + p=w*pgbm+(1-w)*pnn;r2,m,a,b=stat(p)
114 + tag=f"Blend {w:.1f}·GBM+{1-w:.1f}·NN"
115 + print(f"{tag:<30}{r2:>9.4f}{m:>8.1f}%{a:>6.0f}%{b:>6.0f}%")
116 + if best is None or r2>best[0]: best=(r2,w)
117 +print(f"\nMeilleur blend: w_GBM={best[1]:.1f} R²={best[0]:.4f}")
added scripts/11_extras_v5_robustness.py +134 −0
@@ -0,0 +1,134 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Robustesse (par segment) + figures primes de localisation & validation IA."""
4 +import numpy as np, pandas as pd, warnings, statsmodels.formula.api as smf
5 +import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
6 +warnings.filterwarnings("ignore")
7 +plt.rcParams.update({"font.family":"serif","font.size":10,"axes.grid":True,
8 + "grid.alpha":0.25,"axes.axisbelow":True,"figure.dpi":150,"savefig.bbox":"tight"})
9 +C1,C2,C3="#3B6EA5","#C4462E","#4A8C5B"
10 +exec(open("v5_ml.py").read().split("kf=KFold")[0])
11 +amen_=list(AMEN.keys())
12 +bl=np.load("data/processed/v5_blend_oos.npy"); price=df["price"].values; y=np.log(price)
13 +from sklearn.metrics import r2_score
14 +def stat(mask):
15 + yy=y[mask];pp=bl[mask];e=np.abs(np.exp(pp-yy)-1)
16 + return len(yy),r2_score(yy,pp),np.median(e)*100,np.mean(e<=.15)*100
17 +
18 +# ---- robustesse par segment ----
19 +seg=[]
20 +seg.append(("\\textit{Full sample}",np.ones(len(y),bool)))
21 +q1,q2=np.quantile(price,[1/3,2/3])
22 +seg.append(("\\quad Price tercile 1 (low)",price<=q1))
23 +seg.append(("\\quad Price tercile 2 (mid)",(price>q1)&(price<=q2)))
24 +seg.append(("\\quad Price tercile 3 (high)",price>q2))
25 +seg.append(("\\quad Realtor.ca",df["source"].values=="realtor"))
26 +seg.append(("\\quad DuProprio",df["source"].values=="duproprio"))
27 +seg.append(("\\quad Measured area",df["has_sqft"].values==1))
28 +seg.append(("\\quad Imputed area",df["has_sqft"].values==0))
29 +rows=[]
30 +for nm,m in seg:
31 + n,r2,md,w=stat(m)
32 + # R^2 non informatif sur une plage de prix restreinte (terciles) -> n/a
33 + r2c = "n/a$^{\\dagger}$" if "tercile" in nm else f"{r2:.3f}"
34 + rows.append(f" {nm} & {n:,} & {r2c} & {md:.1f}\\% & {w:.0f}\\% \\\\".replace(",","{,}"))
35 +tab=r"""% Auto-generated
36 +\begin{table}[htbp]
37 + \centering
38 + \begin{threeparttable}
39 + \caption{Robustness: Blend Performance Across Subsamples}
40 + \label{tab:robustness}
41 + \begin{tabular}{@{}lcccc@{}}
42 + \toprule
43 + Subsample & $N$ & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% \\
44 + \midrule
45 +"""+"\n".join(rows)+r"""
46 + \bottomrule
47 + \end{tabular}
48 + \begin{tablenotes}[flushleft]
49 + \footnotesize
50 + \item \textit{Notes:} Out-of-sample performance of the text\,+\,vision blend
51 + (Table~\ref{tab:models}, Panel~B) computed on subsamples of the same 5-fold
52 + cross-validated predictions. Price terciles are defined on the actual list price.
53 + ``Measured area'' are listings with a reported or room-reconstructed living area
54 + (priced by Model~A); ``Imputed area'' listings are priced by the area-free model.
55 + $^{\dagger}$Within a single price tercile the range of $\ln P$ is heavily
56 + restricted, so $R^2$---which normalizes by within-group variance---is
57 + uninformative; the median absolute error is the comparable accuracy metric across
58 + terciles.
59 + \end{tablenotes}
60 + \end{threeparttable}
61 +\end{table}
62 +"""
63 +open("paper/tables/robustness.tex","w").write(tab)
64 +print("-> tables/robustness.tex"); [print(f" {nm[:28]:<30} n={stat(m)[0]:5d} R2={stat(m)[1]:.3f} md={stat(m)[2]:.1f}%") for nm,m in seg]
65 +
66 +# ---- OLS v5 pour primes de localisation ----
67 +df["ln_price"]=y
68 +df["ln_sqft"]=np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median()))
69 +df["ln_lot"]=np.log(df["lot_sqft"].fillna(df["lot_sqft"].median()));df["has_lot"]=df["lot_sqft"].notna().astype(int)
70 +df["has_age"]=df["age"].notna().astype(int)
71 +for c in ["beds","baths","storeys","age","age2"]:
72 + df[c]=df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median())
73 +df["parking"]=df["parking"].fillna(0)
74 +form=("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+"+"+".join(amen_)+"+n_images_c+virtual_tour+"+"+".join(rv)+"+"+"+".join(gv)+"+"+"+".join(vv)+"+C(city_fe)")
75 +mA=smf.ols(form,data=df[df["has_sqft"]==1]).fit(cov_type="HC3")
76 +pct=lambda b:(np.exp(b)-1)*100
77 +cf={k.split("T.")[1].rstrip("]"):pct(v) for k,v in mA.params.items() if k.startswith("C(city_fe)")}
78 +s=pd.Series(cf).sort_values()
79 +top=s.tail(12);bot=s.head(8)
80 +fig,ax=plt.subplots(figsize=(6.6,4.6))
81 +sel=pd.concat([bot,top])
82 +cols=[C2 if v<0 else C1 for v in sel.values]
83 +ax.barh(range(len(sel)),sel.values,color=cols,edgecolor="white")
84 +ax.set_yticks(range(len(sel)));ax.set_yticklabels([c.title() for c in sel.index],fontsize=8)
85 +ax.axvline(0,color="black",lw=0.8)
86 +ax.set_xlabel("Location premium vs reference city (\\%)")
87 +ax.set_title("Municipality fixed effects (largest and smallest)",fontsize=11)
88 +fig.savefig("paper/figures/fig_premia.pdf");plt.close(fig);print("-> fig_premia.pdf")
89 +
90 +# ---- validation IA : prix moyen par score visuel ----
91 +ext=df[df["v_is_ext"]==1]
92 +fig,axs=plt.subplots(1,2,figsize=(7.6,3.5))
93 +for ax,col,lab,c in [(axs[0],"v_curb","Vision curb appeal (1--10)",C3),(axs[1],"v_cond","Vision condition score (1--10)",C1)]:
94 + g=ext.groupby(col)["price"].median()/1e3
95 + g=g[g.index>=3]
96 + ax.plot(g.index,g.values,"-o",color=c,lw=2)
97 + ax.set_xlabel(lab);ax.set_ylabel("Median price (\\$000)")
98 +axs[0].set_title("Price rises with visual curb appeal",fontsize=10)
99 +axs[1].set_title("Price rises with visual condition",fontsize=10)
100 +fig.savefig("paper/figures/fig_validation.pdf");plt.close(fig);print("-> fig_validation.pdf")
101 +
102 +# ---- table résumé features IA ----
103 +def sh(col): return df[col].mean()*100
104 +aitab=r"""% Auto-generated
105 +\begin{table}[htbp]
106 + \centering
107 + \begin{threeparttable}
108 + \caption{Machine-Generated Feature Summary}
109 + \label{tab:ai_summary}
110 + \begin{tabular}{@{}lc@{\hskip 1.2cm}lc@{}}
111 + \toprule
112 + Text (LLM) & Share & Vision (façade) & Mean / share \\
113 + \midrule
114 +"""+f""" Recently renovated & {sh('g_recently_renovated'):.0f}\\% & Condition score (1--10) & {df['v_cond'].mean():.1f} \\\\
115 + Renovated kitchen & {sh('g_renovated_kitchen'):.0f}\\% & Curb appeal (1--10) & {df['v_curb'].mean():.1f} \\\\
116 + Hardwood floor & {sh('g_hardwood'):.0f}\\% & Luxury cue & {sh('v_luxury'):.0f}\\% \\\\
117 + Quartz/granite counter & {sh('g_quartz_granite'):.0f}\\% & Garage visible & {sh('v_garage_vis'):.0f}\\% \\\\
118 + New construction & {sh('g_new_construction'):.0f}\\% & Brick façade & {sh('v_mat_brique'):.0f}\\% \\\\
119 + Income property & {sh('g_income_property'):.0f}\\% & Waterfront setting & {sh('v_set_bord_eau'):.0f}\\% \\\\
120 + Water view & {sh('g_view_water'):.0f}\\% & Exterior shot detected & {sh('v_is_ext'):.0f}\\% \\\\
121 +""" + r""" \bottomrule
122 + \end{tabular}
123 + \begin{tablenotes}[flushleft]
124 + \footnotesize
125 + \item \textit{Notes:} Prevalence of selected machine-generated attributes across
126 + the """ + f"{len(df):,}".replace(",","{,}") + r""" cleaned listings. Text attributes are extracted from the
127 + description by \texttt{gpt-4o-mini}; vision attributes from the façade photograph by
128 + \texttt{claude-haiku-4-5}. Scores are 1--10 means; the remainder are shares.
129 + \end{tablenotes}
130 + \end{threeparttable}
131 +\end{table}
132 +"""
133 +open("paper/tables/ai_summary.tex","w").write(aitab)
134 +print("-> tables/ai_summary.tex")
added scripts/12_ablation_extras.py +47 −0
@@ -0,0 +1,47 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Ablation CV complète (4 couches x OLS/GBM) pour cohérence du tableau ablation.tex."""
4 +import json, warnings
5 +import numpy as np, pandas as pd
6 +from sklearn.linear_model import LinearRegression
7 +from sklearn.ensemble import HistGradientBoostingRegressor
8 +from sklearn.model_selection import KFold, cross_val_predict
9 +from sklearn.metrics import r2_score
10 +warnings.filterwarnings("ignore")
11 +
12 +exec(open("v5_ml.py").read().split("kf=KFold")[0])
13 +kf = KFold(n_splits=5, shuffle=True, random_state=42)
14 +amen_ = list(AMEN.keys())
15 +
16 +L1 = ["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2"] + amen_
17 +L2 = L1 + ["n_images_c","virtual_tour"] + rv
18 +L3 = L2 + gv
19 +L4 = L3 + vv
20 +print("feats:", len(L1), len(L2), len(L3), len(L4))
21 +
22 +Xcat = pd.get_dummies(df[cats], prefix=cats, dtype=float)
23 +dfi = df.copy()
24 +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]:
25 + dfi[c] = dfi.groupby("ptype")[c].transform(lambda s: s.fillna(s.median()))
26 + dfi[c] = dfi[c].fillna(dfi[c].median())
27 +dfi["parking"] = dfi["parking"].fillna(0)
28 +
29 +def stats(pred):
30 + e = np.abs(np.exp(pred - y) - 1)
31 + return {"r2": float(r2_score(y, pred)), "mdae": float(np.median(e) * 100),
32 + "w15": float(np.mean(e <= .15) * 100), "w25": float(np.mean(e <= .25) * 100)}
33 +def cv_ols(numcols):
34 + X = pd.concat([dfi[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1)
35 + X = X.fillna(X.median())
36 + return stats(cross_val_predict(LinearRegression(), X, y, cv=kf, n_jobs=-1))
37 +def cv_gbm(numcols):
38 + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1)
39 + m = HistGradientBoostingRegressor(max_iter=600, learning_rate=0.05, max_leaf_nodes=63, l2_regularization=1.0, random_state=42)
40 + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1))
41 +
42 +out = {}
43 +for name, cols in [("L1", L1), ("L2", L2), ("L3", L3), ("L4", L4)]:
44 + out[name] = {"nfeat": len(cols), "ols": cv_ols(cols), "gbm": cv_gbm(cols)}
45 + print(name, out[name])
46 +json.dump(out, open("results/ablation_extras.json", "w"), indent=1)
47 +print("-> ablation_extras.json")
added scripts/13_revision_extras.py +112 −0
@@ -0,0 +1,112 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Estimations complémentaires pour la révision JHE :
4 + - OLS A/B avec SE HC3 ET clusterisées par municipalité
5 + - ablation in-sample du modèle A (base / +texte / +texte+vision)
6 + - effets fixes de ville extrêmes
7 + - CV 5-fold OLS et RF pour les panels A (texte) et B (texte+vision)
8 + Résultats -> revision_extras.json
9 +"""
10 +import json, warnings
11 +import numpy as np, pandas as pd
12 +import statsmodels.formula.api as smf
13 +from sklearn.linear_model import LinearRegression
14 +from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
15 +from sklearn.model_selection import KFold, cross_val_predict
16 +from sklearn.metrics import r2_score
17 +warnings.filterwarnings("ignore")
18 +
19 +# pipeline v5 jusqu'avant la CV (construit df, AMEN, rv, gv, vv, NUM_v4, NUM_v5, MISS, cats, y, kf)
20 +exec(open("v5_ml.py").read().split("kf=KFold")[0])
21 +amen_ = list(AMEN.keys())
22 +out = {}
23 +
24 +# ---------- OLS A/B : HC3 vs cluster ----------
25 +df["ln_price"] = np.log(df["price"])
26 +df["ln_sqft"] = np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median()))
27 +df["ln_lot"] = np.log(df["lot_sqft"].fillna(df["lot_sqft"].median())); df["has_lot"] = df["lot_sqft"].notna().astype(int)
28 +df["has_age"] = df["age"].notna().astype(int)
29 +for c in ["beds","baths","storeys","age","age2"]:
30 + df[c] = df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median())
31 +df["parking"] = df["parking"].fillna(0)
32 +
33 +form = ("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+"
34 + + "+".join(amen_) + "+n_images_c+virtual_tour+" + "+".join(rv) + "+" + "+".join(gv) + "+" + "+".join(vv) + "+C(city_fe)")
35 +dfA = df[df["has_sqft"] == 1]
36 +mA = smf.ols(form, data=dfA).fit(cov_type="HC3")
37 +mB = smf.ols(form.replace("ln_sqft+",""), data=df).fit(cov_type="HC3")
38 +mAc = smf.ols(form, data=dfA).fit(cov_type="cluster", cov_kwds={"groups": dfA["city_fe"]})
39 +mBc = smf.ols(form.replace("ln_sqft+",""), data=df).fit(cov_type="cluster", cov_kwds={"groups": df["city_fe"]})
40 +
41 +keys = ["ln_sqft","ln_lot","beds","baths","parking","age","age2",
42 + "central_air","transit","waterfront",
43 + "g_cond","g_renovated_kitchen","g_hardwood","g_new_construction","g_income_property",
44 + "v_curb","v_reno","v_luxury","v_garage_vis",
45 + "v_mat_brique","v_mat_pierre","v_mat_vinyle","v_set_rural","v_land","v_cond"]
46 +def grab(m, mc):
47 + o = {}
48 + for k in keys:
49 + if k in m.params.index:
50 + o[k] = {"b": float(m.params[k]), "se_hc3": float(m.bse[k]), "p_hc3": float(m.pvalues[k]),
51 + "se_cl": float(mc.bse[k]), "p_cl": float(mc.pvalues[k])}
52 + return o
53 +out["modelA"] = {"r2": mA.rsquared, "adj": mA.rsquared_adj, "n": int(mA.nobs), "coef": grab(mA, mAc)}
54 +out["modelB"] = {"r2": mB.rsquared, "adj": mB.rsquared_adj, "n": int(mB.nobs), "coef": grab(mB, mBc)}
55 +print("A R2=%.4f N=%d | B R2=%.4f N=%d" % (mA.rsquared, mA.nobs, mB.rsquared, mB.nobs))
56 +
57 +# ---------- ablation in-sample modèle A ----------
58 +base_form = ("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+"
59 + + "+".join(amen_) + "+n_images_c+virtual_tour+" + "+".join(rv) + "+C(city_fe)")
60 +mA0 = smf.ols(base_form, data=dfA).fit()
61 +mA1 = smf.ols(base_form + "+" + "+".join(gv), data=dfA).fit()
62 +mA2 = smf.ols(base_form + "+" + "+".join(gv) + "+" + "+".join(vv), data=dfA).fit()
63 +out["ablationA_insample"] = {"base": mA0.rsquared, "plus_text": mA1.rsquared, "plus_text_vision": mA2.rsquared}
64 +print("ablation A in-sample:", out["ablationA_insample"])
65 +
66 +# ---------- effets fixes ville extrêmes ----------
67 +fes = {k.split("T.")[1].rstrip("]"): float(v) for k, v in mA.params.items() if k.startswith("C(city_fe)")}
68 +top = sorted(fes.items(), key=lambda kv: -kv[1])[:12]
69 +bot = sorted(fes.items(), key=lambda kv: kv[1])[:8]
70 +out["city_fe_top"] = [(k, (np.exp(v) - 1) * 100) for k, v in top]
71 +out["city_fe_bot"] = [(k, (np.exp(v) - 1) * 100) for k, v in bot]
72 +print("top villes:", out["city_fe_top"][:4])
73 +
74 +# ---------- CV : OLS et RF, panels texte / texte+vision ----------
75 +kf = KFold(n_splits=5, shuffle=True, random_state=42)
76 +Xcat = pd.get_dummies(df[cats], prefix=cats, dtype=float)
77 +dfi = df.copy()
78 +def stats(pred):
79 + e = np.abs(np.exp(pred - y) - 1)
80 + return {"r2": float(r2_score(y, pred)), "mdae": float(np.median(e) * 100),
81 + "w15": float(np.mean(e <= .15) * 100), "w25": float(np.mean(e <= .25) * 100)}
82 +def cv_ols(numcols):
83 + X = pd.concat([dfi[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1)
84 + X = X.fillna(X.median())
85 + return stats(cross_val_predict(LinearRegression(), X, y, cv=kf, n_jobs=-1))
86 +def cv_rf(numcols):
87 + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1)
88 + m = RandomForestRegressor(n_estimators=400, max_features=0.5, min_samples_leaf=3, n_jobs=-1, random_state=42)
89 + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1))
90 +def cv_gbm(numcols):
91 + X = pd.concat([df[numcols].reset_index(drop=True), Xcat.reset_index(drop=True)], axis=1)
92 + m = HistGradientBoostingRegressor(max_iter=600, learning_rate=0.05, max_leaf_nodes=63, l2_regularization=1.0, random_state=42)
93 + return stats(cross_val_predict(m, X, y, cv=kf, n_jobs=-1))
94 +
95 +# matrice OLS imputée médiane par type (comme hedonic_ml)
96 +for c in ["sqft_final","lot_sqft","beds","baths","storeys","age","age2"]:
97 + dfi[c] = dfi.groupby("ptype")[c].transform(lambda s: s.fillna(s.median()))
98 + dfi[c] = dfi[c].fillna(dfi[c].median())
99 +dfi["parking"] = dfi["parking"].fillna(0)
100 +
101 +print(">>> CV OLS v4/v5")
102 +out["cv"] = {"ols_v4": cv_ols(NUM_v4), "ols_v5": cv_ols(NUM_v5)}
103 +print("OLS:", out["cv"]["ols_v4"], out["cv"]["ols_v5"])
104 +print(">>> CV GBM v4/v5 (contrôle)")
105 +out["cv"]["gbm_v4"] = cv_gbm(NUM_v4); out["cv"]["gbm_v5"] = cv_gbm(NUM_v5)
106 +print("GBM:", out["cv"]["gbm_v4"], out["cv"]["gbm_v5"])
107 +print(">>> CV RF v4/v5 (plus long)")
108 +out["cv"]["rf_v4"] = cv_rf(NUM_v4); out["cv"]["rf_v5"] = cv_rf(NUM_v5)
109 +print("RF:", out["cv"]["rf_v4"], out["cv"]["rf_v5"])
110 +
111 +json.dump(out, open("results/revision_extras.json", "w"), indent=1, default=float)
112 +print("-> revision_extras.json")
added scripts/14_figures_v5.py +152 −0
@@ -0,0 +1,152 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Figures + tableau d'ablation détaillé pour le papier v5."""
4 +import numpy as np, pandas as pd, warnings, json
5 +import matplotlib
6 +matplotlib.use("Agg")
7 +import matplotlib.pyplot as plt
8 +from sklearn.linear_model import LinearRegression
9 +from sklearn.ensemble import HistGradientBoostingRegressor
10 +from sklearn.model_selection import KFold, cross_val_predict
11 +from sklearn.metrics import r2_score
12 +from sklearn.inspection import permutation_importance
13 +warnings.filterwarnings("ignore")
14 +plt.rcParams.update({"font.family":"serif","font.size":10,"axes.grid":True,
15 + "grid.alpha":0.25,"axes.axisbelow":True,"figure.dpi":150,"savefig.bbox":"tight"})
16 +C_OLS,C_GBM,C_BL="#3B6EA5","#C4462E","#4A8C5B"
17 +
18 +# build df via v5_ml prefix
19 +exec(open("v5_ml.py").read().split("kf=KFold")[0])
20 +kf=KFold(n_splits=5,shuffle=True,random_state=42)
21 +onehot=pd.get_dummies(df[cats],dtype=float).reset_index(drop=True)
22 +
23 +struct=["sqft_final","has_sqft","lot_sqft","beds","baths","parking","storeys","age","age2"]
24 +amen_=list(AMEN.keys())
25 +S0=struct+amen_
26 +S1=S0+rv+["n_images_c","virtual_tour"]
27 +S2=S1+gv
28 +S3=S2+vv
29 +sets=[("Structural\n+ amenity",S0),("+ Rooms\n+ listing",S1),("+ Text\n(LLM)",S2),("+ Vision\n(façade)",S3)]
30 +
31 +def ols_r2(cols):
32 + Xi=df[cols].copy()
33 + for c in cols: Xi[c]=Xi[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median())
34 + X=pd.concat([Xi.reset_index(drop=True),onehot],axis=1)
35 + p=cross_val_predict(LinearRegression(),X,y,cv=kf,n_jobs=-1);return r2_score(y,p),p
36 +def gbm_r2(cols):
37 + X=pd.concat([df[cols].reset_index(drop=True),onehot],axis=1)
38 + m=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42)
39 + p=cross_val_predict(m,X,y,cv=kf,n_jobs=-1);return r2_score(y,p),p
40 +
41 +ols=[];gbm=[];gpred=None
42 +for nm,cols in sets:
43 + ro,_=ols_r2(cols);rg,pg=gbm_r2(cols);ols.append(ro);gbm.append(rg)
44 + if nm.startswith("+ Vision"): gpred=pg
45 + print(f"{nm.replace(chr(10),' '):<22} OLS={ro:.4f} GBM={rg:.4f}")
46 +
47 +# blend numbers (déjà calculés) : S2 texte, S3 texte+vision
48 +blend={2:0.760,3:0.779}
49 +
50 +# ===== Figure 1 : ablation =====
51 +x=np.arange(len(sets))
52 +fig,ax=plt.subplots(figsize=(6.4,3.9))
53 +ax.plot(x,ols,"-o",color=C_OLS,lw=2,label="OLS (linear)")
54 +ax.plot(x,gbm,"-s",color=C_GBM,lw=2,label="Gradient boosting")
55 +ax.plot([2,3],[blend[2],blend[3]],"-D",color=C_BL,lw=2,label="Blend (NN+GBM)")
56 +for xi,(ro,rg) in enumerate(zip(ols,gbm)):
57 + ax.annotate(f"{rg:.3f}",(xi,rg),textcoords="offset points",xytext=(0,7),ha="center",fontsize=8,color=C_GBM)
58 + ax.annotate(f"{ro:.3f}",(xi,ro),textcoords="offset points",xytext=(0,-13),ha="center",fontsize=8,color=C_OLS)
59 +ax.annotate(f"{blend[3]:.3f}",(3,blend[3]),textcoords="offset points",xytext=(0,7),ha="center",fontsize=8,color=C_BL)
60 +ax.set_xticks(x);ax.set_xticklabels([s[0] for s in sets])
61 +ax.set_ylabel(r"Out-of-sample $R^2$ (5-fold CV)");ax.set_ylim(0.55,0.80)
62 +ax.legend(frameon=False,loc="lower right",fontsize=9)
63 +ax.set_title("Incremental gain from each feature layer",fontsize=11)
64 +fig.savefig("paper/figures/fig_ablation.pdf");plt.close(fig)
65 +print("-> fig_ablation.pdf")
66 +
67 +# ===== Figure 2 : calibration + résidus (blend v5) =====
68 +try: bl=np.load("data/processed/v5_blend_oos.npy")
69 +except Exception: bl=gpred
70 +yp=np.exp(bl);ya=np.exp(y)
71 +fig,axs=plt.subplots(1,2,figsize=(7.6,3.6))
72 +ax=axs[0]
73 +hb=ax.hexbin(ya/1e3,yp/1e3,gridsize=45,bins="log",cmap="Blues",xscale="log",yscale="log",mincnt=1)
74 +lim=[60,6000];ax.plot(lim,lim,"--",color="black",lw=1)
75 +ax.set_xlim(lim);ax.set_ylim(lim);ax.set_xlabel("Actual price (\\$000)");ax.set_ylabel("Predicted price (\\$000)")
76 +ax.set_title(f"Calibration (blend, $R^2$={r2_score(y,bl):.3f})",fontsize=10)
77 +r=(yp/ya-1)*100
78 +ax2=axs[1]
79 +ax2.hist(np.clip(r,-60,60),bins=50,color=C_BL,alpha=0.85,edgecolor="white",lw=0.3)
80 +ax2.axvline(0,color="black",lw=1,ls="--")
81 +ax2.set_xlabel("Prediction error (\\%)");ax2.set_ylabel("Listings")
82 +ax2.set_title(f"Errors (median |.|={np.median(np.abs(r)):.1f}\\%)",fontsize=10)
83 +fig.savefig("paper/figures/fig_calibration.pdf");plt.close(fig)
84 +print("-> fig_calibration.pdf")
85 +
86 +# ===== Figure 3 : importance par bloc (GBM v5) =====
87 +Xfull=pd.concat([df[S3].reset_index(drop=True),onehot],axis=1)
88 +gb=HistGradientBoostingRegressor(max_iter=600,learning_rate=0.05,max_leaf_nodes=63,l2_regularization=1.0,random_state=42).fit(Xfull,y)
89 +idx=np.random.RandomState(0).choice(len(Xfull),4000,replace=False)
90 +pi=permutation_importance(gb,Xfull.iloc[idx],y[idx],n_repeats=3,random_state=0,n_jobs=-1)
91 +imp=pd.Series(pi.importances_mean,index=Xfull.columns).clip(lower=0)
92 +def block(c):
93 + if c.startswith("city_fe") or c.startswith("ptype_fe") or c.startswith("source"): return "Location/type FE"
94 + if c in ("sqft_final","has_sqft","lot_sqft"): return "Size"
95 + if c in ("beds","baths","parking","storeys","age","age2"): return "Structural"
96 + if c in amen_: return "Amenities (rules)"
97 + if c in rv or c in ("n_images_c","virtual_tour"): return "Rooms / listing"
98 + if c in gv: return "Text (LLM)"
99 + if c in vv: return "Vision (façade)"
100 + return "Other"
101 +bl_imp=imp.groupby(imp.index.map(block)).sum().sort_values()
102 +bl_imp=bl_imp/bl_imp.sum()*100
103 +fig,ax=plt.subplots(figsize=(6.4,3.6))
104 +colors=["#8899AA"]*len(bl_imp)
105 +for i,n in enumerate(bl_imp.index):
106 + if n=="Vision (façade)": colors[i]=C_GBM
107 + if n=="Text (LLM)": colors[i]=C_BL
108 +ax.barh(bl_imp.index,bl_imp.values,color=colors,edgecolor="white")
109 +for i,v in enumerate(bl_imp.values): ax.text(v+0.4,i,f"{v:.0f}\\%",va="center",fontsize=8)
110 +ax.set_xlabel("Share of permutation importance (\\%)")
111 +ax.set_title("Feature-block importance (gradient boosting, v5)",fontsize=11)
112 +ax.set_xlim(0,max(bl_imp.values)*1.15)
113 +fig.savefig("paper/figures/fig_importance.pdf");plt.close(fig)
114 +print("-> fig_importance.pdf")
115 +print("blocs:",{k:round(v,1) for k,v in bl_imp.sort_values(ascending=False).items()})
116 +
117 +# ===== Table ablation détaillée =====
118 +def med(p): return np.median(np.abs(np.exp(p-y)-1))*100
119 +rowsL=[]
120 +labels=["Structural + amenity","\\quad + Rooms + listing","\\quad + Text (LLM)","\\quad + Vision (façade)"]
121 +for i,(nm,cols) in enumerate(sets):
122 + ro,po=ols_r2(cols);rg,pg=gbm_r2(cols)
123 + bcell = f"{blend[i]:.3f}" if i in blend else "---"
124 + rowsL.append(f" {labels[i]} & {len(cols)} & {ro:.3f} & {rg:.3f} & {bcell} \\\\")
125 +tab=r"""% Auto-generated v5
126 +\begin{table}[htbp]
127 + \centering
128 + \begin{threeparttable}
129 + \caption{Feature Ablation: Out-of-Sample $R^2$ by Feature Layer}
130 + \label{tab:ablation}
131 + \begin{tabular}{@{}lcccc@{}}
132 + \toprule
133 + Feature set & \# feats & OLS & GBM & Blend \\
134 + \midrule
135 +"""+"\n".join(rowsL)+r"""
136 + \bottomrule
137 + \end{tabular}
138 + \begin{tablenotes}[flushleft]
139 + \footnotesize
140 + \item \textit{Notes:} Each row adds a feature layer to the row above (nested).
141 + Entries are out-of-sample $R^2$ on $\ln(\text{Price})$ from 5-fold cross-validation
142 + with a common seed. ``\# feats'' counts numeric/indicator features before the city,
143 + type, and source fixed effects (added in every specification). The blend averages
144 + the robust neural network and gradient boosting and is reported where both were
145 + trained. GBM and the neural network handle missing living area natively; OLS uses
146 + within-type median imputation.
147 + \end{tablenotes}
148 + \end{threeparttable}
149 +\end{table}
150 +"""
151 +open("paper/tables/ablation.tex","w").write(tab)
152 +print("-> paper/tables/ablation.tex")
added scripts/15_make_tables.py +209 −0
@@ -0,0 +1,209 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Génère les tableaux LaTeX (summary stats + régression) avec vrais chiffres."""
4 +import json, re, warnings
5 +import numpy as np, pandas as pd
6 +import statsmodels.formula.api as smf
7 +warnings.filterwarnings("ignore")
8 +CUR_YEAR = 2026
9 +data = json.load(open("data/processed/maisons_avec_prix.json", encoding="utf-8"))
10 +
11 +def parse_area(s):
12 + if not s or not isinstance(s, str): return np.nan
13 + t=s.lower(); m=("m2" in t) or ("m²" in t)
14 + n=re.sub(r"[^0-9.,\s]","",t).strip().replace(" ","").replace(",","")
15 + try: v=float(n)
16 + except: return np.nan
17 + return (v*10.7639 if m else v) if v>0 else np.nan
18 +def clean_city(c):
19 + if not c or not isinstance(c,str): return None
20 + c=c.split("(")[0].strip(); return re.sub(r"\s+"," ",c).lower().strip() or None
21 +def to_int(x):
22 + try: return int(x)
23 + except: return np.nan
24 +AMEN={"pool":["swimming pool","piscine","pool"],"waterfront":["waterfront","bord de l'eau","navigable","lakefront","riverfront","au bord de"],
25 +"garage":["garage"],"fireplace":["fireplace","foyer","wood stove","poêle"],
26 +"central_air":["central air","air conditionné","climatisation","a/c","heat-pump","thermopompe","air exchanger"],
27 +"finished_bsmt":["totally finished","finished basement","sous-sol fini","basement"],
28 +"near_water":["near water","waterfront","navigable"],"cul_de_sac":["cul-de-sac","cul de sac"],
29 +"golf":["golf"],"ski":["ski"],"near_park":["park","parc"],
30 +"near_school":["school","école","cegep","university","université","elementary","high school"],
31 +"near_highway":["highway","autoroute"],"transit":["public transit","public transportation","transport"],
32 +"no_rear_neighbor":["no backyard neighbors","sans voisin arrière"]}
33 +def side(s):
34 + s=s.lower()
35 + if "ft" in s:
36 + f=re.search(r"([\d.]+)\s*ft",s); i=re.search(r"([\d.]+)\s*in",s)
37 + return (float(f.group(1)) if f else 0)+(float(i.group(1))/12 if i else 0)
38 + m=re.search(r"([\d.,]+)\s*m",s)
39 + if m:
40 + try: return float(m.group(1).replace(",","."))*3.28084
41 + except: return None
42 + return None
43 +def rarea(d):
44 + if not d or "x" not in d: return None
45 + a,_,b=d.partition("x"); la,lb=side(a),side(b)
46 + return la*lb if (la and lb and 1<la<200 and 1<lb<200) else None
47 +RF={"r_office":["office"],"r_family":["family room"],"r_workshop":["workshop"],
48 +"r_winecellar":["wine cellar"],"r_sunroom":["sunroom","solarium"],"r_mezzanine":["mezzanine"],
49 +"r_library":["library"],"r_laundry":["laundry"]}
50 +def prooms(rooms):
51 + o={"room_area":np.nan,"n_rooms":0,"has_primary_bed":0,"has_basement":0,"n_levels_r":0}
52 + for k in RF: o[k]=0
53 + if not rooms: return o
54 + ar=0.0; lv=set()
55 + for r in rooms:
56 + nm=(r.get("name") or "").lower(); l=(r.get("level") or "").lower(); o["n_rooms"]+=1
57 + if l: lv.add(l)
58 + if "basement" in l or "sous-sol" in l: o["has_basement"]=1
59 + if "primary bedroom" in nm: o["has_primary_bed"]=1
60 + for k,kw in RF.items():
61 + if any(w in nm for w in kw): o[k]=1
62 + a=rarea(r.get("dimensions"))
63 + if a: ar+=a
64 + o["room_area"]=ar if ar>0 else np.nan; o["n_levels_r"]=len(lv); return o
65 +rows=[]
66 +for d in data:
67 + p=d.get("property") or {}; ad=d.get("address") or {}; ft=d.get("features") or []
68 + blob=(" ".join(ft)+" "+(d.get("description") or "")+" "+(d.get("title") or "")).lower()
69 + yb=to_int(p.get("yearBuilt")); age=(CUR_YEAR-yb) if (not pd.isna(yb) and 1700<=yb<=CUR_YEAR) else np.nan
70 + pt=p.get("buildingType") or p.get("type"); st=p.get("storeys") or p.get("levels")
71 + try: st=float(str(st).replace(",","."))
72 + except: st=np.nan
73 + rec={"price":d.get("priceNumber"),"source":d.get("source"),"sqft":parse_area(p.get("squareFeet")),
74 + "lot_sqft":parse_area(p.get("lotSize")),"beds":to_int(p.get("bedrooms")),"baths":to_int(p.get("bathrooms")),
75 + "parking":to_int(p.get("parkingSpaces")),"storeys":st,"age":age,"ptype":(pt or "Unknown").strip(),
76 + "city":clean_city(ad.get("city"))}
77 + for n,kw in AMEN.items(): rec[n]=int(any(k in blob for k in kw))
78 + rec["n_images"]=len(d.get("images") or []); rec["virtual_tour"]=int(bool(d.get("virtualTourUrl")))
79 + rec.update(prooms(d.get("rooms"))); rows.append(rec)
80 +df=pd.DataFrame(rows)
81 +df=df[df["price"].notna()]; df=df[(df["price"]>=50_000)&(df["price"]<=15_000_000)]
82 +df.loc[(df["sqft"]<300)|(df["sqft"]>20_000),"sqft"]=np.nan
83 +df.loc[(df["lot_sqft"]<200)|(df["lot_sqft"]>5_000_000),"lot_sqft"]=np.nan
84 +df.loc[df["beds"]>15,"beds"]=np.nan; df.loc[df["baths"]>12,"baths"]=np.nan
85 +df["ppsf"]=df["price"]/df["sqft"]; lo,hi=df["ppsf"].quantile([0.005,0.995])
86 +df=df[~(df["ppsf"].notna()&((df["ppsf"]<lo)|(df["ppsf"]>hi)))]
87 +ov=df[df["sqft"].notna()&df["room_area"].notna()&(df["room_area"]>200)]
88 +ratio=(ov["sqft"]/ov["room_area"]).median(); re_=df["room_area"]*ratio
89 +ok=df["room_area"].notna()&(re_.between(300,20000))
90 +df["sqft_final"]=df["sqft"]; fill=df["sqft_final"].isna()&ok; df.loc[fill,"sqft_final"]=re_[fill]
91 +df["has_sqft"]=df["sqft_final"].notna().astype(int)
92 +df["sqft_imp"]=df.groupby("ptype")["sqft_final"].transform(lambda s:s.fillna(s.median())).fillna(df["sqft_final"].median())
93 +df["has_lot"]=df["lot_sqft"].notna().astype(int); df["lot_sqft"]=df["lot_sqft"].fillna(df["lot_sqft"].median())
94 +df["has_age"]=df["age"].notna().astype(int); df["age"]=df["age"].fillna(df["age"].median()).clip(lower=0)
95 +for c in ["beds","baths","storeys"]: df[c]=df[c].fillna(df[c].median())
96 +df["parking"]=df["parking"].fillna(0)
97 +df["ln_price"]=np.log(df["price"]); df["ln_sqft"]=np.log(df["sqft_imp"])
98 +df["ln_lot"]=np.log(df["lot_sqft"]); df["age2"]=df["age"]**2; df["n_images_c"]=df["n_images"].clip(upper=60)
99 +vc=df["city"].value_counts(); keep=set(vc[vc>=20].index)
100 +df["city_fe"]=df["city"].where(df["city"].isin(keep),"__other__").fillna("__other__")
101 +tc=df["ptype"].value_counts(); df["ptype_fe"]=df["ptype"].where(df["ptype"].isin(set(tc[tc>=30].index)),"Other")
102 +
103 +amen=list(AMEN.keys()); rv=["n_rooms","n_levels_r","has_primary_bed","has_basement","r_office","r_family","r_workshop","r_winecellar","r_sunroom","r_mezzanine","r_library","r_laundry"]
104 +extra=["n_images_c","virtual_tour"]+rv
105 +base="ln_lot + has_lot + beds + baths + parking + storeys + age + age2 + has_age + C(source) + C(ptype_fe) + "+" + ".join(amen)+" + "+" + ".join(extra)+" + C(city_fe)"
106 +fA="ln_price ~ ln_sqft + "+base; fB="ln_price ~ "+base
107 +dfA=df[df["has_sqft"]==1].copy()
108 +mA=smf.ols(fA,data=dfA).fit(cov_type="HC3"); mB=smf.ols(fB,data=df).fit(cov_type="HC3")
109 +sdA,sdB=np.std(mA.resid),np.std(mB.resid)
110 +
111 +# ---------- summary stats table ----------
112 +def q(s,p): return s.quantile(p)
113 +def fmt(x,d=0):
114 + if d==0: return f"{x:,.0f}".replace(",","{,}")
115 + return f"{x:,.{d}f}".replace(",","{,}")
116 +svars=[("Price (CAD)","price",0),("Living area (ft$^2$)","sqft_final",0),
117 +("Lot area (ft$^2$)","lot_sqft",0),("Bedrooms","beds",1),("Bathrooms","baths",1),
118 +("Parking spaces","parking",1),("Storeys","storeys",1),("Age (years)","age",1),
119 +("Rooms (Realtor)","n_rooms",1),("Photos","n_images",0)]
120 +lines=[]
121 +for lab,c,d in svars:
122 + s=df[c].dropna();
123 + lines.append(f"{lab} & {len(s):,} & {fmt(s.mean(),d)} & {fmt(s.std(),d)} & {fmt(q(s,.25),d)} & {fmt(q(s,.5),d)} & {fmt(q(s,.75),d)} \\\\".replace(",","{,}"))
124 +share_r=(df.source=="realtor").mean()*100
125 +tbl_sum=r"""% Auto-generated
126 +\begin{table}[htbp]
127 + \centering
128 + \begin{threeparttable}
129 + \caption{Summary Statistics}
130 + \label{tab:summary_stats}
131 + \begin{tabular}{@{}lcccccc@{}}
132 + \toprule
133 + Variable & $N$ & Mean & Std.\ Dev. & $p_{25}$ & Median & $p_{75}$ \\
134 + \midrule
135 +"""+"\n".join(" "+l for l in lines)+r"""
136 + \bottomrule
137 + \end{tabular}
138 + \begin{tablenotes}[flushleft]
139 + \footnotesize
140 + \item \textit{Notes:} Descriptive statistics for the cleaned sample of """+f"{len(df):,}".replace(",","{,}")+r""" single-family and townhouse listings (Realtor.ca and DuProprio), scraped on 3~July~2026. """+f"{share_r:.0f}"+r"""\% of listings originate from Realtor.ca. Living area is the reported area, or reconstructed from room dimensions where missing. Continuous variables are bounded by the outlier rules described in Section~\ref{sec:data}; price-per-square-foot is trimmed at the 0.5th and 99.5th percentiles.
141 + \end{tablenotes}
142 + \end{threeparttable}
143 +\end{table}
144 +"""
145 +open("paper/tables/summary_stats.tex","w").write(tbl_sum)
146 +
147 +# ---------- regression table ----------
148 +def star(p): return "^{***}" if p<.01 else "^{**}" if p<.05 else "^{*}" if p<.1 else ""
149 +disp=[("ln_sqft",r"$\ln(\text{Living area})$"),("ln_lot",r"$\ln(\text{Lot area})$"),
150 +("beds","Bedrooms"),("baths","Bathrooms"),("parking","Parking"),("age","Age"),
151 +("central_air","Central A/C"),("transit","Public transit"),("waterfront","Waterfront"),
152 +("pool","Pool"),("garage","Garage"),("fireplace","Fireplace"),("near_school","Near school"),
153 +("has_primary_bed","Primary bedroom"),("r_winecellar","Wine cellar"),
154 +("n_images_c","Photos (count)"),("virtual_tour","Virtual tour"),
155 +("C(source)[T.realtor]","Realtor.ca (vs DuProprio)")]
156 +def cell(m,key):
157 + if key not in m.params.index: return "",""
158 + b=m.params[key]; se=m.bse[key]; p=m.pvalues[key]
159 + sign="$-$" if b<0 else ""
160 + return f"{sign}{abs(b):.3f}${star(p)}$", f"({se:.3f})"
161 +rowsL=[]
162 +for key,lab in disp:
163 + a1,a2=cell(mA,key); b1,b2=cell(mB,key)
164 + rowsL.append(f" {lab} & {a1} & {b1} \\\\")
165 + rowsL.append(f" & {a2} & {b2} \\\\[3pt]")
166 +tbl_reg=r"""% Auto-generated
167 +\begin{table}[htbp]
168 + \centering
169 + \begin{threeparttable}
170 + \caption{Hedonic Regression Results}
171 + \label{tab:regression}
172 + \begin{tabular}{@{}lcc@{}}
173 + \toprule
174 + & (1) Model~A & (2) Model~B \\
175 + & \textit{rich} & \textit{broad} \\
176 + \midrule
177 +"""+"\n".join(rowsL)+r"""
178 + \midrule
179 + City FE & Yes & Yes \\
180 + Property-type FE & Yes & Yes \\
181 + Living area & Yes & No \\
182 + Observations & """+f"{int(mA.nobs):,}".replace(",","{,}")+r""" & """+f"{int(mB.nobs):,}".replace(",","{,}")+r""" \\
183 + $R^2$ & """+f"{mA.rsquared:.3f}"+r""" & """+f"{mB.rsquared:.3f}"+r""" \\
184 + Adjusted $R^2$ & """+f"{mA.rsquared_adj:.3f}"+r""" & """+f"{mB.rsquared_adj:.3f}"+r""" \\
185 + \bottomrule
186 + \end{tabular}
187 + \begin{tablenotes}[flushleft]
188 + \footnotesize
189 + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic}. The dependent variable is $\ln(\text{Price})$. Model~A includes $\ln(\text{Living area})$ and is estimated on listings with a measured (reported or room-reconstructed) area; Model~B omits living area and is estimated on the full cleaned sample. All specifications include city and property-type fixed effects and the full set of amenity, room-detail, and listing-signal controls (Section~\ref{sec:methodology}); only selected coefficients are reported. Heteroskedasticity-robust (HC3) standard errors in parentheses. $^{***}$, $^{**}$, $^{*}$ denote significance at the 1\%, 5\%, and 10\% levels.
190 + \end{tablenotes}
191 + \end{threeparttable}
192 +\end{table}
193 +"""
194 +open("paper/tables/regression.tex","w").write(tbl_reg)
195 +
196 +# key numbers for the text
197 +predA=np.exp(mA.predict(df))*np.exp(sdA**2/2); predB=np.exp(mB.predict(df))*np.exp(sdB**2/2)
198 +df["pred"]=np.where(df["has_sqft"]==1,predA,predB); df["rp"]=(df["price"]/df["pred"]-1)*100
199 +print("N_clean",len(df),"N_A",int(mA.nobs),"N_B",int(mB.nobs))
200 +print("R2_A %.4f adj %.4f errpct %.1f"%(mA.rsquared,mA.rsquared_adj,(np.exp(sdA)-1)*100))
201 +print("R2_B %.4f adj %.4f errpct %.1f"%(mB.rsquared,mB.rsquared_adj,(np.exp(sdB)-1)*100))
202 +print("median|err| %.1f within15 %.0f within25 %.0f"%(df.rp.abs().median(),(df.rp.abs()<=15).mean()*100,(df.rp.abs()<=25).mean()*100))
203 +print("nb_cities_fe",df.city_fe.nunique(),"median_price",int(df.price.median()))
204 +cf={k.split("T.")[1].rstrip("]"):v for k,v in mA.params.items() if k.startswith("C(city_fe)")}
205 +top=pd.Series(cf).sort_values(ascending=False)
206 +print("TOP:",[(c,round((np.exp(b)-1)*100)) for c,b in top.head(5).items()])
207 +print("BOT:",[(c,round((np.exp(b)-1)*100)) for c,b in top.tail(3).items()])
208 +print("share_realtor %.1f"%((df.source=='realtor').mean()*100))
209 +print("Tables written to paper/tables/")
added scripts/16_make_tables_v5.py +124 −0
@@ -0,0 +1,124 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Régénère les tableaux LaTeX du papier en v5 (texte GPT + vision)."""
4 +import numpy as np, pandas as pd, statsmodels.formula.api as smf, warnings
5 +warnings.filterwarnings("ignore")
6 +# construire df via le pipeline de v5_ml (jusqu'avant la CV)
7 +exec(open("v5_ml.py").read().split("kf=KFold")[0])
8 +amen_=list(AMEN.keys())
9 +df["ln_price"]=np.log(df["price"])
10 +df["ln_sqft"]=np.log(df["sqft_final"].fillna(df.groupby("ptype")["sqft_final"].transform("median")).fillna(df["sqft_final"].median()))
11 +df["ln_lot"]=np.log(df["lot_sqft"].fillna(df["lot_sqft"].median()));df["has_lot"]=df["lot_sqft"].notna().astype(int)
12 +df["has_age"]=df["age"].notna().astype(int)
13 +for c in ["beds","baths","storeys","age","age2"]:
14 + df[c]=df[c].fillna(df.groupby("ptype")[c].transform("median")).fillna(df[c].median())
15 +df["parking"]=df["parking"].fillna(0)
16 +form=("ln_price ~ ln_sqft+ln_lot+has_lot+beds+baths+parking+storeys+age+age2+has_age+C(source)+C(ptype_fe)+"
17 + +"+".join(amen_)+"+n_images_c+virtual_tour+"+"+".join(rv)+"+"+"+".join(gv)+"+"+"+".join(vv)+"+C(city_fe)")
18 +mA=smf.ols(form,data=df[df["has_sqft"]==1]).fit(cov_type="HC3")
19 +mB=smf.ols(form.replace("ln_sqft+",""),data=df).fit(cov_type="HC3")
20 +print("A R2=%.4f adj=%.4f N=%d | B R2=%.4f adj=%.4f N=%d"%(mA.rsquared,mA.rsquared_adj,mA.nobs,mB.rsquared,mB.rsquared_adj,mB.nobs))
21 +
22 +def star(p): return "^{***}" if p<.01 else "^{**}" if p<.05 else "^{*}" if p<.1 else ""
23 +def cell(m,k):
24 + if k not in m.params.index: return "",""
25 + b=m.params[k];se=m.bse[k];p=m.pvalues[k];s="$-$" if b<0 else ""
26 + return f"{s}{abs(b):.3f}${star(p)}$",f"({se:.3f})"
27 +disp=[("ln_sqft",r"$\ln(\text{Living area})$"),("ln_lot",r"$\ln(\text{Lot area})$"),
28 +("beds","Bedrooms"),("baths","Bathrooms"),("parking","Parking"),("age","Age"),
29 +("__A__","\\textit{Amenities \\& text (LLM)}"),
30 +("central_air","Central A/C"),("transit","Public transit"),("waterfront","Waterfront"),
31 +("g_cond","Condition (LLM, ord.)"),("g_renovated_kitchen","Reno.\\ kitchen (LLM)"),
32 +("g_hardwood","Hardwood floor (LLM)"),("g_new_construction","New construction (LLM)"),
33 +("g_income_property","Income property (LLM)"),
34 +("__V__","\\textit{Vision (façade)}"),
35 +("v_curb","Curb appeal (1--10)"),("v_reno","Renovation level (ord.)"),
36 +("v_luxury","Luxury (visual)"),("v_garage_vis","Garage visible"),
37 +("v_mat_brique","Brick façade"),("v_mat_pierre","Stone façade"),
38 +("v_mat_vinyle","Vinyl façade"),("v_set_rural","Rural setting")]
39 +L=[]
40 +for k,lab in disp:
41 + if k in ("__A__","__V__"):
42 + L.append(f" \\multicolumn{{3}}{{@{{}}l}}{{{lab}}} \\\\[2pt]");continue
43 + a1,a2=cell(mA,k);b1,b2=cell(mB,k)
44 + L.append(f" {lab} & {a1} & {b1} \\\\")
45 + L.append(f" & {a2} & {b2} \\\\[3pt]")
46 +reg=r"""% Auto-generated v5
47 +\begin{table}[htbp]
48 + \centering
49 + \begin{threeparttable}
50 + \caption{Hedonic Regression Results (Text + Vision Features)}
51 + \label{tab:regression}
52 + \begin{tabular}{@{}lcc@{}}
53 + \toprule
54 + & (1) Model~A & (2) Model~B \\
55 + & \textit{rich} & \textit{broad} \\
56 + \midrule
57 +"""+"\n".join(L)+r"""
58 + \midrule
59 + City FE / Type FE & Yes & Yes \\
60 + Living area & Yes & No \\
61 + Observations & """+f"{int(mA.nobs):,}".replace(",","{,}")+r""" & """+f"{int(mB.nobs):,}".replace(",","{,}")+r""" \\
62 + $R^2$ & """+f"{mA.rsquared:.3f}"+r""" & """+f"{mB.rsquared:.3f}"+r""" \\
63 + Adjusted $R^2$ & """+f"{mA.rsquared_adj:.3f}"+r""" & """+f"{mB.rsquared_adj:.3f}"+r""" \\
64 + \bottomrule
65 + \end{tabular}
66 + \begin{tablenotes}[flushleft]
67 + \footnotesize
68 + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full
69 + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample);
70 + Model~B omits it (full sample). ``LLM'' variables are extracted from the
71 + description text; ``Vision'' variables from the façade photograph. Curb appeal and
72 + renovation level enter as scores; luxury, garage, and material/setting are
73 + indicators. All specifications include city and property-type fixed effects and the
74 + full amenity, room-detail, and listing-signal controls (only selected coefficients
75 + shown). HC3 standard errors in parentheses. $^{***}$, $^{**}$, $^{*}$: significance
76 + at 1\%, 5\%, 10\%.
77 + \end{tablenotes}
78 + \end{threeparttable}
79 +\end{table}
80 +"""
81 +open("paper/tables/regression.tex","w").write(reg)
82 +print("-> paper/tables/regression.tex (v5)")
83 +
84 +# ---- models.tex : deux panels (texte / +vision) avec chiffres CV ----
85 +models=r"""% Auto-generated v5
86 +\begin{table}[htbp]
87 + \centering
88 + \begin{threeparttable}
89 + \caption{Predictive Performance: From OLS to Deep Learning}
90 + \label{tab:models}
91 + \begin{tabular}{@{}lcccc@{}}
92 + \toprule
93 + Estimator & $R^2_{\text{OOS}}$ & Median AE & Within $\pm$15\% & Within $\pm$25\% \\
94 + \midrule
95 + \multicolumn{5}{@{}l}{\textit{Panel A: Structural + amenity + LLM-text features}} \\[3pt]
96 + OLS (linear hedonic) & 0.628 & 17.9\% & 43\% & 64\% \\
97 + Random Forest & 0.678 & 18.0\% & 43\% & 64\% \\
98 + Neural network (plain MLP) & 0.711 & 16.1\% & 47\% & 69\% \\
99 + Neural network (robust ResNet) & 0.741 & 15.2\% & 49\% & 70\% \\
100 + Gradient boosting (HistGBM) & 0.743 & 15.4\% & 49\% & 70\% \\
101 + Blend (NN + GBM) & 0.760 & 14.7\% & 51\% & 72\% \\[4pt]
102 + \multicolumn{5}{@{}l}{\textit{Panel B: \ + Vision (façade) features}} \\[3pt]
103 + Gradient boosting (HistGBM) & 0.770 & 14.5\% & 51\% & 71\% \\
104 + Neural network (robust ResNet) & 0.747 & 14.8\% & 51\% & 71\% \\
105 + \textbf{Blend (NN + GBM)} & \textbf{0.779} & \textbf{14.0\%} & \textbf{53\%} & \textbf{73\%} \\
106 + \bottomrule
107 + \end{tabular}
108 + \begin{tablenotes}[flushleft]
109 + \footnotesize
110 + \item \textit{Notes:} Out-of-sample performance from 5-fold cross-validation with a
111 + common seed (identical folds across rows). $R^2_{\text{OOS}}$ is on
112 + $\ln(\text{Price})$; ``Median AE'' is the median absolute percentage error in
113 + levels (Equation~\ref{eq:cvmetrics}). Panel~A uses structural, amenity, room-detail,
114 + listing-signal, and LLM text-derived features; Panel~B adds the twelve façade
115 + vision attributes. City, type, and source enter as one-hot indicators for the
116 + linear and tree models and as learned embeddings for the neural networks. The tree
117 + and neural models handle missing living area natively. The best model is the
118 + text\,+\,vision blend.
119 + \end{tablenotes}
120 + \end{threeparttable}
121 +\end{table}
122 +"""
123 +open("paper/tables/models.tex","w").write(models)
124 +print("-> paper/tables/models.tex (v5, 2 panels)")
added scripts/17_gen_regression_table.py +85 −0
@@ -0,0 +1,85 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +"""Régénère paper/tables/regression.tex à partir de revision_extras.json
4 + avec SE clusterisées par municipalité (et stars associées)."""
5 +import json
6 +
7 +R = json.load(open("results/revision_extras.json"))
8 +
9 +def star(p):
10 + return "^{***}" if p < .01 else "^{**}" if p < .05 else "^{*}" if p < .1 else ""
11 +
12 +def cell(model, k):
13 + c = R[model]["coef"].get(k)
14 + if not c:
15 + return "", ""
16 + b, se, p = c["b"], c["se_cl"], c["p_cl"]
17 + s = "$-$" if b < 0 else ""
18 + st = star(p)
19 + coef = f"{s}{abs(b):.3f}" + (f"${st}$" if st else "")
20 + return coef, f"({se:.3f})"
21 +
22 +disp = [("ln_sqft", r"$\ln(\text{Living area})$"), ("ln_lot", r"$\ln(\text{Lot area})$"),
23 + ("beds", "Bedrooms"), ("baths", "Bathrooms"), ("parking", "Parking"), ("age", "Age"),
24 + ("__A__", r"\textit{Amenities \& text (LLM)}"),
25 + ("central_air", "Central A/C"), ("transit", "Public transit"), ("waterfront", "Waterfront"),
26 + ("g_cond", "Condition (LLM, ord.)"), ("g_renovated_kitchen", r"Reno.\ kitchen (LLM)"),
27 + ("g_hardwood", "Hardwood floor (LLM)"), ("g_new_construction", "New construction (LLM)"),
28 + ("g_income_property", "Income property (LLM)"),
29 + ("__V__", r"\textit{Vision (façade)}"),
30 + ("v_curb", "Curb appeal (1--10)"), ("v_reno", "Renovation level (ord.)"),
31 + ("v_luxury", "Luxury (visual)"), ("v_garage_vis", "Garage visible"),
32 + ("v_mat_brique", "Brick façade"), ("v_mat_pierre", "Stone façade"),
33 + ("v_mat_vinyle", "Vinyl façade"), ("v_set_rural", "Rural setting")]
34 +
35 +L = []
36 +for k, lab in disp:
37 + if k in ("__A__", "__V__"):
38 + L.append(f" \\multicolumn{{3}}{{@{{}}l}}{{{lab}}} \\\\[2pt]")
39 + continue
40 + a1, a2 = cell("modelA", k)
41 + b1, b2 = cell("modelB", k)
42 + L.append(f" {lab} & {a1} & {b1} \\\\")
43 + L.append(f" & {a2} & {b2} \\\\[3pt]")
44 +
45 +nA, nB = R["modelA"]["n"], R["modelB"]["n"]
46 +tex = r"""% Auto-generated (revision: clustered SEs)
47 +\begin{table}[htbp]
48 + \centering
49 + \begin{threeparttable}
50 + \caption{Hedonic Regression Results (Text + Vision Features)}
51 + \label{tab:regression}
52 + \begin{tabular}{@{}lcc@{}}
53 + \toprule
54 + & (1) Model~A & (2) Model~B \\
55 + & \textit{rich} & \textit{broad} \\
56 + \midrule
57 +""" + "\n".join(L) + f"""
58 + \\midrule
59 + City FE / Type FE & Yes & Yes \\\\
60 + Living area & Yes & No \\\\
61 + Observations & {nA:,} & {nB:,} \\\\
62 + $R^2$ & {R['modelA']['r2']:.3f} & {R['modelB']['r2']:.3f} \\\\
63 + Adjusted $R^2$ & {R['modelA']['adj']:.3f} & {R['modelB']['adj']:.3f} \\\\
64 + \\bottomrule
65 + \\end{{tabular}}
66 +""".replace(f"{nA:,}", f"{nA:,}".replace(",", "{,}")).replace(f"{nB:,}", f"{nB:,}".replace(",", "{,}")) + r""" \begin{tablenotes}[flushleft]
67 + \footnotesize
68 + \item \textit{Notes:} OLS estimates of Equation~\eqref{eq:hedonic} with the full
69 + feature set. Model~A includes $\ln(\text{Living area})$ (measured-area subsample);
70 + Model~B omits it (full sample). ``LLM'' variables are extracted from the
71 + description text; ``Vision'' variables from the façade photograph. Curb appeal and
72 + renovation level enter as scores; luxury, garage, and material/setting are
73 + indicators. All specifications include city and property-type fixed effects and the
74 + full amenity, room-detail, and listing-signal controls (only selected coefficients
75 + shown). Standard errors clustered by municipality (195 clusters) in parentheses.
76 + Heteroskedasticity-robust (HC3) standard errors are uniformly smaller and yield
77 + the same significance pattern for the text- and vision-derived attributes, except
78 + that curb appeal in Model~B is significant under HC3 but not under clustering.
79 + $^{***}$, $^{**}$, $^{*}$: significance at 1\%, 5\%, 10\%.
80 + \end{tablenotes}
81 + \end{threeparttable}
82 +\end{table}
83 +"""
84 +open("paper/tables/regression.tex", "w").write(tex)
85 +print("-> paper/tables/regression.tex (clustered SEs)")
86