Working paper UQO WP 11 (LaTeX), script de calcul réparti, générateur de tables
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
122 changed files +9,870 −5
added
M3U96aomparables-validation/.gitignore
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +out/ | |
| 2 | +data/ | |
| 3 | +*.pyc | |
| 4 | +__pycache__/ | |
| 5 | +.DS_Store | |
added
M3U96aomparables-validation/README.md
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +# Validation empirique de la méthode des comparables (Vrai Prix) | |
| 2 | + | |
| 3 | +Projet de recherche : rétro-évaluation à grande échelle de la méthode de comparaison des ventes sur les transactions | |
| 4 | +résidentielles unifamiliales du Québec (Vrai Prix, 2021-01 → 2026-09). Protocole : `docs/PROTOCOLE.md`. | |
| 5 | +Rapport de résultats : `docs/RAPPORT.md` / `docs/RAPPORT.pdf`. | |
| 6 | + | |
| 7 | +## Organisation | |
| 8 | + | |
| 9 | +``` | |
| 10 | +pipeline/ 01_dataset → 02_index → 03_models → 04_backtest → 05_analyze → 06_report | |
| 11 | +docs/ PROTOCOLE.md, RAPPORT.md(.pdf), header.tex | |
| 12 | +out/<version>/ sales.parquet, index_asof.npy, models_*.json, est_E*.npy, targets.parquet, tables/, fig/, results.json | |
| 13 | +data/ (nœud seulement) vraiprix-20261010.db — instantané de la base de production Vrai Prix | |
| 14 | +ref/ MUN.csv (répertoire des municipalités MAMH : code → MRC → région) | |
| 15 | +``` | |
| 16 | + | |
| 17 | +## Exécution | |
| 18 | + | |
| 19 | +Le calcul se fait sur **M2U64** (`~/comparables-validation`, venv `~/uqo-eval-terrains/env`, Python 3.14 : pandas, numpy, scipy, | |
| 20 | +lightgbm, matplotlib). Le laptop garde le code, les docs et une copie des sorties. | |
| 21 | + | |
| 22 | +```bash | |
| 23 | +rsync -a --exclude out --exclude data --exclude .git ~/Desktop/Validation-Comparables/ M2U64:comparables-validation/ | |
| 24 | +ssh M2U64 'cd ~/comparables-validation && CV_VERSION=2026-10-10 ./run_all.sh' # ~ 30 min | |
| 25 | +./make_pdf.sh # rapatrie out/ + docs/ et compile le PDF | |
| 26 | +``` | |
| 27 | + | |
| 28 | +Variables : `CV_VERSION` (dossier de sortie), `CV_SAMPLE=3000` (échantillon de cibles pour un essai), `CV_PROCS` (processus, 20 par défaut). | |
| 29 | + | |
| 30 | +## Données | |
| 31 | + | |
| 32 | +Instantané `vraiprix.db` pris sur M3U96a (`~/apps/vrai-prix/data`) le 2026-10-10 : 766 596 transactions, 3,75 M d'unités du rôle. | |
| 33 | +Jeu propre : ventes `unifamilial` jumelées au rôle, règles R0-R11 journalisées dans `out/<v>/exclusions.csv`. | |
added
M3U96aomparables-validation/docs/PROTOCOLE.md
+101 −0
@@ -0,0 +1,101 @@ | ||
| 1 | +# Revisiter la méthode de comparaison des ventes : protocole de validation empirique à grande échelle | |
| 2 | + | |
| 3 | +**Titre de travail (article A)** — *Revisiting the Sales Comparison Approach: A Large-Scale Empirical Validation of Comparable Selection, Price Adjustments, and Value Reconciliation* | |
| 4 | + | |
| 5 | +**Variante** — *Does the Sales Comparison Approach Actually Work? Evidence from Large-Scale Residential Property Transactions* | |
| 6 | + | |
| 7 | +Données : transactions historiques **Vrai Prix** (registre foncier du Québec jumelé au rôle d'évaluation), instantané du 2026-10-10. | |
| 8 | + | |
| 9 | +## 1. Question de recherche | |
| 10 | + | |
| 11 | +La méthode des comparables, telle qu'enseignée et pratiquée par les évaluateurs agréés, produit-elle réellement les estimations les plus fiables ? Et surtout, quelles règles de **sélection**, d'**ajustement**, de **pondération** et de **réconciliation** des comparables sont empiriquement justifiées ? | |
| 12 | + | |
| 13 | +| Dimension | Question testée | Où dans le pipeline | | |
| 14 | +|---|---|---| | |
| 15 | +| Sélection | Quels comparables sont réellement les plus pertinents (proximité, similarité physique, « moins ajustés ») ? | E1 (facteur `sel`), E2 | | |
| 16 | +| Nombre | Faut-il 1, 3, 5, 10, 20 ou 50 ventes ? | E1 (facteur `k`) | | |
| 17 | +| Ajustements | Quels ajustements sont statistiquement justifiés (grille conventionnelle, hédonique, locale, non linéaire, ratio au rôle) ? | E2 (facteur `adj`) | | |
| 18 | +| Pondération | Égalitaire, distance, récence, similarité ? | E2 (facteur `w`) | | |
| 19 | +| Réconciliation | Médiane, moyenne, moyenne pondérée, combinaison optimisée (M9) ? | E2 (facteur `recon`), M9 | | |
| 20 | +| Fiabilité | Dans quels marchés (densité, région, segment de prix, hétérogénéité) la méthode tient-elle ou défaille-t-elle ? | strates, H2-H5 | | |
| 21 | + | |
| 22 | +## 2. Protocole expérimental (rétro-évaluation « point dans le temps ») | |
| 23 | + | |
| 24 | +Pour chaque transaction cible : | |
| 25 | + | |
| 26 | +1. **Masquage** du prix de vente réel. | |
| 27 | +2. **Reconstitution du marché à la date d'évaluation** : seules les ventes conclues *strictement avant* la date de la cible sont admissibles (le bien lui-même, s'il a déjà été vendu, est exclu du bassin : on évalue par comparables, pas par vente répétée). L'indice temporel, les coefficients hédoniques et les modèles d'apprentissage sont ré-estimés avec les seules ventes antérieures à la période d'évaluation (indice : ré-estimé chaque mois avec toutes les ventes des mois précédents ; modèles : estimés sur les ventes < 2024-01 pour la validation 2024 et < 2025-01 pour le test 2025-26). | |
| 28 | +3. **Application** de chaque configuration (sélection × rayon × fenêtre × nombre × ajustement × pondération × réconciliation) sur exactement les mêmes cibles. | |
| 29 | +4. **Comparaison** avec le prix réel : erreur absolue en pourcentage, biais, part des estimations à ±5/10/20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur), dispersion des prix ajustés (intervalle de valeur). | |
| 30 | + | |
| 31 | +### Séparation chronologique | |
| 32 | + | |
| 33 | +| Période | Rôle | Ventes | | |
| 34 | +|---|---|---| | |
| 35 | +| 2021-01 → 2023-12 | développement (estimation des grilles d'ajustement et des modèles) | ~ 200 000 | | |
| 36 | +| 2024 | validation : choix des configurations et des hyperparamètres | ~ 65 000 cibles | | |
| 37 | +| 2025-01 → 2026-09 | évaluation finale hors échantillon | ~ 125 000 cibles | | |
| 38 | + | |
| 39 | +Les comparables d'une cible de 2025 peuvent être des ventes de 2024 ou de 2025 antérieures à sa date : c'est l'information dont disposait un évaluateur ce jour-là (nous ignorons le délai de publication au registre foncier, ce qui avantage légèrement toutes les méthodes de la même façon). | |
| 40 | + | |
| 41 | +## 3. Expériences | |
| 42 | + | |
| 43 | +### E1 — surface empirique de performance Erreur(k, d, t) | |
| 44 | + | |
| 45 | +- k ∈ {1, 3, 5, 7, 10, 15, 20, 30, 50} comparables ; | |
| 46 | +- d ∈ {0,5 ; 1 ; 2 ; 5 ; 10 ; 25} km (rayon maximal) ; | |
| 47 | +- t ∈ {3, 6, 12, 24} mois (ancienneté maximale) ; | |
| 48 | +- sélection ∈ {`geo` les plus proches, `sim` les plus semblables physiquement, `mix` compromis} ; | |
| 49 | +- ajustement ∈ {`none` temps seulement, `hedo` grille hédonique} ; réconciliation = médiane. | |
| 50 | + | |
| 51 | +1 296 configurations × toutes les cibles. | |
| 52 | + | |
| 53 | +### E2 — plan factoriel des règles (trois ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois)) | |
| 54 | + | |
| 55 | +- sélection ∈ {`geo`, `sim`, `mix`, `adj` = comparables nécessitant le moins d'ajustements} ; | |
| 56 | +- ajustement ∈ {`raw` aucun, `none` temps seul, `conv` grille conventionnelle stylisée, `hedo` grille hédonique globale (within-municipalité), `hedo_reg` grille régionale rétrécie, `local` régression sur le voisinage (≤ 10 km, ≥ 40 ventes, ridge vers la grille régionale), `gbm` ajustements non linéaires (LightGBM), `role` ratio prix/rôle (comparables de la même municipalité et du même rôle)} ; | |
| 57 | +- pondération ∈ {égale, 1/(0,25 + d), exp(−mois/6), 1/(0,1 + ajustement brut), produit des trois} ; | |
| 58 | +- réconciliation ∈ {médiane (pondérée), moyenne (pondérée) en dollars}. | |
| 59 | + | |
| 60 | +960 configurations. | |
| 61 | + | |
| 62 | +### Compétition de modèles | |
| 63 | + | |
| 64 | +| Modèle | Définition opérationnelle | | |
| 65 | +|---|---| | |
| 66 | +| M1 | médiane des 3 ventes les plus proches (≤ 2 km, ≤ 6 mois), temps seul | | |
| 67 | +| M2 | moyenne des 5 plus proches (≤ 5 km, ≤ 12 mois), temps seul | | |
| 68 | +| M3 | 5 comparables choisis par similarité physique | | |
| 69 | +| M4 | 5 plus proches pondérés par la distance | | |
| 70 | +| M5 | 5 plus proches pondérés par la récence | | |
| 71 | +| M6 | grille d'ajustement conventionnelle stylisée (½ du prix au m² pour l'aire, ½ de la valeur foncière au m² pour le terrain plafonnée à 15 %, −0,5 %/an d'âge plafonné à 20 %, ±5 % détaché/attaché) | | |
| 72 | +| M7 | grille d'ajustement hédonique (coefficients within-municipalité estimés sur les ventes antérieures) | | |
| 73 | +| M8a / M8b | grille locale (régression sur le voisinage) / ajustements non linéaires (GBM) | | |
| 74 | +| M9 | réconciliation optimisée : combinaison convexe (NNLS) de sept estimations par comparables, poids appris en 2024 | | |
| 75 | +| M10 | modèle hédonique global direct (β within + effets région, MRC, municipalité, unité de voisinage) | | |
| 76 | +| M11a / M11b | gradient boosting sans / avec la valeur au rôle | | |
| 77 | +| M12 | hybride : combinaison convexe de M11b et des estimations par comparables ajustées (GBM, hédonique) | | |
| 78 | +| M13 | ratio au rôle : rôle de la cible × ratio prix/rôle médian des 5 plus proches du même rôle | | |
| 79 | +| M0 | meilleure configuration de E1 ∪ E2 choisie sur la validation 2024, évaluée en 2025-26 | | |
| 80 | + | |
| 81 | +### Hypothèses | |
| 82 | + | |
| 83 | +- **H1** — un comparable très proche mais peu semblable peut être moins informatif qu'un comparable plus éloigné mais très semblable. | |
| 84 | +- **H2** — l'ajout de comparables améliore la précision jusqu'à un seuil, au-delà duquel l'hétérogénéité domine ; le seuil dépend de la densité. | |
| 85 | +- **H3** — les ajustements empiriques locaux surpassent les grilles fixes mais deviennent instables dans les petits échantillons. | |
| 86 | +- **H4** — la pondération optimale dépend de la liquidité du sous-marché. | |
| 87 | +- **H5** — les méthodes simples et transparentes rivalisent avec l'apprentissage automatique dans les marchés homogènes. | |
| 88 | + | |
| 89 | +### Inférence | |
| 90 | + | |
| 91 | +Différences d'erreurs appariées (mêmes cibles) avec intervalles de confiance par **bootstrap par grappes** (municipalité × trimestre), qui respecte les dépendances spatiales et temporelles. | |
| 92 | + | |
| 93 | +## 4. Ce que les données permettent — et ne permettent pas | |
| 94 | + | |
| 95 | +Caractéristiques observées pour chaque vente (registre foncier ⋈ rôle) : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable (étages), superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle (en vigueur à la vente) et sa part terrain. **Non observés** : garage, piscine, nombre de chambres et de salles de bains, qualité et état, rénovations. Les ajustements correspondants ne peuvent donc pas être testés ici ; les erreurs résiduelles les contiennent. | |
| 96 | + | |
| 97 | +Nous validons la **précision prédictive** de règles professionnelles *simulées*, pas le jugement réel d'un évaluateur (il faudrait ses dossiers, ses comparables et ses ajustements). La distinction est maintenue partout dans le rapport. | |
| 98 | + | |
| 99 | +## 5. Article B (suite envisagée) | |
| 100 | + | |
| 101 | +Vers une méthode des comparables optimisée empiriquement : algorithme transparent de sélection/ajustement entraîné sur les ventes historiques et validé sur les ventes futures ; règles conditionnelles au type de bien, à la densité et à la qualité des données ; intégration dans UQO Éval pour montrer aux étudiants la comparaison entre règles traditionnelles et règles estimées. | |
added
M3U96aomparables-validation/docs/header.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% En-tête LaTeX — Grille empirique d'ajustement des terrains (UQO Éval) | |
| 2 | +\usepackage{fontspec} | |
| 3 | +\setmainfont{Helvetica Neue}[Ligatures=TeX] | |
| 4 | +\setsansfont{Helvetica Neue} | |
| 5 | +\setmonofont{Menlo}[Scale=0.82] | |
| 6 | +\usepackage{newunicodechar} | |
| 7 | +\newunicodechar{→}{\ensuremath{\rightarrow}}\newunicodechar{≥}{\ensuremath{\geq}}\newunicodechar{≤}{\ensuremath{\leq}}\newunicodechar{×}{\ensuremath{\times}}\newunicodechar{−}{\ensuremath{-}}\newunicodechar{≈}{\ensuremath{\approx}}\newunicodechar{∂}{\ensuremath{\partial}}\newunicodechar{⋈}{\ensuremath{\bowtie}}\newunicodechar{Δ}{\ensuremath{\Delta}}\newunicodechar{τ}{\ensuremath{\tau}}\newunicodechar{σ}{\ensuremath{\sigma}}\newunicodechar{β}{\ensuremath{\beta}}\newunicodechar{θ}{\ensuremath{\theta}}\newunicodechar{γ}{\ensuremath{\gamma}}\newunicodechar{α}{\ensuremath{\alpha}}\newunicodechar{δ}{\ensuremath{\delta}}\newunicodechar{ε}{\ensuremath{\varepsilon}}\newunicodechar{²}{\textsuperscript{2}}\newunicodechar{⇒}{\ensuremath{\Rightarrow}}\newunicodechar{▶}{\ensuremath{\blacktriangleright}}\newunicodechar{│}{|}\newunicodechar{▼}{\ensuremath{\blacktriangledown}}\newunicodechar{─}{-}\newunicodechar{┌}{+}\newunicodechar{┐}{+}\newunicodechar{└}{+}\newunicodechar{┘}{+}\newunicodechar{├}{+}\newunicodechar{┤}{+} | |
| 8 | +\usepackage{xcolor} | |
| 9 | +\definecolor{uqo}{HTML}{006085}\definecolor{turq}{HTML}{2BCCAE}\definecolor{corail}{HTML}{EA6047}\definecolor{encre}{HTML}{16181B}\definecolor{papier}{HTML}{F4F5F3} | |
| 10 | +\usepackage{titlesec} | |
| 11 | +\titleformat{\section}{\Large\bfseries\color{uqo}}{}{0em}{}[{\color{uqo}\titlerule[0.8pt]}] | |
| 12 | +\titleformat{\subsection}{\large\bfseries\color{uqo}}{}{0em}{} | |
| 13 | +\titlespacing*{\section}{0pt}{18pt plus 4pt}{8pt} | |
| 14 | +\usepackage{etoolbox} | |
| 15 | +\AtBeginEnvironment{longtable}{\scriptsize\renewcommand{\arraystretch}{1.18}} | |
| 16 | +\usepackage{colortbl}\arrayrulecolor{uqo!35} | |
| 17 | +\usepackage{tcolorbox} | |
| 18 | +\renewenvironment{quote}{\begin{tcolorbox}[colback=papier,colframe=turq,boxrule=0pt,leftrule=3pt,arc=1pt,left=6pt,right=6pt,top=4pt,bottom=4pt]}{\end{tcolorbox}} | |
| 19 | +\usepackage{fancyhdr} | |
| 20 | +\pagestyle{fancy}\fancyhf{} | |
| 21 | +\fancyhead[L]{\footnotesize\color{gray}Grille empirique d'ajustement des terrains — UQO Éval}\fancyhead[R]{\footnotesize\color{gray}IMM1003 · IMM1033} | |
| 22 | +\fancyfoot[C]{\footnotesize\color{gray}\thepage}\renewcommand{\headrulewidth}{0.3pt} | |
| 23 | +\usepackage{caption}\captionsetup{font=small,labelfont={bf,color=uqo}} | |
| 24 | +\usepackage{float}\floatplacement{figure}{H} | |
| 25 | +\setlength{\emergencystretch}{3em}\hyphenpenalty=300\tolerance=1500 | |
| 26 | +\usepackage{microtype} | |
| 27 | +\makeatletter\renewcommand{\maketitle}{\begin{center}{\color{uqo}\Huge\bfseries\@title\par}\vspace{8pt}{\normalsize\color{gray}\@author\par\@date\par}\end{center}\vspace{10pt}}\makeatother | |
| 28 | +\usepackage{polyglossia}\setdefaultlanguage[frenchpart=false]{french} | |
added
M3U96aomparables-validation/make_pdf.sh
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Rapatrie les résultats depuis M2U64 et produit docs/RAPPORT.pdf (pandoc → XeLaTeX) | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +V=${CV_VERSION:-2026-10-10} | |
| 6 | +mkdir -p out/$V | |
| 7 | +rsync -az --exclude 'est_E*.npy' --exclude 'sd_E2.npy' --exclude '*_pred_*.npy' --exclude 'sales.parquet' M2U64:comparables-validation/out/$V/ out/$V/ | |
| 8 | +rsync -az M2U64:comparables-validation/docs/ docs/ 2>/dev/null || true | |
| 9 | +cd docs | |
| 10 | +pandoc RAPPORT.md -o RAPPORT.pdf --pdf-engine=xelatex -H header.tex --toc --toc-depth=2 \ | |
| 11 | + -V geometry:"a4paper,margin=17mm,top=20mm,bottom=20mm" -V fontsize=10pt -V linkcolor=uqo -V urlcolor=uqo -V toccolor=uqo -V colorlinks=true \ | |
| 12 | + -V toc-title="Table des matières" --resource-path=.:../out/$V/fig 2>&1 | grep -v "Missing character\|Overfull\|Underfull" | tail -8 || true | |
| 13 | +echo "PDF : $(pwd)/RAPPORT.pdf ($(du -h RAPPORT.pdf | cut -f1))" | |
added
M3U96aomparables-validation/pipeline/01_dataset.py
+97 −0
@@ -0,0 +1,97 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""01 — Jeu de données : transactions unifamiliales Vrai Prix ⋈ rôle, nettoyage journalisé, | |
| 3 | +reventes, géographie, périodes. Sorties : sales.parquet, exclusions.csv, dataset_stats.json.""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +con = connect() | |
| 7 | +log(f"lecture {DB}") | |
| 8 | +sql = """ | |
| 9 | +select t.id, t.date, t.amount, t.lat, t.lng, t.city, t.property_type, t.year_built, t.floor_area, | |
| 10 | + t.building_type, t.id_provinc, t.valeur_role as role_vente, t.land_area, | |
| 11 | + u.code_mun, u.municipalite, u.type_prop as u_type, u.annee_construction, u.aire_etages_m2, | |
| 12 | + u.superficie_terrain_m2, u.front_terrain_m, u.nb_etages, u.nb_logements, u.lien_physique, | |
| 13 | + u.genre_construction, u.unite_voisinage, u.dat_cond_marche, u.valeur_terrain, u.valeur_batiment, | |
| 14 | + u.valeur_role as role_actuel, u.valeur_anterieure as role_anterieur, u.cubf | |
| 15 | +from transactions t left join units u on u.id_provinc = t.id_provinc | |
| 16 | +where t.property_type = 'unifamilial' | |
| 17 | +""" | |
| 18 | +df = pd.read_sql(sql, con) | |
| 19 | +log(f"{len(df):,} transactions unifamiliales brutes") | |
| 20 | +n0 = len(df) | |
| 21 | +excl = [] # (règle, n) | |
| 22 | +def drop(mask, rule): | |
| 23 | + global df | |
| 24 | + n = int(mask.sum()); excl.append((rule, n)); df = df[~mask].copy(); log(f" {rule}: -{n:,} → {len(df):,}") | |
| 25 | + | |
| 26 | +df["date"] = pd.to_datetime(df["date"]) | |
| 27 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 28 | +drop(df["id_provinc"].isna() | df["code_mun"].isna(), "R0 sans jumelage au rôle (id_provinc / code_mun)") | |
| 29 | +drop(df.duplicated(["id_provinc", "date"]), "R1 doublons (même unité, même date)") | |
| 30 | +drop(~df["u_type"].isin(["unifamilial"]) | (df["nb_logements"].fillna(1) > 1), "R2 unité du rôle non unifamiliale (chalet, plex, terrain…) ou > 1 logement") | |
| 31 | +drop(df["lat"].isna() | df["lng"].isna() | (df["lat"] < 44.5) | (df["lat"] > 63) | (df["lng"] > -56) | (df["lng"] < -80), "R3 coordonnées absentes ou hors Québec") | |
| 32 | +fa = df["floor_area"].fillna(df["aire_etages_m2"]) | |
| 33 | +df["floor_area"] = fa | |
| 34 | +drop(fa.isna() | (fa < 40) | (fa > 600), "R4 aire habitable absente ou hors [40, 600] m²") | |
| 35 | +la = df["land_area"].fillna(df["superficie_terrain_m2"]) | |
| 36 | +df["land_area"] = la | |
| 37 | +drop(la.isna() | (la < 80) | (la > 20000), "R5 superficie de terrain absente ou hors [80, 20 000] m²") | |
| 38 | +yb = df["year_built"].fillna(df["annee_construction"]) | |
| 39 | +df["year_built"] = yb | |
| 40 | +drop(yb.isna() | (yb < 1850) | (yb > df["date"].dt.year), "R6 année de construction absente, < 1850 ou postérieure à la vente (vente de terrain)") | |
| 41 | +drop((df["year_built"] == df["date"].dt.year), "R6b vente l'année même de la construction (prix du terrain ou d'une construction inachevée : médiane 346 k$ et ratio au rôle 1,15 contre 450-480 k$ et 1,3 aux âges 1-4)") | |
| 42 | +drop((df["amount"] < 50000) | (df["amount"] > 3_000_000), "R7 prix hors [50 k$, 3 M$]") | |
| 43 | +ratio = df["amount"] / df["role_vente"] | |
| 44 | +drop(df["role_vente"].isna() | (df["role_vente"] <= 0) | (ratio < 0.33) | (ratio > 3.0), "R8 rôle absent ou ratio prix/rôle hors [0,33 ; 3] (transactions présumées non représentatives)") | |
| 45 | +# reventes rapides avec forte variation (flips / rénovations : les caractéristiques enregistrées ne décrivent plus le bien) | |
| 46 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 47 | +g = df.groupby("id_provinc") | |
| 48 | +prev_date = g["date"].shift(1); prev_amt = g["amount"].shift(1) | |
| 49 | +dd = (df["date"] - prev_date).dt.days | |
| 50 | +chg = df["amount"] / prev_amt - 1 | |
| 51 | +flip = (dd <= 180) & (chg.abs() > 0.40) | |
| 52 | +drop(flip.fillna(False), "R9 revente ≤ 180 jours avec variation > 40 % (flip)") | |
| 53 | +drop(df["lien_physique"].isna() | df["genre_construction"].isna(), "R10 lien physique ou genre de construction absent au rôle") | |
| 54 | + | |
| 55 | +# ----------------------------------------------------------------- variables dérivées | |
| 56 | +rep = load_repertoire() | |
| 57 | +df = df.merge(rep, on="code_mun", how="left") | |
| 58 | +drop(df["region"].isna(), "R11 municipalité absente du répertoire MAMH") | |
| 59 | +df["sale_year"] = df["date"].dt.year | |
| 60 | +df["month"] = month_index(df["date"]) | |
| 61 | +df["day"] = (df["date"] - pd.Timestamp("2021-01-01")).dt.days.astype(int) | |
| 62 | +df["age"] = (df["sale_year"] - df["year_built"]).clip(lower=0) | |
| 63 | +df["ln_price"] = np.log(df["amount"]) | |
| 64 | +df["ln_floor"] = np.log(df["floor_area"]) | |
| 65 | +df["ln_land"] = np.log(df["land_area"]) | |
| 66 | +df["lien"] = df["lien_physique"].map(LIEN).fillna(0).astype(int) | |
| 67 | +df["genre"] = df["genre_construction"].map(GENRE).fillna(0).astype(int) | |
| 68 | +df["btype"] = df["building_type"].map(BTYPE).fillna(-1).astype(int) | |
| 69 | +df["etages"] = df["nb_etages"].fillna(1).clip(1, 3) | |
| 70 | +df["x_km"], df["y_km"] = to_xy_km(df["lat"], df["lng"]) | |
| 71 | +# millésime du rôle au moment de la vente (rôle courant / antérieur / autre) | |
| 72 | +df["role_vintage"] = np.where(np.isclose(df["role_vente"], df["role_actuel"].fillna(-1)), 2, | |
| 73 | + np.where(np.isclose(df["role_vente"], df["role_anterieur"].fillna(-1)), 1, 0)) | |
| 74 | +df["ratio_role"] = df["amount"] / df["role_vente"] | |
| 75 | +df["part_terrain"] = (df["valeur_terrain"] / df["role_actuel"]).clip(0.02, 0.95).fillna(0.2) | |
| 76 | +df["neuf"] = (df["age"] <= 3).astype(int) | |
| 77 | +df["period"] = np.select([df["date"] <= TRAIN_END, df["date"] <= VAL_END], ["train", "val"], "test") | |
| 78 | +df = df.sort_values("date").reset_index(drop=True) | |
| 79 | +df["idx"] = np.arange(len(df), dtype=np.int32) | |
| 80 | + | |
| 81 | +keep = ["idx", "id", "id_provinc", "date", "day", "month", "sale_year", "period", "amount", "ln_price", | |
| 82 | + "lat", "lng", "x_km", "y_km", "code_mun", "mun_nom", "mrc", "mrc_nom", "region", "region_nom", | |
| 83 | + "unite_voisinage", "floor_area", "ln_floor", "land_area", "ln_land", "year_built", "age", "neuf", | |
| 84 | + "lien", "genre", "btype", "etages", "role_vente", "role_vintage", "ratio_role", "part_terrain"] | |
| 85 | +df = df[keep] | |
| 86 | +save_parquet(df, "sales.parquet") | |
| 87 | +pd.DataFrame(excl, columns=["regle", "n"]).to_csv(OUT / "exclusions.csv", index=False) | |
| 88 | +stats = { | |
| 89 | + "brut": n0, "propre": len(df), "exclusions": excl, | |
| 90 | + "par_periode": df["period"].value_counts().to_dict(), | |
| 91 | + "par_annee": df["sale_year"].value_counts().sort_index().to_dict(), | |
| 92 | + "par_region": df.groupby("region_nom").size().sort_values(ascending=False).to_dict(), | |
| 93 | + "prix": {k: float(v) for k, v in df["amount"].describe().items()}, | |
| 94 | + "dates": [str(df["date"].min().date()), str(df["date"].max().date())], | |
| 95 | +} | |
| 96 | +save_json(stats, "dataset_stats.json") | |
| 97 | +log(f"jeu propre : {len(df):,} ventes ({stats['par_periode']})") | |
added
M3U96aomparables-validation/pipeline/02_index.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""02 — Indice temporel régional « point dans le temps ». | |
| 3 | +Pour chaque mois d'évaluation m, on ré-estime un indice hédonique mensuel par région administrative | |
| 4 | +avec toutes les ventes antérieures au mois m (fenêtre expansive) (jamais le mois m lui-même). Niveau de référence = m-1, | |
| 5 | +dernier mois entièrement connu. Sortie : index_asof.npy [m_eval, région, mois] (log), regions.json, | |
| 6 | +et un indice « plein échantillon » descriptif (index_full.csv).""" | |
| 7 | +from common import * | |
| 8 | + | |
| 9 | +df = load_parquet("sales.parquet") | |
| 10 | +regions = sorted(df["region"].unique().tolist()) | |
| 11 | +rid = {r: i for i, r in enumerate(regions)} | |
| 12 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 13 | +M = int(df["month"].max()) + 1 | |
| 14 | +M_EVAL0 = int(month_index(pd.Series([VAL_START]))[0]) | |
| 15 | +WINDOW = 999 # fenêtre « expansive » : toutes les ventes antérieures au mois m | |
| 16 | +SHRINK = 40.0 # n/(n+SHRINK) : rétrécissement de l'effet mensuel régional vers l'effet provincial | |
| 17 | + | |
| 18 | +X_cols = ["ln_floor", "ln_land", "age"] | |
| 19 | +def design(d): | |
| 20 | + X = np.column_stack([np.ones(len(d)), d["ln_floor"], d["ln_land"], d["age"], d["age"] ** 2 / 100, | |
| 21 | + (d["lien"] == 1), (d["lien"] >= 2), (d["genre"] == 1), (d["genre"] == 2), (d["genre"] == 3), | |
| 22 | + d["etages"], d["neuf"]]).astype(float) | |
| 23 | + return X | |
| 24 | + | |
| 25 | +def fit_month_effects(d, months): | |
| 26 | + """OLS à effets fixes de municipalité (estimateur within) : ln P ~ X + effets mensuels (base = dernier mois), | |
| 27 | + toutes les variables étant centrées par municipalité pour que l'indice ne soit pas contaminé par les | |
| 28 | + déplacements du mélange des ventes entre municipalités chères et bon marché. Renvoie effets et n par mois.""" | |
| 29 | + X = design(d)[:, 1:] | |
| 30 | + mm = d["month"].to_numpy() | |
| 31 | + base = months[-1] | |
| 32 | + dums = np.column_stack([(mm == m).astype(float) for m in months if m != base]) | |
| 33 | + A = np.hstack([X, dums]); y = d["ln_price"].to_numpy(float) | |
| 34 | + codes, inv = np.unique(d["code_mun"].to_numpy(), return_inverse=True) | |
| 35 | + cnt = np.bincount(inv).astype(float) | |
| 36 | + mA = np.zeros((len(codes), A.shape[1])); np.add.at(mA, inv, A); A = A - mA[inv] / cnt[inv, None] | |
| 37 | + my = np.bincount(inv, y) / cnt; y = y - my[inv] | |
| 38 | + beta, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 39 | + eff = {base: 0.0} | |
| 40 | + k = X.shape[1] | |
| 41 | + for j, m in enumerate([m for m in months if m != base]): | |
| 42 | + eff[m] = float(beta[k + j]) | |
| 43 | + n = {m: int((mm == m).sum()) for m in months} | |
| 44 | + return eff, n | |
| 45 | + | |
| 46 | +idx = np.full((M - M_EVAL0, len(regions), M), np.nan, dtype=np.float32) | |
| 47 | +log(f"indice point-dans-le-temps : mois d'évaluation {M_EVAL0}..{M-1}, {len(regions)} régions") | |
| 48 | +for m in range(M_EVAL0, M): | |
| 49 | + months = list(range(max(0, m - WINDOW), m)) | |
| 50 | + win = df[(df["month"] >= months[0]) & (df["month"] < m)] | |
| 51 | + prov_eff, _ = fit_month_effects(win, months) | |
| 52 | + for r in regions: | |
| 53 | + dr = win[win["region"] == r] | |
| 54 | + if len(dr) < 200: | |
| 55 | + for mo in months: idx[m - M_EVAL0, rid[r], mo] = prov_eff[mo] | |
| 56 | + continue | |
| 57 | + eff, n = fit_month_effects(dr, months) | |
| 58 | + for mo in months: | |
| 59 | + w = n[mo] / (n[mo] + SHRINK) | |
| 60 | + idx[m - M_EVAL0, rid[r], mo] = w * eff[mo] + (1 - w) * prov_eff[mo] | |
| 61 | + if m % 6 == 0: log(f" mois {m} ({len(win):,} ventes dans la fenêtre)") | |
| 62 | +np.save(OUT / "index_asof.npy", idx) | |
| 63 | +save_json({"regions": regions, "M_EVAL0": M_EVAL0, "M": M, "window": WINDOW, "shrink": SHRINK}, "index_meta.json") | |
| 64 | + | |
| 65 | +# indice descriptif plein échantillon (pour le rapport seulement — pas utilisé dans la rétro-évaluation) | |
| 66 | +rows = [] | |
| 67 | +months = list(range(0, M)) | |
| 68 | +eff_p, n_p = fit_month_effects(df, months) | |
| 69 | +for mo in months: | |
| 70 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": "QC", "log_idx": eff_p[mo] - eff_p[months[0]], "n": n_p[mo]}) | |
| 71 | +for r in regions: | |
| 72 | + dr = df[df["region"] == r] | |
| 73 | + if len(dr) < 2000: continue | |
| 74 | + eff, n = fit_month_effects(dr, months) | |
| 75 | + for mo in months: | |
| 76 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": r, "log_idx": eff[mo] - eff[months[0]], "n": n[mo]}) | |
| 77 | +pd.DataFrame(rows).to_csv(OUT / "index_full.csv", index=False) | |
| 78 | +log("indice terminé") | |
added
M3U96aomparables-validation/pipeline/03_models.py
+119 −0
@@ -0,0 +1,119 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""03 — Modèles d'ajustement et modèles de référence, ré-estimés chaque trimestre avec les seules ventes | |
| 3 | +antérieures au trimestre (as-of 2024-01, 2024-04, …, 2026-07) — comme le ferait un AVM ré-entraîné régulièrement : | |
| 4 | + • grille hédonique globale (coefficients) et régionale rétrécie → ajustements B / C | |
| 5 | + • LightGBM (objectif L1 = médiane) sur le prix ajusté dans le temps (avec et sans rôle) → ajustements C-GBM, M11 | |
| 6 | + • hédonique globale directe avec effets de municipalité → M10 | |
| 7 | +Sorties : models_<période>.json, gbm_pred_<période>.npy (prédiction f pour TOUTES les ventes).""" | |
| 8 | +import pickle | |
| 9 | +from common import * | |
| 10 | +import lightgbm as lgb | |
| 11 | + | |
| 12 | +df = load_parquet("sales.parquet") | |
| 13 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 14 | +rid = {r: i for i, r in enumerate(regions)} | |
| 15 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 16 | +idx = np.load(OUT / "index_asof.npy") | |
| 17 | + | |
| 18 | +FEATS = ["ln_floor", "ln_land", "age", "age2", "lien1", "lien2", "genre1", "genre2", "genre3", "etages", "neuf"] | |
| 19 | +def feats(d): | |
| 20 | + return pd.DataFrame({ | |
| 21 | + "ln_floor": d["ln_floor"], "ln_land": d["ln_land"], "age": d["age"], "age2": d["age"] ** 2 / 100, | |
| 22 | + "lien1": (d["lien"] == 1).astype(float), "lien2": (d["lien"] >= 2).astype(float), | |
| 23 | + "genre1": (d["genre"] == 1).astype(float), "genre2": (d["genre"] == 2).astype(float), "genre3": (d["genre"] == 3).astype(float), | |
| 24 | + "etages": d["etages"].astype(float), "neuf": d["neuf"].astype(float)}, index=d.index) | |
| 25 | + | |
| 26 | +def time_adjusted(d, m_asof): | |
| 27 | + """ln P ramené au niveau du mois m_asof-1 avec l'indice connu au mois m_asof.""" | |
| 28 | + lvl = idx[m_asof - M_EVAL0] | |
| 29 | + return d["ln_price"].to_numpy() - lvl[d["rid"].to_numpy(), d["month"].to_numpy()] + lvl[d["rid"].to_numpy(), m_asof - 1] | |
| 30 | + | |
| 31 | +def ols(X, y): | |
| 32 | + A = np.column_stack([np.ones(len(X)), X]) | |
| 33 | + b, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 34 | + return b | |
| 35 | + | |
| 36 | +ASOF = list(range(M_EVAL0, int(df["month"].max()) + 1, 3)) # ré-estimation trimestrielle (2024-01, 2024-04, …) | |
| 37 | +save_json({"asof": ASOF}, "asof.json") | |
| 38 | +TODO = [int(a) for a in os.environ["CV_ASOF"].split(",")] if os.environ.get("CV_ASOF") else ASOF # sous-ensemble (répartition sur plusieurs nœuds) | |
| 39 | +for m_asof in TODO: | |
| 40 | + period = str(m_asof); start = f"{2021 + m_asof // 12}-{m_asof % 12 + 1:02d}-01" | |
| 41 | + tr = df[df["month"] < m_asof].copy() | |
| 42 | + y = time_adjusted(tr, m_asof) | |
| 43 | + F = feats(tr) | |
| 44 | + log(f"[{period}] entraînement sur {len(tr):,} ventes < {start} (indice connu au mois {m_asof})") | |
| 45 | + # --- grille hédonique globale : estimateur « within » (effets fixes de municipalité), pour que les β | |
| 46 | + # mesurent des différences entre biens voisins et non entre marchés | |
| 47 | + def within(frame, cols, by): | |
| 48 | + return frame[cols] - frame.groupby(by)[cols].transform("mean") | |
| 49 | + # effets mensuels résiduels (base = dernier mois connu m_asof-1) : l'indice régional ne retire pas toute la | |
| 50 | + # tendance commune ; sans eux, les modèles globaux sous-estiment les ventes récentes | |
| 51 | + months_tr = sorted(tr["month"].unique()); MCOLS = [f"m{m}" for m in months_tr if m != m_asof - 1] | |
| 52 | + Fm = pd.concat([F] + [pd.Series((tr["month"] == m).astype(float), index=tr.index, name=f"m{m}") for m in months_tr if m != m_asof - 1], axis=1) | |
| 53 | + Fw = within(pd.concat([Fm, tr[["code_mun"]]], axis=1), FEATS + MCOLS, "code_mun") | |
| 54 | + yw = y - pd.Series(y, index=tr.index).groupby(tr["code_mun"]).transform("mean").to_numpy() | |
| 55 | + bw_all, *_ = np.linalg.lstsq(Fw.to_numpy(), yw, rcond=None) | |
| 56 | + bw = bw_all[:len(FEATS)]; bm = bw_all[len(FEATS):] | |
| 57 | + beta_global = dict(zip(FEATS, bw.tolist())) | |
| 58 | + xb = F.to_numpy() @ bw + Fm[MCOLS].to_numpy() @ bm | |
| 59 | + resid = y - xb | |
| 60 | + r2w = 1 - np.var(yw - Fw.to_numpy() @ bw_all) / np.var(yw) | |
| 61 | + Fw = Fw[FEATS] | |
| 62 | + log(f" hédonique within-municipalité : R² within = {r2w:.3f} ; β = { {k: round(v, 3) for k, v in beta_global.items()} }") | |
| 63 | + # --- grille régionale (within-municipalité) rétrécie vers la globale (n/(n+300)) | |
| 64 | + beta_region = {} | |
| 65 | + for r in regions: | |
| 66 | + sel = (tr["region"] == r).to_numpy() | |
| 67 | + if sel.sum() < 100: beta_region[r] = beta_global; continue | |
| 68 | + b, *_ = np.linalg.lstsq(Fw.to_numpy()[sel], yw[sel], rcond=None) | |
| 69 | + w = sel.sum() / (sel.sum() + 300.0) | |
| 70 | + beta_region[r] = {k: float(w * b[j] + (1 - w) * beta_global[k]) for j, k in enumerate(FEATS)} | |
| 71 | + # --- M10 : hédonique directe = Xβ + effet de région + MRC + municipalité + unité de voisinage (rétrécis en cascade) | |
| 72 | + reg_eff = pd.Series(resid, index=tr.index).groupby(tr["region"]).mean().to_dict() | |
| 73 | + tr["res_g"] = resid - tr["region"].map(reg_eff).to_numpy() | |
| 74 | + mrc_eff = tr.groupby("mrc")["res_g"].agg(["sum", "count"]) | |
| 75 | + mrc_eff = (mrc_eff["sum"] / (mrc_eff["count"] + 30)).to_dict() | |
| 76 | + tr["res_m"] = tr["res_g"] - tr["mrc"].map(mrc_eff).fillna(0) | |
| 77 | + mun_eff = tr.groupby("code_mun")["res_m"].agg(["sum", "count"]) | |
| 78 | + mun_eff = (mun_eff["sum"] / (mun_eff["count"] + 20)).to_dict() | |
| 79 | + tr["res_u"] = tr["res_m"] - tr["code_mun"].map(mun_eff).fillna(0) | |
| 80 | + # correction médiane : la MCO vise la moyenne du log ; la distribution est asymétrique (queue basse), on recentre sur la médiane | |
| 81 | + uv_tmp = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].transform(lambda v: v.sum() / (len(v) + 10)) | |
| 82 | + median_shift = float(np.median(tr["res_u"] - uv_tmp)) | |
| 83 | + uv = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].agg(["sum", "count"]) | |
| 84 | + uv_eff = {f"{a}|{b}": v for (a, b), v in (uv["sum"] / (uv["count"] + 10)).items()} | |
| 85 | + # --- LightGBM (prix ajusté dans le temps) avec rôle et sans rôle | |
| 86 | + def gbm_frame(d): | |
| 87 | + G = feats(d) | |
| 88 | + G["lat"] = d["lat"]; G["lng"] = d["lng"]; G["x_km"] = d["x_km"]; G["y_km"] = d["y_km"] | |
| 89 | + G["rid"] = d["rid"]; G["btype"] = d["btype"] | |
| 90 | + G["ln_role"] = np.log(d["role_vente"]); G["part_terrain"] = d["part_terrain"]; G["role_vintage"] = d["role_vintage"] | |
| 91 | + G["month"] = np.minimum(d["month"], m_asof - 1).astype(float) # tendance résiduelle ; hors échantillon = dernier mois connu | |
| 92 | + return G | |
| 93 | + G = gbm_frame(tr) | |
| 94 | + params = dict(objective="regression_l1", learning_rate=0.05, num_leaves=127, min_data_in_leaf=40, feature_fraction=0.8, | |
| 95 | + bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=int(os.environ.get("CV_THREADS", 20))) | |
| 96 | + models = {} | |
| 97 | + for name, cols in [("gbm", [c for c in G.columns if c not in ("ln_role", "part_terrain", "role_vintage")]), | |
| 98 | + ("gbm_role", list(G.columns))]: | |
| 99 | + ds = lgb.Dataset(G[cols], y, categorical_feature=["rid", "btype"] if "btype" in cols else ["rid"]) | |
| 100 | + # validation interne : derniers 10 % chronologiques pour fixer le nombre d'arbres | |
| 101 | + n = len(G); cut = int(n * 0.9) | |
| 102 | + dtr = lgb.Dataset(G[cols].iloc[:cut], y[:cut], categorical_feature=["rid", "btype"]) | |
| 103 | + dva = lgb.Dataset(G[cols].iloc[cut:], y[cut:], reference=dtr) | |
| 104 | + bst = lgb.train(params, dtr, num_boost_round=3000, valid_sets=[dva], callbacks=[lgb.early_stopping(100, verbose=False)]) | |
| 105 | + best = max(bst.best_iteration, 300) # plancher : l'arrêt précoce sur la tranche finale (mois partiels) peut être trop court | |
| 106 | + bst = lgb.train(params, ds, num_boost_round=best) | |
| 107 | + models[name] = bst | |
| 108 | + Gall = gbm_frame(df); Gall["month"] = float(m_asof - 1) # f évalué au niveau du dernier mois connu pour toutes les ventes | |
| 109 | + pred_all = bst.predict(Gall[cols]) | |
| 110 | + np.save(OUT / f"{name}_pred_{period}.npy", pred_all.astype(np.float32)) | |
| 111 | + rr = y - bst.predict(G[cols]); r2 = 1 - np.var(rr) / np.var(y) | |
| 112 | + log(f" {name}: {best} arbres, R² (in-sample) = {r2:.3f}, résidu médian = {np.median(rr) * 100:.2f} %") | |
| 113 | + bst.save_model(str(OUT / f"{name}_{period}.txt")) | |
| 114 | + save_json({"period": period, "m_asof": m_asof, "start": start, "n_train": len(tr), "feats": FEATS, | |
| 115 | + "beta_global": beta_global, "intercept_global": 0.0, "region_fe": reg_eff, | |
| 116 | + "beta_region": beta_region, "mrc_eff": mrc_eff, "mun_eff": mun_eff, "uv_eff": uv_eff, | |
| 117 | + "r2_within": float(r2w), "sd_resid_u": float(tr["res_u"].std()), "median_shift": median_shift}, | |
| 118 | + f"models_{period}.json") | |
| 119 | +log("modèles terminés") | |
added
M3U96aomparables-validation/pipeline/04_backtest.py
+193 −0
@@ -0,0 +1,193 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04 — Moteur de rétro-évaluation. | |
| 3 | +Pour chaque vente cible (2024 = validation, 2025-2026 = test), le prix est masqué ; seules les ventes | |
| 4 | +conclues STRICTEMENT AVANT la date cible (et l'indice/les modèles connus à cette date) sont utilisées. | |
| 5 | +On évalue d'un coup toutes les configurations (sélection × rayon × fenêtre × nombre × ajustement × | |
| 6 | +pondération × réconciliation) sur exactement les mêmes cibles. | |
| 7 | +Sorties : targets.parquet, configs_E1.csv/est_E1.npy (surface k,d,T), configs_E2.csv/est_E2.npy/sd_E2.npy | |
| 8 | +(factoriel sélection × ajustement × pondération × réconciliation), n_avail.npy.""" | |
| 9 | +import os | |
| 10 | +for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"): | |
| 11 | + os.environ[_v] = "1" # un seul fil BLAS par processus (fork + BLAS multi-fils = interblocage sur macOS) | |
| 12 | +import multiprocessing as mp, time | |
| 13 | +from scipy.spatial import cKDTree | |
| 14 | +from common import * | |
| 15 | + | |
| 16 | +df = load_parquet("sales.parquet") | |
| 17 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 18 | +rid_map = {r: i for i, r in enumerate(regions)} | |
| 19 | +IDX = np.load(OUT / "index_asof.npy") | |
| 20 | +ASOF = load_json("asof.json")["asof"] | |
| 21 | +MODELS = {a: load_json(f"models_{a}.json") for a in ASOF} | |
| 22 | +FEATS = MODELS[ASOF[0]]["feats"] | |
| 23 | +GBM = {a: np.load(OUT / f"gbm_pred_{a}.npy") for a in ASOF} | |
| 24 | + | |
| 25 | +# ----------------------------------------------------------------- tableaux globaux (partagés par fork) | |
| 26 | +N = len(df) | |
| 27 | +DAY = df["day"].to_numpy(np.int32); MONTH = df["month"].to_numpy(np.int32) | |
| 28 | +RID = df["region"].map(rid_map).to_numpy(np.int32) | |
| 29 | +X = df["x_km"].to_numpy(np.float64); Y = df["y_km"].to_numpy(np.float64) | |
| 30 | +LNP = df["ln_price"].to_numpy(np.float64); P = df["amount"].to_numpy(np.float64) | |
| 31 | +FLOOR = df["floor_area"].to_numpy(np.float64); LAND = df["land_area"].to_numpy(np.float64) | |
| 32 | +AGE = df["age"].to_numpy(np.float64); LIEN = df["lien"].to_numpy(np.int8); ETAGES = df["etages"].to_numpy(np.float64) | |
| 33 | +GENRE = df["genre"].to_numpy(np.int8) | |
| 34 | +PROV = pd.factorize(df["id_provinc"])[0].astype(np.int32) | |
| 35 | +MUN = pd.factorize(df["code_mun"])[0].astype(np.int32) | |
| 36 | +VINT = df["role_vintage"].to_numpy(np.int8); LNROLE = np.log(df["role_vente"].to_numpy(np.float64)) | |
| 37 | +PART = df["part_terrain"].to_numpy(np.float64) | |
| 38 | +NEUF = df["neuf"].to_numpy(np.float64) | |
| 39 | +# matrice des caractéristiques hédoniques (même ordre que FEATS) | |
| 40 | +XF = np.column_stack([np.log(FLOOR), np.log(LAND), AGE, AGE ** 2 / 100, (LIEN == 1), (LIEN >= 2), | |
| 41 | + (GENRE == 1), (GENRE == 2), (GENRE == 3), ETAGES, NEUF]).astype(np.float64) | |
| 42 | +BETA_G = {a: np.array([MODELS[a]["beta_global"][f] for f in FEATS]) for a in ASOF} | |
| 43 | +BETA_R = {a: np.array([[MODELS[a]["beta_region"][r][f] for f in FEATS] for r in regions]) for a in ASOF} | |
| 44 | +ASOF_ARR = np.array(ASOF) | |
| 45 | +TREE = cKDTree(np.column_stack([X, Y])) | |
| 46 | + | |
| 47 | +D_MAX, T_MAX_DAYS = 25.0, 731 | |
| 48 | +KS = [1, 3, 5, 7, 10, 15, 20, 30, 50] | |
| 49 | +DS = [0.5, 1, 2, 5, 10, 25] | |
| 50 | +TS = [3, 6, 12, 24] | |
| 51 | +SELS = ["geo", "sim", "mix", "adj"] | |
| 52 | +ADJS = ["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"] | |
| 53 | +WRS = [("equal", "median"), ("equal", "mean"), ("dist", "median"), ("dist", "mean"), ("rec", "median"), ("rec", "mean"), | |
| 54 | + ("sim", "median"), ("sim", "mean"), ("all", "median"), ("all", "mean")] | |
| 55 | +ANCHORS = [(3, 2, 6), (5, 5, 12), (10, 10, 24)] | |
| 56 | +E1 = [dict(sel=s, d=d, T=T, k=k, adj=a, w="equal", recon="median") | |
| 57 | + for s in ("geo", "sim", "mix") for d in DS for T in TS for k in KS for a in ("none", "hedo")] | |
| 58 | +E2 = [dict(sel=s, d=d, T=T, k=k, adj=a, w=w, recon=r) | |
| 59 | + for (k, d, T) in ANCHORS for s in SELS for a in ADJS for (w, r) in WRS] | |
| 60 | +LOCAL_COLS = [0, 1, 2, 4, 5, 9] # ln_floor, ln_land, age, lien1, lien2, etages | |
| 61 | +LAMBDA_LOCAL = 50.0 | |
| 62 | + | |
| 63 | +def wmedian(v, w): | |
| 64 | + o = np.argsort(v); v = v[o]; w = w[o] | |
| 65 | + c = np.cumsum(w); return v[np.searchsorted(c, 0.5 * c[-1])] | |
| 66 | + | |
| 67 | +def recon(vals, w, how): | |
| 68 | + ok = ~np.isnan(vals) | |
| 69 | + if not ok.any(): return np.nan, np.nan | |
| 70 | + v = vals[ok]; ww = w[ok] | |
| 71 | + sd = v.std() if len(v) > 1 else np.nan | |
| 72 | + if how == "median": | |
| 73 | + return (np.median(v) if ww.min() == ww.max() else wmedian(v, ww)), sd | |
| 74 | + return np.log(np.sum(ww * np.exp(v)) / ww.sum()), sd | |
| 75 | + | |
| 76 | +def evaluate(i): | |
| 77 | + """Toutes les configurations pour la cible i. Renvoie (est_E1, est_E2, sd_E2, n_avail, info).""" | |
| 78 | + period = int(ASOF_ARR[ASOF_ARR <= MONTH[i]].max()) # dernier trimestre de ré-estimation ≤ mois de la cible | |
| 79 | + e1 = np.full(len(E1), np.nan, np.float32); e2 = np.full(len(E2), np.nan, np.float32) | |
| 80 | + s2 = np.full(len(E2), np.nan, np.float32); nav = np.zeros(len(DS) * len(TS), np.int32) | |
| 81 | + cand = np.asarray(TREE.query_ball_point((X[i], Y[i]), D_MAX), dtype=np.int64) | |
| 82 | + keep = (DAY[cand] < DAY[i]) & (DAY[cand] >= DAY[i] - T_MAX_DAYS) & (PROV[cand] != PROV[i]) | |
| 83 | + cand = cand[keep] | |
| 84 | + info = dict(idx=int(i), asof=period, n_cand=int(len(cand))) | |
| 85 | + if len(cand) == 0: return e1, e2, s2, nav, info | |
| 86 | + dist = np.hypot(X[cand] - X[i], Y[cand] - Y[i]) | |
| 87 | + mago = (DAY[i] - DAY[cand]) / 30.4375 | |
| 88 | + m_eval = MONTH[i]; lvl = IDX[m_eval - M_EVAL0] | |
| 89 | + tadj = np.where(MONTH[cand] == m_eval, 0.0, lvl[RID[cand], m_eval - 1] - lvl[RID[cand], MONTH[cand]]) | |
| 90 | + tadj = np.nan_to_num(tadj, nan=0.0) | |
| 91 | + lnT = LNP[cand] + tadj | |
| 92 | + dX = XF[i] - XF[cand] # cible − comparable | |
| 93 | + # --- ajustements | |
| 94 | + bg = BETA_G[period]; br = BETA_R[period][RID[i]] | |
| 95 | + adj = {} | |
| 96 | + adj["raw"] = LNP[cand] | |
| 97 | + adj["none"] = lnT | |
| 98 | + Pt = np.exp(lnT) | |
| 99 | + d_floor = (FLOOR[i] - FLOOR[cand]) * 0.5 * Pt / FLOOR[cand] | |
| 100 | + d_land = np.clip((LAND[i] - LAND[cand]) * 0.5 * Pt * PART[cand] / LAND[cand], -0.15 * Pt, 0.15 * Pt) | |
| 101 | + d_age = np.clip(-0.005 * Pt * (AGE[i] - AGE[cand]), -0.2 * Pt, 0.2 * Pt) | |
| 102 | + d_lien = np.where((LIEN[i] == 0) & (LIEN[cand] > 0), 0.05 * Pt, np.where((LIEN[i] > 0) & (LIEN[cand] == 0), -0.05 * Pt, 0.0)) | |
| 103 | + tot = np.maximum(d_floor + d_land + d_age + d_lien, -0.6 * Pt) | |
| 104 | + adj["conv"] = np.log(Pt + tot) | |
| 105 | + gross_conv = (np.abs(d_floor) + np.abs(d_land) + np.abs(d_age) + np.abs(d_lien)) / Pt | |
| 106 | + adj["hedo"] = lnT + dX @ bg | |
| 107 | + gross_hedo = np.abs(dX * bg).sum(axis=1) | |
| 108 | + adj["hedo_reg"] = lnT + dX @ br | |
| 109 | + loc = dist <= 10.0 | |
| 110 | + if loc.sum() >= 40: | |
| 111 | + Xl = XF[cand][loc][:, LOCAL_COLS]; yl = lnT[loc] | |
| 112 | + b0 = br[LOCAL_COLS] | |
| 113 | + Xc = Xl - Xl.mean(axis=0); yc = yl - yl.mean() | |
| 114 | + A = Xc.T @ Xc + LAMBDA_LOCAL * np.eye(len(LOCAL_COLS)) | |
| 115 | + bl = np.linalg.solve(A, Xc.T @ yc + LAMBDA_LOCAL * b0) | |
| 116 | + adj["local"] = lnT + dX[:, LOCAL_COLS] @ bl | |
| 117 | + else: | |
| 118 | + adj["local"] = adj["hedo_reg"] | |
| 119 | + f = GBM[period] | |
| 120 | + adj["gbm"] = lnT + (f[i] - f[cand]) | |
| 121 | + same = (MUN[cand] == MUN[i]) & (VINT[cand] == VINT[i]) | |
| 122 | + adj["role"] = np.where(same, LNROLE[i] + (lnT - LNROLE[cand]), np.nan) | |
| 123 | + # --- scores de sélection | |
| 124 | + dissim = np.sqrt((dX[:, 0] / 0.25) ** 2 + (dX[:, 1] / 0.6) ** 2 + (dX[:, 2] / 15) ** 2 | |
| 125 | + + (LIEN[cand] != LIEN[i]) * 1.0 + (ETAGES[cand] != ETAGES[i]) * 0.5) | |
| 126 | + score = {"geo": dist, "sim": dissim + dist / 25.0, "mix": dissim + dist / 2.0, "adj": gross_hedo + dist / 25.0} | |
| 127 | + # --- pondérations | |
| 128 | + wts = {"equal": np.ones(len(cand)), "dist": 1.0 / (0.25 + dist), "rec": np.exp(-mago / 6.0), "sim": 1.0 / (0.1 + gross_hedo)} | |
| 129 | + wts["all"] = wts["dist"] * wts["rec"] * wts["sim"] | |
| 130 | + # --- masques (d, T) et classements | |
| 131 | + top = {} | |
| 132 | + for di, d in enumerate(DS): | |
| 133 | + for ti, T in enumerate(TS): | |
| 134 | + m = (dist <= d) & (mago <= T) | |
| 135 | + nav[di * len(TS) + ti] = int(m.sum()) | |
| 136 | + if not m.any(): continue | |
| 137 | + ids = np.flatnonzero(m) | |
| 138 | + for s in SELS: | |
| 139 | + sc = score[s][ids] | |
| 140 | + if len(ids) > 50: | |
| 141 | + part = np.argpartition(sc, 50)[:50]; part = part[np.argsort(sc[part])] | |
| 142 | + else: | |
| 143 | + part = np.argsort(sc) | |
| 144 | + top[(s, d, T)] = ids[part] | |
| 145 | + # --- E1 | |
| 146 | + for c, cfg in enumerate(E1): | |
| 147 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 148 | + if t is None: continue | |
| 149 | + sub = t[:cfg["k"]] | |
| 150 | + e1[c], _ = recon(adj[cfg["adj"]][sub], wts["equal"][sub], "median") | |
| 151 | + # --- E2 | |
| 152 | + for c, cfg in enumerate(E2): | |
| 153 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 154 | + if t is None: continue | |
| 155 | + sub = t[:cfg["k"]] | |
| 156 | + e2[c], s2[c] = recon(adj[cfg["adj"]][sub], wts[cfg["w"]][sub], cfg["recon"]) | |
| 157 | + return e1, e2, s2, nav, info | |
| 158 | + | |
| 159 | +def worker(chunk): | |
| 160 | + return [evaluate(i) for i in chunk] | |
| 161 | + | |
| 162 | +if __name__ == "__main__": | |
| 163 | + targets = df.index[df["period"].isin(["val", "test"])].to_numpy() | |
| 164 | + if os.environ.get("CV_SAMPLE"): | |
| 165 | + rng = np.random.default_rng(1); targets = np.sort(rng.choice(targets, int(os.environ["CV_SAMPLE"]), replace=False)) | |
| 166 | + SUF = "" | |
| 167 | + if os.environ.get("CV_SHARD"): # répartition sur plusieurs nœuds : CV_SHARD=i/N → cibles i, i+N, i+2N, … | |
| 168 | + i_sh, n_sh = map(int, os.environ["CV_SHARD"].split("/")); targets = targets[i_sh::n_sh]; SUF = f".shard{i_sh}of{n_sh}" | |
| 169 | + log(f"{len(targets):,} cibles ; {len(E1)} configurations E1, {len(E2)} E2 ; {N:,} ventes dans le bassin") | |
| 170 | + pd.DataFrame(E1).to_csv(OUT / "configs_E1.csv", index_label="cfg") | |
| 171 | + pd.DataFrame(E2).to_csv(OUT / "configs_E2.csv", index_label="cfg") | |
| 172 | + est1 = np.full((len(targets), len(E1)), np.nan, np.float32) | |
| 173 | + est2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 174 | + sd2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 175 | + nav = np.zeros((len(targets), len(DS) * len(TS)), np.int32) | |
| 176 | + infos = [] | |
| 177 | + chunks = [targets[j:j + 400] for j in range(0, len(targets), 400)] | |
| 178 | + t0 = time.time(); done = 0 | |
| 179 | + with mp.get_context(os.environ.get("CV_MP", "fork")).Pool(int(os.environ.get("CV_PROCS", 20))) as pool: | |
| 180 | + for res in pool.imap(worker, chunks): | |
| 181 | + for (e1, e2, s2, nv, info) in res: | |
| 182 | + est1[done] = e1; est2[done] = e2; sd2[done] = s2; nav[done] = nv; infos.append(info); done += 1 | |
| 183 | + if done % 20000 < 400: | |
| 184 | + log(f" {done:,}/{len(targets):,} cibles ({time.time() - t0:.0f} s)") | |
| 185 | + np.save(OUT / f"est_E1{SUF}.npy", est1); np.save(OUT / f"est_E2{SUF}.npy", est2); np.save(OUT / f"sd_E2{SUF}.npy", sd2); np.save(OUT / f"n_avail{SUF}.npy", nav) | |
| 186 | + T = df.loc[targets, ["idx", "id", "date", "period", "amount", "ln_price", "region", "region_nom", "mrc", "mrc_nom", "code_mun", | |
| 187 | + "mun_nom", "unite_voisinage", "floor_area", "land_area", "age", "lien", "etages", "role_vente", "month", "lat", "lng"]].reset_index(drop=True) | |
| 188 | + T["n_cand"] = [x["n_cand"] for x in infos]; T["asof"] = [x["asof"] for x in infos] | |
| 189 | + T["n_2km_12m"] = nav[:, DS.index(2) * len(TS) + TS.index(12)] | |
| 190 | + T["n_5km_12m"] = nav[:, DS.index(5) * len(TS) + TS.index(12)] | |
| 191 | + save_parquet(T, f"targets{SUF}.parquet") | |
| 192 | + pd.DataFrame({"DS": [DS], "TS": [TS], "KS": [KS]}).to_json(OUT / "grid_meta.json") | |
| 193 | + log(f"rétro-évaluation terminée en {time.time() - t0:.0f} s") | |
added
M3U96aomparables-validation/pipeline/04b_gather.py
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04b — Rassemble les fragments (shards) de la rétro-évaluation calculés sur plusieurs nœuds, dans l'ordre des cibles.""" | |
| 3 | +import glob | |
| 4 | +from common import * | |
| 5 | +sh = sorted(glob.glob(str(OUT / "targets.shard*.parquet"))) | |
| 6 | +log(f"{len(sh)} fragments") | |
| 7 | +Ts = [pd.read_parquet(p) for p in sh] | |
| 8 | +parts = {} | |
| 9 | +for name in ["est_E1", "est_E2", "sd_E2", "n_avail"]: | |
| 10 | + parts[name] = [np.load(p.replace("targets.", f"{name}.").replace(".parquet", ".npy")) for p in sh] | |
| 11 | +T = pd.concat(Ts, ignore_index=True); order = np.argsort(T["idx"].to_numpy(), kind="stable") | |
| 12 | +T = T.iloc[order].reset_index(drop=True); save_parquet(T, "targets.parquet") | |
| 13 | +for name, arrs in parts.items(): | |
| 14 | + A = np.concatenate(arrs, axis=0)[order]; np.save(OUT / f"{name}.npy", A); log(f" {name}: {A.shape}") | |
| 15 | +log("rassemblement terminé") | |
added
M3U96aomparables-validation/pipeline/05_analyze.py
+358 −0
@@ -0,0 +1,358 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""05 — Analyse : métriques par configuration (validation 2024 → choix ; test 2025-26 → verdict), | |
| 3 | +surface Erreur(k, d, t), effets des facteurs (sélection, ajustement, pondération, réconciliation), | |
| 4 | +compétition de modèles M1-M13, hypothèses H1-H5, strates (région, densité, prix), intervalles, | |
| 5 | +règles conditionnelles. Sorties : tables/*.csv, results.json, fig/*.pdf""" | |
| 6 | +from scipy.optimize import nnls | |
| 7 | +import matplotlib; matplotlib.use("Agg") | |
| 8 | +import matplotlib.pyplot as plt | |
| 9 | +from matplotlib import rcParams | |
| 10 | +from common import * | |
| 11 | + | |
| 12 | +# ---------------------------------------------------------------- style (palette de référence, marques fines) | |
| 13 | +PAL = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"] | |
| 14 | +SEQ = ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#184f95", "#0d366b"] | |
| 15 | +rcParams.update({"font.family": "Helvetica Neue", "font.size": 9, "axes.edgecolor": "#c3c2b7", "axes.linewidth": 0.6, | |
| 16 | + "axes.grid": True, "grid.color": "#e8e7e3", "grid.linewidth": 0.5, "axes.spines.top": False, | |
| 17 | + "axes.spines.right": False, "legend.frameon": False, "figure.dpi": 150, "savefig.bbox": "tight", | |
| 18 | + "axes.titleweight": "bold", "axes.titlesize": 10, "axes.labelcolor": "#52514e", "xtick.color": "#52514e", | |
| 19 | + "ytick.color": "#52514e", "figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb"}) | |
| 20 | +from matplotlib.colors import LinearSegmentedColormap | |
| 21 | +CMAP = LinearSegmentedColormap.from_list("seqblue", SEQ) | |
| 22 | +TAB = OUT / "tables"; TAB.mkdir(exist_ok=True); FIG = OUT / "fig" | |
| 23 | +def savefig(fig, name): fig.savefig(FIG / f"{name}.pdf"); fig.savefig(FIG / f"{name}.png"); plt.close(fig) | |
| 24 | + | |
| 25 | +# ---------------------------------------------------------------- chargement | |
| 26 | +T = load_parquet("targets.parquet"); S = load_parquet("sales.parquet").set_index("idx") | |
| 27 | +E1 = pd.read_csv(OUT / "configs_E1.csv"); E2 = pd.read_csv(OUT / "configs_E2.csv") | |
| 28 | +est1 = np.load(OUT / "est_E1.npy"); est2 = np.load(OUT / "est_E2.npy"); sd2 = np.load(OUT / "sd_E2.npy"); nav = np.load(OUT / "n_avail.npy") | |
| 29 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"]; IDX = np.load(OUT / "index_asof.npy") | |
| 30 | +rid = {r: i for i, r in enumerate(regions)} | |
| 31 | +lnP = T["ln_price"].to_numpy(np.float32) | |
| 32 | +VAL = (T["period"] == "val").to_numpy(); TEST = (T["period"] == "test").to_numpy() | |
| 33 | +log(f"{len(T):,} cibles (val {VAL.sum():,}, test {TEST.sum():,}) ; E1 {est1.shape[1]} ; E2 {est2.shape[1]}") | |
| 34 | + | |
| 35 | +def err_of(est): return (np.exp(est - lnP[:, None]) - 1).astype(np.float32) | |
| 36 | +def metrics(err, sel): | |
| 37 | + e = err[sel]; a = np.abs(e) | |
| 38 | + ok = ~np.isnan(a); n = ok.sum(axis=0) | |
| 39 | + with np.errstate(all="ignore"): | |
| 40 | + return pd.DataFrame({"mdape": np.nanmedian(a, axis=0) * 100, "mape": np.nanmean(a, axis=0) * 100, | |
| 41 | + "biais": np.nanmedian(e, axis=0) * 100, "biais_moy": np.nanmean(e, axis=0) * 100, | |
| 42 | + "p5": np.nanmean(a <= 0.05, axis=0) * 100, "p10": np.nanmean(a <= 0.10, axis=0) * 100, | |
| 43 | + "p20": np.nanmean(a <= 0.20, axis=0) * 100, "couverture": n / sel.sum() * 100, "n": n}) | |
| 44 | +err1 = err_of(est1); err2 = err_of(est2) | |
| 45 | +del est1 | |
| 46 | +res = {} | |
| 47 | + | |
| 48 | +# ---------------------------------------------------------------- densité & strates | |
| 49 | +T["dens"] = pd.cut(T["n_2km_12m"], [-1, 4, 19, 49, 199, 10 ** 9], labels=["< 5", "5-19", "20-49", "50-199", "≥ 200"]) | |
| 50 | +T["prix_q"] = pd.qcut(T["amount"], 5, labels=["Q1 (bas)", "Q2", "Q3", "Q4", "Q5 (haut)"]) | |
| 51 | +T["trimestre"] = pd.to_datetime(T["date"]).dt.to_period("Q").astype(str) | |
| 52 | +T["age_cl"] = pd.cut(T["age"], [-1, 3, 15, 40, 70, 1000], labels=["≤ 3 ans", "4-15", "16-40", "41-70", "> 70"]) | |
| 53 | +T["cluster"] = pd.factorize(T["code_mun"].astype(str) + "|" + T["trimestre"])[0] | |
| 54 | + | |
| 55 | +# ---------------------------------------------------------------- E1 : surface Erreur(k, d, t) | |
| 56 | +mv = metrics(err1, VAL).add_suffix("_val"); mt = metrics(err1, TEST).add_suffix("_test") | |
| 57 | +surf = pd.concat([E1, mv, mt], axis=1) | |
| 58 | +surf.to_csv(TAB / "surface_E1.csv", index=False) | |
| 59 | +ok = surf["couverture_val"] >= 97 | |
| 60 | +best_e1 = surf[ok].sort_values("mdape_val").iloc[0] | |
| 61 | +res["E1_best_val"] = best_e1.to_dict() | |
| 62 | +log(f"E1 meilleure (val, couverture ≥ 97 %) : {best_e1[['sel','d','T','k','adj']].to_dict()} → val {best_e1.mdape_val:.2f} %, test {best_e1.mdape_test:.2f} %") | |
| 63 | +for (s, a), g in surf[ok].groupby(["sel", "adj"]): | |
| 64 | + b = g.sort_values("mdape_val").iloc[0] | |
| 65 | + res.setdefault("E1_best_by_sel_adj", {})[f"{s}|{a}"] = dict(k=int(b.k), d=float(b.d), T=int(b["T"]), mdape_val=float(b.mdape_val), mdape_test=float(b.mdape_test), couverture_test=float(b.couverture_test)) | |
| 66 | + | |
| 67 | +# figure : MdAPE vs k selon d (T = 12, geo, hedo, test) | |
| 68 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.4), sharey=True) | |
| 69 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 70 | + for j, d in enumerate([0.5, 1, 2, 5, 10, 25]): | |
| 71 | + g = surf[(surf.sel == "geo") & (surf["T"] == 12) & (surf.adj == a) & (surf.d == d)].sort_values("k") | |
| 72 | + ax.plot(g.k, g.mdape_test, color=PAL[j], lw=2, marker="o", ms=4, label=f"d ≤ {d} km") | |
| 73 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]) | |
| 74 | + ax.set_xlabel("nombre de comparables k"); ax.set_title({"none": "sans ajustement (temps seulement)", "hedo": "grille hédonique"}[a]) | |
| 75 | +axes[0].set_ylabel("MdAPE, test 2025-26 (%)"); axes[1].legend(ncol=2, fontsize=8) | |
| 76 | +savefig(fig, "f1_mdape_vs_k") | |
| 77 | +# figure : carte d × T au meilleur k (geo, hedo) | |
| 78 | +fig, axes = plt.subplots(1, 2, figsize=(8, 3.2)) | |
| 79 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 80 | + g = surf[(surf.sel == "geo") & (surf.adj == a)].groupby(["d", "T"])["mdape_test"].min().unstack("T") | |
| 81 | + im = ax.imshow(g.to_numpy(), cmap=CMAP, aspect="auto") | |
| 82 | + ax.set_xticks(range(len(g.columns))); ax.set_xticklabels([f"{t} mois" for t in g.columns]); ax.set_yticks(range(len(g.index))); ax.set_yticklabels([f"{d} km" for d in g.index]) | |
| 83 | + for i in range(g.shape[0]): | |
| 84 | + for j in range(g.shape[1]): | |
| 85 | + v = g.iloc[i, j]; ax.text(j, i, f"{v:.1f}", ha="center", va="center", fontsize=7.5, color="white" if v < np.nanpercentile(g.to_numpy(), 40) else "#0b0b0b") | |
| 86 | + ax.grid(False); ax.set_title({"none": "sans ajustement", "hedo": "grille hédonique"}[a] + " — MdAPE au meilleur k") | |
| 87 | +savefig(fig, "f2_surface_d_T") | |
| 88 | + | |
| 89 | +# ---------------------------------------------------------------- E2 : facteurs | |
| 90 | +mv2 = metrics(err2, VAL).add_suffix("_val"); mt2 = metrics(err2, TEST).add_suffix("_test") | |
| 91 | +fac = pd.concat([E2, mv2, mt2], axis=1); fac.to_csv(TAB / "facteurs_E2.csv", index=False) | |
| 92 | +anchor = (fac.k == 5) & (fac.d == 5) & (fac["T"] == 12) | |
| 93 | +base = anchor & (fac.sel == "geo") & (fac.w == "equal") & (fac.recon == "median") | |
| 94 | +res["E2_adjust_anchor"] = fac[base].set_index("adj")[["mdape_val", "mdape_test", "mape_test", "biais_test", "p10_test", "couverture_test"]].to_dict("index") | |
| 95 | +res["E2_sel_anchor"] = fac[anchor & (fac.adj == "hedo") & (fac.w == "equal") & (fac.recon == "median")].set_index("sel")[["mdape_val", "mdape_test", "p10_test"]].to_dict("index") | |
| 96 | +res["E2_wr_anchor"] = {f"{r.w}|{r.recon}": dict(mdape_val=r.mdape_val, mdape_test=r.mdape_test, p10_test=r.p10_test) | |
| 97 | + for r in fac[anchor & (fac.adj == "hedo") & (fac.sel == "geo")].itertuples()} | |
| 98 | +# effets marginaux moyens (moyenne de MdAPE test sur toutes les autres modalités) | |
| 99 | +marg = {} | |
| 100 | +for f in ["sel", "adj", "w", "recon"]: | |
| 101 | + marg[f] = fac.groupby(f)["mdape_test"].mean().round(2).to_dict() | |
| 102 | +res["E2_marginal"] = marg | |
| 103 | +anc_tab = fac.groupby(["k", "d", "T"])["mdape_test"].min().round(2).to_dict() | |
| 104 | +res["E2_anchor_best"] = {f"{k}|{d}|{t}": v for (k, d, t), v in anc_tab.items()} | |
| 105 | +# figure : ajustements à l'ancre (val et test) | |
| 106 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 107 | +g = fac[base].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 108 | +lab = {"raw": "aucun", "none": "temps seul", "conv": "grille conv.", "hedo": "hédonique", "hedo_reg": "hédo. régionale", "local": "locale", "gbm": "GBM", "role": "ratio au rôle"} | |
| 109 | +x = np.arange(len(g)); ax.bar(x - 0.2, g.mdape_val, 0.38, color=PAL[0], label="validation 2024"); ax.bar(x + 0.2, g.mdape_test, 0.38, color=PAL[1], label="test 2025-26") | |
| 110 | +for i, (v1, v2) in enumerate(zip(g.mdape_val, g.mdape_test)): | |
| 111 | + ax.text(i - 0.2, v1 + 0.2, f"{v1:.1f}", ha="center", fontsize=7); ax.text(i + 0.2, v2 + 0.2, f"{v2:.1f}", ha="center", fontsize=7) | |
| 112 | +ax.set_xticks(x); ax.set_xticklabels([lab[a] for a in g.index]); ax.set_ylabel("MdAPE (%)"); ax.legend(); ax.set_title("Type d'ajustement — 5 comparables les plus proches, ≤ 5 km, ≤ 12 mois, médiane") | |
| 113 | +savefig(fig, "f3_ajustements") | |
| 114 | + | |
| 115 | +# ---------------------------------------------------------------- modèles de référence (M10, M11) et compétition | |
| 116 | +def level_shift(m_asof): | |
| 117 | + mt_ = T["month"].to_numpy(); r = T["region"].map(rid).to_numpy() | |
| 118 | + return IDX[mt_ - M_EVAL0, r, mt_ - 1] - IDX[mt_ - M_EVAL0, r, m_asof - 1] | |
| 119 | +Sx = S.loc[T["idx"]] | |
| 120 | +XF = np.column_stack([Sx.ln_floor, Sx.ln_land, Sx.age, Sx.age ** 2 / 100, Sx.lien == 1, Sx.lien >= 2, Sx.genre == 1, Sx.genre == 2, Sx.genre == 3, Sx.etages, Sx.neuf]).astype(float) | |
| 121 | +M10 = np.full(len(T), np.nan); M11 = np.full(len(T), np.nan); M11b = np.full(len(T), np.nan) | |
| 122 | +for a in load_json("asof.json")["asof"]: | |
| 123 | + mask = (T["asof"] == a).to_numpy(); period = str(a) | |
| 124 | + if not mask.any(): continue | |
| 125 | + mod = load_json(f"models_{period}.json"); beta = np.array([mod["beta_global"][f] for f in mod["feats"]]) | |
| 126 | + shift = level_shift(mod["m_asof"]) | |
| 127 | + fe = (T["region"].map(mod["region_fe"]).fillna(0) + T["mrc"].map(mod["mrc_eff"]).fillna(0) + T["code_mun"].map(mod["mun_eff"]).fillna(0) | |
| 128 | + + (T["code_mun"].astype(str) + "|" + T["unite_voisinage"].astype(str)).map(mod["uv_eff"]).fillna(0)).to_numpy() | |
| 129 | + M10[mask] = (XF @ beta + fe + shift + mod["median_shift"])[mask] | |
| 130 | + M11[mask] = (np.load(OUT / f"gbm_role_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 131 | + M11b[mask] = (np.load(OUT / f"gbm_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 132 | + | |
| 133 | +def e2col(sel, k, d, Tm, adj, w, recon): | |
| 134 | + m = (E2.sel == sel) & (E2.k == k) & (E2.d == d) & (E2["T"] == Tm) & (E2.adj == adj) & (E2.w == w) & (E2.recon == recon) | |
| 135 | + return int(E2.index[m][0]) | |
| 136 | +def e1col(sel, k, d, Tm, adj): | |
| 137 | + m = (E1.sel == sel) & (E1.k == k) & (E1.d == d) & (E1["T"] == Tm) & (E1.adj == adj) | |
| 138 | + return int(E1.index[m][0]) | |
| 139 | +MODELS = { | |
| 140 | + "M1 médiane des 3 plus proches (temps seul)": ("E2", e2col("geo", 3, 2, 6, "none", "equal", "median")), | |
| 141 | + "M2 moyenne des 5 plus proches (temps seul)": ("E2", e2col("geo", 5, 5, 12, "none", "equal", "mean")), | |
| 142 | + "M3 sélection par similarité physique": ("E2", e2col("sim", 5, 5, 12, "none", "equal", "median")), | |
| 143 | + "M4 pondération par distance": ("E2", e2col("geo", 5, 5, 12, "none", "dist", "mean")), | |
| 144 | + "M5 pondération par récence": ("E2", e2col("geo", 5, 5, 12, "none", "rec", "mean")), | |
| 145 | + "M6 grille d'ajustement conventionnelle": ("E2", e2col("geo", 5, 5, 12, "conv", "equal", "median")), | |
| 146 | + "M7 grille d'ajustement hédonique": ("E2", e2col("geo", 5, 5, 12, "hedo", "equal", "median")), | |
| 147 | + "M8a grille locale (régression sur le voisinage)": ("E2", e2col("geo", 5, 5, 12, "local", "equal", "median")), | |
| 148 | + "M8b ajustements non linéaires (GBM)": ("E2", e2col("geo", 5, 5, 12, "gbm", "equal", "median")), | |
| 149 | + "M13 ratio au rôle (5 plus proches, même rôle)": ("E2", e2col("geo", 5, 5, 12, "role", "equal", "median")), | |
| 150 | + "M10 hédonique globale directe": ("X", M10), | |
| 151 | + "M11a gradient boosting (sans rôle)": ("X", M11b), | |
| 152 | + "M11b gradient boosting (avec rôle)": ("X", M11), | |
| 153 | +} | |
| 154 | +def est_of(spec): | |
| 155 | + kind, c = spec | |
| 156 | + return est2[:, c].astype(float) if kind == "E2" else c | |
| 157 | +# M0 : meilleure configuration empirique choisie sur la validation (E1 ∪ E2, couverture ≥ 90 %) | |
| 158 | +cand_all = pd.concat([surf.assign(src="E1"), fac.assign(src="E2")], ignore_index=True) | |
| 159 | +cand_ok = cand_all[cand_all.couverture_val >= 97].sort_values("mdape_val") | |
| 160 | +b0 = cand_ok.iloc[0] | |
| 161 | +M0_est = (est2[:, int(b0.cfg)] if b0.src == "E2" else np.load(OUT / "est_E1.npy", mmap_mode="r")[:, int(b0.cfg)]).astype(float) | |
| 162 | +res["M0"] = b0[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_test"]].to_dict() | |
| 163 | +res["M0_sans_contrainte"] = cand_all[cand_all.couverture_val >= 50].sort_values("mdape_val").iloc[0][["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_val", "couverture_test"]].to_dict() | |
| 164 | +MODELS["M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"] = ("X", M0_est) | |
| 165 | +# M9 : réconciliation optimisée = combinaison convexe (NNLS sur la validation) d'estimations de comparables | |
| 166 | +stack_cols = {a: e2col("geo", 5, 5, 12, a, "equal", "median") for a in ["none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]} | |
| 167 | +def stack(names, mats, fit_mask): | |
| 168 | + """Combinaison convexe apprise par NNLS sur la validation ; si une composante manque pour une cible, | |
| 169 | + les poids des composantes disponibles sont renormalisés.""" | |
| 170 | + A = np.column_stack(mats); okr = ~np.isnan(A).any(axis=1) | |
| 171 | + w, _ = nnls(A[okr & fit_mask], lnP[okr & fit_mask].astype(float)); w = w / w.sum() | |
| 172 | + W = np.where(np.isnan(A), 0.0, w[None, :]); den = W.sum(axis=1) | |
| 173 | + out = np.nansum(A * W, axis=1) / np.where(den > 0, den, np.nan) | |
| 174 | + return out, dict(zip(names, w.round(3))) | |
| 175 | +M9_est, w9 = stack(list(stack_cols), [est2[:, c].astype(float) for c in stack_cols.values()], VAL) | |
| 176 | +res["M9_weights"] = w9; MODELS["M9 réconciliation optimisée (poids appris en 2024)"] = ("X", M9_est) | |
| 177 | +M12_est, w12 = stack(["gbm_role", "comps_gbm", "comps_hedo"], [M11, est2[:, stack_cols["gbm"]].astype(float), est2[:, stack_cols["hedo"]].astype(float)], VAL) | |
| 178 | +res["M12_weights"] = w12; MODELS["M12 hybride comparables + GBM (poids appris en 2024)"] = ("X", M12_est) | |
| 179 | + | |
| 180 | +rows = [] | |
| 181 | +ERR = {} | |
| 182 | +for name, spec in MODELS.items(): | |
| 183 | + e = (np.exp(est_of(spec) - lnP) - 1).astype(np.float32); ERR[name] = e | |
| 184 | + for per, mask in [("val", VAL), ("test", TEST)]: | |
| 185 | + m = metrics(e[:, None], mask).iloc[0].to_dict(); m.update(modele=name, periode=per); rows.append(m) | |
| 186 | +comp = pd.DataFrame(rows); comp.to_csv(TAB / "competition_modeles.csv", index=False) | |
| 187 | +pd.DataFrame({n.split(" ")[0]: est_of(s_) for n, s_ in MODELS.items()}).assign(idx=T["idx"].to_numpy()).to_parquet(OUT / "models_est.parquet", index=False) | |
| 188 | +res["competition"] = comp[comp.periode == "test"].set_index("modele")[["mdape", "mape", "biais", "p5", "p10", "p20", "couverture"]].round(2).to_dict("index") | |
| 189 | +res["competition_val"] = comp[comp.periode == "val"].set_index("modele")[["mdape", "p10", "couverture"]].round(2).to_dict("index") | |
| 190 | +log("compétition (test) :\n" + comp[comp.periode == "test"][["modele", "mdape", "mape", "biais", "p10", "couverture"]].round(2).to_string(index=False)) | |
| 191 | +fig, ax = plt.subplots(figsize=(7.5, 5)) | |
| 192 | +g = comp[comp.periode == "test"].sort_values("mdape", ascending=False) | |
| 193 | +cols = [PAL[0] if ("M1" in n and "M1 " not in n) or n.startswith(("M10", "M11", "M12")) else PAL[2] if n.startswith(("M0", "M9")) else PAL[1] for n in g.modele] | |
| 194 | +cols = [PAL[0] if n.startswith(("M10", "M11", "M12")) else (PAL[2] if n.startswith(("M0", "M9")) else PAL[1]) for n in g.modele] | |
| 195 | +ax.barh(g.modele, g.mdape, color=cols, height=0.65) | |
| 196 | +for i, v in enumerate(g.mdape): ax.text(v + 0.15, i, f"{v:.1f} %", va="center", fontsize=8) | |
| 197 | +ax.set_xlabel("MdAPE, test 2025-26 (%)"); ax.set_title("Compétition de modèles (orange : comparables ; vert : optimisés ; bleu : références externes)") | |
| 198 | +savefig(fig, "f4_competition") | |
| 199 | + | |
| 200 | +# ---------------------------------------------------------------- bootstrap par grappes (municipalité × trimestre) des différences appariées | |
| 201 | +rng = np.random.default_rng(7) | |
| 202 | +def paired(eA, eB, mask, B=400): | |
| 203 | + a = np.abs(eA[mask]); b = np.abs(eB[mask]); cl = T["cluster"].to_numpy()[mask] | |
| 204 | + okp = ~np.isnan(a) & ~np.isnan(b); a, b, cl = a[okp], b[okp], cl[okp] | |
| 205 | + d = a - b | |
| 206 | + cl_ids, inv = np.unique(cl, return_inverse=True); nc = len(cl_ids) | |
| 207 | + sums = np.bincount(inv, d, nc); cnts = np.bincount(inv, minlength=nc) | |
| 208 | + boots_mean = []; boots_med = [] | |
| 209 | + order = np.argsort(inv); d_sorted = d[order]; starts = np.searchsorted(inv[order], np.arange(nc)); ends = np.append(starts[1:], len(d)) | |
| 210 | + for _ in range(B): | |
| 211 | + pick = rng.integers(0, nc, nc) | |
| 212 | + boots_mean.append(sums[pick].sum() / cnts[pick].sum()) | |
| 213 | + if _ < 150: | |
| 214 | + idxs = np.concatenate([np.arange(starts[p], ends[p]) for p in pick]) | |
| 215 | + boots_med.append(np.median(a[order][idxs]) - np.median(b[order][idxs])) | |
| 216 | + return dict(n=int(len(d)), diff_mape=float(d.mean() * 100), ci_mape=[float(np.percentile(boots_mean, 2.5) * 100), float(np.percentile(boots_mean, 97.5) * 100)], | |
| 217 | + diff_mdape=float((np.median(a) - np.median(b)) * 100), ci_mdape=[float(np.percentile(boots_med, 2.5) * 100), float(np.percentile(boots_med, 97.5) * 100)], | |
| 218 | + part_A_meilleur=float((a < b).mean() * 100)) | |
| 219 | +PAIRS = [ | |
| 220 | + ("M7 grille d'ajustement hédonique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 221 | + ("M7 grille d'ajustement hédonique", "M6 grille d'ajustement conventionnelle"), | |
| 222 | + ("M8a grille locale (régression sur le voisinage)", "M7 grille d'ajustement hédonique"), | |
| 223 | + ("M8b ajustements non linéaires (GBM)", "M7 grille d'ajustement hédonique"), | |
| 224 | + ("M11b gradient boosting (avec rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 225 | + ("M11a gradient boosting (sans rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 226 | + ("M12 hybride comparables + GBM (poids appris en 2024)", "M11b gradient boosting (avec rôle)"), | |
| 227 | + ("M13 ratio au rôle (5 plus proches, même rôle)", "M7 grille d'ajustement hédonique"), | |
| 228 | + ("M3 sélection par similarité physique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 229 | + ("M9 réconciliation optimisée (poids appris en 2024)", "M7 grille d'ajustement hédonique"), | |
| 230 | + ("M1 médiane des 3 plus proches (temps seul)", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 231 | +] | |
| 232 | +res["paires"] = {f"{a} vs {b}": paired(ERR[a], ERR[b], TEST) for a, b in PAIRS} | |
| 233 | +pd.DataFrame([dict(A=a, B=b, **res["paires"][f"{a} vs {b}"]) for a, b in PAIRS]).to_csv(TAB / "paires_bootstrap.csv", index=False) | |
| 234 | + | |
| 235 | +# ---------------------------------------------------------------- strates | |
| 236 | +def strata(err, by): | |
| 237 | + out = [] | |
| 238 | + for lev, g in T[TEST].groupby(by, observed=True): | |
| 239 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 240 | + r = metrics(err[:, None], m).iloc[0].to_dict(); r[by] = str(lev); out.append(r) | |
| 241 | + return pd.DataFrame(out) | |
| 242 | +KEY = ["M2 moyenne des 5 plus proches (temps seul)", "M7 grille d'ajustement hédonique", "M8a grille locale (régression sur le voisinage)", | |
| 243 | + "M13 ratio au rôle (5 plus proches, même rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)", "M10 hédonique globale directe", "M11b gradient boosting (avec rôle)", "M12 hybride comparables + GBM (poids appris en 2024)"] | |
| 244 | +for by in ["region_nom", "dens", "prix_q", "trimestre", "age_cl", "lien"]: | |
| 245 | + tabs = [] | |
| 246 | + for name in KEY: | |
| 247 | + s = strata(ERR[name], by); s["modele"] = name.split(" ")[0]; tabs.append(s) | |
| 248 | + st = pd.concat(tabs); st.to_csv(TAB / f"strates_{by}.csv", index=False) | |
| 249 | + res[f"strates_{by}"] = st.pivot(index=by, columns="modele", values="mdape").round(2).to_dict("index") | |
| 250 | + res[f"strates_{by}_n"] = st[st.modele == "M7"].set_index(by)["n"].to_dict() | |
| 251 | +# figure : densité | |
| 252 | +fig, ax = plt.subplots(figsize=(6.5, 3.4)) | |
| 253 | +st = pd.read_csv(TAB / "strates_dens.csv") | |
| 254 | +order = ["< 5", "5-19", "20-49", "50-199", "≥ 200"] | |
| 255 | +for j, (name, lab_) in enumerate([("M2", "M2 5 plus proches"), ("M7", "M7 grille hédonique"), ("M0", "M0 meilleure config."), ("M11b", "M11b GBM"), ("M12", "M12 hybride")]): | |
| 256 | + g = st[st.modele == name].set_index("dens").loc[order] | |
| 257 | + ax.plot(order, g.mdape, color=PAL[j], lw=2, marker="o", ms=4, label=lab_) | |
| 258 | +ax.set_xlabel("ventes disponibles dans un rayon de 2 km au cours des 12 mois précédents"); ax.set_ylabel("MdAPE, test (%)"); ax.legend(fontsize=8); ax.set_title("Précision selon la liquidité du sous-marché") | |
| 259 | +savefig(fig, "f5_densite") | |
| 260 | + | |
| 261 | +# ---------------------------------------------------------------- hypothèses | |
| 262 | +H = {} | |
| 263 | +# H1 : proche mais dissemblable vs loin mais semblable (k = 5, T = 12, sans ajustement) | |
| 264 | +H["H1"] = {"geo_d1": surf.loc[e1col("geo", 5, 1, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 265 | + "sim_d10": surf.loc[e1col("sim", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 266 | + "geo_d10": surf.loc[e1col("geo", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 267 | + "mix_d10": surf.loc[e1col("mix", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 268 | + "sim_d10_hedo": surf.loc[e1col("sim", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 269 | + "geo_d10_hedo": surf.loc[e1col("geo", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 270 | + "mix_d10_hedo": surf.loc[e1col("mix", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict()} | |
| 271 | +# H2 : k optimal selon la densité (geo, d = 10, T = 12, hedo et none) | |
| 272 | +h2 = {} | |
| 273 | +for a in ["none", "hedo"]: | |
| 274 | + for lev in order: | |
| 275 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 276 | + cols_ = [e1col("geo", k, 10, 12, a) for k in [1, 3, 5, 7, 10, 15, 20, 30, 50]] | |
| 277 | + md = np.nanmedian(np.abs(err1[m][:, cols_]), axis=0) * 100 | |
| 278 | + h2[f"{a}|{lev}"] = dict(zip([1, 3, 5, 7, 10, 15, 20, 30, 50], md.round(2))) | |
| 279 | +H["H2"] = h2 | |
| 280 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.3), sharey=True) | |
| 281 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 282 | + for j, lev in enumerate(order): | |
| 283 | + v = h2[f"{a}|{lev}"]; ax.plot(list(v), list(v.values()), color=PAL[j], lw=2, marker="o", ms=3.5, label=f"densité {lev}") | |
| 284 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]); ax.set_xlabel("k"); ax.set_title({"none": "temps seul", "hedo": "grille hédonique"}[a] + " — d ≤ 10 km, 12 mois") | |
| 285 | +axes[0].set_ylabel("MdAPE, test (%)"); axes[1].legend(fontsize=8) | |
| 286 | +savefig(fig, "f6_k_par_densite") | |
| 287 | +# H3 : local vs hédonique fixe selon la densité, aux trois ancres | |
| 288 | +h3 = {} | |
| 289 | +for (k, d, Tm) in [(3, 2, 6), (5, 5, 12), (10, 10, 24)]: | |
| 290 | + for lev in order: | |
| 291 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 292 | + h3[f"{k}|{d}|{Tm}|{lev}"] = {a: float(np.nanmedian(np.abs(err2[m][:, e2col("geo", k, d, Tm, a, "equal", "median")])) * 100) for a in ["conv", "hedo", "hedo_reg", "local", "gbm"]} | |
| 293 | +H["H3"] = h3 | |
| 294 | +# H4 : pondération selon la densité (ancre 5/5/12, hedo) | |
| 295 | +h4 = {} | |
| 296 | +for lev in order: | |
| 297 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 298 | + h4[lev] = {f"{w}|{r}": float(np.nanmedian(np.abs(err2[m][:, e2col("geo", 5, 5, 12, "hedo", w, r)])) * 100) for (w, r) in [("equal", "median"), ("equal", "mean"), ("dist", "mean"), ("rec", "mean"), ("sim", "mean"), ("all", "mean"), ("all", "median")]} | |
| 299 | +H["H4"] = h4 | |
| 300 | +# H5 : méthodes simples vs ML selon l'homogénéité (dispersion des prix ajustés des 10 voisins à 10 km / 24 mois) | |
| 301 | +het = sd2[:, e2col("geo", 10, 10, 24, "hedo", "equal", "median")].astype(float) | |
| 302 | +T["heterog"] = pd.qcut(pd.Series(np.where(np.isnan(het), np.nanmax(het), het)), 3, labels=["homogène", "moyen", "hétérogène"]) | |
| 303 | +h5 = {} | |
| 304 | +for lev in ["homogène", "moyen", "hétérogène"]: | |
| 305 | + m = TEST & (T["heterog"] == lev).to_numpy() | |
| 306 | + h5[lev] = {n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY} | |
| 307 | + h5[lev]["sd_median"] = float(np.nanmedian(het[m])) | |
| 308 | +H["H5"] = h5 | |
| 309 | +res["hypotheses"] = H | |
| 310 | + | |
| 311 | +# ---------------------------------------------------------------- intervalles d'incertitude (couverture) | |
| 312 | +intervals = {} | |
| 313 | +for a in ["none", "hedo", "gbm"]: | |
| 314 | + for (k, d, Tm) in [(5, 5, 12), (10, 10, 24)]: | |
| 315 | + c = e2col("geo", k, d, Tm, a, "equal", "median"); e = est2[:, c].astype(float); s = sd2[:, c].astype(float) | |
| 316 | + okv = VAL & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 317 | + z_needed = np.abs(lnP[okv] - e[okv]) / s[okv]; z80 = float(np.percentile(z_needed, 80)) | |
| 318 | + okt = TEST & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 319 | + cov_nominal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= 1.2816 * s[okt]) * 100) | |
| 320 | + cov_cal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= z80 * s[okt]) * 100) | |
| 321 | + width = float(np.median(2 * z80 * s[okt]) * 100) | |
| 322 | + intervals[f"{a}|{k}|{d}|{Tm}"] = dict(z80_val=z80, couverture_nominale_80=cov_nominal, couverture_calibree=cov_cal, largeur_mediane_pct=width, n_test=int(okt.sum())) | |
| 323 | +res["intervalles"] = intervals | |
| 324 | + | |
| 325 | +# ---------------------------------------------------------------- règles conditionnelles (système de recommandation) | |
| 326 | +cond = {}; est_cond = np.full(len(T), np.nan) | |
| 327 | +for lev in order: | |
| 328 | + mv_ = VAL & (T["dens"] == lev).to_numpy(); mt_ = TEST & (T["dens"] == lev).to_numpy() | |
| 329 | + m1 = metrics(err1, mv_); m2 = metrics(err2, mv_) | |
| 330 | + allm = pd.concat([pd.concat([E1, m1], axis=1).assign(src="E1"), pd.concat([E2, m2], axis=1).assign(src="E2")], ignore_index=True) | |
| 331 | + b = allm[allm.couverture >= 95].sort_values("mdape").iloc[0] | |
| 332 | + e = (err1 if b.src == "E1" else err2)[:, int(b.cfg)] | |
| 333 | + est_cond[mt_] = e[mt_] | |
| 334 | + cond[lev] = dict(b[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape", "couverture"]].to_dict(), mdape_test=float(np.nanmedian(np.abs(e[mt_])) * 100), n_test=int(mt_.sum())) | |
| 335 | +res["regles_conditionnelles"] = cond | |
| 336 | +res["regles_conditionnelles_global_test"] = dict(mdape=float(np.nanmedian(np.abs(est_cond[TEST])) * 100), p10=float(np.nanmean(np.abs(est_cond[TEST]) <= 0.1) * 100), couverture=float(np.mean(~np.isnan(est_cond[TEST])) * 100)) | |
| 337 | +pd.DataFrame(cond).T.to_csv(TAB / "regles_conditionnelles.csv") | |
| 338 | + | |
| 339 | +# ---------------------------------------------------------------- stabilité temporelle, indice, divers | |
| 340 | +tq = [] | |
| 341 | +for q, g in T[TEST].groupby("trimestre"): | |
| 342 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 343 | + tq.append(dict(trimestre=q, n=int(m.sum()), **{n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY}, | |
| 344 | + **{"biais_" + n.split(" ")[0]: float(np.nanmedian(ERR[n][m]) * 100) for n in KEY})) | |
| 345 | +pd.DataFrame(tq).to_csv(TAB / "stabilite_trimestres.csv", index=False); res["stabilite"] = tq | |
| 346 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 347 | +idxf = pd.read_csv(OUT / "index_full.csv") | |
| 348 | +for j, (r, lab_) in enumerate([("QC", "Québec"), ("06", "Montréal"), ("03", "Capitale-Nationale"), ("07", "Outaouais"), ("05", "Estrie")]): | |
| 349 | + g = idxf[idxf.region.astype(str).str.zfill(2) == r] if r != "QC" else idxf[idxf.region == "QC"] | |
| 350 | + if len(g): ax.plot(pd.to_datetime(g.ym), np.exp(g.log_idx) * 100, color=PAL[j], lw=2 if r == "QC" else 1.4, label=lab_) | |
| 351 | +ax.set_ylabel("indice hédonique (2021-01 = 100)"); ax.legend(fontsize=8); ax.set_title("Indice de prix mensuel (plein échantillon, descriptif)") | |
| 352 | +savefig(fig, "f7_indice") | |
| 353 | +res["dataset"] = load_json("dataset_stats.json") | |
| 354 | +res["n_targets"] = dict(val=int(VAL.sum()), test=int(TEST.sum())) | |
| 355 | +res["densite_distribution"] = T[TEST]["dens"].value_counts().sort_index().to_dict() | |
| 356 | +res["n_cand_median"] = dict(val=float(np.median(T[VAL].n_cand)), test=float(np.median(T[TEST].n_cand))) | |
| 357 | +save_json(res, "results.json") | |
| 358 | +log("analyse terminée") | |
added
M3U96aomparables-validation/pipeline/06_report.py
+313 −0
@@ -0,0 +1,313 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""06 — Rapport : assemble docs/RAPPORT.md à partir de results.json et des tables (texte + tableaux + figures). | |
| 3 | +Les phrases d'interprétation sont calculées à partir des résultats (jamais de chiffre écrit à la main).""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +R = load_json("results.json"); TAB = OUT / "tables" | |
| 7 | +DOCS = ROOT / "docs"; DOCS.mkdir(exist_ok=True) | |
| 8 | +D = R["dataset"]; comp = R["competition"]; compv = R["competition_val"] | |
| 9 | +def f1(x): return f"{x:.1f}".replace(".", ",") | |
| 10 | +def f2(x): return f"{x:.2f}".replace(".", ",") | |
| 11 | +def n_(x): return f"{int(round(x)):,}".replace(",", " ") | |
| 12 | +def pct(x): return f"{f1(x)} %" | |
| 13 | +def md_table(df, cols=None, fmt=None, index=False): | |
| 14 | + if cols: df = df[cols] | |
| 15 | + if index: df = df.reset_index() | |
| 16 | + fmt = fmt or {} | |
| 17 | + head = "| " + " | ".join(str(c) for c in df.columns) + " |\n|" + "|".join("---" for _ in df.columns) + "|\n" | |
| 18 | + rows = [] | |
| 19 | + for _, r in df.iterrows(): | |
| 20 | + cells = [] | |
| 21 | + for c in df.columns: | |
| 22 | + v = r[c] | |
| 23 | + if c in fmt: cells.append(fmt[c](v)) | |
| 24 | + elif isinstance(v, float): cells.append(f1(v)) | |
| 25 | + else: cells.append(str(v)) | |
| 26 | + rows.append("| " + " | ".join(cells) + " |") | |
| 27 | + return head + "\n".join(rows) + "\n" | |
| 28 | +short = lambda name: name.split(" ")[0] | |
| 29 | +LAB = {"raw": "aucun ajustement", "none": "temps seulement", "conv": "grille conventionnelle", "hedo": "grille hédonique", "hedo_reg": "grille hédonique régionale", | |
| 30 | + "local": "régression locale", "gbm": "ajustements GBM", "role": "ratio au rôle", "geo": "les plus proches (distance)", "sim": "les plus semblables (physique)", | |
| 31 | + "mix": "compromis distance / similarité", "adj": "les moins ajustés", "equal": "égale", "dist": "1/distance", "rec": "récence", "all": "distance × récence × similarité", | |
| 32 | + "median": "médiane", "mean": "moyenne"} | |
| 33 | + | |
| 34 | +# ------------------------------------------------------------------ tableaux | |
| 35 | +excl = pd.DataFrame(D["exclusions"], columns=["Règle", "n"]); excl["n"] = excl["n"].map(n_) | |
| 36 | +cm = pd.DataFrame(comp).T; cm.index.name = "Modèle"; cm = cm.reset_index() | |
| 37 | +cm["val"] = cm["Modèle"].map(lambda m: compv[m]["mdape"]) | |
| 38 | +cm = cm.sort_values("mdape") | |
| 39 | +t_comp = md_table(cm.rename(columns={"mdape": "MdAPE test", "val": "MdAPE val.", "mape": "MAPE", "biais": "Biais (médian)", "p5": "±5 %", "p10": "±10 %", "p20": "±20 %", "couverture": "Couverture"}), | |
| 40 | + ["Modèle", "MdAPE val.", "MdAPE test", "MAPE", "Biais (médian)", "±5 %", "±10 %", "±20 %", "Couverture"]) | |
| 41 | +surf = pd.read_csv(TAB / "surface_E1.csv") | |
| 42 | +def surf_tab(sel, adj, T_): | |
| 43 | + g = surf[(surf.sel == sel) & (surf.adj == adj) & (surf["T"] == T_)].pivot(index="d", columns="k", values="mdape_test") | |
| 44 | + g.index = [f"≤ {d} km" for d in g.index]; g.columns = [f"k = {k}" for k in g.columns]; g.index.name = "rayon" | |
| 45 | + return md_table(g.round(1), index=True) | |
| 46 | +kb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.d == 10) & (surf["T"] == 12)].sort_values("k") | |
| 47 | +fac = pd.read_csv(TAB / "facteurs_E2.csv") | |
| 48 | +anchor = fac[(fac.k == 5) & (fac.d == 5) & (fac["T"] == 12)] | |
| 49 | +t_adj = anchor[(anchor.sel == "geo") & (anchor.w == "equal") & (anchor.recon == "median")].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 50 | +t_adj = t_adj.reset_index(); t_adj["Ajustement"] = t_adj["adj"].map(LAB) | |
| 51 | +t_adj_md = md_table(t_adj.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "mape_test": "MAPE test", "biais_test": "Biais", "p10_test": "±10 %", "couverture_test": "Couverture"}), | |
| 52 | + ["Ajustement", "MdAPE val.", "MdAPE test", "MAPE test", "Biais", "±10 %", "Couverture"]) | |
| 53 | +t_sel = anchor[(anchor.adj == "hedo") & (anchor.w == "equal") & (anchor.recon == "median")].copy(); t_sel["Sélection"] = t_sel["sel"].map(LAB) | |
| 54 | +t_sel_md = md_table(t_sel.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Sélection", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 55 | +t_wr = anchor[(anchor.adj == "hedo") & (anchor.sel == "geo")].copy(); t_wr["Pondération"] = t_wr["w"].map(LAB); t_wr["Réconciliation"] = t_wr["recon"].map(LAB) | |
| 56 | +t_wr_md = md_table(t_wr.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Pondération", "Réconciliation", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 57 | +pairs = pd.read_csv(TAB / "paires_bootstrap.csv") | |
| 58 | +pairs["Comparaison"] = pairs.apply(lambda r: f"{short(r.A)} − {short(r.B)}", axis=1) | |
| 59 | +pairs["Δ MdAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mdape)} [{f2(eval(r.ci_mdape)[0])} ; {f2(eval(r.ci_mdape)[1])}]", axis=1) | |
| 60 | +pairs["Δ MAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mape)} [{f2(eval(r.ci_mape)[0])} ; {f2(eval(r.ci_mape)[1])}]", axis=1) | |
| 61 | +pairs["A plus précis (% des cibles)"] = pairs["part_A_meilleur"].map(f1) | |
| 62 | +t_pairs = md_table(pairs, ["Comparaison", "Δ MdAPE (pts) [IC 95 %]", "Δ MAPE (pts) [IC 95 %]", "A plus précis (% des cibles)", "n"], fmt={"n": n_}) | |
| 63 | +def strata_md(by, label): | |
| 64 | + st = pd.read_csv(TAB / f"strates_{by}.csv") | |
| 65 | + g = st.pivot(index=by, columns="modele", values="mdape").round(1) | |
| 66 | + g["n"] = st[st.modele == "M7"].set_index(by)["n"] | |
| 67 | + g = g[["n", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]] | |
| 68 | + if by == "dens": g = g.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 69 | + if by == "region_nom": g = g.sort_values("n", ascending=False) | |
| 70 | + g.index.name = label | |
| 71 | + return md_table(g, index=True, fmt={"n": n_}) | |
| 72 | +H = R["hypotheses"] | |
| 73 | +h2 = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("hedo")}).T; h2.index.name = "densité"; h2.columns = [f"k = {c}" for c in h2.columns] | |
| 74 | +h2_md = md_table(h2.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 75 | +h2n = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("none")}).T; h2n.index.name = "densité"; h2n.columns = [f"k = {c}" for c in h2n.columns] | |
| 76 | +h2n_md = md_table(h2n.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 77 | +h3 = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("5|5|12")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 78 | +h3.columns = [LAB[c] for c in h3.columns]; h3.index.name = "densité"; h3_md = md_table(h3, index=True) | |
| 79 | +h3s = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("3|2|6")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 80 | +h3s.columns = [LAB[c] for c in h3s.columns]; h3s.index.name = "densité"; h3s_md = md_table(h3s, index=True) | |
| 81 | +h4 = pd.DataFrame(H["H4"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]]; h4.columns = [f"{LAB[c.split('|')[0]]} / {LAB[c.split('|')[1]]}" for c in h4.columns]; h4.index.name = "densité"; h4_md = md_table(h4, index=True) | |
| 82 | +h5 = pd.DataFrame(H["H5"]).T; h5.index.name = "sous-marché"; h5 = h5.rename(columns={"sd_median": "σ médian (log)"}); h5_md = md_table(h5[["σ médian (log)", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]], index=True, fmt={"σ médian (log)": lambda v: f"{v:.3f}".replace(".", ",")}) | |
| 83 | +iv = pd.DataFrame(R["intervalles"]).T; iv.index = [f"{LAB[i.split('|')[0]]}, k = {i.split('|')[1]}, ≤ {i.split('|')[2]} km, {i.split('|')[3]} mois" for i in iv.index]; iv.index.name = "configuration" | |
| 84 | +iv_md = md_table(iv.rename(columns={"z80_val": "z calibré (val.)", "couverture_nominale_80": "couverture à z = 1,28", "couverture_calibree": "couverture calibrée", "largeur_mediane_pct": "largeur médiane (%)", "n_test": "n"}), index=True, fmt={"n": n_, "z calibré (val.)": f2}) | |
| 85 | +rc = pd.DataFrame(R["regles_conditionnelles"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 86 | +rc["règle"] = rc.apply(lambda r: f"{LAB[r.sel]}, k = {int(r.k)}, ≤ {r.d} km, {int(r['T'])} mois, {LAB[r.adj]}, pond. {LAB[r.w]}, {LAB[r.recon]}", axis=1) | |
| 87 | +rc.index.name = "densité"; rc_md = md_table(rc.rename(columns={"mdape": "MdAPE val.", "mdape_test": "MdAPE test", "couverture": "couverture val.", "n_test": "n test"}), ["règle", "MdAPE val.", "MdAPE test", "couverture val.", "n test"], index=True, fmt={"n test": n_}) | |
| 88 | +stab = pd.DataFrame(R["stabilite"]); stab_md = md_table(stab[["trimestre", "n", "M2", "M7", "M0", "M11b", "M12", "biais_M7", "biais_M11b"]].rename(columns={"biais_M7": "biais M7", "biais_M11b": "biais M11b"}), fmt={"n": n_}) | |
| 89 | + | |
| 90 | +# ------------------------------------------------------------------ chiffres clés pour le texte | |
| 91 | +best = min(comp, key=lambda m: comp[m]["mdape"]); best_comps = min([m for m in comp if short(m) in ("M1", "M2", "M3", "M4", "M5", "M6", "M7", "M8a", "M8b", "M13", "M0", "M9")], key=lambda m: comp[m]["mdape"]) | |
| 92 | +m0 = R["M0"]; m0u = R["M0_sans_contrainte"] | |
| 93 | +gain_adj = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M7 grille d'ajustement hédonique"]["mdape"] | |
| 94 | +gain_sel = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M3 sélection par similarité physique"]["mdape"] | |
| 95 | +k_curve = {int(r.k): r.mdape_test for r in kb.itertuples()} | |
| 96 | +k_best = min(k_curve, key=k_curve.get) | |
| 97 | +e1b = R["E1_best_val"] | |
| 98 | +pair = R["paires"] | |
| 99 | +def pr(a, b): return pair[f"{a} vs {b}"] | |
| 100 | +M7 = "M7 grille d'ajustement hédonique"; M2 = "M2 moyenne des 5 plus proches (temps seul)"; M6 = "M6 grille d'ajustement conventionnelle" | |
| 101 | +M8a = "M8a grille locale (régression sur le voisinage)"; M8b = "M8b ajustements non linéaires (GBM)"; M11b = "M11b gradient boosting (avec rôle)"; M11a = "M11a gradient boosting (sans rôle)" | |
| 102 | +M0 = "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"; M12 = "M12 hybride comparables + GBM (poids appris en 2024)"; M13 = "M13 ratio au rôle (5 plus proches, même rôle)"; M9 = "M9 réconciliation optimisée (poids appris en 2024)"; M3 = "M3 sélection par similarité physique"; M1 = "M1 médiane des 3 plus proches (temps seul)"; M10 = "M10 hédonique globale directe" | |
| 103 | +dens = R["strates_dens"]; densn = R["strates_dens_n"] | |
| 104 | +h1 = H["H1"] | |
| 105 | +marg = R["E2_marginal"] | |
| 106 | +cond_g = R["regles_conditionnelles_global_test"] | |
| 107 | +nT = R["n_targets"] | |
| 108 | + | |
| 109 | +txt = f"""--- | |
| 110 | +title: "La méthode des comparables à l'épreuve des données : validation empirique à grande échelle sur {n_(D['propre'])} ventes résidentielles québécoises" | |
| 111 | +subtitle: "Sélection, nombre, ajustements, pondération et réconciliation des ventes comparables — rétro-évaluation point-dans-le-temps sur les transactions Vrai Prix 2021-2026" | |
| 112 | +author: "Simon-Pierre Boucher — version {VERSION}" | |
| 113 | +date: "{time.strftime('%d/%m/%Y')}" | |
| 114 | +lang: fr-CA | |
| 115 | +--- | |
| 116 | + | |
| 117 | +# Résumé | |
| 118 | + | |
| 119 | +**Question.** La méthode de comparaison des ventes, telle qu'enseignée et pratiquée en évaluation immobilière, produit-elle des estimations fiables, et lesquelles de ses règles (choix des comparables, nombre, rayon, ancienneté, ajustements, pondération, réconciliation) sont empiriquement justifiées ? | |
| 120 | + | |
| 121 | +**Données et protocole.** {n_(D['propre'])} ventes de résidences unifamiliales du Québec (registre foncier jumelé au rôle d'évaluation, Vrai Prix, {D['dates'][0]} → {D['dates'][1]}), après un nettoyage journalisé de {n_(D['brut'])} transactions brutes. Pour chacune des **{n_(nT['val'] + nT['test'])} ventes cibles** de 2024 (validation, {n_(nT['val'])}) et de 2025-2026 (test hors échantillon, {n_(nT['test'])}), le prix est masqué et la valeur est estimée avec les seules ventes conclues **avant** la date de la cible, un indice de prix ré-estimé chaque mois avec les seules ventes antérieures, et des grilles d'ajustement et modèles ré-estimés chaque trimestre. **2 256 configurations** de la méthode des comparables et treize modèles de référence sont appliqués aux mêmes cibles ; les différences sont testées par bootstrap par grappes (municipalité × trimestre). | |
| 122 | + | |
| 123 | +**Résultats principaux (test 2025-2026).** | |
| 124 | + | |
| 125 | +1. **Les ajustements sont le facteur dominant.** Prendre la moyenne des cinq ventes les plus proches, ajustées seulement pour le temps (M2), donne une erreur absolue médiane (MdAPE) de **{pct(comp[M2]['mdape'])}** ; une grille d'ajustement hédonique (M7) la ramène à **{pct(comp[M7]['mdape'])}** (−{f1(gain_adj)} points, IC 95 % de la différence de MdAPE [{f2(pr(M7, M2)['ci_mdape'][0])} ; {f2(pr(M7, M2)['ci_mdape'][1])}]). Une grille conventionnelle stylisée (M6) fait presque aussi bien ({pct(comp[M6]['mdape'])}) : ce qui compte, c'est d'ajuster, plus que la finesse de la grille. | |
| 126 | +2. **Choisir des comparables semblables vaut autant qu'ajuster.** Sans aucun ajustement physique, sélectionner les cinq ventes les plus **semblables** (aire, terrain, âge, lien physique, étages) plutôt que les cinq plus **proches** fait passer la MdAPE de {pct(comp[M2]['mdape'])} à **{pct(comp[M3]['mdape'])}** (H1 confirmée : un comparable un peu plus éloigné mais semblable vaut mieux qu'un voisin dissemblable). | |
| 127 | +3. **Le nombre optimal de comparables est bien supérieur à trois.** Avec la grille hédonique, la MdAPE décroît de {pct(k_curve[1])} (k = 1) à {pct(k_curve[3])} (k = 3), {pct(k_curve[5])} (k = 5), {pct(k_curve[10])} (k = 10) et atteint son minimum à **k = {k_best}** ({pct(k_curve[k_best])}) avant de remonter ({pct(k_curve[50])} à k = 50) ; l'optimum dépend de la densité du marché (H2 confirmée). | |
| 128 | +4. **Le rayon et la fenêtre temporelle doivent être larges.** La meilleure configuration de la surface Erreur(k, d, t) choisie en 2024 est « {LAB[e1b['sel']]}, k = {int(e1b['k'])}, ≤ {e1b['d']} km, {int(e1b['T'])} mois, {LAB[e1b['adj']]} » ({pct(e1b['mdape_val'])} en validation → **{pct(e1b['mdape_test'])}** en test). Les rayons de 0,5-1 km et les fenêtres de 3 mois, privilégiés par la pratique, sont dominés dès que les comparables sont ajustés. | |
| 129 | +5. **La pondération et le mode de réconciliation comptent peu** (effets marginaux moyens de {f1(min(marg['w'].values()))} à {f1(max(marg['w'].values()))} points selon la pondération, médiane {f1(marg['recon']['median'])} contre moyenne {f1(marg['recon']['mean'])}), bien moins que la sélection ({f1(min(marg['sel'].values()))}-{f1(max(marg['sel'].values()))}) et l'ajustement ({f1(min(marg['adj'].values()))}-{f1(max(marg['adj'].values()))}). | |
| 130 | +6. **La meilleure méthode des comparables, entièrement transparente et choisie en 2024, atteint {pct(comp[M0]['mdape'])} en 2025-2026** (M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]}), contre **{pct(comp[M11b]['mdape'])}** pour un gradient boosting ré-entraîné chaque trimestre avec la valeur au rôle (M11b) et {pct(comp[M11a]['mdape'])} sans le rôle (M11a). L'hybride comparables + GBM (M12) atteint **{pct(comp[M12]['mdape'])}**. Le coût de la transparence est donc de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''} de MdAPE face au meilleur modèle opaque, et il s'annule dans les sous-marchés homogènes (H5). | |
| 131 | +7. **Le ratio au rôle est un comparable redoutable** : rôle de la cible × ratio prix/rôle médian des cinq plus proches ventes du même rôle (M13) donne {pct(comp[M13]['mdape'])}, mieux que toute grille d'ajustement physique, mais seulement pour {pct(comp[M13]['couverture'])} des cibles. | |
| 132 | +8. **La fiabilité dépend d'abord de la liquidité du sous-marché** : la MdAPE de M7 passe de {pct(dens['≥ 200']['M7'])} là où plus de 200 ventes ont eu lieu à moins de 2 km dans l'année à {pct(dens['< 5']['M7'])} là où il y en a eu moins de 5 ; des règles conditionnelles à la densité (système de recommandation) obtiennent {pct(cond_g['mdape'])} globalement avec {pct(cond_g['couverture'])} de couverture. | |
| 133 | + | |
| 134 | +**Portée.** Nous validons la précision prédictive de règles professionnelles *simulées* sur des caractéristiques observables (aire, terrain, âge, étages, lien physique, rôle) ; les ajustements non observables (garage, piscine, chambres, qualité, rénovations) restent dans l'erreur résiduelle, et le jugement réel d'un évaluateur n'est pas reproduit. | |
| 135 | + | |
| 136 | +# 1. Données | |
| 137 | + | |
| 138 | +## 1.1 Source et nettoyage | |
| 139 | + | |
| 140 | +Instantané de la base de production Vrai Prix pris le 2026-10-10 : {n_(766596)} transactions résidentielles et 3,75 M d'unités d'évaluation. Nous retenons les ventes de type « unifamilial » jumelées à une unité du rôle, et appliquons des règles de nettoyage explicites et journalisées — rien n'est éliminé silencieusement. | |
| 141 | + | |
| 142 | +{md_table(excl)} | |
| 143 | +Jeu propre : **{n_(D['propre'])} ventes** ({D['dates'][0]} → {D['dates'][1]}) ; prix médian {n_(D['prix']['50%'])} $, moyenne {n_(D['prix']['mean'])} $. | |
| 144 | + | |
| 145 | +Deux règles méritent un commentaire. R8 écarte les ratios prix/rôle extrêmes (< 0,33 ou > 3) : faute d'indicateur de lien de dépendance entre les parties, c'est notre proxy de « transaction non représentative » — un évaluateur les écarterait aussi. R6b écarte les ventes conclues l'année même de la construction : leur prix médian ({n_(346108)} $) et leur ratio au rôle (1,15) sont ceux de terrains ou de constructions inachevées, non de maisons achevées (450-480 k$ et 1,3 aux âges 1-4 ans). | |
| 146 | + | |
| 147 | +## 1.2 Répartition | |
| 148 | + | |
| 149 | +| Période | Rôle | Ventes | | |
| 150 | +|---|---|---| | |
| 151 | +| 2021-01 → 2023-12 | développement | {n_(D['par_periode']['train'])} | | |
| 152 | +| 2024 | validation (choix des configurations) | {n_(D['par_periode']['val'])} | | |
| 153 | +| 2025-01 → 2026-09 | test hors échantillon | {n_(D['par_periode']['test'])} | | |
| 154 | + | |
| 155 | +Régions les plus représentées : {', '.join(f"{k} ({n_(v)})" for k, v in list(D['par_region'].items())[:6])}. | |
| 156 | + | |
| 157 | +{{ width=85% }} | |
| 158 | + | |
| 159 | +## 1.3 Caractéristiques disponibles et limites | |
| 160 | + | |
| 161 | +Observées pour chaque vente : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable, superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle en vigueur à la vente et sa part terrain. Non observées : garage, piscine, nombre de chambres et de salles de bains, qualité, état, rénovations. Les grilles d'ajustement testées ne portent donc que sur les premières ; les secondes sont une source d'erreur commune à toutes les méthodes. | |
| 162 | + | |
| 163 | +# 2. Protocole | |
| 164 | + | |
| 165 | +## 2.1 Rétro-évaluation point-dans-le-temps | |
| 166 | + | |
| 167 | +Pour chaque cible : (1) le prix est masqué ; (2) le bassin de comparables est constitué des ventes conclues strictement avant la date de la cible (la cible elle-même, si elle a déjà été vendue, est exclue : on évalue par comparables, non par vente répétée), dans un rayon de 25 km et une fenêtre de 24 mois ; (3) l'indice temporel régional utilisé pour ramener chaque comparable au niveau du dernier mois connu est ré-estimé chaque mois avec les seules ventes antérieures (régression hédonique à effets fixes de municipalité et effets mensuels, par région administrative, rétrécie vers l'indice provincial quand les ventes sont rares) ; (4) les coefficients d'ajustement hédoniques, les grilles régionales et les modèles d'apprentissage sont ré-estimés chaque trimestre avec les seules ventes antérieures au trimestre ; (5) chaque configuration produit une valeur et, le cas échéant, une dispersion des prix ajustés. | |
| 168 | + | |
| 169 | +Le niveau de prix du mois de la cible n'est jamais connu : toutes les méthodes travaillent au niveau du mois précédent, ce qui reproduit la situation réelle d'un évaluateur. | |
| 170 | + | |
| 171 | +## 2.2 Configurations testées | |
| 172 | + | |
| 173 | +**E1 — surface Erreur(k, d, t)** : k ∈ {{1, 3, 5, 7, 10, 15, 20, 30, 50}}, d ∈ {{0,5 ; 1 ; 2 ; 5 ; 10 ; 25 km}}, t ∈ {{3, 6, 12, 24 mois}}, sélection ∈ {{plus proches, plus semblables, compromis}}, ajustement ∈ {{temps seul, grille hédonique}}, médiane. 1 296 configurations. | |
| 174 | + | |
| 175 | +**E2 — plan factoriel des règles** aux ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois) : sélection (4) × ajustement (8 : aucun, temps seul, grille conventionnelle, hédonique globale, hédonique régionale, régression locale, GBM, ratio au rôle) × pondération (5) × réconciliation (2). 960 configurations. | |
| 176 | + | |
| 177 | +*Grille conventionnelle stylisée* (M6) : aire habitable ajustée à la moitié du prix au m² du comparable ; terrain à la moitié de la valeur foncière implicite au m² (plafond ±15 %) ; âge à −0,5 % par année d'écart (plafond ±20 %) ; ±5 % entre détaché et attaché. *Grille hédonique* (M7) : coefficients d'une régression du log du prix sur les caractéristiques, à effets fixes de municipalité (within), ré-estimée chaque trimestre. *Régression locale* (M8a) : mêmes coefficients ré-estimés sur les ventes à moins de 10 km des 24 derniers mois (≥ 40 ventes, ridge vers la grille régionale). *GBM* (M8b) : l'ajustement est la différence des prédictions d'un gradient boosting entre la cible et le comparable. *Ratio au rôle* (M13) : valeur au rôle de la cible × ratio prix/rôle du comparable, pour les comparables de la même municipalité et du même rôle triennal. | |
| 178 | + | |
| 179 | +## 2.3 Mesures | |
| 180 | + | |
| 181 | +Erreur absolue en pourcentage APE = |P̂ − P| / P ; MdAPE (médiane), MAPE (moyenne), biais médian signé, parts à ±5, ±10 et ±20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur — une configuration trop restrictive ne « rate » pas, elle s'abstient, et cela doit se voir). Les différences entre méthodes sont des différences d'erreurs appariées sur les mêmes cibles, avec intervalles de confiance à 95 % par bootstrap par grappes municipalité × trimestre (400 réplications). | |
| 182 | + | |
| 183 | +# 3. Résultats | |
| 184 | + | |
| 185 | +## 3.1 Combien de comparables ? À quelle distance ? De quelle ancienneté ? | |
| 186 | + | |
| 187 | +{{ width=100% }} | |
| 188 | + | |
| 189 | +{{ width=90% }} | |
| 190 | + | |
| 191 | +**Sélection par distance, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 192 | + | |
| 193 | +{surf_tab('geo', 'hedo', 12)} | |
| 194 | +**Sélection par distance, temps seulement, fenêtre 12 mois — MdAPE test (%)** | |
| 195 | + | |
| 196 | +{surf_tab('geo', 'none', 12)} | |
| 197 | +**Sélection par similarité physique, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 198 | + | |
| 199 | +{surf_tab('sim', 'hedo', 12)} | |
| 200 | +Lecture. (i) Sans ajustement, l'erreur décroît avec k et avec le rayon tant que l'on reste sous quelques kilomètres, puis se stabilise ; (ii) avec la grille hédonique, elle décroît nettement plus et l'optimum se déplace vers des k et des rayons plus grands ; (iii) la fenêtre de 3 mois est systématiquement dominée par 12-24 mois dès que l'on ajuste pour le temps. Meilleures configurations de E1 par sélection × ajustement (choisies en 2024, évaluées en 2025-2026) : | |
| 201 | + | |
| 202 | +{md_table(pd.DataFrame(R['E1_best_by_sel_adj']).T.reset_index().rename(columns={'index': 'sélection | ajustement', 'mdape_val': 'MdAPE val.', 'mdape_test': 'MdAPE test', 'couverture_test': 'couverture test'}), ['sélection | ajustement', 'k', 'd', 'T', 'MdAPE val.', 'MdAPE test', 'couverture test'], fmt={'k': lambda v: str(int(v)), 'T': lambda v: str(int(v)), 'd': lambda v: f1(v)})} | |
| 203 | +### k optimal selon la densité (H2) | |
| 204 | + | |
| 205 | +{h2_md} | |
| 206 | +*(sélection par distance, ≤ 10 km, 12 mois, grille hédonique ; MdAPE test en %, par nombre de ventes disponibles à moins de 2 km au cours des 12 mois précédents)* | |
| 207 | + | |
| 208 | +Sans ajustement : | |
| 209 | + | |
| 210 | +{h2n_md} | |
| 211 | +{{ width=100% }} | |
| 212 | + | |
| 213 | +## 3.2 Quels ajustements sont justifiés ? | |
| 214 | + | |
| 215 | +{{ width=90% }} | |
| 216 | + | |
| 217 | +{t_adj_md} | |
| 218 | +Trois enseignements. **L'ajustement temporel seul** vaut {f1(t_adj.set_index('adj').loc['raw', 'mdape_test'] - t_adj.set_index('adj').loc['none', 'mdape_test'])} point ; **n'importe quelle grille physique** vaut {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['conv', 'mdape_test'])} à {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['gbm', 'mdape_test'])} points de plus ; **les raffinements** (grille régionale, régression locale, ajustements non linéaires) apportent peu face à la grille hédonique globale, et la régression locale est même {('moins bonne' if t_adj.set_index('adj').loc['local', 'mdape_test'] > t_adj.set_index('adj').loc['hedo', 'mdape_test'] else 'meilleure')} ({pct(t_adj.set_index('adj').loc['local', 'mdape_test'])} contre {pct(t_adj.set_index('adj').loc['hedo', 'mdape_test'])}). | |
| 219 | + | |
| 220 | +### Grilles locales et petits échantillons (H3) | |
| 221 | + | |
| 222 | +MdAPE test (%) par type d'ajustement et densité, ancre (5, 5 km, 12 mois) : | |
| 223 | + | |
| 224 | +{h3_md} | |
| 225 | +Ancre (3, 2 km, 6 mois) : | |
| 226 | + | |
| 227 | +{h3s_md} | |
| 228 | +## 3.3 Comment choisir les comparables ? (H1) | |
| 229 | + | |
| 230 | +Ancre (5, 5 km, 12 mois), grille hédonique, médiane : | |
| 231 | + | |
| 232 | +{t_sel_md} | |
| 233 | +Test direct de H1 (k = 5, 12 mois, temps seulement) : les cinq voisins à moins de 1 km donnent une MdAPE de {pct(h1['geo_d1']['mdape_test'])} (couverture {pct(h1['geo_d1']['couverture_test'])}) ; les cinq ventes les plus semblables à moins de 10 km, {pct(h1['sim_d10']['mdape_test'])} (couverture {pct(h1['sim_d10']['couverture_test'])}) ; les cinq plus proches à moins de 10 km, {pct(h1['geo_d10']['mdape_test'])}. Avec la grille hédonique : plus proches {pct(h1['geo_d10_hedo']['mdape_test'])}, plus semblables {pct(h1['sim_d10_hedo']['mdape_test'])}, compromis {pct(h1['mix_d10_hedo']['mdape_test'])}. | |
| 234 | + | |
| 235 | +## 3.4 Pondération et réconciliation (H4) | |
| 236 | + | |
| 237 | +Ancre (5, 5 km, 12 mois), sélection par distance, grille hédonique : | |
| 238 | + | |
| 239 | +{t_wr_md} | |
| 240 | +Selon la densité (MdAPE test, %) : | |
| 241 | + | |
| 242 | +{h4_md} | |
| 243 | +Effets marginaux moyens (MdAPE test moyenne sur toutes les autres modalités de E2) : sélection {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['sel'].items())} ; ajustement {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['adj'].items())} ; pondération {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['w'].items())} ; réconciliation {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['recon'].items())}. | |
| 244 | + | |
| 245 | +## 3.5 Compétition de modèles | |
| 246 | + | |
| 247 | +{{ width=95% }} | |
| 248 | + | |
| 249 | +{t_comp} | |
| 250 | +M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]} (meilleure des 2 256 configurations en 2024 parmi celles qui couvrent ≥ 97 % des cibles ; sans cette contrainte, la meilleure est « {LAB[m0u['sel']]}, k = {int(m0u['k'])}, ≤ {m0u['d']} km, {int(m0u['T'])} mois, {LAB[m0u['adj']]} » à {pct(m0u['mdape_test'])} mais pour {pct(m0u['couverture_test'])} des cibles seulement). Poids appris de M9 : {', '.join(f"{LAB[k]} {f2(v)}" for k, v in R['M9_weights'].items() if v > 0)}. Poids de M12 : {', '.join(f"{k} {f2(v)}" for k, v in R['M12_weights'].items() if v > 0)}. | |
| 251 | + | |
| 252 | +### Différences appariées (test ; bootstrap par grappes municipalité × trimestre) | |
| 253 | + | |
| 254 | +{t_pairs} | |
| 255 | +Un Δ négatif signifie que A est plus précis que B ; l'intervalle qui exclut zéro signale une différence robuste aux dépendances spatiales et temporelles. | |
| 256 | + | |
| 257 | +## 3.6 Où la méthode tient-elle, où défaille-t-elle ? | |
| 258 | + | |
| 259 | +{{ width=85% }} | |
| 260 | + | |
| 261 | +**Par densité** (ventes à moins de 2 km dans les 12 mois précédents) — MdAPE test (%) : | |
| 262 | + | |
| 263 | +{strata_md('dens', 'densité')} | |
| 264 | +**Par région administrative** : | |
| 265 | + | |
| 266 | +{strata_md('region_nom', 'région')} | |
| 267 | +**Par quintile de prix** : | |
| 268 | + | |
| 269 | +{strata_md('prix_q', 'quintile de prix')} | |
| 270 | +**Par âge du bâtiment** : | |
| 271 | + | |
| 272 | +{strata_md('age_cl', 'âge')} | |
| 273 | +**Par lien physique** (0 détaché, 1 jumelé, 2 en rangée, 3 intégré) : | |
| 274 | + | |
| 275 | +{strata_md('lien', 'lien physique')} | |
| 276 | +### Marchés homogènes et hétérogènes (H5) | |
| 277 | + | |
| 278 | +Terciles de la dispersion (écart-type du log des prix ajustés) des dix ventes les plus proches à ≤ 10 km / 24 mois — MdAPE test (%) : | |
| 279 | + | |
| 280 | +{h5_md} | |
| 281 | +### Stabilité dans le temps | |
| 282 | + | |
| 283 | +{stab_md} | |
| 284 | +Le biais trimestriel commun à toutes les méthodes reflète le fait que le niveau de prix du mois courant est inconnu (toutes travaillent au niveau du mois précédent) et la saisonnalité des prix. | |
| 285 | + | |
| 286 | +## 3.7 Intervalles de valeur | |
| 287 | + | |
| 288 | +La dispersion des prix ajustés des comparables retenus fournit un intervalle de valeur. Avec z = 1,28 (couverture nominale de 80 % si les erreurs étaient normales), puis avec un z calibré sur la validation 2024 : | |
| 289 | + | |
| 290 | +{iv_md} | |
| 291 | +## 3.8 Vers un système de recommandation des comparables | |
| 292 | + | |
| 293 | +Règle choisie en 2024 pour chaque strate de densité (parmi les 2 256 configurations couvrant ≥ 95 % des cibles de la strate), évaluée en 2025-2026 : | |
| 294 | + | |
| 295 | +{rc_md} | |
| 296 | +Globalement, ces règles conditionnelles atteignent une MdAPE de **{pct(cond_g['mdape'])}** ({pct(cond_g['p10'])} des estimations à ±10 %, couverture {pct(cond_g['couverture'])}), contre {pct(comp[M0]['mdape'])} pour la meilleure règle unique. | |
| 297 | + | |
| 298 | +# 4. Discussion | |
| 299 | + | |
| 300 | +**Ce que la méthode des comparables fait bien.** Correctement paramétrée — comparables semblables, nombre d'une dizaine, rayon et fenêtre larges, ajustements hédoniques ou même conventionnels — elle atteint une précision ({pct(comp[M0]['mdape'])}) que ne dépasse le meilleur modèle d'apprentissage ré-entraîné chaque trimestre que de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''}, tout en restant intégralement explicable : chaque valeur est la médiane de quelques ventes identifiées, ajustées par des coefficients lisibles. | |
| 301 | + | |
| 302 | +**Ce qui contredit la pratique.** Trois comparables, un rayon de quelques centaines de mètres et une fenêtre de trois mois — les conventions les plus répandues — sont dominés sur toute la surface Erreur(k, d, t) : {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].mdape_test.iloc[0])} de MdAPE (k = 3, ≤ 1 km, 3 mois, grille hédonique, couverture {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].couverture_test.iloc[0])}) contre {pct(e1b['mdape_test'])} pour la meilleure configuration. La rareté des comparables « parfaits » est un argument pour élargir la recherche et ajuster, non pour se restreindre. | |
| 303 | + | |
| 304 | +**Le rôle d'évaluation comme information.** Le ratio au rôle (M13) et le GBM avec rôle (M11b) dominent leurs équivalents sans rôle : la valeur au rôle encapsule des caractéristiques non observées ici (qualité, dépendances, état) et une appréciation locale que les modèles globaux ne captent pas. Les modèles globaux sans rôle (M10, M11a) présentent d'ailleurs un biais négatif en 2025-2026 : l'appréciation est spatialement hétérogène et un indice régional, même ré-estimé chaque trimestre, la sous-estime là où les ventes se concentrent. | |
| 305 | + | |
| 306 | +**Limites.** (1) Règles professionnelles simulées, non dossiers d'évaluateurs. (2) Caractéristiques limitées à celles du rôle et du registre. (3) Délai de publication des ventes ignoré. (4) Un seul type de bien (unifamilial) ; condos et plex restent à traiter. (5) Les configurations ont été choisies sur 2024 puis évaluées sur 2025-2026 : les écarts entre validation et test (tableaux ci-dessus) mesurent la stabilité des règles, et ils sont faibles. | |
| 307 | + | |
| 308 | +# 5. Reproductibilité | |
| 309 | + | |
| 310 | +Code : `pipeline/01_dataset.py` → `06_report.py` (Python 3.14, pandas, numpy, scipy, lightgbm, matplotlib) ; sorties versionnées `out/{VERSION}/` (configurations `configs_E1.csv`, `configs_E2.csv` ; estimations `est_E1.npy`, `est_E2.npy` ; cibles `targets.parquet` ; tables `tables/*.csv` ; `results.json`). Exécution : `./run_all.sh` sur le nœud de calcul (M2U64), `./make_pdf.sh` sur le poste de travail. | |
| 311 | +""" | |
| 312 | +(DOCS / "RAPPORT.md").write_text(txt, encoding="utf-8") | |
| 313 | +log(f"rapport écrit : {DOCS / 'RAPPORT.md'} ({len(txt.split()):,} mots)") | |
added
M3U96aomparables-validation/pipeline/common.py
+85 −0
@@ -0,0 +1,85 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""Validation empirique de la méthode des comparables — fonctions communes. | |
| 3 | +Données : instantané Vrai Prix (transactions ⋈ units) du 2026-10-10. | |
| 4 | +Toutes les sorties sont versionnées sous out/<version>/. | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | +import json, math, os, sqlite3, time | |
| 8 | +from pathlib import Path | |
| 9 | +import numpy as np | |
| 10 | +import pandas as pd | |
| 11 | + | |
| 12 | +ROOT = Path(os.environ.get("CV_ROOT", Path(__file__).resolve().parents[1])) | |
| 13 | +DB = Path(os.environ.get("CV_DB", ROOT / "data" / "vraiprix-20261010.db")) | |
| 14 | +REF = ROOT / "ref" | |
| 15 | +VERSION = os.environ.get("CV_VERSION", time.strftime("%Y-%m-%d")) | |
| 16 | +OUT = ROOT / "out" / VERSION | |
| 17 | +OUT.mkdir(parents=True, exist_ok=True) | |
| 18 | +(OUT / "fig").mkdir(exist_ok=True) | |
| 19 | + | |
| 20 | +# Périodes (séparation chronologique) | |
| 21 | +TRAIN_END = "2023-12-31" # développement | |
| 22 | +VAL_START, VAL_END = "2024-01-01", "2024-12-31" # choix des configurations | |
| 23 | +TEST_START, TEST_END = "2025-01-01", "2026-09-29" # évaluation finale hors échantillon | |
| 24 | +MONTH0 = "2021-01" # premier mois des données | |
| 25 | + | |
| 26 | +def log(msg: str) -> None: | |
| 27 | + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) | |
| 28 | + | |
| 29 | +def connect(path: Path = DB) -> sqlite3.Connection: | |
| 30 | + return sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=60) | |
| 31 | + | |
| 32 | +def _json_default(o): | |
| 33 | + if isinstance(o, np.integer): return int(o) | |
| 34 | + if isinstance(o, np.floating): return None if np.isnan(o) else float(o) | |
| 35 | + if isinstance(o, np.bool_): return bool(o) | |
| 36 | + if isinstance(o, np.ndarray): return o.tolist() | |
| 37 | + if isinstance(o, pd.Timestamp): return o.strftime("%Y-%m-%d") | |
| 38 | + if isinstance(o, float) and math.isnan(o): return None | |
| 39 | + return str(o) | |
| 40 | + | |
| 41 | +def save_json(obj, name: str) -> Path: | |
| 42 | + p = OUT / name; p.parent.mkdir(parents=True, exist_ok=True) | |
| 43 | + with open(p, "w", encoding="utf-8") as f: | |
| 44 | + json.dump(obj, f, ensure_ascii=False, indent=2, default=_json_default) | |
| 45 | + return p | |
| 46 | + | |
| 47 | +def load_json(name: str): | |
| 48 | + with open(OUT / name, encoding="utf-8") as f: | |
| 49 | + return json.load(f) | |
| 50 | + | |
| 51 | +def save_parquet(df: pd.DataFrame, name: str) -> Path: | |
| 52 | + p = OUT / name; df.to_parquet(p, index=False); return p | |
| 53 | + | |
| 54 | +def load_parquet(name: str) -> pd.DataFrame: | |
| 55 | + return pd.read_parquet(OUT / name) | |
| 56 | + | |
| 57 | +def month_index(dates) -> np.ndarray: | |
| 58 | + """Mois entier depuis MONTH0 (2021-01 = 0).""" | |
| 59 | + d = pd.to_datetime(dates) | |
| 60 | + y0, m0 = map(int, MONTH0.split("-")) | |
| 61 | + return ((d.dt.year - y0) * 12 + (d.dt.month - m0)).to_numpy() | |
| 62 | + | |
| 63 | +def to_xy_km(lat, lng, lat0=46.8, lng0=-71.2): | |
| 64 | + """Projection équirectangulaire locale (km) : erreur < 1 % pour des distances < 50 km.""" | |
| 65 | + x = (np.asarray(lng, float) - lng0) * 111.32 * np.cos(np.radians(np.asarray(lat, float))) | |
| 66 | + y = (np.asarray(lat, float) - lat0) * 110.57 | |
| 67 | + return x, y | |
| 68 | + | |
| 69 | +def load_repertoire() -> pd.DataFrame: | |
| 70 | + mun = pd.read_csv(REF / "MUN.csv", dtype=str, encoding="utf-8-sig") | |
| 71 | + mun = mun.rename(columns=str.lower) | |
| 72 | + out = pd.DataFrame({ | |
| 73 | + "code_mun": mun["mcode"].str.zfill(5), | |
| 74 | + "mun_nom": mun["munnom"], | |
| 75 | + "region": mun["regadm"].str.extract(r"\((\d+)\)")[0], | |
| 76 | + "region_nom": mun["regadm"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 77 | + "mrc": mun["mrc"].str.extract(r"\((\d+)\)")[0], | |
| 78 | + "mrc_nom": mun["mrc"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 79 | + }) | |
| 80 | + return out.drop_duplicates("code_mun") | |
| 81 | + | |
| 82 | +# Codages catégoriels (ordre stable) | |
| 83 | +LIEN = {"Détaché": 0, "Jumelé": 1, "En rangée 1 côté": 2, "En rangée plus de 1 côté": 2, "Intégré": 3} | |
| 84 | +GENRE = {"De plain-pied": 0, "Unimodulaire": 0, "À étages entiers": 1, "À niveaux décalés": 2, "À étage mansardé": 3} | |
| 85 | +BTYPE = {"single-story": 0, "full-story": 1, "mansard": 2, "split-level": 3, "single-wide": 4} | |
added
M3U96aomparables-validation/run_all.sh
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Chaîne complète sur le nœud de calcul (M2U64). Usage : CV_VERSION=2026-10-10 ./run_all.sh [étape de départ] | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +PY=${PY:-$HOME/uqo-eval-terrains/env/bin/python} | |
| 6 | +export CV_VERSION=${CV_VERSION:-$(date +%F)} | |
| 7 | +START=${1:-1} | |
| 8 | +for s in 01_dataset 02_index 03_models 04_backtest 05_analyze 06_report; do | |
| 9 | + n=${s%%_*} | |
| 10 | + [ "$((10#$n))" -lt "$START" ] && continue | |
| 11 | + echo "=== $s ($(date +%T)) ===" | |
| 12 | + (cd pipeline && $PY $s.py) | |
| 13 | +done | |
| 14 | +echo "=== terminé $(date +%T) ===" | |
added
M3U96bomparables-validation/.gitignore
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +out/ | |
| 2 | +data/ | |
| 3 | +*.pyc | |
| 4 | +__pycache__/ | |
| 5 | +.DS_Store | |
added
M3U96bomparables-validation/README.md
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +# Validation empirique de la méthode des comparables (Vrai Prix) | |
| 2 | + | |
| 3 | +Projet de recherche : rétro-évaluation à grande échelle de la méthode de comparaison des ventes sur les transactions | |
| 4 | +résidentielles unifamiliales du Québec (Vrai Prix, 2021-01 → 2026-09). Protocole : `docs/PROTOCOLE.md`. | |
| 5 | +Rapport de résultats : `docs/RAPPORT.md` / `docs/RAPPORT.pdf`. | |
| 6 | + | |
| 7 | +## Organisation | |
| 8 | + | |
| 9 | +``` | |
| 10 | +pipeline/ 01_dataset → 02_index → 03_models → 04_backtest → 05_analyze → 06_report | |
| 11 | +docs/ PROTOCOLE.md, RAPPORT.md(.pdf), header.tex | |
| 12 | +out/<version>/ sales.parquet, index_asof.npy, models_*.json, est_E*.npy, targets.parquet, tables/, fig/, results.json | |
| 13 | +data/ (nœud seulement) vraiprix-20261010.db — instantané de la base de production Vrai Prix | |
| 14 | +ref/ MUN.csv (répertoire des municipalités MAMH : code → MRC → région) | |
| 15 | +``` | |
| 16 | + | |
| 17 | +## Exécution | |
| 18 | + | |
| 19 | +Le calcul se fait sur **M2U64** (`~/comparables-validation`, venv `~/uqo-eval-terrains/env`, Python 3.14 : pandas, numpy, scipy, | |
| 20 | +lightgbm, matplotlib). Le laptop garde le code, les docs et une copie des sorties. | |
| 21 | + | |
| 22 | +```bash | |
| 23 | +rsync -a --exclude out --exclude data --exclude .git ~/Desktop/Validation-Comparables/ M2U64:comparables-validation/ | |
| 24 | +ssh M2U64 'cd ~/comparables-validation && CV_VERSION=2026-10-10 ./run_all.sh' # ~ 30 min | |
| 25 | +./make_pdf.sh # rapatrie out/ + docs/ et compile le PDF | |
| 26 | +``` | |
| 27 | + | |
| 28 | +Variables : `CV_VERSION` (dossier de sortie), `CV_SAMPLE=3000` (échantillon de cibles pour un essai), `CV_PROCS` (processus, 20 par défaut). | |
| 29 | + | |
| 30 | +## Données | |
| 31 | + | |
| 32 | +Instantané `vraiprix.db` pris sur M3U96a (`~/apps/vrai-prix/data`) le 2026-10-10 : 766 596 transactions, 3,75 M d'unités du rôle. | |
| 33 | +Jeu propre : ventes `unifamilial` jumelées au rôle, règles R0-R11 journalisées dans `out/<v>/exclusions.csv`. | |
added
M3U96bomparables-validation/docs/PROTOCOLE.md
+101 −0
@@ -0,0 +1,101 @@ | ||
| 1 | +# Revisiter la méthode de comparaison des ventes : protocole de validation empirique à grande échelle | |
| 2 | + | |
| 3 | +**Titre de travail (article A)** — *Revisiting the Sales Comparison Approach: A Large-Scale Empirical Validation of Comparable Selection, Price Adjustments, and Value Reconciliation* | |
| 4 | + | |
| 5 | +**Variante** — *Does the Sales Comparison Approach Actually Work? Evidence from Large-Scale Residential Property Transactions* | |
| 6 | + | |
| 7 | +Données : transactions historiques **Vrai Prix** (registre foncier du Québec jumelé au rôle d'évaluation), instantané du 2026-10-10. | |
| 8 | + | |
| 9 | +## 1. Question de recherche | |
| 10 | + | |
| 11 | +La méthode des comparables, telle qu'enseignée et pratiquée par les évaluateurs agréés, produit-elle réellement les estimations les plus fiables ? Et surtout, quelles règles de **sélection**, d'**ajustement**, de **pondération** et de **réconciliation** des comparables sont empiriquement justifiées ? | |
| 12 | + | |
| 13 | +| Dimension | Question testée | Où dans le pipeline | | |
| 14 | +|---|---|---| | |
| 15 | +| Sélection | Quels comparables sont réellement les plus pertinents (proximité, similarité physique, « moins ajustés ») ? | E1 (facteur `sel`), E2 | | |
| 16 | +| Nombre | Faut-il 1, 3, 5, 10, 20 ou 50 ventes ? | E1 (facteur `k`) | | |
| 17 | +| Ajustements | Quels ajustements sont statistiquement justifiés (grille conventionnelle, hédonique, locale, non linéaire, ratio au rôle) ? | E2 (facteur `adj`) | | |
| 18 | +| Pondération | Égalitaire, distance, récence, similarité ? | E2 (facteur `w`) | | |
| 19 | +| Réconciliation | Médiane, moyenne, moyenne pondérée, combinaison optimisée (M9) ? | E2 (facteur `recon`), M9 | | |
| 20 | +| Fiabilité | Dans quels marchés (densité, région, segment de prix, hétérogénéité) la méthode tient-elle ou défaille-t-elle ? | strates, H2-H5 | | |
| 21 | + | |
| 22 | +## 2. Protocole expérimental (rétro-évaluation « point dans le temps ») | |
| 23 | + | |
| 24 | +Pour chaque transaction cible : | |
| 25 | + | |
| 26 | +1. **Masquage** du prix de vente réel. | |
| 27 | +2. **Reconstitution du marché à la date d'évaluation** : seules les ventes conclues *strictement avant* la date de la cible sont admissibles (le bien lui-même, s'il a déjà été vendu, est exclu du bassin : on évalue par comparables, pas par vente répétée). L'indice temporel, les coefficients hédoniques et les modèles d'apprentissage sont ré-estimés avec les seules ventes antérieures à la période d'évaluation (indice : ré-estimé chaque mois avec toutes les ventes des mois précédents ; modèles : estimés sur les ventes < 2024-01 pour la validation 2024 et < 2025-01 pour le test 2025-26). | |
| 28 | +3. **Application** de chaque configuration (sélection × rayon × fenêtre × nombre × ajustement × pondération × réconciliation) sur exactement les mêmes cibles. | |
| 29 | +4. **Comparaison** avec le prix réel : erreur absolue en pourcentage, biais, part des estimations à ±5/10/20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur), dispersion des prix ajustés (intervalle de valeur). | |
| 30 | + | |
| 31 | +### Séparation chronologique | |
| 32 | + | |
| 33 | +| Période | Rôle | Ventes | | |
| 34 | +|---|---|---| | |
| 35 | +| 2021-01 → 2023-12 | développement (estimation des grilles d'ajustement et des modèles) | ~ 200 000 | | |
| 36 | +| 2024 | validation : choix des configurations et des hyperparamètres | ~ 65 000 cibles | | |
| 37 | +| 2025-01 → 2026-09 | évaluation finale hors échantillon | ~ 125 000 cibles | | |
| 38 | + | |
| 39 | +Les comparables d'une cible de 2025 peuvent être des ventes de 2024 ou de 2025 antérieures à sa date : c'est l'information dont disposait un évaluateur ce jour-là (nous ignorons le délai de publication au registre foncier, ce qui avantage légèrement toutes les méthodes de la même façon). | |
| 40 | + | |
| 41 | +## 3. Expériences | |
| 42 | + | |
| 43 | +### E1 — surface empirique de performance Erreur(k, d, t) | |
| 44 | + | |
| 45 | +- k ∈ {1, 3, 5, 7, 10, 15, 20, 30, 50} comparables ; | |
| 46 | +- d ∈ {0,5 ; 1 ; 2 ; 5 ; 10 ; 25} km (rayon maximal) ; | |
| 47 | +- t ∈ {3, 6, 12, 24} mois (ancienneté maximale) ; | |
| 48 | +- sélection ∈ {`geo` les plus proches, `sim` les plus semblables physiquement, `mix` compromis} ; | |
| 49 | +- ajustement ∈ {`none` temps seulement, `hedo` grille hédonique} ; réconciliation = médiane. | |
| 50 | + | |
| 51 | +1 296 configurations × toutes les cibles. | |
| 52 | + | |
| 53 | +### E2 — plan factoriel des règles (trois ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois)) | |
| 54 | + | |
| 55 | +- sélection ∈ {`geo`, `sim`, `mix`, `adj` = comparables nécessitant le moins d'ajustements} ; | |
| 56 | +- ajustement ∈ {`raw` aucun, `none` temps seul, `conv` grille conventionnelle stylisée, `hedo` grille hédonique globale (within-municipalité), `hedo_reg` grille régionale rétrécie, `local` régression sur le voisinage (≤ 10 km, ≥ 40 ventes, ridge vers la grille régionale), `gbm` ajustements non linéaires (LightGBM), `role` ratio prix/rôle (comparables de la même municipalité et du même rôle)} ; | |
| 57 | +- pondération ∈ {égale, 1/(0,25 + d), exp(−mois/6), 1/(0,1 + ajustement brut), produit des trois} ; | |
| 58 | +- réconciliation ∈ {médiane (pondérée), moyenne (pondérée) en dollars}. | |
| 59 | + | |
| 60 | +960 configurations. | |
| 61 | + | |
| 62 | +### Compétition de modèles | |
| 63 | + | |
| 64 | +| Modèle | Définition opérationnelle | | |
| 65 | +|---|---| | |
| 66 | +| M1 | médiane des 3 ventes les plus proches (≤ 2 km, ≤ 6 mois), temps seul | | |
| 67 | +| M2 | moyenne des 5 plus proches (≤ 5 km, ≤ 12 mois), temps seul | | |
| 68 | +| M3 | 5 comparables choisis par similarité physique | | |
| 69 | +| M4 | 5 plus proches pondérés par la distance | | |
| 70 | +| M5 | 5 plus proches pondérés par la récence | | |
| 71 | +| M6 | grille d'ajustement conventionnelle stylisée (½ du prix au m² pour l'aire, ½ de la valeur foncière au m² pour le terrain plafonnée à 15 %, −0,5 %/an d'âge plafonné à 20 %, ±5 % détaché/attaché) | | |
| 72 | +| M7 | grille d'ajustement hédonique (coefficients within-municipalité estimés sur les ventes antérieures) | | |
| 73 | +| M8a / M8b | grille locale (régression sur le voisinage) / ajustements non linéaires (GBM) | | |
| 74 | +| M9 | réconciliation optimisée : combinaison convexe (NNLS) de sept estimations par comparables, poids appris en 2024 | | |
| 75 | +| M10 | modèle hédonique global direct (β within + effets région, MRC, municipalité, unité de voisinage) | | |
| 76 | +| M11a / M11b | gradient boosting sans / avec la valeur au rôle | | |
| 77 | +| M12 | hybride : combinaison convexe de M11b et des estimations par comparables ajustées (GBM, hédonique) | | |
| 78 | +| M13 | ratio au rôle : rôle de la cible × ratio prix/rôle médian des 5 plus proches du même rôle | | |
| 79 | +| M0 | meilleure configuration de E1 ∪ E2 choisie sur la validation 2024, évaluée en 2025-26 | | |
| 80 | + | |
| 81 | +### Hypothèses | |
| 82 | + | |
| 83 | +- **H1** — un comparable très proche mais peu semblable peut être moins informatif qu'un comparable plus éloigné mais très semblable. | |
| 84 | +- **H2** — l'ajout de comparables améliore la précision jusqu'à un seuil, au-delà duquel l'hétérogénéité domine ; le seuil dépend de la densité. | |
| 85 | +- **H3** — les ajustements empiriques locaux surpassent les grilles fixes mais deviennent instables dans les petits échantillons. | |
| 86 | +- **H4** — la pondération optimale dépend de la liquidité du sous-marché. | |
| 87 | +- **H5** — les méthodes simples et transparentes rivalisent avec l'apprentissage automatique dans les marchés homogènes. | |
| 88 | + | |
| 89 | +### Inférence | |
| 90 | + | |
| 91 | +Différences d'erreurs appariées (mêmes cibles) avec intervalles de confiance par **bootstrap par grappes** (municipalité × trimestre), qui respecte les dépendances spatiales et temporelles. | |
| 92 | + | |
| 93 | +## 4. Ce que les données permettent — et ne permettent pas | |
| 94 | + | |
| 95 | +Caractéristiques observées pour chaque vente (registre foncier ⋈ rôle) : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable (étages), superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle (en vigueur à la vente) et sa part terrain. **Non observés** : garage, piscine, nombre de chambres et de salles de bains, qualité et état, rénovations. Les ajustements correspondants ne peuvent donc pas être testés ici ; les erreurs résiduelles les contiennent. | |
| 96 | + | |
| 97 | +Nous validons la **précision prédictive** de règles professionnelles *simulées*, pas le jugement réel d'un évaluateur (il faudrait ses dossiers, ses comparables et ses ajustements). La distinction est maintenue partout dans le rapport. | |
| 98 | + | |
| 99 | +## 5. Article B (suite envisagée) | |
| 100 | + | |
| 101 | +Vers une méthode des comparables optimisée empiriquement : algorithme transparent de sélection/ajustement entraîné sur les ventes historiques et validé sur les ventes futures ; règles conditionnelles au type de bien, à la densité et à la qualité des données ; intégration dans UQO Éval pour montrer aux étudiants la comparaison entre règles traditionnelles et règles estimées. | |
added
M3U96bomparables-validation/docs/header.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% En-tête LaTeX — Grille empirique d'ajustement des terrains (UQO Éval) | |
| 2 | +\usepackage{fontspec} | |
| 3 | +\setmainfont{Helvetica Neue}[Ligatures=TeX] | |
| 4 | +\setsansfont{Helvetica Neue} | |
| 5 | +\setmonofont{Menlo}[Scale=0.82] | |
| 6 | +\usepackage{newunicodechar} | |
| 7 | +\newunicodechar{→}{\ensuremath{\rightarrow}}\newunicodechar{≥}{\ensuremath{\geq}}\newunicodechar{≤}{\ensuremath{\leq}}\newunicodechar{×}{\ensuremath{\times}}\newunicodechar{−}{\ensuremath{-}}\newunicodechar{≈}{\ensuremath{\approx}}\newunicodechar{∂}{\ensuremath{\partial}}\newunicodechar{⋈}{\ensuremath{\bowtie}}\newunicodechar{Δ}{\ensuremath{\Delta}}\newunicodechar{τ}{\ensuremath{\tau}}\newunicodechar{σ}{\ensuremath{\sigma}}\newunicodechar{β}{\ensuremath{\beta}}\newunicodechar{θ}{\ensuremath{\theta}}\newunicodechar{γ}{\ensuremath{\gamma}}\newunicodechar{α}{\ensuremath{\alpha}}\newunicodechar{δ}{\ensuremath{\delta}}\newunicodechar{ε}{\ensuremath{\varepsilon}}\newunicodechar{²}{\textsuperscript{2}}\newunicodechar{⇒}{\ensuremath{\Rightarrow}}\newunicodechar{▶}{\ensuremath{\blacktriangleright}}\newunicodechar{│}{|}\newunicodechar{▼}{\ensuremath{\blacktriangledown}}\newunicodechar{─}{-}\newunicodechar{┌}{+}\newunicodechar{┐}{+}\newunicodechar{└}{+}\newunicodechar{┘}{+}\newunicodechar{├}{+}\newunicodechar{┤}{+} | |
| 8 | +\usepackage{xcolor} | |
| 9 | +\definecolor{uqo}{HTML}{006085}\definecolor{turq}{HTML}{2BCCAE}\definecolor{corail}{HTML}{EA6047}\definecolor{encre}{HTML}{16181B}\definecolor{papier}{HTML}{F4F5F3} | |
| 10 | +\usepackage{titlesec} | |
| 11 | +\titleformat{\section}{\Large\bfseries\color{uqo}}{}{0em}{}[{\color{uqo}\titlerule[0.8pt]}] | |
| 12 | +\titleformat{\subsection}{\large\bfseries\color{uqo}}{}{0em}{} | |
| 13 | +\titlespacing*{\section}{0pt}{18pt plus 4pt}{8pt} | |
| 14 | +\usepackage{etoolbox} | |
| 15 | +\AtBeginEnvironment{longtable}{\scriptsize\renewcommand{\arraystretch}{1.18}} | |
| 16 | +\usepackage{colortbl}\arrayrulecolor{uqo!35} | |
| 17 | +\usepackage{tcolorbox} | |
| 18 | +\renewenvironment{quote}{\begin{tcolorbox}[colback=papier,colframe=turq,boxrule=0pt,leftrule=3pt,arc=1pt,left=6pt,right=6pt,top=4pt,bottom=4pt]}{\end{tcolorbox}} | |
| 19 | +\usepackage{fancyhdr} | |
| 20 | +\pagestyle{fancy}\fancyhf{} | |
| 21 | +\fancyhead[L]{\footnotesize\color{gray}Grille empirique d'ajustement des terrains — UQO Éval}\fancyhead[R]{\footnotesize\color{gray}IMM1003 · IMM1033} | |
| 22 | +\fancyfoot[C]{\footnotesize\color{gray}\thepage}\renewcommand{\headrulewidth}{0.3pt} | |
| 23 | +\usepackage{caption}\captionsetup{font=small,labelfont={bf,color=uqo}} | |
| 24 | +\usepackage{float}\floatplacement{figure}{H} | |
| 25 | +\setlength{\emergencystretch}{3em}\hyphenpenalty=300\tolerance=1500 | |
| 26 | +\usepackage{microtype} | |
| 27 | +\makeatletter\renewcommand{\maketitle}{\begin{center}{\color{uqo}\Huge\bfseries\@title\par}\vspace{8pt}{\normalsize\color{gray}\@author\par\@date\par}\end{center}\vspace{10pt}}\makeatother | |
| 28 | +\usepackage{polyglossia}\setdefaultlanguage[frenchpart=false]{french} | |
added
M3U96bomparables-validation/make_pdf.sh
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Rapatrie les résultats depuis M2U64 et produit docs/RAPPORT.pdf (pandoc → XeLaTeX) | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +V=${CV_VERSION:-2026-10-10} | |
| 6 | +mkdir -p out/$V | |
| 7 | +rsync -az --exclude 'est_E*.npy' --exclude 'sd_E2.npy' --exclude '*_pred_*.npy' --exclude 'sales.parquet' M2U64:comparables-validation/out/$V/ out/$V/ | |
| 8 | +rsync -az M2U64:comparables-validation/docs/ docs/ 2>/dev/null || true | |
| 9 | +cd docs | |
| 10 | +pandoc RAPPORT.md -o RAPPORT.pdf --pdf-engine=xelatex -H header.tex --toc --toc-depth=2 \ | |
| 11 | + -V geometry:"a4paper,margin=17mm,top=20mm,bottom=20mm" -V fontsize=10pt -V linkcolor=uqo -V urlcolor=uqo -V toccolor=uqo -V colorlinks=true \ | |
| 12 | + -V toc-title="Table des matières" --resource-path=.:../out/$V/fig 2>&1 | grep -v "Missing character\|Overfull\|Underfull" | tail -8 || true | |
| 13 | +echo "PDF : $(pwd)/RAPPORT.pdf ($(du -h RAPPORT.pdf | cut -f1))" | |
added
M3U96bomparables-validation/pipeline/01_dataset.py
+97 −0
@@ -0,0 +1,97 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""01 — Jeu de données : transactions unifamiliales Vrai Prix ⋈ rôle, nettoyage journalisé, | |
| 3 | +reventes, géographie, périodes. Sorties : sales.parquet, exclusions.csv, dataset_stats.json.""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +con = connect() | |
| 7 | +log(f"lecture {DB}") | |
| 8 | +sql = """ | |
| 9 | +select t.id, t.date, t.amount, t.lat, t.lng, t.city, t.property_type, t.year_built, t.floor_area, | |
| 10 | + t.building_type, t.id_provinc, t.valeur_role as role_vente, t.land_area, | |
| 11 | + u.code_mun, u.municipalite, u.type_prop as u_type, u.annee_construction, u.aire_etages_m2, | |
| 12 | + u.superficie_terrain_m2, u.front_terrain_m, u.nb_etages, u.nb_logements, u.lien_physique, | |
| 13 | + u.genre_construction, u.unite_voisinage, u.dat_cond_marche, u.valeur_terrain, u.valeur_batiment, | |
| 14 | + u.valeur_role as role_actuel, u.valeur_anterieure as role_anterieur, u.cubf | |
| 15 | +from transactions t left join units u on u.id_provinc = t.id_provinc | |
| 16 | +where t.property_type = 'unifamilial' | |
| 17 | +""" | |
| 18 | +df = pd.read_sql(sql, con) | |
| 19 | +log(f"{len(df):,} transactions unifamiliales brutes") | |
| 20 | +n0 = len(df) | |
| 21 | +excl = [] # (règle, n) | |
| 22 | +def drop(mask, rule): | |
| 23 | + global df | |
| 24 | + n = int(mask.sum()); excl.append((rule, n)); df = df[~mask].copy(); log(f" {rule}: -{n:,} → {len(df):,}") | |
| 25 | + | |
| 26 | +df["date"] = pd.to_datetime(df["date"]) | |
| 27 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 28 | +drop(df["id_provinc"].isna() | df["code_mun"].isna(), "R0 sans jumelage au rôle (id_provinc / code_mun)") | |
| 29 | +drop(df.duplicated(["id_provinc", "date"]), "R1 doublons (même unité, même date)") | |
| 30 | +drop(~df["u_type"].isin(["unifamilial"]) | (df["nb_logements"].fillna(1) > 1), "R2 unité du rôle non unifamiliale (chalet, plex, terrain…) ou > 1 logement") | |
| 31 | +drop(df["lat"].isna() | df["lng"].isna() | (df["lat"] < 44.5) | (df["lat"] > 63) | (df["lng"] > -56) | (df["lng"] < -80), "R3 coordonnées absentes ou hors Québec") | |
| 32 | +fa = df["floor_area"].fillna(df["aire_etages_m2"]) | |
| 33 | +df["floor_area"] = fa | |
| 34 | +drop(fa.isna() | (fa < 40) | (fa > 600), "R4 aire habitable absente ou hors [40, 600] m²") | |
| 35 | +la = df["land_area"].fillna(df["superficie_terrain_m2"]) | |
| 36 | +df["land_area"] = la | |
| 37 | +drop(la.isna() | (la < 80) | (la > 20000), "R5 superficie de terrain absente ou hors [80, 20 000] m²") | |
| 38 | +yb = df["year_built"].fillna(df["annee_construction"]) | |
| 39 | +df["year_built"] = yb | |
| 40 | +drop(yb.isna() | (yb < 1850) | (yb > df["date"].dt.year), "R6 année de construction absente, < 1850 ou postérieure à la vente (vente de terrain)") | |
| 41 | +drop((df["year_built"] == df["date"].dt.year), "R6b vente l'année même de la construction (prix du terrain ou d'une construction inachevée : médiane 346 k$ et ratio au rôle 1,15 contre 450-480 k$ et 1,3 aux âges 1-4)") | |
| 42 | +drop((df["amount"] < 50000) | (df["amount"] > 3_000_000), "R7 prix hors [50 k$, 3 M$]") | |
| 43 | +ratio = df["amount"] / df["role_vente"] | |
| 44 | +drop(df["role_vente"].isna() | (df["role_vente"] <= 0) | (ratio < 0.33) | (ratio > 3.0), "R8 rôle absent ou ratio prix/rôle hors [0,33 ; 3] (transactions présumées non représentatives)") | |
| 45 | +# reventes rapides avec forte variation (flips / rénovations : les caractéristiques enregistrées ne décrivent plus le bien) | |
| 46 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 47 | +g = df.groupby("id_provinc") | |
| 48 | +prev_date = g["date"].shift(1); prev_amt = g["amount"].shift(1) | |
| 49 | +dd = (df["date"] - prev_date).dt.days | |
| 50 | +chg = df["amount"] / prev_amt - 1 | |
| 51 | +flip = (dd <= 180) & (chg.abs() > 0.40) | |
| 52 | +drop(flip.fillna(False), "R9 revente ≤ 180 jours avec variation > 40 % (flip)") | |
| 53 | +drop(df["lien_physique"].isna() | df["genre_construction"].isna(), "R10 lien physique ou genre de construction absent au rôle") | |
| 54 | + | |
| 55 | +# ----------------------------------------------------------------- variables dérivées | |
| 56 | +rep = load_repertoire() | |
| 57 | +df = df.merge(rep, on="code_mun", how="left") | |
| 58 | +drop(df["region"].isna(), "R11 municipalité absente du répertoire MAMH") | |
| 59 | +df["sale_year"] = df["date"].dt.year | |
| 60 | +df["month"] = month_index(df["date"]) | |
| 61 | +df["day"] = (df["date"] - pd.Timestamp("2021-01-01")).dt.days.astype(int) | |
| 62 | +df["age"] = (df["sale_year"] - df["year_built"]).clip(lower=0) | |
| 63 | +df["ln_price"] = np.log(df["amount"]) | |
| 64 | +df["ln_floor"] = np.log(df["floor_area"]) | |
| 65 | +df["ln_land"] = np.log(df["land_area"]) | |
| 66 | +df["lien"] = df["lien_physique"].map(LIEN).fillna(0).astype(int) | |
| 67 | +df["genre"] = df["genre_construction"].map(GENRE).fillna(0).astype(int) | |
| 68 | +df["btype"] = df["building_type"].map(BTYPE).fillna(-1).astype(int) | |
| 69 | +df["etages"] = df["nb_etages"].fillna(1).clip(1, 3) | |
| 70 | +df["x_km"], df["y_km"] = to_xy_km(df["lat"], df["lng"]) | |
| 71 | +# millésime du rôle au moment de la vente (rôle courant / antérieur / autre) | |
| 72 | +df["role_vintage"] = np.where(np.isclose(df["role_vente"], df["role_actuel"].fillna(-1)), 2, | |
| 73 | + np.where(np.isclose(df["role_vente"], df["role_anterieur"].fillna(-1)), 1, 0)) | |
| 74 | +df["ratio_role"] = df["amount"] / df["role_vente"] | |
| 75 | +df["part_terrain"] = (df["valeur_terrain"] / df["role_actuel"]).clip(0.02, 0.95).fillna(0.2) | |
| 76 | +df["neuf"] = (df["age"] <= 3).astype(int) | |
| 77 | +df["period"] = np.select([df["date"] <= TRAIN_END, df["date"] <= VAL_END], ["train", "val"], "test") | |
| 78 | +df = df.sort_values("date").reset_index(drop=True) | |
| 79 | +df["idx"] = np.arange(len(df), dtype=np.int32) | |
| 80 | + | |
| 81 | +keep = ["idx", "id", "id_provinc", "date", "day", "month", "sale_year", "period", "amount", "ln_price", | |
| 82 | + "lat", "lng", "x_km", "y_km", "code_mun", "mun_nom", "mrc", "mrc_nom", "region", "region_nom", | |
| 83 | + "unite_voisinage", "floor_area", "ln_floor", "land_area", "ln_land", "year_built", "age", "neuf", | |
| 84 | + "lien", "genre", "btype", "etages", "role_vente", "role_vintage", "ratio_role", "part_terrain"] | |
| 85 | +df = df[keep] | |
| 86 | +save_parquet(df, "sales.parquet") | |
| 87 | +pd.DataFrame(excl, columns=["regle", "n"]).to_csv(OUT / "exclusions.csv", index=False) | |
| 88 | +stats = { | |
| 89 | + "brut": n0, "propre": len(df), "exclusions": excl, | |
| 90 | + "par_periode": df["period"].value_counts().to_dict(), | |
| 91 | + "par_annee": df["sale_year"].value_counts().sort_index().to_dict(), | |
| 92 | + "par_region": df.groupby("region_nom").size().sort_values(ascending=False).to_dict(), | |
| 93 | + "prix": {k: float(v) for k, v in df["amount"].describe().items()}, | |
| 94 | + "dates": [str(df["date"].min().date()), str(df["date"].max().date())], | |
| 95 | +} | |
| 96 | +save_json(stats, "dataset_stats.json") | |
| 97 | +log(f"jeu propre : {len(df):,} ventes ({stats['par_periode']})") | |
added
M3U96bomparables-validation/pipeline/02_index.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""02 — Indice temporel régional « point dans le temps ». | |
| 3 | +Pour chaque mois d'évaluation m, on ré-estime un indice hédonique mensuel par région administrative | |
| 4 | +avec toutes les ventes antérieures au mois m (fenêtre expansive) (jamais le mois m lui-même). Niveau de référence = m-1, | |
| 5 | +dernier mois entièrement connu. Sortie : index_asof.npy [m_eval, région, mois] (log), regions.json, | |
| 6 | +et un indice « plein échantillon » descriptif (index_full.csv).""" | |
| 7 | +from common import * | |
| 8 | + | |
| 9 | +df = load_parquet("sales.parquet") | |
| 10 | +regions = sorted(df["region"].unique().tolist()) | |
| 11 | +rid = {r: i for i, r in enumerate(regions)} | |
| 12 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 13 | +M = int(df["month"].max()) + 1 | |
| 14 | +M_EVAL0 = int(month_index(pd.Series([VAL_START]))[0]) | |
| 15 | +WINDOW = 999 # fenêtre « expansive » : toutes les ventes antérieures au mois m | |
| 16 | +SHRINK = 40.0 # n/(n+SHRINK) : rétrécissement de l'effet mensuel régional vers l'effet provincial | |
| 17 | + | |
| 18 | +X_cols = ["ln_floor", "ln_land", "age"] | |
| 19 | +def design(d): | |
| 20 | + X = np.column_stack([np.ones(len(d)), d["ln_floor"], d["ln_land"], d["age"], d["age"] ** 2 / 100, | |
| 21 | + (d["lien"] == 1), (d["lien"] >= 2), (d["genre"] == 1), (d["genre"] == 2), (d["genre"] == 3), | |
| 22 | + d["etages"], d["neuf"]]).astype(float) | |
| 23 | + return X | |
| 24 | + | |
| 25 | +def fit_month_effects(d, months): | |
| 26 | + """OLS à effets fixes de municipalité (estimateur within) : ln P ~ X + effets mensuels (base = dernier mois), | |
| 27 | + toutes les variables étant centrées par municipalité pour que l'indice ne soit pas contaminé par les | |
| 28 | + déplacements du mélange des ventes entre municipalités chères et bon marché. Renvoie effets et n par mois.""" | |
| 29 | + X = design(d)[:, 1:] | |
| 30 | + mm = d["month"].to_numpy() | |
| 31 | + base = months[-1] | |
| 32 | + dums = np.column_stack([(mm == m).astype(float) for m in months if m != base]) | |
| 33 | + A = np.hstack([X, dums]); y = d["ln_price"].to_numpy(float) | |
| 34 | + codes, inv = np.unique(d["code_mun"].to_numpy(), return_inverse=True) | |
| 35 | + cnt = np.bincount(inv).astype(float) | |
| 36 | + mA = np.zeros((len(codes), A.shape[1])); np.add.at(mA, inv, A); A = A - mA[inv] / cnt[inv, None] | |
| 37 | + my = np.bincount(inv, y) / cnt; y = y - my[inv] | |
| 38 | + beta, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 39 | + eff = {base: 0.0} | |
| 40 | + k = X.shape[1] | |
| 41 | + for j, m in enumerate([m for m in months if m != base]): | |
| 42 | + eff[m] = float(beta[k + j]) | |
| 43 | + n = {m: int((mm == m).sum()) for m in months} | |
| 44 | + return eff, n | |
| 45 | + | |
| 46 | +idx = np.full((M - M_EVAL0, len(regions), M), np.nan, dtype=np.float32) | |
| 47 | +log(f"indice point-dans-le-temps : mois d'évaluation {M_EVAL0}..{M-1}, {len(regions)} régions") | |
| 48 | +for m in range(M_EVAL0, M): | |
| 49 | + months = list(range(max(0, m - WINDOW), m)) | |
| 50 | + win = df[(df["month"] >= months[0]) & (df["month"] < m)] | |
| 51 | + prov_eff, _ = fit_month_effects(win, months) | |
| 52 | + for r in regions: | |
| 53 | + dr = win[win["region"] == r] | |
| 54 | + if len(dr) < 200: | |
| 55 | + for mo in months: idx[m - M_EVAL0, rid[r], mo] = prov_eff[mo] | |
| 56 | + continue | |
| 57 | + eff, n = fit_month_effects(dr, months) | |
| 58 | + for mo in months: | |
| 59 | + w = n[mo] / (n[mo] + SHRINK) | |
| 60 | + idx[m - M_EVAL0, rid[r], mo] = w * eff[mo] + (1 - w) * prov_eff[mo] | |
| 61 | + if m % 6 == 0: log(f" mois {m} ({len(win):,} ventes dans la fenêtre)") | |
| 62 | +np.save(OUT / "index_asof.npy", idx) | |
| 63 | +save_json({"regions": regions, "M_EVAL0": M_EVAL0, "M": M, "window": WINDOW, "shrink": SHRINK}, "index_meta.json") | |
| 64 | + | |
| 65 | +# indice descriptif plein échantillon (pour le rapport seulement — pas utilisé dans la rétro-évaluation) | |
| 66 | +rows = [] | |
| 67 | +months = list(range(0, M)) | |
| 68 | +eff_p, n_p = fit_month_effects(df, months) | |
| 69 | +for mo in months: | |
| 70 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": "QC", "log_idx": eff_p[mo] - eff_p[months[0]], "n": n_p[mo]}) | |
| 71 | +for r in regions: | |
| 72 | + dr = df[df["region"] == r] | |
| 73 | + if len(dr) < 2000: continue | |
| 74 | + eff, n = fit_month_effects(dr, months) | |
| 75 | + for mo in months: | |
| 76 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": r, "log_idx": eff[mo] - eff[months[0]], "n": n[mo]}) | |
| 77 | +pd.DataFrame(rows).to_csv(OUT / "index_full.csv", index=False) | |
| 78 | +log("indice terminé") | |
added
M3U96bomparables-validation/pipeline/03_models.py
+119 −0
@@ -0,0 +1,119 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""03 — Modèles d'ajustement et modèles de référence, ré-estimés chaque trimestre avec les seules ventes | |
| 3 | +antérieures au trimestre (as-of 2024-01, 2024-04, …, 2026-07) — comme le ferait un AVM ré-entraîné régulièrement : | |
| 4 | + • grille hédonique globale (coefficients) et régionale rétrécie → ajustements B / C | |
| 5 | + • LightGBM (objectif L1 = médiane) sur le prix ajusté dans le temps (avec et sans rôle) → ajustements C-GBM, M11 | |
| 6 | + • hédonique globale directe avec effets de municipalité → M10 | |
| 7 | +Sorties : models_<période>.json, gbm_pred_<période>.npy (prédiction f pour TOUTES les ventes).""" | |
| 8 | +import pickle | |
| 9 | +from common import * | |
| 10 | +import lightgbm as lgb | |
| 11 | + | |
| 12 | +df = load_parquet("sales.parquet") | |
| 13 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 14 | +rid = {r: i for i, r in enumerate(regions)} | |
| 15 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 16 | +idx = np.load(OUT / "index_asof.npy") | |
| 17 | + | |
| 18 | +FEATS = ["ln_floor", "ln_land", "age", "age2", "lien1", "lien2", "genre1", "genre2", "genre3", "etages", "neuf"] | |
| 19 | +def feats(d): | |
| 20 | + return pd.DataFrame({ | |
| 21 | + "ln_floor": d["ln_floor"], "ln_land": d["ln_land"], "age": d["age"], "age2": d["age"] ** 2 / 100, | |
| 22 | + "lien1": (d["lien"] == 1).astype(float), "lien2": (d["lien"] >= 2).astype(float), | |
| 23 | + "genre1": (d["genre"] == 1).astype(float), "genre2": (d["genre"] == 2).astype(float), "genre3": (d["genre"] == 3).astype(float), | |
| 24 | + "etages": d["etages"].astype(float), "neuf": d["neuf"].astype(float)}, index=d.index) | |
| 25 | + | |
| 26 | +def time_adjusted(d, m_asof): | |
| 27 | + """ln P ramené au niveau du mois m_asof-1 avec l'indice connu au mois m_asof.""" | |
| 28 | + lvl = idx[m_asof - M_EVAL0] | |
| 29 | + return d["ln_price"].to_numpy() - lvl[d["rid"].to_numpy(), d["month"].to_numpy()] + lvl[d["rid"].to_numpy(), m_asof - 1] | |
| 30 | + | |
| 31 | +def ols(X, y): | |
| 32 | + A = np.column_stack([np.ones(len(X)), X]) | |
| 33 | + b, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 34 | + return b | |
| 35 | + | |
| 36 | +ASOF = list(range(M_EVAL0, int(df["month"].max()) + 1, 3)) # ré-estimation trimestrielle (2024-01, 2024-04, …) | |
| 37 | +save_json({"asof": ASOF}, "asof.json") | |
| 38 | +TODO = [int(a) for a in os.environ["CV_ASOF"].split(",")] if os.environ.get("CV_ASOF") else ASOF # sous-ensemble (répartition sur plusieurs nœuds) | |
| 39 | +for m_asof in TODO: | |
| 40 | + period = str(m_asof); start = f"{2021 + m_asof // 12}-{m_asof % 12 + 1:02d}-01" | |
| 41 | + tr = df[df["month"] < m_asof].copy() | |
| 42 | + y = time_adjusted(tr, m_asof) | |
| 43 | + F = feats(tr) | |
| 44 | + log(f"[{period}] entraînement sur {len(tr):,} ventes < {start} (indice connu au mois {m_asof})") | |
| 45 | + # --- grille hédonique globale : estimateur « within » (effets fixes de municipalité), pour que les β | |
| 46 | + # mesurent des différences entre biens voisins et non entre marchés | |
| 47 | + def within(frame, cols, by): | |
| 48 | + return frame[cols] - frame.groupby(by)[cols].transform("mean") | |
| 49 | + # effets mensuels résiduels (base = dernier mois connu m_asof-1) : l'indice régional ne retire pas toute la | |
| 50 | + # tendance commune ; sans eux, les modèles globaux sous-estiment les ventes récentes | |
| 51 | + months_tr = sorted(tr["month"].unique()); MCOLS = [f"m{m}" for m in months_tr if m != m_asof - 1] | |
| 52 | + Fm = pd.concat([F] + [pd.Series((tr["month"] == m).astype(float), index=tr.index, name=f"m{m}") for m in months_tr if m != m_asof - 1], axis=1) | |
| 53 | + Fw = within(pd.concat([Fm, tr[["code_mun"]]], axis=1), FEATS + MCOLS, "code_mun") | |
| 54 | + yw = y - pd.Series(y, index=tr.index).groupby(tr["code_mun"]).transform("mean").to_numpy() | |
| 55 | + bw_all, *_ = np.linalg.lstsq(Fw.to_numpy(), yw, rcond=None) | |
| 56 | + bw = bw_all[:len(FEATS)]; bm = bw_all[len(FEATS):] | |
| 57 | + beta_global = dict(zip(FEATS, bw.tolist())) | |
| 58 | + xb = F.to_numpy() @ bw + Fm[MCOLS].to_numpy() @ bm | |
| 59 | + resid = y - xb | |
| 60 | + r2w = 1 - np.var(yw - Fw.to_numpy() @ bw_all) / np.var(yw) | |
| 61 | + Fw = Fw[FEATS] | |
| 62 | + log(f" hédonique within-municipalité : R² within = {r2w:.3f} ; β = { {k: round(v, 3) for k, v in beta_global.items()} }") | |
| 63 | + # --- grille régionale (within-municipalité) rétrécie vers la globale (n/(n+300)) | |
| 64 | + beta_region = {} | |
| 65 | + for r in regions: | |
| 66 | + sel = (tr["region"] == r).to_numpy() | |
| 67 | + if sel.sum() < 100: beta_region[r] = beta_global; continue | |
| 68 | + b, *_ = np.linalg.lstsq(Fw.to_numpy()[sel], yw[sel], rcond=None) | |
| 69 | + w = sel.sum() / (sel.sum() + 300.0) | |
| 70 | + beta_region[r] = {k: float(w * b[j] + (1 - w) * beta_global[k]) for j, k in enumerate(FEATS)} | |
| 71 | + # --- M10 : hédonique directe = Xβ + effet de région + MRC + municipalité + unité de voisinage (rétrécis en cascade) | |
| 72 | + reg_eff = pd.Series(resid, index=tr.index).groupby(tr["region"]).mean().to_dict() | |
| 73 | + tr["res_g"] = resid - tr["region"].map(reg_eff).to_numpy() | |
| 74 | + mrc_eff = tr.groupby("mrc")["res_g"].agg(["sum", "count"]) | |
| 75 | + mrc_eff = (mrc_eff["sum"] / (mrc_eff["count"] + 30)).to_dict() | |
| 76 | + tr["res_m"] = tr["res_g"] - tr["mrc"].map(mrc_eff).fillna(0) | |
| 77 | + mun_eff = tr.groupby("code_mun")["res_m"].agg(["sum", "count"]) | |
| 78 | + mun_eff = (mun_eff["sum"] / (mun_eff["count"] + 20)).to_dict() | |
| 79 | + tr["res_u"] = tr["res_m"] - tr["code_mun"].map(mun_eff).fillna(0) | |
| 80 | + # correction médiane : la MCO vise la moyenne du log ; la distribution est asymétrique (queue basse), on recentre sur la médiane | |
| 81 | + uv_tmp = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].transform(lambda v: v.sum() / (len(v) + 10)) | |
| 82 | + median_shift = float(np.median(tr["res_u"] - uv_tmp)) | |
| 83 | + uv = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].agg(["sum", "count"]) | |
| 84 | + uv_eff = {f"{a}|{b}": v for (a, b), v in (uv["sum"] / (uv["count"] + 10)).items()} | |
| 85 | + # --- LightGBM (prix ajusté dans le temps) avec rôle et sans rôle | |
| 86 | + def gbm_frame(d): | |
| 87 | + G = feats(d) | |
| 88 | + G["lat"] = d["lat"]; G["lng"] = d["lng"]; G["x_km"] = d["x_km"]; G["y_km"] = d["y_km"] | |
| 89 | + G["rid"] = d["rid"]; G["btype"] = d["btype"] | |
| 90 | + G["ln_role"] = np.log(d["role_vente"]); G["part_terrain"] = d["part_terrain"]; G["role_vintage"] = d["role_vintage"] | |
| 91 | + G["month"] = np.minimum(d["month"], m_asof - 1).astype(float) # tendance résiduelle ; hors échantillon = dernier mois connu | |
| 92 | + return G | |
| 93 | + G = gbm_frame(tr) | |
| 94 | + params = dict(objective="regression_l1", learning_rate=0.05, num_leaves=127, min_data_in_leaf=40, feature_fraction=0.8, | |
| 95 | + bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=int(os.environ.get("CV_THREADS", 20))) | |
| 96 | + models = {} | |
| 97 | + for name, cols in [("gbm", [c for c in G.columns if c not in ("ln_role", "part_terrain", "role_vintage")]), | |
| 98 | + ("gbm_role", list(G.columns))]: | |
| 99 | + ds = lgb.Dataset(G[cols], y, categorical_feature=["rid", "btype"] if "btype" in cols else ["rid"]) | |
| 100 | + # validation interne : derniers 10 % chronologiques pour fixer le nombre d'arbres | |
| 101 | + n = len(G); cut = int(n * 0.9) | |
| 102 | + dtr = lgb.Dataset(G[cols].iloc[:cut], y[:cut], categorical_feature=["rid", "btype"]) | |
| 103 | + dva = lgb.Dataset(G[cols].iloc[cut:], y[cut:], reference=dtr) | |
| 104 | + bst = lgb.train(params, dtr, num_boost_round=3000, valid_sets=[dva], callbacks=[lgb.early_stopping(100, verbose=False)]) | |
| 105 | + best = max(bst.best_iteration, 300) # plancher : l'arrêt précoce sur la tranche finale (mois partiels) peut être trop court | |
| 106 | + bst = lgb.train(params, ds, num_boost_round=best) | |
| 107 | + models[name] = bst | |
| 108 | + Gall = gbm_frame(df); Gall["month"] = float(m_asof - 1) # f évalué au niveau du dernier mois connu pour toutes les ventes | |
| 109 | + pred_all = bst.predict(Gall[cols]) | |
| 110 | + np.save(OUT / f"{name}_pred_{period}.npy", pred_all.astype(np.float32)) | |
| 111 | + rr = y - bst.predict(G[cols]); r2 = 1 - np.var(rr) / np.var(y) | |
| 112 | + log(f" {name}: {best} arbres, R² (in-sample) = {r2:.3f}, résidu médian = {np.median(rr) * 100:.2f} %") | |
| 113 | + bst.save_model(str(OUT / f"{name}_{period}.txt")) | |
| 114 | + save_json({"period": period, "m_asof": m_asof, "start": start, "n_train": len(tr), "feats": FEATS, | |
| 115 | + "beta_global": beta_global, "intercept_global": 0.0, "region_fe": reg_eff, | |
| 116 | + "beta_region": beta_region, "mrc_eff": mrc_eff, "mun_eff": mun_eff, "uv_eff": uv_eff, | |
| 117 | + "r2_within": float(r2w), "sd_resid_u": float(tr["res_u"].std()), "median_shift": median_shift}, | |
| 118 | + f"models_{period}.json") | |
| 119 | +log("modèles terminés") | |
added
M3U96bomparables-validation/pipeline/04_backtest.py
+193 −0
@@ -0,0 +1,193 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04 — Moteur de rétro-évaluation. | |
| 3 | +Pour chaque vente cible (2024 = validation, 2025-2026 = test), le prix est masqué ; seules les ventes | |
| 4 | +conclues STRICTEMENT AVANT la date cible (et l'indice/les modèles connus à cette date) sont utilisées. | |
| 5 | +On évalue d'un coup toutes les configurations (sélection × rayon × fenêtre × nombre × ajustement × | |
| 6 | +pondération × réconciliation) sur exactement les mêmes cibles. | |
| 7 | +Sorties : targets.parquet, configs_E1.csv/est_E1.npy (surface k,d,T), configs_E2.csv/est_E2.npy/sd_E2.npy | |
| 8 | +(factoriel sélection × ajustement × pondération × réconciliation), n_avail.npy.""" | |
| 9 | +import os | |
| 10 | +for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"): | |
| 11 | + os.environ[_v] = "1" # un seul fil BLAS par processus (fork + BLAS multi-fils = interblocage sur macOS) | |
| 12 | +import multiprocessing as mp, time | |
| 13 | +from scipy.spatial import cKDTree | |
| 14 | +from common import * | |
| 15 | + | |
| 16 | +df = load_parquet("sales.parquet") | |
| 17 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 18 | +rid_map = {r: i for i, r in enumerate(regions)} | |
| 19 | +IDX = np.load(OUT / "index_asof.npy") | |
| 20 | +ASOF = load_json("asof.json")["asof"] | |
| 21 | +MODELS = {a: load_json(f"models_{a}.json") for a in ASOF} | |
| 22 | +FEATS = MODELS[ASOF[0]]["feats"] | |
| 23 | +GBM = {a: np.load(OUT / f"gbm_pred_{a}.npy") for a in ASOF} | |
| 24 | + | |
| 25 | +# ----------------------------------------------------------------- tableaux globaux (partagés par fork) | |
| 26 | +N = len(df) | |
| 27 | +DAY = df["day"].to_numpy(np.int32); MONTH = df["month"].to_numpy(np.int32) | |
| 28 | +RID = df["region"].map(rid_map).to_numpy(np.int32) | |
| 29 | +X = df["x_km"].to_numpy(np.float64); Y = df["y_km"].to_numpy(np.float64) | |
| 30 | +LNP = df["ln_price"].to_numpy(np.float64); P = df["amount"].to_numpy(np.float64) | |
| 31 | +FLOOR = df["floor_area"].to_numpy(np.float64); LAND = df["land_area"].to_numpy(np.float64) | |
| 32 | +AGE = df["age"].to_numpy(np.float64); LIEN = df["lien"].to_numpy(np.int8); ETAGES = df["etages"].to_numpy(np.float64) | |
| 33 | +GENRE = df["genre"].to_numpy(np.int8) | |
| 34 | +PROV = pd.factorize(df["id_provinc"])[0].astype(np.int32) | |
| 35 | +MUN = pd.factorize(df["code_mun"])[0].astype(np.int32) | |
| 36 | +VINT = df["role_vintage"].to_numpy(np.int8); LNROLE = np.log(df["role_vente"].to_numpy(np.float64)) | |
| 37 | +PART = df["part_terrain"].to_numpy(np.float64) | |
| 38 | +NEUF = df["neuf"].to_numpy(np.float64) | |
| 39 | +# matrice des caractéristiques hédoniques (même ordre que FEATS) | |
| 40 | +XF = np.column_stack([np.log(FLOOR), np.log(LAND), AGE, AGE ** 2 / 100, (LIEN == 1), (LIEN >= 2), | |
| 41 | + (GENRE == 1), (GENRE == 2), (GENRE == 3), ETAGES, NEUF]).astype(np.float64) | |
| 42 | +BETA_G = {a: np.array([MODELS[a]["beta_global"][f] for f in FEATS]) for a in ASOF} | |
| 43 | +BETA_R = {a: np.array([[MODELS[a]["beta_region"][r][f] for f in FEATS] for r in regions]) for a in ASOF} | |
| 44 | +ASOF_ARR = np.array(ASOF) | |
| 45 | +TREE = cKDTree(np.column_stack([X, Y])) | |
| 46 | + | |
| 47 | +D_MAX, T_MAX_DAYS = 25.0, 731 | |
| 48 | +KS = [1, 3, 5, 7, 10, 15, 20, 30, 50] | |
| 49 | +DS = [0.5, 1, 2, 5, 10, 25] | |
| 50 | +TS = [3, 6, 12, 24] | |
| 51 | +SELS = ["geo", "sim", "mix", "adj"] | |
| 52 | +ADJS = ["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"] | |
| 53 | +WRS = [("equal", "median"), ("equal", "mean"), ("dist", "median"), ("dist", "mean"), ("rec", "median"), ("rec", "mean"), | |
| 54 | + ("sim", "median"), ("sim", "mean"), ("all", "median"), ("all", "mean")] | |
| 55 | +ANCHORS = [(3, 2, 6), (5, 5, 12), (10, 10, 24)] | |
| 56 | +E1 = [dict(sel=s, d=d, T=T, k=k, adj=a, w="equal", recon="median") | |
| 57 | + for s in ("geo", "sim", "mix") for d in DS for T in TS for k in KS for a in ("none", "hedo")] | |
| 58 | +E2 = [dict(sel=s, d=d, T=T, k=k, adj=a, w=w, recon=r) | |
| 59 | + for (k, d, T) in ANCHORS for s in SELS for a in ADJS for (w, r) in WRS] | |
| 60 | +LOCAL_COLS = [0, 1, 2, 4, 5, 9] # ln_floor, ln_land, age, lien1, lien2, etages | |
| 61 | +LAMBDA_LOCAL = 50.0 | |
| 62 | + | |
| 63 | +def wmedian(v, w): | |
| 64 | + o = np.argsort(v); v = v[o]; w = w[o] | |
| 65 | + c = np.cumsum(w); return v[np.searchsorted(c, 0.5 * c[-1])] | |
| 66 | + | |
| 67 | +def recon(vals, w, how): | |
| 68 | + ok = ~np.isnan(vals) | |
| 69 | + if not ok.any(): return np.nan, np.nan | |
| 70 | + v = vals[ok]; ww = w[ok] | |
| 71 | + sd = v.std() if len(v) > 1 else np.nan | |
| 72 | + if how == "median": | |
| 73 | + return (np.median(v) if ww.min() == ww.max() else wmedian(v, ww)), sd | |
| 74 | + return np.log(np.sum(ww * np.exp(v)) / ww.sum()), sd | |
| 75 | + | |
| 76 | +def evaluate(i): | |
| 77 | + """Toutes les configurations pour la cible i. Renvoie (est_E1, est_E2, sd_E2, n_avail, info).""" | |
| 78 | + period = int(ASOF_ARR[ASOF_ARR <= MONTH[i]].max()) # dernier trimestre de ré-estimation ≤ mois de la cible | |
| 79 | + e1 = np.full(len(E1), np.nan, np.float32); e2 = np.full(len(E2), np.nan, np.float32) | |
| 80 | + s2 = np.full(len(E2), np.nan, np.float32); nav = np.zeros(len(DS) * len(TS), np.int32) | |
| 81 | + cand = np.asarray(TREE.query_ball_point((X[i], Y[i]), D_MAX), dtype=np.int64) | |
| 82 | + keep = (DAY[cand] < DAY[i]) & (DAY[cand] >= DAY[i] - T_MAX_DAYS) & (PROV[cand] != PROV[i]) | |
| 83 | + cand = cand[keep] | |
| 84 | + info = dict(idx=int(i), asof=period, n_cand=int(len(cand))) | |
| 85 | + if len(cand) == 0: return e1, e2, s2, nav, info | |
| 86 | + dist = np.hypot(X[cand] - X[i], Y[cand] - Y[i]) | |
| 87 | + mago = (DAY[i] - DAY[cand]) / 30.4375 | |
| 88 | + m_eval = MONTH[i]; lvl = IDX[m_eval - M_EVAL0] | |
| 89 | + tadj = np.where(MONTH[cand] == m_eval, 0.0, lvl[RID[cand], m_eval - 1] - lvl[RID[cand], MONTH[cand]]) | |
| 90 | + tadj = np.nan_to_num(tadj, nan=0.0) | |
| 91 | + lnT = LNP[cand] + tadj | |
| 92 | + dX = XF[i] - XF[cand] # cible − comparable | |
| 93 | + # --- ajustements | |
| 94 | + bg = BETA_G[period]; br = BETA_R[period][RID[i]] | |
| 95 | + adj = {} | |
| 96 | + adj["raw"] = LNP[cand] | |
| 97 | + adj["none"] = lnT | |
| 98 | + Pt = np.exp(lnT) | |
| 99 | + d_floor = (FLOOR[i] - FLOOR[cand]) * 0.5 * Pt / FLOOR[cand] | |
| 100 | + d_land = np.clip((LAND[i] - LAND[cand]) * 0.5 * Pt * PART[cand] / LAND[cand], -0.15 * Pt, 0.15 * Pt) | |
| 101 | + d_age = np.clip(-0.005 * Pt * (AGE[i] - AGE[cand]), -0.2 * Pt, 0.2 * Pt) | |
| 102 | + d_lien = np.where((LIEN[i] == 0) & (LIEN[cand] > 0), 0.05 * Pt, np.where((LIEN[i] > 0) & (LIEN[cand] == 0), -0.05 * Pt, 0.0)) | |
| 103 | + tot = np.maximum(d_floor + d_land + d_age + d_lien, -0.6 * Pt) | |
| 104 | + adj["conv"] = np.log(Pt + tot) | |
| 105 | + gross_conv = (np.abs(d_floor) + np.abs(d_land) + np.abs(d_age) + np.abs(d_lien)) / Pt | |
| 106 | + adj["hedo"] = lnT + dX @ bg | |
| 107 | + gross_hedo = np.abs(dX * bg).sum(axis=1) | |
| 108 | + adj["hedo_reg"] = lnT + dX @ br | |
| 109 | + loc = dist <= 10.0 | |
| 110 | + if loc.sum() >= 40: | |
| 111 | + Xl = XF[cand][loc][:, LOCAL_COLS]; yl = lnT[loc] | |
| 112 | + b0 = br[LOCAL_COLS] | |
| 113 | + Xc = Xl - Xl.mean(axis=0); yc = yl - yl.mean() | |
| 114 | + A = Xc.T @ Xc + LAMBDA_LOCAL * np.eye(len(LOCAL_COLS)) | |
| 115 | + bl = np.linalg.solve(A, Xc.T @ yc + LAMBDA_LOCAL * b0) | |
| 116 | + adj["local"] = lnT + dX[:, LOCAL_COLS] @ bl | |
| 117 | + else: | |
| 118 | + adj["local"] = adj["hedo_reg"] | |
| 119 | + f = GBM[period] | |
| 120 | + adj["gbm"] = lnT + (f[i] - f[cand]) | |
| 121 | + same = (MUN[cand] == MUN[i]) & (VINT[cand] == VINT[i]) | |
| 122 | + adj["role"] = np.where(same, LNROLE[i] + (lnT - LNROLE[cand]), np.nan) | |
| 123 | + # --- scores de sélection | |
| 124 | + dissim = np.sqrt((dX[:, 0] / 0.25) ** 2 + (dX[:, 1] / 0.6) ** 2 + (dX[:, 2] / 15) ** 2 | |
| 125 | + + (LIEN[cand] != LIEN[i]) * 1.0 + (ETAGES[cand] != ETAGES[i]) * 0.5) | |
| 126 | + score = {"geo": dist, "sim": dissim + dist / 25.0, "mix": dissim + dist / 2.0, "adj": gross_hedo + dist / 25.0} | |
| 127 | + # --- pondérations | |
| 128 | + wts = {"equal": np.ones(len(cand)), "dist": 1.0 / (0.25 + dist), "rec": np.exp(-mago / 6.0), "sim": 1.0 / (0.1 + gross_hedo)} | |
| 129 | + wts["all"] = wts["dist"] * wts["rec"] * wts["sim"] | |
| 130 | + # --- masques (d, T) et classements | |
| 131 | + top = {} | |
| 132 | + for di, d in enumerate(DS): | |
| 133 | + for ti, T in enumerate(TS): | |
| 134 | + m = (dist <= d) & (mago <= T) | |
| 135 | + nav[di * len(TS) + ti] = int(m.sum()) | |
| 136 | + if not m.any(): continue | |
| 137 | + ids = np.flatnonzero(m) | |
| 138 | + for s in SELS: | |
| 139 | + sc = score[s][ids] | |
| 140 | + if len(ids) > 50: | |
| 141 | + part = np.argpartition(sc, 50)[:50]; part = part[np.argsort(sc[part])] | |
| 142 | + else: | |
| 143 | + part = np.argsort(sc) | |
| 144 | + top[(s, d, T)] = ids[part] | |
| 145 | + # --- E1 | |
| 146 | + for c, cfg in enumerate(E1): | |
| 147 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 148 | + if t is None: continue | |
| 149 | + sub = t[:cfg["k"]] | |
| 150 | + e1[c], _ = recon(adj[cfg["adj"]][sub], wts["equal"][sub], "median") | |
| 151 | + # --- E2 | |
| 152 | + for c, cfg in enumerate(E2): | |
| 153 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 154 | + if t is None: continue | |
| 155 | + sub = t[:cfg["k"]] | |
| 156 | + e2[c], s2[c] = recon(adj[cfg["adj"]][sub], wts[cfg["w"]][sub], cfg["recon"]) | |
| 157 | + return e1, e2, s2, nav, info | |
| 158 | + | |
| 159 | +def worker(chunk): | |
| 160 | + return [evaluate(i) for i in chunk] | |
| 161 | + | |
| 162 | +if __name__ == "__main__": | |
| 163 | + targets = df.index[df["period"].isin(["val", "test"])].to_numpy() | |
| 164 | + if os.environ.get("CV_SAMPLE"): | |
| 165 | + rng = np.random.default_rng(1); targets = np.sort(rng.choice(targets, int(os.environ["CV_SAMPLE"]), replace=False)) | |
| 166 | + SUF = "" | |
| 167 | + if os.environ.get("CV_SHARD"): # répartition sur plusieurs nœuds : CV_SHARD=i/N → cibles i, i+N, i+2N, … | |
| 168 | + i_sh, n_sh = map(int, os.environ["CV_SHARD"].split("/")); targets = targets[i_sh::n_sh]; SUF = f".shard{i_sh}of{n_sh}" | |
| 169 | + log(f"{len(targets):,} cibles ; {len(E1)} configurations E1, {len(E2)} E2 ; {N:,} ventes dans le bassin") | |
| 170 | + pd.DataFrame(E1).to_csv(OUT / "configs_E1.csv", index_label="cfg") | |
| 171 | + pd.DataFrame(E2).to_csv(OUT / "configs_E2.csv", index_label="cfg") | |
| 172 | + est1 = np.full((len(targets), len(E1)), np.nan, np.float32) | |
| 173 | + est2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 174 | + sd2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 175 | + nav = np.zeros((len(targets), len(DS) * len(TS)), np.int32) | |
| 176 | + infos = [] | |
| 177 | + chunks = [targets[j:j + 400] for j in range(0, len(targets), 400)] | |
| 178 | + t0 = time.time(); done = 0 | |
| 179 | + with mp.get_context(os.environ.get("CV_MP", "fork")).Pool(int(os.environ.get("CV_PROCS", 20))) as pool: | |
| 180 | + for res in pool.imap(worker, chunks): | |
| 181 | + for (e1, e2, s2, nv, info) in res: | |
| 182 | + est1[done] = e1; est2[done] = e2; sd2[done] = s2; nav[done] = nv; infos.append(info); done += 1 | |
| 183 | + if done % 20000 < 400: | |
| 184 | + log(f" {done:,}/{len(targets):,} cibles ({time.time() - t0:.0f} s)") | |
| 185 | + np.save(OUT / f"est_E1{SUF}.npy", est1); np.save(OUT / f"est_E2{SUF}.npy", est2); np.save(OUT / f"sd_E2{SUF}.npy", sd2); np.save(OUT / f"n_avail{SUF}.npy", nav) | |
| 186 | + T = df.loc[targets, ["idx", "id", "date", "period", "amount", "ln_price", "region", "region_nom", "mrc", "mrc_nom", "code_mun", | |
| 187 | + "mun_nom", "unite_voisinage", "floor_area", "land_area", "age", "lien", "etages", "role_vente", "month", "lat", "lng"]].reset_index(drop=True) | |
| 188 | + T["n_cand"] = [x["n_cand"] for x in infos]; T["asof"] = [x["asof"] for x in infos] | |
| 189 | + T["n_2km_12m"] = nav[:, DS.index(2) * len(TS) + TS.index(12)] | |
| 190 | + T["n_5km_12m"] = nav[:, DS.index(5) * len(TS) + TS.index(12)] | |
| 191 | + save_parquet(T, f"targets{SUF}.parquet") | |
| 192 | + pd.DataFrame({"DS": [DS], "TS": [TS], "KS": [KS]}).to_json(OUT / "grid_meta.json") | |
| 193 | + log(f"rétro-évaluation terminée en {time.time() - t0:.0f} s") | |
added
M3U96bomparables-validation/pipeline/04b_gather.py
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04b — Rassemble les fragments (shards) de la rétro-évaluation calculés sur plusieurs nœuds, dans l'ordre des cibles.""" | |
| 3 | +import glob | |
| 4 | +from common import * | |
| 5 | +sh = sorted(glob.glob(str(OUT / "targets.shard*.parquet"))) | |
| 6 | +log(f"{len(sh)} fragments") | |
| 7 | +Ts = [pd.read_parquet(p) for p in sh] | |
| 8 | +parts = {} | |
| 9 | +for name in ["est_E1", "est_E2", "sd_E2", "n_avail"]: | |
| 10 | + parts[name] = [np.load(p.replace("targets.", f"{name}.").replace(".parquet", ".npy")) for p in sh] | |
| 11 | +T = pd.concat(Ts, ignore_index=True); order = np.argsort(T["idx"].to_numpy(), kind="stable") | |
| 12 | +T = T.iloc[order].reset_index(drop=True); save_parquet(T, "targets.parquet") | |
| 13 | +for name, arrs in parts.items(): | |
| 14 | + A = np.concatenate(arrs, axis=0)[order]; np.save(OUT / f"{name}.npy", A); log(f" {name}: {A.shape}") | |
| 15 | +log("rassemblement terminé") | |
added
M3U96bomparables-validation/pipeline/05_analyze.py
+358 −0
@@ -0,0 +1,358 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""05 — Analyse : métriques par configuration (validation 2024 → choix ; test 2025-26 → verdict), | |
| 3 | +surface Erreur(k, d, t), effets des facteurs (sélection, ajustement, pondération, réconciliation), | |
| 4 | +compétition de modèles M1-M13, hypothèses H1-H5, strates (région, densité, prix), intervalles, | |
| 5 | +règles conditionnelles. Sorties : tables/*.csv, results.json, fig/*.pdf""" | |
| 6 | +from scipy.optimize import nnls | |
| 7 | +import matplotlib; matplotlib.use("Agg") | |
| 8 | +import matplotlib.pyplot as plt | |
| 9 | +from matplotlib import rcParams | |
| 10 | +from common import * | |
| 11 | + | |
| 12 | +# ---------------------------------------------------------------- style (palette de référence, marques fines) | |
| 13 | +PAL = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"] | |
| 14 | +SEQ = ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#184f95", "#0d366b"] | |
| 15 | +rcParams.update({"font.family": "Helvetica Neue", "font.size": 9, "axes.edgecolor": "#c3c2b7", "axes.linewidth": 0.6, | |
| 16 | + "axes.grid": True, "grid.color": "#e8e7e3", "grid.linewidth": 0.5, "axes.spines.top": False, | |
| 17 | + "axes.spines.right": False, "legend.frameon": False, "figure.dpi": 150, "savefig.bbox": "tight", | |
| 18 | + "axes.titleweight": "bold", "axes.titlesize": 10, "axes.labelcolor": "#52514e", "xtick.color": "#52514e", | |
| 19 | + "ytick.color": "#52514e", "figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb"}) | |
| 20 | +from matplotlib.colors import LinearSegmentedColormap | |
| 21 | +CMAP = LinearSegmentedColormap.from_list("seqblue", SEQ) | |
| 22 | +TAB = OUT / "tables"; TAB.mkdir(exist_ok=True); FIG = OUT / "fig" | |
| 23 | +def savefig(fig, name): fig.savefig(FIG / f"{name}.pdf"); fig.savefig(FIG / f"{name}.png"); plt.close(fig) | |
| 24 | + | |
| 25 | +# ---------------------------------------------------------------- chargement | |
| 26 | +T = load_parquet("targets.parquet"); S = load_parquet("sales.parquet").set_index("idx") | |
| 27 | +E1 = pd.read_csv(OUT / "configs_E1.csv"); E2 = pd.read_csv(OUT / "configs_E2.csv") | |
| 28 | +est1 = np.load(OUT / "est_E1.npy"); est2 = np.load(OUT / "est_E2.npy"); sd2 = np.load(OUT / "sd_E2.npy"); nav = np.load(OUT / "n_avail.npy") | |
| 29 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"]; IDX = np.load(OUT / "index_asof.npy") | |
| 30 | +rid = {r: i for i, r in enumerate(regions)} | |
| 31 | +lnP = T["ln_price"].to_numpy(np.float32) | |
| 32 | +VAL = (T["period"] == "val").to_numpy(); TEST = (T["period"] == "test").to_numpy() | |
| 33 | +log(f"{len(T):,} cibles (val {VAL.sum():,}, test {TEST.sum():,}) ; E1 {est1.shape[1]} ; E2 {est2.shape[1]}") | |
| 34 | + | |
| 35 | +def err_of(est): return (np.exp(est - lnP[:, None]) - 1).astype(np.float32) | |
| 36 | +def metrics(err, sel): | |
| 37 | + e = err[sel]; a = np.abs(e) | |
| 38 | + ok = ~np.isnan(a); n = ok.sum(axis=0) | |
| 39 | + with np.errstate(all="ignore"): | |
| 40 | + return pd.DataFrame({"mdape": np.nanmedian(a, axis=0) * 100, "mape": np.nanmean(a, axis=0) * 100, | |
| 41 | + "biais": np.nanmedian(e, axis=0) * 100, "biais_moy": np.nanmean(e, axis=0) * 100, | |
| 42 | + "p5": np.nanmean(a <= 0.05, axis=0) * 100, "p10": np.nanmean(a <= 0.10, axis=0) * 100, | |
| 43 | + "p20": np.nanmean(a <= 0.20, axis=0) * 100, "couverture": n / sel.sum() * 100, "n": n}) | |
| 44 | +err1 = err_of(est1); err2 = err_of(est2) | |
| 45 | +del est1 | |
| 46 | +res = {} | |
| 47 | + | |
| 48 | +# ---------------------------------------------------------------- densité & strates | |
| 49 | +T["dens"] = pd.cut(T["n_2km_12m"], [-1, 4, 19, 49, 199, 10 ** 9], labels=["< 5", "5-19", "20-49", "50-199", "≥ 200"]) | |
| 50 | +T["prix_q"] = pd.qcut(T["amount"], 5, labels=["Q1 (bas)", "Q2", "Q3", "Q4", "Q5 (haut)"]) | |
| 51 | +T["trimestre"] = pd.to_datetime(T["date"]).dt.to_period("Q").astype(str) | |
| 52 | +T["age_cl"] = pd.cut(T["age"], [-1, 3, 15, 40, 70, 1000], labels=["≤ 3 ans", "4-15", "16-40", "41-70", "> 70"]) | |
| 53 | +T["cluster"] = pd.factorize(T["code_mun"].astype(str) + "|" + T["trimestre"])[0] | |
| 54 | + | |
| 55 | +# ---------------------------------------------------------------- E1 : surface Erreur(k, d, t) | |
| 56 | +mv = metrics(err1, VAL).add_suffix("_val"); mt = metrics(err1, TEST).add_suffix("_test") | |
| 57 | +surf = pd.concat([E1, mv, mt], axis=1) | |
| 58 | +surf.to_csv(TAB / "surface_E1.csv", index=False) | |
| 59 | +ok = surf["couverture_val"] >= 97 | |
| 60 | +best_e1 = surf[ok].sort_values("mdape_val").iloc[0] | |
| 61 | +res["E1_best_val"] = best_e1.to_dict() | |
| 62 | +log(f"E1 meilleure (val, couverture ≥ 97 %) : {best_e1[['sel','d','T','k','adj']].to_dict()} → val {best_e1.mdape_val:.2f} %, test {best_e1.mdape_test:.2f} %") | |
| 63 | +for (s, a), g in surf[ok].groupby(["sel", "adj"]): | |
| 64 | + b = g.sort_values("mdape_val").iloc[0] | |
| 65 | + res.setdefault("E1_best_by_sel_adj", {})[f"{s}|{a}"] = dict(k=int(b.k), d=float(b.d), T=int(b["T"]), mdape_val=float(b.mdape_val), mdape_test=float(b.mdape_test), couverture_test=float(b.couverture_test)) | |
| 66 | + | |
| 67 | +# figure : MdAPE vs k selon d (T = 12, geo, hedo, test) | |
| 68 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.4), sharey=True) | |
| 69 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 70 | + for j, d in enumerate([0.5, 1, 2, 5, 10, 25]): | |
| 71 | + g = surf[(surf.sel == "geo") & (surf["T"] == 12) & (surf.adj == a) & (surf.d == d)].sort_values("k") | |
| 72 | + ax.plot(g.k, g.mdape_test, color=PAL[j], lw=2, marker="o", ms=4, label=f"d ≤ {d} km") | |
| 73 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]) | |
| 74 | + ax.set_xlabel("nombre de comparables k"); ax.set_title({"none": "sans ajustement (temps seulement)", "hedo": "grille hédonique"}[a]) | |
| 75 | +axes[0].set_ylabel("MdAPE, test 2025-26 (%)"); axes[1].legend(ncol=2, fontsize=8) | |
| 76 | +savefig(fig, "f1_mdape_vs_k") | |
| 77 | +# figure : carte d × T au meilleur k (geo, hedo) | |
| 78 | +fig, axes = plt.subplots(1, 2, figsize=(8, 3.2)) | |
| 79 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 80 | + g = surf[(surf.sel == "geo") & (surf.adj == a)].groupby(["d", "T"])["mdape_test"].min().unstack("T") | |
| 81 | + im = ax.imshow(g.to_numpy(), cmap=CMAP, aspect="auto") | |
| 82 | + ax.set_xticks(range(len(g.columns))); ax.set_xticklabels([f"{t} mois" for t in g.columns]); ax.set_yticks(range(len(g.index))); ax.set_yticklabels([f"{d} km" for d in g.index]) | |
| 83 | + for i in range(g.shape[0]): | |
| 84 | + for j in range(g.shape[1]): | |
| 85 | + v = g.iloc[i, j]; ax.text(j, i, f"{v:.1f}", ha="center", va="center", fontsize=7.5, color="white" if v < np.nanpercentile(g.to_numpy(), 40) else "#0b0b0b") | |
| 86 | + ax.grid(False); ax.set_title({"none": "sans ajustement", "hedo": "grille hédonique"}[a] + " — MdAPE au meilleur k") | |
| 87 | +savefig(fig, "f2_surface_d_T") | |
| 88 | + | |
| 89 | +# ---------------------------------------------------------------- E2 : facteurs | |
| 90 | +mv2 = metrics(err2, VAL).add_suffix("_val"); mt2 = metrics(err2, TEST).add_suffix("_test") | |
| 91 | +fac = pd.concat([E2, mv2, mt2], axis=1); fac.to_csv(TAB / "facteurs_E2.csv", index=False) | |
| 92 | +anchor = (fac.k == 5) & (fac.d == 5) & (fac["T"] == 12) | |
| 93 | +base = anchor & (fac.sel == "geo") & (fac.w == "equal") & (fac.recon == "median") | |
| 94 | +res["E2_adjust_anchor"] = fac[base].set_index("adj")[["mdape_val", "mdape_test", "mape_test", "biais_test", "p10_test", "couverture_test"]].to_dict("index") | |
| 95 | +res["E2_sel_anchor"] = fac[anchor & (fac.adj == "hedo") & (fac.w == "equal") & (fac.recon == "median")].set_index("sel")[["mdape_val", "mdape_test", "p10_test"]].to_dict("index") | |
| 96 | +res["E2_wr_anchor"] = {f"{r.w}|{r.recon}": dict(mdape_val=r.mdape_val, mdape_test=r.mdape_test, p10_test=r.p10_test) | |
| 97 | + for r in fac[anchor & (fac.adj == "hedo") & (fac.sel == "geo")].itertuples()} | |
| 98 | +# effets marginaux moyens (moyenne de MdAPE test sur toutes les autres modalités) | |
| 99 | +marg = {} | |
| 100 | +for f in ["sel", "adj", "w", "recon"]: | |
| 101 | + marg[f] = fac.groupby(f)["mdape_test"].mean().round(2).to_dict() | |
| 102 | +res["E2_marginal"] = marg | |
| 103 | +anc_tab = fac.groupby(["k", "d", "T"])["mdape_test"].min().round(2).to_dict() | |
| 104 | +res["E2_anchor_best"] = {f"{k}|{d}|{t}": v for (k, d, t), v in anc_tab.items()} | |
| 105 | +# figure : ajustements à l'ancre (val et test) | |
| 106 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 107 | +g = fac[base].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 108 | +lab = {"raw": "aucun", "none": "temps seul", "conv": "grille conv.", "hedo": "hédonique", "hedo_reg": "hédo. régionale", "local": "locale", "gbm": "GBM", "role": "ratio au rôle"} | |
| 109 | +x = np.arange(len(g)); ax.bar(x - 0.2, g.mdape_val, 0.38, color=PAL[0], label="validation 2024"); ax.bar(x + 0.2, g.mdape_test, 0.38, color=PAL[1], label="test 2025-26") | |
| 110 | +for i, (v1, v2) in enumerate(zip(g.mdape_val, g.mdape_test)): | |
| 111 | + ax.text(i - 0.2, v1 + 0.2, f"{v1:.1f}", ha="center", fontsize=7); ax.text(i + 0.2, v2 + 0.2, f"{v2:.1f}", ha="center", fontsize=7) | |
| 112 | +ax.set_xticks(x); ax.set_xticklabels([lab[a] for a in g.index]); ax.set_ylabel("MdAPE (%)"); ax.legend(); ax.set_title("Type d'ajustement — 5 comparables les plus proches, ≤ 5 km, ≤ 12 mois, médiane") | |
| 113 | +savefig(fig, "f3_ajustements") | |
| 114 | + | |
| 115 | +# ---------------------------------------------------------------- modèles de référence (M10, M11) et compétition | |
| 116 | +def level_shift(m_asof): | |
| 117 | + mt_ = T["month"].to_numpy(); r = T["region"].map(rid).to_numpy() | |
| 118 | + return IDX[mt_ - M_EVAL0, r, mt_ - 1] - IDX[mt_ - M_EVAL0, r, m_asof - 1] | |
| 119 | +Sx = S.loc[T["idx"]] | |
| 120 | +XF = np.column_stack([Sx.ln_floor, Sx.ln_land, Sx.age, Sx.age ** 2 / 100, Sx.lien == 1, Sx.lien >= 2, Sx.genre == 1, Sx.genre == 2, Sx.genre == 3, Sx.etages, Sx.neuf]).astype(float) | |
| 121 | +M10 = np.full(len(T), np.nan); M11 = np.full(len(T), np.nan); M11b = np.full(len(T), np.nan) | |
| 122 | +for a in load_json("asof.json")["asof"]: | |
| 123 | + mask = (T["asof"] == a).to_numpy(); period = str(a) | |
| 124 | + if not mask.any(): continue | |
| 125 | + mod = load_json(f"models_{period}.json"); beta = np.array([mod["beta_global"][f] for f in mod["feats"]]) | |
| 126 | + shift = level_shift(mod["m_asof"]) | |
| 127 | + fe = (T["region"].map(mod["region_fe"]).fillna(0) + T["mrc"].map(mod["mrc_eff"]).fillna(0) + T["code_mun"].map(mod["mun_eff"]).fillna(0) | |
| 128 | + + (T["code_mun"].astype(str) + "|" + T["unite_voisinage"].astype(str)).map(mod["uv_eff"]).fillna(0)).to_numpy() | |
| 129 | + M10[mask] = (XF @ beta + fe + shift + mod["median_shift"])[mask] | |
| 130 | + M11[mask] = (np.load(OUT / f"gbm_role_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 131 | + M11b[mask] = (np.load(OUT / f"gbm_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 132 | + | |
| 133 | +def e2col(sel, k, d, Tm, adj, w, recon): | |
| 134 | + m = (E2.sel == sel) & (E2.k == k) & (E2.d == d) & (E2["T"] == Tm) & (E2.adj == adj) & (E2.w == w) & (E2.recon == recon) | |
| 135 | + return int(E2.index[m][0]) | |
| 136 | +def e1col(sel, k, d, Tm, adj): | |
| 137 | + m = (E1.sel == sel) & (E1.k == k) & (E1.d == d) & (E1["T"] == Tm) & (E1.adj == adj) | |
| 138 | + return int(E1.index[m][0]) | |
| 139 | +MODELS = { | |
| 140 | + "M1 médiane des 3 plus proches (temps seul)": ("E2", e2col("geo", 3, 2, 6, "none", "equal", "median")), | |
| 141 | + "M2 moyenne des 5 plus proches (temps seul)": ("E2", e2col("geo", 5, 5, 12, "none", "equal", "mean")), | |
| 142 | + "M3 sélection par similarité physique": ("E2", e2col("sim", 5, 5, 12, "none", "equal", "median")), | |
| 143 | + "M4 pondération par distance": ("E2", e2col("geo", 5, 5, 12, "none", "dist", "mean")), | |
| 144 | + "M5 pondération par récence": ("E2", e2col("geo", 5, 5, 12, "none", "rec", "mean")), | |
| 145 | + "M6 grille d'ajustement conventionnelle": ("E2", e2col("geo", 5, 5, 12, "conv", "equal", "median")), | |
| 146 | + "M7 grille d'ajustement hédonique": ("E2", e2col("geo", 5, 5, 12, "hedo", "equal", "median")), | |
| 147 | + "M8a grille locale (régression sur le voisinage)": ("E2", e2col("geo", 5, 5, 12, "local", "equal", "median")), | |
| 148 | + "M8b ajustements non linéaires (GBM)": ("E2", e2col("geo", 5, 5, 12, "gbm", "equal", "median")), | |
| 149 | + "M13 ratio au rôle (5 plus proches, même rôle)": ("E2", e2col("geo", 5, 5, 12, "role", "equal", "median")), | |
| 150 | + "M10 hédonique globale directe": ("X", M10), | |
| 151 | + "M11a gradient boosting (sans rôle)": ("X", M11b), | |
| 152 | + "M11b gradient boosting (avec rôle)": ("X", M11), | |
| 153 | +} | |
| 154 | +def est_of(spec): | |
| 155 | + kind, c = spec | |
| 156 | + return est2[:, c].astype(float) if kind == "E2" else c | |
| 157 | +# M0 : meilleure configuration empirique choisie sur la validation (E1 ∪ E2, couverture ≥ 90 %) | |
| 158 | +cand_all = pd.concat([surf.assign(src="E1"), fac.assign(src="E2")], ignore_index=True) | |
| 159 | +cand_ok = cand_all[cand_all.couverture_val >= 97].sort_values("mdape_val") | |
| 160 | +b0 = cand_ok.iloc[0] | |
| 161 | +M0_est = (est2[:, int(b0.cfg)] if b0.src == "E2" else np.load(OUT / "est_E1.npy", mmap_mode="r")[:, int(b0.cfg)]).astype(float) | |
| 162 | +res["M0"] = b0[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_test"]].to_dict() | |
| 163 | +res["M0_sans_contrainte"] = cand_all[cand_all.couverture_val >= 50].sort_values("mdape_val").iloc[0][["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_val", "couverture_test"]].to_dict() | |
| 164 | +MODELS["M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"] = ("X", M0_est) | |
| 165 | +# M9 : réconciliation optimisée = combinaison convexe (NNLS sur la validation) d'estimations de comparables | |
| 166 | +stack_cols = {a: e2col("geo", 5, 5, 12, a, "equal", "median") for a in ["none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]} | |
| 167 | +def stack(names, mats, fit_mask): | |
| 168 | + """Combinaison convexe apprise par NNLS sur la validation ; si une composante manque pour une cible, | |
| 169 | + les poids des composantes disponibles sont renormalisés.""" | |
| 170 | + A = np.column_stack(mats); okr = ~np.isnan(A).any(axis=1) | |
| 171 | + w, _ = nnls(A[okr & fit_mask], lnP[okr & fit_mask].astype(float)); w = w / w.sum() | |
| 172 | + W = np.where(np.isnan(A), 0.0, w[None, :]); den = W.sum(axis=1) | |
| 173 | + out = np.nansum(A * W, axis=1) / np.where(den > 0, den, np.nan) | |
| 174 | + return out, dict(zip(names, w.round(3))) | |
| 175 | +M9_est, w9 = stack(list(stack_cols), [est2[:, c].astype(float) for c in stack_cols.values()], VAL) | |
| 176 | +res["M9_weights"] = w9; MODELS["M9 réconciliation optimisée (poids appris en 2024)"] = ("X", M9_est) | |
| 177 | +M12_est, w12 = stack(["gbm_role", "comps_gbm", "comps_hedo"], [M11, est2[:, stack_cols["gbm"]].astype(float), est2[:, stack_cols["hedo"]].astype(float)], VAL) | |
| 178 | +res["M12_weights"] = w12; MODELS["M12 hybride comparables + GBM (poids appris en 2024)"] = ("X", M12_est) | |
| 179 | + | |
| 180 | +rows = [] | |
| 181 | +ERR = {} | |
| 182 | +for name, spec in MODELS.items(): | |
| 183 | + e = (np.exp(est_of(spec) - lnP) - 1).astype(np.float32); ERR[name] = e | |
| 184 | + for per, mask in [("val", VAL), ("test", TEST)]: | |
| 185 | + m = metrics(e[:, None], mask).iloc[0].to_dict(); m.update(modele=name, periode=per); rows.append(m) | |
| 186 | +comp = pd.DataFrame(rows); comp.to_csv(TAB / "competition_modeles.csv", index=False) | |
| 187 | +pd.DataFrame({n.split(" ")[0]: est_of(s_) for n, s_ in MODELS.items()}).assign(idx=T["idx"].to_numpy()).to_parquet(OUT / "models_est.parquet", index=False) | |
| 188 | +res["competition"] = comp[comp.periode == "test"].set_index("modele")[["mdape", "mape", "biais", "p5", "p10", "p20", "couverture"]].round(2).to_dict("index") | |
| 189 | +res["competition_val"] = comp[comp.periode == "val"].set_index("modele")[["mdape", "p10", "couverture"]].round(2).to_dict("index") | |
| 190 | +log("compétition (test) :\n" + comp[comp.periode == "test"][["modele", "mdape", "mape", "biais", "p10", "couverture"]].round(2).to_string(index=False)) | |
| 191 | +fig, ax = plt.subplots(figsize=(7.5, 5)) | |
| 192 | +g = comp[comp.periode == "test"].sort_values("mdape", ascending=False) | |
| 193 | +cols = [PAL[0] if ("M1" in n and "M1 " not in n) or n.startswith(("M10", "M11", "M12")) else PAL[2] if n.startswith(("M0", "M9")) else PAL[1] for n in g.modele] | |
| 194 | +cols = [PAL[0] if n.startswith(("M10", "M11", "M12")) else (PAL[2] if n.startswith(("M0", "M9")) else PAL[1]) for n in g.modele] | |
| 195 | +ax.barh(g.modele, g.mdape, color=cols, height=0.65) | |
| 196 | +for i, v in enumerate(g.mdape): ax.text(v + 0.15, i, f"{v:.1f} %", va="center", fontsize=8) | |
| 197 | +ax.set_xlabel("MdAPE, test 2025-26 (%)"); ax.set_title("Compétition de modèles (orange : comparables ; vert : optimisés ; bleu : références externes)") | |
| 198 | +savefig(fig, "f4_competition") | |
| 199 | + | |
| 200 | +# ---------------------------------------------------------------- bootstrap par grappes (municipalité × trimestre) des différences appariées | |
| 201 | +rng = np.random.default_rng(7) | |
| 202 | +def paired(eA, eB, mask, B=400): | |
| 203 | + a = np.abs(eA[mask]); b = np.abs(eB[mask]); cl = T["cluster"].to_numpy()[mask] | |
| 204 | + okp = ~np.isnan(a) & ~np.isnan(b); a, b, cl = a[okp], b[okp], cl[okp] | |
| 205 | + d = a - b | |
| 206 | + cl_ids, inv = np.unique(cl, return_inverse=True); nc = len(cl_ids) | |
| 207 | + sums = np.bincount(inv, d, nc); cnts = np.bincount(inv, minlength=nc) | |
| 208 | + boots_mean = []; boots_med = [] | |
| 209 | + order = np.argsort(inv); d_sorted = d[order]; starts = np.searchsorted(inv[order], np.arange(nc)); ends = np.append(starts[1:], len(d)) | |
| 210 | + for _ in range(B): | |
| 211 | + pick = rng.integers(0, nc, nc) | |
| 212 | + boots_mean.append(sums[pick].sum() / cnts[pick].sum()) | |
| 213 | + if _ < 150: | |
| 214 | + idxs = np.concatenate([np.arange(starts[p], ends[p]) for p in pick]) | |
| 215 | + boots_med.append(np.median(a[order][idxs]) - np.median(b[order][idxs])) | |
| 216 | + return dict(n=int(len(d)), diff_mape=float(d.mean() * 100), ci_mape=[float(np.percentile(boots_mean, 2.5) * 100), float(np.percentile(boots_mean, 97.5) * 100)], | |
| 217 | + diff_mdape=float((np.median(a) - np.median(b)) * 100), ci_mdape=[float(np.percentile(boots_med, 2.5) * 100), float(np.percentile(boots_med, 97.5) * 100)], | |
| 218 | + part_A_meilleur=float((a < b).mean() * 100)) | |
| 219 | +PAIRS = [ | |
| 220 | + ("M7 grille d'ajustement hédonique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 221 | + ("M7 grille d'ajustement hédonique", "M6 grille d'ajustement conventionnelle"), | |
| 222 | + ("M8a grille locale (régression sur le voisinage)", "M7 grille d'ajustement hédonique"), | |
| 223 | + ("M8b ajustements non linéaires (GBM)", "M7 grille d'ajustement hédonique"), | |
| 224 | + ("M11b gradient boosting (avec rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 225 | + ("M11a gradient boosting (sans rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 226 | + ("M12 hybride comparables + GBM (poids appris en 2024)", "M11b gradient boosting (avec rôle)"), | |
| 227 | + ("M13 ratio au rôle (5 plus proches, même rôle)", "M7 grille d'ajustement hédonique"), | |
| 228 | + ("M3 sélection par similarité physique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 229 | + ("M9 réconciliation optimisée (poids appris en 2024)", "M7 grille d'ajustement hédonique"), | |
| 230 | + ("M1 médiane des 3 plus proches (temps seul)", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 231 | +] | |
| 232 | +res["paires"] = {f"{a} vs {b}": paired(ERR[a], ERR[b], TEST) for a, b in PAIRS} | |
| 233 | +pd.DataFrame([dict(A=a, B=b, **res["paires"][f"{a} vs {b}"]) for a, b in PAIRS]).to_csv(TAB / "paires_bootstrap.csv", index=False) | |
| 234 | + | |
| 235 | +# ---------------------------------------------------------------- strates | |
| 236 | +def strata(err, by): | |
| 237 | + out = [] | |
| 238 | + for lev, g in T[TEST].groupby(by, observed=True): | |
| 239 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 240 | + r = metrics(err[:, None], m).iloc[0].to_dict(); r[by] = str(lev); out.append(r) | |
| 241 | + return pd.DataFrame(out) | |
| 242 | +KEY = ["M2 moyenne des 5 plus proches (temps seul)", "M7 grille d'ajustement hédonique", "M8a grille locale (régression sur le voisinage)", | |
| 243 | + "M13 ratio au rôle (5 plus proches, même rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)", "M10 hédonique globale directe", "M11b gradient boosting (avec rôle)", "M12 hybride comparables + GBM (poids appris en 2024)"] | |
| 244 | +for by in ["region_nom", "dens", "prix_q", "trimestre", "age_cl", "lien"]: | |
| 245 | + tabs = [] | |
| 246 | + for name in KEY: | |
| 247 | + s = strata(ERR[name], by); s["modele"] = name.split(" ")[0]; tabs.append(s) | |
| 248 | + st = pd.concat(tabs); st.to_csv(TAB / f"strates_{by}.csv", index=False) | |
| 249 | + res[f"strates_{by}"] = st.pivot(index=by, columns="modele", values="mdape").round(2).to_dict("index") | |
| 250 | + res[f"strates_{by}_n"] = st[st.modele == "M7"].set_index(by)["n"].to_dict() | |
| 251 | +# figure : densité | |
| 252 | +fig, ax = plt.subplots(figsize=(6.5, 3.4)) | |
| 253 | +st = pd.read_csv(TAB / "strates_dens.csv") | |
| 254 | +order = ["< 5", "5-19", "20-49", "50-199", "≥ 200"] | |
| 255 | +for j, (name, lab_) in enumerate([("M2", "M2 5 plus proches"), ("M7", "M7 grille hédonique"), ("M0", "M0 meilleure config."), ("M11b", "M11b GBM"), ("M12", "M12 hybride")]): | |
| 256 | + g = st[st.modele == name].set_index("dens").loc[order] | |
| 257 | + ax.plot(order, g.mdape, color=PAL[j], lw=2, marker="o", ms=4, label=lab_) | |
| 258 | +ax.set_xlabel("ventes disponibles dans un rayon de 2 km au cours des 12 mois précédents"); ax.set_ylabel("MdAPE, test (%)"); ax.legend(fontsize=8); ax.set_title("Précision selon la liquidité du sous-marché") | |
| 259 | +savefig(fig, "f5_densite") | |
| 260 | + | |
| 261 | +# ---------------------------------------------------------------- hypothèses | |
| 262 | +H = {} | |
| 263 | +# H1 : proche mais dissemblable vs loin mais semblable (k = 5, T = 12, sans ajustement) | |
| 264 | +H["H1"] = {"geo_d1": surf.loc[e1col("geo", 5, 1, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 265 | + "sim_d10": surf.loc[e1col("sim", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 266 | + "geo_d10": surf.loc[e1col("geo", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 267 | + "mix_d10": surf.loc[e1col("mix", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 268 | + "sim_d10_hedo": surf.loc[e1col("sim", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 269 | + "geo_d10_hedo": surf.loc[e1col("geo", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 270 | + "mix_d10_hedo": surf.loc[e1col("mix", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict()} | |
| 271 | +# H2 : k optimal selon la densité (geo, d = 10, T = 12, hedo et none) | |
| 272 | +h2 = {} | |
| 273 | +for a in ["none", "hedo"]: | |
| 274 | + for lev in order: | |
| 275 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 276 | + cols_ = [e1col("geo", k, 10, 12, a) for k in [1, 3, 5, 7, 10, 15, 20, 30, 50]] | |
| 277 | + md = np.nanmedian(np.abs(err1[m][:, cols_]), axis=0) * 100 | |
| 278 | + h2[f"{a}|{lev}"] = dict(zip([1, 3, 5, 7, 10, 15, 20, 30, 50], md.round(2))) | |
| 279 | +H["H2"] = h2 | |
| 280 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.3), sharey=True) | |
| 281 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 282 | + for j, lev in enumerate(order): | |
| 283 | + v = h2[f"{a}|{lev}"]; ax.plot(list(v), list(v.values()), color=PAL[j], lw=2, marker="o", ms=3.5, label=f"densité {lev}") | |
| 284 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]); ax.set_xlabel("k"); ax.set_title({"none": "temps seul", "hedo": "grille hédonique"}[a] + " — d ≤ 10 km, 12 mois") | |
| 285 | +axes[0].set_ylabel("MdAPE, test (%)"); axes[1].legend(fontsize=8) | |
| 286 | +savefig(fig, "f6_k_par_densite") | |
| 287 | +# H3 : local vs hédonique fixe selon la densité, aux trois ancres | |
| 288 | +h3 = {} | |
| 289 | +for (k, d, Tm) in [(3, 2, 6), (5, 5, 12), (10, 10, 24)]: | |
| 290 | + for lev in order: | |
| 291 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 292 | + h3[f"{k}|{d}|{Tm}|{lev}"] = {a: float(np.nanmedian(np.abs(err2[m][:, e2col("geo", k, d, Tm, a, "equal", "median")])) * 100) for a in ["conv", "hedo", "hedo_reg", "local", "gbm"]} | |
| 293 | +H["H3"] = h3 | |
| 294 | +# H4 : pondération selon la densité (ancre 5/5/12, hedo) | |
| 295 | +h4 = {} | |
| 296 | +for lev in order: | |
| 297 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 298 | + h4[lev] = {f"{w}|{r}": float(np.nanmedian(np.abs(err2[m][:, e2col("geo", 5, 5, 12, "hedo", w, r)])) * 100) for (w, r) in [("equal", "median"), ("equal", "mean"), ("dist", "mean"), ("rec", "mean"), ("sim", "mean"), ("all", "mean"), ("all", "median")]} | |
| 299 | +H["H4"] = h4 | |
| 300 | +# H5 : méthodes simples vs ML selon l'homogénéité (dispersion des prix ajustés des 10 voisins à 10 km / 24 mois) | |
| 301 | +het = sd2[:, e2col("geo", 10, 10, 24, "hedo", "equal", "median")].astype(float) | |
| 302 | +T["heterog"] = pd.qcut(pd.Series(np.where(np.isnan(het), np.nanmax(het), het)), 3, labels=["homogène", "moyen", "hétérogène"]) | |
| 303 | +h5 = {} | |
| 304 | +for lev in ["homogène", "moyen", "hétérogène"]: | |
| 305 | + m = TEST & (T["heterog"] == lev).to_numpy() | |
| 306 | + h5[lev] = {n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY} | |
| 307 | + h5[lev]["sd_median"] = float(np.nanmedian(het[m])) | |
| 308 | +H["H5"] = h5 | |
| 309 | +res["hypotheses"] = H | |
| 310 | + | |
| 311 | +# ---------------------------------------------------------------- intervalles d'incertitude (couverture) | |
| 312 | +intervals = {} | |
| 313 | +for a in ["none", "hedo", "gbm"]: | |
| 314 | + for (k, d, Tm) in [(5, 5, 12), (10, 10, 24)]: | |
| 315 | + c = e2col("geo", k, d, Tm, a, "equal", "median"); e = est2[:, c].astype(float); s = sd2[:, c].astype(float) | |
| 316 | + okv = VAL & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 317 | + z_needed = np.abs(lnP[okv] - e[okv]) / s[okv]; z80 = float(np.percentile(z_needed, 80)) | |
| 318 | + okt = TEST & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 319 | + cov_nominal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= 1.2816 * s[okt]) * 100) | |
| 320 | + cov_cal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= z80 * s[okt]) * 100) | |
| 321 | + width = float(np.median(2 * z80 * s[okt]) * 100) | |
| 322 | + intervals[f"{a}|{k}|{d}|{Tm}"] = dict(z80_val=z80, couverture_nominale_80=cov_nominal, couverture_calibree=cov_cal, largeur_mediane_pct=width, n_test=int(okt.sum())) | |
| 323 | +res["intervalles"] = intervals | |
| 324 | + | |
| 325 | +# ---------------------------------------------------------------- règles conditionnelles (système de recommandation) | |
| 326 | +cond = {}; est_cond = np.full(len(T), np.nan) | |
| 327 | +for lev in order: | |
| 328 | + mv_ = VAL & (T["dens"] == lev).to_numpy(); mt_ = TEST & (T["dens"] == lev).to_numpy() | |
| 329 | + m1 = metrics(err1, mv_); m2 = metrics(err2, mv_) | |
| 330 | + allm = pd.concat([pd.concat([E1, m1], axis=1).assign(src="E1"), pd.concat([E2, m2], axis=1).assign(src="E2")], ignore_index=True) | |
| 331 | + b = allm[allm.couverture >= 95].sort_values("mdape").iloc[0] | |
| 332 | + e = (err1 if b.src == "E1" else err2)[:, int(b.cfg)] | |
| 333 | + est_cond[mt_] = e[mt_] | |
| 334 | + cond[lev] = dict(b[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape", "couverture"]].to_dict(), mdape_test=float(np.nanmedian(np.abs(e[mt_])) * 100), n_test=int(mt_.sum())) | |
| 335 | +res["regles_conditionnelles"] = cond | |
| 336 | +res["regles_conditionnelles_global_test"] = dict(mdape=float(np.nanmedian(np.abs(est_cond[TEST])) * 100), p10=float(np.nanmean(np.abs(est_cond[TEST]) <= 0.1) * 100), couverture=float(np.mean(~np.isnan(est_cond[TEST])) * 100)) | |
| 337 | +pd.DataFrame(cond).T.to_csv(TAB / "regles_conditionnelles.csv") | |
| 338 | + | |
| 339 | +# ---------------------------------------------------------------- stabilité temporelle, indice, divers | |
| 340 | +tq = [] | |
| 341 | +for q, g in T[TEST].groupby("trimestre"): | |
| 342 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 343 | + tq.append(dict(trimestre=q, n=int(m.sum()), **{n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY}, | |
| 344 | + **{"biais_" + n.split(" ")[0]: float(np.nanmedian(ERR[n][m]) * 100) for n in KEY})) | |
| 345 | +pd.DataFrame(tq).to_csv(TAB / "stabilite_trimestres.csv", index=False); res["stabilite"] = tq | |
| 346 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 347 | +idxf = pd.read_csv(OUT / "index_full.csv") | |
| 348 | +for j, (r, lab_) in enumerate([("QC", "Québec"), ("06", "Montréal"), ("03", "Capitale-Nationale"), ("07", "Outaouais"), ("05", "Estrie")]): | |
| 349 | + g = idxf[idxf.region.astype(str).str.zfill(2) == r] if r != "QC" else idxf[idxf.region == "QC"] | |
| 350 | + if len(g): ax.plot(pd.to_datetime(g.ym), np.exp(g.log_idx) * 100, color=PAL[j], lw=2 if r == "QC" else 1.4, label=lab_) | |
| 351 | +ax.set_ylabel("indice hédonique (2021-01 = 100)"); ax.legend(fontsize=8); ax.set_title("Indice de prix mensuel (plein échantillon, descriptif)") | |
| 352 | +savefig(fig, "f7_indice") | |
| 353 | +res["dataset"] = load_json("dataset_stats.json") | |
| 354 | +res["n_targets"] = dict(val=int(VAL.sum()), test=int(TEST.sum())) | |
| 355 | +res["densite_distribution"] = T[TEST]["dens"].value_counts().sort_index().to_dict() | |
| 356 | +res["n_cand_median"] = dict(val=float(np.median(T[VAL].n_cand)), test=float(np.median(T[TEST].n_cand))) | |
| 357 | +save_json(res, "results.json") | |
| 358 | +log("analyse terminée") | |
added
M3U96bomparables-validation/pipeline/06_report.py
+313 −0
@@ -0,0 +1,313 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""06 — Rapport : assemble docs/RAPPORT.md à partir de results.json et des tables (texte + tableaux + figures). | |
| 3 | +Les phrases d'interprétation sont calculées à partir des résultats (jamais de chiffre écrit à la main).""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +R = load_json("results.json"); TAB = OUT / "tables" | |
| 7 | +DOCS = ROOT / "docs"; DOCS.mkdir(exist_ok=True) | |
| 8 | +D = R["dataset"]; comp = R["competition"]; compv = R["competition_val"] | |
| 9 | +def f1(x): return f"{x:.1f}".replace(".", ",") | |
| 10 | +def f2(x): return f"{x:.2f}".replace(".", ",") | |
| 11 | +def n_(x): return f"{int(round(x)):,}".replace(",", " ") | |
| 12 | +def pct(x): return f"{f1(x)} %" | |
| 13 | +def md_table(df, cols=None, fmt=None, index=False): | |
| 14 | + if cols: df = df[cols] | |
| 15 | + if index: df = df.reset_index() | |
| 16 | + fmt = fmt or {} | |
| 17 | + head = "| " + " | ".join(str(c) for c in df.columns) + " |\n|" + "|".join("---" for _ in df.columns) + "|\n" | |
| 18 | + rows = [] | |
| 19 | + for _, r in df.iterrows(): | |
| 20 | + cells = [] | |
| 21 | + for c in df.columns: | |
| 22 | + v = r[c] | |
| 23 | + if c in fmt: cells.append(fmt[c](v)) | |
| 24 | + elif isinstance(v, float): cells.append(f1(v)) | |
| 25 | + else: cells.append(str(v)) | |
| 26 | + rows.append("| " + " | ".join(cells) + " |") | |
| 27 | + return head + "\n".join(rows) + "\n" | |
| 28 | +short = lambda name: name.split(" ")[0] | |
| 29 | +LAB = {"raw": "aucun ajustement", "none": "temps seulement", "conv": "grille conventionnelle", "hedo": "grille hédonique", "hedo_reg": "grille hédonique régionale", | |
| 30 | + "local": "régression locale", "gbm": "ajustements GBM", "role": "ratio au rôle", "geo": "les plus proches (distance)", "sim": "les plus semblables (physique)", | |
| 31 | + "mix": "compromis distance / similarité", "adj": "les moins ajustés", "equal": "égale", "dist": "1/distance", "rec": "récence", "all": "distance × récence × similarité", | |
| 32 | + "median": "médiane", "mean": "moyenne"} | |
| 33 | + | |
| 34 | +# ------------------------------------------------------------------ tableaux | |
| 35 | +excl = pd.DataFrame(D["exclusions"], columns=["Règle", "n"]); excl["n"] = excl["n"].map(n_) | |
| 36 | +cm = pd.DataFrame(comp).T; cm.index.name = "Modèle"; cm = cm.reset_index() | |
| 37 | +cm["val"] = cm["Modèle"].map(lambda m: compv[m]["mdape"]) | |
| 38 | +cm = cm.sort_values("mdape") | |
| 39 | +t_comp = md_table(cm.rename(columns={"mdape": "MdAPE test", "val": "MdAPE val.", "mape": "MAPE", "biais": "Biais (médian)", "p5": "±5 %", "p10": "±10 %", "p20": "±20 %", "couverture": "Couverture"}), | |
| 40 | + ["Modèle", "MdAPE val.", "MdAPE test", "MAPE", "Biais (médian)", "±5 %", "±10 %", "±20 %", "Couverture"]) | |
| 41 | +surf = pd.read_csv(TAB / "surface_E1.csv") | |
| 42 | +def surf_tab(sel, adj, T_): | |
| 43 | + g = surf[(surf.sel == sel) & (surf.adj == adj) & (surf["T"] == T_)].pivot(index="d", columns="k", values="mdape_test") | |
| 44 | + g.index = [f"≤ {d} km" for d in g.index]; g.columns = [f"k = {k}" for k in g.columns]; g.index.name = "rayon" | |
| 45 | + return md_table(g.round(1), index=True) | |
| 46 | +kb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.d == 10) & (surf["T"] == 12)].sort_values("k") | |
| 47 | +fac = pd.read_csv(TAB / "facteurs_E2.csv") | |
| 48 | +anchor = fac[(fac.k == 5) & (fac.d == 5) & (fac["T"] == 12)] | |
| 49 | +t_adj = anchor[(anchor.sel == "geo") & (anchor.w == "equal") & (anchor.recon == "median")].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 50 | +t_adj = t_adj.reset_index(); t_adj["Ajustement"] = t_adj["adj"].map(LAB) | |
| 51 | +t_adj_md = md_table(t_adj.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "mape_test": "MAPE test", "biais_test": "Biais", "p10_test": "±10 %", "couverture_test": "Couverture"}), | |
| 52 | + ["Ajustement", "MdAPE val.", "MdAPE test", "MAPE test", "Biais", "±10 %", "Couverture"]) | |
| 53 | +t_sel = anchor[(anchor.adj == "hedo") & (anchor.w == "equal") & (anchor.recon == "median")].copy(); t_sel["Sélection"] = t_sel["sel"].map(LAB) | |
| 54 | +t_sel_md = md_table(t_sel.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Sélection", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 55 | +t_wr = anchor[(anchor.adj == "hedo") & (anchor.sel == "geo")].copy(); t_wr["Pondération"] = t_wr["w"].map(LAB); t_wr["Réconciliation"] = t_wr["recon"].map(LAB) | |
| 56 | +t_wr_md = md_table(t_wr.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Pondération", "Réconciliation", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 57 | +pairs = pd.read_csv(TAB / "paires_bootstrap.csv") | |
| 58 | +pairs["Comparaison"] = pairs.apply(lambda r: f"{short(r.A)} − {short(r.B)}", axis=1) | |
| 59 | +pairs["Δ MdAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mdape)} [{f2(eval(r.ci_mdape)[0])} ; {f2(eval(r.ci_mdape)[1])}]", axis=1) | |
| 60 | +pairs["Δ MAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mape)} [{f2(eval(r.ci_mape)[0])} ; {f2(eval(r.ci_mape)[1])}]", axis=1) | |
| 61 | +pairs["A plus précis (% des cibles)"] = pairs["part_A_meilleur"].map(f1) | |
| 62 | +t_pairs = md_table(pairs, ["Comparaison", "Δ MdAPE (pts) [IC 95 %]", "Δ MAPE (pts) [IC 95 %]", "A plus précis (% des cibles)", "n"], fmt={"n": n_}) | |
| 63 | +def strata_md(by, label): | |
| 64 | + st = pd.read_csv(TAB / f"strates_{by}.csv") | |
| 65 | + g = st.pivot(index=by, columns="modele", values="mdape").round(1) | |
| 66 | + g["n"] = st[st.modele == "M7"].set_index(by)["n"] | |
| 67 | + g = g[["n", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]] | |
| 68 | + if by == "dens": g = g.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 69 | + if by == "region_nom": g = g.sort_values("n", ascending=False) | |
| 70 | + g.index.name = label | |
| 71 | + return md_table(g, index=True, fmt={"n": n_}) | |
| 72 | +H = R["hypotheses"] | |
| 73 | +h2 = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("hedo")}).T; h2.index.name = "densité"; h2.columns = [f"k = {c}" for c in h2.columns] | |
| 74 | +h2_md = md_table(h2.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 75 | +h2n = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("none")}).T; h2n.index.name = "densité"; h2n.columns = [f"k = {c}" for c in h2n.columns] | |
| 76 | +h2n_md = md_table(h2n.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 77 | +h3 = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("5|5|12")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 78 | +h3.columns = [LAB[c] for c in h3.columns]; h3.index.name = "densité"; h3_md = md_table(h3, index=True) | |
| 79 | +h3s = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("3|2|6")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 80 | +h3s.columns = [LAB[c] for c in h3s.columns]; h3s.index.name = "densité"; h3s_md = md_table(h3s, index=True) | |
| 81 | +h4 = pd.DataFrame(H["H4"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]]; h4.columns = [f"{LAB[c.split('|')[0]]} / {LAB[c.split('|')[1]]}" for c in h4.columns]; h4.index.name = "densité"; h4_md = md_table(h4, index=True) | |
| 82 | +h5 = pd.DataFrame(H["H5"]).T; h5.index.name = "sous-marché"; h5 = h5.rename(columns={"sd_median": "σ médian (log)"}); h5_md = md_table(h5[["σ médian (log)", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]], index=True, fmt={"σ médian (log)": lambda v: f"{v:.3f}".replace(".", ",")}) | |
| 83 | +iv = pd.DataFrame(R["intervalles"]).T; iv.index = [f"{LAB[i.split('|')[0]]}, k = {i.split('|')[1]}, ≤ {i.split('|')[2]} km, {i.split('|')[3]} mois" for i in iv.index]; iv.index.name = "configuration" | |
| 84 | +iv_md = md_table(iv.rename(columns={"z80_val": "z calibré (val.)", "couverture_nominale_80": "couverture à z = 1,28", "couverture_calibree": "couverture calibrée", "largeur_mediane_pct": "largeur médiane (%)", "n_test": "n"}), index=True, fmt={"n": n_, "z calibré (val.)": f2}) | |
| 85 | +rc = pd.DataFrame(R["regles_conditionnelles"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 86 | +rc["règle"] = rc.apply(lambda r: f"{LAB[r.sel]}, k = {int(r.k)}, ≤ {r.d} km, {int(r['T'])} mois, {LAB[r.adj]}, pond. {LAB[r.w]}, {LAB[r.recon]}", axis=1) | |
| 87 | +rc.index.name = "densité"; rc_md = md_table(rc.rename(columns={"mdape": "MdAPE val.", "mdape_test": "MdAPE test", "couverture": "couverture val.", "n_test": "n test"}), ["règle", "MdAPE val.", "MdAPE test", "couverture val.", "n test"], index=True, fmt={"n test": n_}) | |
| 88 | +stab = pd.DataFrame(R["stabilite"]); stab_md = md_table(stab[["trimestre", "n", "M2", "M7", "M0", "M11b", "M12", "biais_M7", "biais_M11b"]].rename(columns={"biais_M7": "biais M7", "biais_M11b": "biais M11b"}), fmt={"n": n_}) | |
| 89 | + | |
| 90 | +# ------------------------------------------------------------------ chiffres clés pour le texte | |
| 91 | +best = min(comp, key=lambda m: comp[m]["mdape"]); best_comps = min([m for m in comp if short(m) in ("M1", "M2", "M3", "M4", "M5", "M6", "M7", "M8a", "M8b", "M13", "M0", "M9")], key=lambda m: comp[m]["mdape"]) | |
| 92 | +m0 = R["M0"]; m0u = R["M0_sans_contrainte"] | |
| 93 | +gain_adj = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M7 grille d'ajustement hédonique"]["mdape"] | |
| 94 | +gain_sel = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M3 sélection par similarité physique"]["mdape"] | |
| 95 | +k_curve = {int(r.k): r.mdape_test for r in kb.itertuples()} | |
| 96 | +k_best = min(k_curve, key=k_curve.get) | |
| 97 | +e1b = R["E1_best_val"] | |
| 98 | +pair = R["paires"] | |
| 99 | +def pr(a, b): return pair[f"{a} vs {b}"] | |
| 100 | +M7 = "M7 grille d'ajustement hédonique"; M2 = "M2 moyenne des 5 plus proches (temps seul)"; M6 = "M6 grille d'ajustement conventionnelle" | |
| 101 | +M8a = "M8a grille locale (régression sur le voisinage)"; M8b = "M8b ajustements non linéaires (GBM)"; M11b = "M11b gradient boosting (avec rôle)"; M11a = "M11a gradient boosting (sans rôle)" | |
| 102 | +M0 = "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"; M12 = "M12 hybride comparables + GBM (poids appris en 2024)"; M13 = "M13 ratio au rôle (5 plus proches, même rôle)"; M9 = "M9 réconciliation optimisée (poids appris en 2024)"; M3 = "M3 sélection par similarité physique"; M1 = "M1 médiane des 3 plus proches (temps seul)"; M10 = "M10 hédonique globale directe" | |
| 103 | +dens = R["strates_dens"]; densn = R["strates_dens_n"] | |
| 104 | +h1 = H["H1"] | |
| 105 | +marg = R["E2_marginal"] | |
| 106 | +cond_g = R["regles_conditionnelles_global_test"] | |
| 107 | +nT = R["n_targets"] | |
| 108 | + | |
| 109 | +txt = f"""--- | |
| 110 | +title: "La méthode des comparables à l'épreuve des données : validation empirique à grande échelle sur {n_(D['propre'])} ventes résidentielles québécoises" | |
| 111 | +subtitle: "Sélection, nombre, ajustements, pondération et réconciliation des ventes comparables — rétro-évaluation point-dans-le-temps sur les transactions Vrai Prix 2021-2026" | |
| 112 | +author: "Simon-Pierre Boucher — version {VERSION}" | |
| 113 | +date: "{time.strftime('%d/%m/%Y')}" | |
| 114 | +lang: fr-CA | |
| 115 | +--- | |
| 116 | + | |
| 117 | +# Résumé | |
| 118 | + | |
| 119 | +**Question.** La méthode de comparaison des ventes, telle qu'enseignée et pratiquée en évaluation immobilière, produit-elle des estimations fiables, et lesquelles de ses règles (choix des comparables, nombre, rayon, ancienneté, ajustements, pondération, réconciliation) sont empiriquement justifiées ? | |
| 120 | + | |
| 121 | +**Données et protocole.** {n_(D['propre'])} ventes de résidences unifamiliales du Québec (registre foncier jumelé au rôle d'évaluation, Vrai Prix, {D['dates'][0]} → {D['dates'][1]}), après un nettoyage journalisé de {n_(D['brut'])} transactions brutes. Pour chacune des **{n_(nT['val'] + nT['test'])} ventes cibles** de 2024 (validation, {n_(nT['val'])}) et de 2025-2026 (test hors échantillon, {n_(nT['test'])}), le prix est masqué et la valeur est estimée avec les seules ventes conclues **avant** la date de la cible, un indice de prix ré-estimé chaque mois avec les seules ventes antérieures, et des grilles d'ajustement et modèles ré-estimés chaque trimestre. **2 256 configurations** de la méthode des comparables et treize modèles de référence sont appliqués aux mêmes cibles ; les différences sont testées par bootstrap par grappes (municipalité × trimestre). | |
| 122 | + | |
| 123 | +**Résultats principaux (test 2025-2026).** | |
| 124 | + | |
| 125 | +1. **Les ajustements sont le facteur dominant.** Prendre la moyenne des cinq ventes les plus proches, ajustées seulement pour le temps (M2), donne une erreur absolue médiane (MdAPE) de **{pct(comp[M2]['mdape'])}** ; une grille d'ajustement hédonique (M7) la ramène à **{pct(comp[M7]['mdape'])}** (−{f1(gain_adj)} points, IC 95 % de la différence de MdAPE [{f2(pr(M7, M2)['ci_mdape'][0])} ; {f2(pr(M7, M2)['ci_mdape'][1])}]). Une grille conventionnelle stylisée (M6) fait presque aussi bien ({pct(comp[M6]['mdape'])}) : ce qui compte, c'est d'ajuster, plus que la finesse de la grille. | |
| 126 | +2. **Choisir des comparables semblables vaut autant qu'ajuster.** Sans aucun ajustement physique, sélectionner les cinq ventes les plus **semblables** (aire, terrain, âge, lien physique, étages) plutôt que les cinq plus **proches** fait passer la MdAPE de {pct(comp[M2]['mdape'])} à **{pct(comp[M3]['mdape'])}** (H1 confirmée : un comparable un peu plus éloigné mais semblable vaut mieux qu'un voisin dissemblable). | |
| 127 | +3. **Le nombre optimal de comparables est bien supérieur à trois.** Avec la grille hédonique, la MdAPE décroît de {pct(k_curve[1])} (k = 1) à {pct(k_curve[3])} (k = 3), {pct(k_curve[5])} (k = 5), {pct(k_curve[10])} (k = 10) et atteint son minimum à **k = {k_best}** ({pct(k_curve[k_best])}) avant de remonter ({pct(k_curve[50])} à k = 50) ; l'optimum dépend de la densité du marché (H2 confirmée). | |
| 128 | +4. **Le rayon et la fenêtre temporelle doivent être larges.** La meilleure configuration de la surface Erreur(k, d, t) choisie en 2024 est « {LAB[e1b['sel']]}, k = {int(e1b['k'])}, ≤ {e1b['d']} km, {int(e1b['T'])} mois, {LAB[e1b['adj']]} » ({pct(e1b['mdape_val'])} en validation → **{pct(e1b['mdape_test'])}** en test). Les rayons de 0,5-1 km et les fenêtres de 3 mois, privilégiés par la pratique, sont dominés dès que les comparables sont ajustés. | |
| 129 | +5. **La pondération et le mode de réconciliation comptent peu** (effets marginaux moyens de {f1(min(marg['w'].values()))} à {f1(max(marg['w'].values()))} points selon la pondération, médiane {f1(marg['recon']['median'])} contre moyenne {f1(marg['recon']['mean'])}), bien moins que la sélection ({f1(min(marg['sel'].values()))}-{f1(max(marg['sel'].values()))}) et l'ajustement ({f1(min(marg['adj'].values()))}-{f1(max(marg['adj'].values()))}). | |
| 130 | +6. **La meilleure méthode des comparables, entièrement transparente et choisie en 2024, atteint {pct(comp[M0]['mdape'])} en 2025-2026** (M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]}), contre **{pct(comp[M11b]['mdape'])}** pour un gradient boosting ré-entraîné chaque trimestre avec la valeur au rôle (M11b) et {pct(comp[M11a]['mdape'])} sans le rôle (M11a). L'hybride comparables + GBM (M12) atteint **{pct(comp[M12]['mdape'])}**. Le coût de la transparence est donc de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''} de MdAPE face au meilleur modèle opaque, et il s'annule dans les sous-marchés homogènes (H5). | |
| 131 | +7. **Le ratio au rôle est un comparable redoutable** : rôle de la cible × ratio prix/rôle médian des cinq plus proches ventes du même rôle (M13) donne {pct(comp[M13]['mdape'])}, mieux que toute grille d'ajustement physique, mais seulement pour {pct(comp[M13]['couverture'])} des cibles. | |
| 132 | +8. **La fiabilité dépend d'abord de la liquidité du sous-marché** : la MdAPE de M7 passe de {pct(dens['≥ 200']['M7'])} là où plus de 200 ventes ont eu lieu à moins de 2 km dans l'année à {pct(dens['< 5']['M7'])} là où il y en a eu moins de 5 ; des règles conditionnelles à la densité (système de recommandation) obtiennent {pct(cond_g['mdape'])} globalement avec {pct(cond_g['couverture'])} de couverture. | |
| 133 | + | |
| 134 | +**Portée.** Nous validons la précision prédictive de règles professionnelles *simulées* sur des caractéristiques observables (aire, terrain, âge, étages, lien physique, rôle) ; les ajustements non observables (garage, piscine, chambres, qualité, rénovations) restent dans l'erreur résiduelle, et le jugement réel d'un évaluateur n'est pas reproduit. | |
| 135 | + | |
| 136 | +# 1. Données | |
| 137 | + | |
| 138 | +## 1.1 Source et nettoyage | |
| 139 | + | |
| 140 | +Instantané de la base de production Vrai Prix pris le 2026-10-10 : {n_(766596)} transactions résidentielles et 3,75 M d'unités d'évaluation. Nous retenons les ventes de type « unifamilial » jumelées à une unité du rôle, et appliquons des règles de nettoyage explicites et journalisées — rien n'est éliminé silencieusement. | |
| 141 | + | |
| 142 | +{md_table(excl)} | |
| 143 | +Jeu propre : **{n_(D['propre'])} ventes** ({D['dates'][0]} → {D['dates'][1]}) ; prix médian {n_(D['prix']['50%'])} $, moyenne {n_(D['prix']['mean'])} $. | |
| 144 | + | |
| 145 | +Deux règles méritent un commentaire. R8 écarte les ratios prix/rôle extrêmes (< 0,33 ou > 3) : faute d'indicateur de lien de dépendance entre les parties, c'est notre proxy de « transaction non représentative » — un évaluateur les écarterait aussi. R6b écarte les ventes conclues l'année même de la construction : leur prix médian ({n_(346108)} $) et leur ratio au rôle (1,15) sont ceux de terrains ou de constructions inachevées, non de maisons achevées (450-480 k$ et 1,3 aux âges 1-4 ans). | |
| 146 | + | |
| 147 | +## 1.2 Répartition | |
| 148 | + | |
| 149 | +| Période | Rôle | Ventes | | |
| 150 | +|---|---|---| | |
| 151 | +| 2021-01 → 2023-12 | développement | {n_(D['par_periode']['train'])} | | |
| 152 | +| 2024 | validation (choix des configurations) | {n_(D['par_periode']['val'])} | | |
| 153 | +| 2025-01 → 2026-09 | test hors échantillon | {n_(D['par_periode']['test'])} | | |
| 154 | + | |
| 155 | +Régions les plus représentées : {', '.join(f"{k} ({n_(v)})" for k, v in list(D['par_region'].items())[:6])}. | |
| 156 | + | |
| 157 | +{{ width=85% }} | |
| 158 | + | |
| 159 | +## 1.3 Caractéristiques disponibles et limites | |
| 160 | + | |
| 161 | +Observées pour chaque vente : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable, superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle en vigueur à la vente et sa part terrain. Non observées : garage, piscine, nombre de chambres et de salles de bains, qualité, état, rénovations. Les grilles d'ajustement testées ne portent donc que sur les premières ; les secondes sont une source d'erreur commune à toutes les méthodes. | |
| 162 | + | |
| 163 | +# 2. Protocole | |
| 164 | + | |
| 165 | +## 2.1 Rétro-évaluation point-dans-le-temps | |
| 166 | + | |
| 167 | +Pour chaque cible : (1) le prix est masqué ; (2) le bassin de comparables est constitué des ventes conclues strictement avant la date de la cible (la cible elle-même, si elle a déjà été vendue, est exclue : on évalue par comparables, non par vente répétée), dans un rayon de 25 km et une fenêtre de 24 mois ; (3) l'indice temporel régional utilisé pour ramener chaque comparable au niveau du dernier mois connu est ré-estimé chaque mois avec les seules ventes antérieures (régression hédonique à effets fixes de municipalité et effets mensuels, par région administrative, rétrécie vers l'indice provincial quand les ventes sont rares) ; (4) les coefficients d'ajustement hédoniques, les grilles régionales et les modèles d'apprentissage sont ré-estimés chaque trimestre avec les seules ventes antérieures au trimestre ; (5) chaque configuration produit une valeur et, le cas échéant, une dispersion des prix ajustés. | |
| 168 | + | |
| 169 | +Le niveau de prix du mois de la cible n'est jamais connu : toutes les méthodes travaillent au niveau du mois précédent, ce qui reproduit la situation réelle d'un évaluateur. | |
| 170 | + | |
| 171 | +## 2.2 Configurations testées | |
| 172 | + | |
| 173 | +**E1 — surface Erreur(k, d, t)** : k ∈ {{1, 3, 5, 7, 10, 15, 20, 30, 50}}, d ∈ {{0,5 ; 1 ; 2 ; 5 ; 10 ; 25 km}}, t ∈ {{3, 6, 12, 24 mois}}, sélection ∈ {{plus proches, plus semblables, compromis}}, ajustement ∈ {{temps seul, grille hédonique}}, médiane. 1 296 configurations. | |
| 174 | + | |
| 175 | +**E2 — plan factoriel des règles** aux ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois) : sélection (4) × ajustement (8 : aucun, temps seul, grille conventionnelle, hédonique globale, hédonique régionale, régression locale, GBM, ratio au rôle) × pondération (5) × réconciliation (2). 960 configurations. | |
| 176 | + | |
| 177 | +*Grille conventionnelle stylisée* (M6) : aire habitable ajustée à la moitié du prix au m² du comparable ; terrain à la moitié de la valeur foncière implicite au m² (plafond ±15 %) ; âge à −0,5 % par année d'écart (plafond ±20 %) ; ±5 % entre détaché et attaché. *Grille hédonique* (M7) : coefficients d'une régression du log du prix sur les caractéristiques, à effets fixes de municipalité (within), ré-estimée chaque trimestre. *Régression locale* (M8a) : mêmes coefficients ré-estimés sur les ventes à moins de 10 km des 24 derniers mois (≥ 40 ventes, ridge vers la grille régionale). *GBM* (M8b) : l'ajustement est la différence des prédictions d'un gradient boosting entre la cible et le comparable. *Ratio au rôle* (M13) : valeur au rôle de la cible × ratio prix/rôle du comparable, pour les comparables de la même municipalité et du même rôle triennal. | |
| 178 | + | |
| 179 | +## 2.3 Mesures | |
| 180 | + | |
| 181 | +Erreur absolue en pourcentage APE = |P̂ − P| / P ; MdAPE (médiane), MAPE (moyenne), biais médian signé, parts à ±5, ±10 et ±20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur — une configuration trop restrictive ne « rate » pas, elle s'abstient, et cela doit se voir). Les différences entre méthodes sont des différences d'erreurs appariées sur les mêmes cibles, avec intervalles de confiance à 95 % par bootstrap par grappes municipalité × trimestre (400 réplications). | |
| 182 | + | |
| 183 | +# 3. Résultats | |
| 184 | + | |
| 185 | +## 3.1 Combien de comparables ? À quelle distance ? De quelle ancienneté ? | |
| 186 | + | |
| 187 | +{{ width=100% }} | |
| 188 | + | |
| 189 | +{{ width=90% }} | |
| 190 | + | |
| 191 | +**Sélection par distance, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 192 | + | |
| 193 | +{surf_tab('geo', 'hedo', 12)} | |
| 194 | +**Sélection par distance, temps seulement, fenêtre 12 mois — MdAPE test (%)** | |
| 195 | + | |
| 196 | +{surf_tab('geo', 'none', 12)} | |
| 197 | +**Sélection par similarité physique, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 198 | + | |
| 199 | +{surf_tab('sim', 'hedo', 12)} | |
| 200 | +Lecture. (i) Sans ajustement, l'erreur décroît avec k et avec le rayon tant que l'on reste sous quelques kilomètres, puis se stabilise ; (ii) avec la grille hédonique, elle décroît nettement plus et l'optimum se déplace vers des k et des rayons plus grands ; (iii) la fenêtre de 3 mois est systématiquement dominée par 12-24 mois dès que l'on ajuste pour le temps. Meilleures configurations de E1 par sélection × ajustement (choisies en 2024, évaluées en 2025-2026) : | |
| 201 | + | |
| 202 | +{md_table(pd.DataFrame(R['E1_best_by_sel_adj']).T.reset_index().rename(columns={'index': 'sélection | ajustement', 'mdape_val': 'MdAPE val.', 'mdape_test': 'MdAPE test', 'couverture_test': 'couverture test'}), ['sélection | ajustement', 'k', 'd', 'T', 'MdAPE val.', 'MdAPE test', 'couverture test'], fmt={'k': lambda v: str(int(v)), 'T': lambda v: str(int(v)), 'd': lambda v: f1(v)})} | |
| 203 | +### k optimal selon la densité (H2) | |
| 204 | + | |
| 205 | +{h2_md} | |
| 206 | +*(sélection par distance, ≤ 10 km, 12 mois, grille hédonique ; MdAPE test en %, par nombre de ventes disponibles à moins de 2 km au cours des 12 mois précédents)* | |
| 207 | + | |
| 208 | +Sans ajustement : | |
| 209 | + | |
| 210 | +{h2n_md} | |
| 211 | +{{ width=100% }} | |
| 212 | + | |
| 213 | +## 3.2 Quels ajustements sont justifiés ? | |
| 214 | + | |
| 215 | +{{ width=90% }} | |
| 216 | + | |
| 217 | +{t_adj_md} | |
| 218 | +Trois enseignements. **L'ajustement temporel seul** vaut {f1(t_adj.set_index('adj').loc['raw', 'mdape_test'] - t_adj.set_index('adj').loc['none', 'mdape_test'])} point ; **n'importe quelle grille physique** vaut {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['conv', 'mdape_test'])} à {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['gbm', 'mdape_test'])} points de plus ; **les raffinements** (grille régionale, régression locale, ajustements non linéaires) apportent peu face à la grille hédonique globale, et la régression locale est même {('moins bonne' if t_adj.set_index('adj').loc['local', 'mdape_test'] > t_adj.set_index('adj').loc['hedo', 'mdape_test'] else 'meilleure')} ({pct(t_adj.set_index('adj').loc['local', 'mdape_test'])} contre {pct(t_adj.set_index('adj').loc['hedo', 'mdape_test'])}). | |
| 219 | + | |
| 220 | +### Grilles locales et petits échantillons (H3) | |
| 221 | + | |
| 222 | +MdAPE test (%) par type d'ajustement et densité, ancre (5, 5 km, 12 mois) : | |
| 223 | + | |
| 224 | +{h3_md} | |
| 225 | +Ancre (3, 2 km, 6 mois) : | |
| 226 | + | |
| 227 | +{h3s_md} | |
| 228 | +## 3.3 Comment choisir les comparables ? (H1) | |
| 229 | + | |
| 230 | +Ancre (5, 5 km, 12 mois), grille hédonique, médiane : | |
| 231 | + | |
| 232 | +{t_sel_md} | |
| 233 | +Test direct de H1 (k = 5, 12 mois, temps seulement) : les cinq voisins à moins de 1 km donnent une MdAPE de {pct(h1['geo_d1']['mdape_test'])} (couverture {pct(h1['geo_d1']['couverture_test'])}) ; les cinq ventes les plus semblables à moins de 10 km, {pct(h1['sim_d10']['mdape_test'])} (couverture {pct(h1['sim_d10']['couverture_test'])}) ; les cinq plus proches à moins de 10 km, {pct(h1['geo_d10']['mdape_test'])}. Avec la grille hédonique : plus proches {pct(h1['geo_d10_hedo']['mdape_test'])}, plus semblables {pct(h1['sim_d10_hedo']['mdape_test'])}, compromis {pct(h1['mix_d10_hedo']['mdape_test'])}. | |
| 234 | + | |
| 235 | +## 3.4 Pondération et réconciliation (H4) | |
| 236 | + | |
| 237 | +Ancre (5, 5 km, 12 mois), sélection par distance, grille hédonique : | |
| 238 | + | |
| 239 | +{t_wr_md} | |
| 240 | +Selon la densité (MdAPE test, %) : | |
| 241 | + | |
| 242 | +{h4_md} | |
| 243 | +Effets marginaux moyens (MdAPE test moyenne sur toutes les autres modalités de E2) : sélection {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['sel'].items())} ; ajustement {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['adj'].items())} ; pondération {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['w'].items())} ; réconciliation {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['recon'].items())}. | |
| 244 | + | |
| 245 | +## 3.5 Compétition de modèles | |
| 246 | + | |
| 247 | +{{ width=95% }} | |
| 248 | + | |
| 249 | +{t_comp} | |
| 250 | +M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]} (meilleure des 2 256 configurations en 2024 parmi celles qui couvrent ≥ 97 % des cibles ; sans cette contrainte, la meilleure est « {LAB[m0u['sel']]}, k = {int(m0u['k'])}, ≤ {m0u['d']} km, {int(m0u['T'])} mois, {LAB[m0u['adj']]} » à {pct(m0u['mdape_test'])} mais pour {pct(m0u['couverture_test'])} des cibles seulement). Poids appris de M9 : {', '.join(f"{LAB[k]} {f2(v)}" for k, v in R['M9_weights'].items() if v > 0)}. Poids de M12 : {', '.join(f"{k} {f2(v)}" for k, v in R['M12_weights'].items() if v > 0)}. | |
| 251 | + | |
| 252 | +### Différences appariées (test ; bootstrap par grappes municipalité × trimestre) | |
| 253 | + | |
| 254 | +{t_pairs} | |
| 255 | +Un Δ négatif signifie que A est plus précis que B ; l'intervalle qui exclut zéro signale une différence robuste aux dépendances spatiales et temporelles. | |
| 256 | + | |
| 257 | +## 3.6 Où la méthode tient-elle, où défaille-t-elle ? | |
| 258 | + | |
| 259 | +{{ width=85% }} | |
| 260 | + | |
| 261 | +**Par densité** (ventes à moins de 2 km dans les 12 mois précédents) — MdAPE test (%) : | |
| 262 | + | |
| 263 | +{strata_md('dens', 'densité')} | |
| 264 | +**Par région administrative** : | |
| 265 | + | |
| 266 | +{strata_md('region_nom', 'région')} | |
| 267 | +**Par quintile de prix** : | |
| 268 | + | |
| 269 | +{strata_md('prix_q', 'quintile de prix')} | |
| 270 | +**Par âge du bâtiment** : | |
| 271 | + | |
| 272 | +{strata_md('age_cl', 'âge')} | |
| 273 | +**Par lien physique** (0 détaché, 1 jumelé, 2 en rangée, 3 intégré) : | |
| 274 | + | |
| 275 | +{strata_md('lien', 'lien physique')} | |
| 276 | +### Marchés homogènes et hétérogènes (H5) | |
| 277 | + | |
| 278 | +Terciles de la dispersion (écart-type du log des prix ajustés) des dix ventes les plus proches à ≤ 10 km / 24 mois — MdAPE test (%) : | |
| 279 | + | |
| 280 | +{h5_md} | |
| 281 | +### Stabilité dans le temps | |
| 282 | + | |
| 283 | +{stab_md} | |
| 284 | +Le biais trimestriel commun à toutes les méthodes reflète le fait que le niveau de prix du mois courant est inconnu (toutes travaillent au niveau du mois précédent) et la saisonnalité des prix. | |
| 285 | + | |
| 286 | +## 3.7 Intervalles de valeur | |
| 287 | + | |
| 288 | +La dispersion des prix ajustés des comparables retenus fournit un intervalle de valeur. Avec z = 1,28 (couverture nominale de 80 % si les erreurs étaient normales), puis avec un z calibré sur la validation 2024 : | |
| 289 | + | |
| 290 | +{iv_md} | |
| 291 | +## 3.8 Vers un système de recommandation des comparables | |
| 292 | + | |
| 293 | +Règle choisie en 2024 pour chaque strate de densité (parmi les 2 256 configurations couvrant ≥ 95 % des cibles de la strate), évaluée en 2025-2026 : | |
| 294 | + | |
| 295 | +{rc_md} | |
| 296 | +Globalement, ces règles conditionnelles atteignent une MdAPE de **{pct(cond_g['mdape'])}** ({pct(cond_g['p10'])} des estimations à ±10 %, couverture {pct(cond_g['couverture'])}), contre {pct(comp[M0]['mdape'])} pour la meilleure règle unique. | |
| 297 | + | |
| 298 | +# 4. Discussion | |
| 299 | + | |
| 300 | +**Ce que la méthode des comparables fait bien.** Correctement paramétrée — comparables semblables, nombre d'une dizaine, rayon et fenêtre larges, ajustements hédoniques ou même conventionnels — elle atteint une précision ({pct(comp[M0]['mdape'])}) que ne dépasse le meilleur modèle d'apprentissage ré-entraîné chaque trimestre que de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''}, tout en restant intégralement explicable : chaque valeur est la médiane de quelques ventes identifiées, ajustées par des coefficients lisibles. | |
| 301 | + | |
| 302 | +**Ce qui contredit la pratique.** Trois comparables, un rayon de quelques centaines de mètres et une fenêtre de trois mois — les conventions les plus répandues — sont dominés sur toute la surface Erreur(k, d, t) : {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].mdape_test.iloc[0])} de MdAPE (k = 3, ≤ 1 km, 3 mois, grille hédonique, couverture {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].couverture_test.iloc[0])}) contre {pct(e1b['mdape_test'])} pour la meilleure configuration. La rareté des comparables « parfaits » est un argument pour élargir la recherche et ajuster, non pour se restreindre. | |
| 303 | + | |
| 304 | +**Le rôle d'évaluation comme information.** Le ratio au rôle (M13) et le GBM avec rôle (M11b) dominent leurs équivalents sans rôle : la valeur au rôle encapsule des caractéristiques non observées ici (qualité, dépendances, état) et une appréciation locale que les modèles globaux ne captent pas. Les modèles globaux sans rôle (M10, M11a) présentent d'ailleurs un biais négatif en 2025-2026 : l'appréciation est spatialement hétérogène et un indice régional, même ré-estimé chaque trimestre, la sous-estime là où les ventes se concentrent. | |
| 305 | + | |
| 306 | +**Limites.** (1) Règles professionnelles simulées, non dossiers d'évaluateurs. (2) Caractéristiques limitées à celles du rôle et du registre. (3) Délai de publication des ventes ignoré. (4) Un seul type de bien (unifamilial) ; condos et plex restent à traiter. (5) Les configurations ont été choisies sur 2024 puis évaluées sur 2025-2026 : les écarts entre validation et test (tableaux ci-dessus) mesurent la stabilité des règles, et ils sont faibles. | |
| 307 | + | |
| 308 | +# 5. Reproductibilité | |
| 309 | + | |
| 310 | +Code : `pipeline/01_dataset.py` → `06_report.py` (Python 3.14, pandas, numpy, scipy, lightgbm, matplotlib) ; sorties versionnées `out/{VERSION}/` (configurations `configs_E1.csv`, `configs_E2.csv` ; estimations `est_E1.npy`, `est_E2.npy` ; cibles `targets.parquet` ; tables `tables/*.csv` ; `results.json`). Exécution : `./run_all.sh` sur le nœud de calcul (M2U64), `./make_pdf.sh` sur le poste de travail. | |
| 311 | +""" | |
| 312 | +(DOCS / "RAPPORT.md").write_text(txt, encoding="utf-8") | |
| 313 | +log(f"rapport écrit : {DOCS / 'RAPPORT.md'} ({len(txt.split()):,} mots)") | |
added
M3U96bomparables-validation/pipeline/common.py
+85 −0
@@ -0,0 +1,85 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""Validation empirique de la méthode des comparables — fonctions communes. | |
| 3 | +Données : instantané Vrai Prix (transactions ⋈ units) du 2026-10-10. | |
| 4 | +Toutes les sorties sont versionnées sous out/<version>/. | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | +import json, math, os, sqlite3, time | |
| 8 | +from pathlib import Path | |
| 9 | +import numpy as np | |
| 10 | +import pandas as pd | |
| 11 | + | |
| 12 | +ROOT = Path(os.environ.get("CV_ROOT", Path(__file__).resolve().parents[1])) | |
| 13 | +DB = Path(os.environ.get("CV_DB", ROOT / "data" / "vraiprix-20261010.db")) | |
| 14 | +REF = ROOT / "ref" | |
| 15 | +VERSION = os.environ.get("CV_VERSION", time.strftime("%Y-%m-%d")) | |
| 16 | +OUT = ROOT / "out" / VERSION | |
| 17 | +OUT.mkdir(parents=True, exist_ok=True) | |
| 18 | +(OUT / "fig").mkdir(exist_ok=True) | |
| 19 | + | |
| 20 | +# Périodes (séparation chronologique) | |
| 21 | +TRAIN_END = "2023-12-31" # développement | |
| 22 | +VAL_START, VAL_END = "2024-01-01", "2024-12-31" # choix des configurations | |
| 23 | +TEST_START, TEST_END = "2025-01-01", "2026-09-29" # évaluation finale hors échantillon | |
| 24 | +MONTH0 = "2021-01" # premier mois des données | |
| 25 | + | |
| 26 | +def log(msg: str) -> None: | |
| 27 | + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) | |
| 28 | + | |
| 29 | +def connect(path: Path = DB) -> sqlite3.Connection: | |
| 30 | + return sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=60) | |
| 31 | + | |
| 32 | +def _json_default(o): | |
| 33 | + if isinstance(o, np.integer): return int(o) | |
| 34 | + if isinstance(o, np.floating): return None if np.isnan(o) else float(o) | |
| 35 | + if isinstance(o, np.bool_): return bool(o) | |
| 36 | + if isinstance(o, np.ndarray): return o.tolist() | |
| 37 | + if isinstance(o, pd.Timestamp): return o.strftime("%Y-%m-%d") | |
| 38 | + if isinstance(o, float) and math.isnan(o): return None | |
| 39 | + return str(o) | |
| 40 | + | |
| 41 | +def save_json(obj, name: str) -> Path: | |
| 42 | + p = OUT / name; p.parent.mkdir(parents=True, exist_ok=True) | |
| 43 | + with open(p, "w", encoding="utf-8") as f: | |
| 44 | + json.dump(obj, f, ensure_ascii=False, indent=2, default=_json_default) | |
| 45 | + return p | |
| 46 | + | |
| 47 | +def load_json(name: str): | |
| 48 | + with open(OUT / name, encoding="utf-8") as f: | |
| 49 | + return json.load(f) | |
| 50 | + | |
| 51 | +def save_parquet(df: pd.DataFrame, name: str) -> Path: | |
| 52 | + p = OUT / name; df.to_parquet(p, index=False); return p | |
| 53 | + | |
| 54 | +def load_parquet(name: str) -> pd.DataFrame: | |
| 55 | + return pd.read_parquet(OUT / name) | |
| 56 | + | |
| 57 | +def month_index(dates) -> np.ndarray: | |
| 58 | + """Mois entier depuis MONTH0 (2021-01 = 0).""" | |
| 59 | + d = pd.to_datetime(dates) | |
| 60 | + y0, m0 = map(int, MONTH0.split("-")) | |
| 61 | + return ((d.dt.year - y0) * 12 + (d.dt.month - m0)).to_numpy() | |
| 62 | + | |
| 63 | +def to_xy_km(lat, lng, lat0=46.8, lng0=-71.2): | |
| 64 | + """Projection équirectangulaire locale (km) : erreur < 1 % pour des distances < 50 km.""" | |
| 65 | + x = (np.asarray(lng, float) - lng0) * 111.32 * np.cos(np.radians(np.asarray(lat, float))) | |
| 66 | + y = (np.asarray(lat, float) - lat0) * 110.57 | |
| 67 | + return x, y | |
| 68 | + | |
| 69 | +def load_repertoire() -> pd.DataFrame: | |
| 70 | + mun = pd.read_csv(REF / "MUN.csv", dtype=str, encoding="utf-8-sig") | |
| 71 | + mun = mun.rename(columns=str.lower) | |
| 72 | + out = pd.DataFrame({ | |
| 73 | + "code_mun": mun["mcode"].str.zfill(5), | |
| 74 | + "mun_nom": mun["munnom"], | |
| 75 | + "region": mun["regadm"].str.extract(r"\((\d+)\)")[0], | |
| 76 | + "region_nom": mun["regadm"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 77 | + "mrc": mun["mrc"].str.extract(r"\((\d+)\)")[0], | |
| 78 | + "mrc_nom": mun["mrc"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 79 | + }) | |
| 80 | + return out.drop_duplicates("code_mun") | |
| 81 | + | |
| 82 | +# Codages catégoriels (ordre stable) | |
| 83 | +LIEN = {"Détaché": 0, "Jumelé": 1, "En rangée 1 côté": 2, "En rangée plus de 1 côté": 2, "Intégré": 3} | |
| 84 | +GENRE = {"De plain-pied": 0, "Unimodulaire": 0, "À étages entiers": 1, "À niveaux décalés": 2, "À étage mansardé": 3} | |
| 85 | +BTYPE = {"single-story": 0, "full-story": 1, "mansard": 2, "split-level": 3, "single-wide": 4} | |
added
M3U96bomparables-validation/run_all.sh
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Chaîne complète sur le nœud de calcul (M2U64). Usage : CV_VERSION=2026-10-10 ./run_all.sh [étape de départ] | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +PY=${PY:-$HOME/uqo-eval-terrains/env/bin/python} | |
| 6 | +export CV_VERSION=${CV_VERSION:-$(date +%F)} | |
| 7 | +START=${1:-1} | |
| 8 | +for s in 01_dataset 02_index 03_models 04_backtest 05_analyze 06_report; do | |
| 9 | + n=${s%%_*} | |
| 10 | + [ "$((10#$n))" -lt "$START" ] && continue | |
| 11 | + echo "=== $s ($(date +%T)) ===" | |
| 12 | + (cd pipeline && $PY $s.py) | |
| 13 | +done | |
| 14 | +echo "=== terminé $(date +%T) ===" | |
added
M4M64aomparables-validation/.gitignore
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +out/ | |
| 2 | +data/ | |
| 3 | +*.pyc | |
| 4 | +__pycache__/ | |
| 5 | +.DS_Store | |
added
M4M64aomparables-validation/README.md
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +# Validation empirique de la méthode des comparables (Vrai Prix) | |
| 2 | + | |
| 3 | +Projet de recherche : rétro-évaluation à grande échelle de la méthode de comparaison des ventes sur les transactions | |
| 4 | +résidentielles unifamiliales du Québec (Vrai Prix, 2021-01 → 2026-09). Protocole : `docs/PROTOCOLE.md`. | |
| 5 | +Rapport de résultats : `docs/RAPPORT.md` / `docs/RAPPORT.pdf`. | |
| 6 | + | |
| 7 | +## Organisation | |
| 8 | + | |
| 9 | +``` | |
| 10 | +pipeline/ 01_dataset → 02_index → 03_models → 04_backtest → 05_analyze → 06_report | |
| 11 | +docs/ PROTOCOLE.md, RAPPORT.md(.pdf), header.tex | |
| 12 | +out/<version>/ sales.parquet, index_asof.npy, models_*.json, est_E*.npy, targets.parquet, tables/, fig/, results.json | |
| 13 | +data/ (nœud seulement) vraiprix-20261010.db — instantané de la base de production Vrai Prix | |
| 14 | +ref/ MUN.csv (répertoire des municipalités MAMH : code → MRC → région) | |
| 15 | +``` | |
| 16 | + | |
| 17 | +## Exécution | |
| 18 | + | |
| 19 | +Le calcul se fait sur **M2U64** (`~/comparables-validation`, venv `~/uqo-eval-terrains/env`, Python 3.14 : pandas, numpy, scipy, | |
| 20 | +lightgbm, matplotlib). Le laptop garde le code, les docs et une copie des sorties. | |
| 21 | + | |
| 22 | +```bash | |
| 23 | +rsync -a --exclude out --exclude data --exclude .git ~/Desktop/Validation-Comparables/ M2U64:comparables-validation/ | |
| 24 | +ssh M2U64 'cd ~/comparables-validation && CV_VERSION=2026-10-10 ./run_all.sh' # ~ 30 min | |
| 25 | +./make_pdf.sh # rapatrie out/ + docs/ et compile le PDF | |
| 26 | +``` | |
| 27 | + | |
| 28 | +Variables : `CV_VERSION` (dossier de sortie), `CV_SAMPLE=3000` (échantillon de cibles pour un essai), `CV_PROCS` (processus, 20 par défaut). | |
| 29 | + | |
| 30 | +## Données | |
| 31 | + | |
| 32 | +Instantané `vraiprix.db` pris sur M3U96a (`~/apps/vrai-prix/data`) le 2026-10-10 : 766 596 transactions, 3,75 M d'unités du rôle. | |
| 33 | +Jeu propre : ventes `unifamilial` jumelées au rôle, règles R0-R11 journalisées dans `out/<v>/exclusions.csv`. | |
added
M4M64aomparables-validation/docs/PROTOCOLE.md
+101 −0
@@ -0,0 +1,101 @@ | ||
| 1 | +# Revisiter la méthode de comparaison des ventes : protocole de validation empirique à grande échelle | |
| 2 | + | |
| 3 | +**Titre de travail (article A)** — *Revisiting the Sales Comparison Approach: A Large-Scale Empirical Validation of Comparable Selection, Price Adjustments, and Value Reconciliation* | |
| 4 | + | |
| 5 | +**Variante** — *Does the Sales Comparison Approach Actually Work? Evidence from Large-Scale Residential Property Transactions* | |
| 6 | + | |
| 7 | +Données : transactions historiques **Vrai Prix** (registre foncier du Québec jumelé au rôle d'évaluation), instantané du 2026-10-10. | |
| 8 | + | |
| 9 | +## 1. Question de recherche | |
| 10 | + | |
| 11 | +La méthode des comparables, telle qu'enseignée et pratiquée par les évaluateurs agréés, produit-elle réellement les estimations les plus fiables ? Et surtout, quelles règles de **sélection**, d'**ajustement**, de **pondération** et de **réconciliation** des comparables sont empiriquement justifiées ? | |
| 12 | + | |
| 13 | +| Dimension | Question testée | Où dans le pipeline | | |
| 14 | +|---|---|---| | |
| 15 | +| Sélection | Quels comparables sont réellement les plus pertinents (proximité, similarité physique, « moins ajustés ») ? | E1 (facteur `sel`), E2 | | |
| 16 | +| Nombre | Faut-il 1, 3, 5, 10, 20 ou 50 ventes ? | E1 (facteur `k`) | | |
| 17 | +| Ajustements | Quels ajustements sont statistiquement justifiés (grille conventionnelle, hédonique, locale, non linéaire, ratio au rôle) ? | E2 (facteur `adj`) | | |
| 18 | +| Pondération | Égalitaire, distance, récence, similarité ? | E2 (facteur `w`) | | |
| 19 | +| Réconciliation | Médiane, moyenne, moyenne pondérée, combinaison optimisée (M9) ? | E2 (facteur `recon`), M9 | | |
| 20 | +| Fiabilité | Dans quels marchés (densité, région, segment de prix, hétérogénéité) la méthode tient-elle ou défaille-t-elle ? | strates, H2-H5 | | |
| 21 | + | |
| 22 | +## 2. Protocole expérimental (rétro-évaluation « point dans le temps ») | |
| 23 | + | |
| 24 | +Pour chaque transaction cible : | |
| 25 | + | |
| 26 | +1. **Masquage** du prix de vente réel. | |
| 27 | +2. **Reconstitution du marché à la date d'évaluation** : seules les ventes conclues *strictement avant* la date de la cible sont admissibles (le bien lui-même, s'il a déjà été vendu, est exclu du bassin : on évalue par comparables, pas par vente répétée). L'indice temporel, les coefficients hédoniques et les modèles d'apprentissage sont ré-estimés avec les seules ventes antérieures à la période d'évaluation (indice : ré-estimé chaque mois avec toutes les ventes des mois précédents ; modèles : estimés sur les ventes < 2024-01 pour la validation 2024 et < 2025-01 pour le test 2025-26). | |
| 28 | +3. **Application** de chaque configuration (sélection × rayon × fenêtre × nombre × ajustement × pondération × réconciliation) sur exactement les mêmes cibles. | |
| 29 | +4. **Comparaison** avec le prix réel : erreur absolue en pourcentage, biais, part des estimations à ±5/10/20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur), dispersion des prix ajustés (intervalle de valeur). | |
| 30 | + | |
| 31 | +### Séparation chronologique | |
| 32 | + | |
| 33 | +| Période | Rôle | Ventes | | |
| 34 | +|---|---|---| | |
| 35 | +| 2021-01 → 2023-12 | développement (estimation des grilles d'ajustement et des modèles) | ~ 200 000 | | |
| 36 | +| 2024 | validation : choix des configurations et des hyperparamètres | ~ 65 000 cibles | | |
| 37 | +| 2025-01 → 2026-09 | évaluation finale hors échantillon | ~ 125 000 cibles | | |
| 38 | + | |
| 39 | +Les comparables d'une cible de 2025 peuvent être des ventes de 2024 ou de 2025 antérieures à sa date : c'est l'information dont disposait un évaluateur ce jour-là (nous ignorons le délai de publication au registre foncier, ce qui avantage légèrement toutes les méthodes de la même façon). | |
| 40 | + | |
| 41 | +## 3. Expériences | |
| 42 | + | |
| 43 | +### E1 — surface empirique de performance Erreur(k, d, t) | |
| 44 | + | |
| 45 | +- k ∈ {1, 3, 5, 7, 10, 15, 20, 30, 50} comparables ; | |
| 46 | +- d ∈ {0,5 ; 1 ; 2 ; 5 ; 10 ; 25} km (rayon maximal) ; | |
| 47 | +- t ∈ {3, 6, 12, 24} mois (ancienneté maximale) ; | |
| 48 | +- sélection ∈ {`geo` les plus proches, `sim` les plus semblables physiquement, `mix` compromis} ; | |
| 49 | +- ajustement ∈ {`none` temps seulement, `hedo` grille hédonique} ; réconciliation = médiane. | |
| 50 | + | |
| 51 | +1 296 configurations × toutes les cibles. | |
| 52 | + | |
| 53 | +### E2 — plan factoriel des règles (trois ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois)) | |
| 54 | + | |
| 55 | +- sélection ∈ {`geo`, `sim`, `mix`, `adj` = comparables nécessitant le moins d'ajustements} ; | |
| 56 | +- ajustement ∈ {`raw` aucun, `none` temps seul, `conv` grille conventionnelle stylisée, `hedo` grille hédonique globale (within-municipalité), `hedo_reg` grille régionale rétrécie, `local` régression sur le voisinage (≤ 10 km, ≥ 40 ventes, ridge vers la grille régionale), `gbm` ajustements non linéaires (LightGBM), `role` ratio prix/rôle (comparables de la même municipalité et du même rôle)} ; | |
| 57 | +- pondération ∈ {égale, 1/(0,25 + d), exp(−mois/6), 1/(0,1 + ajustement brut), produit des trois} ; | |
| 58 | +- réconciliation ∈ {médiane (pondérée), moyenne (pondérée) en dollars}. | |
| 59 | + | |
| 60 | +960 configurations. | |
| 61 | + | |
| 62 | +### Compétition de modèles | |
| 63 | + | |
| 64 | +| Modèle | Définition opérationnelle | | |
| 65 | +|---|---| | |
| 66 | +| M1 | médiane des 3 ventes les plus proches (≤ 2 km, ≤ 6 mois), temps seul | | |
| 67 | +| M2 | moyenne des 5 plus proches (≤ 5 km, ≤ 12 mois), temps seul | | |
| 68 | +| M3 | 5 comparables choisis par similarité physique | | |
| 69 | +| M4 | 5 plus proches pondérés par la distance | | |
| 70 | +| M5 | 5 plus proches pondérés par la récence | | |
| 71 | +| M6 | grille d'ajustement conventionnelle stylisée (½ du prix au m² pour l'aire, ½ de la valeur foncière au m² pour le terrain plafonnée à 15 %, −0,5 %/an d'âge plafonné à 20 %, ±5 % détaché/attaché) | | |
| 72 | +| M7 | grille d'ajustement hédonique (coefficients within-municipalité estimés sur les ventes antérieures) | | |
| 73 | +| M8a / M8b | grille locale (régression sur le voisinage) / ajustements non linéaires (GBM) | | |
| 74 | +| M9 | réconciliation optimisée : combinaison convexe (NNLS) de sept estimations par comparables, poids appris en 2024 | | |
| 75 | +| M10 | modèle hédonique global direct (β within + effets région, MRC, municipalité, unité de voisinage) | | |
| 76 | +| M11a / M11b | gradient boosting sans / avec la valeur au rôle | | |
| 77 | +| M12 | hybride : combinaison convexe de M11b et des estimations par comparables ajustées (GBM, hédonique) | | |
| 78 | +| M13 | ratio au rôle : rôle de la cible × ratio prix/rôle médian des 5 plus proches du même rôle | | |
| 79 | +| M0 | meilleure configuration de E1 ∪ E2 choisie sur la validation 2024, évaluée en 2025-26 | | |
| 80 | + | |
| 81 | +### Hypothèses | |
| 82 | + | |
| 83 | +- **H1** — un comparable très proche mais peu semblable peut être moins informatif qu'un comparable plus éloigné mais très semblable. | |
| 84 | +- **H2** — l'ajout de comparables améliore la précision jusqu'à un seuil, au-delà duquel l'hétérogénéité domine ; le seuil dépend de la densité. | |
| 85 | +- **H3** — les ajustements empiriques locaux surpassent les grilles fixes mais deviennent instables dans les petits échantillons. | |
| 86 | +- **H4** — la pondération optimale dépend de la liquidité du sous-marché. | |
| 87 | +- **H5** — les méthodes simples et transparentes rivalisent avec l'apprentissage automatique dans les marchés homogènes. | |
| 88 | + | |
| 89 | +### Inférence | |
| 90 | + | |
| 91 | +Différences d'erreurs appariées (mêmes cibles) avec intervalles de confiance par **bootstrap par grappes** (municipalité × trimestre), qui respecte les dépendances spatiales et temporelles. | |
| 92 | + | |
| 93 | +## 4. Ce que les données permettent — et ne permettent pas | |
| 94 | + | |
| 95 | +Caractéristiques observées pour chaque vente (registre foncier ⋈ rôle) : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable (étages), superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle (en vigueur à la vente) et sa part terrain. **Non observés** : garage, piscine, nombre de chambres et de salles de bains, qualité et état, rénovations. Les ajustements correspondants ne peuvent donc pas être testés ici ; les erreurs résiduelles les contiennent. | |
| 96 | + | |
| 97 | +Nous validons la **précision prédictive** de règles professionnelles *simulées*, pas le jugement réel d'un évaluateur (il faudrait ses dossiers, ses comparables et ses ajustements). La distinction est maintenue partout dans le rapport. | |
| 98 | + | |
| 99 | +## 5. Article B (suite envisagée) | |
| 100 | + | |
| 101 | +Vers une méthode des comparables optimisée empiriquement : algorithme transparent de sélection/ajustement entraîné sur les ventes historiques et validé sur les ventes futures ; règles conditionnelles au type de bien, à la densité et à la qualité des données ; intégration dans UQO Éval pour montrer aux étudiants la comparaison entre règles traditionnelles et règles estimées. | |
added
M4M64aomparables-validation/docs/header.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% En-tête LaTeX — Grille empirique d'ajustement des terrains (UQO Éval) | |
| 2 | +\usepackage{fontspec} | |
| 3 | +\setmainfont{Helvetica Neue}[Ligatures=TeX] | |
| 4 | +\setsansfont{Helvetica Neue} | |
| 5 | +\setmonofont{Menlo}[Scale=0.82] | |
| 6 | +\usepackage{newunicodechar} | |
| 7 | +\newunicodechar{→}{\ensuremath{\rightarrow}}\newunicodechar{≥}{\ensuremath{\geq}}\newunicodechar{≤}{\ensuremath{\leq}}\newunicodechar{×}{\ensuremath{\times}}\newunicodechar{−}{\ensuremath{-}}\newunicodechar{≈}{\ensuremath{\approx}}\newunicodechar{∂}{\ensuremath{\partial}}\newunicodechar{⋈}{\ensuremath{\bowtie}}\newunicodechar{Δ}{\ensuremath{\Delta}}\newunicodechar{τ}{\ensuremath{\tau}}\newunicodechar{σ}{\ensuremath{\sigma}}\newunicodechar{β}{\ensuremath{\beta}}\newunicodechar{θ}{\ensuremath{\theta}}\newunicodechar{γ}{\ensuremath{\gamma}}\newunicodechar{α}{\ensuremath{\alpha}}\newunicodechar{δ}{\ensuremath{\delta}}\newunicodechar{ε}{\ensuremath{\varepsilon}}\newunicodechar{²}{\textsuperscript{2}}\newunicodechar{⇒}{\ensuremath{\Rightarrow}}\newunicodechar{▶}{\ensuremath{\blacktriangleright}}\newunicodechar{│}{|}\newunicodechar{▼}{\ensuremath{\blacktriangledown}}\newunicodechar{─}{-}\newunicodechar{┌}{+}\newunicodechar{┐}{+}\newunicodechar{└}{+}\newunicodechar{┘}{+}\newunicodechar{├}{+}\newunicodechar{┤}{+} | |
| 8 | +\usepackage{xcolor} | |
| 9 | +\definecolor{uqo}{HTML}{006085}\definecolor{turq}{HTML}{2BCCAE}\definecolor{corail}{HTML}{EA6047}\definecolor{encre}{HTML}{16181B}\definecolor{papier}{HTML}{F4F5F3} | |
| 10 | +\usepackage{titlesec} | |
| 11 | +\titleformat{\section}{\Large\bfseries\color{uqo}}{}{0em}{}[{\color{uqo}\titlerule[0.8pt]}] | |
| 12 | +\titleformat{\subsection}{\large\bfseries\color{uqo}}{}{0em}{} | |
| 13 | +\titlespacing*{\section}{0pt}{18pt plus 4pt}{8pt} | |
| 14 | +\usepackage{etoolbox} | |
| 15 | +\AtBeginEnvironment{longtable}{\scriptsize\renewcommand{\arraystretch}{1.18}} | |
| 16 | +\usepackage{colortbl}\arrayrulecolor{uqo!35} | |
| 17 | +\usepackage{tcolorbox} | |
| 18 | +\renewenvironment{quote}{\begin{tcolorbox}[colback=papier,colframe=turq,boxrule=0pt,leftrule=3pt,arc=1pt,left=6pt,right=6pt,top=4pt,bottom=4pt]}{\end{tcolorbox}} | |
| 19 | +\usepackage{fancyhdr} | |
| 20 | +\pagestyle{fancy}\fancyhf{} | |
| 21 | +\fancyhead[L]{\footnotesize\color{gray}Grille empirique d'ajustement des terrains — UQO Éval}\fancyhead[R]{\footnotesize\color{gray}IMM1003 · IMM1033} | |
| 22 | +\fancyfoot[C]{\footnotesize\color{gray}\thepage}\renewcommand{\headrulewidth}{0.3pt} | |
| 23 | +\usepackage{caption}\captionsetup{font=small,labelfont={bf,color=uqo}} | |
| 24 | +\usepackage{float}\floatplacement{figure}{H} | |
| 25 | +\setlength{\emergencystretch}{3em}\hyphenpenalty=300\tolerance=1500 | |
| 26 | +\usepackage{microtype} | |
| 27 | +\makeatletter\renewcommand{\maketitle}{\begin{center}{\color{uqo}\Huge\bfseries\@title\par}\vspace{8pt}{\normalsize\color{gray}\@author\par\@date\par}\end{center}\vspace{10pt}}\makeatother | |
| 28 | +\usepackage{polyglossia}\setdefaultlanguage[frenchpart=false]{french} | |
added
M4M64aomparables-validation/make_pdf.sh
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Rapatrie les résultats depuis M2U64 et produit docs/RAPPORT.pdf (pandoc → XeLaTeX) | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +V=${CV_VERSION:-2026-10-10} | |
| 6 | +mkdir -p out/$V | |
| 7 | +rsync -az --exclude 'est_E*.npy' --exclude 'sd_E2.npy' --exclude '*_pred_*.npy' --exclude 'sales.parquet' M2U64:comparables-validation/out/$V/ out/$V/ | |
| 8 | +rsync -az M2U64:comparables-validation/docs/ docs/ 2>/dev/null || true | |
| 9 | +cd docs | |
| 10 | +pandoc RAPPORT.md -o RAPPORT.pdf --pdf-engine=xelatex -H header.tex --toc --toc-depth=2 \ | |
| 11 | + -V geometry:"a4paper,margin=17mm,top=20mm,bottom=20mm" -V fontsize=10pt -V linkcolor=uqo -V urlcolor=uqo -V toccolor=uqo -V colorlinks=true \ | |
| 12 | + -V toc-title="Table des matières" --resource-path=.:../out/$V/fig 2>&1 | grep -v "Missing character\|Overfull\|Underfull" | tail -8 || true | |
| 13 | +echo "PDF : $(pwd)/RAPPORT.pdf ($(du -h RAPPORT.pdf | cut -f1))" | |
added
M4M64aomparables-validation/pipeline/01_dataset.py
+97 −0
@@ -0,0 +1,97 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""01 — Jeu de données : transactions unifamiliales Vrai Prix ⋈ rôle, nettoyage journalisé, | |
| 3 | +reventes, géographie, périodes. Sorties : sales.parquet, exclusions.csv, dataset_stats.json.""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +con = connect() | |
| 7 | +log(f"lecture {DB}") | |
| 8 | +sql = """ | |
| 9 | +select t.id, t.date, t.amount, t.lat, t.lng, t.city, t.property_type, t.year_built, t.floor_area, | |
| 10 | + t.building_type, t.id_provinc, t.valeur_role as role_vente, t.land_area, | |
| 11 | + u.code_mun, u.municipalite, u.type_prop as u_type, u.annee_construction, u.aire_etages_m2, | |
| 12 | + u.superficie_terrain_m2, u.front_terrain_m, u.nb_etages, u.nb_logements, u.lien_physique, | |
| 13 | + u.genre_construction, u.unite_voisinage, u.dat_cond_marche, u.valeur_terrain, u.valeur_batiment, | |
| 14 | + u.valeur_role as role_actuel, u.valeur_anterieure as role_anterieur, u.cubf | |
| 15 | +from transactions t left join units u on u.id_provinc = t.id_provinc | |
| 16 | +where t.property_type = 'unifamilial' | |
| 17 | +""" | |
| 18 | +df = pd.read_sql(sql, con) | |
| 19 | +log(f"{len(df):,} transactions unifamiliales brutes") | |
| 20 | +n0 = len(df) | |
| 21 | +excl = [] # (règle, n) | |
| 22 | +def drop(mask, rule): | |
| 23 | + global df | |
| 24 | + n = int(mask.sum()); excl.append((rule, n)); df = df[~mask].copy(); log(f" {rule}: -{n:,} → {len(df):,}") | |
| 25 | + | |
| 26 | +df["date"] = pd.to_datetime(df["date"]) | |
| 27 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 28 | +drop(df["id_provinc"].isna() | df["code_mun"].isna(), "R0 sans jumelage au rôle (id_provinc / code_mun)") | |
| 29 | +drop(df.duplicated(["id_provinc", "date"]), "R1 doublons (même unité, même date)") | |
| 30 | +drop(~df["u_type"].isin(["unifamilial"]) | (df["nb_logements"].fillna(1) > 1), "R2 unité du rôle non unifamiliale (chalet, plex, terrain…) ou > 1 logement") | |
| 31 | +drop(df["lat"].isna() | df["lng"].isna() | (df["lat"] < 44.5) | (df["lat"] > 63) | (df["lng"] > -56) | (df["lng"] < -80), "R3 coordonnées absentes ou hors Québec") | |
| 32 | +fa = df["floor_area"].fillna(df["aire_etages_m2"]) | |
| 33 | +df["floor_area"] = fa | |
| 34 | +drop(fa.isna() | (fa < 40) | (fa > 600), "R4 aire habitable absente ou hors [40, 600] m²") | |
| 35 | +la = df["land_area"].fillna(df["superficie_terrain_m2"]) | |
| 36 | +df["land_area"] = la | |
| 37 | +drop(la.isna() | (la < 80) | (la > 20000), "R5 superficie de terrain absente ou hors [80, 20 000] m²") | |
| 38 | +yb = df["year_built"].fillna(df["annee_construction"]) | |
| 39 | +df["year_built"] = yb | |
| 40 | +drop(yb.isna() | (yb < 1850) | (yb > df["date"].dt.year), "R6 année de construction absente, < 1850 ou postérieure à la vente (vente de terrain)") | |
| 41 | +drop((df["year_built"] == df["date"].dt.year), "R6b vente l'année même de la construction (prix du terrain ou d'une construction inachevée : médiane 346 k$ et ratio au rôle 1,15 contre 450-480 k$ et 1,3 aux âges 1-4)") | |
| 42 | +drop((df["amount"] < 50000) | (df["amount"] > 3_000_000), "R7 prix hors [50 k$, 3 M$]") | |
| 43 | +ratio = df["amount"] / df["role_vente"] | |
| 44 | +drop(df["role_vente"].isna() | (df["role_vente"] <= 0) | (ratio < 0.33) | (ratio > 3.0), "R8 rôle absent ou ratio prix/rôle hors [0,33 ; 3] (transactions présumées non représentatives)") | |
| 45 | +# reventes rapides avec forte variation (flips / rénovations : les caractéristiques enregistrées ne décrivent plus le bien) | |
| 46 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 47 | +g = df.groupby("id_provinc") | |
| 48 | +prev_date = g["date"].shift(1); prev_amt = g["amount"].shift(1) | |
| 49 | +dd = (df["date"] - prev_date).dt.days | |
| 50 | +chg = df["amount"] / prev_amt - 1 | |
| 51 | +flip = (dd <= 180) & (chg.abs() > 0.40) | |
| 52 | +drop(flip.fillna(False), "R9 revente ≤ 180 jours avec variation > 40 % (flip)") | |
| 53 | +drop(df["lien_physique"].isna() | df["genre_construction"].isna(), "R10 lien physique ou genre de construction absent au rôle") | |
| 54 | + | |
| 55 | +# ----------------------------------------------------------------- variables dérivées | |
| 56 | +rep = load_repertoire() | |
| 57 | +df = df.merge(rep, on="code_mun", how="left") | |
| 58 | +drop(df["region"].isna(), "R11 municipalité absente du répertoire MAMH") | |
| 59 | +df["sale_year"] = df["date"].dt.year | |
| 60 | +df["month"] = month_index(df["date"]) | |
| 61 | +df["day"] = (df["date"] - pd.Timestamp("2021-01-01")).dt.days.astype(int) | |
| 62 | +df["age"] = (df["sale_year"] - df["year_built"]).clip(lower=0) | |
| 63 | +df["ln_price"] = np.log(df["amount"]) | |
| 64 | +df["ln_floor"] = np.log(df["floor_area"]) | |
| 65 | +df["ln_land"] = np.log(df["land_area"]) | |
| 66 | +df["lien"] = df["lien_physique"].map(LIEN).fillna(0).astype(int) | |
| 67 | +df["genre"] = df["genre_construction"].map(GENRE).fillna(0).astype(int) | |
| 68 | +df["btype"] = df["building_type"].map(BTYPE).fillna(-1).astype(int) | |
| 69 | +df["etages"] = df["nb_etages"].fillna(1).clip(1, 3) | |
| 70 | +df["x_km"], df["y_km"] = to_xy_km(df["lat"], df["lng"]) | |
| 71 | +# millésime du rôle au moment de la vente (rôle courant / antérieur / autre) | |
| 72 | +df["role_vintage"] = np.where(np.isclose(df["role_vente"], df["role_actuel"].fillna(-1)), 2, | |
| 73 | + np.where(np.isclose(df["role_vente"], df["role_anterieur"].fillna(-1)), 1, 0)) | |
| 74 | +df["ratio_role"] = df["amount"] / df["role_vente"] | |
| 75 | +df["part_terrain"] = (df["valeur_terrain"] / df["role_actuel"]).clip(0.02, 0.95).fillna(0.2) | |
| 76 | +df["neuf"] = (df["age"] <= 3).astype(int) | |
| 77 | +df["period"] = np.select([df["date"] <= TRAIN_END, df["date"] <= VAL_END], ["train", "val"], "test") | |
| 78 | +df = df.sort_values("date").reset_index(drop=True) | |
| 79 | +df["idx"] = np.arange(len(df), dtype=np.int32) | |
| 80 | + | |
| 81 | +keep = ["idx", "id", "id_provinc", "date", "day", "month", "sale_year", "period", "amount", "ln_price", | |
| 82 | + "lat", "lng", "x_km", "y_km", "code_mun", "mun_nom", "mrc", "mrc_nom", "region", "region_nom", | |
| 83 | + "unite_voisinage", "floor_area", "ln_floor", "land_area", "ln_land", "year_built", "age", "neuf", | |
| 84 | + "lien", "genre", "btype", "etages", "role_vente", "role_vintage", "ratio_role", "part_terrain"] | |
| 85 | +df = df[keep] | |
| 86 | +save_parquet(df, "sales.parquet") | |
| 87 | +pd.DataFrame(excl, columns=["regle", "n"]).to_csv(OUT / "exclusions.csv", index=False) | |
| 88 | +stats = { | |
| 89 | + "brut": n0, "propre": len(df), "exclusions": excl, | |
| 90 | + "par_periode": df["period"].value_counts().to_dict(), | |
| 91 | + "par_annee": df["sale_year"].value_counts().sort_index().to_dict(), | |
| 92 | + "par_region": df.groupby("region_nom").size().sort_values(ascending=False).to_dict(), | |
| 93 | + "prix": {k: float(v) for k, v in df["amount"].describe().items()}, | |
| 94 | + "dates": [str(df["date"].min().date()), str(df["date"].max().date())], | |
| 95 | +} | |
| 96 | +save_json(stats, "dataset_stats.json") | |
| 97 | +log(f"jeu propre : {len(df):,} ventes ({stats['par_periode']})") | |
added
M4M64aomparables-validation/pipeline/02_index.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""02 — Indice temporel régional « point dans le temps ». | |
| 3 | +Pour chaque mois d'évaluation m, on ré-estime un indice hédonique mensuel par région administrative | |
| 4 | +avec toutes les ventes antérieures au mois m (fenêtre expansive) (jamais le mois m lui-même). Niveau de référence = m-1, | |
| 5 | +dernier mois entièrement connu. Sortie : index_asof.npy [m_eval, région, mois] (log), regions.json, | |
| 6 | +et un indice « plein échantillon » descriptif (index_full.csv).""" | |
| 7 | +from common import * | |
| 8 | + | |
| 9 | +df = load_parquet("sales.parquet") | |
| 10 | +regions = sorted(df["region"].unique().tolist()) | |
| 11 | +rid = {r: i for i, r in enumerate(regions)} | |
| 12 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 13 | +M = int(df["month"].max()) + 1 | |
| 14 | +M_EVAL0 = int(month_index(pd.Series([VAL_START]))[0]) | |
| 15 | +WINDOW = 999 # fenêtre « expansive » : toutes les ventes antérieures au mois m | |
| 16 | +SHRINK = 40.0 # n/(n+SHRINK) : rétrécissement de l'effet mensuel régional vers l'effet provincial | |
| 17 | + | |
| 18 | +X_cols = ["ln_floor", "ln_land", "age"] | |
| 19 | +def design(d): | |
| 20 | + X = np.column_stack([np.ones(len(d)), d["ln_floor"], d["ln_land"], d["age"], d["age"] ** 2 / 100, | |
| 21 | + (d["lien"] == 1), (d["lien"] >= 2), (d["genre"] == 1), (d["genre"] == 2), (d["genre"] == 3), | |
| 22 | + d["etages"], d["neuf"]]).astype(float) | |
| 23 | + return X | |
| 24 | + | |
| 25 | +def fit_month_effects(d, months): | |
| 26 | + """OLS à effets fixes de municipalité (estimateur within) : ln P ~ X + effets mensuels (base = dernier mois), | |
| 27 | + toutes les variables étant centrées par municipalité pour que l'indice ne soit pas contaminé par les | |
| 28 | + déplacements du mélange des ventes entre municipalités chères et bon marché. Renvoie effets et n par mois.""" | |
| 29 | + X = design(d)[:, 1:] | |
| 30 | + mm = d["month"].to_numpy() | |
| 31 | + base = months[-1] | |
| 32 | + dums = np.column_stack([(mm == m).astype(float) for m in months if m != base]) | |
| 33 | + A = np.hstack([X, dums]); y = d["ln_price"].to_numpy(float) | |
| 34 | + codes, inv = np.unique(d["code_mun"].to_numpy(), return_inverse=True) | |
| 35 | + cnt = np.bincount(inv).astype(float) | |
| 36 | + mA = np.zeros((len(codes), A.shape[1])); np.add.at(mA, inv, A); A = A - mA[inv] / cnt[inv, None] | |
| 37 | + my = np.bincount(inv, y) / cnt; y = y - my[inv] | |
| 38 | + beta, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 39 | + eff = {base: 0.0} | |
| 40 | + k = X.shape[1] | |
| 41 | + for j, m in enumerate([m for m in months if m != base]): | |
| 42 | + eff[m] = float(beta[k + j]) | |
| 43 | + n = {m: int((mm == m).sum()) for m in months} | |
| 44 | + return eff, n | |
| 45 | + | |
| 46 | +idx = np.full((M - M_EVAL0, len(regions), M), np.nan, dtype=np.float32) | |
| 47 | +log(f"indice point-dans-le-temps : mois d'évaluation {M_EVAL0}..{M-1}, {len(regions)} régions") | |
| 48 | +for m in range(M_EVAL0, M): | |
| 49 | + months = list(range(max(0, m - WINDOW), m)) | |
| 50 | + win = df[(df["month"] >= months[0]) & (df["month"] < m)] | |
| 51 | + prov_eff, _ = fit_month_effects(win, months) | |
| 52 | + for r in regions: | |
| 53 | + dr = win[win["region"] == r] | |
| 54 | + if len(dr) < 200: | |
| 55 | + for mo in months: idx[m - M_EVAL0, rid[r], mo] = prov_eff[mo] | |
| 56 | + continue | |
| 57 | + eff, n = fit_month_effects(dr, months) | |
| 58 | + for mo in months: | |
| 59 | + w = n[mo] / (n[mo] + SHRINK) | |
| 60 | + idx[m - M_EVAL0, rid[r], mo] = w * eff[mo] + (1 - w) * prov_eff[mo] | |
| 61 | + if m % 6 == 0: log(f" mois {m} ({len(win):,} ventes dans la fenêtre)") | |
| 62 | +np.save(OUT / "index_asof.npy", idx) | |
| 63 | +save_json({"regions": regions, "M_EVAL0": M_EVAL0, "M": M, "window": WINDOW, "shrink": SHRINK}, "index_meta.json") | |
| 64 | + | |
| 65 | +# indice descriptif plein échantillon (pour le rapport seulement — pas utilisé dans la rétro-évaluation) | |
| 66 | +rows = [] | |
| 67 | +months = list(range(0, M)) | |
| 68 | +eff_p, n_p = fit_month_effects(df, months) | |
| 69 | +for mo in months: | |
| 70 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": "QC", "log_idx": eff_p[mo] - eff_p[months[0]], "n": n_p[mo]}) | |
| 71 | +for r in regions: | |
| 72 | + dr = df[df["region"] == r] | |
| 73 | + if len(dr) < 2000: continue | |
| 74 | + eff, n = fit_month_effects(dr, months) | |
| 75 | + for mo in months: | |
| 76 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": r, "log_idx": eff[mo] - eff[months[0]], "n": n[mo]}) | |
| 77 | +pd.DataFrame(rows).to_csv(OUT / "index_full.csv", index=False) | |
| 78 | +log("indice terminé") | |
added
M4M64aomparables-validation/pipeline/03_models.py
+119 −0
@@ -0,0 +1,119 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""03 — Modèles d'ajustement et modèles de référence, ré-estimés chaque trimestre avec les seules ventes | |
| 3 | +antérieures au trimestre (as-of 2024-01, 2024-04, …, 2026-07) — comme le ferait un AVM ré-entraîné régulièrement : | |
| 4 | + • grille hédonique globale (coefficients) et régionale rétrécie → ajustements B / C | |
| 5 | + • LightGBM (objectif L1 = médiane) sur le prix ajusté dans le temps (avec et sans rôle) → ajustements C-GBM, M11 | |
| 6 | + • hédonique globale directe avec effets de municipalité → M10 | |
| 7 | +Sorties : models_<période>.json, gbm_pred_<période>.npy (prédiction f pour TOUTES les ventes).""" | |
| 8 | +import pickle | |
| 9 | +from common import * | |
| 10 | +import lightgbm as lgb | |
| 11 | + | |
| 12 | +df = load_parquet("sales.parquet") | |
| 13 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 14 | +rid = {r: i for i, r in enumerate(regions)} | |
| 15 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 16 | +idx = np.load(OUT / "index_asof.npy") | |
| 17 | + | |
| 18 | +FEATS = ["ln_floor", "ln_land", "age", "age2", "lien1", "lien2", "genre1", "genre2", "genre3", "etages", "neuf"] | |
| 19 | +def feats(d): | |
| 20 | + return pd.DataFrame({ | |
| 21 | + "ln_floor": d["ln_floor"], "ln_land": d["ln_land"], "age": d["age"], "age2": d["age"] ** 2 / 100, | |
| 22 | + "lien1": (d["lien"] == 1).astype(float), "lien2": (d["lien"] >= 2).astype(float), | |
| 23 | + "genre1": (d["genre"] == 1).astype(float), "genre2": (d["genre"] == 2).astype(float), "genre3": (d["genre"] == 3).astype(float), | |
| 24 | + "etages": d["etages"].astype(float), "neuf": d["neuf"].astype(float)}, index=d.index) | |
| 25 | + | |
| 26 | +def time_adjusted(d, m_asof): | |
| 27 | + """ln P ramené au niveau du mois m_asof-1 avec l'indice connu au mois m_asof.""" | |
| 28 | + lvl = idx[m_asof - M_EVAL0] | |
| 29 | + return d["ln_price"].to_numpy() - lvl[d["rid"].to_numpy(), d["month"].to_numpy()] + lvl[d["rid"].to_numpy(), m_asof - 1] | |
| 30 | + | |
| 31 | +def ols(X, y): | |
| 32 | + A = np.column_stack([np.ones(len(X)), X]) | |
| 33 | + b, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 34 | + return b | |
| 35 | + | |
| 36 | +ASOF = list(range(M_EVAL0, int(df["month"].max()) + 1, 3)) # ré-estimation trimestrielle (2024-01, 2024-04, …) | |
| 37 | +save_json({"asof": ASOF}, "asof.json") | |
| 38 | +TODO = [int(a) for a in os.environ["CV_ASOF"].split(",")] if os.environ.get("CV_ASOF") else ASOF # sous-ensemble (répartition sur plusieurs nœuds) | |
| 39 | +for m_asof in TODO: | |
| 40 | + period = str(m_asof); start = f"{2021 + m_asof // 12}-{m_asof % 12 + 1:02d}-01" | |
| 41 | + tr = df[df["month"] < m_asof].copy() | |
| 42 | + y = time_adjusted(tr, m_asof) | |
| 43 | + F = feats(tr) | |
| 44 | + log(f"[{period}] entraînement sur {len(tr):,} ventes < {start} (indice connu au mois {m_asof})") | |
| 45 | + # --- grille hédonique globale : estimateur « within » (effets fixes de municipalité), pour que les β | |
| 46 | + # mesurent des différences entre biens voisins et non entre marchés | |
| 47 | + def within(frame, cols, by): | |
| 48 | + return frame[cols] - frame.groupby(by)[cols].transform("mean") | |
| 49 | + # effets mensuels résiduels (base = dernier mois connu m_asof-1) : l'indice régional ne retire pas toute la | |
| 50 | + # tendance commune ; sans eux, les modèles globaux sous-estiment les ventes récentes | |
| 51 | + months_tr = sorted(tr["month"].unique()); MCOLS = [f"m{m}" for m in months_tr if m != m_asof - 1] | |
| 52 | + Fm = pd.concat([F] + [pd.Series((tr["month"] == m).astype(float), index=tr.index, name=f"m{m}") for m in months_tr if m != m_asof - 1], axis=1) | |
| 53 | + Fw = within(pd.concat([Fm, tr[["code_mun"]]], axis=1), FEATS + MCOLS, "code_mun") | |
| 54 | + yw = y - pd.Series(y, index=tr.index).groupby(tr["code_mun"]).transform("mean").to_numpy() | |
| 55 | + bw_all, *_ = np.linalg.lstsq(Fw.to_numpy(), yw, rcond=None) | |
| 56 | + bw = bw_all[:len(FEATS)]; bm = bw_all[len(FEATS):] | |
| 57 | + beta_global = dict(zip(FEATS, bw.tolist())) | |
| 58 | + xb = F.to_numpy() @ bw + Fm[MCOLS].to_numpy() @ bm | |
| 59 | + resid = y - xb | |
| 60 | + r2w = 1 - np.var(yw - Fw.to_numpy() @ bw_all) / np.var(yw) | |
| 61 | + Fw = Fw[FEATS] | |
| 62 | + log(f" hédonique within-municipalité : R² within = {r2w:.3f} ; β = { {k: round(v, 3) for k, v in beta_global.items()} }") | |
| 63 | + # --- grille régionale (within-municipalité) rétrécie vers la globale (n/(n+300)) | |
| 64 | + beta_region = {} | |
| 65 | + for r in regions: | |
| 66 | + sel = (tr["region"] == r).to_numpy() | |
| 67 | + if sel.sum() < 100: beta_region[r] = beta_global; continue | |
| 68 | + b, *_ = np.linalg.lstsq(Fw.to_numpy()[sel], yw[sel], rcond=None) | |
| 69 | + w = sel.sum() / (sel.sum() + 300.0) | |
| 70 | + beta_region[r] = {k: float(w * b[j] + (1 - w) * beta_global[k]) for j, k in enumerate(FEATS)} | |
| 71 | + # --- M10 : hédonique directe = Xβ + effet de région + MRC + municipalité + unité de voisinage (rétrécis en cascade) | |
| 72 | + reg_eff = pd.Series(resid, index=tr.index).groupby(tr["region"]).mean().to_dict() | |
| 73 | + tr["res_g"] = resid - tr["region"].map(reg_eff).to_numpy() | |
| 74 | + mrc_eff = tr.groupby("mrc")["res_g"].agg(["sum", "count"]) | |
| 75 | + mrc_eff = (mrc_eff["sum"] / (mrc_eff["count"] + 30)).to_dict() | |
| 76 | + tr["res_m"] = tr["res_g"] - tr["mrc"].map(mrc_eff).fillna(0) | |
| 77 | + mun_eff = tr.groupby("code_mun")["res_m"].agg(["sum", "count"]) | |
| 78 | + mun_eff = (mun_eff["sum"] / (mun_eff["count"] + 20)).to_dict() | |
| 79 | + tr["res_u"] = tr["res_m"] - tr["code_mun"].map(mun_eff).fillna(0) | |
| 80 | + # correction médiane : la MCO vise la moyenne du log ; la distribution est asymétrique (queue basse), on recentre sur la médiane | |
| 81 | + uv_tmp = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].transform(lambda v: v.sum() / (len(v) + 10)) | |
| 82 | + median_shift = float(np.median(tr["res_u"] - uv_tmp)) | |
| 83 | + uv = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].agg(["sum", "count"]) | |
| 84 | + uv_eff = {f"{a}|{b}": v for (a, b), v in (uv["sum"] / (uv["count"] + 10)).items()} | |
| 85 | + # --- LightGBM (prix ajusté dans le temps) avec rôle et sans rôle | |
| 86 | + def gbm_frame(d): | |
| 87 | + G = feats(d) | |
| 88 | + G["lat"] = d["lat"]; G["lng"] = d["lng"]; G["x_km"] = d["x_km"]; G["y_km"] = d["y_km"] | |
| 89 | + G["rid"] = d["rid"]; G["btype"] = d["btype"] | |
| 90 | + G["ln_role"] = np.log(d["role_vente"]); G["part_terrain"] = d["part_terrain"]; G["role_vintage"] = d["role_vintage"] | |
| 91 | + G["month"] = np.minimum(d["month"], m_asof - 1).astype(float) # tendance résiduelle ; hors échantillon = dernier mois connu | |
| 92 | + return G | |
| 93 | + G = gbm_frame(tr) | |
| 94 | + params = dict(objective="regression_l1", learning_rate=0.05, num_leaves=127, min_data_in_leaf=40, feature_fraction=0.8, | |
| 95 | + bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=int(os.environ.get("CV_THREADS", 20))) | |
| 96 | + models = {} | |
| 97 | + for name, cols in [("gbm", [c for c in G.columns if c not in ("ln_role", "part_terrain", "role_vintage")]), | |
| 98 | + ("gbm_role", list(G.columns))]: | |
| 99 | + ds = lgb.Dataset(G[cols], y, categorical_feature=["rid", "btype"] if "btype" in cols else ["rid"]) | |
| 100 | + # validation interne : derniers 10 % chronologiques pour fixer le nombre d'arbres | |
| 101 | + n = len(G); cut = int(n * 0.9) | |
| 102 | + dtr = lgb.Dataset(G[cols].iloc[:cut], y[:cut], categorical_feature=["rid", "btype"]) | |
| 103 | + dva = lgb.Dataset(G[cols].iloc[cut:], y[cut:], reference=dtr) | |
| 104 | + bst = lgb.train(params, dtr, num_boost_round=3000, valid_sets=[dva], callbacks=[lgb.early_stopping(100, verbose=False)]) | |
| 105 | + best = max(bst.best_iteration, 300) # plancher : l'arrêt précoce sur la tranche finale (mois partiels) peut être trop court | |
| 106 | + bst = lgb.train(params, ds, num_boost_round=best) | |
| 107 | + models[name] = bst | |
| 108 | + Gall = gbm_frame(df); Gall["month"] = float(m_asof - 1) # f évalué au niveau du dernier mois connu pour toutes les ventes | |
| 109 | + pred_all = bst.predict(Gall[cols]) | |
| 110 | + np.save(OUT / f"{name}_pred_{period}.npy", pred_all.astype(np.float32)) | |
| 111 | + rr = y - bst.predict(G[cols]); r2 = 1 - np.var(rr) / np.var(y) | |
| 112 | + log(f" {name}: {best} arbres, R² (in-sample) = {r2:.3f}, résidu médian = {np.median(rr) * 100:.2f} %") | |
| 113 | + bst.save_model(str(OUT / f"{name}_{period}.txt")) | |
| 114 | + save_json({"period": period, "m_asof": m_asof, "start": start, "n_train": len(tr), "feats": FEATS, | |
| 115 | + "beta_global": beta_global, "intercept_global": 0.0, "region_fe": reg_eff, | |
| 116 | + "beta_region": beta_region, "mrc_eff": mrc_eff, "mun_eff": mun_eff, "uv_eff": uv_eff, | |
| 117 | + "r2_within": float(r2w), "sd_resid_u": float(tr["res_u"].std()), "median_shift": median_shift}, | |
| 118 | + f"models_{period}.json") | |
| 119 | +log("modèles terminés") | |
added
M4M64aomparables-validation/pipeline/04_backtest.py
+193 −0
@@ -0,0 +1,193 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04 — Moteur de rétro-évaluation. | |
| 3 | +Pour chaque vente cible (2024 = validation, 2025-2026 = test), le prix est masqué ; seules les ventes | |
| 4 | +conclues STRICTEMENT AVANT la date cible (et l'indice/les modèles connus à cette date) sont utilisées. | |
| 5 | +On évalue d'un coup toutes les configurations (sélection × rayon × fenêtre × nombre × ajustement × | |
| 6 | +pondération × réconciliation) sur exactement les mêmes cibles. | |
| 7 | +Sorties : targets.parquet, configs_E1.csv/est_E1.npy (surface k,d,T), configs_E2.csv/est_E2.npy/sd_E2.npy | |
| 8 | +(factoriel sélection × ajustement × pondération × réconciliation), n_avail.npy.""" | |
| 9 | +import os | |
| 10 | +for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"): | |
| 11 | + os.environ[_v] = "1" # un seul fil BLAS par processus (fork + BLAS multi-fils = interblocage sur macOS) | |
| 12 | +import multiprocessing as mp, time | |
| 13 | +from scipy.spatial import cKDTree | |
| 14 | +from common import * | |
| 15 | + | |
| 16 | +df = load_parquet("sales.parquet") | |
| 17 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 18 | +rid_map = {r: i for i, r in enumerate(regions)} | |
| 19 | +IDX = np.load(OUT / "index_asof.npy") | |
| 20 | +ASOF = load_json("asof.json")["asof"] | |
| 21 | +MODELS = {a: load_json(f"models_{a}.json") for a in ASOF} | |
| 22 | +FEATS = MODELS[ASOF[0]]["feats"] | |
| 23 | +GBM = {a: np.load(OUT / f"gbm_pred_{a}.npy") for a in ASOF} | |
| 24 | + | |
| 25 | +# ----------------------------------------------------------------- tableaux globaux (partagés par fork) | |
| 26 | +N = len(df) | |
| 27 | +DAY = df["day"].to_numpy(np.int32); MONTH = df["month"].to_numpy(np.int32) | |
| 28 | +RID = df["region"].map(rid_map).to_numpy(np.int32) | |
| 29 | +X = df["x_km"].to_numpy(np.float64); Y = df["y_km"].to_numpy(np.float64) | |
| 30 | +LNP = df["ln_price"].to_numpy(np.float64); P = df["amount"].to_numpy(np.float64) | |
| 31 | +FLOOR = df["floor_area"].to_numpy(np.float64); LAND = df["land_area"].to_numpy(np.float64) | |
| 32 | +AGE = df["age"].to_numpy(np.float64); LIEN = df["lien"].to_numpy(np.int8); ETAGES = df["etages"].to_numpy(np.float64) | |
| 33 | +GENRE = df["genre"].to_numpy(np.int8) | |
| 34 | +PROV = pd.factorize(df["id_provinc"])[0].astype(np.int32) | |
| 35 | +MUN = pd.factorize(df["code_mun"])[0].astype(np.int32) | |
| 36 | +VINT = df["role_vintage"].to_numpy(np.int8); LNROLE = np.log(df["role_vente"].to_numpy(np.float64)) | |
| 37 | +PART = df["part_terrain"].to_numpy(np.float64) | |
| 38 | +NEUF = df["neuf"].to_numpy(np.float64) | |
| 39 | +# matrice des caractéristiques hédoniques (même ordre que FEATS) | |
| 40 | +XF = np.column_stack([np.log(FLOOR), np.log(LAND), AGE, AGE ** 2 / 100, (LIEN == 1), (LIEN >= 2), | |
| 41 | + (GENRE == 1), (GENRE == 2), (GENRE == 3), ETAGES, NEUF]).astype(np.float64) | |
| 42 | +BETA_G = {a: np.array([MODELS[a]["beta_global"][f] for f in FEATS]) for a in ASOF} | |
| 43 | +BETA_R = {a: np.array([[MODELS[a]["beta_region"][r][f] for f in FEATS] for r in regions]) for a in ASOF} | |
| 44 | +ASOF_ARR = np.array(ASOF) | |
| 45 | +TREE = cKDTree(np.column_stack([X, Y])) | |
| 46 | + | |
| 47 | +D_MAX, T_MAX_DAYS = 25.0, 731 | |
| 48 | +KS = [1, 3, 5, 7, 10, 15, 20, 30, 50] | |
| 49 | +DS = [0.5, 1, 2, 5, 10, 25] | |
| 50 | +TS = [3, 6, 12, 24] | |
| 51 | +SELS = ["geo", "sim", "mix", "adj"] | |
| 52 | +ADJS = ["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"] | |
| 53 | +WRS = [("equal", "median"), ("equal", "mean"), ("dist", "median"), ("dist", "mean"), ("rec", "median"), ("rec", "mean"), | |
| 54 | + ("sim", "median"), ("sim", "mean"), ("all", "median"), ("all", "mean")] | |
| 55 | +ANCHORS = [(3, 2, 6), (5, 5, 12), (10, 10, 24)] | |
| 56 | +E1 = [dict(sel=s, d=d, T=T, k=k, adj=a, w="equal", recon="median") | |
| 57 | + for s in ("geo", "sim", "mix") for d in DS for T in TS for k in KS for a in ("none", "hedo")] | |
| 58 | +E2 = [dict(sel=s, d=d, T=T, k=k, adj=a, w=w, recon=r) | |
| 59 | + for (k, d, T) in ANCHORS for s in SELS for a in ADJS for (w, r) in WRS] | |
| 60 | +LOCAL_COLS = [0, 1, 2, 4, 5, 9] # ln_floor, ln_land, age, lien1, lien2, etages | |
| 61 | +LAMBDA_LOCAL = 50.0 | |
| 62 | + | |
| 63 | +def wmedian(v, w): | |
| 64 | + o = np.argsort(v); v = v[o]; w = w[o] | |
| 65 | + c = np.cumsum(w); return v[np.searchsorted(c, 0.5 * c[-1])] | |
| 66 | + | |
| 67 | +def recon(vals, w, how): | |
| 68 | + ok = ~np.isnan(vals) | |
| 69 | + if not ok.any(): return np.nan, np.nan | |
| 70 | + v = vals[ok]; ww = w[ok] | |
| 71 | + sd = v.std() if len(v) > 1 else np.nan | |
| 72 | + if how == "median": | |
| 73 | + return (np.median(v) if ww.min() == ww.max() else wmedian(v, ww)), sd | |
| 74 | + return np.log(np.sum(ww * np.exp(v)) / ww.sum()), sd | |
| 75 | + | |
| 76 | +def evaluate(i): | |
| 77 | + """Toutes les configurations pour la cible i. Renvoie (est_E1, est_E2, sd_E2, n_avail, info).""" | |
| 78 | + period = int(ASOF_ARR[ASOF_ARR <= MONTH[i]].max()) # dernier trimestre de ré-estimation ≤ mois de la cible | |
| 79 | + e1 = np.full(len(E1), np.nan, np.float32); e2 = np.full(len(E2), np.nan, np.float32) | |
| 80 | + s2 = np.full(len(E2), np.nan, np.float32); nav = np.zeros(len(DS) * len(TS), np.int32) | |
| 81 | + cand = np.asarray(TREE.query_ball_point((X[i], Y[i]), D_MAX), dtype=np.int64) | |
| 82 | + keep = (DAY[cand] < DAY[i]) & (DAY[cand] >= DAY[i] - T_MAX_DAYS) & (PROV[cand] != PROV[i]) | |
| 83 | + cand = cand[keep] | |
| 84 | + info = dict(idx=int(i), asof=period, n_cand=int(len(cand))) | |
| 85 | + if len(cand) == 0: return e1, e2, s2, nav, info | |
| 86 | + dist = np.hypot(X[cand] - X[i], Y[cand] - Y[i]) | |
| 87 | + mago = (DAY[i] - DAY[cand]) / 30.4375 | |
| 88 | + m_eval = MONTH[i]; lvl = IDX[m_eval - M_EVAL0] | |
| 89 | + tadj = np.where(MONTH[cand] == m_eval, 0.0, lvl[RID[cand], m_eval - 1] - lvl[RID[cand], MONTH[cand]]) | |
| 90 | + tadj = np.nan_to_num(tadj, nan=0.0) | |
| 91 | + lnT = LNP[cand] + tadj | |
| 92 | + dX = XF[i] - XF[cand] # cible − comparable | |
| 93 | + # --- ajustements | |
| 94 | + bg = BETA_G[period]; br = BETA_R[period][RID[i]] | |
| 95 | + adj = {} | |
| 96 | + adj["raw"] = LNP[cand] | |
| 97 | + adj["none"] = lnT | |
| 98 | + Pt = np.exp(lnT) | |
| 99 | + d_floor = (FLOOR[i] - FLOOR[cand]) * 0.5 * Pt / FLOOR[cand] | |
| 100 | + d_land = np.clip((LAND[i] - LAND[cand]) * 0.5 * Pt * PART[cand] / LAND[cand], -0.15 * Pt, 0.15 * Pt) | |
| 101 | + d_age = np.clip(-0.005 * Pt * (AGE[i] - AGE[cand]), -0.2 * Pt, 0.2 * Pt) | |
| 102 | + d_lien = np.where((LIEN[i] == 0) & (LIEN[cand] > 0), 0.05 * Pt, np.where((LIEN[i] > 0) & (LIEN[cand] == 0), -0.05 * Pt, 0.0)) | |
| 103 | + tot = np.maximum(d_floor + d_land + d_age + d_lien, -0.6 * Pt) | |
| 104 | + adj["conv"] = np.log(Pt + tot) | |
| 105 | + gross_conv = (np.abs(d_floor) + np.abs(d_land) + np.abs(d_age) + np.abs(d_lien)) / Pt | |
| 106 | + adj["hedo"] = lnT + dX @ bg | |
| 107 | + gross_hedo = np.abs(dX * bg).sum(axis=1) | |
| 108 | + adj["hedo_reg"] = lnT + dX @ br | |
| 109 | + loc = dist <= 10.0 | |
| 110 | + if loc.sum() >= 40: | |
| 111 | + Xl = XF[cand][loc][:, LOCAL_COLS]; yl = lnT[loc] | |
| 112 | + b0 = br[LOCAL_COLS] | |
| 113 | + Xc = Xl - Xl.mean(axis=0); yc = yl - yl.mean() | |
| 114 | + A = Xc.T @ Xc + LAMBDA_LOCAL * np.eye(len(LOCAL_COLS)) | |
| 115 | + bl = np.linalg.solve(A, Xc.T @ yc + LAMBDA_LOCAL * b0) | |
| 116 | + adj["local"] = lnT + dX[:, LOCAL_COLS] @ bl | |
| 117 | + else: | |
| 118 | + adj["local"] = adj["hedo_reg"] | |
| 119 | + f = GBM[period] | |
| 120 | + adj["gbm"] = lnT + (f[i] - f[cand]) | |
| 121 | + same = (MUN[cand] == MUN[i]) & (VINT[cand] == VINT[i]) | |
| 122 | + adj["role"] = np.where(same, LNROLE[i] + (lnT - LNROLE[cand]), np.nan) | |
| 123 | + # --- scores de sélection | |
| 124 | + dissim = np.sqrt((dX[:, 0] / 0.25) ** 2 + (dX[:, 1] / 0.6) ** 2 + (dX[:, 2] / 15) ** 2 | |
| 125 | + + (LIEN[cand] != LIEN[i]) * 1.0 + (ETAGES[cand] != ETAGES[i]) * 0.5) | |
| 126 | + score = {"geo": dist, "sim": dissim + dist / 25.0, "mix": dissim + dist / 2.0, "adj": gross_hedo + dist / 25.0} | |
| 127 | + # --- pondérations | |
| 128 | + wts = {"equal": np.ones(len(cand)), "dist": 1.0 / (0.25 + dist), "rec": np.exp(-mago / 6.0), "sim": 1.0 / (0.1 + gross_hedo)} | |
| 129 | + wts["all"] = wts["dist"] * wts["rec"] * wts["sim"] | |
| 130 | + # --- masques (d, T) et classements | |
| 131 | + top = {} | |
| 132 | + for di, d in enumerate(DS): | |
| 133 | + for ti, T in enumerate(TS): | |
| 134 | + m = (dist <= d) & (mago <= T) | |
| 135 | + nav[di * len(TS) + ti] = int(m.sum()) | |
| 136 | + if not m.any(): continue | |
| 137 | + ids = np.flatnonzero(m) | |
| 138 | + for s in SELS: | |
| 139 | + sc = score[s][ids] | |
| 140 | + if len(ids) > 50: | |
| 141 | + part = np.argpartition(sc, 50)[:50]; part = part[np.argsort(sc[part])] | |
| 142 | + else: | |
| 143 | + part = np.argsort(sc) | |
| 144 | + top[(s, d, T)] = ids[part] | |
| 145 | + # --- E1 | |
| 146 | + for c, cfg in enumerate(E1): | |
| 147 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 148 | + if t is None: continue | |
| 149 | + sub = t[:cfg["k"]] | |
| 150 | + e1[c], _ = recon(adj[cfg["adj"]][sub], wts["equal"][sub], "median") | |
| 151 | + # --- E2 | |
| 152 | + for c, cfg in enumerate(E2): | |
| 153 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 154 | + if t is None: continue | |
| 155 | + sub = t[:cfg["k"]] | |
| 156 | + e2[c], s2[c] = recon(adj[cfg["adj"]][sub], wts[cfg["w"]][sub], cfg["recon"]) | |
| 157 | + return e1, e2, s2, nav, info | |
| 158 | + | |
| 159 | +def worker(chunk): | |
| 160 | + return [evaluate(i) for i in chunk] | |
| 161 | + | |
| 162 | +if __name__ == "__main__": | |
| 163 | + targets = df.index[df["period"].isin(["val", "test"])].to_numpy() | |
| 164 | + if os.environ.get("CV_SAMPLE"): | |
| 165 | + rng = np.random.default_rng(1); targets = np.sort(rng.choice(targets, int(os.environ["CV_SAMPLE"]), replace=False)) | |
| 166 | + SUF = "" | |
| 167 | + if os.environ.get("CV_SHARD"): # répartition sur plusieurs nœuds : CV_SHARD=i/N → cibles i, i+N, i+2N, … | |
| 168 | + i_sh, n_sh = map(int, os.environ["CV_SHARD"].split("/")); targets = targets[i_sh::n_sh]; SUF = f".shard{i_sh}of{n_sh}" | |
| 169 | + log(f"{len(targets):,} cibles ; {len(E1)} configurations E1, {len(E2)} E2 ; {N:,} ventes dans le bassin") | |
| 170 | + pd.DataFrame(E1).to_csv(OUT / "configs_E1.csv", index_label="cfg") | |
| 171 | + pd.DataFrame(E2).to_csv(OUT / "configs_E2.csv", index_label="cfg") | |
| 172 | + est1 = np.full((len(targets), len(E1)), np.nan, np.float32) | |
| 173 | + est2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 174 | + sd2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 175 | + nav = np.zeros((len(targets), len(DS) * len(TS)), np.int32) | |
| 176 | + infos = [] | |
| 177 | + chunks = [targets[j:j + 400] for j in range(0, len(targets), 400)] | |
| 178 | + t0 = time.time(); done = 0 | |
| 179 | + with mp.get_context(os.environ.get("CV_MP", "fork")).Pool(int(os.environ.get("CV_PROCS", 20))) as pool: | |
| 180 | + for res in pool.imap(worker, chunks): | |
| 181 | + for (e1, e2, s2, nv, info) in res: | |
| 182 | + est1[done] = e1; est2[done] = e2; sd2[done] = s2; nav[done] = nv; infos.append(info); done += 1 | |
| 183 | + if done % 20000 < 400: | |
| 184 | + log(f" {done:,}/{len(targets):,} cibles ({time.time() - t0:.0f} s)") | |
| 185 | + np.save(OUT / f"est_E1{SUF}.npy", est1); np.save(OUT / f"est_E2{SUF}.npy", est2); np.save(OUT / f"sd_E2{SUF}.npy", sd2); np.save(OUT / f"n_avail{SUF}.npy", nav) | |
| 186 | + T = df.loc[targets, ["idx", "id", "date", "period", "amount", "ln_price", "region", "region_nom", "mrc", "mrc_nom", "code_mun", | |
| 187 | + "mun_nom", "unite_voisinage", "floor_area", "land_area", "age", "lien", "etages", "role_vente", "month", "lat", "lng"]].reset_index(drop=True) | |
| 188 | + T["n_cand"] = [x["n_cand"] for x in infos]; T["asof"] = [x["asof"] for x in infos] | |
| 189 | + T["n_2km_12m"] = nav[:, DS.index(2) * len(TS) + TS.index(12)] | |
| 190 | + T["n_5km_12m"] = nav[:, DS.index(5) * len(TS) + TS.index(12)] | |
| 191 | + save_parquet(T, f"targets{SUF}.parquet") | |
| 192 | + pd.DataFrame({"DS": [DS], "TS": [TS], "KS": [KS]}).to_json(OUT / "grid_meta.json") | |
| 193 | + log(f"rétro-évaluation terminée en {time.time() - t0:.0f} s") | |
added
M4M64aomparables-validation/pipeline/04b_gather.py
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04b — Rassemble les fragments (shards) de la rétro-évaluation calculés sur plusieurs nœuds, dans l'ordre des cibles.""" | |
| 3 | +import glob | |
| 4 | +from common import * | |
| 5 | +sh = sorted(glob.glob(str(OUT / "targets.shard*.parquet"))) | |
| 6 | +log(f"{len(sh)} fragments") | |
| 7 | +Ts = [pd.read_parquet(p) for p in sh] | |
| 8 | +parts = {} | |
| 9 | +for name in ["est_E1", "est_E2", "sd_E2", "n_avail"]: | |
| 10 | + parts[name] = [np.load(p.replace("targets.", f"{name}.").replace(".parquet", ".npy")) for p in sh] | |
| 11 | +T = pd.concat(Ts, ignore_index=True); order = np.argsort(T["idx"].to_numpy(), kind="stable") | |
| 12 | +T = T.iloc[order].reset_index(drop=True); save_parquet(T, "targets.parquet") | |
| 13 | +for name, arrs in parts.items(): | |
| 14 | + A = np.concatenate(arrs, axis=0)[order]; np.save(OUT / f"{name}.npy", A); log(f" {name}: {A.shape}") | |
| 15 | +log("rassemblement terminé") | |
added
M4M64aomparables-validation/pipeline/05_analyze.py
+358 −0
@@ -0,0 +1,358 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""05 — Analyse : métriques par configuration (validation 2024 → choix ; test 2025-26 → verdict), | |
| 3 | +surface Erreur(k, d, t), effets des facteurs (sélection, ajustement, pondération, réconciliation), | |
| 4 | +compétition de modèles M1-M13, hypothèses H1-H5, strates (région, densité, prix), intervalles, | |
| 5 | +règles conditionnelles. Sorties : tables/*.csv, results.json, fig/*.pdf""" | |
| 6 | +from scipy.optimize import nnls | |
| 7 | +import matplotlib; matplotlib.use("Agg") | |
| 8 | +import matplotlib.pyplot as plt | |
| 9 | +from matplotlib import rcParams | |
| 10 | +from common import * | |
| 11 | + | |
| 12 | +# ---------------------------------------------------------------- style (palette de référence, marques fines) | |
| 13 | +PAL = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"] | |
| 14 | +SEQ = ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#184f95", "#0d366b"] | |
| 15 | +rcParams.update({"font.family": "Helvetica Neue", "font.size": 9, "axes.edgecolor": "#c3c2b7", "axes.linewidth": 0.6, | |
| 16 | + "axes.grid": True, "grid.color": "#e8e7e3", "grid.linewidth": 0.5, "axes.spines.top": False, | |
| 17 | + "axes.spines.right": False, "legend.frameon": False, "figure.dpi": 150, "savefig.bbox": "tight", | |
| 18 | + "axes.titleweight": "bold", "axes.titlesize": 10, "axes.labelcolor": "#52514e", "xtick.color": "#52514e", | |
| 19 | + "ytick.color": "#52514e", "figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb"}) | |
| 20 | +from matplotlib.colors import LinearSegmentedColormap | |
| 21 | +CMAP = LinearSegmentedColormap.from_list("seqblue", SEQ) | |
| 22 | +TAB = OUT / "tables"; TAB.mkdir(exist_ok=True); FIG = OUT / "fig" | |
| 23 | +def savefig(fig, name): fig.savefig(FIG / f"{name}.pdf"); fig.savefig(FIG / f"{name}.png"); plt.close(fig) | |
| 24 | + | |
| 25 | +# ---------------------------------------------------------------- chargement | |
| 26 | +T = load_parquet("targets.parquet"); S = load_parquet("sales.parquet").set_index("idx") | |
| 27 | +E1 = pd.read_csv(OUT / "configs_E1.csv"); E2 = pd.read_csv(OUT / "configs_E2.csv") | |
| 28 | +est1 = np.load(OUT / "est_E1.npy"); est2 = np.load(OUT / "est_E2.npy"); sd2 = np.load(OUT / "sd_E2.npy"); nav = np.load(OUT / "n_avail.npy") | |
| 29 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"]; IDX = np.load(OUT / "index_asof.npy") | |
| 30 | +rid = {r: i for i, r in enumerate(regions)} | |
| 31 | +lnP = T["ln_price"].to_numpy(np.float32) | |
| 32 | +VAL = (T["period"] == "val").to_numpy(); TEST = (T["period"] == "test").to_numpy() | |
| 33 | +log(f"{len(T):,} cibles (val {VAL.sum():,}, test {TEST.sum():,}) ; E1 {est1.shape[1]} ; E2 {est2.shape[1]}") | |
| 34 | + | |
| 35 | +def err_of(est): return (np.exp(est - lnP[:, None]) - 1).astype(np.float32) | |
| 36 | +def metrics(err, sel): | |
| 37 | + e = err[sel]; a = np.abs(e) | |
| 38 | + ok = ~np.isnan(a); n = ok.sum(axis=0) | |
| 39 | + with np.errstate(all="ignore"): | |
| 40 | + return pd.DataFrame({"mdape": np.nanmedian(a, axis=0) * 100, "mape": np.nanmean(a, axis=0) * 100, | |
| 41 | + "biais": np.nanmedian(e, axis=0) * 100, "biais_moy": np.nanmean(e, axis=0) * 100, | |
| 42 | + "p5": np.nanmean(a <= 0.05, axis=0) * 100, "p10": np.nanmean(a <= 0.10, axis=0) * 100, | |
| 43 | + "p20": np.nanmean(a <= 0.20, axis=0) * 100, "couverture": n / sel.sum() * 100, "n": n}) | |
| 44 | +err1 = err_of(est1); err2 = err_of(est2) | |
| 45 | +del est1 | |
| 46 | +res = {} | |
| 47 | + | |
| 48 | +# ---------------------------------------------------------------- densité & strates | |
| 49 | +T["dens"] = pd.cut(T["n_2km_12m"], [-1, 4, 19, 49, 199, 10 ** 9], labels=["< 5", "5-19", "20-49", "50-199", "≥ 200"]) | |
| 50 | +T["prix_q"] = pd.qcut(T["amount"], 5, labels=["Q1 (bas)", "Q2", "Q3", "Q4", "Q5 (haut)"]) | |
| 51 | +T["trimestre"] = pd.to_datetime(T["date"]).dt.to_period("Q").astype(str) | |
| 52 | +T["age_cl"] = pd.cut(T["age"], [-1, 3, 15, 40, 70, 1000], labels=["≤ 3 ans", "4-15", "16-40", "41-70", "> 70"]) | |
| 53 | +T["cluster"] = pd.factorize(T["code_mun"].astype(str) + "|" + T["trimestre"])[0] | |
| 54 | + | |
| 55 | +# ---------------------------------------------------------------- E1 : surface Erreur(k, d, t) | |
| 56 | +mv = metrics(err1, VAL).add_suffix("_val"); mt = metrics(err1, TEST).add_suffix("_test") | |
| 57 | +surf = pd.concat([E1, mv, mt], axis=1) | |
| 58 | +surf.to_csv(TAB / "surface_E1.csv", index=False) | |
| 59 | +ok = surf["couverture_val"] >= 97 | |
| 60 | +best_e1 = surf[ok].sort_values("mdape_val").iloc[0] | |
| 61 | +res["E1_best_val"] = best_e1.to_dict() | |
| 62 | +log(f"E1 meilleure (val, couverture ≥ 97 %) : {best_e1[['sel','d','T','k','adj']].to_dict()} → val {best_e1.mdape_val:.2f} %, test {best_e1.mdape_test:.2f} %") | |
| 63 | +for (s, a), g in surf[ok].groupby(["sel", "adj"]): | |
| 64 | + b = g.sort_values("mdape_val").iloc[0] | |
| 65 | + res.setdefault("E1_best_by_sel_adj", {})[f"{s}|{a}"] = dict(k=int(b.k), d=float(b.d), T=int(b["T"]), mdape_val=float(b.mdape_val), mdape_test=float(b.mdape_test), couverture_test=float(b.couverture_test)) | |
| 66 | + | |
| 67 | +# figure : MdAPE vs k selon d (T = 12, geo, hedo, test) | |
| 68 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.4), sharey=True) | |
| 69 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 70 | + for j, d in enumerate([0.5, 1, 2, 5, 10, 25]): | |
| 71 | + g = surf[(surf.sel == "geo") & (surf["T"] == 12) & (surf.adj == a) & (surf.d == d)].sort_values("k") | |
| 72 | + ax.plot(g.k, g.mdape_test, color=PAL[j], lw=2, marker="o", ms=4, label=f"d ≤ {d} km") | |
| 73 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]) | |
| 74 | + ax.set_xlabel("nombre de comparables k"); ax.set_title({"none": "sans ajustement (temps seulement)", "hedo": "grille hédonique"}[a]) | |
| 75 | +axes[0].set_ylabel("MdAPE, test 2025-26 (%)"); axes[1].legend(ncol=2, fontsize=8) | |
| 76 | +savefig(fig, "f1_mdape_vs_k") | |
| 77 | +# figure : carte d × T au meilleur k (geo, hedo) | |
| 78 | +fig, axes = plt.subplots(1, 2, figsize=(8, 3.2)) | |
| 79 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 80 | + g = surf[(surf.sel == "geo") & (surf.adj == a)].groupby(["d", "T"])["mdape_test"].min().unstack("T") | |
| 81 | + im = ax.imshow(g.to_numpy(), cmap=CMAP, aspect="auto") | |
| 82 | + ax.set_xticks(range(len(g.columns))); ax.set_xticklabels([f"{t} mois" for t in g.columns]); ax.set_yticks(range(len(g.index))); ax.set_yticklabels([f"{d} km" for d in g.index]) | |
| 83 | + for i in range(g.shape[0]): | |
| 84 | + for j in range(g.shape[1]): | |
| 85 | + v = g.iloc[i, j]; ax.text(j, i, f"{v:.1f}", ha="center", va="center", fontsize=7.5, color="white" if v < np.nanpercentile(g.to_numpy(), 40) else "#0b0b0b") | |
| 86 | + ax.grid(False); ax.set_title({"none": "sans ajustement", "hedo": "grille hédonique"}[a] + " — MdAPE au meilleur k") | |
| 87 | +savefig(fig, "f2_surface_d_T") | |
| 88 | + | |
| 89 | +# ---------------------------------------------------------------- E2 : facteurs | |
| 90 | +mv2 = metrics(err2, VAL).add_suffix("_val"); mt2 = metrics(err2, TEST).add_suffix("_test") | |
| 91 | +fac = pd.concat([E2, mv2, mt2], axis=1); fac.to_csv(TAB / "facteurs_E2.csv", index=False) | |
| 92 | +anchor = (fac.k == 5) & (fac.d == 5) & (fac["T"] == 12) | |
| 93 | +base = anchor & (fac.sel == "geo") & (fac.w == "equal") & (fac.recon == "median") | |
| 94 | +res["E2_adjust_anchor"] = fac[base].set_index("adj")[["mdape_val", "mdape_test", "mape_test", "biais_test", "p10_test", "couverture_test"]].to_dict("index") | |
| 95 | +res["E2_sel_anchor"] = fac[anchor & (fac.adj == "hedo") & (fac.w == "equal") & (fac.recon == "median")].set_index("sel")[["mdape_val", "mdape_test", "p10_test"]].to_dict("index") | |
| 96 | +res["E2_wr_anchor"] = {f"{r.w}|{r.recon}": dict(mdape_val=r.mdape_val, mdape_test=r.mdape_test, p10_test=r.p10_test) | |
| 97 | + for r in fac[anchor & (fac.adj == "hedo") & (fac.sel == "geo")].itertuples()} | |
| 98 | +# effets marginaux moyens (moyenne de MdAPE test sur toutes les autres modalités) | |
| 99 | +marg = {} | |
| 100 | +for f in ["sel", "adj", "w", "recon"]: | |
| 101 | + marg[f] = fac.groupby(f)["mdape_test"].mean().round(2).to_dict() | |
| 102 | +res["E2_marginal"] = marg | |
| 103 | +anc_tab = fac.groupby(["k", "d", "T"])["mdape_test"].min().round(2).to_dict() | |
| 104 | +res["E2_anchor_best"] = {f"{k}|{d}|{t}": v for (k, d, t), v in anc_tab.items()} | |
| 105 | +# figure : ajustements à l'ancre (val et test) | |
| 106 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 107 | +g = fac[base].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 108 | +lab = {"raw": "aucun", "none": "temps seul", "conv": "grille conv.", "hedo": "hédonique", "hedo_reg": "hédo. régionale", "local": "locale", "gbm": "GBM", "role": "ratio au rôle"} | |
| 109 | +x = np.arange(len(g)); ax.bar(x - 0.2, g.mdape_val, 0.38, color=PAL[0], label="validation 2024"); ax.bar(x + 0.2, g.mdape_test, 0.38, color=PAL[1], label="test 2025-26") | |
| 110 | +for i, (v1, v2) in enumerate(zip(g.mdape_val, g.mdape_test)): | |
| 111 | + ax.text(i - 0.2, v1 + 0.2, f"{v1:.1f}", ha="center", fontsize=7); ax.text(i + 0.2, v2 + 0.2, f"{v2:.1f}", ha="center", fontsize=7) | |
| 112 | +ax.set_xticks(x); ax.set_xticklabels([lab[a] for a in g.index]); ax.set_ylabel("MdAPE (%)"); ax.legend(); ax.set_title("Type d'ajustement — 5 comparables les plus proches, ≤ 5 km, ≤ 12 mois, médiane") | |
| 113 | +savefig(fig, "f3_ajustements") | |
| 114 | + | |
| 115 | +# ---------------------------------------------------------------- modèles de référence (M10, M11) et compétition | |
| 116 | +def level_shift(m_asof): | |
| 117 | + mt_ = T["month"].to_numpy(); r = T["region"].map(rid).to_numpy() | |
| 118 | + return IDX[mt_ - M_EVAL0, r, mt_ - 1] - IDX[mt_ - M_EVAL0, r, m_asof - 1] | |
| 119 | +Sx = S.loc[T["idx"]] | |
| 120 | +XF = np.column_stack([Sx.ln_floor, Sx.ln_land, Sx.age, Sx.age ** 2 / 100, Sx.lien == 1, Sx.lien >= 2, Sx.genre == 1, Sx.genre == 2, Sx.genre == 3, Sx.etages, Sx.neuf]).astype(float) | |
| 121 | +M10 = np.full(len(T), np.nan); M11 = np.full(len(T), np.nan); M11b = np.full(len(T), np.nan) | |
| 122 | +for a in load_json("asof.json")["asof"]: | |
| 123 | + mask = (T["asof"] == a).to_numpy(); period = str(a) | |
| 124 | + if not mask.any(): continue | |
| 125 | + mod = load_json(f"models_{period}.json"); beta = np.array([mod["beta_global"][f] for f in mod["feats"]]) | |
| 126 | + shift = level_shift(mod["m_asof"]) | |
| 127 | + fe = (T["region"].map(mod["region_fe"]).fillna(0) + T["mrc"].map(mod["mrc_eff"]).fillna(0) + T["code_mun"].map(mod["mun_eff"]).fillna(0) | |
| 128 | + + (T["code_mun"].astype(str) + "|" + T["unite_voisinage"].astype(str)).map(mod["uv_eff"]).fillna(0)).to_numpy() | |
| 129 | + M10[mask] = (XF @ beta + fe + shift + mod["median_shift"])[mask] | |
| 130 | + M11[mask] = (np.load(OUT / f"gbm_role_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 131 | + M11b[mask] = (np.load(OUT / f"gbm_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 132 | + | |
| 133 | +def e2col(sel, k, d, Tm, adj, w, recon): | |
| 134 | + m = (E2.sel == sel) & (E2.k == k) & (E2.d == d) & (E2["T"] == Tm) & (E2.adj == adj) & (E2.w == w) & (E2.recon == recon) | |
| 135 | + return int(E2.index[m][0]) | |
| 136 | +def e1col(sel, k, d, Tm, adj): | |
| 137 | + m = (E1.sel == sel) & (E1.k == k) & (E1.d == d) & (E1["T"] == Tm) & (E1.adj == adj) | |
| 138 | + return int(E1.index[m][0]) | |
| 139 | +MODELS = { | |
| 140 | + "M1 médiane des 3 plus proches (temps seul)": ("E2", e2col("geo", 3, 2, 6, "none", "equal", "median")), | |
| 141 | + "M2 moyenne des 5 plus proches (temps seul)": ("E2", e2col("geo", 5, 5, 12, "none", "equal", "mean")), | |
| 142 | + "M3 sélection par similarité physique": ("E2", e2col("sim", 5, 5, 12, "none", "equal", "median")), | |
| 143 | + "M4 pondération par distance": ("E2", e2col("geo", 5, 5, 12, "none", "dist", "mean")), | |
| 144 | + "M5 pondération par récence": ("E2", e2col("geo", 5, 5, 12, "none", "rec", "mean")), | |
| 145 | + "M6 grille d'ajustement conventionnelle": ("E2", e2col("geo", 5, 5, 12, "conv", "equal", "median")), | |
| 146 | + "M7 grille d'ajustement hédonique": ("E2", e2col("geo", 5, 5, 12, "hedo", "equal", "median")), | |
| 147 | + "M8a grille locale (régression sur le voisinage)": ("E2", e2col("geo", 5, 5, 12, "local", "equal", "median")), | |
| 148 | + "M8b ajustements non linéaires (GBM)": ("E2", e2col("geo", 5, 5, 12, "gbm", "equal", "median")), | |
| 149 | + "M13 ratio au rôle (5 plus proches, même rôle)": ("E2", e2col("geo", 5, 5, 12, "role", "equal", "median")), | |
| 150 | + "M10 hédonique globale directe": ("X", M10), | |
| 151 | + "M11a gradient boosting (sans rôle)": ("X", M11b), | |
| 152 | + "M11b gradient boosting (avec rôle)": ("X", M11), | |
| 153 | +} | |
| 154 | +def est_of(spec): | |
| 155 | + kind, c = spec | |
| 156 | + return est2[:, c].astype(float) if kind == "E2" else c | |
| 157 | +# M0 : meilleure configuration empirique choisie sur la validation (E1 ∪ E2, couverture ≥ 90 %) | |
| 158 | +cand_all = pd.concat([surf.assign(src="E1"), fac.assign(src="E2")], ignore_index=True) | |
| 159 | +cand_ok = cand_all[cand_all.couverture_val >= 97].sort_values("mdape_val") | |
| 160 | +b0 = cand_ok.iloc[0] | |
| 161 | +M0_est = (est2[:, int(b0.cfg)] if b0.src == "E2" else np.load(OUT / "est_E1.npy", mmap_mode="r")[:, int(b0.cfg)]).astype(float) | |
| 162 | +res["M0"] = b0[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_test"]].to_dict() | |
| 163 | +res["M0_sans_contrainte"] = cand_all[cand_all.couverture_val >= 50].sort_values("mdape_val").iloc[0][["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_val", "couverture_test"]].to_dict() | |
| 164 | +MODELS["M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"] = ("X", M0_est) | |
| 165 | +# M9 : réconciliation optimisée = combinaison convexe (NNLS sur la validation) d'estimations de comparables | |
| 166 | +stack_cols = {a: e2col("geo", 5, 5, 12, a, "equal", "median") for a in ["none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]} | |
| 167 | +def stack(names, mats, fit_mask): | |
| 168 | + """Combinaison convexe apprise par NNLS sur la validation ; si une composante manque pour une cible, | |
| 169 | + les poids des composantes disponibles sont renormalisés.""" | |
| 170 | + A = np.column_stack(mats); okr = ~np.isnan(A).any(axis=1) | |
| 171 | + w, _ = nnls(A[okr & fit_mask], lnP[okr & fit_mask].astype(float)); w = w / w.sum() | |
| 172 | + W = np.where(np.isnan(A), 0.0, w[None, :]); den = W.sum(axis=1) | |
| 173 | + out = np.nansum(A * W, axis=1) / np.where(den > 0, den, np.nan) | |
| 174 | + return out, dict(zip(names, w.round(3))) | |
| 175 | +M9_est, w9 = stack(list(stack_cols), [est2[:, c].astype(float) for c in stack_cols.values()], VAL) | |
| 176 | +res["M9_weights"] = w9; MODELS["M9 réconciliation optimisée (poids appris en 2024)"] = ("X", M9_est) | |
| 177 | +M12_est, w12 = stack(["gbm_role", "comps_gbm", "comps_hedo"], [M11, est2[:, stack_cols["gbm"]].astype(float), est2[:, stack_cols["hedo"]].astype(float)], VAL) | |
| 178 | +res["M12_weights"] = w12; MODELS["M12 hybride comparables + GBM (poids appris en 2024)"] = ("X", M12_est) | |
| 179 | + | |
| 180 | +rows = [] | |
| 181 | +ERR = {} | |
| 182 | +for name, spec in MODELS.items(): | |
| 183 | + e = (np.exp(est_of(spec) - lnP) - 1).astype(np.float32); ERR[name] = e | |
| 184 | + for per, mask in [("val", VAL), ("test", TEST)]: | |
| 185 | + m = metrics(e[:, None], mask).iloc[0].to_dict(); m.update(modele=name, periode=per); rows.append(m) | |
| 186 | +comp = pd.DataFrame(rows); comp.to_csv(TAB / "competition_modeles.csv", index=False) | |
| 187 | +pd.DataFrame({n.split(" ")[0]: est_of(s_) for n, s_ in MODELS.items()}).assign(idx=T["idx"].to_numpy()).to_parquet(OUT / "models_est.parquet", index=False) | |
| 188 | +res["competition"] = comp[comp.periode == "test"].set_index("modele")[["mdape", "mape", "biais", "p5", "p10", "p20", "couverture"]].round(2).to_dict("index") | |
| 189 | +res["competition_val"] = comp[comp.periode == "val"].set_index("modele")[["mdape", "p10", "couverture"]].round(2).to_dict("index") | |
| 190 | +log("compétition (test) :\n" + comp[comp.periode == "test"][["modele", "mdape", "mape", "biais", "p10", "couverture"]].round(2).to_string(index=False)) | |
| 191 | +fig, ax = plt.subplots(figsize=(7.5, 5)) | |
| 192 | +g = comp[comp.periode == "test"].sort_values("mdape", ascending=False) | |
| 193 | +cols = [PAL[0] if ("M1" in n and "M1 " not in n) or n.startswith(("M10", "M11", "M12")) else PAL[2] if n.startswith(("M0", "M9")) else PAL[1] for n in g.modele] | |
| 194 | +cols = [PAL[0] if n.startswith(("M10", "M11", "M12")) else (PAL[2] if n.startswith(("M0", "M9")) else PAL[1]) for n in g.modele] | |
| 195 | +ax.barh(g.modele, g.mdape, color=cols, height=0.65) | |
| 196 | +for i, v in enumerate(g.mdape): ax.text(v + 0.15, i, f"{v:.1f} %", va="center", fontsize=8) | |
| 197 | +ax.set_xlabel("MdAPE, test 2025-26 (%)"); ax.set_title("Compétition de modèles (orange : comparables ; vert : optimisés ; bleu : références externes)") | |
| 198 | +savefig(fig, "f4_competition") | |
| 199 | + | |
| 200 | +# ---------------------------------------------------------------- bootstrap par grappes (municipalité × trimestre) des différences appariées | |
| 201 | +rng = np.random.default_rng(7) | |
| 202 | +def paired(eA, eB, mask, B=400): | |
| 203 | + a = np.abs(eA[mask]); b = np.abs(eB[mask]); cl = T["cluster"].to_numpy()[mask] | |
| 204 | + okp = ~np.isnan(a) & ~np.isnan(b); a, b, cl = a[okp], b[okp], cl[okp] | |
| 205 | + d = a - b | |
| 206 | + cl_ids, inv = np.unique(cl, return_inverse=True); nc = len(cl_ids) | |
| 207 | + sums = np.bincount(inv, d, nc); cnts = np.bincount(inv, minlength=nc) | |
| 208 | + boots_mean = []; boots_med = [] | |
| 209 | + order = np.argsort(inv); d_sorted = d[order]; starts = np.searchsorted(inv[order], np.arange(nc)); ends = np.append(starts[1:], len(d)) | |
| 210 | + for _ in range(B): | |
| 211 | + pick = rng.integers(0, nc, nc) | |
| 212 | + boots_mean.append(sums[pick].sum() / cnts[pick].sum()) | |
| 213 | + if _ < 150: | |
| 214 | + idxs = np.concatenate([np.arange(starts[p], ends[p]) for p in pick]) | |
| 215 | + boots_med.append(np.median(a[order][idxs]) - np.median(b[order][idxs])) | |
| 216 | + return dict(n=int(len(d)), diff_mape=float(d.mean() * 100), ci_mape=[float(np.percentile(boots_mean, 2.5) * 100), float(np.percentile(boots_mean, 97.5) * 100)], | |
| 217 | + diff_mdape=float((np.median(a) - np.median(b)) * 100), ci_mdape=[float(np.percentile(boots_med, 2.5) * 100), float(np.percentile(boots_med, 97.5) * 100)], | |
| 218 | + part_A_meilleur=float((a < b).mean() * 100)) | |
| 219 | +PAIRS = [ | |
| 220 | + ("M7 grille d'ajustement hédonique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 221 | + ("M7 grille d'ajustement hédonique", "M6 grille d'ajustement conventionnelle"), | |
| 222 | + ("M8a grille locale (régression sur le voisinage)", "M7 grille d'ajustement hédonique"), | |
| 223 | + ("M8b ajustements non linéaires (GBM)", "M7 grille d'ajustement hédonique"), | |
| 224 | + ("M11b gradient boosting (avec rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 225 | + ("M11a gradient boosting (sans rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 226 | + ("M12 hybride comparables + GBM (poids appris en 2024)", "M11b gradient boosting (avec rôle)"), | |
| 227 | + ("M13 ratio au rôle (5 plus proches, même rôle)", "M7 grille d'ajustement hédonique"), | |
| 228 | + ("M3 sélection par similarité physique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 229 | + ("M9 réconciliation optimisée (poids appris en 2024)", "M7 grille d'ajustement hédonique"), | |
| 230 | + ("M1 médiane des 3 plus proches (temps seul)", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 231 | +] | |
| 232 | +res["paires"] = {f"{a} vs {b}": paired(ERR[a], ERR[b], TEST) for a, b in PAIRS} | |
| 233 | +pd.DataFrame([dict(A=a, B=b, **res["paires"][f"{a} vs {b}"]) for a, b in PAIRS]).to_csv(TAB / "paires_bootstrap.csv", index=False) | |
| 234 | + | |
| 235 | +# ---------------------------------------------------------------- strates | |
| 236 | +def strata(err, by): | |
| 237 | + out = [] | |
| 238 | + for lev, g in T[TEST].groupby(by, observed=True): | |
| 239 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 240 | + r = metrics(err[:, None], m).iloc[0].to_dict(); r[by] = str(lev); out.append(r) | |
| 241 | + return pd.DataFrame(out) | |
| 242 | +KEY = ["M2 moyenne des 5 plus proches (temps seul)", "M7 grille d'ajustement hédonique", "M8a grille locale (régression sur le voisinage)", | |
| 243 | + "M13 ratio au rôle (5 plus proches, même rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)", "M10 hédonique globale directe", "M11b gradient boosting (avec rôle)", "M12 hybride comparables + GBM (poids appris en 2024)"] | |
| 244 | +for by in ["region_nom", "dens", "prix_q", "trimestre", "age_cl", "lien"]: | |
| 245 | + tabs = [] | |
| 246 | + for name in KEY: | |
| 247 | + s = strata(ERR[name], by); s["modele"] = name.split(" ")[0]; tabs.append(s) | |
| 248 | + st = pd.concat(tabs); st.to_csv(TAB / f"strates_{by}.csv", index=False) | |
| 249 | + res[f"strates_{by}"] = st.pivot(index=by, columns="modele", values="mdape").round(2).to_dict("index") | |
| 250 | + res[f"strates_{by}_n"] = st[st.modele == "M7"].set_index(by)["n"].to_dict() | |
| 251 | +# figure : densité | |
| 252 | +fig, ax = plt.subplots(figsize=(6.5, 3.4)) | |
| 253 | +st = pd.read_csv(TAB / "strates_dens.csv") | |
| 254 | +order = ["< 5", "5-19", "20-49", "50-199", "≥ 200"] | |
| 255 | +for j, (name, lab_) in enumerate([("M2", "M2 5 plus proches"), ("M7", "M7 grille hédonique"), ("M0", "M0 meilleure config."), ("M11b", "M11b GBM"), ("M12", "M12 hybride")]): | |
| 256 | + g = st[st.modele == name].set_index("dens").loc[order] | |
| 257 | + ax.plot(order, g.mdape, color=PAL[j], lw=2, marker="o", ms=4, label=lab_) | |
| 258 | +ax.set_xlabel("ventes disponibles dans un rayon de 2 km au cours des 12 mois précédents"); ax.set_ylabel("MdAPE, test (%)"); ax.legend(fontsize=8); ax.set_title("Précision selon la liquidité du sous-marché") | |
| 259 | +savefig(fig, "f5_densite") | |
| 260 | + | |
| 261 | +# ---------------------------------------------------------------- hypothèses | |
| 262 | +H = {} | |
| 263 | +# H1 : proche mais dissemblable vs loin mais semblable (k = 5, T = 12, sans ajustement) | |
| 264 | +H["H1"] = {"geo_d1": surf.loc[e1col("geo", 5, 1, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 265 | + "sim_d10": surf.loc[e1col("sim", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 266 | + "geo_d10": surf.loc[e1col("geo", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 267 | + "mix_d10": surf.loc[e1col("mix", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 268 | + "sim_d10_hedo": surf.loc[e1col("sim", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 269 | + "geo_d10_hedo": surf.loc[e1col("geo", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 270 | + "mix_d10_hedo": surf.loc[e1col("mix", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict()} | |
| 271 | +# H2 : k optimal selon la densité (geo, d = 10, T = 12, hedo et none) | |
| 272 | +h2 = {} | |
| 273 | +for a in ["none", "hedo"]: | |
| 274 | + for lev in order: | |
| 275 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 276 | + cols_ = [e1col("geo", k, 10, 12, a) for k in [1, 3, 5, 7, 10, 15, 20, 30, 50]] | |
| 277 | + md = np.nanmedian(np.abs(err1[m][:, cols_]), axis=0) * 100 | |
| 278 | + h2[f"{a}|{lev}"] = dict(zip([1, 3, 5, 7, 10, 15, 20, 30, 50], md.round(2))) | |
| 279 | +H["H2"] = h2 | |
| 280 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.3), sharey=True) | |
| 281 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 282 | + for j, lev in enumerate(order): | |
| 283 | + v = h2[f"{a}|{lev}"]; ax.plot(list(v), list(v.values()), color=PAL[j], lw=2, marker="o", ms=3.5, label=f"densité {lev}") | |
| 284 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]); ax.set_xlabel("k"); ax.set_title({"none": "temps seul", "hedo": "grille hédonique"}[a] + " — d ≤ 10 km, 12 mois") | |
| 285 | +axes[0].set_ylabel("MdAPE, test (%)"); axes[1].legend(fontsize=8) | |
| 286 | +savefig(fig, "f6_k_par_densite") | |
| 287 | +# H3 : local vs hédonique fixe selon la densité, aux trois ancres | |
| 288 | +h3 = {} | |
| 289 | +for (k, d, Tm) in [(3, 2, 6), (5, 5, 12), (10, 10, 24)]: | |
| 290 | + for lev in order: | |
| 291 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 292 | + h3[f"{k}|{d}|{Tm}|{lev}"] = {a: float(np.nanmedian(np.abs(err2[m][:, e2col("geo", k, d, Tm, a, "equal", "median")])) * 100) for a in ["conv", "hedo", "hedo_reg", "local", "gbm"]} | |
| 293 | +H["H3"] = h3 | |
| 294 | +# H4 : pondération selon la densité (ancre 5/5/12, hedo) | |
| 295 | +h4 = {} | |
| 296 | +for lev in order: | |
| 297 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 298 | + h4[lev] = {f"{w}|{r}": float(np.nanmedian(np.abs(err2[m][:, e2col("geo", 5, 5, 12, "hedo", w, r)])) * 100) for (w, r) in [("equal", "median"), ("equal", "mean"), ("dist", "mean"), ("rec", "mean"), ("sim", "mean"), ("all", "mean"), ("all", "median")]} | |
| 299 | +H["H4"] = h4 | |
| 300 | +# H5 : méthodes simples vs ML selon l'homogénéité (dispersion des prix ajustés des 10 voisins à 10 km / 24 mois) | |
| 301 | +het = sd2[:, e2col("geo", 10, 10, 24, "hedo", "equal", "median")].astype(float) | |
| 302 | +T["heterog"] = pd.qcut(pd.Series(np.where(np.isnan(het), np.nanmax(het), het)), 3, labels=["homogène", "moyen", "hétérogène"]) | |
| 303 | +h5 = {} | |
| 304 | +for lev in ["homogène", "moyen", "hétérogène"]: | |
| 305 | + m = TEST & (T["heterog"] == lev).to_numpy() | |
| 306 | + h5[lev] = {n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY} | |
| 307 | + h5[lev]["sd_median"] = float(np.nanmedian(het[m])) | |
| 308 | +H["H5"] = h5 | |
| 309 | +res["hypotheses"] = H | |
| 310 | + | |
| 311 | +# ---------------------------------------------------------------- intervalles d'incertitude (couverture) | |
| 312 | +intervals = {} | |
| 313 | +for a in ["none", "hedo", "gbm"]: | |
| 314 | + for (k, d, Tm) in [(5, 5, 12), (10, 10, 24)]: | |
| 315 | + c = e2col("geo", k, d, Tm, a, "equal", "median"); e = est2[:, c].astype(float); s = sd2[:, c].astype(float) | |
| 316 | + okv = VAL & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 317 | + z_needed = np.abs(lnP[okv] - e[okv]) / s[okv]; z80 = float(np.percentile(z_needed, 80)) | |
| 318 | + okt = TEST & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 319 | + cov_nominal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= 1.2816 * s[okt]) * 100) | |
| 320 | + cov_cal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= z80 * s[okt]) * 100) | |
| 321 | + width = float(np.median(2 * z80 * s[okt]) * 100) | |
| 322 | + intervals[f"{a}|{k}|{d}|{Tm}"] = dict(z80_val=z80, couverture_nominale_80=cov_nominal, couverture_calibree=cov_cal, largeur_mediane_pct=width, n_test=int(okt.sum())) | |
| 323 | +res["intervalles"] = intervals | |
| 324 | + | |
| 325 | +# ---------------------------------------------------------------- règles conditionnelles (système de recommandation) | |
| 326 | +cond = {}; est_cond = np.full(len(T), np.nan) | |
| 327 | +for lev in order: | |
| 328 | + mv_ = VAL & (T["dens"] == lev).to_numpy(); mt_ = TEST & (T["dens"] == lev).to_numpy() | |
| 329 | + m1 = metrics(err1, mv_); m2 = metrics(err2, mv_) | |
| 330 | + allm = pd.concat([pd.concat([E1, m1], axis=1).assign(src="E1"), pd.concat([E2, m2], axis=1).assign(src="E2")], ignore_index=True) | |
| 331 | + b = allm[allm.couverture >= 95].sort_values("mdape").iloc[0] | |
| 332 | + e = (err1 if b.src == "E1" else err2)[:, int(b.cfg)] | |
| 333 | + est_cond[mt_] = e[mt_] | |
| 334 | + cond[lev] = dict(b[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape", "couverture"]].to_dict(), mdape_test=float(np.nanmedian(np.abs(e[mt_])) * 100), n_test=int(mt_.sum())) | |
| 335 | +res["regles_conditionnelles"] = cond | |
| 336 | +res["regles_conditionnelles_global_test"] = dict(mdape=float(np.nanmedian(np.abs(est_cond[TEST])) * 100), p10=float(np.nanmean(np.abs(est_cond[TEST]) <= 0.1) * 100), couverture=float(np.mean(~np.isnan(est_cond[TEST])) * 100)) | |
| 337 | +pd.DataFrame(cond).T.to_csv(TAB / "regles_conditionnelles.csv") | |
| 338 | + | |
| 339 | +# ---------------------------------------------------------------- stabilité temporelle, indice, divers | |
| 340 | +tq = [] | |
| 341 | +for q, g in T[TEST].groupby("trimestre"): | |
| 342 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 343 | + tq.append(dict(trimestre=q, n=int(m.sum()), **{n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY}, | |
| 344 | + **{"biais_" + n.split(" ")[0]: float(np.nanmedian(ERR[n][m]) * 100) for n in KEY})) | |
| 345 | +pd.DataFrame(tq).to_csv(TAB / "stabilite_trimestres.csv", index=False); res["stabilite"] = tq | |
| 346 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 347 | +idxf = pd.read_csv(OUT / "index_full.csv") | |
| 348 | +for j, (r, lab_) in enumerate([("QC", "Québec"), ("06", "Montréal"), ("03", "Capitale-Nationale"), ("07", "Outaouais"), ("05", "Estrie")]): | |
| 349 | + g = idxf[idxf.region.astype(str).str.zfill(2) == r] if r != "QC" else idxf[idxf.region == "QC"] | |
| 350 | + if len(g): ax.plot(pd.to_datetime(g.ym), np.exp(g.log_idx) * 100, color=PAL[j], lw=2 if r == "QC" else 1.4, label=lab_) | |
| 351 | +ax.set_ylabel("indice hédonique (2021-01 = 100)"); ax.legend(fontsize=8); ax.set_title("Indice de prix mensuel (plein échantillon, descriptif)") | |
| 352 | +savefig(fig, "f7_indice") | |
| 353 | +res["dataset"] = load_json("dataset_stats.json") | |
| 354 | +res["n_targets"] = dict(val=int(VAL.sum()), test=int(TEST.sum())) | |
| 355 | +res["densite_distribution"] = T[TEST]["dens"].value_counts().sort_index().to_dict() | |
| 356 | +res["n_cand_median"] = dict(val=float(np.median(T[VAL].n_cand)), test=float(np.median(T[TEST].n_cand))) | |
| 357 | +save_json(res, "results.json") | |
| 358 | +log("analyse terminée") | |
added
M4M64aomparables-validation/pipeline/06_report.py
+313 −0
@@ -0,0 +1,313 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""06 — Rapport : assemble docs/RAPPORT.md à partir de results.json et des tables (texte + tableaux + figures). | |
| 3 | +Les phrases d'interprétation sont calculées à partir des résultats (jamais de chiffre écrit à la main).""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +R = load_json("results.json"); TAB = OUT / "tables" | |
| 7 | +DOCS = ROOT / "docs"; DOCS.mkdir(exist_ok=True) | |
| 8 | +D = R["dataset"]; comp = R["competition"]; compv = R["competition_val"] | |
| 9 | +def f1(x): return f"{x:.1f}".replace(".", ",") | |
| 10 | +def f2(x): return f"{x:.2f}".replace(".", ",") | |
| 11 | +def n_(x): return f"{int(round(x)):,}".replace(",", " ") | |
| 12 | +def pct(x): return f"{f1(x)} %" | |
| 13 | +def md_table(df, cols=None, fmt=None, index=False): | |
| 14 | + if cols: df = df[cols] | |
| 15 | + if index: df = df.reset_index() | |
| 16 | + fmt = fmt or {} | |
| 17 | + head = "| " + " | ".join(str(c) for c in df.columns) + " |\n|" + "|".join("---" for _ in df.columns) + "|\n" | |
| 18 | + rows = [] | |
| 19 | + for _, r in df.iterrows(): | |
| 20 | + cells = [] | |
| 21 | + for c in df.columns: | |
| 22 | + v = r[c] | |
| 23 | + if c in fmt: cells.append(fmt[c](v)) | |
| 24 | + elif isinstance(v, float): cells.append(f1(v)) | |
| 25 | + else: cells.append(str(v)) | |
| 26 | + rows.append("| " + " | ".join(cells) + " |") | |
| 27 | + return head + "\n".join(rows) + "\n" | |
| 28 | +short = lambda name: name.split(" ")[0] | |
| 29 | +LAB = {"raw": "aucun ajustement", "none": "temps seulement", "conv": "grille conventionnelle", "hedo": "grille hédonique", "hedo_reg": "grille hédonique régionale", | |
| 30 | + "local": "régression locale", "gbm": "ajustements GBM", "role": "ratio au rôle", "geo": "les plus proches (distance)", "sim": "les plus semblables (physique)", | |
| 31 | + "mix": "compromis distance / similarité", "adj": "les moins ajustés", "equal": "égale", "dist": "1/distance", "rec": "récence", "all": "distance × récence × similarité", | |
| 32 | + "median": "médiane", "mean": "moyenne"} | |
| 33 | + | |
| 34 | +# ------------------------------------------------------------------ tableaux | |
| 35 | +excl = pd.DataFrame(D["exclusions"], columns=["Règle", "n"]); excl["n"] = excl["n"].map(n_) | |
| 36 | +cm = pd.DataFrame(comp).T; cm.index.name = "Modèle"; cm = cm.reset_index() | |
| 37 | +cm["val"] = cm["Modèle"].map(lambda m: compv[m]["mdape"]) | |
| 38 | +cm = cm.sort_values("mdape") | |
| 39 | +t_comp = md_table(cm.rename(columns={"mdape": "MdAPE test", "val": "MdAPE val.", "mape": "MAPE", "biais": "Biais (médian)", "p5": "±5 %", "p10": "±10 %", "p20": "±20 %", "couverture": "Couverture"}), | |
| 40 | + ["Modèle", "MdAPE val.", "MdAPE test", "MAPE", "Biais (médian)", "±5 %", "±10 %", "±20 %", "Couverture"]) | |
| 41 | +surf = pd.read_csv(TAB / "surface_E1.csv") | |
| 42 | +def surf_tab(sel, adj, T_): | |
| 43 | + g = surf[(surf.sel == sel) & (surf.adj == adj) & (surf["T"] == T_)].pivot(index="d", columns="k", values="mdape_test") | |
| 44 | + g.index = [f"≤ {d} km" for d in g.index]; g.columns = [f"k = {k}" for k in g.columns]; g.index.name = "rayon" | |
| 45 | + return md_table(g.round(1), index=True) | |
| 46 | +kb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.d == 10) & (surf["T"] == 12)].sort_values("k") | |
| 47 | +fac = pd.read_csv(TAB / "facteurs_E2.csv") | |
| 48 | +anchor = fac[(fac.k == 5) & (fac.d == 5) & (fac["T"] == 12)] | |
| 49 | +t_adj = anchor[(anchor.sel == "geo") & (anchor.w == "equal") & (anchor.recon == "median")].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 50 | +t_adj = t_adj.reset_index(); t_adj["Ajustement"] = t_adj["adj"].map(LAB) | |
| 51 | +t_adj_md = md_table(t_adj.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "mape_test": "MAPE test", "biais_test": "Biais", "p10_test": "±10 %", "couverture_test": "Couverture"}), | |
| 52 | + ["Ajustement", "MdAPE val.", "MdAPE test", "MAPE test", "Biais", "±10 %", "Couverture"]) | |
| 53 | +t_sel = anchor[(anchor.adj == "hedo") & (anchor.w == "equal") & (anchor.recon == "median")].copy(); t_sel["Sélection"] = t_sel["sel"].map(LAB) | |
| 54 | +t_sel_md = md_table(t_sel.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Sélection", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 55 | +t_wr = anchor[(anchor.adj == "hedo") & (anchor.sel == "geo")].copy(); t_wr["Pondération"] = t_wr["w"].map(LAB); t_wr["Réconciliation"] = t_wr["recon"].map(LAB) | |
| 56 | +t_wr_md = md_table(t_wr.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Pondération", "Réconciliation", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 57 | +pairs = pd.read_csv(TAB / "paires_bootstrap.csv") | |
| 58 | +pairs["Comparaison"] = pairs.apply(lambda r: f"{short(r.A)} − {short(r.B)}", axis=1) | |
| 59 | +pairs["Δ MdAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mdape)} [{f2(eval(r.ci_mdape)[0])} ; {f2(eval(r.ci_mdape)[1])}]", axis=1) | |
| 60 | +pairs["Δ MAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mape)} [{f2(eval(r.ci_mape)[0])} ; {f2(eval(r.ci_mape)[1])}]", axis=1) | |
| 61 | +pairs["A plus précis (% des cibles)"] = pairs["part_A_meilleur"].map(f1) | |
| 62 | +t_pairs = md_table(pairs, ["Comparaison", "Δ MdAPE (pts) [IC 95 %]", "Δ MAPE (pts) [IC 95 %]", "A plus précis (% des cibles)", "n"], fmt={"n": n_}) | |
| 63 | +def strata_md(by, label): | |
| 64 | + st = pd.read_csv(TAB / f"strates_{by}.csv") | |
| 65 | + g = st.pivot(index=by, columns="modele", values="mdape").round(1) | |
| 66 | + g["n"] = st[st.modele == "M7"].set_index(by)["n"] | |
| 67 | + g = g[["n", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]] | |
| 68 | + if by == "dens": g = g.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 69 | + if by == "region_nom": g = g.sort_values("n", ascending=False) | |
| 70 | + g.index.name = label | |
| 71 | + return md_table(g, index=True, fmt={"n": n_}) | |
| 72 | +H = R["hypotheses"] | |
| 73 | +h2 = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("hedo")}).T; h2.index.name = "densité"; h2.columns = [f"k = {c}" for c in h2.columns] | |
| 74 | +h2_md = md_table(h2.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 75 | +h2n = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("none")}).T; h2n.index.name = "densité"; h2n.columns = [f"k = {c}" for c in h2n.columns] | |
| 76 | +h2n_md = md_table(h2n.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 77 | +h3 = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("5|5|12")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 78 | +h3.columns = [LAB[c] for c in h3.columns]; h3.index.name = "densité"; h3_md = md_table(h3, index=True) | |
| 79 | +h3s = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("3|2|6")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 80 | +h3s.columns = [LAB[c] for c in h3s.columns]; h3s.index.name = "densité"; h3s_md = md_table(h3s, index=True) | |
| 81 | +h4 = pd.DataFrame(H["H4"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]]; h4.columns = [f"{LAB[c.split('|')[0]]} / {LAB[c.split('|')[1]]}" for c in h4.columns]; h4.index.name = "densité"; h4_md = md_table(h4, index=True) | |
| 82 | +h5 = pd.DataFrame(H["H5"]).T; h5.index.name = "sous-marché"; h5 = h5.rename(columns={"sd_median": "σ médian (log)"}); h5_md = md_table(h5[["σ médian (log)", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]], index=True, fmt={"σ médian (log)": lambda v: f"{v:.3f}".replace(".", ",")}) | |
| 83 | +iv = pd.DataFrame(R["intervalles"]).T; iv.index = [f"{LAB[i.split('|')[0]]}, k = {i.split('|')[1]}, ≤ {i.split('|')[2]} km, {i.split('|')[3]} mois" for i in iv.index]; iv.index.name = "configuration" | |
| 84 | +iv_md = md_table(iv.rename(columns={"z80_val": "z calibré (val.)", "couverture_nominale_80": "couverture à z = 1,28", "couverture_calibree": "couverture calibrée", "largeur_mediane_pct": "largeur médiane (%)", "n_test": "n"}), index=True, fmt={"n": n_, "z calibré (val.)": f2}) | |
| 85 | +rc = pd.DataFrame(R["regles_conditionnelles"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 86 | +rc["règle"] = rc.apply(lambda r: f"{LAB[r.sel]}, k = {int(r.k)}, ≤ {r.d} km, {int(r['T'])} mois, {LAB[r.adj]}, pond. {LAB[r.w]}, {LAB[r.recon]}", axis=1) | |
| 87 | +rc.index.name = "densité"; rc_md = md_table(rc.rename(columns={"mdape": "MdAPE val.", "mdape_test": "MdAPE test", "couverture": "couverture val.", "n_test": "n test"}), ["règle", "MdAPE val.", "MdAPE test", "couverture val.", "n test"], index=True, fmt={"n test": n_}) | |
| 88 | +stab = pd.DataFrame(R["stabilite"]); stab_md = md_table(stab[["trimestre", "n", "M2", "M7", "M0", "M11b", "M12", "biais_M7", "biais_M11b"]].rename(columns={"biais_M7": "biais M7", "biais_M11b": "biais M11b"}), fmt={"n": n_}) | |
| 89 | + | |
| 90 | +# ------------------------------------------------------------------ chiffres clés pour le texte | |
| 91 | +best = min(comp, key=lambda m: comp[m]["mdape"]); best_comps = min([m for m in comp if short(m) in ("M1", "M2", "M3", "M4", "M5", "M6", "M7", "M8a", "M8b", "M13", "M0", "M9")], key=lambda m: comp[m]["mdape"]) | |
| 92 | +m0 = R["M0"]; m0u = R["M0_sans_contrainte"] | |
| 93 | +gain_adj = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M7 grille d'ajustement hédonique"]["mdape"] | |
| 94 | +gain_sel = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M3 sélection par similarité physique"]["mdape"] | |
| 95 | +k_curve = {int(r.k): r.mdape_test for r in kb.itertuples()} | |
| 96 | +k_best = min(k_curve, key=k_curve.get) | |
| 97 | +e1b = R["E1_best_val"] | |
| 98 | +pair = R["paires"] | |
| 99 | +def pr(a, b): return pair[f"{a} vs {b}"] | |
| 100 | +M7 = "M7 grille d'ajustement hédonique"; M2 = "M2 moyenne des 5 plus proches (temps seul)"; M6 = "M6 grille d'ajustement conventionnelle" | |
| 101 | +M8a = "M8a grille locale (régression sur le voisinage)"; M8b = "M8b ajustements non linéaires (GBM)"; M11b = "M11b gradient boosting (avec rôle)"; M11a = "M11a gradient boosting (sans rôle)" | |
| 102 | +M0 = "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"; M12 = "M12 hybride comparables + GBM (poids appris en 2024)"; M13 = "M13 ratio au rôle (5 plus proches, même rôle)"; M9 = "M9 réconciliation optimisée (poids appris en 2024)"; M3 = "M3 sélection par similarité physique"; M1 = "M1 médiane des 3 plus proches (temps seul)"; M10 = "M10 hédonique globale directe" | |
| 103 | +dens = R["strates_dens"]; densn = R["strates_dens_n"] | |
| 104 | +h1 = H["H1"] | |
| 105 | +marg = R["E2_marginal"] | |
| 106 | +cond_g = R["regles_conditionnelles_global_test"] | |
| 107 | +nT = R["n_targets"] | |
| 108 | + | |
| 109 | +txt = f"""--- | |
| 110 | +title: "La méthode des comparables à l'épreuve des données : validation empirique à grande échelle sur {n_(D['propre'])} ventes résidentielles québécoises" | |
| 111 | +subtitle: "Sélection, nombre, ajustements, pondération et réconciliation des ventes comparables — rétro-évaluation point-dans-le-temps sur les transactions Vrai Prix 2021-2026" | |
| 112 | +author: "Simon-Pierre Boucher — version {VERSION}" | |
| 113 | +date: "{time.strftime('%d/%m/%Y')}" | |
| 114 | +lang: fr-CA | |
| 115 | +--- | |
| 116 | + | |
| 117 | +# Résumé | |
| 118 | + | |
| 119 | +**Question.** La méthode de comparaison des ventes, telle qu'enseignée et pratiquée en évaluation immobilière, produit-elle des estimations fiables, et lesquelles de ses règles (choix des comparables, nombre, rayon, ancienneté, ajustements, pondération, réconciliation) sont empiriquement justifiées ? | |
| 120 | + | |
| 121 | +**Données et protocole.** {n_(D['propre'])} ventes de résidences unifamiliales du Québec (registre foncier jumelé au rôle d'évaluation, Vrai Prix, {D['dates'][0]} → {D['dates'][1]}), après un nettoyage journalisé de {n_(D['brut'])} transactions brutes. Pour chacune des **{n_(nT['val'] + nT['test'])} ventes cibles** de 2024 (validation, {n_(nT['val'])}) et de 2025-2026 (test hors échantillon, {n_(nT['test'])}), le prix est masqué et la valeur est estimée avec les seules ventes conclues **avant** la date de la cible, un indice de prix ré-estimé chaque mois avec les seules ventes antérieures, et des grilles d'ajustement et modèles ré-estimés chaque trimestre. **2 256 configurations** de la méthode des comparables et treize modèles de référence sont appliqués aux mêmes cibles ; les différences sont testées par bootstrap par grappes (municipalité × trimestre). | |
| 122 | + | |
| 123 | +**Résultats principaux (test 2025-2026).** | |
| 124 | + | |
| 125 | +1. **Les ajustements sont le facteur dominant.** Prendre la moyenne des cinq ventes les plus proches, ajustées seulement pour le temps (M2), donne une erreur absolue médiane (MdAPE) de **{pct(comp[M2]['mdape'])}** ; une grille d'ajustement hédonique (M7) la ramène à **{pct(comp[M7]['mdape'])}** (−{f1(gain_adj)} points, IC 95 % de la différence de MdAPE [{f2(pr(M7, M2)['ci_mdape'][0])} ; {f2(pr(M7, M2)['ci_mdape'][1])}]). Une grille conventionnelle stylisée (M6) fait presque aussi bien ({pct(comp[M6]['mdape'])}) : ce qui compte, c'est d'ajuster, plus que la finesse de la grille. | |
| 126 | +2. **Choisir des comparables semblables vaut autant qu'ajuster.** Sans aucun ajustement physique, sélectionner les cinq ventes les plus **semblables** (aire, terrain, âge, lien physique, étages) plutôt que les cinq plus **proches** fait passer la MdAPE de {pct(comp[M2]['mdape'])} à **{pct(comp[M3]['mdape'])}** (H1 confirmée : un comparable un peu plus éloigné mais semblable vaut mieux qu'un voisin dissemblable). | |
| 127 | +3. **Le nombre optimal de comparables est bien supérieur à trois.** Avec la grille hédonique, la MdAPE décroît de {pct(k_curve[1])} (k = 1) à {pct(k_curve[3])} (k = 3), {pct(k_curve[5])} (k = 5), {pct(k_curve[10])} (k = 10) et atteint son minimum à **k = {k_best}** ({pct(k_curve[k_best])}) avant de remonter ({pct(k_curve[50])} à k = 50) ; l'optimum dépend de la densité du marché (H2 confirmée). | |
| 128 | +4. **Le rayon et la fenêtre temporelle doivent être larges.** La meilleure configuration de la surface Erreur(k, d, t) choisie en 2024 est « {LAB[e1b['sel']]}, k = {int(e1b['k'])}, ≤ {e1b['d']} km, {int(e1b['T'])} mois, {LAB[e1b['adj']]} » ({pct(e1b['mdape_val'])} en validation → **{pct(e1b['mdape_test'])}** en test). Les rayons de 0,5-1 km et les fenêtres de 3 mois, privilégiés par la pratique, sont dominés dès que les comparables sont ajustés. | |
| 129 | +5. **La pondération et le mode de réconciliation comptent peu** (effets marginaux moyens de {f1(min(marg['w'].values()))} à {f1(max(marg['w'].values()))} points selon la pondération, médiane {f1(marg['recon']['median'])} contre moyenne {f1(marg['recon']['mean'])}), bien moins que la sélection ({f1(min(marg['sel'].values()))}-{f1(max(marg['sel'].values()))}) et l'ajustement ({f1(min(marg['adj'].values()))}-{f1(max(marg['adj'].values()))}). | |
| 130 | +6. **La meilleure méthode des comparables, entièrement transparente et choisie en 2024, atteint {pct(comp[M0]['mdape'])} en 2025-2026** (M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]}), contre **{pct(comp[M11b]['mdape'])}** pour un gradient boosting ré-entraîné chaque trimestre avec la valeur au rôle (M11b) et {pct(comp[M11a]['mdape'])} sans le rôle (M11a). L'hybride comparables + GBM (M12) atteint **{pct(comp[M12]['mdape'])}**. Le coût de la transparence est donc de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''} de MdAPE face au meilleur modèle opaque, et il s'annule dans les sous-marchés homogènes (H5). | |
| 131 | +7. **Le ratio au rôle est un comparable redoutable** : rôle de la cible × ratio prix/rôle médian des cinq plus proches ventes du même rôle (M13) donne {pct(comp[M13]['mdape'])}, mieux que toute grille d'ajustement physique, mais seulement pour {pct(comp[M13]['couverture'])} des cibles. | |
| 132 | +8. **La fiabilité dépend d'abord de la liquidité du sous-marché** : la MdAPE de M7 passe de {pct(dens['≥ 200']['M7'])} là où plus de 200 ventes ont eu lieu à moins de 2 km dans l'année à {pct(dens['< 5']['M7'])} là où il y en a eu moins de 5 ; des règles conditionnelles à la densité (système de recommandation) obtiennent {pct(cond_g['mdape'])} globalement avec {pct(cond_g['couverture'])} de couverture. | |
| 133 | + | |
| 134 | +**Portée.** Nous validons la précision prédictive de règles professionnelles *simulées* sur des caractéristiques observables (aire, terrain, âge, étages, lien physique, rôle) ; les ajustements non observables (garage, piscine, chambres, qualité, rénovations) restent dans l'erreur résiduelle, et le jugement réel d'un évaluateur n'est pas reproduit. | |
| 135 | + | |
| 136 | +# 1. Données | |
| 137 | + | |
| 138 | +## 1.1 Source et nettoyage | |
| 139 | + | |
| 140 | +Instantané de la base de production Vrai Prix pris le 2026-10-10 : {n_(766596)} transactions résidentielles et 3,75 M d'unités d'évaluation. Nous retenons les ventes de type « unifamilial » jumelées à une unité du rôle, et appliquons des règles de nettoyage explicites et journalisées — rien n'est éliminé silencieusement. | |
| 141 | + | |
| 142 | +{md_table(excl)} | |
| 143 | +Jeu propre : **{n_(D['propre'])} ventes** ({D['dates'][0]} → {D['dates'][1]}) ; prix médian {n_(D['prix']['50%'])} $, moyenne {n_(D['prix']['mean'])} $. | |
| 144 | + | |
| 145 | +Deux règles méritent un commentaire. R8 écarte les ratios prix/rôle extrêmes (< 0,33 ou > 3) : faute d'indicateur de lien de dépendance entre les parties, c'est notre proxy de « transaction non représentative » — un évaluateur les écarterait aussi. R6b écarte les ventes conclues l'année même de la construction : leur prix médian ({n_(346108)} $) et leur ratio au rôle (1,15) sont ceux de terrains ou de constructions inachevées, non de maisons achevées (450-480 k$ et 1,3 aux âges 1-4 ans). | |
| 146 | + | |
| 147 | +## 1.2 Répartition | |
| 148 | + | |
| 149 | +| Période | Rôle | Ventes | | |
| 150 | +|---|---|---| | |
| 151 | +| 2021-01 → 2023-12 | développement | {n_(D['par_periode']['train'])} | | |
| 152 | +| 2024 | validation (choix des configurations) | {n_(D['par_periode']['val'])} | | |
| 153 | +| 2025-01 → 2026-09 | test hors échantillon | {n_(D['par_periode']['test'])} | | |
| 154 | + | |
| 155 | +Régions les plus représentées : {', '.join(f"{k} ({n_(v)})" for k, v in list(D['par_region'].items())[:6])}. | |
| 156 | + | |
| 157 | +{{ width=85% }} | |
| 158 | + | |
| 159 | +## 1.3 Caractéristiques disponibles et limites | |
| 160 | + | |
| 161 | +Observées pour chaque vente : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable, superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle en vigueur à la vente et sa part terrain. Non observées : garage, piscine, nombre de chambres et de salles de bains, qualité, état, rénovations. Les grilles d'ajustement testées ne portent donc que sur les premières ; les secondes sont une source d'erreur commune à toutes les méthodes. | |
| 162 | + | |
| 163 | +# 2. Protocole | |
| 164 | + | |
| 165 | +## 2.1 Rétro-évaluation point-dans-le-temps | |
| 166 | + | |
| 167 | +Pour chaque cible : (1) le prix est masqué ; (2) le bassin de comparables est constitué des ventes conclues strictement avant la date de la cible (la cible elle-même, si elle a déjà été vendue, est exclue : on évalue par comparables, non par vente répétée), dans un rayon de 25 km et une fenêtre de 24 mois ; (3) l'indice temporel régional utilisé pour ramener chaque comparable au niveau du dernier mois connu est ré-estimé chaque mois avec les seules ventes antérieures (régression hédonique à effets fixes de municipalité et effets mensuels, par région administrative, rétrécie vers l'indice provincial quand les ventes sont rares) ; (4) les coefficients d'ajustement hédoniques, les grilles régionales et les modèles d'apprentissage sont ré-estimés chaque trimestre avec les seules ventes antérieures au trimestre ; (5) chaque configuration produit une valeur et, le cas échéant, une dispersion des prix ajustés. | |
| 168 | + | |
| 169 | +Le niveau de prix du mois de la cible n'est jamais connu : toutes les méthodes travaillent au niveau du mois précédent, ce qui reproduit la situation réelle d'un évaluateur. | |
| 170 | + | |
| 171 | +## 2.2 Configurations testées | |
| 172 | + | |
| 173 | +**E1 — surface Erreur(k, d, t)** : k ∈ {{1, 3, 5, 7, 10, 15, 20, 30, 50}}, d ∈ {{0,5 ; 1 ; 2 ; 5 ; 10 ; 25 km}}, t ∈ {{3, 6, 12, 24 mois}}, sélection ∈ {{plus proches, plus semblables, compromis}}, ajustement ∈ {{temps seul, grille hédonique}}, médiane. 1 296 configurations. | |
| 174 | + | |
| 175 | +**E2 — plan factoriel des règles** aux ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois) : sélection (4) × ajustement (8 : aucun, temps seul, grille conventionnelle, hédonique globale, hédonique régionale, régression locale, GBM, ratio au rôle) × pondération (5) × réconciliation (2). 960 configurations. | |
| 176 | + | |
| 177 | +*Grille conventionnelle stylisée* (M6) : aire habitable ajustée à la moitié du prix au m² du comparable ; terrain à la moitié de la valeur foncière implicite au m² (plafond ±15 %) ; âge à −0,5 % par année d'écart (plafond ±20 %) ; ±5 % entre détaché et attaché. *Grille hédonique* (M7) : coefficients d'une régression du log du prix sur les caractéristiques, à effets fixes de municipalité (within), ré-estimée chaque trimestre. *Régression locale* (M8a) : mêmes coefficients ré-estimés sur les ventes à moins de 10 km des 24 derniers mois (≥ 40 ventes, ridge vers la grille régionale). *GBM* (M8b) : l'ajustement est la différence des prédictions d'un gradient boosting entre la cible et le comparable. *Ratio au rôle* (M13) : valeur au rôle de la cible × ratio prix/rôle du comparable, pour les comparables de la même municipalité et du même rôle triennal. | |
| 178 | + | |
| 179 | +## 2.3 Mesures | |
| 180 | + | |
| 181 | +Erreur absolue en pourcentage APE = |P̂ − P| / P ; MdAPE (médiane), MAPE (moyenne), biais médian signé, parts à ±5, ±10 et ±20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur — une configuration trop restrictive ne « rate » pas, elle s'abstient, et cela doit se voir). Les différences entre méthodes sont des différences d'erreurs appariées sur les mêmes cibles, avec intervalles de confiance à 95 % par bootstrap par grappes municipalité × trimestre (400 réplications). | |
| 182 | + | |
| 183 | +# 3. Résultats | |
| 184 | + | |
| 185 | +## 3.1 Combien de comparables ? À quelle distance ? De quelle ancienneté ? | |
| 186 | + | |
| 187 | +{{ width=100% }} | |
| 188 | + | |
| 189 | +{{ width=90% }} | |
| 190 | + | |
| 191 | +**Sélection par distance, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 192 | + | |
| 193 | +{surf_tab('geo', 'hedo', 12)} | |
| 194 | +**Sélection par distance, temps seulement, fenêtre 12 mois — MdAPE test (%)** | |
| 195 | + | |
| 196 | +{surf_tab('geo', 'none', 12)} | |
| 197 | +**Sélection par similarité physique, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 198 | + | |
| 199 | +{surf_tab('sim', 'hedo', 12)} | |
| 200 | +Lecture. (i) Sans ajustement, l'erreur décroît avec k et avec le rayon tant que l'on reste sous quelques kilomètres, puis se stabilise ; (ii) avec la grille hédonique, elle décroît nettement plus et l'optimum se déplace vers des k et des rayons plus grands ; (iii) la fenêtre de 3 mois est systématiquement dominée par 12-24 mois dès que l'on ajuste pour le temps. Meilleures configurations de E1 par sélection × ajustement (choisies en 2024, évaluées en 2025-2026) : | |
| 201 | + | |
| 202 | +{md_table(pd.DataFrame(R['E1_best_by_sel_adj']).T.reset_index().rename(columns={'index': 'sélection | ajustement', 'mdape_val': 'MdAPE val.', 'mdape_test': 'MdAPE test', 'couverture_test': 'couverture test'}), ['sélection | ajustement', 'k', 'd', 'T', 'MdAPE val.', 'MdAPE test', 'couverture test'], fmt={'k': lambda v: str(int(v)), 'T': lambda v: str(int(v)), 'd': lambda v: f1(v)})} | |
| 203 | +### k optimal selon la densité (H2) | |
| 204 | + | |
| 205 | +{h2_md} | |
| 206 | +*(sélection par distance, ≤ 10 km, 12 mois, grille hédonique ; MdAPE test en %, par nombre de ventes disponibles à moins de 2 km au cours des 12 mois précédents)* | |
| 207 | + | |
| 208 | +Sans ajustement : | |
| 209 | + | |
| 210 | +{h2n_md} | |
| 211 | +{{ width=100% }} | |
| 212 | + | |
| 213 | +## 3.2 Quels ajustements sont justifiés ? | |
| 214 | + | |
| 215 | +{{ width=90% }} | |
| 216 | + | |
| 217 | +{t_adj_md} | |
| 218 | +Trois enseignements. **L'ajustement temporel seul** vaut {f1(t_adj.set_index('adj').loc['raw', 'mdape_test'] - t_adj.set_index('adj').loc['none', 'mdape_test'])} point ; **n'importe quelle grille physique** vaut {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['conv', 'mdape_test'])} à {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['gbm', 'mdape_test'])} points de plus ; **les raffinements** (grille régionale, régression locale, ajustements non linéaires) apportent peu face à la grille hédonique globale, et la régression locale est même {('moins bonne' if t_adj.set_index('adj').loc['local', 'mdape_test'] > t_adj.set_index('adj').loc['hedo', 'mdape_test'] else 'meilleure')} ({pct(t_adj.set_index('adj').loc['local', 'mdape_test'])} contre {pct(t_adj.set_index('adj').loc['hedo', 'mdape_test'])}). | |
| 219 | + | |
| 220 | +### Grilles locales et petits échantillons (H3) | |
| 221 | + | |
| 222 | +MdAPE test (%) par type d'ajustement et densité, ancre (5, 5 km, 12 mois) : | |
| 223 | + | |
| 224 | +{h3_md} | |
| 225 | +Ancre (3, 2 km, 6 mois) : | |
| 226 | + | |
| 227 | +{h3s_md} | |
| 228 | +## 3.3 Comment choisir les comparables ? (H1) | |
| 229 | + | |
| 230 | +Ancre (5, 5 km, 12 mois), grille hédonique, médiane : | |
| 231 | + | |
| 232 | +{t_sel_md} | |
| 233 | +Test direct de H1 (k = 5, 12 mois, temps seulement) : les cinq voisins à moins de 1 km donnent une MdAPE de {pct(h1['geo_d1']['mdape_test'])} (couverture {pct(h1['geo_d1']['couverture_test'])}) ; les cinq ventes les plus semblables à moins de 10 km, {pct(h1['sim_d10']['mdape_test'])} (couverture {pct(h1['sim_d10']['couverture_test'])}) ; les cinq plus proches à moins de 10 km, {pct(h1['geo_d10']['mdape_test'])}. Avec la grille hédonique : plus proches {pct(h1['geo_d10_hedo']['mdape_test'])}, plus semblables {pct(h1['sim_d10_hedo']['mdape_test'])}, compromis {pct(h1['mix_d10_hedo']['mdape_test'])}. | |
| 234 | + | |
| 235 | +## 3.4 Pondération et réconciliation (H4) | |
| 236 | + | |
| 237 | +Ancre (5, 5 km, 12 mois), sélection par distance, grille hédonique : | |
| 238 | + | |
| 239 | +{t_wr_md} | |
| 240 | +Selon la densité (MdAPE test, %) : | |
| 241 | + | |
| 242 | +{h4_md} | |
| 243 | +Effets marginaux moyens (MdAPE test moyenne sur toutes les autres modalités de E2) : sélection {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['sel'].items())} ; ajustement {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['adj'].items())} ; pondération {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['w'].items())} ; réconciliation {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['recon'].items())}. | |
| 244 | + | |
| 245 | +## 3.5 Compétition de modèles | |
| 246 | + | |
| 247 | +{{ width=95% }} | |
| 248 | + | |
| 249 | +{t_comp} | |
| 250 | +M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]} (meilleure des 2 256 configurations en 2024 parmi celles qui couvrent ≥ 97 % des cibles ; sans cette contrainte, la meilleure est « {LAB[m0u['sel']]}, k = {int(m0u['k'])}, ≤ {m0u['d']} km, {int(m0u['T'])} mois, {LAB[m0u['adj']]} » à {pct(m0u['mdape_test'])} mais pour {pct(m0u['couverture_test'])} des cibles seulement). Poids appris de M9 : {', '.join(f"{LAB[k]} {f2(v)}" for k, v in R['M9_weights'].items() if v > 0)}. Poids de M12 : {', '.join(f"{k} {f2(v)}" for k, v in R['M12_weights'].items() if v > 0)}. | |
| 251 | + | |
| 252 | +### Différences appariées (test ; bootstrap par grappes municipalité × trimestre) | |
| 253 | + | |
| 254 | +{t_pairs} | |
| 255 | +Un Δ négatif signifie que A est plus précis que B ; l'intervalle qui exclut zéro signale une différence robuste aux dépendances spatiales et temporelles. | |
| 256 | + | |
| 257 | +## 3.6 Où la méthode tient-elle, où défaille-t-elle ? | |
| 258 | + | |
| 259 | +{{ width=85% }} | |
| 260 | + | |
| 261 | +**Par densité** (ventes à moins de 2 km dans les 12 mois précédents) — MdAPE test (%) : | |
| 262 | + | |
| 263 | +{strata_md('dens', 'densité')} | |
| 264 | +**Par région administrative** : | |
| 265 | + | |
| 266 | +{strata_md('region_nom', 'région')} | |
| 267 | +**Par quintile de prix** : | |
| 268 | + | |
| 269 | +{strata_md('prix_q', 'quintile de prix')} | |
| 270 | +**Par âge du bâtiment** : | |
| 271 | + | |
| 272 | +{strata_md('age_cl', 'âge')} | |
| 273 | +**Par lien physique** (0 détaché, 1 jumelé, 2 en rangée, 3 intégré) : | |
| 274 | + | |
| 275 | +{strata_md('lien', 'lien physique')} | |
| 276 | +### Marchés homogènes et hétérogènes (H5) | |
| 277 | + | |
| 278 | +Terciles de la dispersion (écart-type du log des prix ajustés) des dix ventes les plus proches à ≤ 10 km / 24 mois — MdAPE test (%) : | |
| 279 | + | |
| 280 | +{h5_md} | |
| 281 | +### Stabilité dans le temps | |
| 282 | + | |
| 283 | +{stab_md} | |
| 284 | +Le biais trimestriel commun à toutes les méthodes reflète le fait que le niveau de prix du mois courant est inconnu (toutes travaillent au niveau du mois précédent) et la saisonnalité des prix. | |
| 285 | + | |
| 286 | +## 3.7 Intervalles de valeur | |
| 287 | + | |
| 288 | +La dispersion des prix ajustés des comparables retenus fournit un intervalle de valeur. Avec z = 1,28 (couverture nominale de 80 % si les erreurs étaient normales), puis avec un z calibré sur la validation 2024 : | |
| 289 | + | |
| 290 | +{iv_md} | |
| 291 | +## 3.8 Vers un système de recommandation des comparables | |
| 292 | + | |
| 293 | +Règle choisie en 2024 pour chaque strate de densité (parmi les 2 256 configurations couvrant ≥ 95 % des cibles de la strate), évaluée en 2025-2026 : | |
| 294 | + | |
| 295 | +{rc_md} | |
| 296 | +Globalement, ces règles conditionnelles atteignent une MdAPE de **{pct(cond_g['mdape'])}** ({pct(cond_g['p10'])} des estimations à ±10 %, couverture {pct(cond_g['couverture'])}), contre {pct(comp[M0]['mdape'])} pour la meilleure règle unique. | |
| 297 | + | |
| 298 | +# 4. Discussion | |
| 299 | + | |
| 300 | +**Ce que la méthode des comparables fait bien.** Correctement paramétrée — comparables semblables, nombre d'une dizaine, rayon et fenêtre larges, ajustements hédoniques ou même conventionnels — elle atteint une précision ({pct(comp[M0]['mdape'])}) que ne dépasse le meilleur modèle d'apprentissage ré-entraîné chaque trimestre que de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''}, tout en restant intégralement explicable : chaque valeur est la médiane de quelques ventes identifiées, ajustées par des coefficients lisibles. | |
| 301 | + | |
| 302 | +**Ce qui contredit la pratique.** Trois comparables, un rayon de quelques centaines de mètres et une fenêtre de trois mois — les conventions les plus répandues — sont dominés sur toute la surface Erreur(k, d, t) : {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].mdape_test.iloc[0])} de MdAPE (k = 3, ≤ 1 km, 3 mois, grille hédonique, couverture {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].couverture_test.iloc[0])}) contre {pct(e1b['mdape_test'])} pour la meilleure configuration. La rareté des comparables « parfaits » est un argument pour élargir la recherche et ajuster, non pour se restreindre. | |
| 303 | + | |
| 304 | +**Le rôle d'évaluation comme information.** Le ratio au rôle (M13) et le GBM avec rôle (M11b) dominent leurs équivalents sans rôle : la valeur au rôle encapsule des caractéristiques non observées ici (qualité, dépendances, état) et une appréciation locale que les modèles globaux ne captent pas. Les modèles globaux sans rôle (M10, M11a) présentent d'ailleurs un biais négatif en 2025-2026 : l'appréciation est spatialement hétérogène et un indice régional, même ré-estimé chaque trimestre, la sous-estime là où les ventes se concentrent. | |
| 305 | + | |
| 306 | +**Limites.** (1) Règles professionnelles simulées, non dossiers d'évaluateurs. (2) Caractéristiques limitées à celles du rôle et du registre. (3) Délai de publication des ventes ignoré. (4) Un seul type de bien (unifamilial) ; condos et plex restent à traiter. (5) Les configurations ont été choisies sur 2024 puis évaluées sur 2025-2026 : les écarts entre validation et test (tableaux ci-dessus) mesurent la stabilité des règles, et ils sont faibles. | |
| 307 | + | |
| 308 | +# 5. Reproductibilité | |
| 309 | + | |
| 310 | +Code : `pipeline/01_dataset.py` → `06_report.py` (Python 3.14, pandas, numpy, scipy, lightgbm, matplotlib) ; sorties versionnées `out/{VERSION}/` (configurations `configs_E1.csv`, `configs_E2.csv` ; estimations `est_E1.npy`, `est_E2.npy` ; cibles `targets.parquet` ; tables `tables/*.csv` ; `results.json`). Exécution : `./run_all.sh` sur le nœud de calcul (M2U64), `./make_pdf.sh` sur le poste de travail. | |
| 311 | +""" | |
| 312 | +(DOCS / "RAPPORT.md").write_text(txt, encoding="utf-8") | |
| 313 | +log(f"rapport écrit : {DOCS / 'RAPPORT.md'} ({len(txt.split()):,} mots)") | |
added
M4M64aomparables-validation/pipeline/common.py
+85 −0
@@ -0,0 +1,85 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""Validation empirique de la méthode des comparables — fonctions communes. | |
| 3 | +Données : instantané Vrai Prix (transactions ⋈ units) du 2026-10-10. | |
| 4 | +Toutes les sorties sont versionnées sous out/<version>/. | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | +import json, math, os, sqlite3, time | |
| 8 | +from pathlib import Path | |
| 9 | +import numpy as np | |
| 10 | +import pandas as pd | |
| 11 | + | |
| 12 | +ROOT = Path(os.environ.get("CV_ROOT", Path(__file__).resolve().parents[1])) | |
| 13 | +DB = Path(os.environ.get("CV_DB", ROOT / "data" / "vraiprix-20261010.db")) | |
| 14 | +REF = ROOT / "ref" | |
| 15 | +VERSION = os.environ.get("CV_VERSION", time.strftime("%Y-%m-%d")) | |
| 16 | +OUT = ROOT / "out" / VERSION | |
| 17 | +OUT.mkdir(parents=True, exist_ok=True) | |
| 18 | +(OUT / "fig").mkdir(exist_ok=True) | |
| 19 | + | |
| 20 | +# Périodes (séparation chronologique) | |
| 21 | +TRAIN_END = "2023-12-31" # développement | |
| 22 | +VAL_START, VAL_END = "2024-01-01", "2024-12-31" # choix des configurations | |
| 23 | +TEST_START, TEST_END = "2025-01-01", "2026-09-29" # évaluation finale hors échantillon | |
| 24 | +MONTH0 = "2021-01" # premier mois des données | |
| 25 | + | |
| 26 | +def log(msg: str) -> None: | |
| 27 | + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) | |
| 28 | + | |
| 29 | +def connect(path: Path = DB) -> sqlite3.Connection: | |
| 30 | + return sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=60) | |
| 31 | + | |
| 32 | +def _json_default(o): | |
| 33 | + if isinstance(o, np.integer): return int(o) | |
| 34 | + if isinstance(o, np.floating): return None if np.isnan(o) else float(o) | |
| 35 | + if isinstance(o, np.bool_): return bool(o) | |
| 36 | + if isinstance(o, np.ndarray): return o.tolist() | |
| 37 | + if isinstance(o, pd.Timestamp): return o.strftime("%Y-%m-%d") | |
| 38 | + if isinstance(o, float) and math.isnan(o): return None | |
| 39 | + return str(o) | |
| 40 | + | |
| 41 | +def save_json(obj, name: str) -> Path: | |
| 42 | + p = OUT / name; p.parent.mkdir(parents=True, exist_ok=True) | |
| 43 | + with open(p, "w", encoding="utf-8") as f: | |
| 44 | + json.dump(obj, f, ensure_ascii=False, indent=2, default=_json_default) | |
| 45 | + return p | |
| 46 | + | |
| 47 | +def load_json(name: str): | |
| 48 | + with open(OUT / name, encoding="utf-8") as f: | |
| 49 | + return json.load(f) | |
| 50 | + | |
| 51 | +def save_parquet(df: pd.DataFrame, name: str) -> Path: | |
| 52 | + p = OUT / name; df.to_parquet(p, index=False); return p | |
| 53 | + | |
| 54 | +def load_parquet(name: str) -> pd.DataFrame: | |
| 55 | + return pd.read_parquet(OUT / name) | |
| 56 | + | |
| 57 | +def month_index(dates) -> np.ndarray: | |
| 58 | + """Mois entier depuis MONTH0 (2021-01 = 0).""" | |
| 59 | + d = pd.to_datetime(dates) | |
| 60 | + y0, m0 = map(int, MONTH0.split("-")) | |
| 61 | + return ((d.dt.year - y0) * 12 + (d.dt.month - m0)).to_numpy() | |
| 62 | + | |
| 63 | +def to_xy_km(lat, lng, lat0=46.8, lng0=-71.2): | |
| 64 | + """Projection équirectangulaire locale (km) : erreur < 1 % pour des distances < 50 km.""" | |
| 65 | + x = (np.asarray(lng, float) - lng0) * 111.32 * np.cos(np.radians(np.asarray(lat, float))) | |
| 66 | + y = (np.asarray(lat, float) - lat0) * 110.57 | |
| 67 | + return x, y | |
| 68 | + | |
| 69 | +def load_repertoire() -> pd.DataFrame: | |
| 70 | + mun = pd.read_csv(REF / "MUN.csv", dtype=str, encoding="utf-8-sig") | |
| 71 | + mun = mun.rename(columns=str.lower) | |
| 72 | + out = pd.DataFrame({ | |
| 73 | + "code_mun": mun["mcode"].str.zfill(5), | |
| 74 | + "mun_nom": mun["munnom"], | |
| 75 | + "region": mun["regadm"].str.extract(r"\((\d+)\)")[0], | |
| 76 | + "region_nom": mun["regadm"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 77 | + "mrc": mun["mrc"].str.extract(r"\((\d+)\)")[0], | |
| 78 | + "mrc_nom": mun["mrc"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 79 | + }) | |
| 80 | + return out.drop_duplicates("code_mun") | |
| 81 | + | |
| 82 | +# Codages catégoriels (ordre stable) | |
| 83 | +LIEN = {"Détaché": 0, "Jumelé": 1, "En rangée 1 côté": 2, "En rangée plus de 1 côté": 2, "Intégré": 3} | |
| 84 | +GENRE = {"De plain-pied": 0, "Unimodulaire": 0, "À étages entiers": 1, "À niveaux décalés": 2, "À étage mansardé": 3} | |
| 85 | +BTYPE = {"single-story": 0, "full-story": 1, "mansard": 2, "split-level": 3, "single-wide": 4} | |
added
M4M64aomparables-validation/run_all.sh
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Chaîne complète sur le nœud de calcul (M2U64). Usage : CV_VERSION=2026-10-10 ./run_all.sh [étape de départ] | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +PY=${PY:-$HOME/uqo-eval-terrains/env/bin/python} | |
| 6 | +export CV_VERSION=${CV_VERSION:-$(date +%F)} | |
| 7 | +START=${1:-1} | |
| 8 | +for s in 01_dataset 02_index 03_models 04_backtest 05_analyze 06_report; do | |
| 9 | + n=${s%%_*} | |
| 10 | + [ "$((10#$n))" -lt "$START" ] && continue | |
| 11 | + echo "=== $s ($(date +%T)) ===" | |
| 12 | + (cd pipeline && $PY $s.py) | |
| 13 | +done | |
| 14 | +echo "=== terminé $(date +%T) ===" | |
added
M4M64bomparables-validation/.gitignore
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +out/ | |
| 2 | +data/ | |
| 3 | +*.pyc | |
| 4 | +__pycache__/ | |
| 5 | +.DS_Store | |
added
M4M64bomparables-validation/README.md
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +# Validation empirique de la méthode des comparables (Vrai Prix) | |
| 2 | + | |
| 3 | +Projet de recherche : rétro-évaluation à grande échelle de la méthode de comparaison des ventes sur les transactions | |
| 4 | +résidentielles unifamiliales du Québec (Vrai Prix, 2021-01 → 2026-09). Protocole : `docs/PROTOCOLE.md`. | |
| 5 | +Rapport de résultats : `docs/RAPPORT.md` / `docs/RAPPORT.pdf`. | |
| 6 | + | |
| 7 | +## Organisation | |
| 8 | + | |
| 9 | +``` | |
| 10 | +pipeline/ 01_dataset → 02_index → 03_models → 04_backtest → 05_analyze → 06_report | |
| 11 | +docs/ PROTOCOLE.md, RAPPORT.md(.pdf), header.tex | |
| 12 | +out/<version>/ sales.parquet, index_asof.npy, models_*.json, est_E*.npy, targets.parquet, tables/, fig/, results.json | |
| 13 | +data/ (nœud seulement) vraiprix-20261010.db — instantané de la base de production Vrai Prix | |
| 14 | +ref/ MUN.csv (répertoire des municipalités MAMH : code → MRC → région) | |
| 15 | +``` | |
| 16 | + | |
| 17 | +## Exécution | |
| 18 | + | |
| 19 | +Le calcul se fait sur **M2U64** (`~/comparables-validation`, venv `~/uqo-eval-terrains/env`, Python 3.14 : pandas, numpy, scipy, | |
| 20 | +lightgbm, matplotlib). Le laptop garde le code, les docs et une copie des sorties. | |
| 21 | + | |
| 22 | +```bash | |
| 23 | +rsync -a --exclude out --exclude data --exclude .git ~/Desktop/Validation-Comparables/ M2U64:comparables-validation/ | |
| 24 | +ssh M2U64 'cd ~/comparables-validation && CV_VERSION=2026-10-10 ./run_all.sh' # ~ 30 min | |
| 25 | +./make_pdf.sh # rapatrie out/ + docs/ et compile le PDF | |
| 26 | +``` | |
| 27 | + | |
| 28 | +Variables : `CV_VERSION` (dossier de sortie), `CV_SAMPLE=3000` (échantillon de cibles pour un essai), `CV_PROCS` (processus, 20 par défaut). | |
| 29 | + | |
| 30 | +## Données | |
| 31 | + | |
| 32 | +Instantané `vraiprix.db` pris sur M3U96a (`~/apps/vrai-prix/data`) le 2026-10-10 : 766 596 transactions, 3,75 M d'unités du rôle. | |
| 33 | +Jeu propre : ventes `unifamilial` jumelées au rôle, règles R0-R11 journalisées dans `out/<v>/exclusions.csv`. | |
added
M4M64bomparables-validation/docs/PROTOCOLE.md
+101 −0
@@ -0,0 +1,101 @@ | ||
| 1 | +# Revisiter la méthode de comparaison des ventes : protocole de validation empirique à grande échelle | |
| 2 | + | |
| 3 | +**Titre de travail (article A)** — *Revisiting the Sales Comparison Approach: A Large-Scale Empirical Validation of Comparable Selection, Price Adjustments, and Value Reconciliation* | |
| 4 | + | |
| 5 | +**Variante** — *Does the Sales Comparison Approach Actually Work? Evidence from Large-Scale Residential Property Transactions* | |
| 6 | + | |
| 7 | +Données : transactions historiques **Vrai Prix** (registre foncier du Québec jumelé au rôle d'évaluation), instantané du 2026-10-10. | |
| 8 | + | |
| 9 | +## 1. Question de recherche | |
| 10 | + | |
| 11 | +La méthode des comparables, telle qu'enseignée et pratiquée par les évaluateurs agréés, produit-elle réellement les estimations les plus fiables ? Et surtout, quelles règles de **sélection**, d'**ajustement**, de **pondération** et de **réconciliation** des comparables sont empiriquement justifiées ? | |
| 12 | + | |
| 13 | +| Dimension | Question testée | Où dans le pipeline | | |
| 14 | +|---|---|---| | |
| 15 | +| Sélection | Quels comparables sont réellement les plus pertinents (proximité, similarité physique, « moins ajustés ») ? | E1 (facteur `sel`), E2 | | |
| 16 | +| Nombre | Faut-il 1, 3, 5, 10, 20 ou 50 ventes ? | E1 (facteur `k`) | | |
| 17 | +| Ajustements | Quels ajustements sont statistiquement justifiés (grille conventionnelle, hédonique, locale, non linéaire, ratio au rôle) ? | E2 (facteur `adj`) | | |
| 18 | +| Pondération | Égalitaire, distance, récence, similarité ? | E2 (facteur `w`) | | |
| 19 | +| Réconciliation | Médiane, moyenne, moyenne pondérée, combinaison optimisée (M9) ? | E2 (facteur `recon`), M9 | | |
| 20 | +| Fiabilité | Dans quels marchés (densité, région, segment de prix, hétérogénéité) la méthode tient-elle ou défaille-t-elle ? | strates, H2-H5 | | |
| 21 | + | |
| 22 | +## 2. Protocole expérimental (rétro-évaluation « point dans le temps ») | |
| 23 | + | |
| 24 | +Pour chaque transaction cible : | |
| 25 | + | |
| 26 | +1. **Masquage** du prix de vente réel. | |
| 27 | +2. **Reconstitution du marché à la date d'évaluation** : seules les ventes conclues *strictement avant* la date de la cible sont admissibles (le bien lui-même, s'il a déjà été vendu, est exclu du bassin : on évalue par comparables, pas par vente répétée). L'indice temporel, les coefficients hédoniques et les modèles d'apprentissage sont ré-estimés avec les seules ventes antérieures à la période d'évaluation (indice : ré-estimé chaque mois avec toutes les ventes des mois précédents ; modèles : estimés sur les ventes < 2024-01 pour la validation 2024 et < 2025-01 pour le test 2025-26). | |
| 28 | +3. **Application** de chaque configuration (sélection × rayon × fenêtre × nombre × ajustement × pondération × réconciliation) sur exactement les mêmes cibles. | |
| 29 | +4. **Comparaison** avec le prix réel : erreur absolue en pourcentage, biais, part des estimations à ±5/10/20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur), dispersion des prix ajustés (intervalle de valeur). | |
| 30 | + | |
| 31 | +### Séparation chronologique | |
| 32 | + | |
| 33 | +| Période | Rôle | Ventes | | |
| 34 | +|---|---|---| | |
| 35 | +| 2021-01 → 2023-12 | développement (estimation des grilles d'ajustement et des modèles) | ~ 200 000 | | |
| 36 | +| 2024 | validation : choix des configurations et des hyperparamètres | ~ 65 000 cibles | | |
| 37 | +| 2025-01 → 2026-09 | évaluation finale hors échantillon | ~ 125 000 cibles | | |
| 38 | + | |
| 39 | +Les comparables d'une cible de 2025 peuvent être des ventes de 2024 ou de 2025 antérieures à sa date : c'est l'information dont disposait un évaluateur ce jour-là (nous ignorons le délai de publication au registre foncier, ce qui avantage légèrement toutes les méthodes de la même façon). | |
| 40 | + | |
| 41 | +## 3. Expériences | |
| 42 | + | |
| 43 | +### E1 — surface empirique de performance Erreur(k, d, t) | |
| 44 | + | |
| 45 | +- k ∈ {1, 3, 5, 7, 10, 15, 20, 30, 50} comparables ; | |
| 46 | +- d ∈ {0,5 ; 1 ; 2 ; 5 ; 10 ; 25} km (rayon maximal) ; | |
| 47 | +- t ∈ {3, 6, 12, 24} mois (ancienneté maximale) ; | |
| 48 | +- sélection ∈ {`geo` les plus proches, `sim` les plus semblables physiquement, `mix` compromis} ; | |
| 49 | +- ajustement ∈ {`none` temps seulement, `hedo` grille hédonique} ; réconciliation = médiane. | |
| 50 | + | |
| 51 | +1 296 configurations × toutes les cibles. | |
| 52 | + | |
| 53 | +### E2 — plan factoriel des règles (trois ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois)) | |
| 54 | + | |
| 55 | +- sélection ∈ {`geo`, `sim`, `mix`, `adj` = comparables nécessitant le moins d'ajustements} ; | |
| 56 | +- ajustement ∈ {`raw` aucun, `none` temps seul, `conv` grille conventionnelle stylisée, `hedo` grille hédonique globale (within-municipalité), `hedo_reg` grille régionale rétrécie, `local` régression sur le voisinage (≤ 10 km, ≥ 40 ventes, ridge vers la grille régionale), `gbm` ajustements non linéaires (LightGBM), `role` ratio prix/rôle (comparables de la même municipalité et du même rôle)} ; | |
| 57 | +- pondération ∈ {égale, 1/(0,25 + d), exp(−mois/6), 1/(0,1 + ajustement brut), produit des trois} ; | |
| 58 | +- réconciliation ∈ {médiane (pondérée), moyenne (pondérée) en dollars}. | |
| 59 | + | |
| 60 | +960 configurations. | |
| 61 | + | |
| 62 | +### Compétition de modèles | |
| 63 | + | |
| 64 | +| Modèle | Définition opérationnelle | | |
| 65 | +|---|---| | |
| 66 | +| M1 | médiane des 3 ventes les plus proches (≤ 2 km, ≤ 6 mois), temps seul | | |
| 67 | +| M2 | moyenne des 5 plus proches (≤ 5 km, ≤ 12 mois), temps seul | | |
| 68 | +| M3 | 5 comparables choisis par similarité physique | | |
| 69 | +| M4 | 5 plus proches pondérés par la distance | | |
| 70 | +| M5 | 5 plus proches pondérés par la récence | | |
| 71 | +| M6 | grille d'ajustement conventionnelle stylisée (½ du prix au m² pour l'aire, ½ de la valeur foncière au m² pour le terrain plafonnée à 15 %, −0,5 %/an d'âge plafonné à 20 %, ±5 % détaché/attaché) | | |
| 72 | +| M7 | grille d'ajustement hédonique (coefficients within-municipalité estimés sur les ventes antérieures) | | |
| 73 | +| M8a / M8b | grille locale (régression sur le voisinage) / ajustements non linéaires (GBM) | | |
| 74 | +| M9 | réconciliation optimisée : combinaison convexe (NNLS) de sept estimations par comparables, poids appris en 2024 | | |
| 75 | +| M10 | modèle hédonique global direct (β within + effets région, MRC, municipalité, unité de voisinage) | | |
| 76 | +| M11a / M11b | gradient boosting sans / avec la valeur au rôle | | |
| 77 | +| M12 | hybride : combinaison convexe de M11b et des estimations par comparables ajustées (GBM, hédonique) | | |
| 78 | +| M13 | ratio au rôle : rôle de la cible × ratio prix/rôle médian des 5 plus proches du même rôle | | |
| 79 | +| M0 | meilleure configuration de E1 ∪ E2 choisie sur la validation 2024, évaluée en 2025-26 | | |
| 80 | + | |
| 81 | +### Hypothèses | |
| 82 | + | |
| 83 | +- **H1** — un comparable très proche mais peu semblable peut être moins informatif qu'un comparable plus éloigné mais très semblable. | |
| 84 | +- **H2** — l'ajout de comparables améliore la précision jusqu'à un seuil, au-delà duquel l'hétérogénéité domine ; le seuil dépend de la densité. | |
| 85 | +- **H3** — les ajustements empiriques locaux surpassent les grilles fixes mais deviennent instables dans les petits échantillons. | |
| 86 | +- **H4** — la pondération optimale dépend de la liquidité du sous-marché. | |
| 87 | +- **H5** — les méthodes simples et transparentes rivalisent avec l'apprentissage automatique dans les marchés homogènes. | |
| 88 | + | |
| 89 | +### Inférence | |
| 90 | + | |
| 91 | +Différences d'erreurs appariées (mêmes cibles) avec intervalles de confiance par **bootstrap par grappes** (municipalité × trimestre), qui respecte les dépendances spatiales et temporelles. | |
| 92 | + | |
| 93 | +## 4. Ce que les données permettent — et ne permettent pas | |
| 94 | + | |
| 95 | +Caractéristiques observées pour chaque vente (registre foncier ⋈ rôle) : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable (étages), superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle (en vigueur à la vente) et sa part terrain. **Non observés** : garage, piscine, nombre de chambres et de salles de bains, qualité et état, rénovations. Les ajustements correspondants ne peuvent donc pas être testés ici ; les erreurs résiduelles les contiennent. | |
| 96 | + | |
| 97 | +Nous validons la **précision prédictive** de règles professionnelles *simulées*, pas le jugement réel d'un évaluateur (il faudrait ses dossiers, ses comparables et ses ajustements). La distinction est maintenue partout dans le rapport. | |
| 98 | + | |
| 99 | +## 5. Article B (suite envisagée) | |
| 100 | + | |
| 101 | +Vers une méthode des comparables optimisée empiriquement : algorithme transparent de sélection/ajustement entraîné sur les ventes historiques et validé sur les ventes futures ; règles conditionnelles au type de bien, à la densité et à la qualité des données ; intégration dans UQO Éval pour montrer aux étudiants la comparaison entre règles traditionnelles et règles estimées. | |
added
M4M64bomparables-validation/docs/header.tex
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +% En-tête LaTeX — Grille empirique d'ajustement des terrains (UQO Éval) | |
| 2 | +\usepackage{fontspec} | |
| 3 | +\setmainfont{Helvetica Neue}[Ligatures=TeX] | |
| 4 | +\setsansfont{Helvetica Neue} | |
| 5 | +\setmonofont{Menlo}[Scale=0.82] | |
| 6 | +\usepackage{newunicodechar} | |
| 7 | +\newunicodechar{→}{\ensuremath{\rightarrow}}\newunicodechar{≥}{\ensuremath{\geq}}\newunicodechar{≤}{\ensuremath{\leq}}\newunicodechar{×}{\ensuremath{\times}}\newunicodechar{−}{\ensuremath{-}}\newunicodechar{≈}{\ensuremath{\approx}}\newunicodechar{∂}{\ensuremath{\partial}}\newunicodechar{⋈}{\ensuremath{\bowtie}}\newunicodechar{Δ}{\ensuremath{\Delta}}\newunicodechar{τ}{\ensuremath{\tau}}\newunicodechar{σ}{\ensuremath{\sigma}}\newunicodechar{β}{\ensuremath{\beta}}\newunicodechar{θ}{\ensuremath{\theta}}\newunicodechar{γ}{\ensuremath{\gamma}}\newunicodechar{α}{\ensuremath{\alpha}}\newunicodechar{δ}{\ensuremath{\delta}}\newunicodechar{ε}{\ensuremath{\varepsilon}}\newunicodechar{²}{\textsuperscript{2}}\newunicodechar{⇒}{\ensuremath{\Rightarrow}}\newunicodechar{▶}{\ensuremath{\blacktriangleright}}\newunicodechar{│}{|}\newunicodechar{▼}{\ensuremath{\blacktriangledown}}\newunicodechar{─}{-}\newunicodechar{┌}{+}\newunicodechar{┐}{+}\newunicodechar{└}{+}\newunicodechar{┘}{+}\newunicodechar{├}{+}\newunicodechar{┤}{+} | |
| 8 | +\usepackage{xcolor} | |
| 9 | +\definecolor{uqo}{HTML}{006085}\definecolor{turq}{HTML}{2BCCAE}\definecolor{corail}{HTML}{EA6047}\definecolor{encre}{HTML}{16181B}\definecolor{papier}{HTML}{F4F5F3} | |
| 10 | +\usepackage{titlesec} | |
| 11 | +\titleformat{\section}{\Large\bfseries\color{uqo}}{}{0em}{}[{\color{uqo}\titlerule[0.8pt]}] | |
| 12 | +\titleformat{\subsection}{\large\bfseries\color{uqo}}{}{0em}{} | |
| 13 | +\titlespacing*{\section}{0pt}{18pt plus 4pt}{8pt} | |
| 14 | +\usepackage{etoolbox} | |
| 15 | +\AtBeginEnvironment{longtable}{\scriptsize\renewcommand{\arraystretch}{1.18}} | |
| 16 | +\usepackage{colortbl}\arrayrulecolor{uqo!35} | |
| 17 | +\usepackage{tcolorbox} | |
| 18 | +\renewenvironment{quote}{\begin{tcolorbox}[colback=papier,colframe=turq,boxrule=0pt,leftrule=3pt,arc=1pt,left=6pt,right=6pt,top=4pt,bottom=4pt]}{\end{tcolorbox}} | |
| 19 | +\usepackage{fancyhdr} | |
| 20 | +\pagestyle{fancy}\fancyhf{} | |
| 21 | +\fancyhead[L]{\footnotesize\color{gray}Grille empirique d'ajustement des terrains — UQO Éval}\fancyhead[R]{\footnotesize\color{gray}IMM1003 · IMM1033} | |
| 22 | +\fancyfoot[C]{\footnotesize\color{gray}\thepage}\renewcommand{\headrulewidth}{0.3pt} | |
| 23 | +\usepackage{caption}\captionsetup{font=small,labelfont={bf,color=uqo}} | |
| 24 | +\usepackage{float}\floatplacement{figure}{H} | |
| 25 | +\setlength{\emergencystretch}{3em}\hyphenpenalty=300\tolerance=1500 | |
| 26 | +\usepackage{microtype} | |
| 27 | +\makeatletter\renewcommand{\maketitle}{\begin{center}{\color{uqo}\Huge\bfseries\@title\par}\vspace{8pt}{\normalsize\color{gray}\@author\par\@date\par}\end{center}\vspace{10pt}}\makeatother | |
| 28 | +\usepackage{polyglossia}\setdefaultlanguage[frenchpart=false]{french} | |
added
M4M64bomparables-validation/make_pdf.sh
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Rapatrie les résultats depuis M2U64 et produit docs/RAPPORT.pdf (pandoc → XeLaTeX) | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +V=${CV_VERSION:-2026-10-10} | |
| 6 | +mkdir -p out/$V | |
| 7 | +rsync -az --exclude 'est_E*.npy' --exclude 'sd_E2.npy' --exclude '*_pred_*.npy' --exclude 'sales.parquet' M2U64:comparables-validation/out/$V/ out/$V/ | |
| 8 | +rsync -az M2U64:comparables-validation/docs/ docs/ 2>/dev/null || true | |
| 9 | +cd docs | |
| 10 | +pandoc RAPPORT.md -o RAPPORT.pdf --pdf-engine=xelatex -H header.tex --toc --toc-depth=2 \ | |
| 11 | + -V geometry:"a4paper,margin=17mm,top=20mm,bottom=20mm" -V fontsize=10pt -V linkcolor=uqo -V urlcolor=uqo -V toccolor=uqo -V colorlinks=true \ | |
| 12 | + -V toc-title="Table des matières" --resource-path=.:../out/$V/fig 2>&1 | grep -v "Missing character\|Overfull\|Underfull" | tail -8 || true | |
| 13 | +echo "PDF : $(pwd)/RAPPORT.pdf ($(du -h RAPPORT.pdf | cut -f1))" | |
added
M4M64bomparables-validation/pipeline/01_dataset.py
+97 −0
@@ -0,0 +1,97 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""01 — Jeu de données : transactions unifamiliales Vrai Prix ⋈ rôle, nettoyage journalisé, | |
| 3 | +reventes, géographie, périodes. Sorties : sales.parquet, exclusions.csv, dataset_stats.json.""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +con = connect() | |
| 7 | +log(f"lecture {DB}") | |
| 8 | +sql = """ | |
| 9 | +select t.id, t.date, t.amount, t.lat, t.lng, t.city, t.property_type, t.year_built, t.floor_area, | |
| 10 | + t.building_type, t.id_provinc, t.valeur_role as role_vente, t.land_area, | |
| 11 | + u.code_mun, u.municipalite, u.type_prop as u_type, u.annee_construction, u.aire_etages_m2, | |
| 12 | + u.superficie_terrain_m2, u.front_terrain_m, u.nb_etages, u.nb_logements, u.lien_physique, | |
| 13 | + u.genre_construction, u.unite_voisinage, u.dat_cond_marche, u.valeur_terrain, u.valeur_batiment, | |
| 14 | + u.valeur_role as role_actuel, u.valeur_anterieure as role_anterieur, u.cubf | |
| 15 | +from transactions t left join units u on u.id_provinc = t.id_provinc | |
| 16 | +where t.property_type = 'unifamilial' | |
| 17 | +""" | |
| 18 | +df = pd.read_sql(sql, con) | |
| 19 | +log(f"{len(df):,} transactions unifamiliales brutes") | |
| 20 | +n0 = len(df) | |
| 21 | +excl = [] # (règle, n) | |
| 22 | +def drop(mask, rule): | |
| 23 | + global df | |
| 24 | + n = int(mask.sum()); excl.append((rule, n)); df = df[~mask].copy(); log(f" {rule}: -{n:,} → {len(df):,}") | |
| 25 | + | |
| 26 | +df["date"] = pd.to_datetime(df["date"]) | |
| 27 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 28 | +drop(df["id_provinc"].isna() | df["code_mun"].isna(), "R0 sans jumelage au rôle (id_provinc / code_mun)") | |
| 29 | +drop(df.duplicated(["id_provinc", "date"]), "R1 doublons (même unité, même date)") | |
| 30 | +drop(~df["u_type"].isin(["unifamilial"]) | (df["nb_logements"].fillna(1) > 1), "R2 unité du rôle non unifamiliale (chalet, plex, terrain…) ou > 1 logement") | |
| 31 | +drop(df["lat"].isna() | df["lng"].isna() | (df["lat"] < 44.5) | (df["lat"] > 63) | (df["lng"] > -56) | (df["lng"] < -80), "R3 coordonnées absentes ou hors Québec") | |
| 32 | +fa = df["floor_area"].fillna(df["aire_etages_m2"]) | |
| 33 | +df["floor_area"] = fa | |
| 34 | +drop(fa.isna() | (fa < 40) | (fa > 600), "R4 aire habitable absente ou hors [40, 600] m²") | |
| 35 | +la = df["land_area"].fillna(df["superficie_terrain_m2"]) | |
| 36 | +df["land_area"] = la | |
| 37 | +drop(la.isna() | (la < 80) | (la > 20000), "R5 superficie de terrain absente ou hors [80, 20 000] m²") | |
| 38 | +yb = df["year_built"].fillna(df["annee_construction"]) | |
| 39 | +df["year_built"] = yb | |
| 40 | +drop(yb.isna() | (yb < 1850) | (yb > df["date"].dt.year), "R6 année de construction absente, < 1850 ou postérieure à la vente (vente de terrain)") | |
| 41 | +drop((df["year_built"] == df["date"].dt.year), "R6b vente l'année même de la construction (prix du terrain ou d'une construction inachevée : médiane 346 k$ et ratio au rôle 1,15 contre 450-480 k$ et 1,3 aux âges 1-4)") | |
| 42 | +drop((df["amount"] < 50000) | (df["amount"] > 3_000_000), "R7 prix hors [50 k$, 3 M$]") | |
| 43 | +ratio = df["amount"] / df["role_vente"] | |
| 44 | +drop(df["role_vente"].isna() | (df["role_vente"] <= 0) | (ratio < 0.33) | (ratio > 3.0), "R8 rôle absent ou ratio prix/rôle hors [0,33 ; 3] (transactions présumées non représentatives)") | |
| 45 | +# reventes rapides avec forte variation (flips / rénovations : les caractéristiques enregistrées ne décrivent plus le bien) | |
| 46 | +df = df.sort_values(["id_provinc", "date"]).reset_index(drop=True) | |
| 47 | +g = df.groupby("id_provinc") | |
| 48 | +prev_date = g["date"].shift(1); prev_amt = g["amount"].shift(1) | |
| 49 | +dd = (df["date"] - prev_date).dt.days | |
| 50 | +chg = df["amount"] / prev_amt - 1 | |
| 51 | +flip = (dd <= 180) & (chg.abs() > 0.40) | |
| 52 | +drop(flip.fillna(False), "R9 revente ≤ 180 jours avec variation > 40 % (flip)") | |
| 53 | +drop(df["lien_physique"].isna() | df["genre_construction"].isna(), "R10 lien physique ou genre de construction absent au rôle") | |
| 54 | + | |
| 55 | +# ----------------------------------------------------------------- variables dérivées | |
| 56 | +rep = load_repertoire() | |
| 57 | +df = df.merge(rep, on="code_mun", how="left") | |
| 58 | +drop(df["region"].isna(), "R11 municipalité absente du répertoire MAMH") | |
| 59 | +df["sale_year"] = df["date"].dt.year | |
| 60 | +df["month"] = month_index(df["date"]) | |
| 61 | +df["day"] = (df["date"] - pd.Timestamp("2021-01-01")).dt.days.astype(int) | |
| 62 | +df["age"] = (df["sale_year"] - df["year_built"]).clip(lower=0) | |
| 63 | +df["ln_price"] = np.log(df["amount"]) | |
| 64 | +df["ln_floor"] = np.log(df["floor_area"]) | |
| 65 | +df["ln_land"] = np.log(df["land_area"]) | |
| 66 | +df["lien"] = df["lien_physique"].map(LIEN).fillna(0).astype(int) | |
| 67 | +df["genre"] = df["genre_construction"].map(GENRE).fillna(0).astype(int) | |
| 68 | +df["btype"] = df["building_type"].map(BTYPE).fillna(-1).astype(int) | |
| 69 | +df["etages"] = df["nb_etages"].fillna(1).clip(1, 3) | |
| 70 | +df["x_km"], df["y_km"] = to_xy_km(df["lat"], df["lng"]) | |
| 71 | +# millésime du rôle au moment de la vente (rôle courant / antérieur / autre) | |
| 72 | +df["role_vintage"] = np.where(np.isclose(df["role_vente"], df["role_actuel"].fillna(-1)), 2, | |
| 73 | + np.where(np.isclose(df["role_vente"], df["role_anterieur"].fillna(-1)), 1, 0)) | |
| 74 | +df["ratio_role"] = df["amount"] / df["role_vente"] | |
| 75 | +df["part_terrain"] = (df["valeur_terrain"] / df["role_actuel"]).clip(0.02, 0.95).fillna(0.2) | |
| 76 | +df["neuf"] = (df["age"] <= 3).astype(int) | |
| 77 | +df["period"] = np.select([df["date"] <= TRAIN_END, df["date"] <= VAL_END], ["train", "val"], "test") | |
| 78 | +df = df.sort_values("date").reset_index(drop=True) | |
| 79 | +df["idx"] = np.arange(len(df), dtype=np.int32) | |
| 80 | + | |
| 81 | +keep = ["idx", "id", "id_provinc", "date", "day", "month", "sale_year", "period", "amount", "ln_price", | |
| 82 | + "lat", "lng", "x_km", "y_km", "code_mun", "mun_nom", "mrc", "mrc_nom", "region", "region_nom", | |
| 83 | + "unite_voisinage", "floor_area", "ln_floor", "land_area", "ln_land", "year_built", "age", "neuf", | |
| 84 | + "lien", "genre", "btype", "etages", "role_vente", "role_vintage", "ratio_role", "part_terrain"] | |
| 85 | +df = df[keep] | |
| 86 | +save_parquet(df, "sales.parquet") | |
| 87 | +pd.DataFrame(excl, columns=["regle", "n"]).to_csv(OUT / "exclusions.csv", index=False) | |
| 88 | +stats = { | |
| 89 | + "brut": n0, "propre": len(df), "exclusions": excl, | |
| 90 | + "par_periode": df["period"].value_counts().to_dict(), | |
| 91 | + "par_annee": df["sale_year"].value_counts().sort_index().to_dict(), | |
| 92 | + "par_region": df.groupby("region_nom").size().sort_values(ascending=False).to_dict(), | |
| 93 | + "prix": {k: float(v) for k, v in df["amount"].describe().items()}, | |
| 94 | + "dates": [str(df["date"].min().date()), str(df["date"].max().date())], | |
| 95 | +} | |
| 96 | +save_json(stats, "dataset_stats.json") | |
| 97 | +log(f"jeu propre : {len(df):,} ventes ({stats['par_periode']})") | |
added
M4M64bomparables-validation/pipeline/02_index.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""02 — Indice temporel régional « point dans le temps ». | |
| 3 | +Pour chaque mois d'évaluation m, on ré-estime un indice hédonique mensuel par région administrative | |
| 4 | +avec toutes les ventes antérieures au mois m (fenêtre expansive) (jamais le mois m lui-même). Niveau de référence = m-1, | |
| 5 | +dernier mois entièrement connu. Sortie : index_asof.npy [m_eval, région, mois] (log), regions.json, | |
| 6 | +et un indice « plein échantillon » descriptif (index_full.csv).""" | |
| 7 | +from common import * | |
| 8 | + | |
| 9 | +df = load_parquet("sales.parquet") | |
| 10 | +regions = sorted(df["region"].unique().tolist()) | |
| 11 | +rid = {r: i for i, r in enumerate(regions)} | |
| 12 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 13 | +M = int(df["month"].max()) + 1 | |
| 14 | +M_EVAL0 = int(month_index(pd.Series([VAL_START]))[0]) | |
| 15 | +WINDOW = 999 # fenêtre « expansive » : toutes les ventes antérieures au mois m | |
| 16 | +SHRINK = 40.0 # n/(n+SHRINK) : rétrécissement de l'effet mensuel régional vers l'effet provincial | |
| 17 | + | |
| 18 | +X_cols = ["ln_floor", "ln_land", "age"] | |
| 19 | +def design(d): | |
| 20 | + X = np.column_stack([np.ones(len(d)), d["ln_floor"], d["ln_land"], d["age"], d["age"] ** 2 / 100, | |
| 21 | + (d["lien"] == 1), (d["lien"] >= 2), (d["genre"] == 1), (d["genre"] == 2), (d["genre"] == 3), | |
| 22 | + d["etages"], d["neuf"]]).astype(float) | |
| 23 | + return X | |
| 24 | + | |
| 25 | +def fit_month_effects(d, months): | |
| 26 | + """OLS à effets fixes de municipalité (estimateur within) : ln P ~ X + effets mensuels (base = dernier mois), | |
| 27 | + toutes les variables étant centrées par municipalité pour que l'indice ne soit pas contaminé par les | |
| 28 | + déplacements du mélange des ventes entre municipalités chères et bon marché. Renvoie effets et n par mois.""" | |
| 29 | + X = design(d)[:, 1:] | |
| 30 | + mm = d["month"].to_numpy() | |
| 31 | + base = months[-1] | |
| 32 | + dums = np.column_stack([(mm == m).astype(float) for m in months if m != base]) | |
| 33 | + A = np.hstack([X, dums]); y = d["ln_price"].to_numpy(float) | |
| 34 | + codes, inv = np.unique(d["code_mun"].to_numpy(), return_inverse=True) | |
| 35 | + cnt = np.bincount(inv).astype(float) | |
| 36 | + mA = np.zeros((len(codes), A.shape[1])); np.add.at(mA, inv, A); A = A - mA[inv] / cnt[inv, None] | |
| 37 | + my = np.bincount(inv, y) / cnt; y = y - my[inv] | |
| 38 | + beta, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 39 | + eff = {base: 0.0} | |
| 40 | + k = X.shape[1] | |
| 41 | + for j, m in enumerate([m for m in months if m != base]): | |
| 42 | + eff[m] = float(beta[k + j]) | |
| 43 | + n = {m: int((mm == m).sum()) for m in months} | |
| 44 | + return eff, n | |
| 45 | + | |
| 46 | +idx = np.full((M - M_EVAL0, len(regions), M), np.nan, dtype=np.float32) | |
| 47 | +log(f"indice point-dans-le-temps : mois d'évaluation {M_EVAL0}..{M-1}, {len(regions)} régions") | |
| 48 | +for m in range(M_EVAL0, M): | |
| 49 | + months = list(range(max(0, m - WINDOW), m)) | |
| 50 | + win = df[(df["month"] >= months[0]) & (df["month"] < m)] | |
| 51 | + prov_eff, _ = fit_month_effects(win, months) | |
| 52 | + for r in regions: | |
| 53 | + dr = win[win["region"] == r] | |
| 54 | + if len(dr) < 200: | |
| 55 | + for mo in months: idx[m - M_EVAL0, rid[r], mo] = prov_eff[mo] | |
| 56 | + continue | |
| 57 | + eff, n = fit_month_effects(dr, months) | |
| 58 | + for mo in months: | |
| 59 | + w = n[mo] / (n[mo] + SHRINK) | |
| 60 | + idx[m - M_EVAL0, rid[r], mo] = w * eff[mo] + (1 - w) * prov_eff[mo] | |
| 61 | + if m % 6 == 0: log(f" mois {m} ({len(win):,} ventes dans la fenêtre)") | |
| 62 | +np.save(OUT / "index_asof.npy", idx) | |
| 63 | +save_json({"regions": regions, "M_EVAL0": M_EVAL0, "M": M, "window": WINDOW, "shrink": SHRINK}, "index_meta.json") | |
| 64 | + | |
| 65 | +# indice descriptif plein échantillon (pour le rapport seulement — pas utilisé dans la rétro-évaluation) | |
| 66 | +rows = [] | |
| 67 | +months = list(range(0, M)) | |
| 68 | +eff_p, n_p = fit_month_effects(df, months) | |
| 69 | +for mo in months: | |
| 70 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": "QC", "log_idx": eff_p[mo] - eff_p[months[0]], "n": n_p[mo]}) | |
| 71 | +for r in regions: | |
| 72 | + dr = df[df["region"] == r] | |
| 73 | + if len(dr) < 2000: continue | |
| 74 | + eff, n = fit_month_effects(dr, months) | |
| 75 | + for mo in months: | |
| 76 | + rows.append({"month": mo, "ym": f"{2021 + mo // 12}-{mo % 12 + 1:02d}", "region": r, "log_idx": eff[mo] - eff[months[0]], "n": n[mo]}) | |
| 77 | +pd.DataFrame(rows).to_csv(OUT / "index_full.csv", index=False) | |
| 78 | +log("indice terminé") | |
added
M4M64bomparables-validation/pipeline/03_models.py
+119 −0
@@ -0,0 +1,119 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""03 — Modèles d'ajustement et modèles de référence, ré-estimés chaque trimestre avec les seules ventes | |
| 3 | +antérieures au trimestre (as-of 2024-01, 2024-04, …, 2026-07) — comme le ferait un AVM ré-entraîné régulièrement : | |
| 4 | + • grille hédonique globale (coefficients) et régionale rétrécie → ajustements B / C | |
| 5 | + • LightGBM (objectif L1 = médiane) sur le prix ajusté dans le temps (avec et sans rôle) → ajustements C-GBM, M11 | |
| 6 | + • hédonique globale directe avec effets de municipalité → M10 | |
| 7 | +Sorties : models_<période>.json, gbm_pred_<période>.npy (prédiction f pour TOUTES les ventes).""" | |
| 8 | +import pickle | |
| 9 | +from common import * | |
| 10 | +import lightgbm as lgb | |
| 11 | + | |
| 12 | +df = load_parquet("sales.parquet") | |
| 13 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 14 | +rid = {r: i for i, r in enumerate(regions)} | |
| 15 | +df["rid"] = df["region"].map(rid).astype(int) | |
| 16 | +idx = np.load(OUT / "index_asof.npy") | |
| 17 | + | |
| 18 | +FEATS = ["ln_floor", "ln_land", "age", "age2", "lien1", "lien2", "genre1", "genre2", "genre3", "etages", "neuf"] | |
| 19 | +def feats(d): | |
| 20 | + return pd.DataFrame({ | |
| 21 | + "ln_floor": d["ln_floor"], "ln_land": d["ln_land"], "age": d["age"], "age2": d["age"] ** 2 / 100, | |
| 22 | + "lien1": (d["lien"] == 1).astype(float), "lien2": (d["lien"] >= 2).astype(float), | |
| 23 | + "genre1": (d["genre"] == 1).astype(float), "genre2": (d["genre"] == 2).astype(float), "genre3": (d["genre"] == 3).astype(float), | |
| 24 | + "etages": d["etages"].astype(float), "neuf": d["neuf"].astype(float)}, index=d.index) | |
| 25 | + | |
| 26 | +def time_adjusted(d, m_asof): | |
| 27 | + """ln P ramené au niveau du mois m_asof-1 avec l'indice connu au mois m_asof.""" | |
| 28 | + lvl = idx[m_asof - M_EVAL0] | |
| 29 | + return d["ln_price"].to_numpy() - lvl[d["rid"].to_numpy(), d["month"].to_numpy()] + lvl[d["rid"].to_numpy(), m_asof - 1] | |
| 30 | + | |
| 31 | +def ols(X, y): | |
| 32 | + A = np.column_stack([np.ones(len(X)), X]) | |
| 33 | + b, *_ = np.linalg.lstsq(A, y, rcond=None) | |
| 34 | + return b | |
| 35 | + | |
| 36 | +ASOF = list(range(M_EVAL0, int(df["month"].max()) + 1, 3)) # ré-estimation trimestrielle (2024-01, 2024-04, …) | |
| 37 | +save_json({"asof": ASOF}, "asof.json") | |
| 38 | +TODO = [int(a) for a in os.environ["CV_ASOF"].split(",")] if os.environ.get("CV_ASOF") else ASOF # sous-ensemble (répartition sur plusieurs nœuds) | |
| 39 | +for m_asof in TODO: | |
| 40 | + period = str(m_asof); start = f"{2021 + m_asof // 12}-{m_asof % 12 + 1:02d}-01" | |
| 41 | + tr = df[df["month"] < m_asof].copy() | |
| 42 | + y = time_adjusted(tr, m_asof) | |
| 43 | + F = feats(tr) | |
| 44 | + log(f"[{period}] entraînement sur {len(tr):,} ventes < {start} (indice connu au mois {m_asof})") | |
| 45 | + # --- grille hédonique globale : estimateur « within » (effets fixes de municipalité), pour que les β | |
| 46 | + # mesurent des différences entre biens voisins et non entre marchés | |
| 47 | + def within(frame, cols, by): | |
| 48 | + return frame[cols] - frame.groupby(by)[cols].transform("mean") | |
| 49 | + # effets mensuels résiduels (base = dernier mois connu m_asof-1) : l'indice régional ne retire pas toute la | |
| 50 | + # tendance commune ; sans eux, les modèles globaux sous-estiment les ventes récentes | |
| 51 | + months_tr = sorted(tr["month"].unique()); MCOLS = [f"m{m}" for m in months_tr if m != m_asof - 1] | |
| 52 | + Fm = pd.concat([F] + [pd.Series((tr["month"] == m).astype(float), index=tr.index, name=f"m{m}") for m in months_tr if m != m_asof - 1], axis=1) | |
| 53 | + Fw = within(pd.concat([Fm, tr[["code_mun"]]], axis=1), FEATS + MCOLS, "code_mun") | |
| 54 | + yw = y - pd.Series(y, index=tr.index).groupby(tr["code_mun"]).transform("mean").to_numpy() | |
| 55 | + bw_all, *_ = np.linalg.lstsq(Fw.to_numpy(), yw, rcond=None) | |
| 56 | + bw = bw_all[:len(FEATS)]; bm = bw_all[len(FEATS):] | |
| 57 | + beta_global = dict(zip(FEATS, bw.tolist())) | |
| 58 | + xb = F.to_numpy() @ bw + Fm[MCOLS].to_numpy() @ bm | |
| 59 | + resid = y - xb | |
| 60 | + r2w = 1 - np.var(yw - Fw.to_numpy() @ bw_all) / np.var(yw) | |
| 61 | + Fw = Fw[FEATS] | |
| 62 | + log(f" hédonique within-municipalité : R² within = {r2w:.3f} ; β = { {k: round(v, 3) for k, v in beta_global.items()} }") | |
| 63 | + # --- grille régionale (within-municipalité) rétrécie vers la globale (n/(n+300)) | |
| 64 | + beta_region = {} | |
| 65 | + for r in regions: | |
| 66 | + sel = (tr["region"] == r).to_numpy() | |
| 67 | + if sel.sum() < 100: beta_region[r] = beta_global; continue | |
| 68 | + b, *_ = np.linalg.lstsq(Fw.to_numpy()[sel], yw[sel], rcond=None) | |
| 69 | + w = sel.sum() / (sel.sum() + 300.0) | |
| 70 | + beta_region[r] = {k: float(w * b[j] + (1 - w) * beta_global[k]) for j, k in enumerate(FEATS)} | |
| 71 | + # --- M10 : hédonique directe = Xβ + effet de région + MRC + municipalité + unité de voisinage (rétrécis en cascade) | |
| 72 | + reg_eff = pd.Series(resid, index=tr.index).groupby(tr["region"]).mean().to_dict() | |
| 73 | + tr["res_g"] = resid - tr["region"].map(reg_eff).to_numpy() | |
| 74 | + mrc_eff = tr.groupby("mrc")["res_g"].agg(["sum", "count"]) | |
| 75 | + mrc_eff = (mrc_eff["sum"] / (mrc_eff["count"] + 30)).to_dict() | |
| 76 | + tr["res_m"] = tr["res_g"] - tr["mrc"].map(mrc_eff).fillna(0) | |
| 77 | + mun_eff = tr.groupby("code_mun")["res_m"].agg(["sum", "count"]) | |
| 78 | + mun_eff = (mun_eff["sum"] / (mun_eff["count"] + 20)).to_dict() | |
| 79 | + tr["res_u"] = tr["res_m"] - tr["code_mun"].map(mun_eff).fillna(0) | |
| 80 | + # correction médiane : la MCO vise la moyenne du log ; la distribution est asymétrique (queue basse), on recentre sur la médiane | |
| 81 | + uv_tmp = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].transform(lambda v: v.sum() / (len(v) + 10)) | |
| 82 | + median_shift = float(np.median(tr["res_u"] - uv_tmp)) | |
| 83 | + uv = tr.groupby(["code_mun", "unite_voisinage"])["res_u"].agg(["sum", "count"]) | |
| 84 | + uv_eff = {f"{a}|{b}": v for (a, b), v in (uv["sum"] / (uv["count"] + 10)).items()} | |
| 85 | + # --- LightGBM (prix ajusté dans le temps) avec rôle et sans rôle | |
| 86 | + def gbm_frame(d): | |
| 87 | + G = feats(d) | |
| 88 | + G["lat"] = d["lat"]; G["lng"] = d["lng"]; G["x_km"] = d["x_km"]; G["y_km"] = d["y_km"] | |
| 89 | + G["rid"] = d["rid"]; G["btype"] = d["btype"] | |
| 90 | + G["ln_role"] = np.log(d["role_vente"]); G["part_terrain"] = d["part_terrain"]; G["role_vintage"] = d["role_vintage"] | |
| 91 | + G["month"] = np.minimum(d["month"], m_asof - 1).astype(float) # tendance résiduelle ; hors échantillon = dernier mois connu | |
| 92 | + return G | |
| 93 | + G = gbm_frame(tr) | |
| 94 | + params = dict(objective="regression_l1", learning_rate=0.05, num_leaves=127, min_data_in_leaf=40, feature_fraction=0.8, | |
| 95 | + bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=int(os.environ.get("CV_THREADS", 20))) | |
| 96 | + models = {} | |
| 97 | + for name, cols in [("gbm", [c for c in G.columns if c not in ("ln_role", "part_terrain", "role_vintage")]), | |
| 98 | + ("gbm_role", list(G.columns))]: | |
| 99 | + ds = lgb.Dataset(G[cols], y, categorical_feature=["rid", "btype"] if "btype" in cols else ["rid"]) | |
| 100 | + # validation interne : derniers 10 % chronologiques pour fixer le nombre d'arbres | |
| 101 | + n = len(G); cut = int(n * 0.9) | |
| 102 | + dtr = lgb.Dataset(G[cols].iloc[:cut], y[:cut], categorical_feature=["rid", "btype"]) | |
| 103 | + dva = lgb.Dataset(G[cols].iloc[cut:], y[cut:], reference=dtr) | |
| 104 | + bst = lgb.train(params, dtr, num_boost_round=3000, valid_sets=[dva], callbacks=[lgb.early_stopping(100, verbose=False)]) | |
| 105 | + best = bst.best_iteration | |
| 106 | + bst = lgb.train(params, ds, num_boost_round=best) | |
| 107 | + models[name] = bst | |
| 108 | + Gall = gbm_frame(df); Gall["month"] = float(m_asof - 1) # f évalué au niveau du dernier mois connu pour toutes les ventes | |
| 109 | + pred_all = bst.predict(Gall[cols]) | |
| 110 | + np.save(OUT / f"{name}_pred_{period}.npy", pred_all.astype(np.float32)) | |
| 111 | + rr = y - bst.predict(G[cols]); r2 = 1 - np.var(rr) / np.var(y) | |
| 112 | + log(f" {name}: {best} arbres, R² (in-sample) = {r2:.3f}, résidu médian = {np.median(rr) * 100:.2f} %") | |
| 113 | + bst.save_model(str(OUT / f"{name}_{period}.txt")) | |
| 114 | + save_json({"period": period, "m_asof": m_asof, "start": start, "n_train": len(tr), "feats": FEATS, | |
| 115 | + "beta_global": beta_global, "intercept_global": 0.0, "region_fe": reg_eff, | |
| 116 | + "beta_region": beta_region, "mrc_eff": mrc_eff, "mun_eff": mun_eff, "uv_eff": uv_eff, | |
| 117 | + "r2_within": float(r2w), "sd_resid_u": float(tr["res_u"].std()), "median_shift": median_shift}, | |
| 118 | + f"models_{period}.json") | |
| 119 | +log("modèles terminés") | |
added
M4M64bomparables-validation/pipeline/04_backtest.py
+193 −0
@@ -0,0 +1,193 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04 — Moteur de rétro-évaluation. | |
| 3 | +Pour chaque vente cible (2024 = validation, 2025-2026 = test), le prix est masqué ; seules les ventes | |
| 4 | +conclues STRICTEMENT AVANT la date cible (et l'indice/les modèles connus à cette date) sont utilisées. | |
| 5 | +On évalue d'un coup toutes les configurations (sélection × rayon × fenêtre × nombre × ajustement × | |
| 6 | +pondération × réconciliation) sur exactement les mêmes cibles. | |
| 7 | +Sorties : targets.parquet, configs_E1.csv/est_E1.npy (surface k,d,T), configs_E2.csv/est_E2.npy/sd_E2.npy | |
| 8 | +(factoriel sélection × ajustement × pondération × réconciliation), n_avail.npy.""" | |
| 9 | +import os | |
| 10 | +for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"): | |
| 11 | + os.environ[_v] = "1" # un seul fil BLAS par processus (fork + BLAS multi-fils = interblocage sur macOS) | |
| 12 | +import multiprocessing as mp, time | |
| 13 | +from scipy.spatial import cKDTree | |
| 14 | +from common import * | |
| 15 | + | |
| 16 | +df = load_parquet("sales.parquet") | |
| 17 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"] | |
| 18 | +rid_map = {r: i for i, r in enumerate(regions)} | |
| 19 | +IDX = np.load(OUT / "index_asof.npy") | |
| 20 | +ASOF = load_json("asof.json")["asof"] | |
| 21 | +MODELS = {a: load_json(f"models_{a}.json") for a in ASOF} | |
| 22 | +FEATS = MODELS[ASOF[0]]["feats"] | |
| 23 | +GBM = {a: np.load(OUT / f"gbm_pred_{a}.npy") for a in ASOF} | |
| 24 | + | |
| 25 | +# ----------------------------------------------------------------- tableaux globaux (partagés par fork) | |
| 26 | +N = len(df) | |
| 27 | +DAY = df["day"].to_numpy(np.int32); MONTH = df["month"].to_numpy(np.int32) | |
| 28 | +RID = df["region"].map(rid_map).to_numpy(np.int32) | |
| 29 | +X = df["x_km"].to_numpy(np.float64); Y = df["y_km"].to_numpy(np.float64) | |
| 30 | +LNP = df["ln_price"].to_numpy(np.float64); P = df["amount"].to_numpy(np.float64) | |
| 31 | +FLOOR = df["floor_area"].to_numpy(np.float64); LAND = df["land_area"].to_numpy(np.float64) | |
| 32 | +AGE = df["age"].to_numpy(np.float64); LIEN = df["lien"].to_numpy(np.int8); ETAGES = df["etages"].to_numpy(np.float64) | |
| 33 | +GENRE = df["genre"].to_numpy(np.int8) | |
| 34 | +PROV = pd.factorize(df["id_provinc"])[0].astype(np.int32) | |
| 35 | +MUN = pd.factorize(df["code_mun"])[0].astype(np.int32) | |
| 36 | +VINT = df["role_vintage"].to_numpy(np.int8); LNROLE = np.log(df["role_vente"].to_numpy(np.float64)) | |
| 37 | +PART = df["part_terrain"].to_numpy(np.float64) | |
| 38 | +NEUF = df["neuf"].to_numpy(np.float64) | |
| 39 | +# matrice des caractéristiques hédoniques (même ordre que FEATS) | |
| 40 | +XF = np.column_stack([np.log(FLOOR), np.log(LAND), AGE, AGE ** 2 / 100, (LIEN == 1), (LIEN >= 2), | |
| 41 | + (GENRE == 1), (GENRE == 2), (GENRE == 3), ETAGES, NEUF]).astype(np.float64) | |
| 42 | +BETA_G = {a: np.array([MODELS[a]["beta_global"][f] for f in FEATS]) for a in ASOF} | |
| 43 | +BETA_R = {a: np.array([[MODELS[a]["beta_region"][r][f] for f in FEATS] for r in regions]) for a in ASOF} | |
| 44 | +ASOF_ARR = np.array(ASOF) | |
| 45 | +TREE = cKDTree(np.column_stack([X, Y])) | |
| 46 | + | |
| 47 | +D_MAX, T_MAX_DAYS = 25.0, 731 | |
| 48 | +KS = [1, 3, 5, 7, 10, 15, 20, 30, 50] | |
| 49 | +DS = [0.5, 1, 2, 5, 10, 25] | |
| 50 | +TS = [3, 6, 12, 24] | |
| 51 | +SELS = ["geo", "sim", "mix", "adj"] | |
| 52 | +ADJS = ["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"] | |
| 53 | +WRS = [("equal", "median"), ("equal", "mean"), ("dist", "median"), ("dist", "mean"), ("rec", "median"), ("rec", "mean"), | |
| 54 | + ("sim", "median"), ("sim", "mean"), ("all", "median"), ("all", "mean")] | |
| 55 | +ANCHORS = [(3, 2, 6), (5, 5, 12), (10, 10, 24)] | |
| 56 | +E1 = [dict(sel=s, d=d, T=T, k=k, adj=a, w="equal", recon="median") | |
| 57 | + for s in ("geo", "sim", "mix") for d in DS for T in TS for k in KS for a in ("none", "hedo")] | |
| 58 | +E2 = [dict(sel=s, d=d, T=T, k=k, adj=a, w=w, recon=r) | |
| 59 | + for (k, d, T) in ANCHORS for s in SELS for a in ADJS for (w, r) in WRS] | |
| 60 | +LOCAL_COLS = [0, 1, 2, 4, 5, 9] # ln_floor, ln_land, age, lien1, lien2, etages | |
| 61 | +LAMBDA_LOCAL = 50.0 | |
| 62 | + | |
| 63 | +def wmedian(v, w): | |
| 64 | + o = np.argsort(v); v = v[o]; w = w[o] | |
| 65 | + c = np.cumsum(w); return v[np.searchsorted(c, 0.5 * c[-1])] | |
| 66 | + | |
| 67 | +def recon(vals, w, how): | |
| 68 | + ok = ~np.isnan(vals) | |
| 69 | + if not ok.any(): return np.nan, np.nan | |
| 70 | + v = vals[ok]; ww = w[ok] | |
| 71 | + sd = v.std() if len(v) > 1 else np.nan | |
| 72 | + if how == "median": | |
| 73 | + return (np.median(v) if ww.min() == ww.max() else wmedian(v, ww)), sd | |
| 74 | + return np.log(np.sum(ww * np.exp(v)) / ww.sum()), sd | |
| 75 | + | |
| 76 | +def evaluate(i): | |
| 77 | + """Toutes les configurations pour la cible i. Renvoie (est_E1, est_E2, sd_E2, n_avail, info).""" | |
| 78 | + period = int(ASOF_ARR[ASOF_ARR <= MONTH[i]].max()) # dernier trimestre de ré-estimation ≤ mois de la cible | |
| 79 | + e1 = np.full(len(E1), np.nan, np.float32); e2 = np.full(len(E2), np.nan, np.float32) | |
| 80 | + s2 = np.full(len(E2), np.nan, np.float32); nav = np.zeros(len(DS) * len(TS), np.int32) | |
| 81 | + cand = np.asarray(TREE.query_ball_point((X[i], Y[i]), D_MAX), dtype=np.int64) | |
| 82 | + keep = (DAY[cand] < DAY[i]) & (DAY[cand] >= DAY[i] - T_MAX_DAYS) & (PROV[cand] != PROV[i]) | |
| 83 | + cand = cand[keep] | |
| 84 | + info = dict(idx=int(i), asof=period, n_cand=int(len(cand))) | |
| 85 | + if len(cand) == 0: return e1, e2, s2, nav, info | |
| 86 | + dist = np.hypot(X[cand] - X[i], Y[cand] - Y[i]) | |
| 87 | + mago = (DAY[i] - DAY[cand]) / 30.4375 | |
| 88 | + m_eval = MONTH[i]; lvl = IDX[m_eval - M_EVAL0] | |
| 89 | + tadj = np.where(MONTH[cand] == m_eval, 0.0, lvl[RID[cand], m_eval - 1] - lvl[RID[cand], MONTH[cand]]) | |
| 90 | + tadj = np.nan_to_num(tadj, nan=0.0) | |
| 91 | + lnT = LNP[cand] + tadj | |
| 92 | + dX = XF[i] - XF[cand] # cible − comparable | |
| 93 | + # --- ajustements | |
| 94 | + bg = BETA_G[period]; br = BETA_R[period][RID[i]] | |
| 95 | + adj = {} | |
| 96 | + adj["raw"] = LNP[cand] | |
| 97 | + adj["none"] = lnT | |
| 98 | + Pt = np.exp(lnT) | |
| 99 | + d_floor = (FLOOR[i] - FLOOR[cand]) * 0.5 * Pt / FLOOR[cand] | |
| 100 | + d_land = np.clip((LAND[i] - LAND[cand]) * 0.5 * Pt * PART[cand] / LAND[cand], -0.15 * Pt, 0.15 * Pt) | |
| 101 | + d_age = np.clip(-0.005 * Pt * (AGE[i] - AGE[cand]), -0.2 * Pt, 0.2 * Pt) | |
| 102 | + d_lien = np.where((LIEN[i] == 0) & (LIEN[cand] > 0), 0.05 * Pt, np.where((LIEN[i] > 0) & (LIEN[cand] == 0), -0.05 * Pt, 0.0)) | |
| 103 | + tot = np.maximum(d_floor + d_land + d_age + d_lien, -0.6 * Pt) | |
| 104 | + adj["conv"] = np.log(Pt + tot) | |
| 105 | + gross_conv = (np.abs(d_floor) + np.abs(d_land) + np.abs(d_age) + np.abs(d_lien)) / Pt | |
| 106 | + adj["hedo"] = lnT + dX @ bg | |
| 107 | + gross_hedo = np.abs(dX * bg).sum(axis=1) | |
| 108 | + adj["hedo_reg"] = lnT + dX @ br | |
| 109 | + loc = dist <= 10.0 | |
| 110 | + if loc.sum() >= 40: | |
| 111 | + Xl = XF[cand][loc][:, LOCAL_COLS]; yl = lnT[loc] | |
| 112 | + b0 = br[LOCAL_COLS] | |
| 113 | + Xc = Xl - Xl.mean(axis=0); yc = yl - yl.mean() | |
| 114 | + A = Xc.T @ Xc + LAMBDA_LOCAL * np.eye(len(LOCAL_COLS)) | |
| 115 | + bl = np.linalg.solve(A, Xc.T @ yc + LAMBDA_LOCAL * b0) | |
| 116 | + adj["local"] = lnT + dX[:, LOCAL_COLS] @ bl | |
| 117 | + else: | |
| 118 | + adj["local"] = adj["hedo_reg"] | |
| 119 | + f = GBM[period] | |
| 120 | + adj["gbm"] = lnT + (f[i] - f[cand]) | |
| 121 | + same = (MUN[cand] == MUN[i]) & (VINT[cand] == VINT[i]) | |
| 122 | + adj["role"] = np.where(same, LNROLE[i] + (lnT - LNROLE[cand]), np.nan) | |
| 123 | + # --- scores de sélection | |
| 124 | + dissim = np.sqrt((dX[:, 0] / 0.25) ** 2 + (dX[:, 1] / 0.6) ** 2 + (dX[:, 2] / 15) ** 2 | |
| 125 | + + (LIEN[cand] != LIEN[i]) * 1.0 + (ETAGES[cand] != ETAGES[i]) * 0.5) | |
| 126 | + score = {"geo": dist, "sim": dissim + dist / 25.0, "mix": dissim + dist / 2.0, "adj": gross_hedo + dist / 25.0} | |
| 127 | + # --- pondérations | |
| 128 | + wts = {"equal": np.ones(len(cand)), "dist": 1.0 / (0.25 + dist), "rec": np.exp(-mago / 6.0), "sim": 1.0 / (0.1 + gross_hedo)} | |
| 129 | + wts["all"] = wts["dist"] * wts["rec"] * wts["sim"] | |
| 130 | + # --- masques (d, T) et classements | |
| 131 | + top = {} | |
| 132 | + for di, d in enumerate(DS): | |
| 133 | + for ti, T in enumerate(TS): | |
| 134 | + m = (dist <= d) & (mago <= T) | |
| 135 | + nav[di * len(TS) + ti] = int(m.sum()) | |
| 136 | + if not m.any(): continue | |
| 137 | + ids = np.flatnonzero(m) | |
| 138 | + for s in SELS: | |
| 139 | + sc = score[s][ids] | |
| 140 | + if len(ids) > 50: | |
| 141 | + part = np.argpartition(sc, 50)[:50]; part = part[np.argsort(sc[part])] | |
| 142 | + else: | |
| 143 | + part = np.argsort(sc) | |
| 144 | + top[(s, d, T)] = ids[part] | |
| 145 | + # --- E1 | |
| 146 | + for c, cfg in enumerate(E1): | |
| 147 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 148 | + if t is None: continue | |
| 149 | + sub = t[:cfg["k"]] | |
| 150 | + e1[c], _ = recon(adj[cfg["adj"]][sub], wts["equal"][sub], "median") | |
| 151 | + # --- E2 | |
| 152 | + for c, cfg in enumerate(E2): | |
| 153 | + t = top.get((cfg["sel"], cfg["d"], cfg["T"])) | |
| 154 | + if t is None: continue | |
| 155 | + sub = t[:cfg["k"]] | |
| 156 | + e2[c], s2[c] = recon(adj[cfg["adj"]][sub], wts[cfg["w"]][sub], cfg["recon"]) | |
| 157 | + return e1, e2, s2, nav, info | |
| 158 | + | |
| 159 | +def worker(chunk): | |
| 160 | + return [evaluate(i) for i in chunk] | |
| 161 | + | |
| 162 | +if __name__ == "__main__": | |
| 163 | + targets = df.index[df["period"].isin(["val", "test"])].to_numpy() | |
| 164 | + if os.environ.get("CV_SAMPLE"): | |
| 165 | + rng = np.random.default_rng(1); targets = np.sort(rng.choice(targets, int(os.environ["CV_SAMPLE"]), replace=False)) | |
| 166 | + SUF = "" | |
| 167 | + if os.environ.get("CV_SHARD"): # répartition sur plusieurs nœuds : CV_SHARD=i/N → cibles i, i+N, i+2N, … | |
| 168 | + i_sh, n_sh = map(int, os.environ["CV_SHARD"].split("/")); targets = targets[i_sh::n_sh]; SUF = f".shard{i_sh}of{n_sh}" | |
| 169 | + log(f"{len(targets):,} cibles ; {len(E1)} configurations E1, {len(E2)} E2 ; {N:,} ventes dans le bassin") | |
| 170 | + pd.DataFrame(E1).to_csv(OUT / "configs_E1.csv", index_label="cfg") | |
| 171 | + pd.DataFrame(E2).to_csv(OUT / "configs_E2.csv", index_label="cfg") | |
| 172 | + est1 = np.full((len(targets), len(E1)), np.nan, np.float32) | |
| 173 | + est2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 174 | + sd2 = np.full((len(targets), len(E2)), np.nan, np.float32) | |
| 175 | + nav = np.zeros((len(targets), len(DS) * len(TS)), np.int32) | |
| 176 | + infos = [] | |
| 177 | + chunks = [targets[j:j + 400] for j in range(0, len(targets), 400)] | |
| 178 | + t0 = time.time(); done = 0 | |
| 179 | + with mp.get_context(os.environ.get("CV_MP", "fork")).Pool(int(os.environ.get("CV_PROCS", 20))) as pool: | |
| 180 | + for res in pool.imap(worker, chunks): | |
| 181 | + for (e1, e2, s2, nv, info) in res: | |
| 182 | + est1[done] = e1; est2[done] = e2; sd2[done] = s2; nav[done] = nv; infos.append(info); done += 1 | |
| 183 | + if done % 20000 < 400: | |
| 184 | + log(f" {done:,}/{len(targets):,} cibles ({time.time() - t0:.0f} s)") | |
| 185 | + np.save(OUT / f"est_E1{SUF}.npy", est1); np.save(OUT / f"est_E2{SUF}.npy", est2); np.save(OUT / f"sd_E2{SUF}.npy", sd2); np.save(OUT / f"n_avail{SUF}.npy", nav) | |
| 186 | + T = df.loc[targets, ["idx", "id", "date", "period", "amount", "ln_price", "region", "region_nom", "mrc", "mrc_nom", "code_mun", | |
| 187 | + "mun_nom", "unite_voisinage", "floor_area", "land_area", "age", "lien", "etages", "role_vente", "month", "lat", "lng"]].reset_index(drop=True) | |
| 188 | + T["n_cand"] = [x["n_cand"] for x in infos]; T["asof"] = [x["asof"] for x in infos] | |
| 189 | + T["n_2km_12m"] = nav[:, DS.index(2) * len(TS) + TS.index(12)] | |
| 190 | + T["n_5km_12m"] = nav[:, DS.index(5) * len(TS) + TS.index(12)] | |
| 191 | + save_parquet(T, f"targets{SUF}.parquet") | |
| 192 | + pd.DataFrame({"DS": [DS], "TS": [TS], "KS": [KS]}).to_json(OUT / "grid_meta.json") | |
| 193 | + log(f"rétro-évaluation terminée en {time.time() - t0:.0f} s") | |
added
M4M64bomparables-validation/pipeline/04b_gather.py
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04b — Rassemble les fragments (shards) de la rétro-évaluation calculés sur plusieurs nœuds, dans l'ordre des cibles.""" | |
| 3 | +import glob | |
| 4 | +from common import * | |
| 5 | +sh = sorted(glob.glob(str(OUT / "targets.shard*.parquet"))) | |
| 6 | +log(f"{len(sh)} fragments") | |
| 7 | +Ts = [pd.read_parquet(p) for p in sh] | |
| 8 | +parts = {} | |
| 9 | +for name in ["est_E1", "est_E2", "sd_E2", "n_avail"]: | |
| 10 | + parts[name] = [np.load(p.replace("targets.", f"{name}.").replace(".parquet", ".npy")) for p in sh] | |
| 11 | +T = pd.concat(Ts, ignore_index=True); order = np.argsort(T["idx"].to_numpy(), kind="stable") | |
| 12 | +T = T.iloc[order].reset_index(drop=True); save_parquet(T, "targets.parquet") | |
| 13 | +for name, arrs in parts.items(): | |
| 14 | + A = np.concatenate(arrs, axis=0)[order]; np.save(OUT / f"{name}.npy", A); log(f" {name}: {A.shape}") | |
| 15 | +log("rassemblement terminé") | |
added
M4M64bomparables-validation/pipeline/05_analyze.py
+358 −0
@@ -0,0 +1,358 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""05 — Analyse : métriques par configuration (validation 2024 → choix ; test 2025-26 → verdict), | |
| 3 | +surface Erreur(k, d, t), effets des facteurs (sélection, ajustement, pondération, réconciliation), | |
| 4 | +compétition de modèles M1-M13, hypothèses H1-H5, strates (région, densité, prix), intervalles, | |
| 5 | +règles conditionnelles. Sorties : tables/*.csv, results.json, fig/*.pdf""" | |
| 6 | +from scipy.optimize import nnls | |
| 7 | +import matplotlib; matplotlib.use("Agg") | |
| 8 | +import matplotlib.pyplot as plt | |
| 9 | +from matplotlib import rcParams | |
| 10 | +from common import * | |
| 11 | + | |
| 12 | +# ---------------------------------------------------------------- style (palette de référence, marques fines) | |
| 13 | +PAL = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"] | |
| 14 | +SEQ = ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#184f95", "#0d366b"] | |
| 15 | +rcParams.update({"font.family": "Helvetica Neue", "font.size": 9, "axes.edgecolor": "#c3c2b7", "axes.linewidth": 0.6, | |
| 16 | + "axes.grid": True, "grid.color": "#e8e7e3", "grid.linewidth": 0.5, "axes.spines.top": False, | |
| 17 | + "axes.spines.right": False, "legend.frameon": False, "figure.dpi": 150, "savefig.bbox": "tight", | |
| 18 | + "axes.titleweight": "bold", "axes.titlesize": 10, "axes.labelcolor": "#52514e", "xtick.color": "#52514e", | |
| 19 | + "ytick.color": "#52514e", "figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb"}) | |
| 20 | +from matplotlib.colors import LinearSegmentedColormap | |
| 21 | +CMAP = LinearSegmentedColormap.from_list("seqblue", SEQ) | |
| 22 | +TAB = OUT / "tables"; TAB.mkdir(exist_ok=True); FIG = OUT / "fig" | |
| 23 | +def savefig(fig, name): fig.savefig(FIG / f"{name}.pdf"); fig.savefig(FIG / f"{name}.png"); plt.close(fig) | |
| 24 | + | |
| 25 | +# ---------------------------------------------------------------- chargement | |
| 26 | +T = load_parquet("targets.parquet"); S = load_parquet("sales.parquet").set_index("idx") | |
| 27 | +E1 = pd.read_csv(OUT / "configs_E1.csv"); E2 = pd.read_csv(OUT / "configs_E2.csv") | |
| 28 | +est1 = np.load(OUT / "est_E1.npy"); est2 = np.load(OUT / "est_E2.npy"); sd2 = np.load(OUT / "sd_E2.npy"); nav = np.load(OUT / "n_avail.npy") | |
| 29 | +meta = load_json("index_meta.json"); regions = meta["regions"]; M_EVAL0 = meta["M_EVAL0"]; IDX = np.load(OUT / "index_asof.npy") | |
| 30 | +rid = {r: i for i, r in enumerate(regions)} | |
| 31 | +lnP = T["ln_price"].to_numpy(np.float32) | |
| 32 | +VAL = (T["period"] == "val").to_numpy(); TEST = (T["period"] == "test").to_numpy() | |
| 33 | +log(f"{len(T):,} cibles (val {VAL.sum():,}, test {TEST.sum():,}) ; E1 {est1.shape[1]} ; E2 {est2.shape[1]}") | |
| 34 | + | |
| 35 | +def err_of(est): return (np.exp(est - lnP[:, None]) - 1).astype(np.float32) | |
| 36 | +def metrics(err, sel): | |
| 37 | + e = err[sel]; a = np.abs(e) | |
| 38 | + ok = ~np.isnan(a); n = ok.sum(axis=0) | |
| 39 | + with np.errstate(all="ignore"): | |
| 40 | + return pd.DataFrame({"mdape": np.nanmedian(a, axis=0) * 100, "mape": np.nanmean(a, axis=0) * 100, | |
| 41 | + "biais": np.nanmedian(e, axis=0) * 100, "biais_moy": np.nanmean(e, axis=0) * 100, | |
| 42 | + "p5": np.nanmean(a <= 0.05, axis=0) * 100, "p10": np.nanmean(a <= 0.10, axis=0) * 100, | |
| 43 | + "p20": np.nanmean(a <= 0.20, axis=0) * 100, "couverture": n / sel.sum() * 100, "n": n}) | |
| 44 | +err1 = err_of(est1); err2 = err_of(est2) | |
| 45 | +del est1 | |
| 46 | +res = {} | |
| 47 | + | |
| 48 | +# ---------------------------------------------------------------- densité & strates | |
| 49 | +T["dens"] = pd.cut(T["n_2km_12m"], [-1, 4, 19, 49, 199, 10 ** 9], labels=["< 5", "5-19", "20-49", "50-199", "≥ 200"]) | |
| 50 | +T["prix_q"] = pd.qcut(T["amount"], 5, labels=["Q1 (bas)", "Q2", "Q3", "Q4", "Q5 (haut)"]) | |
| 51 | +T["trimestre"] = pd.to_datetime(T["date"]).dt.to_period("Q").astype(str) | |
| 52 | +T["age_cl"] = pd.cut(T["age"], [-1, 3, 15, 40, 70, 1000], labels=["≤ 3 ans", "4-15", "16-40", "41-70", "> 70"]) | |
| 53 | +T["cluster"] = pd.factorize(T["code_mun"].astype(str) + "|" + T["trimestre"])[0] | |
| 54 | + | |
| 55 | +# ---------------------------------------------------------------- E1 : surface Erreur(k, d, t) | |
| 56 | +mv = metrics(err1, VAL).add_suffix("_val"); mt = metrics(err1, TEST).add_suffix("_test") | |
| 57 | +surf = pd.concat([E1, mv, mt], axis=1) | |
| 58 | +surf.to_csv(TAB / "surface_E1.csv", index=False) | |
| 59 | +ok = surf["couverture_val"] >= 97 | |
| 60 | +best_e1 = surf[ok].sort_values("mdape_val").iloc[0] | |
| 61 | +res["E1_best_val"] = best_e1.to_dict() | |
| 62 | +log(f"E1 meilleure (val, couverture ≥ 97 %) : {best_e1[['sel','d','T','k','adj']].to_dict()} → val {best_e1.mdape_val:.2f} %, test {best_e1.mdape_test:.2f} %") | |
| 63 | +for (s, a), g in surf[ok].groupby(["sel", "adj"]): | |
| 64 | + b = g.sort_values("mdape_val").iloc[0] | |
| 65 | + res.setdefault("E1_best_by_sel_adj", {})[f"{s}|{a}"] = dict(k=int(b.k), d=float(b.d), T=int(b["T"]), mdape_val=float(b.mdape_val), mdape_test=float(b.mdape_test), couverture_test=float(b.couverture_test)) | |
| 66 | + | |
| 67 | +# figure : MdAPE vs k selon d (T = 12, geo, hedo, test) | |
| 68 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.4), sharey=True) | |
| 69 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 70 | + for j, d in enumerate([0.5, 1, 2, 5, 10, 25]): | |
| 71 | + g = surf[(surf.sel == "geo") & (surf["T"] == 12) & (surf.adj == a) & (surf.d == d)].sort_values("k") | |
| 72 | + ax.plot(g.k, g.mdape_test, color=PAL[j], lw=2, marker="o", ms=4, label=f"d ≤ {d} km") | |
| 73 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]) | |
| 74 | + ax.set_xlabel("nombre de comparables k"); ax.set_title({"none": "sans ajustement (temps seulement)", "hedo": "grille hédonique"}[a]) | |
| 75 | +axes[0].set_ylabel("MdAPE, test 2025-26 (%)"); axes[1].legend(ncol=2, fontsize=8) | |
| 76 | +savefig(fig, "f1_mdape_vs_k") | |
| 77 | +# figure : carte d × T au meilleur k (geo, hedo) | |
| 78 | +fig, axes = plt.subplots(1, 2, figsize=(8, 3.2)) | |
| 79 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 80 | + g = surf[(surf.sel == "geo") & (surf.adj == a)].groupby(["d", "T"])["mdape_test"].min().unstack("T") | |
| 81 | + im = ax.imshow(g.to_numpy(), cmap=CMAP, aspect="auto") | |
| 82 | + ax.set_xticks(range(len(g.columns))); ax.set_xticklabels([f"{t} mois" for t in g.columns]); ax.set_yticks(range(len(g.index))); ax.set_yticklabels([f"{d} km" for d in g.index]) | |
| 83 | + for i in range(g.shape[0]): | |
| 84 | + for j in range(g.shape[1]): | |
| 85 | + v = g.iloc[i, j]; ax.text(j, i, f"{v:.1f}", ha="center", va="center", fontsize=7.5, color="white" if v < np.nanpercentile(g.to_numpy(), 40) else "#0b0b0b") | |
| 86 | + ax.grid(False); ax.set_title({"none": "sans ajustement", "hedo": "grille hédonique"}[a] + " — MdAPE au meilleur k") | |
| 87 | +savefig(fig, "f2_surface_d_T") | |
| 88 | + | |
| 89 | +# ---------------------------------------------------------------- E2 : facteurs | |
| 90 | +mv2 = metrics(err2, VAL).add_suffix("_val"); mt2 = metrics(err2, TEST).add_suffix("_test") | |
| 91 | +fac = pd.concat([E2, mv2, mt2], axis=1); fac.to_csv(TAB / "facteurs_E2.csv", index=False) | |
| 92 | +anchor = (fac.k == 5) & (fac.d == 5) & (fac["T"] == 12) | |
| 93 | +base = anchor & (fac.sel == "geo") & (fac.w == "equal") & (fac.recon == "median") | |
| 94 | +res["E2_adjust_anchor"] = fac[base].set_index("adj")[["mdape_val", "mdape_test", "mape_test", "biais_test", "p10_test", "couverture_test"]].to_dict("index") | |
| 95 | +res["E2_sel_anchor"] = fac[anchor & (fac.adj == "hedo") & (fac.w == "equal") & (fac.recon == "median")].set_index("sel")[["mdape_val", "mdape_test", "p10_test"]].to_dict("index") | |
| 96 | +res["E2_wr_anchor"] = {f"{r.w}|{r.recon}": dict(mdape_val=r.mdape_val, mdape_test=r.mdape_test, p10_test=r.p10_test) | |
| 97 | + for r in fac[anchor & (fac.adj == "hedo") & (fac.sel == "geo")].itertuples()} | |
| 98 | +# effets marginaux moyens (moyenne de MdAPE test sur toutes les autres modalités) | |
| 99 | +marg = {} | |
| 100 | +for f in ["sel", "adj", "w", "recon"]: | |
| 101 | + marg[f] = fac.groupby(f)["mdape_test"].mean().round(2).to_dict() | |
| 102 | +res["E2_marginal"] = marg | |
| 103 | +anc_tab = fac.groupby(["k", "d", "T"])["mdape_test"].min().round(2).to_dict() | |
| 104 | +res["E2_anchor_best"] = {f"{k}|{d}|{t}": v for (k, d, t), v in anc_tab.items()} | |
| 105 | +# figure : ajustements à l'ancre (val et test) | |
| 106 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 107 | +g = fac[base].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 108 | +lab = {"raw": "aucun", "none": "temps seul", "conv": "grille conv.", "hedo": "hédonique", "hedo_reg": "hédo. régionale", "local": "locale", "gbm": "GBM", "role": "ratio au rôle"} | |
| 109 | +x = np.arange(len(g)); ax.bar(x - 0.2, g.mdape_val, 0.38, color=PAL[0], label="validation 2024"); ax.bar(x + 0.2, g.mdape_test, 0.38, color=PAL[1], label="test 2025-26") | |
| 110 | +for i, (v1, v2) in enumerate(zip(g.mdape_val, g.mdape_test)): | |
| 111 | + ax.text(i - 0.2, v1 + 0.2, f"{v1:.1f}", ha="center", fontsize=7); ax.text(i + 0.2, v2 + 0.2, f"{v2:.1f}", ha="center", fontsize=7) | |
| 112 | +ax.set_xticks(x); ax.set_xticklabels([lab[a] for a in g.index]); ax.set_ylabel("MdAPE (%)"); ax.legend(); ax.set_title("Type d'ajustement — 5 comparables les plus proches, ≤ 5 km, ≤ 12 mois, médiane") | |
| 113 | +savefig(fig, "f3_ajustements") | |
| 114 | + | |
| 115 | +# ---------------------------------------------------------------- modèles de référence (M10, M11) et compétition | |
| 116 | +def level_shift(m_asof): | |
| 117 | + mt_ = T["month"].to_numpy(); r = T["region"].map(rid).to_numpy() | |
| 118 | + return IDX[mt_ - M_EVAL0, r, mt_ - 1] - IDX[mt_ - M_EVAL0, r, m_asof - 1] | |
| 119 | +Sx = S.loc[T["idx"]] | |
| 120 | +XF = np.column_stack([Sx.ln_floor, Sx.ln_land, Sx.age, Sx.age ** 2 / 100, Sx.lien == 1, Sx.lien >= 2, Sx.genre == 1, Sx.genre == 2, Sx.genre == 3, Sx.etages, Sx.neuf]).astype(float) | |
| 121 | +M10 = np.full(len(T), np.nan); M11 = np.full(len(T), np.nan); M11b = np.full(len(T), np.nan) | |
| 122 | +for a in load_json("asof.json")["asof"]: | |
| 123 | + mask = (T["asof"] == a).to_numpy(); period = str(a) | |
| 124 | + if not mask.any(): continue | |
| 125 | + mod = load_json(f"models_{period}.json"); beta = np.array([mod["beta_global"][f] for f in mod["feats"]]) | |
| 126 | + shift = level_shift(mod["m_asof"]) | |
| 127 | + fe = (T["region"].map(mod["region_fe"]).fillna(0) + T["mrc"].map(mod["mrc_eff"]).fillna(0) + T["code_mun"].map(mod["mun_eff"]).fillna(0) | |
| 128 | + + (T["code_mun"].astype(str) + "|" + T["unite_voisinage"].astype(str)).map(mod["uv_eff"]).fillna(0)).to_numpy() | |
| 129 | + M10[mask] = (XF @ beta + fe + shift + mod["median_shift"])[mask] | |
| 130 | + M11[mask] = (np.load(OUT / f"gbm_role_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 131 | + M11b[mask] = (np.load(OUT / f"gbm_pred_{period}.npy")[T["idx"].to_numpy()] + shift)[mask] | |
| 132 | + | |
| 133 | +def e2col(sel, k, d, Tm, adj, w, recon): | |
| 134 | + m = (E2.sel == sel) & (E2.k == k) & (E2.d == d) & (E2["T"] == Tm) & (E2.adj == adj) & (E2.w == w) & (E2.recon == recon) | |
| 135 | + return int(E2.index[m][0]) | |
| 136 | +def e1col(sel, k, d, Tm, adj): | |
| 137 | + m = (E1.sel == sel) & (E1.k == k) & (E1.d == d) & (E1["T"] == Tm) & (E1.adj == adj) | |
| 138 | + return int(E1.index[m][0]) | |
| 139 | +MODELS = { | |
| 140 | + "M1 médiane des 3 plus proches (temps seul)": ("E2", e2col("geo", 3, 2, 6, "none", "equal", "median")), | |
| 141 | + "M2 moyenne des 5 plus proches (temps seul)": ("E2", e2col("geo", 5, 5, 12, "none", "equal", "mean")), | |
| 142 | + "M3 sélection par similarité physique": ("E2", e2col("sim", 5, 5, 12, "none", "equal", "median")), | |
| 143 | + "M4 pondération par distance": ("E2", e2col("geo", 5, 5, 12, "none", "dist", "mean")), | |
| 144 | + "M5 pondération par récence": ("E2", e2col("geo", 5, 5, 12, "none", "rec", "mean")), | |
| 145 | + "M6 grille d'ajustement conventionnelle": ("E2", e2col("geo", 5, 5, 12, "conv", "equal", "median")), | |
| 146 | + "M7 grille d'ajustement hédonique": ("E2", e2col("geo", 5, 5, 12, "hedo", "equal", "median")), | |
| 147 | + "M8a grille locale (régression sur le voisinage)": ("E2", e2col("geo", 5, 5, 12, "local", "equal", "median")), | |
| 148 | + "M8b ajustements non linéaires (GBM)": ("E2", e2col("geo", 5, 5, 12, "gbm", "equal", "median")), | |
| 149 | + "M13 ratio au rôle (5 plus proches, même rôle)": ("E2", e2col("geo", 5, 5, 12, "role", "equal", "median")), | |
| 150 | + "M10 hédonique globale directe": ("X", M10), | |
| 151 | + "M11a gradient boosting (sans rôle)": ("X", M11b), | |
| 152 | + "M11b gradient boosting (avec rôle)": ("X", M11), | |
| 153 | +} | |
| 154 | +def est_of(spec): | |
| 155 | + kind, c = spec | |
| 156 | + return est2[:, c].astype(float) if kind == "E2" else c | |
| 157 | +# M0 : meilleure configuration empirique choisie sur la validation (E1 ∪ E2, couverture ≥ 90 %) | |
| 158 | +cand_all = pd.concat([surf.assign(src="E1"), fac.assign(src="E2")], ignore_index=True) | |
| 159 | +cand_ok = cand_all[cand_all.couverture_val >= 97].sort_values("mdape_val") | |
| 160 | +b0 = cand_ok.iloc[0] | |
| 161 | +M0_est = (est2[:, int(b0.cfg)] if b0.src == "E2" else np.load(OUT / "est_E1.npy", mmap_mode="r")[:, int(b0.cfg)]).astype(float) | |
| 162 | +res["M0"] = b0[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_test"]].to_dict() | |
| 163 | +res["M0_sans_contrainte"] = cand_all[cand_all.couverture_val >= 50].sort_values("mdape_val").iloc[0][["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape_val", "mdape_test", "couverture_val", "couverture_test"]].to_dict() | |
| 164 | +MODELS["M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"] = ("X", M0_est) | |
| 165 | +# M9 : réconciliation optimisée = combinaison convexe (NNLS sur la validation) d'estimations de comparables | |
| 166 | +stack_cols = {a: e2col("geo", 5, 5, 12, a, "equal", "median") for a in ["none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]} | |
| 167 | +def stack(names, mats, fit_mask): | |
| 168 | + """Combinaison convexe apprise par NNLS sur la validation ; si une composante manque pour une cible, | |
| 169 | + les poids des composantes disponibles sont renormalisés.""" | |
| 170 | + A = np.column_stack(mats); okr = ~np.isnan(A).any(axis=1) | |
| 171 | + w, _ = nnls(A[okr & fit_mask], lnP[okr & fit_mask].astype(float)); w = w / w.sum() | |
| 172 | + W = np.where(np.isnan(A), 0.0, w[None, :]); den = W.sum(axis=1) | |
| 173 | + out = np.nansum(A * W, axis=1) / np.where(den > 0, den, np.nan) | |
| 174 | + return out, dict(zip(names, w.round(3))) | |
| 175 | +M9_est, w9 = stack(list(stack_cols), [est2[:, c].astype(float) for c in stack_cols.values()], VAL) | |
| 176 | +res["M9_weights"] = w9; MODELS["M9 réconciliation optimisée (poids appris en 2024)"] = ("X", M9_est) | |
| 177 | +M12_est, w12 = stack(["gbm_role", "comps_gbm", "comps_hedo"], [M11, est2[:, stack_cols["gbm"]].astype(float), est2[:, stack_cols["hedo"]].astype(float)], VAL) | |
| 178 | +res["M12_weights"] = w12; MODELS["M12 hybride comparables + GBM (poids appris en 2024)"] = ("X", M12_est) | |
| 179 | + | |
| 180 | +rows = [] | |
| 181 | +ERR = {} | |
| 182 | +for name, spec in MODELS.items(): | |
| 183 | + e = (np.exp(est_of(spec) - lnP) - 1).astype(np.float32); ERR[name] = e | |
| 184 | + for per, mask in [("val", VAL), ("test", TEST)]: | |
| 185 | + m = metrics(e[:, None], mask).iloc[0].to_dict(); m.update(modele=name, periode=per); rows.append(m) | |
| 186 | +comp = pd.DataFrame(rows); comp.to_csv(TAB / "competition_modeles.csv", index=False) | |
| 187 | +pd.DataFrame({n.split(" ")[0]: est_of(s_) for n, s_ in MODELS.items()}).assign(idx=T["idx"].to_numpy()).to_parquet(OUT / "models_est.parquet", index=False) | |
| 188 | +res["competition"] = comp[comp.periode == "test"].set_index("modele")[["mdape", "mape", "biais", "p5", "p10", "p20", "couverture"]].round(2).to_dict("index") | |
| 189 | +res["competition_val"] = comp[comp.periode == "val"].set_index("modele")[["mdape", "p10", "couverture"]].round(2).to_dict("index") | |
| 190 | +log("compétition (test) :\n" + comp[comp.periode == "test"][["modele", "mdape", "mape", "biais", "p10", "couverture"]].round(2).to_string(index=False)) | |
| 191 | +fig, ax = plt.subplots(figsize=(7.5, 5)) | |
| 192 | +g = comp[comp.periode == "test"].sort_values("mdape", ascending=False) | |
| 193 | +cols = [PAL[0] if ("M1" in n and "M1 " not in n) or n.startswith(("M10", "M11", "M12")) else PAL[2] if n.startswith(("M0", "M9")) else PAL[1] for n in g.modele] | |
| 194 | +cols = [PAL[0] if n.startswith(("M10", "M11", "M12")) else (PAL[2] if n.startswith(("M0", "M9")) else PAL[1]) for n in g.modele] | |
| 195 | +ax.barh(g.modele, g.mdape, color=cols, height=0.65) | |
| 196 | +for i, v in enumerate(g.mdape): ax.text(v + 0.15, i, f"{v:.1f} %", va="center", fontsize=8) | |
| 197 | +ax.set_xlabel("MdAPE, test 2025-26 (%)"); ax.set_title("Compétition de modèles (orange : comparables ; vert : optimisés ; bleu : références externes)") | |
| 198 | +savefig(fig, "f4_competition") | |
| 199 | + | |
| 200 | +# ---------------------------------------------------------------- bootstrap par grappes (municipalité × trimestre) des différences appariées | |
| 201 | +rng = np.random.default_rng(7) | |
| 202 | +def paired(eA, eB, mask, B=400): | |
| 203 | + a = np.abs(eA[mask]); b = np.abs(eB[mask]); cl = T["cluster"].to_numpy()[mask] | |
| 204 | + okp = ~np.isnan(a) & ~np.isnan(b); a, b, cl = a[okp], b[okp], cl[okp] | |
| 205 | + d = a - b | |
| 206 | + cl_ids, inv = np.unique(cl, return_inverse=True); nc = len(cl_ids) | |
| 207 | + sums = np.bincount(inv, d, nc); cnts = np.bincount(inv, minlength=nc) | |
| 208 | + boots_mean = []; boots_med = [] | |
| 209 | + order = np.argsort(inv); d_sorted = d[order]; starts = np.searchsorted(inv[order], np.arange(nc)); ends = np.append(starts[1:], len(d)) | |
| 210 | + for _ in range(B): | |
| 211 | + pick = rng.integers(0, nc, nc) | |
| 212 | + boots_mean.append(sums[pick].sum() / cnts[pick].sum()) | |
| 213 | + if _ < 150: | |
| 214 | + idxs = np.concatenate([np.arange(starts[p], ends[p]) for p in pick]) | |
| 215 | + boots_med.append(np.median(a[order][idxs]) - np.median(b[order][idxs])) | |
| 216 | + return dict(n=int(len(d)), diff_mape=float(d.mean() * 100), ci_mape=[float(np.percentile(boots_mean, 2.5) * 100), float(np.percentile(boots_mean, 97.5) * 100)], | |
| 217 | + diff_mdape=float((np.median(a) - np.median(b)) * 100), ci_mdape=[float(np.percentile(boots_med, 2.5) * 100), float(np.percentile(boots_med, 97.5) * 100)], | |
| 218 | + part_A_meilleur=float((a < b).mean() * 100)) | |
| 219 | +PAIRS = [ | |
| 220 | + ("M7 grille d'ajustement hédonique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 221 | + ("M7 grille d'ajustement hédonique", "M6 grille d'ajustement conventionnelle"), | |
| 222 | + ("M8a grille locale (régression sur le voisinage)", "M7 grille d'ajustement hédonique"), | |
| 223 | + ("M8b ajustements non linéaires (GBM)", "M7 grille d'ajustement hédonique"), | |
| 224 | + ("M11b gradient boosting (avec rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 225 | + ("M11a gradient boosting (sans rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"), | |
| 226 | + ("M12 hybride comparables + GBM (poids appris en 2024)", "M11b gradient boosting (avec rôle)"), | |
| 227 | + ("M13 ratio au rôle (5 plus proches, même rôle)", "M7 grille d'ajustement hédonique"), | |
| 228 | + ("M3 sélection par similarité physique", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 229 | + ("M9 réconciliation optimisée (poids appris en 2024)", "M7 grille d'ajustement hédonique"), | |
| 230 | + ("M1 médiane des 3 plus proches (temps seul)", "M2 moyenne des 5 plus proches (temps seul)"), | |
| 231 | +] | |
| 232 | +res["paires"] = {f"{a} vs {b}": paired(ERR[a], ERR[b], TEST) for a, b in PAIRS} | |
| 233 | +pd.DataFrame([dict(A=a, B=b, **res["paires"][f"{a} vs {b}"]) for a, b in PAIRS]).to_csv(TAB / "paires_bootstrap.csv", index=False) | |
| 234 | + | |
| 235 | +# ---------------------------------------------------------------- strates | |
| 236 | +def strata(err, by): | |
| 237 | + out = [] | |
| 238 | + for lev, g in T[TEST].groupby(by, observed=True): | |
| 239 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 240 | + r = metrics(err[:, None], m).iloc[0].to_dict(); r[by] = str(lev); out.append(r) | |
| 241 | + return pd.DataFrame(out) | |
| 242 | +KEY = ["M2 moyenne des 5 plus proches (temps seul)", "M7 grille d'ajustement hédonique", "M8a grille locale (régression sur le voisinage)", | |
| 243 | + "M13 ratio au rôle (5 plus proches, même rôle)", "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)", "M10 hédonique globale directe", "M11b gradient boosting (avec rôle)", "M12 hybride comparables + GBM (poids appris en 2024)"] | |
| 244 | +for by in ["region_nom", "dens", "prix_q", "trimestre", "age_cl", "lien"]: | |
| 245 | + tabs = [] | |
| 246 | + for name in KEY: | |
| 247 | + s = strata(ERR[name], by); s["modele"] = name.split(" ")[0]; tabs.append(s) | |
| 248 | + st = pd.concat(tabs); st.to_csv(TAB / f"strates_{by}.csv", index=False) | |
| 249 | + res[f"strates_{by}"] = st.pivot(index=by, columns="modele", values="mdape").round(2).to_dict("index") | |
| 250 | + res[f"strates_{by}_n"] = st[st.modele == "M7"].set_index(by)["n"].to_dict() | |
| 251 | +# figure : densité | |
| 252 | +fig, ax = plt.subplots(figsize=(6.5, 3.4)) | |
| 253 | +st = pd.read_csv(TAB / "strates_dens.csv") | |
| 254 | +order = ["< 5", "5-19", "20-49", "50-199", "≥ 200"] | |
| 255 | +for j, (name, lab_) in enumerate([("M2", "M2 5 plus proches"), ("M7", "M7 grille hédonique"), ("M0", "M0 meilleure config."), ("M11b", "M11b GBM"), ("M12", "M12 hybride")]): | |
| 256 | + g = st[st.modele == name].set_index("dens").loc[order] | |
| 257 | + ax.plot(order, g.mdape, color=PAL[j], lw=2, marker="o", ms=4, label=lab_) | |
| 258 | +ax.set_xlabel("ventes disponibles dans un rayon de 2 km au cours des 12 mois précédents"); ax.set_ylabel("MdAPE, test (%)"); ax.legend(fontsize=8); ax.set_title("Précision selon la liquidité du sous-marché") | |
| 259 | +savefig(fig, "f5_densite") | |
| 260 | + | |
| 261 | +# ---------------------------------------------------------------- hypothèses | |
| 262 | +H = {} | |
| 263 | +# H1 : proche mais dissemblable vs loin mais semblable (k = 5, T = 12, sans ajustement) | |
| 264 | +H["H1"] = {"geo_d1": surf.loc[e1col("geo", 5, 1, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 265 | + "sim_d10": surf.loc[e1col("sim", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 266 | + "geo_d10": surf.loc[e1col("geo", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 267 | + "mix_d10": surf.loc[e1col("mix", 5, 10, 12, "none"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 268 | + "sim_d10_hedo": surf.loc[e1col("sim", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 269 | + "geo_d10_hedo": surf.loc[e1col("geo", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict(), | |
| 270 | + "mix_d10_hedo": surf.loc[e1col("mix", 5, 10, 12, "hedo"), ["mdape_test", "couverture_test"]].to_dict()} | |
| 271 | +# H2 : k optimal selon la densité (geo, d = 10, T = 12, hedo et none) | |
| 272 | +h2 = {} | |
| 273 | +for a in ["none", "hedo"]: | |
| 274 | + for lev in order: | |
| 275 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 276 | + cols_ = [e1col("geo", k, 10, 12, a) for k in [1, 3, 5, 7, 10, 15, 20, 30, 50]] | |
| 277 | + md = np.nanmedian(np.abs(err1[m][:, cols_]), axis=0) * 100 | |
| 278 | + h2[f"{a}|{lev}"] = dict(zip([1, 3, 5, 7, 10, 15, 20, 30, 50], md.round(2))) | |
| 279 | +H["H2"] = h2 | |
| 280 | +fig, axes = plt.subplots(1, 2, figsize=(9, 3.3), sharey=True) | |
| 281 | +for ax, a in zip(axes, ["none", "hedo"]): | |
| 282 | + for j, lev in enumerate(order): | |
| 283 | + v = h2[f"{a}|{lev}"]; ax.plot(list(v), list(v.values()), color=PAL[j], lw=2, marker="o", ms=3.5, label=f"densité {lev}") | |
| 284 | + ax.set_xscale("log"); ax.set_xticks([1, 3, 5, 10, 20, 50]); ax.set_xticklabels([1, 3, 5, 10, 20, 50]); ax.set_xlabel("k"); ax.set_title({"none": "temps seul", "hedo": "grille hédonique"}[a] + " — d ≤ 10 km, 12 mois") | |
| 285 | +axes[0].set_ylabel("MdAPE, test (%)"); axes[1].legend(fontsize=8) | |
| 286 | +savefig(fig, "f6_k_par_densite") | |
| 287 | +# H3 : local vs hédonique fixe selon la densité, aux trois ancres | |
| 288 | +h3 = {} | |
| 289 | +for (k, d, Tm) in [(3, 2, 6), (5, 5, 12), (10, 10, 24)]: | |
| 290 | + for lev in order: | |
| 291 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 292 | + h3[f"{k}|{d}|{Tm}|{lev}"] = {a: float(np.nanmedian(np.abs(err2[m][:, e2col("geo", k, d, Tm, a, "equal", "median")])) * 100) for a in ["conv", "hedo", "hedo_reg", "local", "gbm"]} | |
| 293 | +H["H3"] = h3 | |
| 294 | +# H4 : pondération selon la densité (ancre 5/5/12, hedo) | |
| 295 | +h4 = {} | |
| 296 | +for lev in order: | |
| 297 | + m = TEST & (T["dens"] == lev).to_numpy() | |
| 298 | + h4[lev] = {f"{w}|{r}": float(np.nanmedian(np.abs(err2[m][:, e2col("geo", 5, 5, 12, "hedo", w, r)])) * 100) for (w, r) in [("equal", "median"), ("equal", "mean"), ("dist", "mean"), ("rec", "mean"), ("sim", "mean"), ("all", "mean"), ("all", "median")]} | |
| 299 | +H["H4"] = h4 | |
| 300 | +# H5 : méthodes simples vs ML selon l'homogénéité (dispersion des prix ajustés des 10 voisins à 10 km / 24 mois) | |
| 301 | +het = sd2[:, e2col("geo", 10, 10, 24, "hedo", "equal", "median")].astype(float) | |
| 302 | +T["heterog"] = pd.qcut(pd.Series(np.where(np.isnan(het), np.nanmax(het), het)), 3, labels=["homogène", "moyen", "hétérogène"]) | |
| 303 | +h5 = {} | |
| 304 | +for lev in ["homogène", "moyen", "hétérogène"]: | |
| 305 | + m = TEST & (T["heterog"] == lev).to_numpy() | |
| 306 | + h5[lev] = {n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY} | |
| 307 | + h5[lev]["sd_median"] = float(np.nanmedian(het[m])) | |
| 308 | +H["H5"] = h5 | |
| 309 | +res["hypotheses"] = H | |
| 310 | + | |
| 311 | +# ---------------------------------------------------------------- intervalles d'incertitude (couverture) | |
| 312 | +intervals = {} | |
| 313 | +for a in ["none", "hedo", "gbm"]: | |
| 314 | + for (k, d, Tm) in [(5, 5, 12), (10, 10, 24)]: | |
| 315 | + c = e2col("geo", k, d, Tm, a, "equal", "median"); e = est2[:, c].astype(float); s = sd2[:, c].astype(float) | |
| 316 | + okv = VAL & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 317 | + z_needed = np.abs(lnP[okv] - e[okv]) / s[okv]; z80 = float(np.percentile(z_needed, 80)) | |
| 318 | + okt = TEST & ~np.isnan(e) & ~np.isnan(s) & (s > 0) | |
| 319 | + cov_nominal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= 1.2816 * s[okt]) * 100) | |
| 320 | + cov_cal = float(np.mean(np.abs(lnP[okt] - e[okt]) <= z80 * s[okt]) * 100) | |
| 321 | + width = float(np.median(2 * z80 * s[okt]) * 100) | |
| 322 | + intervals[f"{a}|{k}|{d}|{Tm}"] = dict(z80_val=z80, couverture_nominale_80=cov_nominal, couverture_calibree=cov_cal, largeur_mediane_pct=width, n_test=int(okt.sum())) | |
| 323 | +res["intervalles"] = intervals | |
| 324 | + | |
| 325 | +# ---------------------------------------------------------------- règles conditionnelles (système de recommandation) | |
| 326 | +cond = {}; est_cond = np.full(len(T), np.nan) | |
| 327 | +for lev in order: | |
| 328 | + mv_ = VAL & (T["dens"] == lev).to_numpy(); mt_ = TEST & (T["dens"] == lev).to_numpy() | |
| 329 | + m1 = metrics(err1, mv_); m2 = metrics(err2, mv_) | |
| 330 | + allm = pd.concat([pd.concat([E1, m1], axis=1).assign(src="E1"), pd.concat([E2, m2], axis=1).assign(src="E2")], ignore_index=True) | |
| 331 | + b = allm[allm.couverture >= 95].sort_values("mdape").iloc[0] | |
| 332 | + e = (err1 if b.src == "E1" else err2)[:, int(b.cfg)] | |
| 333 | + est_cond[mt_] = e[mt_] | |
| 334 | + cond[lev] = dict(b[["src", "sel", "d", "T", "k", "adj", "w", "recon", "mdape", "couverture"]].to_dict(), mdape_test=float(np.nanmedian(np.abs(e[mt_])) * 100), n_test=int(mt_.sum())) | |
| 335 | +res["regles_conditionnelles"] = cond | |
| 336 | +res["regles_conditionnelles_global_test"] = dict(mdape=float(np.nanmedian(np.abs(est_cond[TEST])) * 100), p10=float(np.nanmean(np.abs(est_cond[TEST]) <= 0.1) * 100), couverture=float(np.mean(~np.isnan(est_cond[TEST])) * 100)) | |
| 337 | +pd.DataFrame(cond).T.to_csv(TAB / "regles_conditionnelles.csv") | |
| 338 | + | |
| 339 | +# ---------------------------------------------------------------- stabilité temporelle, indice, divers | |
| 340 | +tq = [] | |
| 341 | +for q, g in T[TEST].groupby("trimestre"): | |
| 342 | + m = np.zeros(len(T), bool); m[g.index] = True | |
| 343 | + tq.append(dict(trimestre=q, n=int(m.sum()), **{n.split(" ")[0]: float(np.nanmedian(np.abs(ERR[n][m])) * 100) for n in KEY}, | |
| 344 | + **{"biais_" + n.split(" ")[0]: float(np.nanmedian(ERR[n][m]) * 100) for n in KEY})) | |
| 345 | +pd.DataFrame(tq).to_csv(TAB / "stabilite_trimestres.csv", index=False); res["stabilite"] = tq | |
| 346 | +fig, ax = plt.subplots(figsize=(7, 3.2)) | |
| 347 | +idxf = pd.read_csv(OUT / "index_full.csv") | |
| 348 | +for j, (r, lab_) in enumerate([("QC", "Québec"), ("06", "Montréal"), ("03", "Capitale-Nationale"), ("07", "Outaouais"), ("05", "Estrie")]): | |
| 349 | + g = idxf[idxf.region.astype(str).str.zfill(2) == r] if r != "QC" else idxf[idxf.region == "QC"] | |
| 350 | + if len(g): ax.plot(pd.to_datetime(g.ym), np.exp(g.log_idx) * 100, color=PAL[j], lw=2 if r == "QC" else 1.4, label=lab_) | |
| 351 | +ax.set_ylabel("indice hédonique (2021-01 = 100)"); ax.legend(fontsize=8); ax.set_title("Indice de prix mensuel (plein échantillon, descriptif)") | |
| 352 | +savefig(fig, "f7_indice") | |
| 353 | +res["dataset"] = load_json("dataset_stats.json") | |
| 354 | +res["n_targets"] = dict(val=int(VAL.sum()), test=int(TEST.sum())) | |
| 355 | +res["densite_distribution"] = T[TEST]["dens"].value_counts().sort_index().to_dict() | |
| 356 | +res["n_cand_median"] = dict(val=float(np.median(T[VAL].n_cand)), test=float(np.median(T[TEST].n_cand))) | |
| 357 | +save_json(res, "results.json") | |
| 358 | +log("analyse terminée") | |
added
M4M64bomparables-validation/pipeline/06_report.py
+313 −0
@@ -0,0 +1,313 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""06 — Rapport : assemble docs/RAPPORT.md à partir de results.json et des tables (texte + tableaux + figures). | |
| 3 | +Les phrases d'interprétation sont calculées à partir des résultats (jamais de chiffre écrit à la main).""" | |
| 4 | +from common import * | |
| 5 | + | |
| 6 | +R = load_json("results.json"); TAB = OUT / "tables" | |
| 7 | +DOCS = ROOT / "docs"; DOCS.mkdir(exist_ok=True) | |
| 8 | +D = R["dataset"]; comp = R["competition"]; compv = R["competition_val"] | |
| 9 | +def f1(x): return f"{x:.1f}".replace(".", ",") | |
| 10 | +def f2(x): return f"{x:.2f}".replace(".", ",") | |
| 11 | +def n_(x): return f"{int(round(x)):,}".replace(",", " ") | |
| 12 | +def pct(x): return f"{f1(x)} %" | |
| 13 | +def md_table(df, cols=None, fmt=None, index=False): | |
| 14 | + if cols: df = df[cols] | |
| 15 | + if index: df = df.reset_index() | |
| 16 | + fmt = fmt or {} | |
| 17 | + head = "| " + " | ".join(str(c) for c in df.columns) + " |\n|" + "|".join("---" for _ in df.columns) + "|\n" | |
| 18 | + rows = [] | |
| 19 | + for _, r in df.iterrows(): | |
| 20 | + cells = [] | |
| 21 | + for c in df.columns: | |
| 22 | + v = r[c] | |
| 23 | + if c in fmt: cells.append(fmt[c](v)) | |
| 24 | + elif isinstance(v, float): cells.append(f1(v)) | |
| 25 | + else: cells.append(str(v)) | |
| 26 | + rows.append("| " + " | ".join(cells) + " |") | |
| 27 | + return head + "\n".join(rows) + "\n" | |
| 28 | +short = lambda name: name.split(" ")[0] | |
| 29 | +LAB = {"raw": "aucun ajustement", "none": "temps seulement", "conv": "grille conventionnelle", "hedo": "grille hédonique", "hedo_reg": "grille hédonique régionale", | |
| 30 | + "local": "régression locale", "gbm": "ajustements GBM", "role": "ratio au rôle", "geo": "les plus proches (distance)", "sim": "les plus semblables (physique)", | |
| 31 | + "mix": "compromis distance / similarité", "adj": "les moins ajustés", "equal": "égale", "dist": "1/distance", "rec": "récence", "all": "distance × récence × similarité", | |
| 32 | + "median": "médiane", "mean": "moyenne"} | |
| 33 | + | |
| 34 | +# ------------------------------------------------------------------ tableaux | |
| 35 | +excl = pd.DataFrame(D["exclusions"], columns=["Règle", "n"]); excl["n"] = excl["n"].map(n_) | |
| 36 | +cm = pd.DataFrame(comp).T; cm.index.name = "Modèle"; cm = cm.reset_index() | |
| 37 | +cm["val"] = cm["Modèle"].map(lambda m: compv[m]["mdape"]) | |
| 38 | +cm = cm.sort_values("mdape") | |
| 39 | +t_comp = md_table(cm.rename(columns={"mdape": "MdAPE test", "val": "MdAPE val.", "mape": "MAPE", "biais": "Biais (médian)", "p5": "±5 %", "p10": "±10 %", "p20": "±20 %", "couverture": "Couverture"}), | |
| 40 | + ["Modèle", "MdAPE val.", "MdAPE test", "MAPE", "Biais (médian)", "±5 %", "±10 %", "±20 %", "Couverture"]) | |
| 41 | +surf = pd.read_csv(TAB / "surface_E1.csv") | |
| 42 | +def surf_tab(sel, adj, T_): | |
| 43 | + g = surf[(surf.sel == sel) & (surf.adj == adj) & (surf["T"] == T_)].pivot(index="d", columns="k", values="mdape_test") | |
| 44 | + g.index = [f"≤ {d} km" for d in g.index]; g.columns = [f"k = {k}" for k in g.columns]; g.index.name = "rayon" | |
| 45 | + return md_table(g.round(1), index=True) | |
| 46 | +kb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.d == 10) & (surf["T"] == 12)].sort_values("k") | |
| 47 | +fac = pd.read_csv(TAB / "facteurs_E2.csv") | |
| 48 | +anchor = fac[(fac.k == 5) & (fac.d == 5) & (fac["T"] == 12)] | |
| 49 | +t_adj = anchor[(anchor.sel == "geo") & (anchor.w == "equal") & (anchor.recon == "median")].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]] | |
| 50 | +t_adj = t_adj.reset_index(); t_adj["Ajustement"] = t_adj["adj"].map(LAB) | |
| 51 | +t_adj_md = md_table(t_adj.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "mape_test": "MAPE test", "biais_test": "Biais", "p10_test": "±10 %", "couverture_test": "Couverture"}), | |
| 52 | + ["Ajustement", "MdAPE val.", "MdAPE test", "MAPE test", "Biais", "±10 %", "Couverture"]) | |
| 53 | +t_sel = anchor[(anchor.adj == "hedo") & (anchor.w == "equal") & (anchor.recon == "median")].copy(); t_sel["Sélection"] = t_sel["sel"].map(LAB) | |
| 54 | +t_sel_md = md_table(t_sel.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Sélection", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 55 | +t_wr = anchor[(anchor.adj == "hedo") & (anchor.sel == "geo")].copy(); t_wr["Pondération"] = t_wr["w"].map(LAB); t_wr["Réconciliation"] = t_wr["recon"].map(LAB) | |
| 56 | +t_wr_md = md_table(t_wr.rename(columns={"mdape_val": "MdAPE val.", "mdape_test": "MdAPE test", "p10_test": "±10 %", "biais_test": "Biais"}), ["Pondération", "Réconciliation", "MdAPE val.", "MdAPE test", "Biais", "±10 %"]) | |
| 57 | +pairs = pd.read_csv(TAB / "paires_bootstrap.csv") | |
| 58 | +pairs["Comparaison"] = pairs.apply(lambda r: f"{short(r.A)} − {short(r.B)}", axis=1) | |
| 59 | +pairs["Δ MdAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mdape)} [{f2(eval(r.ci_mdape)[0])} ; {f2(eval(r.ci_mdape)[1])}]", axis=1) | |
| 60 | +pairs["Δ MAPE (pts) [IC 95 %]"] = pairs.apply(lambda r: f"{f2(r.diff_mape)} [{f2(eval(r.ci_mape)[0])} ; {f2(eval(r.ci_mape)[1])}]", axis=1) | |
| 61 | +pairs["A plus précis (% des cibles)"] = pairs["part_A_meilleur"].map(f1) | |
| 62 | +t_pairs = md_table(pairs, ["Comparaison", "Δ MdAPE (pts) [IC 95 %]", "Δ MAPE (pts) [IC 95 %]", "A plus précis (% des cibles)", "n"], fmt={"n": n_}) | |
| 63 | +def strata_md(by, label): | |
| 64 | + st = pd.read_csv(TAB / f"strates_{by}.csv") | |
| 65 | + g = st.pivot(index=by, columns="modele", values="mdape").round(1) | |
| 66 | + g["n"] = st[st.modele == "M7"].set_index(by)["n"] | |
| 67 | + g = g[["n", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]] | |
| 68 | + if by == "dens": g = g.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 69 | + if by == "region_nom": g = g.sort_values("n", ascending=False) | |
| 70 | + g.index.name = label | |
| 71 | + return md_table(g, index=True, fmt={"n": n_}) | |
| 72 | +H = R["hypotheses"] | |
| 73 | +h2 = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("hedo")}).T; h2.index.name = "densité"; h2.columns = [f"k = {c}" for c in h2.columns] | |
| 74 | +h2_md = md_table(h2.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 75 | +h2n = pd.DataFrame({k.split("|")[1]: v for k, v in H["H2"].items() if k.startswith("none")}).T; h2n.index.name = "densité"; h2n.columns = [f"k = {c}" for c in h2n.columns] | |
| 76 | +h2n_md = md_table(h2n.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]], index=True) | |
| 77 | +h3 = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("5|5|12")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 78 | +h3.columns = [LAB[c] for c in h3.columns]; h3.index.name = "densité"; h3_md = md_table(h3, index=True) | |
| 79 | +h3s = pd.DataFrame({k.split("|")[3]: v for k, v in H["H3"].items() if k.startswith("3|2|6")}).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 80 | +h3s.columns = [LAB[c] for c in h3s.columns]; h3s.index.name = "densité"; h3s_md = md_table(h3s, index=True) | |
| 81 | +h4 = pd.DataFrame(H["H4"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]]; h4.columns = [f"{LAB[c.split('|')[0]]} / {LAB[c.split('|')[1]]}" for c in h4.columns]; h4.index.name = "densité"; h4_md = md_table(h4, index=True) | |
| 82 | +h5 = pd.DataFrame(H["H5"]).T; h5.index.name = "sous-marché"; h5 = h5.rename(columns={"sd_median": "σ médian (log)"}); h5_md = md_table(h5[["σ médian (log)", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]], index=True, fmt={"σ médian (log)": lambda v: f"{v:.3f}".replace(".", ",")}) | |
| 83 | +iv = pd.DataFrame(R["intervalles"]).T; iv.index = [f"{LAB[i.split('|')[0]]}, k = {i.split('|')[1]}, ≤ {i.split('|')[2]} km, {i.split('|')[3]} mois" for i in iv.index]; iv.index.name = "configuration" | |
| 84 | +iv_md = md_table(iv.rename(columns={"z80_val": "z calibré (val.)", "couverture_nominale_80": "couverture à z = 1,28", "couverture_calibree": "couverture calibrée", "largeur_mediane_pct": "largeur médiane (%)", "n_test": "n"}), index=True, fmt={"n": n_, "z calibré (val.)": f2}) | |
| 85 | +rc = pd.DataFrame(R["regles_conditionnelles"]).T.loc[["< 5", "5-19", "20-49", "50-199", "≥ 200"]] | |
| 86 | +rc["règle"] = rc.apply(lambda r: f"{LAB[r.sel]}, k = {int(r.k)}, ≤ {r.d} km, {int(r['T'])} mois, {LAB[r.adj]}, pond. {LAB[r.w]}, {LAB[r.recon]}", axis=1) | |
| 87 | +rc.index.name = "densité"; rc_md = md_table(rc.rename(columns={"mdape": "MdAPE val.", "mdape_test": "MdAPE test", "couverture": "couverture val.", "n_test": "n test"}), ["règle", "MdAPE val.", "MdAPE test", "couverture val.", "n test"], index=True, fmt={"n test": n_}) | |
| 88 | +stab = pd.DataFrame(R["stabilite"]); stab_md = md_table(stab[["trimestre", "n", "M2", "M7", "M0", "M11b", "M12", "biais_M7", "biais_M11b"]].rename(columns={"biais_M7": "biais M7", "biais_M11b": "biais M11b"}), fmt={"n": n_}) | |
| 89 | + | |
| 90 | +# ------------------------------------------------------------------ chiffres clés pour le texte | |
| 91 | +best = min(comp, key=lambda m: comp[m]["mdape"]); best_comps = min([m for m in comp if short(m) in ("M1", "M2", "M3", "M4", "M5", "M6", "M7", "M8a", "M8b", "M13", "M0", "M9")], key=lambda m: comp[m]["mdape"]) | |
| 92 | +m0 = R["M0"]; m0u = R["M0_sans_contrainte"] | |
| 93 | +gain_adj = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M7 grille d'ajustement hédonique"]["mdape"] | |
| 94 | +gain_sel = comp["M2 moyenne des 5 plus proches (temps seul)"]["mdape"] - comp["M3 sélection par similarité physique"]["mdape"] | |
| 95 | +k_curve = {int(r.k): r.mdape_test for r in kb.itertuples()} | |
| 96 | +k_best = min(k_curve, key=k_curve.get) | |
| 97 | +e1b = R["E1_best_val"] | |
| 98 | +pair = R["paires"] | |
| 99 | +def pr(a, b): return pair[f"{a} vs {b}"] | |
| 100 | +M7 = "M7 grille d'ajustement hédonique"; M2 = "M2 moyenne des 5 plus proches (temps seul)"; M6 = "M6 grille d'ajustement conventionnelle" | |
| 101 | +M8a = "M8a grille locale (régression sur le voisinage)"; M8b = "M8b ajustements non linéaires (GBM)"; M11b = "M11b gradient boosting (avec rôle)"; M11a = "M11a gradient boosting (sans rôle)" | |
| 102 | +M0 = "M0 meilleure configuration empirique (choisie en 2024, couverture ≥ 97 %)"; M12 = "M12 hybride comparables + GBM (poids appris en 2024)"; M13 = "M13 ratio au rôle (5 plus proches, même rôle)"; M9 = "M9 réconciliation optimisée (poids appris en 2024)"; M3 = "M3 sélection par similarité physique"; M1 = "M1 médiane des 3 plus proches (temps seul)"; M10 = "M10 hédonique globale directe" | |
| 103 | +dens = R["strates_dens"]; densn = R["strates_dens_n"] | |
| 104 | +h1 = H["H1"] | |
| 105 | +marg = R["E2_marginal"] | |
| 106 | +cond_g = R["regles_conditionnelles_global_test"] | |
| 107 | +nT = R["n_targets"] | |
| 108 | + | |
| 109 | +txt = f"""--- | |
| 110 | +title: "La méthode des comparables à l'épreuve des données : validation empirique à grande échelle sur {n_(D['propre'])} ventes résidentielles québécoises" | |
| 111 | +subtitle: "Sélection, nombre, ajustements, pondération et réconciliation des ventes comparables — rétro-évaluation point-dans-le-temps sur les transactions Vrai Prix 2021-2026" | |
| 112 | +author: "Simon-Pierre Boucher — version {VERSION}" | |
| 113 | +date: "{time.strftime('%d/%m/%Y')}" | |
| 114 | +lang: fr-CA | |
| 115 | +--- | |
| 116 | + | |
| 117 | +# Résumé | |
| 118 | + | |
| 119 | +**Question.** La méthode de comparaison des ventes, telle qu'enseignée et pratiquée en évaluation immobilière, produit-elle des estimations fiables, et lesquelles de ses règles (choix des comparables, nombre, rayon, ancienneté, ajustements, pondération, réconciliation) sont empiriquement justifiées ? | |
| 120 | + | |
| 121 | +**Données et protocole.** {n_(D['propre'])} ventes de résidences unifamiliales du Québec (registre foncier jumelé au rôle d'évaluation, Vrai Prix, {D['dates'][0]} → {D['dates'][1]}), après un nettoyage journalisé de {n_(D['brut'])} transactions brutes. Pour chacune des **{n_(nT['val'] + nT['test'])} ventes cibles** de 2024 (validation, {n_(nT['val'])}) et de 2025-2026 (test hors échantillon, {n_(nT['test'])}), le prix est masqué et la valeur est estimée avec les seules ventes conclues **avant** la date de la cible, un indice de prix ré-estimé chaque mois avec les seules ventes antérieures, et des grilles d'ajustement et modèles ré-estimés chaque trimestre. **2 256 configurations** de la méthode des comparables et treize modèles de référence sont appliqués aux mêmes cibles ; les différences sont testées par bootstrap par grappes (municipalité × trimestre). | |
| 122 | + | |
| 123 | +**Résultats principaux (test 2025-2026).** | |
| 124 | + | |
| 125 | +1. **Les ajustements sont le facteur dominant.** Prendre la moyenne des cinq ventes les plus proches, ajustées seulement pour le temps (M2), donne une erreur absolue médiane (MdAPE) de **{pct(comp[M2]['mdape'])}** ; une grille d'ajustement hédonique (M7) la ramène à **{pct(comp[M7]['mdape'])}** (−{f1(gain_adj)} points, IC 95 % de la différence de MdAPE [{f2(pr(M7, M2)['ci_mdape'][0])} ; {f2(pr(M7, M2)['ci_mdape'][1])}]). Une grille conventionnelle stylisée (M6) fait presque aussi bien ({pct(comp[M6]['mdape'])}) : ce qui compte, c'est d'ajuster, plus que la finesse de la grille. | |
| 126 | +2. **Choisir des comparables semblables vaut autant qu'ajuster.** Sans aucun ajustement physique, sélectionner les cinq ventes les plus **semblables** (aire, terrain, âge, lien physique, étages) plutôt que les cinq plus **proches** fait passer la MdAPE de {pct(comp[M2]['mdape'])} à **{pct(comp[M3]['mdape'])}** (H1 confirmée : un comparable un peu plus éloigné mais semblable vaut mieux qu'un voisin dissemblable). | |
| 127 | +3. **Le nombre optimal de comparables est bien supérieur à trois.** Avec la grille hédonique, la MdAPE décroît de {pct(k_curve[1])} (k = 1) à {pct(k_curve[3])} (k = 3), {pct(k_curve[5])} (k = 5), {pct(k_curve[10])} (k = 10) et atteint son minimum à **k = {k_best}** ({pct(k_curve[k_best])}) avant de remonter ({pct(k_curve[50])} à k = 50) ; l'optimum dépend de la densité du marché (H2 confirmée). | |
| 128 | +4. **Le rayon et la fenêtre temporelle doivent être larges.** La meilleure configuration de la surface Erreur(k, d, t) choisie en 2024 est « {LAB[e1b['sel']]}, k = {int(e1b['k'])}, ≤ {e1b['d']} km, {int(e1b['T'])} mois, {LAB[e1b['adj']]} » ({pct(e1b['mdape_val'])} en validation → **{pct(e1b['mdape_test'])}** en test). Les rayons de 0,5-1 km et les fenêtres de 3 mois, privilégiés par la pratique, sont dominés dès que les comparables sont ajustés. | |
| 129 | +5. **La pondération et le mode de réconciliation comptent peu** (effets marginaux moyens de {f1(min(marg['w'].values()))} à {f1(max(marg['w'].values()))} points selon la pondération, médiane {f1(marg['recon']['median'])} contre moyenne {f1(marg['recon']['mean'])}), bien moins que la sélection ({f1(min(marg['sel'].values()))}-{f1(max(marg['sel'].values()))}) et l'ajustement ({f1(min(marg['adj'].values()))}-{f1(max(marg['adj'].values()))}). | |
| 130 | +6. **La meilleure méthode des comparables, entièrement transparente et choisie en 2024, atteint {pct(comp[M0]['mdape'])} en 2025-2026** (M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]}), contre **{pct(comp[M11b]['mdape'])}** pour un gradient boosting ré-entraîné chaque trimestre avec la valeur au rôle (M11b) et {pct(comp[M11a]['mdape'])} sans le rôle (M11a). L'hybride comparables + GBM (M12) atteint **{pct(comp[M12]['mdape'])}**. Le coût de la transparence est donc de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''} de MdAPE face au meilleur modèle opaque, et il s'annule dans les sous-marchés homogènes (H5). | |
| 131 | +7. **Le ratio au rôle est un comparable redoutable** : rôle de la cible × ratio prix/rôle médian des cinq plus proches ventes du même rôle (M13) donne {pct(comp[M13]['mdape'])}, mieux que toute grille d'ajustement physique, mais seulement pour {pct(comp[M13]['couverture'])} des cibles. | |
| 132 | +8. **La fiabilité dépend d'abord de la liquidité du sous-marché** : la MdAPE de M7 passe de {pct(dens['≥ 200']['M7'])} là où plus de 200 ventes ont eu lieu à moins de 2 km dans l'année à {pct(dens['< 5']['M7'])} là où il y en a eu moins de 5 ; des règles conditionnelles à la densité (système de recommandation) obtiennent {pct(cond_g['mdape'])} globalement avec {pct(cond_g['couverture'])} de couverture. | |
| 133 | + | |
| 134 | +**Portée.** Nous validons la précision prédictive de règles professionnelles *simulées* sur des caractéristiques observables (aire, terrain, âge, étages, lien physique, rôle) ; les ajustements non observables (garage, piscine, chambres, qualité, rénovations) restent dans l'erreur résiduelle, et le jugement réel d'un évaluateur n'est pas reproduit. | |
| 135 | + | |
| 136 | +# 1. Données | |
| 137 | + | |
| 138 | +## 1.1 Source et nettoyage | |
| 139 | + | |
| 140 | +Instantané de la base de production Vrai Prix pris le 2026-10-10 : {n_(766596)} transactions résidentielles et 3,75 M d'unités d'évaluation. Nous retenons les ventes de type « unifamilial » jumelées à une unité du rôle, et appliquons des règles de nettoyage explicites et journalisées — rien n'est éliminé silencieusement. | |
| 141 | + | |
| 142 | +{md_table(excl)} | |
| 143 | +Jeu propre : **{n_(D['propre'])} ventes** ({D['dates'][0]} → {D['dates'][1]}) ; prix médian {n_(D['prix']['50%'])} $, moyenne {n_(D['prix']['mean'])} $. | |
| 144 | + | |
| 145 | +Deux règles méritent un commentaire. R8 écarte les ratios prix/rôle extrêmes (< 0,33 ou > 3) : faute d'indicateur de lien de dépendance entre les parties, c'est notre proxy de « transaction non représentative » — un évaluateur les écarterait aussi. R6b écarte les ventes conclues l'année même de la construction : leur prix médian ({n_(346108)} $) et leur ratio au rôle (1,15) sont ceux de terrains ou de constructions inachevées, non de maisons achevées (450-480 k$ et 1,3 aux âges 1-4 ans). | |
| 146 | + | |
| 147 | +## 1.2 Répartition | |
| 148 | + | |
| 149 | +| Période | Rôle | Ventes | | |
| 150 | +|---|---|---| | |
| 151 | +| 2021-01 → 2023-12 | développement | {n_(D['par_periode']['train'])} | | |
| 152 | +| 2024 | validation (choix des configurations) | {n_(D['par_periode']['val'])} | | |
| 153 | +| 2025-01 → 2026-09 | test hors échantillon | {n_(D['par_periode']['test'])} | | |
| 154 | + | |
| 155 | +Régions les plus représentées : {', '.join(f"{k} ({n_(v)})" for k, v in list(D['par_region'].items())[:6])}. | |
| 156 | + | |
| 157 | +{{ width=85% }} | |
| 158 | + | |
| 159 | +## 1.3 Caractéristiques disponibles et limites | |
| 160 | + | |
| 161 | +Observées pour chaque vente : date, prix, coordonnées, municipalité, unité de voisinage, aire habitable, superficie du terrain, année de construction, nombre d'étages, lien physique (détaché, jumelé, en rangée), genre de construction, valeur au rôle en vigueur à la vente et sa part terrain. Non observées : garage, piscine, nombre de chambres et de salles de bains, qualité, état, rénovations. Les grilles d'ajustement testées ne portent donc que sur les premières ; les secondes sont une source d'erreur commune à toutes les méthodes. | |
| 162 | + | |
| 163 | +# 2. Protocole | |
| 164 | + | |
| 165 | +## 2.1 Rétro-évaluation point-dans-le-temps | |
| 166 | + | |
| 167 | +Pour chaque cible : (1) le prix est masqué ; (2) le bassin de comparables est constitué des ventes conclues strictement avant la date de la cible (la cible elle-même, si elle a déjà été vendue, est exclue : on évalue par comparables, non par vente répétée), dans un rayon de 25 km et une fenêtre de 24 mois ; (3) l'indice temporel régional utilisé pour ramener chaque comparable au niveau du dernier mois connu est ré-estimé chaque mois avec les seules ventes antérieures (régression hédonique à effets fixes de municipalité et effets mensuels, par région administrative, rétrécie vers l'indice provincial quand les ventes sont rares) ; (4) les coefficients d'ajustement hédoniques, les grilles régionales et les modèles d'apprentissage sont ré-estimés chaque trimestre avec les seules ventes antérieures au trimestre ; (5) chaque configuration produit une valeur et, le cas échéant, une dispersion des prix ajustés. | |
| 168 | + | |
| 169 | +Le niveau de prix du mois de la cible n'est jamais connu : toutes les méthodes travaillent au niveau du mois précédent, ce qui reproduit la situation réelle d'un évaluateur. | |
| 170 | + | |
| 171 | +## 2.2 Configurations testées | |
| 172 | + | |
| 173 | +**E1 — surface Erreur(k, d, t)** : k ∈ {{1, 3, 5, 7, 10, 15, 20, 30, 50}}, d ∈ {{0,5 ; 1 ; 2 ; 5 ; 10 ; 25 km}}, t ∈ {{3, 6, 12, 24 mois}}, sélection ∈ {{plus proches, plus semblables, compromis}}, ajustement ∈ {{temps seul, grille hédonique}}, médiane. 1 296 configurations. | |
| 174 | + | |
| 175 | +**E2 — plan factoriel des règles** aux ancres (k, d, t) = (3, 2 km, 6 mois), (5, 5 km, 12 mois), (10, 10 km, 24 mois) : sélection (4) × ajustement (8 : aucun, temps seul, grille conventionnelle, hédonique globale, hédonique régionale, régression locale, GBM, ratio au rôle) × pondération (5) × réconciliation (2). 960 configurations. | |
| 176 | + | |
| 177 | +*Grille conventionnelle stylisée* (M6) : aire habitable ajustée à la moitié du prix au m² du comparable ; terrain à la moitié de la valeur foncière implicite au m² (plafond ±15 %) ; âge à −0,5 % par année d'écart (plafond ±20 %) ; ±5 % entre détaché et attaché. *Grille hédonique* (M7) : coefficients d'une régression du log du prix sur les caractéristiques, à effets fixes de municipalité (within), ré-estimée chaque trimestre. *Régression locale* (M8a) : mêmes coefficients ré-estimés sur les ventes à moins de 10 km des 24 derniers mois (≥ 40 ventes, ridge vers la grille régionale). *GBM* (M8b) : l'ajustement est la différence des prédictions d'un gradient boosting entre la cible et le comparable. *Ratio au rôle* (M13) : valeur au rôle de la cible × ratio prix/rôle du comparable, pour les comparables de la même municipalité et du même rôle triennal. | |
| 178 | + | |
| 179 | +## 2.3 Mesures | |
| 180 | + | |
| 181 | +Erreur absolue en pourcentage APE = |P̂ − P| / P ; MdAPE (médiane), MAPE (moyenne), biais médian signé, parts à ±5, ±10 et ±20 %, couverture (part des cibles pour lesquelles la configuration produit une valeur — une configuration trop restrictive ne « rate » pas, elle s'abstient, et cela doit se voir). Les différences entre méthodes sont des différences d'erreurs appariées sur les mêmes cibles, avec intervalles de confiance à 95 % par bootstrap par grappes municipalité × trimestre (400 réplications). | |
| 182 | + | |
| 183 | +# 3. Résultats | |
| 184 | + | |
| 185 | +## 3.1 Combien de comparables ? À quelle distance ? De quelle ancienneté ? | |
| 186 | + | |
| 187 | +{{ width=100% }} | |
| 188 | + | |
| 189 | +{{ width=90% }} | |
| 190 | + | |
| 191 | +**Sélection par distance, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 192 | + | |
| 193 | +{surf_tab('geo', 'hedo', 12)} | |
| 194 | +**Sélection par distance, temps seulement, fenêtre 12 mois — MdAPE test (%)** | |
| 195 | + | |
| 196 | +{surf_tab('geo', 'none', 12)} | |
| 197 | +**Sélection par similarité physique, grille hédonique, fenêtre 12 mois — MdAPE test (%)** | |
| 198 | + | |
| 199 | +{surf_tab('sim', 'hedo', 12)} | |
| 200 | +Lecture. (i) Sans ajustement, l'erreur décroît avec k et avec le rayon tant que l'on reste sous quelques kilomètres, puis se stabilise ; (ii) avec la grille hédonique, elle décroît nettement plus et l'optimum se déplace vers des k et des rayons plus grands ; (iii) la fenêtre de 3 mois est systématiquement dominée par 12-24 mois dès que l'on ajuste pour le temps. Meilleures configurations de E1 par sélection × ajustement (choisies en 2024, évaluées en 2025-2026) : | |
| 201 | + | |
| 202 | +{md_table(pd.DataFrame(R['E1_best_by_sel_adj']).T.reset_index().rename(columns={'index': 'sélection | ajustement', 'mdape_val': 'MdAPE val.', 'mdape_test': 'MdAPE test', 'couverture_test': 'couverture test'}), ['sélection | ajustement', 'k', 'd', 'T', 'MdAPE val.', 'MdAPE test', 'couverture test'], fmt={'k': lambda v: str(int(v)), 'T': lambda v: str(int(v)), 'd': lambda v: f1(v)})} | |
| 203 | +### k optimal selon la densité (H2) | |
| 204 | + | |
| 205 | +{h2_md} | |
| 206 | +*(sélection par distance, ≤ 10 km, 12 mois, grille hédonique ; MdAPE test en %, par nombre de ventes disponibles à moins de 2 km au cours des 12 mois précédents)* | |
| 207 | + | |
| 208 | +Sans ajustement : | |
| 209 | + | |
| 210 | +{h2n_md} | |
| 211 | +{{ width=100% }} | |
| 212 | + | |
| 213 | +## 3.2 Quels ajustements sont justifiés ? | |
| 214 | + | |
| 215 | +{{ width=90% }} | |
| 216 | + | |
| 217 | +{t_adj_md} | |
| 218 | +Trois enseignements. **L'ajustement temporel seul** vaut {f1(t_adj.set_index('adj').loc['raw', 'mdape_test'] - t_adj.set_index('adj').loc['none', 'mdape_test'])} point ; **n'importe quelle grille physique** vaut {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['conv', 'mdape_test'])} à {f1(t_adj.set_index('adj').loc['none', 'mdape_test'] - t_adj.set_index('adj').loc['gbm', 'mdape_test'])} points de plus ; **les raffinements** (grille régionale, régression locale, ajustements non linéaires) apportent peu face à la grille hédonique globale, et la régression locale est même {('moins bonne' if t_adj.set_index('adj').loc['local', 'mdape_test'] > t_adj.set_index('adj').loc['hedo', 'mdape_test'] else 'meilleure')} ({pct(t_adj.set_index('adj').loc['local', 'mdape_test'])} contre {pct(t_adj.set_index('adj').loc['hedo', 'mdape_test'])}). | |
| 219 | + | |
| 220 | +### Grilles locales et petits échantillons (H3) | |
| 221 | + | |
| 222 | +MdAPE test (%) par type d'ajustement et densité, ancre (5, 5 km, 12 mois) : | |
| 223 | + | |
| 224 | +{h3_md} | |
| 225 | +Ancre (3, 2 km, 6 mois) : | |
| 226 | + | |
| 227 | +{h3s_md} | |
| 228 | +## 3.3 Comment choisir les comparables ? (H1) | |
| 229 | + | |
| 230 | +Ancre (5, 5 km, 12 mois), grille hédonique, médiane : | |
| 231 | + | |
| 232 | +{t_sel_md} | |
| 233 | +Test direct de H1 (k = 5, 12 mois, temps seulement) : les cinq voisins à moins de 1 km donnent une MdAPE de {pct(h1['geo_d1']['mdape_test'])} (couverture {pct(h1['geo_d1']['couverture_test'])}) ; les cinq ventes les plus semblables à moins de 10 km, {pct(h1['sim_d10']['mdape_test'])} (couverture {pct(h1['sim_d10']['couverture_test'])}) ; les cinq plus proches à moins de 10 km, {pct(h1['geo_d10']['mdape_test'])}. Avec la grille hédonique : plus proches {pct(h1['geo_d10_hedo']['mdape_test'])}, plus semblables {pct(h1['sim_d10_hedo']['mdape_test'])}, compromis {pct(h1['mix_d10_hedo']['mdape_test'])}. | |
| 234 | + | |
| 235 | +## 3.4 Pondération et réconciliation (H4) | |
| 236 | + | |
| 237 | +Ancre (5, 5 km, 12 mois), sélection par distance, grille hédonique : | |
| 238 | + | |
| 239 | +{t_wr_md} | |
| 240 | +Selon la densité (MdAPE test, %) : | |
| 241 | + | |
| 242 | +{h4_md} | |
| 243 | +Effets marginaux moyens (MdAPE test moyenne sur toutes les autres modalités de E2) : sélection {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['sel'].items())} ; ajustement {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['adj'].items())} ; pondération {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['w'].items())} ; réconciliation {', '.join(f"{LAB[k]} {f1(v)}" for k, v in marg['recon'].items())}. | |
| 244 | + | |
| 245 | +## 3.5 Compétition de modèles | |
| 246 | + | |
| 247 | +{{ width=95% }} | |
| 248 | + | |
| 249 | +{t_comp} | |
| 250 | +M0 : {LAB[m0['sel']]}, k = {int(m0['k'])}, ≤ {m0['d']} km, {int(m0['T'])} mois, {LAB[m0['adj']]}, pondération {LAB[m0['w']]}, {LAB[m0['recon']]} (meilleure des 2 256 configurations en 2024 parmi celles qui couvrent ≥ 97 % des cibles ; sans cette contrainte, la meilleure est « {LAB[m0u['sel']]}, k = {int(m0u['k'])}, ≤ {m0u['d']} km, {int(m0u['T'])} mois, {LAB[m0u['adj']]} » à {pct(m0u['mdape_test'])} mais pour {pct(m0u['couverture_test'])} des cibles seulement). Poids appris de M9 : {', '.join(f"{LAB[k]} {f2(v)}" for k, v in R['M9_weights'].items() if v > 0)}. Poids de M12 : {', '.join(f"{k} {f2(v)}" for k, v in R['M12_weights'].items() if v > 0)}. | |
| 251 | + | |
| 252 | +### Différences appariées (test ; bootstrap par grappes municipalité × trimestre) | |
| 253 | + | |
| 254 | +{t_pairs} | |
| 255 | +Un Δ négatif signifie que A est plus précis que B ; l'intervalle qui exclut zéro signale une différence robuste aux dépendances spatiales et temporelles. | |
| 256 | + | |
| 257 | +## 3.6 Où la méthode tient-elle, où défaille-t-elle ? | |
| 258 | + | |
| 259 | +{{ width=85% }} | |
| 260 | + | |
| 261 | +**Par densité** (ventes à moins de 2 km dans les 12 mois précédents) — MdAPE test (%) : | |
| 262 | + | |
| 263 | +{strata_md('dens', 'densité')} | |
| 264 | +**Par région administrative** : | |
| 265 | + | |
| 266 | +{strata_md('region_nom', 'région')} | |
| 267 | +**Par quintile de prix** : | |
| 268 | + | |
| 269 | +{strata_md('prix_q', 'quintile de prix')} | |
| 270 | +**Par âge du bâtiment** : | |
| 271 | + | |
| 272 | +{strata_md('age_cl', 'âge')} | |
| 273 | +**Par lien physique** (0 détaché, 1 jumelé, 2 en rangée, 3 intégré) : | |
| 274 | + | |
| 275 | +{strata_md('lien', 'lien physique')} | |
| 276 | +### Marchés homogènes et hétérogènes (H5) | |
| 277 | + | |
| 278 | +Terciles de la dispersion (écart-type du log des prix ajustés) des dix ventes les plus proches à ≤ 10 km / 24 mois — MdAPE test (%) : | |
| 279 | + | |
| 280 | +{h5_md} | |
| 281 | +### Stabilité dans le temps | |
| 282 | + | |
| 283 | +{stab_md} | |
| 284 | +Le biais trimestriel commun à toutes les méthodes reflète le fait que le niveau de prix du mois courant est inconnu (toutes travaillent au niveau du mois précédent) et la saisonnalité des prix. | |
| 285 | + | |
| 286 | +## 3.7 Intervalles de valeur | |
| 287 | + | |
| 288 | +La dispersion des prix ajustés des comparables retenus fournit un intervalle de valeur. Avec z = 1,28 (couverture nominale de 80 % si les erreurs étaient normales), puis avec un z calibré sur la validation 2024 : | |
| 289 | + | |
| 290 | +{iv_md} | |
| 291 | +## 3.8 Vers un système de recommandation des comparables | |
| 292 | + | |
| 293 | +Règle choisie en 2024 pour chaque strate de densité (parmi les 2 256 configurations couvrant ≥ 95 % des cibles de la strate), évaluée en 2025-2026 : | |
| 294 | + | |
| 295 | +{rc_md} | |
| 296 | +Globalement, ces règles conditionnelles atteignent une MdAPE de **{pct(cond_g['mdape'])}** ({pct(cond_g['p10'])} des estimations à ±10 %, couverture {pct(cond_g['couverture'])}), contre {pct(comp[M0]['mdape'])} pour la meilleure règle unique. | |
| 297 | + | |
| 298 | +# 4. Discussion | |
| 299 | + | |
| 300 | +**Ce que la méthode des comparables fait bien.** Correctement paramétrée — comparables semblables, nombre d'une dizaine, rayon et fenêtre larges, ajustements hédoniques ou même conventionnels — elle atteint une précision ({pct(comp[M0]['mdape'])}) que ne dépasse le meilleur modèle d'apprentissage ré-entraîné chaque trimestre que de {f1(comp[M0]['mdape'] - comp[M11b]['mdape'])} point{'s' if abs(comp[M0]['mdape'] - comp[M11b]['mdape']) >= 2 else ''}, tout en restant intégralement explicable : chaque valeur est la médiane de quelques ventes identifiées, ajustées par des coefficients lisibles. | |
| 301 | + | |
| 302 | +**Ce qui contredit la pratique.** Trois comparables, un rayon de quelques centaines de mètres et une fenêtre de trois mois — les conventions les plus répandues — sont dominés sur toute la surface Erreur(k, d, t) : {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].mdape_test.iloc[0])} de MdAPE (k = 3, ≤ 1 km, 3 mois, grille hédonique, couverture {pct(surf[(surf.sel == 'geo') & (surf.adj == 'hedo') & (surf.k == 3) & (surf.d == 1) & (surf['T'] == 3)].couverture_test.iloc[0])}) contre {pct(e1b['mdape_test'])} pour la meilleure configuration. La rareté des comparables « parfaits » est un argument pour élargir la recherche et ajuster, non pour se restreindre. | |
| 303 | + | |
| 304 | +**Le rôle d'évaluation comme information.** Le ratio au rôle (M13) et le GBM avec rôle (M11b) dominent leurs équivalents sans rôle : la valeur au rôle encapsule des caractéristiques non observées ici (qualité, dépendances, état) et une appréciation locale que les modèles globaux ne captent pas. Les modèles globaux sans rôle (M10, M11a) présentent d'ailleurs un biais négatif en 2025-2026 : l'appréciation est spatialement hétérogène et un indice régional, même ré-estimé chaque trimestre, la sous-estime là où les ventes se concentrent. | |
| 305 | + | |
| 306 | +**Limites.** (1) Règles professionnelles simulées, non dossiers d'évaluateurs. (2) Caractéristiques limitées à celles du rôle et du registre. (3) Délai de publication des ventes ignoré. (4) Un seul type de bien (unifamilial) ; condos et plex restent à traiter. (5) Les configurations ont été choisies sur 2024 puis évaluées sur 2025-2026 : les écarts entre validation et test (tableaux ci-dessus) mesurent la stabilité des règles, et ils sont faibles. | |
| 307 | + | |
| 308 | +# 5. Reproductibilité | |
| 309 | + | |
| 310 | +Code : `pipeline/01_dataset.py` → `06_report.py` (Python 3.14, pandas, numpy, scipy, lightgbm, matplotlib) ; sorties versionnées `out/{VERSION}/` (configurations `configs_E1.csv`, `configs_E2.csv` ; estimations `est_E1.npy`, `est_E2.npy` ; cibles `targets.parquet` ; tables `tables/*.csv` ; `results.json`). Exécution : `./run_all.sh` sur le nœud de calcul (M2U64), `./make_pdf.sh` sur le poste de travail. | |
| 311 | +""" | |
| 312 | +(DOCS / "RAPPORT.md").write_text(txt, encoding="utf-8") | |
| 313 | +log(f"rapport écrit : {DOCS / 'RAPPORT.md'} ({len(txt.split()):,} mots)") | |
added
M4M64bomparables-validation/pipeline/common.py
+85 −0
@@ -0,0 +1,85 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""Validation empirique de la méthode des comparables — fonctions communes. | |
| 3 | +Données : instantané Vrai Prix (transactions ⋈ units) du 2026-10-10. | |
| 4 | +Toutes les sorties sont versionnées sous out/<version>/. | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | +import json, math, os, sqlite3, time | |
| 8 | +from pathlib import Path | |
| 9 | +import numpy as np | |
| 10 | +import pandas as pd | |
| 11 | + | |
| 12 | +ROOT = Path(os.environ.get("CV_ROOT", Path(__file__).resolve().parents[1])) | |
| 13 | +DB = Path(os.environ.get("CV_DB", ROOT / "data" / "vraiprix-20261010.db")) | |
| 14 | +REF = ROOT / "ref" | |
| 15 | +VERSION = os.environ.get("CV_VERSION", time.strftime("%Y-%m-%d")) | |
| 16 | +OUT = ROOT / "out" / VERSION | |
| 17 | +OUT.mkdir(parents=True, exist_ok=True) | |
| 18 | +(OUT / "fig").mkdir(exist_ok=True) | |
| 19 | + | |
| 20 | +# Périodes (séparation chronologique) | |
| 21 | +TRAIN_END = "2023-12-31" # développement | |
| 22 | +VAL_START, VAL_END = "2024-01-01", "2024-12-31" # choix des configurations | |
| 23 | +TEST_START, TEST_END = "2025-01-01", "2026-09-29" # évaluation finale hors échantillon | |
| 24 | +MONTH0 = "2021-01" # premier mois des données | |
| 25 | + | |
| 26 | +def log(msg: str) -> None: | |
| 27 | + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) | |
| 28 | + | |
| 29 | +def connect(path: Path = DB) -> sqlite3.Connection: | |
| 30 | + return sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=60) | |
| 31 | + | |
| 32 | +def _json_default(o): | |
| 33 | + if isinstance(o, np.integer): return int(o) | |
| 34 | + if isinstance(o, np.floating): return None if np.isnan(o) else float(o) | |
| 35 | + if isinstance(o, np.bool_): return bool(o) | |
| 36 | + if isinstance(o, np.ndarray): return o.tolist() | |
| 37 | + if isinstance(o, pd.Timestamp): return o.strftime("%Y-%m-%d") | |
| 38 | + if isinstance(o, float) and math.isnan(o): return None | |
| 39 | + return str(o) | |
| 40 | + | |
| 41 | +def save_json(obj, name: str) -> Path: | |
| 42 | + p = OUT / name; p.parent.mkdir(parents=True, exist_ok=True) | |
| 43 | + with open(p, "w", encoding="utf-8") as f: | |
| 44 | + json.dump(obj, f, ensure_ascii=False, indent=2, default=_json_default) | |
| 45 | + return p | |
| 46 | + | |
| 47 | +def load_json(name: str): | |
| 48 | + with open(OUT / name, encoding="utf-8") as f: | |
| 49 | + return json.load(f) | |
| 50 | + | |
| 51 | +def save_parquet(df: pd.DataFrame, name: str) -> Path: | |
| 52 | + p = OUT / name; df.to_parquet(p, index=False); return p | |
| 53 | + | |
| 54 | +def load_parquet(name: str) -> pd.DataFrame: | |
| 55 | + return pd.read_parquet(OUT / name) | |
| 56 | + | |
| 57 | +def month_index(dates) -> np.ndarray: | |
| 58 | + """Mois entier depuis MONTH0 (2021-01 = 0).""" | |
| 59 | + d = pd.to_datetime(dates) | |
| 60 | + y0, m0 = map(int, MONTH0.split("-")) | |
| 61 | + return ((d.dt.year - y0) * 12 + (d.dt.month - m0)).to_numpy() | |
| 62 | + | |
| 63 | +def to_xy_km(lat, lng, lat0=46.8, lng0=-71.2): | |
| 64 | + """Projection équirectangulaire locale (km) : erreur < 1 % pour des distances < 50 km.""" | |
| 65 | + x = (np.asarray(lng, float) - lng0) * 111.32 * np.cos(np.radians(np.asarray(lat, float))) | |
| 66 | + y = (np.asarray(lat, float) - lat0) * 110.57 | |
| 67 | + return x, y | |
| 68 | + | |
| 69 | +def load_repertoire() -> pd.DataFrame: | |
| 70 | + mun = pd.read_csv(REF / "MUN.csv", dtype=str, encoding="utf-8-sig") | |
| 71 | + mun = mun.rename(columns=str.lower) | |
| 72 | + out = pd.DataFrame({ | |
| 73 | + "code_mun": mun["mcode"].str.zfill(5), | |
| 74 | + "mun_nom": mun["munnom"], | |
| 75 | + "region": mun["regadm"].str.extract(r"\((\d+)\)")[0], | |
| 76 | + "region_nom": mun["regadm"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 77 | + "mrc": mun["mrc"].str.extract(r"\((\d+)\)")[0], | |
| 78 | + "mrc_nom": mun["mrc"].str.replace(r"\s*\(\d+\)", "", regex=True), | |
| 79 | + }) | |
| 80 | + return out.drop_duplicates("code_mun") | |
| 81 | + | |
| 82 | +# Codages catégoriels (ordre stable) | |
| 83 | +LIEN = {"Détaché": 0, "Jumelé": 1, "En rangée 1 côté": 2, "En rangée plus de 1 côté": 2, "Intégré": 3} | |
| 84 | +GENRE = {"De plain-pied": 0, "Unimodulaire": 0, "À étages entiers": 1, "À niveaux décalés": 2, "À étage mansardé": 3} | |
| 85 | +BTYPE = {"single-story": 0, "full-story": 1, "mansard": 2, "split-level": 3, "single-wide": 4} | |
added
M4M64bomparables-validation/run_all.sh
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Chaîne complète sur le nœud de calcul (M2U64). Usage : CV_VERSION=2026-10-10 ./run_all.sh [étape de départ] | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")" | |
| 5 | +PY=${PY:-$HOME/uqo-eval-terrains/env/bin/python} | |
| 6 | +export CV_VERSION=${CV_VERSION:-$(date +%F)} | |
| 7 | +START=${1:-1} | |
| 8 | +for s in 01_dataset 02_index 03_models 04_backtest 05_analyze 06_report; do | |
| 9 | + n=${s%%_*} | |
| 10 | + [ "$((10#$n))" -lt "$START" ] && continue | |
| 11 | + echo "=== $s ($(date +%T)) ===" | |
| 12 | + (cd pipeline && $PY $s.py) | |
| 13 | +done | |
| 14 | +echo "=== terminé $(date +%T) ===" | |
modified
README.md
+16 −0
@@ -31,3 +31,19 @@ Variables : `CV_VERSION` (dossier de sortie), `CV_SAMPLE=3000` (échantillon de | ||
| 31 | 31 | |
| 32 | 32 | Instantané `vraiprix.db` pris sur M3U96a (`~/apps/vrai-prix/data`) le 2026-10-10 : 766 596 transactions, 3,75 M d'unités du rôle. |
| 33 | 33 | Jeu propre : ventes `unifamilial` jumelées au rôle, règles R0-R11 journalisées dans `out/<v>/exclusions.csv`. |
| 34 | + | |
| 35 | +## Working paper (UQO WP 11) | |
| 36 | + | |
| 37 | +`paper/` : article en anglais au gabarit UQO (`main.tex`, `sections/*.tex`, `appendix/`, `references.bib`). | |
| 38 | +Les chiffres du texte sont des macros (`tables/numbers.tex`) et les tableaux sont générés depuis `out/<version>/` : | |
| 39 | + | |
| 40 | +```bash | |
| 41 | +./make_pdf.sh # rapatrie out/ et compile docs/RAPPORT.pdf (français) | |
| 42 | +cd paper && python3 make_paper_tables.py && latexmk -pdf main.tex # tables + macros + PDF du WP | |
| 43 | +``` | |
| 44 | + | |
| 45 | +## Calcul réparti | |
| 46 | + | |
| 47 | +`run_distributed.sh` : distribue les modèles trimestriels aux nœuds, lance la rétro-évaluation en 15 fragments | |
| 48 | +(M3U96a ×4, M3U96b ×4, M2U64 ×3, M4M64a ×2, M4M64b ×2), rassemble sur M2U64 (`04b_gather.py`) puis exécute 05 et 06. | |
| 49 | +Ré-estimation trimestrielle répartie : `CV_ASOF=54,57 CV_THREADS=28 ../env/bin/python 03_models.py` sur chaque nœud. | |
added
paper/.latexmkrc
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +$pdf_mode = 1; | |
| 2 | +$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S'; | |
| 3 | +$bibtex_use = 2; | |
| 4 | +$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg'; | |
| 5 | +@default_files = ('main.tex'); | |
added
paper/Makefile
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +.PHONY: all clean distclean | |
| 2 | + | |
| 3 | +all: | |
| 4 | + latexmk | |
| 5 | + | |
| 6 | +clean: | |
| 7 | + latexmk -c | |
| 8 | + | |
| 9 | +distclean: | |
| 10 | + latexmk -C | |
| 11 | + rm -f *.synctex.gz *.run.xml *.bbl *.bcf | |
added
paper/appendix/appendix.tex
+62 −0
@@ -0,0 +1,62 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\begin{appendices} | |
| 3 | +\section{Additional tables} | |
| 4 | +\label{app:tables} | |
| 5 | +% ============================================================================ | |
| 6 | + | |
| 7 | +\begin{table}[H] | |
| 8 | +\centering | |
| 9 | +\caption{Coverage of the error surface: share of test targets (\%) for which a value is returned, by radius and window} | |
| 10 | +\label{tab:surface_cov} | |
| 11 | +\input{tables/surface_cov} | |
| 12 | +\end{table} | |
| 13 | + | |
| 14 | +\begin{table}[H] | |
| 15 | +\centering | |
| 16 | +\caption{Error surface without physical adjustment: out-of-sample MdAPE (\%), selection by distance, time adjustment only, twelve-month window} | |
| 17 | +\label{tab:surface_none} | |
| 18 | +\input{tables/surface_geo_none} | |
| 19 | +\end{table} | |
| 20 | + | |
| 21 | +\begin{table}[H] | |
| 22 | +\centering | |
| 23 | +\caption{Error surface with selection by similarity: out-of-sample MdAPE (\%), hedonic grid, twelve-month window} | |
| 24 | +\label{tab:surface_sim} | |
| 25 | +\input{tables/surface_sim_hedo} | |
| 26 | +\end{table} | |
| 27 | + | |
| 28 | +\begin{table}[H] | |
| 29 | +\centering | |
| 30 | +\caption{Best configuration of the error surface by selection and adjustment (chosen on 2024, evaluated on 2025--2026)} | |
| 31 | +\label{tab:e1best} | |
| 32 | +\input{tables/e1_best} | |
| 33 | +\end{table} | |
| 34 | + | |
| 35 | +\begin{table}[H] | |
| 36 | +\centering | |
| 37 | +\caption{Out-of-sample MdAPE (\%) by administrative region} | |
| 38 | +\label{tab:regions} | |
| 39 | +\footnotesize | |
| 40 | +\input{tables/regions} | |
| 41 | +\end{table} | |
| 42 | + | |
| 43 | +\begin{table}[H] | |
| 44 | +\centering | |
| 45 | +\caption{Out-of-sample MdAPE (\%) by price quintile, age and physical link} | |
| 46 | +\label{tab:price} | |
| 47 | +\footnotesize | |
| 48 | +\input{tables/strata_other} | |
| 49 | +\end{table} | |
| 50 | + | |
| 51 | +\section{Implementation details} | |
| 52 | +\label{app:impl} | |
| 53 | + | |
| 54 | +\paragraph{Computation.} The backtest evaluates \nConfigs{} configurations on \nTargetsAll{} targets. For each target the candidate pool is retrieved with a $k$-d tree on projected coordinates, the time, conventional, hedonic, regional, local, GBM and ratio adjustments are computed once for every candidate, and the selection--window--count--weighting--reconciliation combinations are evaluated as operations on the resulting vectors. The computation is embarrassingly parallel over targets and was distributed over five Apple-silicon nodes of a local cluster; a full run takes under ten minutes. The quarterly re-estimation of the adjustment grids and gradient-boosting benchmarks (eleven as-of dates) takes a comparable time. All code is in Python (pandas, NumPy, SciPy, LightGBM) and is available from the author. | |
| 55 | + | |
| 56 | +\paragraph{Local grid.} The local regression (adjustment scheme f) estimates, on the subject's candidates within 10~km and 24 months, the coefficients of log living area, log lot area, age, the two attached-dwelling indicators and the number of storeys, by ridge regression with penalty $\lambda=50$ towards the regional grid; with fewer than 40 candidates the regional grid is used. | |
| 57 | + | |
| 58 | +\paragraph{Gradient boosting.} LightGBM with 127 leaves, minimum 40 observations per leaf, learning rate 0.05, feature and bagging fractions 0.8, $L_2$ regularisation 1, L1 objective, number of trees by early stopping (patience 100) on the chronologically last 10\% of the training sample with a floor of 300 trees, then refit on the whole training sample. Features: log living area, log lot area, age, age$^2$, physical-link and construction-type indicators, storeys, new-build indicator, latitude, longitude, projected coordinates, region, building type, month of sale (capped at the last training month), and for M11b the log assessed value, its land share and its roll vintage. | |
| 59 | + | |
| 60 | +\paragraph{Cluster bootstrap.} Municipality--quarter cells of the test period are resampled with replacement 400 times; for each replication the mean of the paired APE differences is recomputed from cell sums, and for 150 replications the difference in medians is recomputed from the resampled observations. | |
| 61 | + | |
| 62 | +\end{appendices} | |
added
paper/figures/f1_mdape_vs_k.pdf
+0 −0
Binary file not shown.
added
paper/figures/f2_surface_d_T.pdf
+0 −0
Binary file not shown.
added
paper/figures/f3_ajustements.pdf
+0 −0
Binary file not shown.
added
paper/figures/f4_competition.pdf
+0 −0
Binary file not shown.
added
paper/figures/f5_densite.pdf
+0 −0
Binary file not shown.
added
paper/figures/f6_k_par_densite.pdf
+0 −0
Binary file not shown.
added
paper/figures/f7_indice.pdf
+0 −0
Binary file not shown.
added
paper/figures/main_figure.tex
+20 −0
@@ -0,0 +1,20 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Figure: Main Result | |
| 3 | +% ============================================================================ | |
| 4 | +\begin{figure}[htbp] | |
| 5 | + \centering | |
| 6 | + % --- Replace placeholder with actual figure file --- | |
| 7 | + % \includegraphics[width=0.85\textwidth]{figures/main_result.pdf} | |
| 8 | + % | |
| 9 | + % Placeholder box (remove when replacing with actual figure) | |
| 10 | + \fbox{\parbox{0.85\textwidth}{\centering\vspace{4cm} | |
| 11 | + \textit{[Insert figure here: figures/main\_result.pdf]} | |
| 12 | + \vspace{4cm}}} | |
| 13 | + % | |
| 14 | + \caption{Main Result --- Effect of $X$ on $Y$ Over Time} | |
| 15 | + \label{fig:main_result} | |
| 16 | + \begin{minipage}{0.85\textwidth} | |
| 17 | + \vspace{0.3cm} | |
| 18 | + \footnotesize\textit{Notes:} This figure plots \ldots{} over the sample period January~1990 to December~2023. The solid line represents \ldots{} The shaded area indicates the 95\% confidence interval. The dashed vertical line marks \ldots{} | |
| 19 | + \end{minipage} | |
| 20 | +\end{figure} | |
added
paper/main.aux
+205 −0
@@ -0,0 +1,205 @@ | ||
| 1 | +\relax | |
| 2 | +\providecommand \babel@aux [2]{\global \let \babel@toc \@gobbletwo } | |
| 3 | +\@nameuse{bbl@beforestart} | |
| 4 | +\providecommand\hyper@newdestlabel[2]{} | |
| 5 | +\providecommand\HyField@AuxAddToFields[1]{} | |
| 6 | +\providecommand\HyField@AuxAddToCoFields[2]{} | |
| 7 | +\bibstyle{aer} | |
| 8 | +\babel@aux{english}{} | |
| 9 | +\citation{appraisal2020,iaao2017} | |
| 10 | +\citation{colwell1983,vandell1991,gau1992,isakson2002} | |
| 11 | +\@writefile{toc}{\contentsline {section}{\numberline {1}Introduction}{3}{section.1}\protected@file@percent } | |
| 12 | +\newlabel{sec:introduction}{{1}{3}{Introduction}{section.1}{}} | |
| 13 | +\citation{colwell1983,vandell1991,gau1992,gau1994,isakson2002} | |
| 14 | +\citation{kang1991,lentz1998,pace1998} | |
| 15 | +\citation{mccluskey2013,mayer2019,bourassa2010,pace2020} | |
| 16 | +\citation{appraisal2020} | |
| 17 | +\citation{iaao2018} | |
| 18 | +\citation{diaz1990,diaz2001} | |
| 19 | +\citation{colwell1983} | |
| 20 | +\citation{vandell1991} | |
| 21 | +\citation{gau1992,gau1994} | |
| 22 | +\citation{isakson1986,isakson2002} | |
| 23 | +\citation{lentz1998} | |
| 24 | +\@writefile{toc}{\contentsline {section}{\numberline {2}Related literature}{5}{section.2}\protected@file@percent } | |
| 25 | +\newlabel{sec:literature}{{2}{5}{Related literature}{section.2}{}} | |
| 26 | +\@writefile{toc}{\contentsline {subsection}{\numberline {2.1}The sales comparison approach and its conventions}{5}{subsection.2.1}\protected@file@percent } | |
| 27 | +\citation{rosen1974} | |
| 28 | +\citation{kang1991,pace1998} | |
| 29 | +\citation{kang1991} | |
| 30 | +\citation{pace1998} | |
| 31 | +\citation{pace1997} | |
| 32 | +\citation{bourassa2010,case2004} | |
| 33 | +\citation{bailey1963} | |
| 34 | +\citation{case1989} | |
| 35 | +\citation{hill2013} | |
| 36 | +\citation{gloudemans1999} | |
| 37 | +\citation{pagourtzi2003} | |
| 38 | +\citation{mccluskey2013} | |
| 39 | +\citation{mayer2019} | |
| 40 | +\citation{bourassa2010} | |
| 41 | +\citation{case2004} | |
| 42 | +\citation{pace2020} | |
| 43 | +\citation{iaao2018} | |
| 44 | +\@writefile{toc}{\contentsline {subsection}{\numberline {2.2}Estimating adjustments: grids, hedonics and local models}{6}{subsection.2.2}\protected@file@percent } | |
| 45 | +\citation{mayer2019} | |
| 46 | +\citation{diaz1990,diaz2001,northcraft1987} | |
| 47 | +\@writefile{toc}{\contentsline {subsection}{\numberline {2.3}Automated valuation and machine learning}{7}{subsection.2.3}\protected@file@percent } | |
| 48 | +\@writefile{toc}{\contentsline {subsection}{\numberline {2.4}Appraiser behaviour and the limits of simulation}{7}{subsection.2.4}\protected@file@percent } | |
| 49 | +\@writefile{toc}{\contentsline {section}{\numberline {3}Data}{8}{section.3}\protected@file@percent } | |
| 50 | +\newlabel{sec:data}{{3}{8}{Data}{section.3}{}} | |
| 51 | +\@writefile{toc}{\contentsline {subsection}{\numberline {3.1}The Vrai Prix transaction database}{8}{subsection.3.1}\protected@file@percent } | |
| 52 | +\@writefile{toc}{\contentsline {subsection}{\numberline {3.2}Sample construction}{8}{subsection.3.2}\protected@file@percent } | |
| 53 | +\@writefile{lot}{\contentsline {table}{\numberline {1}{\ignorespaces Sample construction}}{9}{table.caption.1}\protected@file@percent } | |
| 54 | +\providecommand*\caption@xref[2]{\@setref\relax\@undefined{#1}} | |
| 55 | +\newlabel{tab:exclusions}{{1}{9}{Sample construction}{table.caption.1}{}} | |
| 56 | +\@writefile{toc}{\contentsline {subsection}{\numberline {3.3}Chronological partition and descriptive statistics}{9}{subsection.3.3}\protected@file@percent } | |
| 57 | +\@writefile{lot}{\contentsline {table}{\numberline {2}{\ignorespaces Descriptive statistics by period}}{10}{table.caption.2}\protected@file@percent } | |
| 58 | +\newlabel{tab:descriptives}{{2}{10}{Descriptive statistics by period}{table.caption.2}{}} | |
| 59 | +\@writefile{lof}{\contentsline {figure}{\numberline {1}{\ignorespaces Hedonic price index by region, full sample (descriptive). Monthly effects from a within-municipality regression of log price on characteristics; January 2021 $=100$. The point-in-time indices used in the backtest are re-estimated every month from prior sales only.}}{10}{figure.caption.3}\protected@file@percent } | |
| 60 | +\newlabel{fig:index}{{1}{10}{Hedonic price index by region, full sample (descriptive). Monthly effects from a within-municipality regression of log price on characteristics; January 2021 $=100$. The point-in-time indices used in the backtest are re-estimated every month from prior sales only}{figure.caption.3}{}} | |
| 61 | +\@writefile{toc}{\contentsline {subsection}{\numberline {3.4}Market density}{11}{subsection.3.4}\protected@file@percent } | |
| 62 | +\@writefile{lot}{\contentsline {table}{\numberline {3}{\ignorespaces Distribution of test targets by market density}}{11}{table.caption.4}\protected@file@percent } | |
| 63 | +\newlabel{tab:density_dist}{{3}{11}{Distribution of test targets by market density}{table.caption.4}{}} | |
| 64 | +\@writefile{toc}{\contentsline {section}{\numberline {4}Methodology}{11}{section.4}\protected@file@percent } | |
| 65 | +\newlabel{sec:methodology}{{4}{11}{Methodology}{section.4}{}} | |
| 66 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.1}Point-in-time backtesting}{11}{subsection.4.1}\protected@file@percent } | |
| 67 | +\newlabel{eq:pool}{{1}{11}{Point-in-time backtesting}{equation.1}{}} | |
| 68 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.2}Point-in-time price index}{12}{subsection.4.2}\protected@file@percent } | |
| 69 | +\newlabel{sec:index}{{4.2}{12}{Point-in-time price index}{subsection.4.2}{}} | |
| 70 | +\newlabel{eq:index}{{2}{12}{Point-in-time price index}{equation.2}{}} | |
| 71 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.3}Configurations of the sales comparison approach}{12}{subsection.4.3}\protected@file@percent } | |
| 72 | +\newlabel{sec:config}{{4.3}{12}{Configurations of the sales comparison approach}{subsection.4.3}{}} | |
| 73 | +\@writefile{toc}{\contentsline {paragraph}{Selection.}{12}{subsection.4.3}\protected@file@percent } | |
| 74 | +\@writefile{toc}{\contentsline {paragraph}{Adjustment.}{12}{subsection.4.3}\protected@file@percent } | |
| 75 | +\@writefile{toc}{\contentsline {paragraph}{Weighting and reconciliation.}{13}{Item.8}\protected@file@percent } | |
| 76 | +\@writefile{toc}{\contentsline {paragraph}{Experiments.}{13}{Item.8}\protected@file@percent } | |
| 77 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.4}Benchmark models}{13}{subsection.4.4}\protected@file@percent } | |
| 78 | +\newlabel{sec:benchmarks}{{4.4}{13}{Benchmark models}{subsection.4.4}{}} | |
| 79 | +\citation{cameron2008} | |
| 80 | +\@writefile{lot}{\contentsline {table}{\numberline {4}{\ignorespaces Models in the horse race}}{14}{table.caption.5}\protected@file@percent } | |
| 81 | +\newlabel{tab:models}{{4}{14}{Models in the horse race}{table.caption.5}{}} | |
| 82 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.5}Accuracy measures and inference}{14}{subsection.4.5}\protected@file@percent } | |
| 83 | +\@writefile{toc}{\contentsline {subsection}{\numberline {4.6}Hypotheses}{15}{subsection.4.6}\protected@file@percent } | |
| 84 | +\@writefile{toc}{\contentsline {section}{\numberline {5}Results}{15}{section.5}\protected@file@percent } | |
| 85 | +\newlabel{sec:results}{{5}{15}{Results}{section.5}{}} | |
| 86 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.1}How many comparables, how far, how far back?}{15}{subsection.5.1}\protected@file@percent } | |
| 87 | +\newlabel{sec:surface}{{5.1}{15}{How many comparables, how far, how far back?}{subsection.5.1}{}} | |
| 88 | +\@writefile{lof}{\contentsline {figure}{\numberline {2}{\ignorespaces Out-of-sample MdAPE (2025--2026) by number of comparables and search radius; twelve-month window, selection by distance, median reconciliation. Left: time adjustment only. Right: hedonic adjustment grid.}}{16}{figure.caption.6}\protected@file@percent } | |
| 89 | +\newlabel{fig:surface_k}{{2}{16}{Out-of-sample MdAPE (2025--2026) by number of comparables and search radius; twelve-month window, selection by distance, median reconciliation. Left: time adjustment only. Right: hedonic adjustment grid}{figure.caption.6}{}} | |
| 90 | +\@writefile{lof}{\contentsline {figure}{\numberline {3}{\ignorespaces Best attainable out-of-sample MdAPE (over $k$) for each search radius and look-back window; selection by distance.}}{16}{figure.caption.7}\protected@file@percent } | |
| 91 | +\newlabel{fig:surface_dt}{{3}{16}{Best attainable out-of-sample MdAPE (over $k$) for each search radius and look-back window; selection by distance}{figure.caption.7}{}} | |
| 92 | +\@writefile{lot}{\contentsline {table}{\numberline {5}{\ignorespaces Error surface $\mathrm {Error}(k,d)$: out-of-sample MdAPE (\%), selection by distance, hedonic grid, twelve-month window}}{17}{table.caption.8}\protected@file@percent } | |
| 93 | +\newlabel{tab:surface}{{5}{17}{Error surface $\mathrm {Error}(k,d)$: out-of-sample MdAPE (\%), selection by distance, hedonic grid, twelve-month window}{table.caption.8}{}} | |
| 94 | +\citation{colwell1983} | |
| 95 | +\@writefile{lot}{\contentsline {table}{\numberline {6}{\ignorespaces Optimal number of comparables by market density (H2): out-of-sample MdAPE (\%)}}{18}{table.caption.9}\protected@file@percent } | |
| 96 | +\newlabel{tab:h2}{{6}{18}{Optimal number of comparables by market density (H2): out-of-sample MdAPE (\%)}{table.caption.9}{}} | |
| 97 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.2}Which adjustments are warranted?}{18}{subsection.5.2}\protected@file@percent } | |
| 98 | +\newlabel{sec:adjustments}{{5.2}{18}{Which adjustments are warranted?}{subsection.5.2}{}} | |
| 99 | +\@writefile{lot}{\contentsline {table}{\numberline {7}{\ignorespaces Adjustment schemes at the $(5,\,5\text { km},\,12\text { months})$ anchor}}{18}{table.caption.10}\protected@file@percent } | |
| 100 | +\newlabel{tab:adjustments}{{7}{18}{Adjustment schemes at the $(5,\,5\text { km},\,12\text { months})$ anchor}{table.caption.10}{}} | |
| 101 | +\@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces MdAPE by adjustment scheme, validation (2024) and test (2025--2026); five nearest sales, $\le 5$~km, $\le 12$ months, median.}}{19}{figure.caption.11}\protected@file@percent } | |
| 102 | +\newlabel{fig:adjustments}{{4}{19}{MdAPE by adjustment scheme, validation (2024) and test (2025--2026); five nearest sales, $\le 5$~km, $\le 12$ months, median}{figure.caption.11}{}} | |
| 103 | +\@writefile{lot}{\contentsline {table}{\numberline {8}{\ignorespaces Adjustment schemes by market density (H3): out-of-sample MdAPE (\%)}}{20}{table.caption.12}\protected@file@percent } | |
| 104 | +\newlabel{tab:h3}{{8}{20}{Adjustment schemes by market density (H3): out-of-sample MdAPE (\%)}{table.caption.12}{}} | |
| 105 | +\@writefile{lot}{\contentsline {table}{\numberline {9}{\ignorespaces Hedonic adjustment grid (within-municipality coefficients, log price)}}{21}{table.caption.13}\protected@file@percent } | |
| 106 | +\newlabel{tab:grid}{{9}{21}{Hedonic adjustment grid (within-municipality coefficients, log price)}{table.caption.13}{}} | |
| 107 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.3}How should comparables be selected? (H1)}{21}{subsection.5.3}\protected@file@percent } | |
| 108 | +\newlabel{sec:selection}{{5.3}{21}{How should comparables be selected? (H1)}{subsection.5.3}{}} | |
| 109 | +\@writefile{lot}{\contentsline {table}{\numberline {10}{\ignorespaces Selection criteria at the $(5,\,5\text { km},\,12\text { months})$ anchor, hedonic grid}}{21}{table.caption.14}\protected@file@percent } | |
| 110 | +\newlabel{tab:selection}{{10}{21}{Selection criteria at the $(5,\,5\text { km},\,12\text { months})$ anchor, hedonic grid}{table.caption.14}{}} | |
| 111 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.4}Weighting and reconciliation (H4)}{22}{subsection.5.4}\protected@file@percent } | |
| 112 | +\newlabel{sec:weighting}{{5.4}{22}{Weighting and reconciliation (H4)}{subsection.5.4}{}} | |
| 113 | +\@writefile{lot}{\contentsline {table}{\numberline {11}{\ignorespaces Weighting and reconciliation at the $(5,\,5\text { km},\,12\text { months})$ anchor, hedonic grid}}{22}{table.caption.15}\protected@file@percent } | |
| 114 | +\newlabel{tab:weighting}{{11}{22}{Weighting and reconciliation at the $(5,\,5\text { km},\,12\text { months})$ anchor, hedonic grid}{table.caption.15}{}} | |
| 115 | +\@writefile{lot}{\contentsline {table}{\numberline {12}{\ignorespaces Weighting by market density (H4): out-of-sample MdAPE (\%)}}{23}{table.caption.16}\protected@file@percent } | |
| 116 | +\newlabel{tab:h4}{{12}{23}{Weighting by market density (H4): out-of-sample MdAPE (\%)}{table.caption.16}{}} | |
| 117 | +\@writefile{lot}{\contentsline {table}{\numberline {13}{\ignorespaces Marginal effects of the four rule factors in the E2 design}}{23}{table.caption.17}\protected@file@percent } | |
| 118 | +\newlabel{tab:marginal}{{13}{23}{Marginal effects of the four rule factors in the E2 design}{table.caption.17}{}} | |
| 119 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.5}The horse race}{24}{subsection.5.5}\protected@file@percent } | |
| 120 | +\newlabel{sec:horserace}{{5.5}{24}{The horse race}{subsection.5.5}{}} | |
| 121 | +\@writefile{lot}{\contentsline {table}{\numberline {14}{\ignorespaces Horse race: out-of-sample accuracy, 2025--2026}}{24}{table.caption.18}\protected@file@percent } | |
| 122 | +\newlabel{tab:competition}{{14}{24}{Horse race: out-of-sample accuracy, 2025--2026}{table.caption.18}{}} | |
| 123 | +\@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces Out-of-sample MdAPE by model, 2025--2026. Orange: configurations of the sales comparison approach; green: empirically optimised configurations; blue: external benchmarks.}}{25}{figure.caption.19}\protected@file@percent } | |
| 124 | +\newlabel{fig:competition}{{5}{25}{Out-of-sample MdAPE by model, 2025--2026. Orange: configurations of the sales comparison approach; green: empirically optimised configurations; blue: external benchmarks}{figure.caption.19}{}} | |
| 125 | +\@writefile{lot}{\contentsline {table}{\numberline {15}{\ignorespaces Paired differences in accuracy with cluster-bootstrap 95\% confidence intervals}}{26}{table.caption.20}\protected@file@percent } | |
| 126 | +\newlabel{tab:pairs}{{15}{26}{Paired differences in accuracy with cluster-bootstrap 95\% confidence intervals}{table.caption.20}{}} | |
| 127 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.6}Where does the method hold, and where does it fail?}{26}{subsection.5.6}\protected@file@percent } | |
| 128 | +\newlabel{sec:strata}{{5.6}{26}{Where does the method hold, and where does it fail?}{subsection.5.6}{}} | |
| 129 | +\@writefile{lot}{\contentsline {table}{\numberline {16}{\ignorespaces Out-of-sample MdAPE (\%) by market density}}{27}{table.caption.21}\protected@file@percent } | |
| 130 | +\newlabel{tab:density}{{16}{27}{Out-of-sample MdAPE (\%) by market density}{table.caption.21}{}} | |
| 131 | +\@writefile{lof}{\contentsline {figure}{\numberline {6}{\ignorespaces Out-of-sample MdAPE by market density (sales within 2~km in the preceding twelve months), selected models.}}{27}{figure.caption.22}\protected@file@percent } | |
| 132 | +\newlabel{fig:density}{{6}{27}{Out-of-sample MdAPE by market density (sales within 2~km in the preceding twelve months), selected models}{figure.caption.22}{}} | |
| 133 | +\@writefile{lot}{\contentsline {table}{\numberline {17}{\ignorespaces Accuracy by submarket heterogeneity (H5): out-of-sample MdAPE (\%)}}{28}{table.caption.23}\protected@file@percent } | |
| 134 | +\newlabel{tab:h5}{{17}{28}{Accuracy by submarket heterogeneity (H5): out-of-sample MdAPE (\%)}{table.caption.23}{}} | |
| 135 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.7}Value intervals}{28}{subsection.5.7}\protected@file@percent } | |
| 136 | +\newlabel{sec:intervals}{{5.7}{28}{Value intervals}{subsection.5.7}{}} | |
| 137 | +\@writefile{lot}{\contentsline {table}{\numberline {18}{\ignorespaces Coverage of value intervals derived from the dispersion of adjusted indications}}{28}{table.caption.24}\protected@file@percent } | |
| 138 | +\newlabel{tab:intervals}{{18}{28}{Coverage of value intervals derived from the dispersion of adjusted indications}{table.caption.24}{}} | |
| 139 | +\@writefile{toc}{\contentsline {subsection}{\numberline {5.8}Towards an empirical comparable-recommendation system}{28}{subsection.5.8}\protected@file@percent } | |
| 140 | +\newlabel{sec:recommend}{{5.8}{28}{Towards an empirical comparable-recommendation system}{subsection.5.8}{}} | |
| 141 | +\@writefile{lot}{\contentsline {table}{\numberline {19}{\ignorespaces Density-conditional rules chosen on 2024 and evaluated on 2025--2026}}{29}{table.caption.25}\protected@file@percent } | |
| 142 | +\newlabel{tab:cond}{{19}{29}{Density-conditional rules chosen on 2024 and evaluated on 2025--2026}{table.caption.25}{}} | |
| 143 | +\@writefile{toc}{\contentsline {section}{\numberline {6}Robustness}{29}{section.6}\protected@file@percent } | |
| 144 | +\newlabel{sec:robustness}{{6}{29}{Robustness}{section.6}{}} | |
| 145 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.1}Stability over time}{29}{subsection.6.1}\protected@file@percent } | |
| 146 | +\@writefile{lot}{\contentsline {table}{\numberline {20}{\ignorespaces Stability over the test period: MdAPE and bias (\%) by quarter}}{29}{table.caption.26}\protected@file@percent } | |
| 147 | +\newlabel{tab:stability}{{20}{29}{Stability over the test period: MdAPE and bias (\%) by quarter}{table.caption.26}{}} | |
| 148 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.2}Coverage and the common-support comparison}{30}{subsection.6.2}\protected@file@percent } | |
| 149 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.3}Exclusion rules}{30}{subsection.6.3}\protected@file@percent } | |
| 150 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.4}Alternative accuracy measures}{30}{subsection.6.4}\protected@file@percent } | |
| 151 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.5}Index construction}{30}{subsection.6.5}\protected@file@percent } | |
| 152 | +\@writefile{toc}{\contentsline {subsection}{\numberline {6.6}Benchmark specification}{31}{subsection.6.6}\protected@file@percent } | |
| 153 | +\@writefile{toc}{\contentsline {section}{\numberline {7}Conclusion}{31}{section.7}\protected@file@percent } | |
| 154 | +\newlabel{sec:conclusion}{{7}{31}{Conclusion}{section.7}{}} | |
| 155 | +\bibdata{references} | |
| 156 | +\bibcite{appraisal2020}{{1}{2020}{{Appraisal Institute}}{{Appraisal Institute}}} | |
| 157 | +\bibcite{bailey1963}{{2}{1963}{{Bailey et al.}}{{Bailey, Muth and Nourse}}} | |
| 158 | +\bibcite{bourassa2010}{{3}{2010}{{Bourassa et al.}}{{Bourassa, Cantoni and Hoesli}}} | |
| 159 | +\bibcite{cameron2008}{{4}{2008}{{Cameron et al.}}{{Cameron, Gelbach and Miller}}} | |
| 160 | +\bibcite{case2004}{{5}{2004}{{Case et al.}}{{Case, Clapp, Dubin and Rodriguez}}} | |
| 161 | +\bibcite{case1989}{{6}{1989}{{Case and Shiller}}{{Case and Shiller}}} | |
| 162 | +\bibcite{colwell1983}{{7}{1983}{{Colwell et al.}}{{Colwell, Cannaday and Wu}}} | |
| 163 | +\bibcite{diaz1990}{{8}{1990}{{Diaz}}{{Diaz}}} | |
| 164 | +\bibcite{diaz2001}{{9}{2001}{{Diaz and Hansz}}{{Diaz and Hansz}}} | |
| 165 | +\bibcite{gau1992}{{10}{1992}{{Gau et al.}}{{Gau, Lai and Wang}}} | |
| 166 | +\bibcite{gau1994}{{11}{1994}{{Gau et al.}}{{Gau, Lai and Wang}}} | |
| 167 | +\bibcite{gloudemans1999}{{12}{1999}{{Gloudemans}}{{Gloudemans}}} | |
| 168 | +\bibcite{hill2013}{{13}{2013}{{Hill}}{{Hill}}} | |
| 169 | +\bibcite{iaao2017}{{14}{2017}{{International Association of Assessing Officers}}{{International Association of Assessing Officers}}} | |
| 170 | +\bibcite{iaao2018}{{15}{2018}{{International Association of Assessing Officers}}{{International Association of Assessing Officers}}} | |
| 171 | +\bibcite{isakson1986}{{16}{1986}{{Isakson}}{{Isakson}}} | |
| 172 | +\bibcite{isakson2002}{{17}{2002}{{Isakson}}{{Isakson}}} | |
| 173 | +\bibcite{kang1991}{{18}{1991}{{Kang and Reichert}}{{Kang and Reichert}}} | |
| 174 | +\bibcite{lentz1998}{{19}{1998}{{Lentz and Wang}}{{Lentz and Wang}}} | |
| 175 | +\bibcite{mayer2019}{{20}{2019}{{Mayer et al.}}{{Mayer, Bourassa, Hoesli and Scognamiglio}}} | |
| 176 | +\bibcite{mccluskey2013}{{21}{2013}{{McCluskey et al.}}{{McCluskey, McCord, Davis, Haran and McIlhatton}}} | |
| 177 | +\bibcite{northcraft1987}{{22}{1987}{{Northcraft and Neale}}{{Northcraft and Neale}}} | |
| 178 | +\bibcite{pace1998}{{23}{1998}{{Pace}}{{Pace}}} | |
| 179 | +\bibcite{pace2020}{{24}{2020}{{Pace and Hayunga}}{{Pace and Hayunga}}} | |
| 180 | +\bibcite{pace1997}{{25}{1997}{{Pace and Gilley}}{{Pace and Gilley}}} | |
| 181 | +\bibcite{pagourtzi2003}{{26}{2003}{{Pagourtzi et al.}}{{Pagourtzi, Assimakopoulos, Hatzichristos and French}}} | |
| 182 | +\bibcite{rosen1974}{{27}{1974}{{Rosen}}{{Rosen}}} | |
| 183 | +\bibcite{vandell1991}{{28}{1991}{{Vandell}}{{Vandell}}} | |
| 184 | +\@writefile{toc}{\contentsline {section}{Appendices}{35}{section*.27}\protected@file@percent } | |
| 185 | +\@writefile{toc}{\contentsline {section}{\numberline {A}Additional tables}{35}{appendix.1.A}\protected@file@percent } | |
| 186 | +\newlabel{app:tables}{{A}{35}{Additional tables}{appendix.1.A}{}} | |
| 187 | +\@writefile{lot}{\contentsline {table}{\numberline {21}{\ignorespaces Coverage of the error surface: share of test targets (\%) for which a value is returned, by radius and window}}{35}{table.caption.28}\protected@file@percent } | |
| 188 | +\newlabel{tab:surface_cov}{{21}{35}{Coverage of the error surface: share of test targets (\%) for which a value is returned, by radius and window}{table.caption.28}{}} | |
| 189 | +\@writefile{lot}{\contentsline {table}{\numberline {22}{\ignorespaces Error surface without physical adjustment: out-of-sample MdAPE (\%), selection by distance, time adjustment only, twelve-month window}}{35}{table.caption.29}\protected@file@percent } | |
| 190 | +\newlabel{tab:surface_none}{{22}{35}{Error surface without physical adjustment: out-of-sample MdAPE (\%), selection by distance, time adjustment only, twelve-month window}{table.caption.29}{}} | |
| 191 | +\@writefile{lot}{\contentsline {table}{\numberline {23}{\ignorespaces Error surface with selection by similarity: out-of-sample MdAPE (\%), hedonic grid, twelve-month window}}{36}{table.caption.30}\protected@file@percent } | |
| 192 | +\newlabel{tab:surface_sim}{{23}{36}{Error surface with selection by similarity: out-of-sample MdAPE (\%), hedonic grid, twelve-month window}{table.caption.30}{}} | |
| 193 | +\@writefile{lot}{\contentsline {table}{\numberline {24}{\ignorespaces Best configuration of the error surface by selection and adjustment (chosen on 2024, evaluated on 2025--2026)}}{36}{table.caption.31}\protected@file@percent } | |
| 194 | +\newlabel{tab:e1best}{{24}{36}{Best configuration of the error surface by selection and adjustment (chosen on 2024, evaluated on 2025--2026)}{table.caption.31}{}} | |
| 195 | +\@writefile{lot}{\contentsline {table}{\numberline {25}{\ignorespaces Out-of-sample MdAPE (\%) by administrative region}}{37}{table.caption.32}\protected@file@percent } | |
| 196 | +\newlabel{tab:regions}{{25}{37}{Out-of-sample MdAPE (\%) by administrative region}{table.caption.32}{}} | |
| 197 | +\@writefile{lot}{\contentsline {table}{\numberline {26}{\ignorespaces Out-of-sample MdAPE (\%) by price quintile, age and physical link}}{37}{table.caption.33}\protected@file@percent } | |
| 198 | +\newlabel{tab:price}{{26}{37}{Out-of-sample MdAPE (\%) by price quintile, age and physical link}{table.caption.33}{}} | |
| 199 | +\@writefile{toc}{\contentsline {section}{\numberline {B}Implementation details}{38}{appendix.1.B}\protected@file@percent } | |
| 200 | +\newlabel{app:impl}{{B}{38}{Implementation details}{appendix.1.B}{}} | |
| 201 | +\@writefile{toc}{\contentsline {paragraph}{Computation.}{38}{appendix.1.B}\protected@file@percent } | |
| 202 | +\@writefile{toc}{\contentsline {paragraph}{Local grid.}{38}{appendix.1.B}\protected@file@percent } | |
| 203 | +\@writefile{toc}{\contentsline {paragraph}{Gradient boosting.}{38}{appendix.1.B}\protected@file@percent } | |
| 204 | +\@writefile{toc}{\contentsline {paragraph}{Cluster bootstrap.}{38}{appendix.1.B}\protected@file@percent } | |
| 205 | +\gdef \@abspage@last{38} | |
added
paper/main.bbl
+148 −0
@@ -0,0 +1,148 @@ | ||
| 1 | +\ifx\undefined\bysame | |
| 2 | +\newcommand{\bysame}{\leavevmode\hbox to\leftmargin{\hrulefill\,\,}} | |
| 3 | +\fi | |
| 4 | +\begin{thebibliography}{xx} | |
| 5 | + | |
| 6 | +\harvarditem[{Appraisal Institute}]{{Appraisal Institute}}{2020}{appraisal2020} | |
| 7 | +{\bf {Appraisal Institute}}, {\it The Appraisal of Real Estate}, 15th ed., | |
| 8 | + Chicago: Appraisal Institute, 2020. | |
| 9 | + | |
| 10 | +\harvarditem[Bailey et al.]{Bailey, Muth and Nourse}{1963}{bailey1963} | |
| 11 | +{\bf Bailey, Martin~J., Richard~F. Muth, and Hugh~O. Nourse}, ``A Regression | |
| 12 | + Method for Real Estate Price Index Construction,'' {\it Journal of the | |
| 13 | + American Statistical Association}, 1963, {\it 58} (304), 933--942. | |
| 14 | + | |
| 15 | +\harvarditem[Bourassa et al.]{Bourassa, Cantoni and Hoesli}{2010}{bourassa2010} | |
| 16 | +{\bf Bourassa, Steven~C., Eva Cantoni, and Martin Hoesli}, ``Predicting House | |
| 17 | + Prices with Spatial Dependence: A Comparison of Alternative Methods,'' {\it | |
| 18 | + Journal of Real Estate Research}, 2010, {\it 32} (2), 139--159. | |
| 19 | + | |
| 20 | +\harvarditem[Cameron et al.]{Cameron, Gelbach and Miller}{2008}{cameron2008} | |
| 21 | +{\bf Cameron, A.~Colin, Jonah~B. Gelbach, and Douglas~L. Miller}, | |
| 22 | + ``Bootstrap-Based Improvements for Inference with Clustered Errors,'' {\it | |
| 23 | + Review of Economics and Statistics}, 2008, {\it 90} (3), 414--427. | |
| 24 | + | |
| 25 | +\harvarditem[Case et al.]{Case, Clapp, Dubin and Rodriguez}{2004}{case2004} | |
| 26 | +{\bf Case, Bradford, John Clapp, Robin Dubin, and Mauricio Rodriguez}, | |
| 27 | + ``Modeling Spatial and Temporal House Price Patterns: A Comparison of Four | |
| 28 | + Models,'' {\it Journal of Real Estate Finance and Economics}, 2004, {\it 29} | |
| 29 | + (2), 167--191. | |
| 30 | + | |
| 31 | +\harvarditem[Case and Shiller]{Case and Shiller}{1989}{case1989} | |
| 32 | +{\bf Case, Karl~E. and Robert~J. Shiller}, ``The Efficiency of the Market for | |
| 33 | + Single-Family Homes,'' {\it American Economic Review}, 1989, {\it 79} (1), | |
| 34 | + 125--137. | |
| 35 | + | |
| 36 | +\harvarditem[Colwell et al.]{Colwell, Cannaday and Wu}{1983}{colwell1983} | |
| 37 | +{\bf Colwell, Peter~F., Roger~E. Cannaday, and Chunchi Wu}, ``The Analytical | |
| 38 | + Foundations of Adjustment Grid Methods,'' {\it Real Estate Economics}, 1983, | |
| 39 | + {\it 11} (1), 11--29. | |
| 40 | + | |
| 41 | +\harvarditem[Diaz]{Diaz}{1990}{diaz1990} | |
| 42 | +{\bf Diaz, Julian}, ``How Appraisers Do Their Work: A Test of the Appraisal | |
| 43 | + Process and the Development of a Descriptive Model,'' {\it Journal of Real | |
| 44 | + Estate Research}, 1990, {\it 5} (1), 1--15. | |
| 45 | + | |
| 46 | +\harvarditem[Diaz and Hansz]{Diaz and Hansz}{2001}{diaz2001} | |
| 47 | +{\bf \bysame{} and J.~Andrew Hansz}, ``The Use of Reference Points in Valuation | |
| 48 | + Judgment,'' {\it Journal of Property Research}, 2001, {\it 18} (2), 141--148. | |
| 49 | + | |
| 50 | +\harvarditem[Gau et al.]{Gau, Lai and Wang}{1992}{gau1992} | |
| 51 | +{\bf Gau, George~W., Tsong-Yue Lai, and Ko~Wang}, ``Optimal Comparable | |
| 52 | + Selection and Weighting in Real Property Valuation: An Extension,'' {\it Real | |
| 53 | + Estate Economics}, 1992, {\it 20} (1), 107--123. | |
| 54 | + | |
| 55 | +\harvarditem[Gau et al.]{Gau, Lai and Wang}{1994}{gau1994} | |
| 56 | +{\bf \bysame{}, \bysame{}, and \bysame{}}, ``A Further Discussion of Optimal | |
| 57 | + Comparable Selection and Weighting, and a Response to {Green},'' {\it Real | |
| 58 | + Estate Economics}, 1994, {\it 22} (4), 655--663. | |
| 59 | + | |
| 60 | +\harvarditem[Gloudemans]{Gloudemans}{1999}{gloudemans1999} | |
| 61 | +{\bf Gloudemans, Robert~J.}, {\it Mass Appraisal of Real Property}, Chicago: | |
| 62 | + International Association of Assessing Officers, 1999. | |
| 63 | + | |
| 64 | +\harvarditem[Hill]{Hill}{2013}{hill2013} | |
| 65 | +{\bf Hill, Robert~J.}, ``Hedonic Price Indexes for Residential Housing: A | |
| 66 | + Survey, Evaluation and Taxonomy,'' {\it Journal of Economic Surveys}, 2013, | |
| 67 | + {\it 27} (5), 879--914. | |
| 68 | + | |
| 69 | +\harvarditem[{International Association of Assessing Officers}]{{International | |
| 70 | + Association of Assessing Officers}}{2017}{iaao2017} | |
| 71 | +{\bf {International Association of Assessing Officers}}, ``Standard on Mass | |
| 72 | + Appraisal of Real Property,'' Technical Report, IAAO, Kansas City 2017. | |
| 73 | + | |
| 74 | +\harvarditem[{International Association of Assessing Officers}]{{International | |
| 75 | + Association of Assessing Officers}}{2018}{iaao2018} | |
| 76 | +{\bf \bysame{}}, ``Standard on Automated Valuation Models ({AVM}s),'' Technical | |
| 77 | + Report, IAAO, Kansas City 2018. | |
| 78 | + | |
| 79 | +\harvarditem[Isakson]{Isakson}{1986}{isakson1986} | |
| 80 | +{\bf Isakson, Hans~R.}, ``The Nearest Neighbors Appraisal Technique: An | |
| 81 | + Alternative to the Adjustment Grid Methods,'' {\it Real Estate Economics}, | |
| 82 | + 1986, {\it 14} (2), 274--286. | |
| 83 | + | |
| 84 | +\harvarditem[Isakson]{Isakson}{2002}{isakson2002} | |
| 85 | +{\bf \bysame{}}, ``The Linear Algebra of the Sales Comparison Approach,'' {\it | |
| 86 | + Journal of Real Estate Research}, 2002, {\it 24} (2), 117--128. | |
| 87 | + | |
| 88 | +\harvarditem[Kang and Reichert]{Kang and Reichert}{1991}{kang1991} | |
| 89 | +{\bf Kang, Han-Bin and Alan~K. Reichert}, ``An Empirical Analysis of Hedonic | |
| 90 | + Regression and Grid-Adjustment Techniques in Real Estate Appraisal,'' {\it | |
| 91 | + Real Estate Economics}, 1991, {\it 19} (1), 70--91. | |
| 92 | + | |
| 93 | +\harvarditem[Lentz and Wang]{Lentz and Wang}{1998}{lentz1998} | |
| 94 | +{\bf Lentz, George~H. and Ko~Wang}, ``Residential Appraisal and the Lending | |
| 95 | + Process: A Survey of Issues,'' {\it Journal of Real Estate Research}, 1998, | |
| 96 | + {\it 15} (1), 11--39. | |
| 97 | + | |
| 98 | +\harvarditem[Mayer et al.]{Mayer, Bourassa, Hoesli and | |
| 99 | + Scognamiglio}{2019}{mayer2019} | |
| 100 | +{\bf Mayer, Michael, Steven~C. Bourassa, Martin Hoesli, and Donato | |
| 101 | + Scognamiglio}, ``Estimation and Updating Methods for Hedonic Valuation,'' | |
| 102 | + {\it Journal of European Real Estate Research}, 2019, {\it 12} (1), 134--150. | |
| 103 | + | |
| 104 | +\harvarditem[McCluskey et al.]{McCluskey, McCord, Davis, Haran and | |
| 105 | + McIlhatton}{2013}{mccluskey2013} | |
| 106 | +{\bf McCluskey, William~J., Michael McCord, Peadar~T. Davis, Martin Haran, and | |
| 107 | + David McIlhatton}, ``Prediction Accuracy in Mass Appraisal: A Comparison of | |
| 108 | + Modern Approaches,'' {\it Journal of Property Research}, 2013, {\it 30} (4), | |
| 109 | + 239--265. | |
| 110 | + | |
| 111 | +\harvarditem[Northcraft and Neale]{Northcraft and Neale}{1987}{northcraft1987} | |
| 112 | +{\bf Northcraft, Gregory~B. and Margaret~A. Neale}, ``Experts, Amateurs, and | |
| 113 | + Real Estate: An Anchoring-and-Adjustment Perspective on Property Pricing | |
| 114 | + Decisions,'' {\it Organizational Behavior and Human Decision Processes}, | |
| 115 | + 1987, {\it 39} (1), 84--97. | |
| 116 | + | |
| 117 | +\harvarditem[Pace]{Pace}{1998}{pace1998} | |
| 118 | +{\bf Pace, R.~Kelley}, ``Appraisal Using Generalized Additive Models,'' {\it | |
| 119 | + Journal of Real Estate Research}, 1998, {\it 15} (1), 77--99. | |
| 120 | + | |
| 121 | +\harvarditem[Pace and Hayunga]{Pace and Hayunga}{2020}{pace2020} | |
| 122 | +{\bf \bysame{} and Darren Hayunga}, ``Examining the Information Content of | |
| 123 | + Residuals from Hedonic and Spatial Models Using Trees and Forests,'' {\it | |
| 124 | + Journal of Real Estate Finance and Economics}, 2020, {\it 60} (1--2), | |
| 125 | + 170--180. | |
| 126 | + | |
| 127 | +\harvarditem[Pace and Gilley]{Pace and Gilley}{1997}{pace1997} | |
| 128 | +{\bf \bysame{} and Otis~W. Gilley}, ``Using the Spatial Configuration of the | |
| 129 | + Data to Improve Estimation,'' {\it Journal of Real Estate Finance and | |
| 130 | + Economics}, 1997, {\it 14} (3), 333--340. | |
| 131 | + | |
| 132 | +\harvarditem[Pagourtzi et al.]{Pagourtzi, Assimakopoulos, Hatzichristos and | |
| 133 | + French}{2003}{pagourtzi2003} | |
| 134 | +{\bf Pagourtzi, Elli, Vassilis Assimakopoulos, Thomas Hatzichristos, and Nick | |
| 135 | + French}, ``Real Estate Appraisal: A Review of Valuation Methods,'' {\it | |
| 136 | + Journal of Property Investment \& Finance}, 2003, {\it 21} (4), 383--401. | |
| 137 | + | |
| 138 | +\harvarditem[Rosen]{Rosen}{1974}{rosen1974} | |
| 139 | +{\bf Rosen, Sherwin}, ``Hedonic Prices and Implicit Markets: Product | |
| 140 | + Differentiation in Pure Competition,'' {\it Journal of Political Economy}, | |
| 141 | + 1974, {\it 82} (1), 34--55. | |
| 142 | + | |
| 143 | +\harvarditem[Vandell]{Vandell}{1991}{vandell1991} | |
| 144 | +{\bf Vandell, Kerry~D.}, ``Optimal Comparable Selection and Weighting in Real | |
| 145 | + Property Valuation,'' {\it Real Estate Economics}, 1991, {\it 19} (2), | |
| 146 | + 213--239. | |
| 147 | + | |
| 148 | +\end{thebibliography} | |
added
paper/main.blg
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +This is BibTeX, Version 0.99e (TeX Live 2026) | |
| 2 | +Capacity: max_strings=200000, hash_size=200000, hash_prime=170003 | |
| 3 | +The top-level auxiliary file: main.aux | |
| 4 | +The style file: aer.bst | |
| 5 | +Reallocated singl_function (elt_size=4) to 100 items from 50. | |
| 6 | +Reallocated singl_function (elt_size=4) to 100 items from 50. | |
| 7 | +Database file #1: references.bib | |
| 8 | +You've used 28 entries, | |
| 9 | + 2984 wiz_defined-function locations, | |
| 10 | + 821 strings with 8352 characters, | |
| 11 | +and the built_in function-call counts, 24844 in all, are: | |
| 12 | += -- 4009 | |
| 13 | +> -- 1349 | |
| 14 | +< -- 399 | |
| 15 | ++ -- 438 | |
| 16 | +- -- 240 | |
| 17 | +* -- 1622 | |
| 18 | +:= -- 5571 | |
| 19 | +add.period$ -- 28 | |
| 20 | +call.type$ -- 28 | |
| 21 | +change.case$ -- 197 | |
| 22 | +chr.to.int$ -- 28 | |
| 23 | +cite$ -- 28 | |
| 24 | +duplicate$ -- 311 | |
| 25 | +empty$ -- 769 | |
| 26 | +format.name$ -- 670 | |
| 27 | +if$ -- 6155 | |
| 28 | +int.to.chr$ -- 1 | |
| 29 | +int.to.str$ -- 0 | |
| 30 | +missing$ -- 28 | |
| 31 | +newline$ -- 90 | |
| 32 | +num.names$ -- 336 | |
| 33 | +pop$ -- 425 | |
| 34 | +preamble$ -- 1 | |
| 35 | +purify$ -- 224 | |
| 36 | +quote$ -- 0 | |
| 37 | +skip$ -- 0 | |
| 38 | +stack$ -- 0 | |
| 39 | +substring$ -- 881 | |
| 40 | +swap$ -- 104 | |
| 41 | +text.length$ -- 0 | |
| 42 | +text.prefix$ -- 0 | |
| 43 | +top$ -- 0 | |
| 44 | +type$ -- 108 | |
| 45 | +warning$ -- 0 | |
| 46 | +while$ -- 328 | |
| 47 | +width$ -- 0 | |
| 48 | +write$ -- 476 | |
added
paper/main.fdb_latexmk
+200 −0
@@ -0,0 +1,200 @@ | ||
| 1 | +# Fdb version 4 | |
| 2 | +["bibtex main"] 1791669590.6914 "main.aux" "main.bbl" "main" 1791669591.58141 0 | |
| 3 | + "./references.bib" 1791669372.52602 8208 317d4516e8f0f47f17cb222b3493cf06 "" | |
| 4 | + "/usr/local/texlive/2026/texmf-dist/bibtex/bst/economic/aer.bst" 1389482635 26812 2ae6e48450a8c17a93d98067b7084a7a "" | |
| 5 | + "main.aux" 1791669591.43573 22155 66f06235bf01cd09404880f862499c71 "pdflatex" | |
| 6 | + (generated) | |
| 7 | + "main.bbl" | |
| 8 | + "main.blg" | |
| 9 | + (rewritten before read) | |
| 10 | +["pdflatex"] 1791669590.75035 "main.tex" "main.pdf" "main" 1791669591.58152 0 | |
| 11 | + "/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/base/8r.enc" 1165713224 4850 80dc9bab7f31fb78a000ccfed0e27cab "" | |
| 12 | + "/usr/local/texlive/2026/texmf-dist/fonts/map/fontname/texfonts.map" 1577235249 3524 cb3e574dea2d1052e39280babc910dc8 "" | |
| 13 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm" 1136768653 1408 5937f58aa508ea2cea4901c07d10f5fe "" | |
| 14 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm" 1136768653 1544 23a042a74981a3e4b6ce2e350e390409 "" | |
| 15 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm" 1136768653 2172 fd0c924230362ff848a33632ed45dc23 "" | |
| 16 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm" 1136768653 4524 6bce29db5bc272ba5f332261583fee9c "" | |
| 17 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm" 1136768653 6880 f19b8995b61c334d78fc734065f6b4d4 "" | |
| 18 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm" 1136768653 4408 25b74d011a4c66b7f212c0cc3c90061b "" | |
| 19 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm" 1136768653 6672 e3ab9e37e925f3045c9005e6d1473d56 "" | |
| 20 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmrc8t.tfm" 1136768653 16648 7e91c60ebf91b9c60e16ef52b3070f9c "" | |
| 21 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm" 1136768653 2288 f478fc8fed18759effb59f3dad7f3084 "" | |
| 22 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm" 1136768653 4640 532ca3305aad10cc01d769f3f91f1029 "" | |
| 23 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8t.tfm" 1136768653 6944 94c55ad86e6ea2826f78ba2240d50df9 "" | |
| 24 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm" 1136768653 2232 db256afffc8202da192b4641df14d602 "" | |
| 25 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm" 1136768653 2172 1d00c2a0d10f23031be62329457a870c "" | |
| 26 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm" 1136768653 1032 20febbd0f0c9a48eb78616f897008286 "" | |
| 27 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm" 1136768653 1520 ad7b3c1a480a03b3e41b5fbb13d938f2 "" | |
| 28 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/jknappen/ec/ecrm1200.tfm" 1136768653 3584 f80ddd985bd00e29e9a6047ebd9d4781 "" | |
| 29 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm" 1246382020 916 f87d7c45f9c908e672703b83b72241a3 "" | |
| 30 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm" 1246382020 908 2921f8a10601f252058503cc6570e581 "" | |
| 31 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmex10.tfm" 1136768653 992 662f679a0b3d2d53c1b94050fdaa3f50 "" | |
| 32 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmmi10.tfm" 1136768653 1528 abec98dbc43e172678c11b3b9031252a "" | |
| 33 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr10.tfm" 1136768653 1296 45809c5a464d5f32c8f98ba97c1bb47f "" | |
| 34 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr12.tfm" 1136768653 1288 655e228510b4c2a1abe905c368440826 "" | |
| 35 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmsy10.tfm" 1136768653 1124 6c73e740cf17375f03eec0ee63599741 "" | |
| 36 | + "/usr/local/texlive/2026/texmf-dist/fonts/tfm/public/rsfs/rsfs10.tfm" 1229303445 688 37338d6ab346c2f1466b29e195316aa4 "" | |
| 37 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb" 1248133631 30251 6afa5cb1d0204815a708a080681d4674 "" | |
| 38 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb" 1248133631 36299 5f9df58c2139e7edcf37c8fca4bd384d "" | |
| 39 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb" 1248133631 35752 024fb6c41858982481f6968b5fc26508 "" | |
| 40 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb" 1248133631 32569 5e5ddc8df908dea60932f3c484a54c0d "" | |
| 41 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/symbols/msbm10.pfb" 1248133631 34694 ad62b13721ee8eda1dcc8993c8bd7041 "" | |
| 42 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/public/rsfs/rsfs10.pfb" 1229303445 16077 4737ac34f0fb5608550f3780a0202c22 "" | |
| 43 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb" 1136849748 33709 b09d2e140b7e807d3a97058263ab6693 "" | |
| 44 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmb8a.pfb" 1136849748 44729 811d6c62865936705a31c797a1d5dada "" | |
| 45 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmr8a.pfb" 1136849748 46026 6dab18b61c907687b520c72847215a68 "" | |
| 46 | + "/usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8a.pfb" 1136849748 45458 a3faba884469519614ca56ba5f6b1de1 "" | |
| 47 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb7t.vf" 1136768653 1372 788387fea833ef5963f4c5bffe33eb89 "" | |
| 48 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf" 1136768653 2340 df9c920cc5688ebbf16a93f45ce7bdd3 "" | |
| 49 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmr8t.vf" 1136768653 2348 91706c542228501c410c266421fbe30c "" | |
| 50 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmrc8t.vf" 1136768653 3608 78e4c999d9e69f03f448158d01821741 "" | |
| 51 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmri8t.vf" 1136768653 2328 6cd7df782b09b29cfc4d93e55b6b9a59 "" | |
| 52 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf" 1136768653 1132 27520247d3fe18d4266a226b461885c2 "" | |
| 53 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf" 1136768653 1108 d271d6f9de4122c3f8d3b65666167fac "" | |
| 54 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7v.vf" 1136768653 1012 046369ac6a83af997c3aa05a43256ad5 "" | |
| 55 | + "/usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7y.vf" 1136768653 964 5673178ff30617b900214de28ab32b38 "" | |
| 56 | + "/usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii" 1461363279 71627 94eb9990bed73c364d7f53f960cc8c5b "" | |
| 57 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf" 1496785618 7008 9ff5fdcc865b01beca2b0fe4a46231d4 "" | |
| 58 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/babel/babel.sty" 1773085155 147019 a132d5463be1676209afefa6c8d009c6 "" | |
| 59 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-en.ini" 1763931111 3936 fef897bbcfe8e9d18c93a3d2d53cf9d8 "" | |
| 60 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.tex" 1752438283 374 a9c05f002d9437e6de45ac1cc23d56c2 "" | |
| 61 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/babel/txtbabel.def" 1768770623 5231 c1599e5bf7d2ee42743212a7f8d20a56 "" | |
| 62 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty" 1576625341 40635 c40361e206be584d448876bba8a64a3b "" | |
| 63 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/bitset/bitset.sty" 1576016050 33961 6b5c75130e435b2bfdb9f480a09a39f9 "" | |
| 64 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty" 1576625223 8371 9d55b8bd010bc717624922fb3477d92e "" | |
| 65 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/iftex.sty" 1734129479 7984 7dbb9280f03c0a315425f1b4f35d43ee "" | |
| 66 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/ifvtex.sty" 1572645307 1057 525c2192b5febbd8c1f662c9468335bb "" | |
| 67 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/infwarerr/infwarerr.sty" 1575499628 8356 7bbb2c2373aa810be568c29e333da8ed "" | |
| 68 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/intcalc/intcalc.sty" 1576625065 31769 002a487f55041f8e805cfbf6385ffd97 "" | |
| 69 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty" 1576878844 5412 d5a2436094cd7be85769db90f29250a6 "" | |
| 70 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty" 1701727651 17865 1a9bd36b4f98178fa551aca822290953 "" | |
| 71 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/pdfescape/pdfescape.sty" 1576015897 19007 15924f7228aca6c6d184b115f4baa231 "" | |
| 72 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty" 1593379760 20089 80423eac55aa175305d35b49e04fe23b "" | |
| 73 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/stringenc/stringenc.sty" 1575152242 21514 b7557edcee22835ef6b03ede1802dad4 "" | |
| 74 | + "/usr/local/texlive/2026/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty" 1576624663 7008 f92eaa0a3872ed622bbf538217cd2ab7 "" | |
| 75 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amscls/amsthm.sty" 1591045760 12594 0d51ac3a545aaaa555021326ff22a6cc "" | |
| 76 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amsfonts.sty" 1359763108 5949 3f3fd50a8cc94c3d4cbf4fc66cd3df1c "" | |
| 77 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amssymb.sty" 1359763108 13829 94730e64147574077f8ecfea9bb69af4 "" | |
| 78 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsbsy.sty" 1748806692 2222 27db7d52163edae53881b71ff62e754e "" | |
| 79 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsgen.sty" 1748806692 4173 1b3e76addfb8afcb47db4811d66e1dc6 "" | |
| 80 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsmath.sty" 1761946296 88471 b1bb09142edddebd46ba986341b867bd "" | |
| 81 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsopn.sty" 1748806692 4474 c510a88aa5f51b8c773b50a7ee92befd "" | |
| 82 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amstext.sty" 1748806692 2444 9983e1d0683f102e3b190c64a49313aa "" | |
| 83 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/appendix/appendix.sty" 1581200180 8878 d9f65b39ca82f1d70030390eca653b1c "" | |
| 84 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/base/article.cls" 1748806692 20144 b966087dda3b194755eb460d32e2ef75 "" | |
| 85 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/base/fontenc.sty" 1761946296 5275 6f9d359641b36842524cdb97716ab75f "" | |
| 86 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/base/inputenc.sty" 1738182759 5048 0270515b828149155424600fd2d58ac5 "" | |
| 87 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo" 1748806692 8449 8dc66c6c313c8eb2d774af83bca435dd "" | |
| 88 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/booktabs/booktabs.sty" 1579038678 6078 f1cb470c9199e7110a27851508ed7a5c "" | |
| 89 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption.sty" 1696191071 56128 c2ccf1a29d78c33bc553880402e4fb9a "" | |
| 90 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption3.sty" 1696191071 72619 ee90b6612147680fd73c3b1406a74245 "" | |
| 91 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/caption/subcaption.sty" 1690576852 12494 0c0cdb824278a4d51cefeb2e79901315 "" | |
| 92 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/enumitem/enumitem.sty" 1738874546 52272 63d293bc0d496619edb57585740861a2 "" | |
| 93 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty" 1579991033 13886 d1306dcf79a944f6988e688c1785f9ce "" | |
| 94 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/etoolbox/etoolbox.sty" 1759437024 46885 8953c67ffba03252c6090aa19568b8ba "" | |
| 95 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty" 1739048179 31715 19e60610b63819fe670dfa1cd84a4e94 "" | |
| 96 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/float/float.sty" 1137110151 6749 16d2656a1984957e674b149555f1ea1d "" | |
| 97 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/geometry/geometry.sty" 1772916678 41904 7711079366884e0d15963c9d40019853 "" | |
| 98 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg" 1459978653 1213 620bba36b25224fa9b7e1ccb4ecb76fd "" | |
| 99 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg" 1465944070 1224 978390e9c2234eab29404bc21b268d1e "" | |
| 100 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def" 1759176675 19626 23e2822b9b2b5005f4c549ca98b9334d "" | |
| 101 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/color.sty" 1748806692 7245 a7e8457a46cda4920df85d975267efb4 "" | |
| 102 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def" 1748806692 5009 f48e1006c67ca6c50094e2f41e884d72 "" | |
| 103 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphics.sty" 1748806692 18363 69bb4f5538964bfea50d1e6d89cbe69f "" | |
| 104 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphicx.sty" 1748806692 8118 43b99e52946c33a23f5f43b52d5cc5ec "" | |
| 105 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/keyval.sty" 1748806692 2671 d9941f4bf4750e9b0603c9a2ec54693b "" | |
| 106 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx" 1667332637 2885 9c645d672ae17285bba324998918efd8 "" | |
| 107 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/trig.sty" 1748806692 4023 e66acf578d6b564c4670fb57ff336a7a "" | |
| 108 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hycolor/hycolor.sty" 1580250785 17914 4c28a13fc3d975e6e81c9bea1d697276 "" | |
| 109 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def" 1769721417 48140 0d317d7fb0c7460a10b7b2713db57305 "" | |
| 110 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hyperref.sty" 1769721417 223349 c7928c099a8656537a829ba316c95536 "" | |
| 111 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/nameref.sty" 1769721417 11459 697f11f6c439d25d39d2674b99566af4 "" | |
| 112 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def" 1769721417 14249 b94983bbccc8d5739c16cc91d1fd1c3b "" | |
| 113 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def" 1769721417 117118 2e3ba580751de5583beacf2e5fee69a9 "" | |
| 114 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/kvoptions/kvoptions.sty" 1655478651 22555 6d8e155cfef6d82c3d5c742fea7c992e "" | |
| 115 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty" 1665067230 13815 760b0c02f691ea230f5359c4e1de23a7 "" | |
| 116 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def" 1761248321 30662 bfd6e864f4ffc5018b0e2b6260c3181c "" | |
| 117 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg" 1279039959 678 4792914a8f45be57bb98413425e4c7af "" | |
| 118 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def" 1772383374 49656 1c61dbb6f95479ba3d6c0033b53590e2 "" | |
| 119 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg" 1772383374 27642 f0cea12315babf4d40608f4eeb5c8458 "" | |
| 120 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.sty" 1772383374 102845 043c56602c7d94c8a716319df1af5479 "" | |
| 121 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg" 1772226603 5929 2b35ae0f0fb46984dfffa2bc9d09de5c "" | |
| 122 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg" 1772226603 5594 992ef5c3f8fd1168bb7101a957333065 "" | |
| 123 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg" 1772226603 12427 a6802929d6bd2a4ba6d4f5db602da2b4 "" | |
| 124 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/multirow/multirow.sty" 1731446765 6696 886c9f3087d0b973ed2c19aa79cb3023 "" | |
| 125 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/natbib/natbib.sty" 1291685959 45456 1c8843383c0bd05870c45fa0ebea6cc2 "" | |
| 126 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/mathptmx.sty" 1586716065 4631 6e41de2b7a83dfa5d2c4b0a2fe01f046 "" | |
| 127 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd" 1137110629 411 12564a37a279e4e0b533cdf5e03eeb7c "" | |
| 128 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd" 1137110629 348 f4ce75d394e7d9ac12ca7aac4045ed77 "" | |
| 129 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd" 1137110629 329 c8cddcc90b6f567b28408eb374773c9c "" | |
| 130 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd" 1137110629 961 15056f4a61917ceed3a44e4ac11fcc52 "" | |
| 131 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd" 1137110629 329 aee7226812ba4138ac67a018466b488d "" | |
| 132 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd" 1137110629 774 61d7da1e9f9e74989b196d147e623736 "" | |
| 133 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/refcount/refcount.sty" 1576624809 9878 9e94e8fa600d95f9c7731bb21dfb67a4 "" | |
| 134 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty" 1750533675 9684 a33a14b82ce60d6e77cb9be689d79ee6 "" | |
| 135 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/setspace/setspace.sty" 1670275497 22490 8cac309b79a4c53a4ffce4b1b07aead0 "" | |
| 136 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/siunitx/siunitx.sty" 1752176849 344221 f0c98c737eca26294f5039e8afde731f "" | |
| 137 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/threeparttable/threeparttable.sty" 1267981840 13506 a4e71a27db1a69b6fabada5beebf0844 "" | |
| 138 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/titlesec/titlesec.sty" 1736023606 48766 87a17a4ef312a39cd43896e34a679a56 "" | |
| 139 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/tools/array.sty" 1761946296 15651 9d7c62df82cb29a555c00550babfe364 "" | |
| 140 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/tools/tabularx.sty" 1748806692 7243 a2c17f18e2c9b702b84fad03d5f9c78b "" | |
| 141 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-dictionary-english.trsl" 1644096163 5588 0c1628daf15f4411ff1f463114c634a3 "" | |
| 142 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations.sty" 1644096163 44247 2188b95d0ee74e31eca4d316263c58a7 "" | |
| 143 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/url/url.sty" 1388531844 12796 8edb7d69a20b857904dd0ea757c14ec9 "" | |
| 144 | + "/usr/local/texlive/2026/texmf-dist/tex/latex/xcolor/xcolor.sty" 1727642399 55384 b454dec21c2d9f45ec0b793f0995b992 "" | |
| 145 | + "/usr/local/texlive/2026/texmf-dist/web2c/texmf.cnf" 1770478701 43630 e9fa012910491fe5ae08a72c3c2997a7 "" | |
| 146 | + "/usr/local/texlive/2026/texmf-var/fonts/map/pdftex/updmap/pdftex.map" 1776992453 5573038 b99aad5e4898a9034e953632f940512b "" | |
| 147 | + "/usr/local/texlive/2026/texmf-var/web2c/pdftex/pdflatex.fmt" 1774209607 3636776 dc5fa9b6b1453b9d196cd42edf876004 "" | |
| 148 | + "/usr/local/texlive/2026/texmf.cnf" 1774209515 577 648cd9f92f9c0eac486f4c71c85cfbbe "" | |
| 149 | + "appendix/appendix.tex" 1791669372.50415 3692 978627ae3304068b6095a969913dfa4e "" | |
| 150 | + "figures/f1_mdape_vs_k.pdf" 1791669587.98363 36037 e355c81c5fedba32e832a24c3f4408be "" | |
| 151 | + "figures/f2_surface_d_T.pdf" 1791669587.98375 25994 1cbd1b2bd396b69eb2de3ef24917c2ad "" | |
| 152 | + "figures/f3_ajustements.pdf" 1791669587.98385 36136 d300894232678abe72aa465a8d4514d1 "" | |
| 153 | + "figures/f4_competition.pdf" 1791669587.98395 40740 22b7c55d5bd5e089cec626a605271727 "" | |
| 154 | + "figures/f5_densite.pdf" 1791669587.98405 35609 e96ada411d9e6fa6195cdf974bc5e2d9 "" | |
| 155 | + "figures/f7_indice.pdf" 1791669587.98426 32594 b053e3f3f6425f0cbdca6080697aae6d "" | |
| 156 | + "main.aux" 1791669591.43573 22155 66f06235bf01cd09404880f862499c71 "pdflatex" | |
| 157 | + "main.bbl" 1791669590.74854 7426 189002991017ce2edb4cbc51edc5c910 "bibtex main" | |
| 158 | + "main.out" 1791669591.43699 7416 bf1dd9fe3c2a7a6e19ac40b70b222a94 "pdflatex" | |
| 159 | + "main.tex" 1791668967.77061 5795 731299ff1ece15e1ee4fe9c5b5a60161 "" | |
| 160 | + "sections/conclusion.tex" 1791669308.39232 3338 78d7447ef4108c4c3317e331a2d362ca "" | |
| 161 | + "sections/data.tex" 1791669168.2267 6652 c60d1f77ad533b10b3a19fd4c74c920c "" | |
| 162 | + "sections/introduction.tex" 1791669076.68532 8175 2bbbe12a7b1eb46260e5fb05ad8871e6 "" | |
| 163 | + "sections/literature.tex" 1791669397.57194 7416 fcf0f91922c2a753315e7447161f83ad "" | |
| 164 | + "sections/methodology.tex" 1791669168.22958 12016 6e19e7205805f92dd2c552cdac0f1ea6 "" | |
| 165 | + "sections/results.tex" 1791669308.38701 20700 cf4e58d2ab9874499c3dd14aeae553bc "" | |
| 166 | + "sections/robustness.tex" 1791669308.38987 4679 bf493ecdc1a68ee7068bc6c075e89794 "" | |
| 167 | + "sections/titlepage.tex" 1791668938.15793 1458 1242e8e90e0d8dba3af4619bac574330 "" | |
| 168 | + "tables/adjustments.tex" 1791669587.93708 619 4da35be663bc2987d7e2e6e6160033a7 "" | |
| 169 | + "tables/competition.tex" 1791669587.97048 1609 fedc41b78e516ff6279b721b259ecee8 "" | |
| 170 | + "tables/cond_rules.tex" 1791669587.98284 728 9cd4052824e8c66a496224326b8bd730 "" | |
| 171 | + "tables/density.tex" 1791669587.97424 490 6548c4e056702791a45a92d4c99ded08 "" | |
| 172 | + "tables/density_dist.tex" 1791669587.92592 252 a22d5551c517c29a57c714a97962e9ed "" | |
| 173 | + "tables/descriptives.tex" 1791669587.92562 554 92426b58e569ae8bc50b170cb2602356 "" | |
| 174 | + "tables/e1_best.tex" 1791669587.93359 541 f725375dfefb305a29cc241720be4165 "" | |
| 175 | + "tables/exclusions.tex" 1791669587.92509 912 38d67ac94c92d67670ec913b07170893 "" | |
| 176 | + "tables/grid.tex" 1791669587.96621 468 ca25d30bb81416d1e88594d7477d619a "" | |
| 177 | + "tables/h2.tex" 1791669587.93419 571 500c1470f6bf48c977871ef111677206 "" | |
| 178 | + "tables/h3.tex" 1791669587.93803 862 c86bda54ae7744821e2da1eef7c20d7a "" | |
| 179 | + "tables/h4.tex" 1791669587.969 549 2c056e3ed64fcd1e813e12958d5646d9 "" | |
| 180 | + "tables/h5.tex" 1791669587.98134 389 7f44b67c4104e20155c316efeffc648f "" | |
| 181 | + "tables/intervals.tex" 1791669587.98192 686 6057227c93b778b03e0cb3ee6c93b590 "" | |
| 182 | + "tables/marginal.tex" 1791669587.96935 779 1aeae827a0c45f64f982fd5c71eaf1fb "" | |
| 183 | + "tables/numbers.tex" 1791669587.98525 3128 68e923e2cd1be3ef00574bcc221e5bdc "" | |
| 184 | + "tables/pairs.tex" 1791669587.97237 970 ed0ba5a37367eb621b0191989a08e0e9 "" | |
| 185 | + "tables/regions.tex" 1791669587.97662 1478 99b0aa03520a52b944bf2fd38dfbb9fc "" | |
| 186 | + "tables/selection.tex" 1791669587.96724 304 f4292253fb696f6609af5d60de099448 "" | |
| 187 | + "tables/stability.tex" 1791669587.98348 588 681cdb7ff7cb69b84248f56ff21294d6 "" | |
| 188 | + "tables/strata_other.tex" 1791669587.98078 1273 5cb3a1299a0cf09db5d00470f6f90e97 "" | |
| 189 | + "tables/surface_cov.tex" 1791669587.93279 383 2e8f2138b19787afa928c1f6b97ffee2 "" | |
| 190 | + "tables/surface_geo_hedo.tex" 1791669587.92972 679 538fad263bc8a0b1ee338c1af3b12776 "" | |
| 191 | + "tables/surface_geo_none.tex" 1791669587.93091 679 044182fe5d472eee588250e0f7d67800 "" | |
| 192 | + "tables/surface_sim_hedo.tex" 1791669587.93194 679 7e1a2230ac2699c84858722e48fa34eb "" | |
| 193 | + "tables/weighting.tex" 1791669587.96827 710 62650f8a7708ca522dec3f5779199f04 "" | |
| 194 | + "uq_logo.jpg" 1791668938.162 58719 668023c2daab9af750d1f0889564af82 "" | |
| 195 | + (generated) | |
| 196 | + "main.aux" | |
| 197 | + "main.log" | |
| 198 | + "main.out" | |
| 199 | + "main.pdf" | |
| 200 | + (rewritten before read) | |
added
paper/main.fls
+603 −0
@@ -0,0 +1,603 @@ | ||
| 1 | +PWD /Users/simon-pierreboucher/Desktop/Validation-Comparables/paper | |
| 2 | +INPUT /usr/local/texlive/2026/texmf.cnf | |
| 3 | +INPUT /usr/local/texlive/2026/texmf-dist/web2c/texmf.cnf | |
| 4 | +INPUT /usr/local/texlive/2026/texmf-var/web2c/pdftex/pdflatex.fmt | |
| 5 | +INPUT main.tex | |
| 6 | +OUTPUT main.log | |
| 7 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/article.cls | |
| 8 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/article.cls | |
| 9 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo | |
| 10 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo | |
| 11 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo | |
| 12 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/map/fontname/texfonts.map | |
| 13 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr12.tfm | |
| 14 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/inputenc.sty | |
| 15 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/inputenc.sty | |
| 16 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/fontenc.sty | |
| 17 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/base/fontenc.sty | |
| 18 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/jknappen/ec/ecrm1200.tfm | |
| 19 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/babel.sty | |
| 20 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/babel.sty | |
| 21 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/txtbabel.def | |
| 22 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.tex | |
| 23 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.tex | |
| 24 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.tex | |
| 25 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-en.ini | |
| 26 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf | |
| 27 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf | |
| 28 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf | |
| 29 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/geometry/geometry.sty | |
| 30 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/geometry/geometry.sty | |
| 31 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/keyval.sty | |
| 32 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/keyval.sty | |
| 33 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/iftex/ifvtex.sty | |
| 34 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/iftex/ifvtex.sty | |
| 35 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/iftex/iftex.sty | |
| 36 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/iftex/iftex.sty | |
| 37 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/mathptmx.sty | |
| 38 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/mathptmx.sty | |
| 39 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.sty | |
| 40 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.sty | |
| 41 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/etoolbox/etoolbox.sty | |
| 42 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/etoolbox/etoolbox.sty | |
| 43 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def | |
| 44 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def | |
| 45 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def | |
| 46 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg | |
| 47 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg | |
| 48 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg | |
| 49 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/setspace/setspace.sty | |
| 50 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/setspace/setspace.sty | |
| 51 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsmath.sty | |
| 52 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsmath.sty | |
| 53 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsopn.sty | |
| 54 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amstext.sty | |
| 55 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amstext.sty | |
| 56 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsgen.sty | |
| 57 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsgen.sty | |
| 58 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsbsy.sty | |
| 59 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsbsy.sty | |
| 60 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsopn.sty | |
| 61 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amssymb.sty | |
| 62 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amssymb.sty | |
| 63 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amsfonts.sty | |
| 64 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amsfonts.sty | |
| 65 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amscls/amsthm.sty | |
| 66 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/amscls/amsthm.sty | |
| 67 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/booktabs/booktabs.sty | |
| 68 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/booktabs/booktabs.sty | |
| 69 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/tools/tabularx.sty | |
| 70 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/tools/tabularx.sty | |
| 71 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/tools/array.sty | |
| 72 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/tools/array.sty | |
| 73 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/multirow/multirow.sty | |
| 74 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/multirow/multirow.sty | |
| 75 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/threeparttable/threeparttable.sty | |
| 76 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/threeparttable/threeparttable.sty | |
| 77 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/siunitx/siunitx.sty | |
| 78 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/siunitx/siunitx.sty | |
| 79 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations.sty | |
| 80 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations.sty | |
| 81 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty | |
| 82 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty | |
| 83 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/infwarerr/infwarerr.sty | |
| 84 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/infwarerr/infwarerr.sty | |
| 85 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty | |
| 86 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty | |
| 87 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphicx.sty | |
| 88 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphicx.sty | |
| 89 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphics.sty | |
| 90 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphics.sty | |
| 91 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/trig.sty | |
| 92 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/trig.sty | |
| 93 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg | |
| 94 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg | |
| 95 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg | |
| 96 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def | |
| 97 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def | |
| 98 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def | |
| 99 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/subcaption.sty | |
| 100 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/subcaption.sty | |
| 101 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption.sty | |
| 102 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption.sty | |
| 103 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption3.sty | |
| 104 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption3.sty | |
| 105 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/float/float.sty | |
| 106 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/float/float.sty | |
| 107 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/xcolor/xcolor.sty | |
| 108 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/xcolor/xcolor.sty | |
| 109 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg | |
| 110 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg | |
| 111 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg | |
| 112 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx | |
| 113 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx | |
| 114 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx | |
| 115 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def | |
| 116 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def | |
| 117 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def | |
| 118 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hyperref.sty | |
| 119 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hyperref.sty | |
| 120 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty | |
| 121 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty | |
| 122 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty | |
| 123 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty | |
| 124 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/pdfescape/pdfescape.sty | |
| 125 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/pdfescape/pdfescape.sty | |
| 126 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hycolor/hycolor.sty | |
| 127 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hycolor/hycolor.sty | |
| 128 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/nameref.sty | |
| 129 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/nameref.sty | |
| 130 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/refcount/refcount.sty | |
| 131 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/refcount/refcount.sty | |
| 132 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty | |
| 133 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty | |
| 134 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/kvoptions/kvoptions.sty | |
| 135 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/kvoptions/kvoptions.sty | |
| 136 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/stringenc/stringenc.sty | |
| 137 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/stringenc/stringenc.sty | |
| 138 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def | |
| 139 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def | |
| 140 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def | |
| 141 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/intcalc/intcalc.sty | |
| 142 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/intcalc/intcalc.sty | |
| 143 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def | |
| 144 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def | |
| 145 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def | |
| 146 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/url/url.sty | |
| 147 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/url/url.sty | |
| 148 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/bitset/bitset.sty | |
| 149 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/bitset/bitset.sty | |
| 150 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty | |
| 151 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty | |
| 152 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def | |
| 153 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def | |
| 154 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def | |
| 155 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty | |
| 156 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty | |
| 157 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty | |
| 158 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty | |
| 159 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/natbib/natbib.sty | |
| 160 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/natbib/natbib.sty | |
| 161 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty | |
| 162 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty | |
| 163 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/titlesec/titlesec.sty | |
| 164 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/titlesec/titlesec.sty | |
| 165 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/appendix/appendix.sty | |
| 166 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/appendix/appendix.sty | |
| 167 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/enumitem/enumitem.sty | |
| 168 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/enumitem/enumitem.sty | |
| 169 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/graphics/color.sty | |
| 170 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd | |
| 171 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd | |
| 172 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd | |
| 173 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 174 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def | |
| 175 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def | |
| 176 | +INPUT ./main.aux | |
| 177 | +INPUT ./main.aux | |
| 178 | +INPUT main.aux | |
| 179 | +OUTPUT main.aux | |
| 180 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg | |
| 181 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg | |
| 182 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg | |
| 183 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-dictionary-english.trsl | |
| 184 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-dictionary-english.trsl | |
| 185 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-dictionary-english.trsl | |
| 186 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii | |
| 187 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii | |
| 188 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii | |
| 189 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty | |
| 190 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty | |
| 191 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg | |
| 192 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg | |
| 193 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg | |
| 194 | +INPUT ./main.out | |
| 195 | +INPUT ./main.out | |
| 196 | +INPUT main.out | |
| 197 | +INPUT main.out | |
| 198 | +OUTPUT main.pdf | |
| 199 | +INPUT ./main.out | |
| 200 | +INPUT ./main.out | |
| 201 | +OUTPUT main.out | |
| 202 | +INPUT ./tables/numbers.tex | |
| 203 | +INPUT ./tables/numbers.tex | |
| 204 | +INPUT ./tables/numbers.tex | |
| 205 | +INPUT ./tables/numbers.tex | |
| 206 | +INPUT tables/numbers.tex | |
| 207 | +INPUT ./sections/titlepage.tex | |
| 208 | +INPUT ./sections/titlepage.tex | |
| 209 | +INPUT ./sections/titlepage.tex | |
| 210 | +INPUT ./sections/titlepage.tex | |
| 211 | +INPUT sections/titlepage.tex | |
| 212 | +INPUT ./uq_logo.jpg | |
| 213 | +INPUT ./uq_logo.jpg | |
| 214 | +INPUT ./uq_logo.jpg | |
| 215 | +INPUT ./uq_logo.jpg | |
| 216 | +INPUT ./uq_logo.jpg | |
| 217 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 218 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmrc8t.tfm | |
| 219 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 220 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm | |
| 221 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 222 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8t.tfm | |
| 223 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 224 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmrc8t.vf | |
| 225 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 226 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 227 | +INPUT /usr/local/texlive/2026/texmf-var/fonts/map/pdftex/updmap/pdftex.map | |
| 228 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/base/8r.enc | |
| 229 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf | |
| 230 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm | |
| 231 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmri8t.vf | |
| 232 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 233 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmr8t.vf | |
| 234 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 235 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmr8t.vf | |
| 236 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 237 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmr8t.vf | |
| 238 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 239 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm | |
| 240 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf | |
| 241 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm | |
| 242 | +INPUT ./sections/introduction.tex | |
| 243 | +INPUT ./sections/introduction.tex | |
| 244 | +INPUT ./sections/introduction.tex | |
| 245 | +INPUT ./sections/introduction.tex | |
| 246 | +INPUT sections/introduction.tex | |
| 247 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm | |
| 248 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8t.tfm | |
| 249 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd | |
| 250 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd | |
| 251 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd | |
| 252 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 253 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 254 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 255 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd | |
| 256 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd | |
| 257 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd | |
| 258 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 259 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 260 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 261 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd | |
| 262 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd | |
| 263 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd | |
| 264 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 265 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 266 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 267 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd | |
| 268 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd | |
| 269 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd | |
| 270 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 271 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 272 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 273 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd | |
| 274 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd | |
| 275 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd | |
| 276 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 277 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 278 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 279 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 280 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 281 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 282 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 283 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg | |
| 284 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg | |
| 285 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg | |
| 286 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 287 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 288 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 289 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg | |
| 290 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg | |
| 291 | +INPUT /usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg | |
| 292 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 293 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 294 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf | |
| 295 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm | |
| 296 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmri8t.vf | |
| 297 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 298 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf | |
| 299 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 300 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr10.tfm | |
| 301 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf | |
| 302 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm | |
| 303 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmmi10.tfm | |
| 304 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7y.vf | |
| 305 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmsy10.tfm | |
| 306 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/rsfs/rsfs10.tfm | |
| 307 | +INPUT ./sections/literature.tex | |
| 308 | +INPUT ./sections/literature.tex | |
| 309 | +INPUT ./sections/literature.tex | |
| 310 | +INPUT ./sections/literature.tex | |
| 311 | +INPUT sections/literature.tex | |
| 312 | +INPUT ./sections/data.tex | |
| 313 | +INPUT ./sections/data.tex | |
| 314 | +INPUT ./sections/data.tex | |
| 315 | +INPUT ./sections/data.tex | |
| 316 | +INPUT sections/data.tex | |
| 317 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm | |
| 318 | +INPUT ./tables/exclusions.tex | |
| 319 | +INPUT ./tables/exclusions.tex | |
| 320 | +INPUT ./tables/exclusions.tex | |
| 321 | +INPUT ./tables/exclusions.tex | |
| 322 | +INPUT tables/exclusions.tex | |
| 323 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf | |
| 324 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm | |
| 325 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf | |
| 326 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 327 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 328 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr10.tfm | |
| 329 | +INPUT ./tables/descriptives.tex | |
| 330 | +INPUT ./tables/descriptives.tex | |
| 331 | +INPUT ./tables/descriptives.tex | |
| 332 | +INPUT ./tables/descriptives.tex | |
| 333 | +INPUT tables/descriptives.tex | |
| 334 | +INPUT ./figures/f7_indice.pdf | |
| 335 | +INPUT ./figures/f7_indice.pdf | |
| 336 | +INPUT ./figures/f7_indice.pdf | |
| 337 | +INPUT ./figures/f7_indice.pdf | |
| 338 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 339 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 340 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 341 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 342 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 343 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 344 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 345 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 346 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 347 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 348 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 349 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 350 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 351 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 352 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 353 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 354 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 355 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 356 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 357 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 358 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 359 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 360 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 361 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 362 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf | |
| 363 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 364 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr10.tfm | |
| 365 | +INPUT ./tables/density_dist.tex | |
| 366 | +INPUT ./tables/density_dist.tex | |
| 367 | +INPUT ./tables/density_dist.tex | |
| 368 | +INPUT ./tables/density_dist.tex | |
| 369 | +INPUT tables/density_dist.tex | |
| 370 | +INPUT ./sections/methodology.tex | |
| 371 | +INPUT ./sections/methodology.tex | |
| 372 | +INPUT ./sections/methodology.tex | |
| 373 | +INPUT ./sections/methodology.tex | |
| 374 | +INPUT sections/methodology.tex | |
| 375 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 376 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 377 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf | |
| 378 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm | |
| 379 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmmi10.tfm | |
| 380 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 381 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb7t.vf | |
| 382 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7y.vf | |
| 383 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmsy10.tfm | |
| 384 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/rsfs/rsfs10.tfm | |
| 385 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7v.vf | |
| 386 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmex10.tfm | |
| 387 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmex10.tfm | |
| 388 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 389 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7y.vf | |
| 390 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmsy10.tfm | |
| 391 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/rsfs/rsfs10.tfm | |
| 392 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf | |
| 393 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm | |
| 394 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmmi10.tfm | |
| 395 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 396 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf | |
| 397 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm | |
| 398 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmmi10.tfm | |
| 399 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 400 | +INPUT ./sections/results.tex | |
| 401 | +INPUT ./sections/results.tex | |
| 402 | +INPUT ./sections/results.tex | |
| 403 | +INPUT ./sections/results.tex | |
| 404 | +INPUT sections/results.tex | |
| 405 | +INPUT ./figures/f1_mdape_vs_k.pdf | |
| 406 | +INPUT ./figures/f1_mdape_vs_k.pdf | |
| 407 | +INPUT ./figures/f1_mdape_vs_k.pdf | |
| 408 | +INPUT ./figures/f1_mdape_vs_k.pdf | |
| 409 | +INPUT ./figures/f2_surface_d_T.pdf | |
| 410 | +INPUT ./figures/f2_surface_d_T.pdf | |
| 411 | +INPUT ./figures/f2_surface_d_T.pdf | |
| 412 | +INPUT ./figures/f2_surface_d_T.pdf | |
| 413 | +INPUT ./tables/surface_geo_hedo.tex | |
| 414 | +INPUT ./tables/surface_geo_hedo.tex | |
| 415 | +INPUT ./tables/surface_geo_hedo.tex | |
| 416 | +INPUT ./tables/surface_geo_hedo.tex | |
| 417 | +INPUT tables/surface_geo_hedo.tex | |
| 418 | +INPUT ./tables/h2.tex | |
| 419 | +INPUT ./tables/h2.tex | |
| 420 | +INPUT ./tables/h2.tex | |
| 421 | +INPUT ./tables/h2.tex | |
| 422 | +INPUT tables/h2.tex | |
| 423 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 424 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7t.tfm | |
| 425 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 426 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7m.tfm | |
| 427 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 428 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7y.tfm | |
| 429 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 430 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/zptmcm7v.tfm | |
| 431 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 432 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb7t.tfm | |
| 433 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 434 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri7t.tfm | |
| 435 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 436 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msam10.tfm | |
| 437 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 438 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/amsfonts/symbols/msbm10.tfm | |
| 439 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmr8t.vf | |
| 440 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 441 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 442 | +INPUT ./tables/adjustments.tex | |
| 443 | +INPUT ./tables/adjustments.tex | |
| 444 | +INPUT ./tables/adjustments.tex | |
| 445 | +INPUT ./tables/adjustments.tex | |
| 446 | +INPUT tables/adjustments.tex | |
| 447 | +INPUT ./figures/f3_ajustements.pdf | |
| 448 | +INPUT ./figures/f3_ajustements.pdf | |
| 449 | +INPUT ./figures/f3_ajustements.pdf | |
| 450 | +INPUT ./figures/f3_ajustements.pdf | |
| 451 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7m.vf | |
| 452 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/psyro.tfm | |
| 453 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8r.tfm | |
| 454 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7y.vf | |
| 455 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 456 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/rsfs/rsfs10.tfm | |
| 457 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf | |
| 458 | +INPUT ./tables/h3.tex | |
| 459 | +INPUT ./tables/h3.tex | |
| 460 | +INPUT ./tables/h3.tex | |
| 461 | +INPUT ./tables/h3.tex | |
| 462 | +INPUT tables/h3.tex | |
| 463 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8t.tfm | |
| 464 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmri8t.tfm | |
| 465 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 466 | +INPUT ./tables/grid.tex | |
| 467 | +INPUT ./tables/grid.tex | |
| 468 | +INPUT ./tables/grid.tex | |
| 469 | +INPUT ./tables/grid.tex | |
| 470 | +INPUT tables/grid.tex | |
| 471 | +INPUT ./tables/selection.tex | |
| 472 | +INPUT ./tables/selection.tex | |
| 473 | +INPUT ./tables/selection.tex | |
| 474 | +INPUT ./tables/selection.tex | |
| 475 | +INPUT tables/selection.tex | |
| 476 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/zptmcm7t.vf | |
| 477 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/symbol/psyr.tfm | |
| 478 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8r.tfm | |
| 479 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/public/cm/cmr10.tfm | |
| 480 | +INPUT ./tables/weighting.tex | |
| 481 | +INPUT ./tables/weighting.tex | |
| 482 | +INPUT ./tables/weighting.tex | |
| 483 | +INPUT ./tables/weighting.tex | |
| 484 | +INPUT tables/weighting.tex | |
| 485 | +INPUT ./tables/h4.tex | |
| 486 | +INPUT ./tables/h4.tex | |
| 487 | +INPUT ./tables/h4.tex | |
| 488 | +INPUT ./tables/h4.tex | |
| 489 | +INPUT tables/h4.tex | |
| 490 | +INPUT ./tables/marginal.tex | |
| 491 | +INPUT ./tables/marginal.tex | |
| 492 | +INPUT ./tables/marginal.tex | |
| 493 | +INPUT ./tables/marginal.tex | |
| 494 | +INPUT tables/marginal.tex | |
| 495 | +INPUT ./tables/competition.tex | |
| 496 | +INPUT ./tables/competition.tex | |
| 497 | +INPUT ./tables/competition.tex | |
| 498 | +INPUT ./tables/competition.tex | |
| 499 | +INPUT tables/competition.tex | |
| 500 | +INPUT ./figures/f4_competition.pdf | |
| 501 | +INPUT ./figures/f4_competition.pdf | |
| 502 | +INPUT ./figures/f4_competition.pdf | |
| 503 | +INPUT ./figures/f4_competition.pdf | |
| 504 | +INPUT ./tables/pairs.tex | |
| 505 | +INPUT ./tables/pairs.tex | |
| 506 | +INPUT ./tables/pairs.tex | |
| 507 | +INPUT ./tables/pairs.tex | |
| 508 | +INPUT tables/pairs.tex | |
| 509 | +INPUT ./tables/density.tex | |
| 510 | +INPUT ./tables/density.tex | |
| 511 | +INPUT ./tables/density.tex | |
| 512 | +INPUT ./tables/density.tex | |
| 513 | +INPUT tables/density.tex | |
| 514 | +INPUT ./figures/f5_densite.pdf | |
| 515 | +INPUT ./figures/f5_densite.pdf | |
| 516 | +INPUT ./figures/f5_densite.pdf | |
| 517 | +INPUT ./figures/f5_densite.pdf | |
| 518 | +INPUT ./tables/h5.tex | |
| 519 | +INPUT ./tables/h5.tex | |
| 520 | +INPUT ./tables/h5.tex | |
| 521 | +INPUT ./tables/h5.tex | |
| 522 | +INPUT tables/h5.tex | |
| 523 | +INPUT ./tables/intervals.tex | |
| 524 | +INPUT ./tables/intervals.tex | |
| 525 | +INPUT ./tables/intervals.tex | |
| 526 | +INPUT ./tables/intervals.tex | |
| 527 | +INPUT tables/intervals.tex | |
| 528 | +INPUT ./tables/cond_rules.tex | |
| 529 | +INPUT ./tables/cond_rules.tex | |
| 530 | +INPUT ./tables/cond_rules.tex | |
| 531 | +INPUT ./tables/cond_rules.tex | |
| 532 | +INPUT tables/cond_rules.tex | |
| 533 | +INPUT ./sections/robustness.tex | |
| 534 | +INPUT ./sections/robustness.tex | |
| 535 | +INPUT ./sections/robustness.tex | |
| 536 | +INPUT ./sections/robustness.tex | |
| 537 | +INPUT sections/robustness.tex | |
| 538 | +INPUT ./tables/stability.tex | |
| 539 | +INPUT ./tables/stability.tex | |
| 540 | +INPUT ./tables/stability.tex | |
| 541 | +INPUT ./tables/stability.tex | |
| 542 | +INPUT tables/stability.tex | |
| 543 | +INPUT ./sections/conclusion.tex | |
| 544 | +INPUT ./sections/conclusion.tex | |
| 545 | +INPUT ./sections/conclusion.tex | |
| 546 | +INPUT ./sections/conclusion.tex | |
| 547 | +INPUT sections/conclusion.tex | |
| 548 | +INPUT ./main.bbl | |
| 549 | +INPUT ./main.bbl | |
| 550 | +INPUT main.bbl | |
| 551 | +INPUT ./appendix/appendix.tex | |
| 552 | +INPUT ./appendix/appendix.tex | |
| 553 | +INPUT ./appendix/appendix.tex | |
| 554 | +INPUT ./appendix/appendix.tex | |
| 555 | +INPUT appendix/appendix.tex | |
| 556 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmr8t.tfm | |
| 557 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8t.tfm | |
| 558 | +INPUT ./tables/surface_cov.tex | |
| 559 | +INPUT ./tables/surface_cov.tex | |
| 560 | +INPUT ./tables/surface_cov.tex | |
| 561 | +INPUT ./tables/surface_cov.tex | |
| 562 | +INPUT tables/surface_cov.tex | |
| 563 | +INPUT ./tables/surface_geo_none.tex | |
| 564 | +INPUT ./tables/surface_geo_none.tex | |
| 565 | +INPUT ./tables/surface_geo_none.tex | |
| 566 | +INPUT ./tables/surface_geo_none.tex | |
| 567 | +INPUT tables/surface_geo_none.tex | |
| 568 | +INPUT ./tables/surface_sim_hedo.tex | |
| 569 | +INPUT ./tables/surface_sim_hedo.tex | |
| 570 | +INPUT ./tables/surface_sim_hedo.tex | |
| 571 | +INPUT ./tables/surface_sim_hedo.tex | |
| 572 | +INPUT tables/surface_sim_hedo.tex | |
| 573 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/vf/adobe/times/ptmb8t.vf | |
| 574 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/tfm/adobe/times/ptmb8r.tfm | |
| 575 | +INPUT ./tables/e1_best.tex | |
| 576 | +INPUT ./tables/e1_best.tex | |
| 577 | +INPUT ./tables/e1_best.tex | |
| 578 | +INPUT ./tables/e1_best.tex | |
| 579 | +INPUT tables/e1_best.tex | |
| 580 | +INPUT ./tables/regions.tex | |
| 581 | +INPUT ./tables/regions.tex | |
| 582 | +INPUT ./tables/regions.tex | |
| 583 | +INPUT ./tables/regions.tex | |
| 584 | +INPUT tables/regions.tex | |
| 585 | +INPUT ./tables/strata_other.tex | |
| 586 | +INPUT ./tables/strata_other.tex | |
| 587 | +INPUT ./tables/strata_other.tex | |
| 588 | +INPUT ./tables/strata_other.tex | |
| 589 | +INPUT tables/strata_other.tex | |
| 590 | +INPUT main.aux | |
| 591 | +INPUT ./main.out | |
| 592 | +INPUT ./main.out | |
| 593 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb | |
| 594 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb | |
| 595 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb | |
| 596 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb | |
| 597 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/symbols/msbm10.pfb | |
| 598 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/public/rsfs/rsfs10.pfb | |
| 599 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb | |
| 600 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb | |
| 601 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmb8a.pfb | |
| 602 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmr8a.pfb | |
| 603 | +INPUT /usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8a.pfb | |
added
paper/main.log
+930 −0
@@ -0,0 +1,930 @@ | ||
| 1 | +This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 10 OCT 2026 17:59 | |
| 2 | +entering extended mode | |
| 3 | + restricted \write18 enabled. | |
| 4 | + %&-line parsing enabled. | |
| 5 | +**main.tex | |
| 6 | +(./main.tex | |
| 7 | +LaTeX2e <2025-11-01> | |
| 8 | +L3 programming layer <2026-01-19> | |
| 9 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/article.cls | |
| 10 | +Document Class: article 2025/01/22 v1.4n Standard LaTeX document class | |
| 11 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo | |
| 12 | +File: size12.clo 2025/01/22 v1.4n Standard LaTeX file (size option) | |
| 13 | +) | |
| 14 | +\c@part=\count275 | |
| 15 | +\c@section=\count276 | |
| 16 | +\c@subsection=\count277 | |
| 17 | +\c@subsubsection=\count278 | |
| 18 | +\c@paragraph=\count279 | |
| 19 | +\c@subparagraph=\count280 | |
| 20 | +\c@figure=\count281 | |
| 21 | +\c@table=\count282 | |
| 22 | +\abovecaptionskip=\skip49 | |
| 23 | +\belowcaptionskip=\skip50 | |
| 24 | +\bibindent=\dimen148 | |
| 25 | +) | |
| 26 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/inputenc.sty | |
| 27 | +Package: inputenc 2024/02/08 v1.3d Input encoding file | |
| 28 | +\inpenc@prehook=\toks17 | |
| 29 | +\inpenc@posthook=\toks18 | |
| 30 | +) | |
| 31 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/fontenc.sty | |
| 32 | +Package: fontenc 2025/07/18 v2.1d Standard LaTeX package | |
| 33 | +) | |
| 34 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/babel.sty | |
| 35 | +Package: babel 2026/03/08 v26.4 The multilingual framework for LuaLaTeX, pdfLaT | |
| 36 | +eX and XeLaTeX | |
| 37 | +\babel@savecnt=\count283 | |
| 38 | +LaTeX Encoding Info: Redeclaring text command \ij (encoding OT1) on input li | |
| 39 | +ne 2078. | |
| 40 | +LaTeX Encoding Info: Redeclaring text command \IJ (encoding OT1) on input li | |
| 41 | +ne 2080. | |
| 42 | +LaTeX Encoding Info: Redeclaring text command \ij (encoding T1) on input lin | |
| 43 | +e 2082. | |
| 44 | +LaTeX Encoding Info: Redeclaring text command \IJ (encoding T1) on input lin | |
| 45 | +e 2083. | |
| 46 | +LaTeX Encoding Info: Ignoring declaration for text command \ij (encoding ?) | |
| 47 | +on input line 2084. | |
| 48 | +LaTeX Encoding Info: Ignoring declaration for text command \IJ (encoding ?) | |
| 49 | +on input line 2086. | |
| 50 | +LaTeX Encoding Info: Ignoring declaration for text command \SS (encoding ?) | |
| 51 | +on input line 2111. | |
| 52 | +\U@D=\dimen149 | |
| 53 | +\l@unhyphenated=\language91 | |
| 54 | + | |
| 55 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/txtbabel.def) | |
| 56 | +\bbl@readstream=\read2 | |
| 57 | +\bbl@dirlevel=\count284 | |
| 58 | + | |
| 59 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.t | |
| 60 | +ex) | |
| 61 | +Package babel Info: Importing font and identification data for english | |
| 62 | +(babel) from babel-en.ini. Reported on input line 4330. | |
| 63 | + (/usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf | |
| 64 | +Language: english 2017/06/06 v3.3r English support from the babel system | |
| 65 | +Package babel Info: Hyphen rules for 'canadian' set to \l@english | |
| 66 | +(babel) (\language0). Reported on input line 102. | |
| 67 | +Package babel Info: Hyphen rules for 'australian' set to \l@ukenglish | |
| 68 | +(babel) (\language22). Reported on input line 105. | |
| 69 | +Package babel Info: Hyphen rules for 'newzealand' set to \l@ukenglish | |
| 70 | +(babel) (\language22). Reported on input line 108. | |
| 71 | +)) (/usr/local/texlive/2026/texmf-dist/tex/latex/geometry/geometry.sty | |
| 72 | +Package: geometry 2026/03/07 v6.0 Page Geometry | |
| 73 | + | |
| 74 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/keyval.sty | |
| 75 | +Package: keyval 2022/05/29 v1.15 key=value parser (DPC) | |
| 76 | +\KV@toks@=\toks19 | |
| 77 | +) | |
| 78 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/ifvtex.sty | |
| 79 | +Package: ifvtex 2019/10/25 v1.7 ifvtex legacy package. Use iftex instead. | |
| 80 | + | |
| 81 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/iftex.sty | |
| 82 | +Package: iftex 2024/12/12 v1.0g TeX engine tests | |
| 83 | +)) | |
| 84 | +\Gm@cnth=\count285 | |
| 85 | +\Gm@cntv=\count286 | |
| 86 | +\c@Gm@tempcnt=\count287 | |
| 87 | +\Gm@bindingoffset=\dimen150 | |
| 88 | +\Gm@wd@mp=\dimen151 | |
| 89 | +\Gm@odd@mp=\dimen152 | |
| 90 | +\Gm@even@mp=\dimen153 | |
| 91 | +\Gm@layoutwidth=\dimen154 | |
| 92 | +\Gm@layoutheight=\dimen155 | |
| 93 | +\Gm@layouthoffset=\dimen156 | |
| 94 | +\Gm@layoutvoffset=\dimen157 | |
| 95 | +\Gm@dimlist=\toks20 | |
| 96 | +) | |
| 97 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/mathptmx.sty | |
| 98 | +Package: mathptmx 2020/03/25 PSNFSS-v9.3 Times w/ Math, improved (SPQR, WaS) | |
| 99 | +LaTeX Font Info: Redeclaring symbol font `operators' on input line 28. | |
| 100 | +LaTeX Font Info: Overwriting symbol font `operators' in version `normal' | |
| 101 | +(Font) OT1/cmr/m/n --> OT1/ztmcm/m/n on input line 28. | |
| 102 | +LaTeX Font Info: Overwriting symbol font `operators' in version `bold' | |
| 103 | +(Font) OT1/cmr/bx/n --> OT1/ztmcm/m/n on input line 28. | |
| 104 | +LaTeX Font Info: Redeclaring symbol font `letters' on input line 29. | |
| 105 | +LaTeX Font Info: Overwriting symbol font `letters' in version `normal' | |
| 106 | +(Font) OML/cmm/m/it --> OML/ztmcm/m/it on input line 29. | |
| 107 | +LaTeX Font Info: Overwriting symbol font `letters' in version `bold' | |
| 108 | +(Font) OML/cmm/b/it --> OML/ztmcm/m/it on input line 29. | |
| 109 | +LaTeX Font Info: Redeclaring symbol font `symbols' on input line 30. | |
| 110 | +LaTeX Font Info: Overwriting symbol font `symbols' in version `normal' | |
| 111 | +(Font) OMS/cmsy/m/n --> OMS/ztmcm/m/n on input line 30. | |
| 112 | +LaTeX Font Info: Overwriting symbol font `symbols' in version `bold' | |
| 113 | +(Font) OMS/cmsy/b/n --> OMS/ztmcm/m/n on input line 30. | |
| 114 | +LaTeX Font Info: Redeclaring symbol font `largesymbols' on input line 31. | |
| 115 | +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal' | |
| 116 | +(Font) OMX/cmex/m/n --> OMX/ztmcm/m/n on input line 31. | |
| 117 | +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold' | |
| 118 | +(Font) OMX/cmex/m/n --> OMX/ztmcm/m/n on input line 31. | |
| 119 | +\symbold=\mathgroup4 | |
| 120 | +\symitalic=\mathgroup5 | |
| 121 | +LaTeX Font Info: Redeclaring math alphabet \mathbf on input line 34. | |
| 122 | +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal' | |
| 123 | +(Font) OT1/cmr/bx/n --> OT1/ptm/bx/n on input line 34. | |
| 124 | +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold' | |
| 125 | +(Font) OT1/cmr/bx/n --> OT1/ptm/bx/n on input line 34. | |
| 126 | +LaTeX Font Info: Redeclaring math alphabet \mathit on input line 35. | |
| 127 | +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal' | |
| 128 | +(Font) OT1/cmr/m/it --> OT1/ptm/m/it on input line 35. | |
| 129 | +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold' | |
| 130 | +(Font) OT1/cmr/bx/it --> OT1/ptm/m/it on input line 35. | |
| 131 | +LaTeX Info: Redefining \hbar on input line 50. | |
| 132 | +) | |
| 133 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.sty | |
| 134 | +Package: microtype 2026/03/01 v3.2d Micro-typographical refinements (RS) | |
| 135 | + | |
| 136 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/etoolbox/etoolbox.sty | |
| 137 | +Package: etoolbox 2025/10/02 v2.5m e-TeX tools for LaTeX (JAW) | |
| 138 | +\etb@tempcnta=\count288 | |
| 139 | +) | |
| 140 | +\MT@toks=\toks21 | |
| 141 | +\MT@tempbox=\box53 | |
| 142 | +\MT@count=\count289 | |
| 143 | +LaTeX Info: Redefining \noprotrusionifhmode on input line 1084. | |
| 144 | +LaTeX Info: Redefining \leftprotrusion on input line 1085. | |
| 145 | +\MT@prot@toks=\toks22 | |
| 146 | +LaTeX Info: Redefining \rightprotrusion on input line 1104. | |
| 147 | +LaTeX Info: Redefining \textls on input line 1449. | |
| 148 | +\MT@outer@kern=\dimen158 | |
| 149 | +LaTeX Info: Redefining \microtypecontext on input line 2053. | |
| 150 | +LaTeX Info: Redefining \textmicrotypecontext on input line 2070. | |
| 151 | +\MT@listname@count=\count290 | |
| 152 | + | |
| 153 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def | |
| 154 | +File: microtype-pdftex.def 2026/03/01 v3.2d Definitions specific to pdftex (RS) | |
| 155 | + | |
| 156 | +LaTeX Info: Redefining \lsstyle on input line 944. | |
| 157 | +LaTeX Info: Redefining \lslig on input line 944. | |
| 158 | +\MT@outer@space=\skip51 | |
| 159 | +) | |
| 160 | +Package microtype Info: Loading configuration file microtype.cfg. | |
| 161 | + | |
| 162 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg | |
| 163 | +File: microtype.cfg 2026/03/01 v3.2d microtype main configuration file (RS) | |
| 164 | +) | |
| 165 | +LaTeX Info: Redefining \microtypesetup on input line 3065. | |
| 166 | +) | |
| 167 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/setspace/setspace.sty | |
| 168 | +Package: setspace 2022/12/04 v6.7b set line spacing | |
| 169 | +) | |
| 170 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsmath.sty | |
| 171 | +Package: amsmath 2025/07/09 v2.17z AMS math features | |
| 172 | +\@mathmargin=\skip52 | |
| 173 | + | |
| 174 | +For additional information on amsmath, use the `?' option. | |
| 175 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amstext.sty | |
| 176 | +Package: amstext 2024/11/17 v2.01 AMS text | |
| 177 | + | |
| 178 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsgen.sty | |
| 179 | +File: amsgen.sty 1999/11/30 v2.0 generic functions | |
| 180 | +\@emptytoks=\toks23 | |
| 181 | +\ex@=\dimen159 | |
| 182 | +)) | |
| 183 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsbsy.sty | |
| 184 | +Package: amsbsy 1999/11/29 v1.2d Bold Symbols | |
| 185 | +\pmbraise@=\dimen160 | |
| 186 | +) | |
| 187 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsopn.sty | |
| 188 | +Package: amsopn 2022/04/08 v2.04 operator names | |
| 189 | +) | |
| 190 | +\inf@bad=\count291 | |
| 191 | +LaTeX Info: Redefining \frac on input line 233. | |
| 192 | +\uproot@=\count292 | |
| 193 | +\leftroot@=\count293 | |
| 194 | +LaTeX Info: Redefining \overline on input line 398. | |
| 195 | +LaTeX Info: Redefining \colon on input line 409. | |
| 196 | +\classnum@=\count294 | |
| 197 | +\DOTSCASE@=\count295 | |
| 198 | +LaTeX Info: Redefining \ldots on input line 495. | |
| 199 | +LaTeX Info: Redefining \dots on input line 498. | |
| 200 | +LaTeX Info: Redefining \cdots on input line 619. | |
| 201 | +\Mathstrutbox@=\box54 | |
| 202 | +\strutbox@=\box55 | |
| 203 | +LaTeX Info: Redefining \big on input line 721. | |
| 204 | +LaTeX Info: Redefining \Big on input line 722. | |
| 205 | +LaTeX Info: Redefining \bigg on input line 723. | |
| 206 | +LaTeX Info: Redefining \Bigg on input line 724. | |
| 207 | +\big@size=\dimen161 | |
| 208 | +LaTeX Font Info: Redeclaring font encoding OML on input line 742. | |
| 209 | +LaTeX Font Info: Redeclaring font encoding OMS on input line 743. | |
| 210 | +\macc@depth=\count296 | |
| 211 | +LaTeX Info: Redefining \bmod on input line 904. | |
| 212 | +LaTeX Info: Redefining \pmod on input line 909. | |
| 213 | +LaTeX Info: Redefining \smash on input line 939. | |
| 214 | +LaTeX Info: Redefining \relbar on input line 969. | |
| 215 | +LaTeX Info: Redefining \Relbar on input line 970. | |
| 216 | +\c@MaxMatrixCols=\count297 | |
| 217 | +\dotsspace@=\muskip17 | |
| 218 | +\c@parentequation=\count298 | |
| 219 | +\dspbrk@lvl=\count299 | |
| 220 | +\tag@help=\toks24 | |
| 221 | +\row@=\count300 | |
| 222 | +\column@=\count301 | |
| 223 | +\maxfields@=\count302 | |
| 224 | +\andhelp@=\toks25 | |
| 225 | +\eqnshift@=\dimen162 | |
| 226 | +\alignsep@=\dimen163 | |
| 227 | +\tagshift@=\dimen164 | |
| 228 | +\tagwidth@=\dimen165 | |
| 229 | +\totwidth@=\dimen166 | |
| 230 | +\lineht@=\dimen167 | |
| 231 | +\@envbody=\toks26 | |
| 232 | +\multlinegap=\skip53 | |
| 233 | +\multlinetaggap=\skip54 | |
| 234 | +\mathdisplay@stack=\toks27 | |
| 235 | +LaTeX Info: Redefining \[ on input line 2950. | |
| 236 | +LaTeX Info: Redefining \] on input line 2951. | |
| 237 | +) | |
| 238 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amssymb.sty | |
| 239 | +Package: amssymb 2013/01/14 v3.01 AMS font symbols | |
| 240 | + | |
| 241 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amsfonts.sty | |
| 242 | +Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support | |
| 243 | +\symAMSa=\mathgroup6 | |
| 244 | +\symAMSb=\mathgroup7 | |
| 245 | +LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' | |
| 246 | +(Font) U/euf/m/n --> U/euf/b/n on input line 106. | |
| 247 | +)) | |
| 248 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amscls/amsthm.sty | |
| 249 | +Package: amsthm 2020/05/29 v2.20.6 | |
| 250 | +\thm@style=\toks28 | |
| 251 | +\thm@bodyfont=\toks29 | |
| 252 | +\thm@headfont=\toks30 | |
| 253 | +\thm@notefont=\toks31 | |
| 254 | +\thm@headpunct=\toks32 | |
| 255 | +\thm@preskip=\skip55 | |
| 256 | +\thm@postskip=\skip56 | |
| 257 | +\thm@headsep=\skip57 | |
| 258 | +\dth@everypar=\toks33 | |
| 259 | +) | |
| 260 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/booktabs/booktabs.sty | |
| 261 | +Package: booktabs 2020/01/12 v1.61803398 Publication quality tables | |
| 262 | +\heavyrulewidth=\dimen168 | |
| 263 | +\lightrulewidth=\dimen169 | |
| 264 | +\cmidrulewidth=\dimen170 | |
| 265 | +\belowrulesep=\dimen171 | |
| 266 | +\belowbottomsep=\dimen172 | |
| 267 | +\aboverulesep=\dimen173 | |
| 268 | +\abovetopsep=\dimen174 | |
| 269 | +\cmidrulesep=\dimen175 | |
| 270 | +\cmidrulekern=\dimen176 | |
| 271 | +\defaultaddspace=\dimen177 | |
| 272 | +\@cmidla=\count303 | |
| 273 | +\@cmidlb=\count304 | |
| 274 | +\@aboverulesep=\dimen178 | |
| 275 | +\@belowrulesep=\dimen179 | |
| 276 | +\@thisruleclass=\count305 | |
| 277 | +\@lastruleclass=\count306 | |
| 278 | +\@thisrulewidth=\dimen180 | |
| 279 | +) | |
| 280 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/tabularx.sty | |
| 281 | +Package: tabularx 2023/12/11 v2.12a `tabularx' package (DPC) | |
| 282 | + | |
| 283 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/array.sty | |
| 284 | +Package: array 2025/09/25 v2.6n Tabular extension package (FMi) | |
| 285 | +\col@sep=\dimen181 | |
| 286 | +\ar@mcellbox=\box56 | |
| 287 | +\extrarowheight=\dimen182 | |
| 288 | +\NC@list=\toks34 | |
| 289 | +\extratabsurround=\skip58 | |
| 290 | +\backup@length=\skip59 | |
| 291 | +\ar@cellbox=\box57 | |
| 292 | +) | |
| 293 | +\TX@col@width=\dimen183 | |
| 294 | +\TX@old@table=\dimen184 | |
| 295 | +\TX@old@col=\dimen185 | |
| 296 | +\TX@target=\dimen186 | |
| 297 | +\TX@delta=\dimen187 | |
| 298 | +\TX@cols=\count307 | |
| 299 | +\TX@ftn=\toks35 | |
| 300 | +) | |
| 301 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/multirow/multirow.sty | |
| 302 | +Package: multirow 2024/11/12 v2.9 Span multiple rows of a table | |
| 303 | +\multirow@colwidth=\skip60 | |
| 304 | +\multirow@cntb=\count308 | |
| 305 | +\multirow@dima=\skip61 | |
| 306 | +\bigstrutjot=\dimen188 | |
| 307 | +) | |
| 308 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/threeparttable/threeparttable.sty | |
| 309 | +Package: threeparttable 2003/06/13 v 3.0 | |
| 310 | +\@tempboxb=\box58 | |
| 311 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/siunitx/siunitx.sty | |
| 312 | +Package: siunitx 2025-07-09 v3.4.14 A comprehensive (SI) units package | |
| 313 | +\l__siunitx_number_uncert_offset_int=\count309 | |
| 314 | +\l__siunitx_number_exponent_fixed_int=\count310 | |
| 315 | +\l__siunitx_number_min_decimal_int=\count311 | |
| 316 | +\l__siunitx_number_min_integer_int=\count312 | |
| 317 | +\l__siunitx_number_round_precision_int=\count313 | |
| 318 | +\l__siunitx_number_lower_threshold_int=\count314 | |
| 319 | +\l__siunitx_number_upper_threshold_int=\count315 | |
| 320 | +\l__siunitx_number_group_first_int=\count316 | |
| 321 | +\l__siunitx_number_group_size_int=\count317 | |
| 322 | +\l__siunitx_number_group_minimum_int=\count318 | |
| 323 | +\l__siunitx_angle_tmp_dim=\dimen189 | |
| 324 | +\l__siunitx_angle_marker_box=\box59 | |
| 325 | +\l__siunitx_angle_unit_box=\box60 | |
| 326 | +\l__siunitx_compound_count_int=\count319 | |
| 327 | + | |
| 328 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations.sty | |
| 329 | +Package: translations 2022/02/05 v1.12 internationalization of LaTeX2e packages | |
| 330 | + (CN) | |
| 331 | + | |
| 332 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty | |
| 333 | +Package: pdftexcmds 2020-06-27 v0.33 Utility functions of pdfTeX for LuaTeX (HO | |
| 334 | +) | |
| 335 | + | |
| 336 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/infwarerr/infwarerr.sty | |
| 337 | +Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO) | |
| 338 | +) | |
| 339 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty | |
| 340 | +Package: ltxcmds 2023-12-04 v1.26 LaTeX kernel commands for general use (HO) | |
| 341 | +) | |
| 342 | +Package pdftexcmds Info: \pdf@primitive is available. | |
| 343 | +Package pdftexcmds Info: \pdf@ifprimitive is available. | |
| 344 | +Package pdftexcmds Info: \pdfdraftmode found. | |
| 345 | +)) | |
| 346 | +\l__siunitx_table_tmp_box=\box61 | |
| 347 | +\l__siunitx_table_tmp_dim=\dimen190 | |
| 348 | +\l__siunitx_table_column_width_dim=\dimen191 | |
| 349 | +\l__siunitx_table_integer_box=\box62 | |
| 350 | +\l__siunitx_table_decimal_box=\box63 | |
| 351 | +\l__siunitx_table_uncert_box=\box64 | |
| 352 | +\l__siunitx_table_before_box=\box65 | |
| 353 | +\l__siunitx_table_after_box=\box66 | |
| 354 | +\l__siunitx_table_before_dim=\dimen192 | |
| 355 | +\l__siunitx_table_carry_dim=\dimen193 | |
| 356 | +\l__siunitx_unit_tmp_int=\count320 | |
| 357 | +\l__siunitx_unit_position_int=\count321 | |
| 358 | +\l__siunitx_unit_total_int=\count322 | |
| 359 | +) | |
| 360 | + | |
| 361 | +Package siunitx Info: Option "input-symbols" has been deprecated in this | |
| 362 | +(siunitx) release. | |
| 363 | +(siunitx) | |
| 364 | +(siunitx) Use "input-digits" as a replacement. | |
| 365 | + | |
| 366 | + | |
| 367 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphicx.sty | |
| 368 | +Package: graphicx 2024/12/31 v1.2e Enhanced LaTeX Graphics (DPC,SPQR) | |
| 369 | + | |
| 370 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphics.sty | |
| 371 | +Package: graphics 2024/08/06 v1.4g Standard LaTeX Graphics (DPC,SPQR) | |
| 372 | + | |
| 373 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/trig.sty | |
| 374 | +Package: trig 2023/12/02 v1.11 sin cos tan (DPC) | |
| 375 | +) | |
| 376 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg | |
| 377 | +File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration | |
| 378 | +) | |
| 379 | +Package graphics Info: Driver file: pdftex.def on input line 106. | |
| 380 | + | |
| 381 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def | |
| 382 | +File: pdftex.def 2025/09/29 v1.2d Graphics/color driver for pdftex | |
| 383 | +)) | |
| 384 | +\Gin@req@height=\dimen194 | |
| 385 | +\Gin@req@width=\dimen195 | |
| 386 | +) | |
| 387 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/subcaption.sty | |
| 388 | +Package: subcaption 2023/07/28 v1.6b Sub-captions (AR) | |
| 389 | + | |
| 390 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption.sty | |
| 391 | +Package: caption 2023/08/05 v3.6o Customizing captions (AR) | |
| 392 | + | |
| 393 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption3.sty | |
| 394 | +Package: caption3 2023/07/31 v2.4d caption3 kernel (AR) | |
| 395 | +\caption@tempdima=\dimen196 | |
| 396 | +\captionmargin=\dimen197 | |
| 397 | +\caption@leftmargin=\dimen198 | |
| 398 | +\caption@rightmargin=\dimen199 | |
| 399 | +\caption@width=\dimen256 | |
| 400 | +\caption@indent=\dimen257 | |
| 401 | +\caption@parindent=\dimen258 | |
| 402 | +\caption@hangindent=\dimen259 | |
| 403 | +Package caption Info: Standard document class detected. | |
| 404 | +) | |
| 405 | +\c@caption@flags=\count323 | |
| 406 | +\c@continuedfloat=\count324 | |
| 407 | +Package caption Info: threeparttable package is loaded. | |
| 408 | +) | |
| 409 | +Package caption Info: New subtype `subfigure' on input line 238. | |
| 410 | +\c@subfigure=\count325 | |
| 411 | +Package caption Info: New subtype `subtable' on input line 238. | |
| 412 | +\c@subtable=\count326 | |
| 413 | +) | |
| 414 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/float/float.sty | |
| 415 | +Package: float 2001/11/08 v1.3d Float enhancements (AL) | |
| 416 | +\c@float@type=\count327 | |
| 417 | +\float@exts=\toks36 | |
| 418 | +\float@box=\box67 | |
| 419 | +\@float@everytoks=\toks37 | |
| 420 | +\@floatcapt=\box68 | |
| 421 | +) | |
| 422 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/xcolor/xcolor.sty | |
| 423 | +Package: xcolor 2024/09/29 v3.02 LaTeX color extensions (UK) | |
| 424 | + | |
| 425 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg | |
| 426 | +File: color.cfg 2016/01/02 v1.6 sample color configuration | |
| 427 | +) | |
| 428 | +Package xcolor Info: Driver file: pdftex.def on input line 274. | |
| 429 | + | |
| 430 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx) | |
| 431 | +Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1349. | |
| 432 | +Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1353. | |
| 433 | +Package xcolor Info: Model `RGB' extended on input line 1365. | |
| 434 | +Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1367. | |
| 435 | +Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1368. | |
| 436 | +Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1369. | |
| 437 | +Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1370. | |
| 438 | +Package xcolor Info: Model `Gray' substituted by `gray' on input line 1371. | |
| 439 | +Package xcolor Info: Model `wave' substituted by `hsb' on input line 1372. | |
| 440 | +) | |
| 441 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def | |
| 442 | +File: dvipsnam.def 2016/06/17 v3.0m Driver-dependent file (DPC,SPQR) | |
| 443 | +) | |
| 444 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hyperref.sty | |
| 445 | +Package: hyperref 2026-01-29 v7.01p Hypertext links for LaTeX | |
| 446 | + | |
| 447 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty | |
| 448 | +Package: kvsetkeys 2022-10-05 v1.19 Key value parser (HO) | |
| 449 | +) | |
| 450 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty | |
| 451 | +Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO) | |
| 452 | +) | |
| 453 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pdfescape/pdfescape.sty | |
| 454 | +Package: pdfescape 2019/12/09 v1.15 Implements pdfTeX's escape features (HO) | |
| 455 | +) | |
| 456 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hycolor/hycolor.sty | |
| 457 | +Package: hycolor 2020-01-27 v1.10 Color options for hyperref/bookmark (HO) | |
| 458 | +) | |
| 459 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/nameref.sty | |
| 460 | +Package: nameref 2026-01-29 v2.58 Cross-referencing by name of section | |
| 461 | + | |
| 462 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/refcount/refcount.sty | |
| 463 | +Package: refcount 2019/12/15 v3.6 Data extraction from label references (HO) | |
| 464 | +) | |
| 465 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/gettitlestring/gettitlestring.s | |
| 466 | +ty | |
| 467 | +Package: gettitlestring 2019/12/15 v1.6 Cleanup title references (HO) | |
| 468 | + (/usr/local/texlive/2026/texmf-dist/tex/latex/kvoptions/kvoptions.sty | |
| 469 | +Package: kvoptions 2022-06-15 v3.15 Key value format for package options (HO) | |
| 470 | +)) | |
| 471 | +\c@section@level=\count328 | |
| 472 | +) | |
| 473 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/stringenc/stringenc.sty | |
| 474 | +Package: stringenc 2019/11/29 v1.12 Convert strings between diff. encodings (HO | |
| 475 | +) | |
| 476 | +) | |
| 477 | +\@linkdim=\dimen260 | |
| 478 | +\Hy@linkcounter=\count329 | |
| 479 | +\Hy@pagecounter=\count330 | |
| 480 | + | |
| 481 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def | |
| 482 | +File: pd1enc.def 2026-01-29 v7.01p Hyperref: PDFDocEncoding definition (HO) | |
| 483 | +Now handling font encoding PD1 ... | |
| 484 | +... no UTF-8 mapping file for font encoding PD1 | |
| 485 | +) | |
| 486 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/intcalc/intcalc.sty | |
| 487 | +Package: intcalc 2019/12/15 v1.3 Expandable calculations with integers (HO) | |
| 488 | +) | |
| 489 | +\Hy@SavedSpaceFactor=\count331 | |
| 490 | + | |
| 491 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def | |
| 492 | +File: puenc.def 2026-01-29 v7.01p Hyperref: PDF Unicode definition (HO) | |
| 493 | +Now handling font encoding PU ... | |
| 494 | +... no UTF-8 mapping file for font encoding PU | |
| 495 | +) | |
| 496 | +Package hyperref Info: Option `colorlinks' set `true' on input line 4072. | |
| 497 | +Package hyperref Info: Option `breaklinks' set `true' on input line 4072. | |
| 498 | +Package hyperref Info: Hyper figures OFF on input line 4201. | |
| 499 | +Package hyperref Info: Link nesting OFF on input line 4206. | |
| 500 | +Package hyperref Info: Hyper index ON on input line 4209. | |
| 501 | +Package hyperref Info: Plain pages OFF on input line 4216. | |
| 502 | +Package hyperref Info: Backreferencing OFF on input line 4221. | |
| 503 | +Package hyperref Info: Implicit mode ON; LaTeX internals redefined. | |
| 504 | +Package hyperref Info: Bookmarks ON on input line 4468. | |
| 505 | +\c@Hy@tempcnt=\count332 | |
| 506 | + | |
| 507 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/url/url.sty | |
| 508 | +\Urlmuskip=\muskip18 | |
| 509 | +Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc. | |
| 510 | +) | |
| 511 | +LaTeX Info: Redefining \url on input line 4807. | |
| 512 | +\XeTeXLinkMargin=\dimen261 | |
| 513 | + | |
| 514 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/bitset/bitset.sty | |
| 515 | +Package: bitset 2019/12/09 v1.3 Handle bit-vector datatype (HO) | |
| 516 | + | |
| 517 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty | |
| 518 | +Package: bigintcalc 2019/12/15 v1.5 Expandable calculations on big integers (HO | |
| 519 | +) | |
| 520 | +)) | |
| 521 | +\Fld@menulength=\count333 | |
| 522 | +\Field@Width=\dimen262 | |
| 523 | +\Fld@charsize=\dimen263 | |
| 524 | +Package hyperref Info: Hyper figures OFF on input line 6084. | |
| 525 | +Package hyperref Info: Link nesting OFF on input line 6089. | |
| 526 | +Package hyperref Info: Hyper index ON on input line 6092. | |
| 527 | +Package hyperref Info: backreferencing OFF on input line 6099. | |
| 528 | +Package hyperref Info: Link coloring ON on input line 6102. | |
| 529 | +Package hyperref Info: Link coloring with OCG OFF on input line 6109. | |
| 530 | +Package hyperref Info: PDF/A mode OFF on input line 6114. | |
| 531 | +\Hy@abspage=\count334 | |
| 532 | +\c@Item=\count335 | |
| 533 | +\c@Hfootnote=\count336 | |
| 534 | +) | |
| 535 | +Package hyperref Info: Driver (autodetected): hpdftex. | |
| 536 | + | |
| 537 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def | |
| 538 | +File: hpdftex.def 2026-01-29 v7.01p Hyperref driver for pdfTeX | |
| 539 | +\Fld@listcount=\count337 | |
| 540 | +\c@bookmark@seq@number=\count338 | |
| 541 | + | |
| 542 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty | |
| 543 | +Package: rerunfilecheck 2025-06-21 v1.11 Rerun checks for auxiliary files (HO) | |
| 544 | + | |
| 545 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty | |
| 546 | +Package: uniquecounter 2019/12/15 v1.4 Provide unlimited unique counter (HO) | |
| 547 | +) | |
| 548 | +Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2 | |
| 549 | +84. | |
| 550 | +) | |
| 551 | +\Hy@SectionHShift=\skip62 | |
| 552 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/natbib/natbib.sty | |
| 553 | +Package: natbib 2010/09/13 8.31b (PWD, AO) | |
| 554 | +\bibhang=\skip63 | |
| 555 | +\bibsep=\skip64 | |
| 556 | +LaTeX Info: Redefining \cite on input line 694. | |
| 557 | +\c@NAT@ctr=\count339 | |
| 558 | +) | |
| 559 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty | |
| 560 | +Package: fancyhdr 2025/02/07 v5.2 Extensive control of page headers and footers | |
| 561 | + | |
| 562 | +\f@nch@headwidth=\skip65 | |
| 563 | +\f@nch@offset@elh=\skip66 | |
| 564 | +\f@nch@offset@erh=\skip67 | |
| 565 | +\f@nch@offset@olh=\skip68 | |
| 566 | +\f@nch@offset@orh=\skip69 | |
| 567 | +\f@nch@offset@elf=\skip70 | |
| 568 | +\f@nch@offset@erf=\skip71 | |
| 569 | +\f@nch@offset@olf=\skip72 | |
| 570 | +\f@nch@offset@orf=\skip73 | |
| 571 | +\f@nch@height=\skip74 | |
| 572 | +\f@nch@footalignment=\skip75 | |
| 573 | +\f@nch@widthL=\skip76 | |
| 574 | +\f@nch@widthC=\skip77 | |
| 575 | +\f@nch@widthR=\skip78 | |
| 576 | +\@temptokenb=\toks38 | |
| 577 | +) | |
| 578 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/titlesec/titlesec.sty | |
| 579 | +Package: titlesec 2025/01/04 v2.17 Sectioning titles | |
| 580 | +\ttl@box=\box69 | |
| 581 | +\beforetitleunit=\skip79 | |
| 582 | +\aftertitleunit=\skip80 | |
| 583 | +\ttl@plus=\dimen264 | |
| 584 | +\ttl@minus=\dimen265 | |
| 585 | +\ttl@toksa=\toks39 | |
| 586 | +\titlewidth=\dimen266 | |
| 587 | +\titlewidthlast=\dimen267 | |
| 588 | +\titlewidthfirst=\dimen268 | |
| 589 | +) | |
| 590 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/appendix/appendix.sty | |
| 591 | +Package: appendix 2020/02/08 v1.2c extra appendix facilities | |
| 592 | +\c@@pps=\count340 | |
| 593 | +\c@@ppsavesec=\count341 | |
| 594 | +\c@@ppsaveapp=\count342 | |
| 595 | +) | |
| 596 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/enumitem/enumitem.sty | |
| 597 | +Package: enumitem 2025/02/06 v3.11 Customized lists | |
| 598 | +\labelindent=\skip81 | |
| 599 | +\enit@outerparindent=\dimen269 | |
| 600 | +\enit@toks=\toks40 | |
| 601 | +\enit@inbox=\box70 | |
| 602 | +\enit@count@id=\count343 | |
| 603 | +\enitdp@description=\count344 | |
| 604 | +) | |
| 605 | +LaTeX Font Info: Trying to load font information for T1+ptm on input line 12 | |
| 606 | +0. | |
| 607 | + | |
| 608 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd | |
| 609 | +File: t1ptm.fd 2001/06/04 font definitions for T1/ptm. | |
| 610 | +) | |
| 611 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def | |
| 612 | +File: l3backend-pdftex.def 2025-10-09 L3 backend support: PDF output (pdfTeX) | |
| 613 | +\l__color_backend_stack_int=\count345 | |
| 614 | +) | |
| 615 | +(./main.aux) | |
| 616 | +\openout1 = `main.aux'. | |
| 617 | + | |
| 618 | +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 120. | |
| 619 | +LaTeX Font Info: ... okay on input line 120. | |
| 620 | +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 120. | |
| 621 | +LaTeX Font Info: ... okay on input line 120. | |
| 622 | +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 120. | |
| 623 | +LaTeX Font Info: ... okay on input line 120. | |
| 624 | +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 120. | |
| 625 | +LaTeX Font Info: ... okay on input line 120. | |
| 626 | +LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 120. | |
| 627 | +LaTeX Font Info: ... okay on input line 120. | |
| 628 | +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 120. | |
| 629 | +LaTeX Font Info: ... okay on input line 120. | |
| 630 | +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 120. | |
| 631 | +LaTeX Font Info: ... okay on input line 120. | |
| 632 | +LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 120. | |
| 633 | +LaTeX Font Info: ... okay on input line 120. | |
| 634 | +LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 120. | |
| 635 | +LaTeX Font Info: ... okay on input line 120. | |
| 636 | + | |
| 637 | +*geometry* driver: auto-detecting | |
| 638 | +*geometry* detected driver: pdftex | |
| 639 | +*geometry* verbose mode - [ preamble ] result: | |
| 640 | +* driver: pdftex | |
| 641 | +* paper: letterpaper | |
| 642 | +* layout: <same size as paper> | |
| 643 | +* layoutoffset:(h,v)=(0.0pt,0.0pt) | |
| 644 | +* modes: | |
| 645 | +* h-part:(L,W,R)=(72.26999pt, 469.75502pt, 72.26999pt) | |
| 646 | +* v-part:(T,H,B)=(72.26999pt, 650.43001pt, 72.26999pt) | |
| 647 | +* \paperwidth=614.295pt | |
| 648 | +* \paperheight=794.96999pt | |
| 649 | +* \textwidth=469.75502pt | |
| 650 | +* \textheight=650.43001pt | |
| 651 | +* \oddsidemargin=0.0pt | |
| 652 | +* \evensidemargin=0.0pt | |
| 653 | +* \topmargin=-37.0pt | |
| 654 | +* \headheight=12.0pt | |
| 655 | +* \headsep=25.0pt | |
| 656 | +* \topskip=12.0pt | |
| 657 | +* \footskip=30.0pt | |
| 658 | +* \marginparwidth=44.0pt | |
| 659 | +* \marginparsep=10.0pt | |
| 660 | +* \columnsep=10.0pt | |
| 661 | +* \skip\footins=10.8pt plus 4.0pt minus 2.0pt | |
| 662 | +* \hoffset=0.0pt | |
| 663 | +* \voffset=0.0pt | |
| 664 | +* \mag=1000 | |
| 665 | +* \@twocolumnfalse | |
| 666 | +* \@twosidefalse | |
| 667 | +* \@mparswitchfalse | |
| 668 | +* \@reversemarginfalse | |
| 669 | +* (1in=72.27pt=25.4mm, 1cm=28.453pt) | |
| 670 | + | |
| 671 | +LaTeX Info: Redefining \microtypecontext on input line 120. | |
| 672 | +Package microtype Info: Applying patch `item' on input line 120. | |
| 673 | +Package microtype Info: Applying patch `toc' on input line 120. | |
| 674 | +Package microtype Info: Applying patch `eqnum' on input line 120. | |
| 675 | +Package microtype Info: Applying patch `footnote' on input line 120. | |
| 676 | +Package microtype Info: Applying patch `verbatim' on input line 120. | |
| 677 | +LaTeX Info: Redefining \microtypesetup on input line 120. | |
| 678 | +Package microtype Info: Generating PDF output. | |
| 679 | +Package microtype Info: Character protrusion enabled (level 2). | |
| 680 | +Package microtype Info: Using default protrusion set `alltext'. | |
| 681 | +Package microtype Info: Automatic font expansion enabled (level 2), | |
| 682 | +(microtype) stretch: 20, shrink: 20, step: 1, non-selected. | |
| 683 | +Package microtype Info: Using default expansion set `alltext-nott'. | |
| 684 | +Package microtype Info: Patching command `\showhyphens'. | |
| 685 | +Package microtype Info: No adjustment of tracking. | |
| 686 | +Package microtype Info: No adjustment of interword spacing. | |
| 687 | +Package microtype Info: No adjustment of character kerning. | |
| 688 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg | |
| 689 | +File: mt-ptm.cfg 2006/04/20 v1.7 microtype config. file: Times (RS) | |
| 690 | +) | |
| 691 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-d | |
| 692 | +ictionary-english.trsl | |
| 693 | +File: translations-basic-dictionary-english.trsl (english translation file `tra | |
| 694 | +nslations-basic-dictionary') | |
| 695 | +) | |
| 696 | +Package translations Info: loading dictionary `translations-basic-dictionary' f | |
| 697 | +or `english'. on input line 120. | |
| 698 | + | |
| 699 | +(/usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii | |
| 700 | +[Loading MPS to PDF converter (version 2006.09.02).] | |
| 701 | +\scratchcounter=\count346 | |
| 702 | +\scratchdimen=\dimen270 | |
| 703 | +\scratchbox=\box71 | |
| 704 | +\nofMPsegments=\count347 | |
| 705 | +\nofMParguments=\count348 | |
| 706 | +\everyMPshowfont=\toks41 | |
| 707 | +\MPscratchCnt=\count349 | |
| 708 | +\MPscratchDim=\dimen271 | |
| 709 | +\MPnumerator=\count350 | |
| 710 | +\makeMPintoPDFobject=\count351 | |
| 711 | +\everyMPtoPDFconversion=\toks42 | |
| 712 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty | |
| 713 | +Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf | |
| 714 | +Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4 | |
| 715 | +85. | |
| 716 | + | |
| 717 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg | |
| 718 | +File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv | |
| 719 | +e | |
| 720 | +)) | |
| 721 | +Package caption Info: Begin \AtBeginDocument code. | |
| 722 | +Package caption Info: float package is loaded. | |
| 723 | +Package caption Info: hyperref package is loaded. | |
| 724 | +Package caption Info: End \AtBeginDocument code. | |
| 725 | +Package hyperref Info: Link coloring ON on input line 120. | |
| 726 | + | |
| 727 | +(./main.out) (./main.out) | |
| 728 | +\@outlinefile=\write3 | |
| 729 | +\openout3 = `main.out'. | |
| 730 | + | |
| 731 | + (./tables/numbers.tex) (./sections/titlepage.tex | |
| 732 | +<uq_logo.jpg, id=156, 722.7pt x 361.35pt> | |
| 733 | +File: uq_logo.jpg Graphic file (type jpg) | |
| 734 | +<use uq_logo.jpg> | |
| 735 | +Package pdftex.def Info: uq_logo.jpg used on input line 9. | |
| 736 | +(pdftex.def) Requested size: 99.58464pt x 49.78915pt. | |
| 737 | + | |
| 738 | +[1 | |
| 739 | + | |
| 740 | +{/usr/local/texlive/2026/texmf-var/fonts/map/pdftex/updmap/pdftex.map}{/usr/loc | |
| 741 | +al/texlive/2026/texmf-dist/fonts/enc/dvips/base/8r.enc} <./uq_logo.jpg>] | |
| 742 | +[2]) (./sections/introduction.tex | |
| 743 | +LaTeX Font Info: Trying to load font information for OT1+ztmcm on input line | |
| 744 | + 10. | |
| 745 | + | |
| 746 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd | |
| 747 | +File: ot1ztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OT1/ztmcm. | |
| 748 | +) | |
| 749 | +Package microtype Info: Loading generic protrusion settings for font family | |
| 750 | +(microtype) `ztmcm' (encoding: OT1). | |
| 751 | +(microtype) For optimal results, create family-specific settings. | |
| 752 | +(microtype) See the microtype manual for details. | |
| 753 | +LaTeX Font Info: Trying to load font information for OML+ztmcm on input line | |
| 754 | + 10. | |
| 755 | + | |
| 756 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd | |
| 757 | +File: omlztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OML/ztmcm. | |
| 758 | +) | |
| 759 | +LaTeX Font Info: Trying to load font information for OMS+ztmcm on input line | |
| 760 | + 10. | |
| 761 | + | |
| 762 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd | |
| 763 | +File: omsztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OMS/ztmcm. | |
| 764 | +) | |
| 765 | +LaTeX Font Info: Trying to load font information for OMX+ztmcm on input line | |
| 766 | + 10. | |
| 767 | + | |
| 768 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd | |
| 769 | +File: omxztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OMX/ztmcm. | |
| 770 | +) | |
| 771 | +LaTeX Font Info: Trying to load font information for OT1+ptm on input line 1 | |
| 772 | +0. | |
| 773 | + | |
| 774 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd | |
| 775 | +File: ot1ptm.fd 2001/06/04 font definitions for OT1/ptm. | |
| 776 | +) | |
| 777 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <12> not available | |
| 778 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 10. | |
| 779 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <9> not available | |
| 780 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 10. | |
| 781 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <7> not available | |
| 782 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 10. | |
| 783 | + | |
| 784 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg | |
| 785 | +File: mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS) | |
| 786 | +) | |
| 787 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg | |
| 788 | +File: mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS) | |
| 789 | +) [3] | |
| 790 | +[4]) (./sections/literature.tex [5] [6]) (./sections/data.tex [7] [8] | |
| 791 | +(./tables/exclusions.tex) [9] (./tables/descriptives.tex) | |
| 792 | +Overfull \hbox (60.22191pt too wide) in paragraph at lines 1--36 | |
| 793 | + [][] | |
| 794 | + [] | |
| 795 | + | |
| 796 | +<figures/f7_indice.pdf, id=350, 443.14055pt x 225.88692pt> | |
| 797 | +File: figures/f7_indice.pdf Graphic file (type pdf) | |
| 798 | +<use figures/f7_indice.pdf> | |
| 799 | +Package pdftex.def Info: figures/f7_indice.pdf used on input line 40. | |
| 800 | +(pdftex.def) Requested size: 399.29463pt x 203.54105pt. | |
| 801 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <10.95> not available | |
| 802 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 41. | |
| 803 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <8> not available | |
| 804 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 41. | |
| 805 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <6> not available | |
| 806 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 41. | |
| 807 | +[10 <./figures/f7_indice.pdf>] (./tables/density_dist.tex)) | |
| 808 | +(./sections/methodology.tex | |
| 809 | + | |
| 810 | +LaTeX Warning: Reference `sec:adjust' on page 11 undefined on input line 13. | |
| 811 | + | |
| 812 | +[11] [12] [13] | |
| 813 | +Overfull \hbox (95.51924pt too wide) in paragraph at lines 61--82 | |
| 814 | + [][] | |
| 815 | + [] | |
| 816 | + | |
| 817 | +[14]) (./sections/results.tex | |
| 818 | +<figures/f1_mdape_vs_k.pdf, id=468, 550.14034pt x 251.5558pt> | |
| 819 | +File: figures/f1_mdape_vs_k.pdf Graphic file (type pdf) | |
| 820 | +<use figures/f1_mdape_vs_k.pdf> | |
| 821 | +Package pdftex.def Info: figures/f1_mdape_vs_k.pdf used on input line 15. | |
| 822 | +(pdftex.def) Requested size: 469.75502pt x 214.80223pt. | |
| 823 | + [15] | |
| 824 | +<figures/f2_surface_d_T.pdf, id=477, 502.01927pt x 225.88692pt> | |
| 825 | +File: figures/f2_surface_d_T.pdf Graphic file (type pdf) | |
| 826 | +<use figures/f2_surface_d_T.pdf> | |
| 827 | +Package pdftex.def Info: figures/f2_surface_d_T.pdf used on input line 22. | |
| 828 | +(pdftex.def) Requested size: 422.77664pt x 190.23659pt. | |
| 829 | + (./tables/surface_geo_hedo.tex) [16 <./figures/f1_mdape_vs_k.pdf> <./figures/f | |
| 830 | +2_surface_d_T.pdf>] (./tables/h2.tex) | |
| 831 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <10> not available | |
| 832 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 47. | |
| 833 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <7.4> not available | |
| 834 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 47. | |
| 835 | + | |
| 836 | +[17] (./tables/adjustments.tex) | |
| 837 | +Overfull \hbox (20.0581pt too wide) in paragraph at lines 1--61 | |
| 838 | + [][] | |
| 839 | + [] | |
| 840 | + | |
| 841 | +<figures/f3_ajustements.pdf, id=638, 445.64993pt x 225.88692pt> | |
| 842 | +File: figures/f3_ajustements.pdf Graphic file (type pdf) | |
| 843 | +<use figures/f3_ajustements.pdf> | |
| 844 | +Package pdftex.def Info: figures/f3_ajustements.pdf used on input line 68. | |
| 845 | +(pdftex.def) Requested size: 422.77664pt x 214.3018pt. | |
| 846 | +[18] (./tables/h3.tex) [19 <./figures/f3_ajustements.pdf>] (./tables/grid.tex) | |
| 847 | +[20] (./tables/selection.tex) [21] (./tables/weighting.tex) | |
| 848 | +Overfull \hbox (64.52856pt too wide) in paragraph at lines 1--121 | |
| 849 | + [][] | |
| 850 | + [] | |
| 851 | + | |
| 852 | +(./tables/h4.tex) | |
| 853 | +Overfull \hbox (200.30655pt too wide) in paragraph at lines 1--128 | |
| 854 | + [][] | |
| 855 | + [] | |
| 856 | + | |
| 857 | +[22] (./tables/marginal.tex) [23] (./tables/competition.tex) | |
| 858 | +Overfull \hbox (222.59087pt too wide) in paragraph at lines 1--151 | |
| 859 | + [][] | |
| 860 | + [] | |
| 861 | + | |
| 862 | +<figures/f4_competition.pdf, id=766, 754.11583pt x 340.59245pt> | |
| 863 | +File: figures/f4_competition.pdf Graphic file (type pdf) | |
| 864 | +<use figures/f4_competition.pdf> | |
| 865 | +Package pdftex.def Info: figures/f4_competition.pdf used on input line 158. | |
| 866 | +(pdftex.def) Requested size: 446.26582pt x 201.55585pt. | |
| 867 | +[24] (./tables/pairs.tex) [25 <./figures/f4_competition.pdf>] | |
| 868 | +(./tables/density.tex) [26] | |
| 869 | +<figures/f5_densite.pdf, id=887, 410.11719pt x 251.5558pt> | |
| 870 | +File: figures/f5_densite.pdf Graphic file (type pdf) | |
| 871 | +<use figures/f5_densite.pdf> | |
| 872 | +Package pdftex.def Info: figures/f5_densite.pdf used on input line 192. | |
| 873 | +(pdftex.def) Requested size: 399.29463pt x 244.93004pt. | |
| 874 | + (./tables/h5.tex) [27 <./figures/f5_densite.pdf>] | |
| 875 | +(./tables/intervals.tex) | |
| 876 | +Overfull \hbox (260.2727pt too wide) in paragraph at lines 1--219 | |
| 877 | + [][] | |
| 878 | + [] | |
| 879 | + | |
| 880 | +[28] (./tables/cond_rules.tex) | |
| 881 | +Overfull \hbox (163.44707pt too wide) in paragraph at lines 1--232 | |
| 882 | + [][] | |
| 883 | + [] | |
| 884 | + | |
| 885 | +) (./sections/robustness.tex (./tables/stability.tex) [29]) | |
| 886 | +(./sections/conclusion.tex [30]) [31] [32] (./main.bbl [33] [34]) | |
| 887 | +(./appendix/appendix.tex (./tables/surface_cov.tex) | |
| 888 | +(./tables/surface_geo_none.tex) (./tables/surface_sim_hedo.tex) [35] | |
| 889 | +(./tables/e1_best.tex) | |
| 890 | +Overfull \hbox (37.67392pt too wide) in paragraph at lines 1--33 | |
| 891 | + [][] | |
| 892 | + [] | |
| 893 | + | |
| 894 | +(./tables/regions.tex) [36] (./tables/strata_other.tex) [37]) [38] (./main.aux) | |
| 895 | + *********** | |
| 896 | +LaTeX2e <2025-11-01> | |
| 897 | +L3 programming layer <2026-01-19> | |
| 898 | + *********** | |
| 899 | + | |
| 900 | +LaTeX Warning: There were undefined references. | |
| 901 | + | |
| 902 | +Package rerunfilecheck Info: File `main.out' has not changed. | |
| 903 | +(rerunfilecheck) Checksum: BF1DD9FE3C2A7A6E19AC40B70B222A94;7416. | |
| 904 | + ) | |
| 905 | +Here is how much of TeX's memory you used: | |
| 906 | + 22556 strings out of 467525 | |
| 907 | + 419168 string characters out of 5418982 | |
| 908 | + 869339 words of memory out of 5000000 | |
| 909 | + 50445 multiletter control sequences out of 15000+600000 | |
| 910 | + 740916 words of font info for 465 fonts, out of 8000000 for 9000 | |
| 911 | + 1141 hyphenation exceptions out of 8191 | |
| 912 | + 75i,12n,84p,2312b,1226s stack positions out of 10000i,1000n,20000p,200000b,200000s | |
| 913 | +</usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb | |
| 914 | +></usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb> | |
| 915 | +</usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></ | |
| 916 | +usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb></u | |
| 917 | +sr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/symbols/msbm10.pfb | |
| 918 | +></usr/local/texlive/2026/texmf-dist/fonts/type1/public/rsfs/rsfs10.pfb></usr/l | |
| 919 | +ocal/texlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texliv | |
| 920 | +e/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026/texm | |
| 921 | +f-dist/fonts/type1/urw/times/utmb8a.pfb></usr/local/texlive/2026/texmf-dist/fon | |
| 922 | +ts/type1/urw/times/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/u | |
| 923 | +rw/times/utmri8a.pfb> | |
| 924 | +Output written on main.pdf (38 pages, 440556 bytes). | |
| 925 | +PDF statistics: | |
| 926 | + 1127 PDF objects out of 1200 (max. 8388607) | |
| 927 | + 648 compressed objects within 7 object streams | |
| 928 | + 147 named destinations out of 1000 (max. 500000) | |
| 929 | + 104788 words of extra memory for PDF output out of 106986 (max. 10000000) | |
| 930 | + | |
added
paper/main.out
+38 −0
@@ -0,0 +1,38 @@ | ||
| 1 | +\BOOKMARK [1][-]{section.1}{\376\377\000I\000n\000t\000r\000o\000d\000u\000c\000t\000i\000o\000n}{}% 1 | |
| 2 | +\BOOKMARK [1][-]{section.2}{\376\377\000R\000e\000l\000a\000t\000e\000d\000\040\000l\000i\000t\000e\000r\000a\000t\000u\000r\000e}{}% 2 | |
| 3 | +\BOOKMARK [2][-]{subsection.2.1}{\376\377\000T\000h\000e\000\040\000s\000a\000l\000e\000s\000\040\000c\000o\000m\000p\000a\000r\000i\000s\000o\000n\000\040\000a\000p\000p\000r\000o\000a\000c\000h\000\040\000a\000n\000d\000\040\000i\000t\000s\000\040\000c\000o\000n\000v\000e\000n\000t\000i\000o\000n\000s}{section.2}% 3 | |
| 4 | +\BOOKMARK [2][-]{subsection.2.2}{\376\377\000E\000s\000t\000i\000m\000a\000t\000i\000n\000g\000\040\000a\000d\000j\000u\000s\000t\000m\000e\000n\000t\000s\000:\000\040\000g\000r\000i\000d\000s\000,\000\040\000h\000e\000d\000o\000n\000i\000c\000s\000\040\000a\000n\000d\000\040\000l\000o\000c\000a\000l\000\040\000m\000o\000d\000e\000l\000s}{section.2}% 4 | |
| 5 | +\BOOKMARK [2][-]{subsection.2.3}{\376\377\000A\000u\000t\000o\000m\000a\000t\000e\000d\000\040\000v\000a\000l\000u\000a\000t\000i\000o\000n\000\040\000a\000n\000d\000\040\000m\000a\000c\000h\000i\000n\000e\000\040\000l\000e\000a\000r\000n\000i\000n\000g}{section.2}% 5 | |
| 6 | +\BOOKMARK [2][-]{subsection.2.4}{\376\377\000A\000p\000p\000r\000a\000i\000s\000e\000r\000\040\000b\000e\000h\000a\000v\000i\000o\000u\000r\000\040\000a\000n\000d\000\040\000t\000h\000e\000\040\000l\000i\000m\000i\000t\000s\000\040\000o\000f\000\040\000s\000i\000m\000u\000l\000a\000t\000i\000o\000n}{section.2}% 6 | |
| 7 | +\BOOKMARK [1][-]{section.3}{\376\377\000D\000a\000t\000a}{}% 7 | |
| 8 | +\BOOKMARK [2][-]{subsection.3.1}{\376\377\000T\000h\000e\000\040\000V\000r\000a\000i\000\040\000P\000r\000i\000x\000\040\000t\000r\000a\000n\000s\000a\000c\000t\000i\000o\000n\000\040\000d\000a\000t\000a\000b\000a\000s\000e}{section.3}% 8 | |
| 9 | +\BOOKMARK [2][-]{subsection.3.2}{\376\377\000S\000a\000m\000p\000l\000e\000\040\000c\000o\000n\000s\000t\000r\000u\000c\000t\000i\000o\000n}{section.3}% 9 | |
| 10 | +\BOOKMARK [2][-]{subsection.3.3}{\376\377\000C\000h\000r\000o\000n\000o\000l\000o\000g\000i\000c\000a\000l\000\040\000p\000a\000r\000t\000i\000t\000i\000o\000n\000\040\000a\000n\000d\000\040\000d\000e\000s\000c\000r\000i\000p\000t\000i\000v\000e\000\040\000s\000t\000a\000t\000i\000s\000t\000i\000c\000s}{section.3}% 10 | |
| 11 | +\BOOKMARK [2][-]{subsection.3.4}{\376\377\000M\000a\000r\000k\000e\000t\000\040\000d\000e\000n\000s\000i\000t\000y}{section.3}% 11 | |
| 12 | +\BOOKMARK [1][-]{section.4}{\376\377\000M\000e\000t\000h\000o\000d\000o\000l\000o\000g\000y}{}% 12 | |
| 13 | +\BOOKMARK [2][-]{subsection.4.1}{\376\377\000P\000o\000i\000n\000t\000-\000i\000n\000-\000t\000i\000m\000e\000\040\000b\000a\000c\000k\000t\000e\000s\000t\000i\000n\000g}{section.4}% 13 | |
| 14 | +\BOOKMARK [2][-]{subsection.4.2}{\376\377\000P\000o\000i\000n\000t\000-\000i\000n\000-\000t\000i\000m\000e\000\040\000p\000r\000i\000c\000e\000\040\000i\000n\000d\000e\000x}{section.4}% 14 | |
| 15 | +\BOOKMARK [2][-]{subsection.4.3}{\376\377\000C\000o\000n\000f\000i\000g\000u\000r\000a\000t\000i\000o\000n\000s\000\040\000o\000f\000\040\000t\000h\000e\000\040\000s\000a\000l\000e\000s\000\040\000c\000o\000m\000p\000a\000r\000i\000s\000o\000n\000\040\000a\000p\000p\000r\000o\000a\000c\000h}{section.4}% 15 | |
| 16 | +\BOOKMARK [2][-]{subsection.4.4}{\376\377\000B\000e\000n\000c\000h\000m\000a\000r\000k\000\040\000m\000o\000d\000e\000l\000s}{section.4}% 16 | |
| 17 | +\BOOKMARK [2][-]{subsection.4.5}{\376\377\000A\000c\000c\000u\000r\000a\000c\000y\000\040\000m\000e\000a\000s\000u\000r\000e\000s\000\040\000a\000n\000d\000\040\000i\000n\000f\000e\000r\000e\000n\000c\000e}{section.4}% 17 | |
| 18 | +\BOOKMARK [2][-]{subsection.4.6}{\376\377\000H\000y\000p\000o\000t\000h\000e\000s\000e\000s}{section.4}% 18 | |
| 19 | +\BOOKMARK [1][-]{section.5}{\376\377\000R\000e\000s\000u\000l\000t\000s}{}% 19 | |
| 20 | +\BOOKMARK [2][-]{subsection.5.1}{\376\377\000H\000o\000w\000\040\000m\000a\000n\000y\000\040\000c\000o\000m\000p\000a\000r\000a\000b\000l\000e\000s\000,\000\040\000h\000o\000w\000\040\000f\000a\000r\000,\000\040\000h\000o\000w\000\040\000f\000a\000r\000\040\000b\000a\000c\000k\000?}{section.5}% 20 | |
| 21 | +\BOOKMARK [2][-]{subsection.5.2}{\376\377\000W\000h\000i\000c\000h\000\040\000a\000d\000j\000u\000s\000t\000m\000e\000n\000t\000s\000\040\000a\000r\000e\000\040\000w\000a\000r\000r\000a\000n\000t\000e\000d\000?}{section.5}% 21 | |
| 22 | +\BOOKMARK [2][-]{subsection.5.3}{\376\377\000H\000o\000w\000\040\000s\000h\000o\000u\000l\000d\000\040\000c\000o\000m\000p\000a\000r\000a\000b\000l\000e\000s\000\040\000b\000e\000\040\000s\000e\000l\000e\000c\000t\000e\000d\000?\000\040\000\050\000H\0001\000\051}{section.5}% 22 | |
| 23 | +\BOOKMARK [2][-]{subsection.5.4}{\376\377\000W\000e\000i\000g\000h\000t\000i\000n\000g\000\040\000a\000n\000d\000\040\000r\000e\000c\000o\000n\000c\000i\000l\000i\000a\000t\000i\000o\000n\000\040\000\050\000H\0004\000\051}{section.5}% 23 | |
| 24 | +\BOOKMARK [2][-]{subsection.5.5}{\376\377\000T\000h\000e\000\040\000h\000o\000r\000s\000e\000\040\000r\000a\000c\000e}{section.5}% 24 | |
| 25 | +\BOOKMARK [2][-]{subsection.5.6}{\376\377\000W\000h\000e\000r\000e\000\040\000d\000o\000e\000s\000\040\000t\000h\000e\000\040\000m\000e\000t\000h\000o\000d\000\040\000h\000o\000l\000d\000,\000\040\000a\000n\000d\000\040\000w\000h\000e\000r\000e\000\040\000d\000o\000e\000s\000\040\000i\000t\000\040\000f\000a\000i\000l\000?}{section.5}% 25 | |
| 26 | +\BOOKMARK [2][-]{subsection.5.7}{\376\377\000V\000a\000l\000u\000e\000\040\000i\000n\000t\000e\000r\000v\000a\000l\000s}{section.5}% 26 | |
| 27 | +\BOOKMARK [2][-]{subsection.5.8}{\376\377\000T\000o\000w\000a\000r\000d\000s\000\040\000a\000n\000\040\000e\000m\000p\000i\000r\000i\000c\000a\000l\000\040\000c\000o\000m\000p\000a\000r\000a\000b\000l\000e\000-\000r\000e\000c\000o\000m\000m\000e\000n\000d\000a\000t\000i\000o\000n\000\040\000s\000y\000s\000t\000e\000m}{section.5}% 27 | |
| 28 | +\BOOKMARK [1][-]{section.6}{\376\377\000R\000o\000b\000u\000s\000t\000n\000e\000s\000s}{}% 28 | |
| 29 | +\BOOKMARK [2][-]{subsection.6.1}{\376\377\000S\000t\000a\000b\000i\000l\000i\000t\000y\000\040\000o\000v\000e\000r\000\040\000t\000i\000m\000e}{section.6}% 29 | |
| 30 | +\BOOKMARK [2][-]{subsection.6.2}{\376\377\000C\000o\000v\000e\000r\000a\000g\000e\000\040\000a\000n\000d\000\040\000t\000h\000e\000\040\000c\000o\000m\000m\000o\000n\000-\000s\000u\000p\000p\000o\000r\000t\000\040\000c\000o\000m\000p\000a\000r\000i\000s\000o\000n}{section.6}% 30 | |
| 31 | +\BOOKMARK [2][-]{subsection.6.3}{\376\377\000E\000x\000c\000l\000u\000s\000i\000o\000n\000\040\000r\000u\000l\000e\000s}{section.6}% 31 | |
| 32 | +\BOOKMARK [2][-]{subsection.6.4}{\376\377\000A\000l\000t\000e\000r\000n\000a\000t\000i\000v\000e\000\040\000a\000c\000c\000u\000r\000a\000c\000y\000\040\000m\000e\000a\000s\000u\000r\000e\000s}{section.6}% 32 | |
| 33 | +\BOOKMARK [2][-]{subsection.6.5}{\376\377\000I\000n\000d\000e\000x\000\040\000c\000o\000n\000s\000t\000r\000u\000c\000t\000i\000o\000n}{section.6}% 33 | |
| 34 | +\BOOKMARK [2][-]{subsection.6.6}{\376\377\000B\000e\000n\000c\000h\000m\000a\000r\000k\000\040\000s\000p\000e\000c\000i\000f\000i\000c\000a\000t\000i\000o\000n}{section.6}% 34 | |
| 35 | +\BOOKMARK [1][-]{section.7}{\376\377\000C\000o\000n\000c\000l\000u\000s\000i\000o\000n}{}% 35 | |
| 36 | +\BOOKMARK [1][-]{section*.27}{\376\377\000A\000p\000p\000e\000n\000d\000i\000c\000e\000s}{}% 36 | |
| 37 | +\BOOKMARK [1][-]{appendix.1.A}{\376\377\000A\000d\000d\000i\000t\000i\000o\000n\000a\000l\000\040\000t\000a\000b\000l\000e\000s}{}% 37 | |
| 38 | +\BOOKMARK [1][-]{appendix.1.B}{\376\377\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000\040\000d\000e\000t\000a\000i\000l\000s}{}% 38 | |
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.synctex.gz
+0 −0
Binary file not shown.
added
paper/main.tex
+142 −0
@@ -0,0 +1,142 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% UQO Working Paper Template — Economics / Finance | |
| 3 | +% ============================================================================ | |
| 4 | +\documentclass[12pt,letterpaper]{article} | |
| 5 | + | |
| 6 | +% --- Encoding & Language --- | |
| 7 | +\usepackage[utf8]{inputenc} | |
| 8 | +\usepackage[T1]{fontenc} | |
| 9 | +\usepackage[english]{babel} | |
| 10 | + | |
| 11 | +% --- Page Layout --- | |
| 12 | +\usepackage[ | |
| 13 | + letterpaper, | |
| 14 | + top=1in, | |
| 15 | + bottom=1in, | |
| 16 | + left=1in, | |
| 17 | + right=1in | |
| 18 | +]{geometry} | |
| 19 | + | |
| 20 | +% --- Typography --- | |
| 21 | +\usepackage{mathptmx} % Times-like font (text + math) | |
| 22 | +\usepackage{microtype} % Improved typographic refinements | |
| 23 | +\usepackage{setspace} | |
| 24 | +\onehalfspacing % 1.5 line spacing | |
| 25 | + | |
| 26 | +% --- Math --- | |
| 27 | +\usepackage{amsmath,amssymb,amsthm} | |
| 28 | + | |
| 29 | +% --- Tables --- | |
| 30 | +\usepackage{booktabs} | |
| 31 | +\usepackage{tabularx} | |
| 32 | +\usepackage{multirow} | |
| 33 | +\usepackage{threeparttable} | |
| 34 | +\usepackage{siunitx} | |
| 35 | +\sisetup{ | |
| 36 | + input-symbols = {()}, | |
| 37 | + table-format = -1.3, | |
| 38 | + parse-numbers = false | |
| 39 | +} | |
| 40 | + | |
| 41 | +% --- Figures --- | |
| 42 | +\usepackage{graphicx} | |
| 43 | +\usepackage{subcaption} | |
| 44 | +\graphicspath{{figures/}} | |
| 45 | +\usepackage{float} | |
| 46 | + | |
| 47 | +% --- Colors & Links --- | |
| 48 | +\usepackage[dvipsnames]{xcolor} | |
| 49 | +\usepackage[ | |
| 50 | + colorlinks = true, | |
| 51 | + linkcolor = NavyBlue, | |
| 52 | + citecolor = NavyBlue, | |
| 53 | + urlcolor = NavyBlue, | |
| 54 | + breaklinks = true | |
| 55 | +]{hyperref} | |
| 56 | + | |
| 57 | +% --- Bibliography --- | |
| 58 | +\usepackage[authoryear,round,semicolon]{natbib} | |
| 59 | +\bibliographystyle{aer} | |
| 60 | + | |
| 61 | +% --- Captions --- | |
| 62 | +\usepackage[ | |
| 63 | + font = small, | |
| 64 | + labelfont = bf, | |
| 65 | + labelsep = period, | |
| 66 | + justification = justified, | |
| 67 | + singlelinecheck = false | |
| 68 | +]{caption} | |
| 69 | + | |
| 70 | +% --- Headers & Footers --- | |
| 71 | +\usepackage{fancyhdr} | |
| 72 | +\pagestyle{fancy} | |
| 73 | +\fancyhf{} | |
| 74 | +\renewcommand{\headrulewidth}{0pt} | |
| 75 | +\fancyfoot[C]{\thepage} | |
| 76 | + | |
| 77 | +% --- Section Formatting --- | |
| 78 | +\usepackage{titlesec} | |
| 79 | +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{} | |
| 80 | +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{} | |
| 81 | +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{} | |
| 82 | + | |
| 83 | +% --- Appendix Support --- | |
| 84 | +\usepackage[toc,page]{appendix} | |
| 85 | + | |
| 86 | +% --- Misc --- | |
| 87 | +\usepackage{enumitem} | |
| 88 | +\setlist{nosep,leftmargin=*} | |
| 89 | + | |
| 90 | +% ============================================================================ | |
| 91 | +% METADATA — Edit these fields for each working paper | |
| 92 | +% ============================================================================ | |
| 93 | +\newcommand{\WPnumber}{11} | |
| 94 | +\newcommand{\WPtitle}{Does the Sales Comparison Approach Actually Work? A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation} | |
| 95 | +\newcommand{\WPsubtitle}{Evidence from 394,000 Single-Family Transactions in Quebec, 2021--2026} | |
| 96 | +\newcommand{\WPdate}{October 2026} | |
| 97 | +\newcommand{\WPversion}{1.0} | |
| 98 | +\newcommand{\WPabstract}{% | |
| 99 | + The sales comparison approach is the dominant method of residential valuation, yet its operating rules---how many comparables to use, how far and how far back to search, which adjustments to apply, how to weight and reconcile---rest largely on professional convention. We subject these rules to a large-scale, point-in-time backtest on \nSalesClean{} arm's-length single-family transactions in Quebec (2021--2026) matched to the municipal assessment roll. For each of \nTargetsAll{} target sales in 2024--2026, the price is masked and \nConfigs{} configurations of the approach---together with hedonic, gradient-boosting, assessment-ratio and hybrid benchmarks---are applied using only information available on the valuation date; configurations are selected on 2024 and evaluated out of sample on 2025--2026. Adjustments dominate: the median absolute error of the five nearest sales falls from \mdapeMtwo{} with a time adjustment only to \mdapeMseven{} with a hedonic adjustment grid, and a stylized conventional grid captures most of that gain. Selecting physically similar rather than merely nearby comparables delivers a comparable gain without any adjustment. The optimal number of comparables is well above the conventional three and increases with adjustment quality and market density; narrow search radii and short time windows are dominated. Weighting and reconciliation rules matter little. The best transparent configuration chosen in 2024 reaches \mdapeMzero{} out of sample, against \mdapeMelevenB{} for a quarterly-retrained gradient-boosting model that uses the assessed value and \mdapeMtwelve{} for a hybrid; the cost of transparency is small and vanishes in homogeneous submarkets. Accuracy is governed primarily by submarket liquidity, and density-conditional rules improve on any single rule. The results turn a convention-based method into a set of empirically validated, reproducible rules. | |
| 100 | +} | |
| 101 | +\newcommand{\WPkeywords}{Sales comparison approach, comparable selection, adjustment grid, automated valuation models, hedonic pricing, backtesting, residential real estate, Quebec} | |
| 102 | +\newcommand{\WPjel}{R31, R32, C52, C53, G12} | |
| 103 | + | |
| 104 | +% --- Author --- | |
| 105 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | |
| 106 | +\newcommand{\WPaffiliation}{% | |
| 107 | + Département des sciences administratives\\ | |
| 108 | + Université du Québec en Outaouais% | |
| 109 | +} | |
| 110 | +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca} | |
| 111 | +\newcommand{\WPaddress}{% | |
| 112 | + Gatineau -- Pavillon Alexandre-Taché\\ | |
| 113 | + 283, boulevard Alexandre-Taché\\ | |
| 114 | + Gatineau, Québec, Canada J9A 1L8% | |
| 115 | +} | |
| 116 | + | |
| 117 | +% ============================================================================ | |
| 118 | +% DOCUMENT | |
| 119 | +% ============================================================================ | |
| 120 | +\begin{document} | |
| 121 | + | |
| 122 | +% --- Title Page --- | |
| 123 | +\input{tables/numbers} | |
| 124 | +\input{sections/titlepage} | |
| 125 | + | |
| 126 | +% --- Main Body --- | |
| 127 | +\input{sections/introduction} | |
| 128 | +\input{sections/literature} | |
| 129 | +\input{sections/data} | |
| 130 | +\input{sections/methodology} | |
| 131 | +\input{sections/results} | |
| 132 | +\input{sections/robustness} | |
| 133 | +\input{sections/conclusion} | |
| 134 | + | |
| 135 | +% --- References --- | |
| 136 | +\newpage | |
| 137 | +\bibliography{references} | |
| 138 | + | |
| 139 | +% --- Appendix --- | |
| 140 | +\input{appendix/appendix} | |
| 141 | + | |
| 142 | +\end{document} | |
added
paper/make_paper_tables.py
+207 −0
@@ -0,0 +1,207 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""Génère tables/*.tex, tables/numbers.tex (macros) et copie les figures depuis out/<version>/ pour le working paper.""" | |
| 3 | +import json, os, shutil, sys | |
| 4 | +from pathlib import Path | |
| 5 | +import numpy as np, pandas as pd | |
| 6 | +ROOT = Path(__file__).resolve().parents[1]; V = os.environ.get("CV_VERSION", "2026-10-10") | |
| 7 | +OUT = ROOT / "out" / V; TAB = OUT / "tables"; P = Path(__file__).resolve().parent | |
| 8 | +(P / "tables").mkdir(exist_ok=True); (P / "figures").mkdir(exist_ok=True) | |
| 9 | +R = json.load(open(OUT / "results.json")) | |
| 10 | +comp = R["competition"]; compv = R["competition_val"] | |
| 11 | +def key(prefix): return next(k for k in comp if k.startswith(prefix + " ")) | |
| 12 | +M = {k: key(k) for k in ["M0", "M1", "M2", "M3", "M4", "M5", "M6", "M7", "M8a", "M8b", "M9", "M10", "M11a", "M11b", "M12", "M13"]} | |
| 13 | +def f1(x): return f"{x:.1f}" | |
| 14 | +def f2(x): return f"{x:.2f}" | |
| 15 | +def n_(x): return f"{int(round(x)):,}" | |
| 16 | +def pc(x): return f"{x:.1f}\\%" | |
| 17 | +def esc(s): | |
| 18 | + s = str(s) | |
| 19 | + if "$" in s or "\\" in s: return s # déjà en LaTeX | |
| 20 | + return s.replace("%", "\\%").replace("&", "\\&").replace("_", "\\_").replace("≥", "$\\ge$").replace("≤", "$\\le$").replace("<", "$<$").replace(">", "$>$") | |
| 21 | +LAB = {"raw": "none", "none": "time only", "conv": "conventional grid", "hedo": "hedonic grid", "hedo_reg": "regional grid", "local": "local grid", "gbm": "GBM adjustments", | |
| 22 | + "role": "assessment ratio", "geo": "nearest", "sim": "most similar", "mix": "compromise", "adj": "least adjusted", "equal": "equal", "dist": "1/distance", | |
| 23 | + "rec": "recency", "all": "distance $\\times$ recency $\\times$ similarity", "median": "median", "mean": "mean"} | |
| 24 | +DENS = ["< 5", "5-19", "20-49", "50-199", "≥ 200"]; DENS_EN = {"< 5": "$<5$", "5-19": "5--19", "20-49": "20--49", "50-199": "50--199", "≥ 200": "$\\ge 200$"} | |
| 25 | +def tabular(df, cols, align=None, fmt=None, bold_min_cols=None): | |
| 26 | + fmt = fmt or {}; align = align or ("l" + "r" * (len(cols) - 1)) | |
| 27 | + lines = ["\\begin{tabular}{" + align + "}", "\\toprule", " & ".join(esc(c) for c in cols) + " \\\\", "\\midrule"] | |
| 28 | + for _, r in df.iterrows(): | |
| 29 | + cells = [] | |
| 30 | + rowmin = None | |
| 31 | + if bold_min_cols: rowmin = min(float(r[c]) for c in bold_min_cols if pd.notna(r[c])) | |
| 32 | + for c in cols: | |
| 33 | + v = r[c] | |
| 34 | + if c in fmt: t = fmt[c](v) | |
| 35 | + elif isinstance(v, (float, np.floating)): t = f1(v) if pd.notna(v) else "--" | |
| 36 | + else: t = esc(v) | |
| 37 | + if bold_min_cols and c in bold_min_cols and pd.notna(v) and float(v) == rowmin: t = "\\textbf{" + t + "}" | |
| 38 | + cells.append(t) | |
| 39 | + lines.append(" & ".join(cells) + " \\\\") | |
| 40 | + lines += ["\\bottomrule", "\\end{tabular}"] | |
| 41 | + return "\n".join(lines) + "\n" | |
| 42 | +def write(name, txt): (P / "tables" / f"{name}.tex").write_text(txt) | |
| 43 | + | |
| 44 | +# ------------------------------------------------------------------ tables | |
| 45 | +D = R["dataset"] | |
| 46 | +ex = pd.DataFrame(D["exclusions"], columns=["Rule", "Removed"]); ex["Rule"] = ex["Rule"].str.replace(r"^R\d+b? ", "", regex=True) | |
| 47 | +EN = ["Not matched to the assessment roll", "Duplicates (same unit, same date)", "Roll unit not a single dwelling or more than one dwelling", "Coordinates missing or outside Quebec", | |
| 48 | + "Living area missing or outside [40, 600] m$^2$", "Lot area missing or outside [80, 20,000] m$^2$", "Year built missing, before 1850 or after the sale (land sale)", | |
| 49 | + "Sale in the year of construction (lot or unfinished building)", "Price outside [\\$50,000, \\$3,000,000]", "Assessed value missing or price/assessment ratio outside [0.33, 3]", | |
| 50 | + "Resale within 180 days with price change above 40\\% (flip)", "Physical link or construction type missing", "Municipality absent from the MAMH directory"] | |
| 51 | +ex["Rule"] = EN[:len(ex)]; ex["Removed"] = ex["Removed"].map(n_) | |
| 52 | +rows = [["Single-family transactions, raw", n_(D["brut"])]] + ex.values.tolist() + [["Final sample", n_(D["propre"])]] | |
| 53 | +write("exclusions", tabular(pd.DataFrame(rows, columns=["Step", "n"]), ["Step", "n"], "lr")) | |
| 54 | +ds = R.get("descriptives") or {p: dict(n=0, prix_med=0, prix_moy=0, aire_med=0, terrain_med=0, age_med=0, detache=0, ratio_role_med=0) for p in ["train", "val", "test"]} | |
| 55 | +dd = pd.DataFrame({"Statistic": ["Sales", "Median price (\\$)", "Mean price (\\$)", "Median living area (m$^2$)", "Median lot area (m$^2$)", "Median age (years)", "Detached (\\%)", "Median price/assessment ratio", "Median sales within 2 km, 12 months", "Median candidate pool (25 km, 24 months)"], | |
| 56 | + **{{"train": "Development 2021--2023", "val": "Validation 2024", "test": "Test 2025--2026"}[p]: [n_(ds[p]["n"]), n_(ds[p]["prix_med"]), n_(ds[p]["prix_moy"]), f1(ds[p]["aire_med"]), n_(ds[p]["terrain_med"]), f1(ds[p]["age_med"]), f1(ds[p]["detache"]), f2(ds[p]["ratio_role_med"]), (n_(ds[p]["dens_med"]) if "dens_med" in ds[p] else "--"), (n_(ds[p]["n_cand_med"]) if "n_cand_med" in ds[p] else "--")] for p in ["train", "val", "test"]}}) | |
| 57 | +write("descriptives", tabular(dd, list(dd.columns), "lrrr")) | |
| 58 | +dist = R["densite_distribution"]; tot = sum(dist.values()) | |
| 59 | +dd = pd.DataFrame({"Density (sales within 2 km, 12 months)": [DENS_EN[d] for d in DENS], "Test targets": [n_(dist[d]) for d in DENS], "Share (\\%)": [f1(dist[d] / tot * 100) for d in DENS]}) | |
| 60 | +write("density_dist", tabular(dd, list(dd.columns), "lrr")) | |
| 61 | +surf = pd.read_csv(TAB / "surface_E1.csv") | |
| 62 | +def surf_tab(sel, adj, T_, val="mdape_test"): | |
| 63 | + g = surf[(surf.sel == sel) & (surf.adj == adj) & (surf["T"] == T_)].pivot(index="d", columns="k", values=val) | |
| 64 | + df = g.reset_index(); df["d"] = df["d"].map(lambda d: f"$\\le {d:g}$ km"); cols = ["d"] + [c for c in g.columns] | |
| 65 | + df.columns = ["Radius"] + [f"$k={c}$" for c in g.columns] | |
| 66 | + return tabular(df, list(df.columns), "l" + "r" * len(g.columns), bold_min_cols=list(df.columns[1:])) | |
| 67 | +write("surface_geo_hedo", surf_tab("geo", "hedo", 12)); write("surface_geo_none", surf_tab("geo", "none", 12)); write("surface_sim_hedo", surf_tab("sim", "hedo", 12)) | |
| 68 | +g = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.k == 5)].pivot(index="d", columns="T", values="couverture_test").reset_index() | |
| 69 | +g["d"] = g["d"].map(lambda d: f"$\\le {d:g}$ km"); g.columns = ["Radius"] + [f"{c} months" for c in g.columns[1:]] | |
| 70 | +write("surface_cov", tabular(g, list(g.columns), "lrrrr")) | |
| 71 | +eb = pd.DataFrame(R["E1_best_by_sel_adj"]).T.reset_index() | |
| 72 | +eb["Selection"] = eb["index"].map(lambda s: LAB[s.split("|")[0]]); eb["Adjustment"] = eb["index"].map(lambda s: LAB[s.split("|")[1]]) | |
| 73 | +eb = eb.rename(columns={"k": "$k$", "d": "$d$ (km)", "T": "$t$ (months)", "mdape_val": "MdAPE 2024", "mdape_test": "MdAPE 2025--26", "couverture_test": "Coverage"}) | |
| 74 | +write("e1_best", tabular(eb, ["Selection", "Adjustment", "$k$", "$d$ (km)", "$t$ (months)", "MdAPE 2024", "MdAPE 2025--26", "Coverage"], "llrrrrrr", fmt={"$k$": lambda v: str(int(v)), "$t$ (months)": lambda v: str(int(v)), "$d$ (km)": lambda v: f"{v:g}"})) | |
| 75 | +H = R["hypotheses"] | |
| 76 | +h2 = pd.DataFrame({DENS_EN[k.split("|")[1]]: v for k, v in H["H2"].items() if k.startswith("hedo")}).T.loc[[DENS_EN[d] for d in DENS]].reset_index() | |
| 77 | +h2.columns = ["Density"] + [f"$k={c}$" for c in h2.columns[1:]] | |
| 78 | +write("h2", tabular(h2, list(h2.columns), "l" + "r" * (len(h2.columns) - 1), bold_min_cols=list(h2.columns[1:]))) | |
| 79 | +fac = pd.read_csv(TAB / "facteurs_E2.csv"); anchor = fac[(fac.k == 5) & (fac.d == 5) & (fac["T"] == 12)] | |
| 80 | +adj = anchor[(anchor.sel == "geo") & (anchor.w == "equal") & (anchor.recon == "median")].set_index("adj").loc[["raw", "none", "conv", "hedo", "hedo_reg", "local", "gbm", "role"]].reset_index() | |
| 81 | +adj["Adjustment"] = adj["adj"].map(LAB) | |
| 82 | +adj = adj.rename(columns={"mdape_val": "MdAPE 2024", "mdape_test": "MdAPE 2025--26", "mape_test": "MAPE", "biais_test": "Bias", "p10_test": "Within $\\pm10\\%$", "couverture_test": "Coverage"}) | |
| 83 | +write("adjustments", tabular(adj, ["Adjustment", "MdAPE 2024", "MdAPE 2025--26", "MAPE", "Bias", "Within $\\pm10\\%$", "Coverage"], "lrrrrrr")) | |
| 84 | +def h3tab(prefix): | |
| 85 | + h = pd.DataFrame({DENS_EN[k.split("|")[3]]: v for k, v in H["H3"].items() if k.startswith(prefix)}).T.loc[[DENS_EN[d] for d in DENS]].reset_index() | |
| 86 | + h.columns = ["Density"] + [LAB[c] for c in h.columns[1:]]; return h | |
| 87 | +h3a, h3b = h3tab("5|5|12"), h3tab("3|2|6") | |
| 88 | +t = tabular(h3a, list(h3a.columns), "lrrrrr", bold_min_cols=list(h3a.columns[1:])) | |
| 89 | +t2 = tabular(h3b, list(h3b.columns), "lrrrrr", bold_min_cols=list(h3b.columns[1:])) | |
| 90 | +t = t.replace("\\midrule", "\\midrule\n\\multicolumn{6}{l}{\\emph{Anchor $(5,\\,5\\text{ km},\\,12\\text{ months})$}} \\\\", 1) | |
| 91 | +body2 = t2.split("\\midrule\n", 1)[1].replace("\\bottomrule\n\\end{tabular}\n", "") | |
| 92 | +t = t.replace("\\bottomrule", "\\midrule\n\\multicolumn{6}{l}{\\emph{Anchor $(3,\\,2\\text{ km},\\,6\\text{ months})$}} \\\\\n" + body2 + "\\bottomrule") | |
| 93 | +write("h3", t) | |
| 94 | +mod = json.load(open(OUT / "models_48.json")); b = mod["beta_global"] | |
| 95 | +NAMES = {"ln_floor": "log living area", "ln_land": "log lot area", "age": "age (years)", "age2": "age$^2$/100", "lien1": "semi-detached", "lien2": "row / integrated", "genre1": "construction type: full storeys", "genre2": "split-level", "genre3": "mansard", "etages": "storeys", "neuf": "built within 3 years"} | |
| 96 | +gd = pd.DataFrame({"Characteristic": [NAMES[k] for k in mod["feats"]], "Coefficient": [f"{b[k]:.3f}" for k in mod["feats"]]}) | |
| 97 | +write("grid", tabular(gd, list(gd.columns), "lr") + f"\\par\\footnotesize Within-$R^2$ = {mod['r2_within']:.3f}; $n$ = {n_(mod['n_train'])} sales before {mod['start']}.\n") | |
| 98 | +sel = anchor[(anchor.adj == "hedo") & (anchor.w == "equal") & (anchor.recon == "median")].copy(); sel["Selection"] = sel["sel"].map(LAB) | |
| 99 | +sel = sel.rename(columns={"mdape_val": "MdAPE 2024", "mdape_test": "MdAPE 2025--26", "biais_test": "Bias", "p10_test": "Within $\\pm10\\%$"}) | |
| 100 | +write("selection", tabular(sel, ["Selection", "MdAPE 2024", "MdAPE 2025--26", "Bias", "Within $\\pm10\\%$"], "lrrrr")) | |
| 101 | +wr = anchor[(anchor.adj == "hedo") & (anchor.sel == "geo")].copy(); wr["Weighting"] = wr["w"].map(LAB); wr["Reconciliation"] = wr["recon"].map(LAB) | |
| 102 | +wr = wr.rename(columns={"mdape_val": "MdAPE 2024", "mdape_test": "MdAPE 2025--26", "biais_test": "Bias", "p10_test": "Within $\\pm10\\%$"}) | |
| 103 | +write("weighting", tabular(wr, ["Weighting", "Reconciliation", "MdAPE 2024", "MdAPE 2025--26", "Bias", "Within $\\pm10\\%$"], "llrrrr")) | |
| 104 | +h4 = pd.DataFrame(H["H4"]).T.loc[DENS].reset_index(); h4["index"] = h4["index"].map(DENS_EN) | |
| 105 | +h4.columns = ["Density"] + [f"{LAB[c.split('|')[0]]} / {LAB[c.split('|')[1]]}".replace("distance $\\times$ recency $\\times$ similarity", "product") for c in h4.columns[1:]] | |
| 106 | +write("h4", tabular(h4, list(h4.columns), "l" + "r" * (len(h4.columns) - 1), bold_min_cols=list(h4.columns[1:]))) | |
| 107 | +marg = R["E2_marginal"]; rows = [] | |
| 108 | +for f, lab in [("sel", "Selection"), ("adj", "Adjustment"), ("w", "Weighting"), ("recon", "Reconciliation")]: | |
| 109 | + for k, v in marg[f].items(): rows.append([lab, LAB[k], v]) | |
| 110 | +mg = pd.DataFrame(rows, columns=["Factor", "Level", "Mean MdAPE"]); write("marginal", tabular(mg, list(mg.columns), "llr")) | |
| 111 | +cm = pd.DataFrame(comp).T; cm["val"] = [compv[m]["mdape"] for m in cm.index]; cm = cm.reset_index().rename(columns={"index": "Model"}) | |
| 112 | +cm["Model"] = cm["Model"].str.replace(" (choisie en 2024, couverture ≥ 97 %)", " (best configuration, chosen on 2024)", regex=False) | |
| 113 | +TR = {"M1 médiane des 3 plus proches (temps seul)": "M1 median of 3 nearest, time only", "M2 moyenne des 5 plus proches (temps seul)": "M2 mean of 5 nearest, time only", | |
| 114 | + "M3 sélection par similarité physique": "M3 5 most similar, time only", "M4 pondération par distance": "M4 5 nearest, distance-weighted", "M5 pondération par récence": "M5 5 nearest, recency-weighted", | |
| 115 | + "M6 grille d'ajustement conventionnelle": "M6 conventional grid", "M7 grille d'ajustement hédonique": "M7 hedonic grid", "M8a grille locale (régression sur le voisinage)": "M8a local grid", | |
| 116 | + "M8b ajustements non linéaires (GBM)": "M8b GBM adjustments", "M13 ratio au rôle (5 plus proches, même rôle)": "M13 assessment ratio", "M10 hédonique globale directe": "M10 global hedonic model", | |
| 117 | + "M11a gradient boosting (sans rôle)": "M11a gradient boosting, no assessed value", "M11b gradient boosting (avec rôle)": "M11b gradient boosting, with assessed value", | |
| 118 | + "M9 réconciliation optimisée (poids appris en 2024)": "M9 optimised reconciliation", "M12 hybride comparables + GBM (poids appris en 2024)": "M12 hybrid comparables + GBM"} | |
| 119 | +cm["Model"] = cm["Model"].map(lambda m: TR.get(m, m)); cm = cm.sort_values("mdape") | |
| 120 | +cm = cm.rename(columns={"val": "MdAPE 2024", "mdape": "MdAPE 2025--26", "mape": "MAPE", "biais": "Bias", "p5": "$\\pm5\\%$", "p10": "$\\pm10\\%$", "p20": "$\\pm20\\%$", "couverture": "Coverage"}) | |
| 121 | +write("competition", tabular(cm, ["Model", "MdAPE 2024", "MdAPE 2025--26", "MAPE", "Bias", "$\\pm5\\%$", "$\\pm10\\%$", "$\\pm20\\%$", "Coverage"], "lrrrrrrrr")) | |
| 122 | +pr = pd.read_csv(TAB / "paires_bootstrap.csv"); sh = lambda s: s.split(" ")[0] | |
| 123 | +pr["Comparison"] = pr.apply(lambda r: f"{sh(r.A)} $-$ {sh(r.B)}", axis=1) | |
| 124 | +pr["$\\Delta$ MdAPE [95\\% CI]"] = pr.apply(lambda r: f"{r.diff_mdape:.2f} [{eval(r.ci_mdape)[0]:.2f}; {eval(r.ci_mdape)[1]:.2f}]", axis=1) | |
| 125 | +pr["$\\Delta$ MAPE [95\\% CI]"] = pr.apply(lambda r: f"{r.diff_mape:.2f} [{eval(r.ci_mape)[0]:.2f}; {eval(r.ci_mape)[1]:.2f}]", axis=1) | |
| 126 | +pr["$A$ more accurate (\\%)"] = pr["part_A_meilleur"].map(f1); pr["$n$"] = pr["n"].map(n_) | |
| 127 | +write("pairs", tabular(pr, ["Comparison", "$\\Delta$ MdAPE [95\\% CI]", "$\\Delta$ MAPE [95\\% CI]", "$A$ more accurate (\\%)", "$n$"], "lrrrr")) | |
| 128 | +def strata(by, label, order=None): | |
| 129 | + st = pd.read_csv(TAB / f"strates_{by}.csv"); g = st.pivot(index=by, columns="modele", values="mdape").round(1) | |
| 130 | + g["n"] = st[st.modele == "M7"].set_index(by)["n"]; g = g[["n", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"]] | |
| 131 | + if order: g = g.loc[order] | |
| 132 | + g = g.reset_index(); g[by] = g[by].map(lambda x: DENS_EN.get(x, x)); g = g.rename(columns={by: label, "n": "$n$"}) | |
| 133 | + return tabular(g, list(g.columns), "lr" + "r" * 8, fmt={"$n$": n_}) | |
| 134 | +write("density", strata("dens", "Density", DENS)) | |
| 135 | +st = pd.read_csv(TAB / "strates_region_nom.csv"); reg_order = st[st.modele == "M7"].sort_values("n", ascending=False)["region_nom"].tolist() | |
| 136 | +write("regions", strata("region_nom", "Region", reg_order)) | |
| 137 | +parts = [] | |
| 138 | +for by, lab in [("prix_q", "Price quintile"), ("age_cl", "Age"), ("lien", "Physical link")]: | |
| 139 | + t = strata(by, lab); parts.append(t) | |
| 140 | +LIEN = {"0": "detached", "1": "semi-detached", "2": "row", "3": "integrated"}; AGE = {"≤ 3 ans": "$\\le 3$ years", "4-15": "4--15", "16-40": "16--40", "41-70": "41--70", "> 70": "$>70$"} | |
| 141 | +PQ = {"Q1 (bas)": "Q1 (low)", "Q5 (haut)": "Q5 (high)"} | |
| 142 | +txt = parts[0] | |
| 143 | +for p_ in parts[1:]: | |
| 144 | + body = p_.split("\\midrule\n", 1)[1].replace("\\bottomrule\n\\end{tabular}\n", "") | |
| 145 | + head = p_.split("\\toprule\n", 1)[1].split(" \\\\", 1)[0] | |
| 146 | + txt = txt.replace("\\bottomrule", "\\midrule\n" + head + " \\\\\n\\midrule\n" + body + "\\bottomrule") | |
| 147 | +for k, v in {**LIEN, **AGE, **PQ}.items(): txt = txt.replace(f"\n{esc(k)} &", f"\n{v} &") | |
| 148 | +write("strata_other", txt) | |
| 149 | +h5 = pd.DataFrame(H["H5"]).T.reset_index().rename(columns={"index": "Submarket", "sd_median": "Median $\\hat\\sigma$"}) | |
| 150 | +h5["Submarket"] = h5["Submarket"].map({"homogène": "homogeneous", "moyen": "intermediate", "hétérogène": "heterogeneous"}) | |
| 151 | +write("h5", tabular(h5, ["Submarket", "Median $\\hat\\sigma$", "M2", "M7", "M8a", "M13", "M0", "M10", "M11b", "M12"], "lrrrrrrrrr", fmt={"Median $\\hat\\sigma$": lambda v: f"{v:.3f}"})) | |
| 152 | +iv = pd.DataFrame(R["intervalles"]).T.reset_index() | |
| 153 | +iv["Configuration"] = iv["index"].map(lambda i: f"{LAB[i.split('|')[0]]}, $k={i.split('|')[1]}$, $\\le {i.split('|')[2]}$ km, {i.split('|')[3]} months") | |
| 154 | +iv = iv.rename(columns={"z80_val": "$z$ (calibrated on 2024)", "couverture_nominale_80": "Coverage at $z=1.28$", "couverture_calibree": "Coverage, calibrated $z$", "largeur_mediane_pct": "Median width (\\%)", "n_test": "$n$"}) | |
| 155 | +write("intervals", tabular(iv, ["Configuration", "$z$ (calibrated on 2024)", "Coverage at $z=1.28$", "Coverage, calibrated $z$", "Median width (\\%)", "$n$"], "lrrrrr", fmt={"$n$": n_, "$z$ (calibrated on 2024)": f2})) | |
| 156 | +rc = pd.DataFrame(R["regles_conditionnelles"]).T.loc[DENS].reset_index(); rc["Density"] = rc["index"].map(DENS_EN) | |
| 157 | +rc["Rule"] = rc.apply(lambda r: f"{LAB[r.sel]}, $k={int(r.k)}$, $\\le {float(r.d):g}$ km, {int(r['T'])} mo, {LAB[r.adj]}, {LAB[r.w]} weights, {LAB[r.recon]}", axis=1) | |
| 158 | +rc = rc.rename(columns={"mdape": "MdAPE 2024", "mdape_test": "MdAPE 2025--26", "couverture": "Coverage 2024", "n_test": "$n$ test"}) | |
| 159 | +write("cond_rules", tabular(rc, ["Density", "Rule", "MdAPE 2024", "MdAPE 2025--26", "Coverage 2024", "$n$ test"], "llrrrr", fmt={"$n$ test": n_})) | |
| 160 | +stab = pd.DataFrame(R["stabilite"]).rename(columns={"trimestre": "Quarter", "n": "$n$", "biais_M7": "Bias M7", "biais_M11b": "Bias M11b"}) | |
| 161 | +write("stability", tabular(stab, ["Quarter", "$n$", "M2", "M7", "M0", "M11b", "M12", "Bias M7", "Bias M11b"], "lrrrrrrrr", fmt={"$n$": n_})) | |
| 162 | +for f in ["f1_mdape_vs_k", "f2_surface_d_T", "f3_ajustements", "f4_competition", "f5_densite", "f6_k_par_densite", "f7_indice"]: | |
| 163 | + shutil.copy(OUT / "fig" / f"{f}.pdf", P / "figures" / f"{f}.pdf") | |
| 164 | + | |
| 165 | +# ------------------------------------------------------------------ macros | |
| 166 | +m = {} | |
| 167 | +m["nSalesRaw"] = n_(D["brut"]); m["nSalesClean"] = n_(D["propre"]); m["nTrain"] = n_(D["par_periode"]["train"]); m["nVal"] = n_(D["par_periode"]["val"]); m["nTest"] = n_(D["par_periode"]["test"]) | |
| 168 | +m["nTargetsVal"] = n_(R["n_targets"]["val"]); m["nTargetsTest"] = n_(R["n_targets"]["test"]); m["nTargetsAll"] = n_(R["n_targets"]["val"] + R["n_targets"]["test"]) | |
| 169 | +m["nConfigs"] = "2,256"; m["nTransactionsDb"] = "766,596"; m["nUnits"] = "3.75 million"; m["dateMin"] = D["dates"][0]; m["dateMax"] = D["dates"][1]; m["medianPrice"] = n_(D["prix"]["50%"]) | |
| 170 | +for k, name in [("M0", "mdapeMzero"), ("M2", "mdapeMtwo"), ("M3", "mdapeMthree"), ("M6", "mdapeMsix"), ("M7", "mdapeMseven"), ("M9", "mdapeMnine"), ("M10", "mdapeMten"), ("M11a", "mdapeMelevenA"), ("M11b", "mdapeMelevenB"), ("M12", "mdapeMtwelve"), ("M13", "mdapeMthirteen")]: | |
| 171 | + m[name] = pc(comp[M[k]]["mdape"]) | |
| 172 | +m["convMdape"] = pc(comp[M["M6"]]["mdape"]); m["localMdape"] = pc(comp[M["M8a"]]["mdape"]); m["covMthirteen"] = pc(comp[M["M13"]]["couverture"]) | |
| 173 | +m["pTenMzero"] = pc(comp[M["M0"]]["p10"]); m["pTenMelevenB"] = pc(comp[M["M11b"]]["p10"]) | |
| 174 | +r15 = [comp[M[k]]["mdape"] for k in ["M1", "M2", "M4", "M5"]]; m["mdapeMoneRange"] = f"{min(r15):.1f}--{max(r15):.1f}\\%" | |
| 175 | +rg = [comp[M[k]]["mdape"] for k in ["M3", "M6", "M7"]]; m["mdapeGridRange"] = f"{min(rg):.1f}--{max(rg):.1f}\\%" | |
| 176 | +m["transparencyCost"] = f1(comp[M["M0"]]["mdape"] - comp[M["M11b"]]["mdape"]); m["transparencyCostNoRole"] = f1(comp[M["M0"]]["mdape"] - comp[M["M11a"]]["mdape"]) | |
| 177 | +m["roleGainRatio"] = f1(comp[M["M7"]]["mdape"] - comp[M["M13"]]["mdape"]); m["roleGainGbm"] = f1(comp[M["M11a"]]["mdape"] - comp[M["M11b"]]["mdape"]) | |
| 178 | +ea = R["E2_adjust_anchor"] | |
| 179 | +m["gainTime"] = f1(ea["raw"]["mdape_test"] - ea["none"]["mdape_test"]) | |
| 180 | +grids = [ea[a]["mdape_test"] for a in ["conv", "hedo", "hedo_reg", "local", "gbm"]] | |
| 181 | +m["gainGridLo"] = f1(ea["none"]["mdape_test"] - max(grids)); m["gainGridHi"] = f1(ea["none"]["mdape_test"] - min(grids)) | |
| 182 | +m["localVsHedo"] = "slightly worse" if ea["local"]["mdape_test"] > ea["hedo"]["mdape_test"] else "slightly better" | |
| 183 | +m["gbmVsHedo"] = f1(ea["hedo"]["mdape_test"] - ea["gbm"]["mdape_test"]) | |
| 184 | +es = R["E2_sel_anchor"]; m["gainSelHedo"] = f1(es["geo"]["mdape_test"] - es["sim"]["mdape_test"]) | |
| 185 | +wrv = [v["mdape_test"] for v in R["E2_wr_anchor"].values()]; m["wrRange"] = f1(max(wrv) - min(wrv)) | |
| 186 | +mg_ = R["E2_marginal"] | |
| 187 | +for f, nm in [("sel", "Sel"), ("adj", "Adj"), ("w", "W")]: | |
| 188 | + m[f"marg{nm}Lo"] = f1(min(mg_[f].values())); m[f"marg{nm}Hi"] = f1(max(mg_[f].values())) | |
| 189 | +m["margReconMedian"] = f1(mg_["recon"]["median"]); m["margReconMean"] = f1(mg_["recon"]["mean"]) | |
| 190 | +kb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.d == 10) & (surf["T"] == 12)].set_index("k")["mdape_test"] | |
| 191 | +m["kOpt"] = str(int(kb.idxmin())); m["mdapeKone"] = pc(kb[1]); m["mdapeKthree"] = pc(kb[3]); m["mdapeKfive"] = pc(kb[5]); m["mdapeKten"] = pc(kb[10]); m["mdapeKopt"] = pc(kb.min()); m["mdapeKfifty"] = pc(kb[50]) | |
| 192 | +nb = surf[(surf.sel == "geo") & (surf.adj == "hedo") & (surf.k == 3) & (surf.d == 1) & (surf["T"] == 3)].iloc[0] | |
| 193 | +m["narrowMdape"] = pc(nb.mdape_test); m["narrowCov"] = pc(nb.couverture_test) | |
| 194 | +e1b = R["E1_best_val"]; m["bestSel"] = LAB[e1b["sel"]]; m["bestK"] = str(int(e1b["k"])); m["bestD"] = f"{float(e1b['d']):g}"; m["bestT"] = str(int(e1b["T"])); m["bestAdj"] = LAB[e1b["adj"]] | |
| 195 | +m["bestValMdape"] = pc(e1b["mdape_val"]); m["bestTestMdape"] = pc(e1b["mdape_test"]) | |
| 196 | +m0 = R["M0"]; m["mZeroDesc"] = f"{LAB[m0['sel']]}, $k={int(m0['k'])}$, $\\le {float(m0['d']):g}$ km, {int(m0['T'])} months, {LAB[m0['adj']]}, {LAB[m0['w']]} weights, {LAB[m0['recon']]}" | |
| 197 | +mu = R["M0_sans_contrainte"]; m["mZeroUncDesc"] = f"{LAB[mu['sel']]}, $k={int(mu['k'])}$, $\\le {float(mu['d']):g}$ km, {int(mu['T'])} months, {LAB[mu['adj']]}"; m["mZeroUncMdape"] = pc(mu["mdape_test"]); m["mZeroUncCov"] = pc(mu["couverture_test"]) | |
| 198 | +dn = R["strates_dens"]; m["densHighMseven"] = pc(dn["≥ 200"]["M7"]); m["densLowMseven"] = pc(dn["< 5"]["M7"]) | |
| 199 | +h1 = H["H1"]; m["hOneGeoDone"] = pc(h1["geo_d1"]["mdape_test"]); m["hOneGeoDoneCov"] = pc(h1["geo_d1"]["couverture_test"]); m["hOneSimDten"] = pc(h1["sim_d10"]["mdape_test"]) | |
| 200 | +m["hOneGeoDtenHedo"] = pc(h1["geo_d10_hedo"]["mdape_test"]); m["hOneSimDtenHedo"] = pc(h1["sim_d10_hedo"]["mdape_test"]); m["hOneMixDtenHedo"] = pc(h1["mix_d10_hedo"]["mdape_test"]) | |
| 201 | +h5 = H["H5"]; m["hFiveGapHomog"] = f1(h5["homogène"]["M0"] - h5["homogène"]["M11b"]); m["hFiveGapHeter"] = f1(h5["hétérogène"]["M0"] - h5["hétérogène"]["M11b"]) | |
| 202 | +ivh = R["intervalles"]["hedo|10|10|24"]; m["ivWidth"] = pc(ivh["largeur_mediane_pct"]) | |
| 203 | +cg = R["regles_conditionnelles_global_test"]; m["condMdape"] = pc(cg["mdape"]); m["condPten"] = pc(cg["p10"]); m["condCov"] = pc(cg["couverture"]) | |
| 204 | +m["betaFloor"] = f"{b['ln_floor']:.2f}"; m["betaLand"] = f"{b['ln_land']:.2f}"; m["betaAge"] = f"{abs(b['age']) * 100:.1f}\\%"; m["betaSemi"] = f"{abs(b['lien1']) * 100:.1f}\\%"; m["betaRow"] = f"{abs(b['lien2']) * 100:.1f}\\%" | |
| 205 | +lines = [f"\\newcommand{{\\{k}}}{{{v}}}" for k, v in m.items()] | |
| 206 | +(P / "tables" / "numbers.tex").write_text("% généré par make_paper_tables.py\n" + "\n".join(lines) + "\n") | |
| 207 | +print(f"{len(m)} macros, tables et figures écrites") | |
added
paper/references.bib
+276 −0
@@ -0,0 +1,276 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% References — Working Paper 11 | |
| 3 | +% ============================================================================ | |
| 4 | + | |
| 5 | +@book{appraisal2020, | |
| 6 | + author = {{Appraisal Institute}}, | |
| 7 | + title = {The Appraisal of Real Estate}, | |
| 8 | + edition = {15th}, | |
| 9 | + publisher = {Appraisal Institute}, | |
| 10 | + address = {Chicago}, | |
| 11 | + year = {2020} | |
| 12 | +} | |
| 13 | + | |
| 14 | +@techreport{iaao2017, | |
| 15 | + author = {{International Association of Assessing Officers}}, | |
| 16 | + title = {Standard on Mass Appraisal of Real Property}, | |
| 17 | + institution = {IAAO}, | |
| 18 | + address = {Kansas City}, | |
| 19 | + year = {2017} | |
| 20 | +} | |
| 21 | + | |
| 22 | +@techreport{iaao2018, | |
| 23 | + author = {{International Association of Assessing Officers}}, | |
| 24 | + title = {Standard on Automated Valuation Models ({AVM}s)}, | |
| 25 | + institution = {IAAO}, | |
| 26 | + address = {Kansas City}, | |
| 27 | + year = {2018} | |
| 28 | +} | |
| 29 | + | |
| 30 | +@article{colwell1983, | |
| 31 | + author = {Colwell, Peter F. and Cannaday, Roger E. and Wu, Chunchi}, | |
| 32 | + title = {The Analytical Foundations of Adjustment Grid Methods}, | |
| 33 | + journal = {Real Estate Economics}, | |
| 34 | + volume = {11}, | |
| 35 | + number = {1}, | |
| 36 | + pages = {11--29}, | |
| 37 | + year = {1983} | |
| 38 | +} | |
| 39 | + | |
| 40 | +@article{vandell1991, | |
| 41 | + author = {Vandell, Kerry D.}, | |
| 42 | + title = {Optimal Comparable Selection and Weighting in Real Property Valuation}, | |
| 43 | + journal = {Real Estate Economics}, | |
| 44 | + volume = {19}, | |
| 45 | + number = {2}, | |
| 46 | + pages = {213--239}, | |
| 47 | + year = {1991} | |
| 48 | +} | |
| 49 | + | |
| 50 | +@article{gau1992, | |
| 51 | + author = {Gau, George W. and Lai, Tsong-Yue and Wang, Ko}, | |
| 52 | + title = {Optimal Comparable Selection and Weighting in Real Property Valuation: An Extension}, | |
| 53 | + journal = {Real Estate Economics}, | |
| 54 | + volume = {20}, | |
| 55 | + number = {1}, | |
| 56 | + pages = {107--123}, | |
| 57 | + year = {1992} | |
| 58 | +} | |
| 59 | + | |
| 60 | +@article{gau1994, | |
| 61 | + author = {Gau, George W. and Lai, Tsong-Yue and Wang, Ko}, | |
| 62 | + title = {A Further Discussion of Optimal Comparable Selection and Weighting, and a Response to {Green}}, | |
| 63 | + journal = {Real Estate Economics}, | |
| 64 | + volume = {22}, | |
| 65 | + number = {4}, | |
| 66 | + pages = {655--663}, | |
| 67 | + year = {1994} | |
| 68 | +} | |
| 69 | + | |
| 70 | +@article{isakson1986, | |
| 71 | + author = {Isakson, Hans R.}, | |
| 72 | + title = {The Nearest Neighbors Appraisal Technique: An Alternative to the Adjustment Grid Methods}, | |
| 73 | + journal = {Real Estate Economics}, | |
| 74 | + volume = {14}, | |
| 75 | + number = {2}, | |
| 76 | + pages = {274--286}, | |
| 77 | + year = {1986} | |
| 78 | +} | |
| 79 | + | |
| 80 | +@article{isakson2002, | |
| 81 | + author = {Isakson, Hans R.}, | |
| 82 | + title = {The Linear Algebra of the Sales Comparison Approach}, | |
| 83 | + journal = {Journal of Real Estate Research}, | |
| 84 | + volume = {24}, | |
| 85 | + number = {2}, | |
| 86 | + pages = {117--128}, | |
| 87 | + year = {2002} | |
| 88 | +} | |
| 89 | + | |
| 90 | +@article{lentz1998, | |
| 91 | + author = {Lentz, George H. and Wang, Ko}, | |
| 92 | + title = {Residential Appraisal and the Lending Process: A Survey of Issues}, | |
| 93 | + journal = {Journal of Real Estate Research}, | |
| 94 | + volume = {15}, | |
| 95 | + number = {1}, | |
| 96 | + pages = {11--39}, | |
| 97 | + year = {1998} | |
| 98 | +} | |
| 99 | + | |
| 100 | +@article{kang1991, | |
| 101 | + author = {Kang, Han-Bin and Reichert, Alan K.}, | |
| 102 | + title = {An Empirical Analysis of Hedonic Regression and Grid-Adjustment Techniques in Real Estate Appraisal}, | |
| 103 | + journal = {Real Estate Economics}, | |
| 104 | + volume = {19}, | |
| 105 | + number = {1}, | |
| 106 | + pages = {70--91}, | |
| 107 | + year = {1991} | |
| 108 | +} | |
| 109 | + | |
| 110 | +@article{pace1998, | |
| 111 | + author = {Pace, R. Kelley}, | |
| 112 | + title = {Appraisal Using Generalized Additive Models}, | |
| 113 | + journal = {Journal of Real Estate Research}, | |
| 114 | + volume = {15}, | |
| 115 | + number = {1}, | |
| 116 | + pages = {77--99}, | |
| 117 | + year = {1998} | |
| 118 | +} | |
| 119 | + | |
| 120 | +@article{pace1997, | |
| 121 | + author = {Pace, R. Kelley and Gilley, Otis W.}, | |
| 122 | + title = {Using the Spatial Configuration of the Data to Improve Estimation}, | |
| 123 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 124 | + volume = {14}, | |
| 125 | + number = {3}, | |
| 126 | + pages = {333--340}, | |
| 127 | + year = {1997} | |
| 128 | +} | |
| 129 | + | |
| 130 | +@article{rosen1974, | |
| 131 | + author = {Rosen, Sherwin}, | |
| 132 | + title = {Hedonic Prices and Implicit Markets: Product Differentiation in Pure Competition}, | |
| 133 | + journal = {Journal of Political Economy}, | |
| 134 | + volume = {82}, | |
| 135 | + number = {1}, | |
| 136 | + pages = {34--55}, | |
| 137 | + year = {1974} | |
| 138 | +} | |
| 139 | + | |
| 140 | +@article{bourassa2010, | |
| 141 | + author = {Bourassa, Steven C. and Cantoni, Eva and Hoesli, Martin}, | |
| 142 | + title = {Predicting House Prices with Spatial Dependence: A Comparison of Alternative Methods}, | |
| 143 | + journal = {Journal of Real Estate Research}, | |
| 144 | + volume = {32}, | |
| 145 | + number = {2}, | |
| 146 | + pages = {139--159}, | |
| 147 | + year = {2010} | |
| 148 | +} | |
| 149 | + | |
| 150 | +@article{case2004, | |
| 151 | + author = {Case, Bradford and Clapp, John and Dubin, Robin and Rodriguez, Mauricio}, | |
| 152 | + title = {Modeling Spatial and Temporal House Price Patterns: A Comparison of Four Models}, | |
| 153 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 154 | + volume = {29}, | |
| 155 | + number = {2}, | |
| 156 | + pages = {167--191}, | |
| 157 | + year = {2004} | |
| 158 | +} | |
| 159 | + | |
| 160 | +@article{bailey1963, | |
| 161 | + author = {Bailey, Martin J. and Muth, Richard F. and Nourse, Hugh O.}, | |
| 162 | + title = {A Regression Method for Real Estate Price Index Construction}, | |
| 163 | + journal = {Journal of the American Statistical Association}, | |
| 164 | + volume = {58}, | |
| 165 | + number = {304}, | |
| 166 | + pages = {933--942}, | |
| 167 | + year = {1963} | |
| 168 | +} | |
| 169 | + | |
| 170 | +@article{case1989, | |
| 171 | + author = {Case, Karl E. and Shiller, Robert J.}, | |
| 172 | + title = {The Efficiency of the Market for Single-Family Homes}, | |
| 173 | + journal = {American Economic Review}, | |
| 174 | + volume = {79}, | |
| 175 | + number = {1}, | |
| 176 | + pages = {125--137}, | |
| 177 | + year = {1989} | |
| 178 | +} | |
| 179 | + | |
| 180 | +@article{hill2013, | |
| 181 | + author = {Hill, Robert J.}, | |
| 182 | + title = {Hedonic Price Indexes for Residential Housing: A Survey, Evaluation and Taxonomy}, | |
| 183 | + journal = {Journal of Economic Surveys}, | |
| 184 | + volume = {27}, | |
| 185 | + number = {5}, | |
| 186 | + pages = {879--914}, | |
| 187 | + year = {2013} | |
| 188 | +} | |
| 189 | + | |
| 190 | +@book{gloudemans1999, | |
| 191 | + author = {Gloudemans, Robert J.}, | |
| 192 | + title = {Mass Appraisal of Real Property}, | |
| 193 | + publisher = {International Association of Assessing Officers}, | |
| 194 | + address = {Chicago}, | |
| 195 | + year = {1999} | |
| 196 | +} | |
| 197 | + | |
| 198 | +@article{pagourtzi2003, | |
| 199 | + author = {Pagourtzi, Elli and Assimakopoulos, Vassilis and Hatzichristos, Thomas and French, Nick}, | |
| 200 | + title = {Real Estate Appraisal: A Review of Valuation Methods}, | |
| 201 | + journal = {Journal of Property Investment \& Finance}, | |
| 202 | + volume = {21}, | |
| 203 | + number = {4}, | |
| 204 | + pages = {383--401}, | |
| 205 | + year = {2003} | |
| 206 | +} | |
| 207 | + | |
| 208 | +@article{mccluskey2013, | |
| 209 | + author = {McCluskey, William J. and McCord, Michael and Davis, Peadar T. and Haran, Martin and McIlhatton, David}, | |
| 210 | + title = {Prediction Accuracy in Mass Appraisal: A Comparison of Modern Approaches}, | |
| 211 | + journal = {Journal of Property Research}, | |
| 212 | + volume = {30}, | |
| 213 | + number = {4}, | |
| 214 | + pages = {239--265}, | |
| 215 | + year = {2013} | |
| 216 | +} | |
| 217 | + | |
| 218 | +@article{mayer2019, | |
| 219 | + author = {Mayer, Michael and Bourassa, Steven C. and Hoesli, Martin and Scognamiglio, Donato}, | |
| 220 | + title = {Estimation and Updating Methods for Hedonic Valuation}, | |
| 221 | + journal = {Journal of European Real Estate Research}, | |
| 222 | + volume = {12}, | |
| 223 | + number = {1}, | |
| 224 | + pages = {134--150}, | |
| 225 | + year = {2019} | |
| 226 | +} | |
| 227 | + | |
| 228 | +@article{pace2020, | |
| 229 | + author = {Pace, R. Kelley and Hayunga, Darren}, | |
| 230 | + title = {Examining the Information Content of Residuals from Hedonic and Spatial Models Using Trees and Forests}, | |
| 231 | + journal = {Journal of Real Estate Finance and Economics}, | |
| 232 | + volume = {60}, | |
| 233 | + number = {1--2}, | |
| 234 | + pages = {170--180}, | |
| 235 | + year = {2020} | |
| 236 | +} | |
| 237 | + | |
| 238 | +@article{diaz1990, | |
| 239 | + author = {Diaz, Julian}, | |
| 240 | + title = {How Appraisers Do Their Work: A Test of the Appraisal Process and the Development of a Descriptive Model}, | |
| 241 | + journal = {Journal of Real Estate Research}, | |
| 242 | + volume = {5}, | |
| 243 | + number = {1}, | |
| 244 | + pages = {1--15}, | |
| 245 | + year = {1990} | |
| 246 | +} | |
| 247 | + | |
| 248 | +@article{diaz2001, | |
| 249 | + author = {Diaz, Julian and Hansz, J. Andrew}, | |
| 250 | + title = {The Use of Reference Points in Valuation Judgment}, | |
| 251 | + journal = {Journal of Property Research}, | |
| 252 | + volume = {18}, | |
| 253 | + number = {2}, | |
| 254 | + pages = {141--148}, | |
| 255 | + year = {2001} | |
| 256 | +} | |
| 257 | + | |
| 258 | +@article{northcraft1987, | |
| 259 | + author = {Northcraft, Gregory B. and Neale, Margaret A.}, | |
| 260 | + title = {Experts, Amateurs, and Real Estate: An Anchoring-and-Adjustment Perspective on Property Pricing Decisions}, | |
| 261 | + journal = {Organizational Behavior and Human Decision Processes}, | |
| 262 | + volume = {39}, | |
| 263 | + number = {1}, | |
| 264 | + pages = {84--97}, | |
| 265 | + year = {1987} | |
| 266 | +} | |
| 267 | + | |
| 268 | +@article{cameron2008, | |
| 269 | + author = {Cameron, A. Colin and Gelbach, Jonah B. and Miller, Douglas L.}, | |
| 270 | + title = {Bootstrap-Based Improvements for Inference with Clustered Errors}, | |
| 271 | + journal = {Review of Economics and Statistics}, | |
| 272 | + volume = {90}, | |
| 273 | + number = {3}, | |
| 274 | + pages = {414--427}, | |
| 275 | + year = {2008} | |
| 276 | +} | |
added
paper/sections/conclusion.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Conclusion} | |
| 3 | +\label{sec:conclusion} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +The sales comparison approach is usually defended on the grounds of its logic and its transparency and criticised on the grounds of its subjectivity. This paper has asked a different question: taken as a set of explicit rules, how accurate is it, and which rules are empirically justified? Using the universe of single-family transactions in Quebec and a backtest that respects the information available on each valuation date, we have measured the accuracy of \nConfigs{} configurations of the method and of the natural alternatives on \nTargetsTest{} out-of-sample sales. | |
| 7 | + | |
| 8 | +The answers are clear enough to be taught. Adjust: a grid---even a crude one---is worth several points of error. Choose similar comparables, not merely near ones, and search widely in space and time. Use more comparables than three: ten to twenty when a grid is applied, more in dense markets and fewer in sparse ones. Do not agonise over weights or over the choice between median and mean. Use the assessed value: the ratio method is the most accurate single adjustment scheme where it applies. And report an interval, calibrated so that it means what it says. A transparent method configured on these principles and frozen in 2024 reaches a median absolute error of \mdapeMzero{} on 2025--2026 sales, within \transparencyCost{} points of the best quarterly-retrained machine-learning model given the same data, and within less than that in homogeneous submarkets. | |
| 9 | + | |
| 10 | +The results have implications for practice, for education and for regulation. For practice, they suggest that the appraiser's effort is best spent on selecting similar comparables and on estimating adjustments from the largest homogeneous pool of evidence, rather than on reconciling a handful of nearby sales by judgement. For education, they provide a set of validated rules that can be taught alongside the conventions, and a laboratory---the backtesting engine---in which students can test their own. For regulation and for the design of automated valuation models, they show that a comparable-based method can be made as accurate as an opaque model in most of the market while remaining fully explainable, and that density-conditional rules dominate any single rule. | |
| 11 | + | |
| 12 | +Three limitations bound the conclusions. We simulate rules; we do not observe appraisers, whose files may contain information (condition, renovations, listing history) that would improve on any rule, and whose judgement may also degrade it. We observe the characteristics of the assessment roll, not those of listings or inspections, so that adjustments for garages, pools, bedrooms and quality are untested and their effects remain in the residual error. And we study single-family dwellings in one province; condominiums, income properties and other markets may yield different optima, though we expect the qualitative hierarchy---adjustment and selection first, number and radius second, weighting last---to carry over. A companion paper develops the recommendation system of Section~\ref{sec:recommend} into a transparent, empirically optimised comparable-sales algorithm and integrates it into a teaching platform. | |
added
paper/sections/data.tex
+54 −0
@@ -0,0 +1,54 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Data} | |
| 3 | +\label{sec:data} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{The Vrai Prix transaction database} | |
| 7 | + | |
| 8 | +Our data come from Vrai Prix, a research database that assembles every residential transfer registered in the Quebec land register (\emph{Registre foncier du Québec}) since January 2021 and links each transfer to the corresponding unit of the municipal assessment roll (\emph{rôle d'évaluation foncière}). The snapshot used here was taken on 10 October 2026 and contains \nTransactionsDb{} transactions and \nUnits{} assessment units covering the whole province. For each transaction we observe the registration date, the price, the civic address and coordinates, the property type, and---through the link to the roll---the physical characteristics recorded by the municipal assessor: living area (sum of floor areas), lot area, year of construction, number of storeys, physical link (detached, semi-detached, row, integrated), construction type, the neighbourhood unit used by the assessor, and the assessed value in force at the date of sale together with its land and building components. Assessed values are revised on a three-year cycle that differs across municipalities; we record, for each sale, whether the assessed value in force belongs to the current roll, the previous roll, or an older one. | |
| 9 | + | |
| 10 | +Two features of this database are essential for our purpose. First, it is a population, not a sample: every registered sale is present, so that the comparable pool available on any valuation date is the one an appraiser with full access to the register would have had. Second, the characteristics are those of the assessment roll, harmonised across the province by the \emph{Manuel d'évaluation foncière du Québec}, rather than listing descriptions whose coding varies by broker. The price of this harmonisation is that some characteristics commonly adjusted for in practice---garage, pool, number of bedrooms and bathrooms, interior quality, condition, renovations---are not recorded in the roll extract and are therefore unavailable. We return to this limitation in Section~\ref{sec:conclusion}. | |
| 11 | + | |
| 12 | +\subsection{Sample construction} | |
| 13 | + | |
| 14 | +We restrict attention to single-family dwellings, the segment for which the sales comparison approach is the reference method. Starting from \nSalesRaw{} transactions coded as single-family, we apply a sequence of explicit exclusion rules, each logged with the number of observations it removes (Table~\ref{tab:exclusions}). The rules remove transactions that cannot be matched to the roll; duplicates; units whose roll record is not a single dwelling; implausible or missing living area, lot area or year of construction; sales closed before the dwelling was built (which are land sales) or in the year of construction (whose prices are, on inspection, those of lots or unfinished buildings: a median of \$346{,}108 and a price-to-assessment ratio of 1.15, against \$450{,}000--480{,}000 and 1.30 for dwellings aged one to four years); prices outside \$50{,}000--\$3{,}000{,}000; and resales within 180 days with a price change above 40\%, which signal renovations or flips for which the recorded characteristics no longer describe the dwelling. | |
| 15 | + | |
| 16 | +The most consequential rule is the exclusion of sales whose price-to-assessment ratio lies outside $[0.33, 3]$. The register does not flag non-arm's-length transfers (intra-family sales, transfers at nominal prices, distress sales), and an extreme ratio is the only observable indicator. An appraiser would screen such sales out of the comparable pool; we therefore exclude them both as targets and as comparables, and report the count. All other rules are standard. The final sample comprises \nSalesClean{} sales between \dateMin{} and \dateMax{}, with a median price of \$\medianPrice{}. | |
| 17 | + | |
| 18 | +\begin{table}[H] | |
| 19 | +\centering | |
| 20 | +\caption{Sample construction} | |
| 21 | +\label{tab:exclusions} | |
| 22 | +\input{tables/exclusions} | |
| 23 | +\end{table} | |
| 24 | + | |
| 25 | +\subsection{Chronological partition and descriptive statistics} | |
| 26 | + | |
| 27 | +The sample is partitioned chronologically. Sales from January 2021 to December 2023 (\nTrain{} observations) serve to develop the adjustment grids and benchmark models. Sales in 2024 (\nVal{} observations) are the \emph{validation} targets, on which configurations and hyperparameters are chosen. Sales from January 2025 to September 2026 (\nTest{} observations) are the \emph{test} targets, on which all results reported as out of sample are computed. Every sale in the sample, whatever its period, is eligible as a comparable for any later target; what distinguishes the periods is the role of the sale as a target. | |
| 28 | + | |
| 29 | +Table~\ref{tab:descriptives} reports descriptive statistics by period and Figure~\ref{fig:index} shows the hedonic price index by region over the sample. The period covers the end of the pandemic boom, a correction in 2022--2023 and a renewed rise in 2024--2026 that is strongly heterogeneous across regions: prices in the Montréal region rose by roughly half as much as in Quebec City or the Montérégie over 2025--2026. This heterogeneity is a feature, not a nuisance: it is precisely the environment in which the quality of time adjustments and the locality of comparables matter. | |
| 30 | + | |
| 31 | +\begin{table}[H] | |
| 32 | +\centering | |
| 33 | +\caption{Descriptive statistics by period} | |
| 34 | +\label{tab:descriptives} | |
| 35 | +\input{tables/descriptives} | |
| 36 | +\end{table} | |
| 37 | + | |
| 38 | +\begin{figure}[H] | |
| 39 | +\centering | |
| 40 | +\includegraphics[width=0.85\textwidth]{f7_indice.pdf} | |
| 41 | +\caption{Hedonic price index by region, full sample (descriptive). Monthly effects from a within-municipality regression of log price on characteristics; January 2021 $=100$. The point-in-time indices used in the backtest are re-estimated every month from prior sales only.} | |
| 42 | +\label{fig:index} | |
| 43 | +\end{figure} | |
| 44 | + | |
| 45 | +\subsection{Market density} | |
| 46 | + | |
| 47 | +Because the accuracy of any comparable-based method depends on the supply of comparables, we characterise each target by the number of sample sales that occurred within 2~km during the twelve months preceding its date. This \emph{density} measure is computed from the target's own comparable pool and is therefore available at valuation time. We group targets into five strata---fewer than 5, 5--19, 20--49, 50--199 and 200 or more sales---which range from rural municipalities and small towns to the dense suburbs of Montréal and Quebec City. Table~\ref{tab:density_dist} gives the distribution of test targets across strata. | |
| 48 | + | |
| 49 | +\begin{table}[H] | |
| 50 | +\centering | |
| 51 | +\caption{Distribution of test targets by market density} | |
| 52 | +\label{tab:density_dist} | |
| 53 | +\input{tables/density_dist} | |
| 54 | +\end{table} | |
added
paper/sections/introduction.tex
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Introduction} | |
| 3 | +\label{sec:introduction} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +The sales comparison approach is the workhorse of residential property valuation. Appraisers, assessors, lenders and courts rely on it daily: a small set of recent sales of similar properties is selected, each sale price is adjusted for the differences between the comparable and the subject, and the adjusted prices are reconciled into a single opinion of value \citep{appraisal2020,iaao2017}. The approach is taught in every valuation curriculum and codified in professional standards, and its logic---that the market itself reveals what buyers pay for a bundle of characteristics---is intuitively compelling. Yet the \emph{operating rules} that turn this logic into a number are, to a remarkable degree, matters of convention rather than evidence. How many comparables should be used? Three is the traditional answer; some standards say ``at least three.'' How far should one search, and how far back in time? Which adjustments are warranted, and of what magnitude? Should comparables be weighted equally, by proximity, by recency, or by similarity? Should adjusted prices be averaged, or should the appraiser ``reconcile'' them by judgement? Each of these questions has been debated in the professional and academic literature for four decades \citep{colwell1983,vandell1991,gau1992,isakson2002}, but the empirical evidence remains thin, dated, and almost always drawn from small samples in a single metropolitan area. | |
| 7 | + | |
| 8 | +This paper provides, to our knowledge, the largest and most systematic empirical validation of the sales comparison approach to date. We exploit a new research database---Vrai Prix---that links the universe of registered residential transactions in Quebec to the municipal assessment roll, giving us \nSalesClean{} arm's-length single-family sales from January 2021 to September 2026 with harmonised physical characteristics, coordinates, and assessed values. On this base we build a \emph{point-in-time backtesting} engine. For each of \nTargetsAll{} target sales occurring in 2024, 2025 and 2026, the sale price is masked and the property is valued as an appraiser would have had to value it on the day of the sale: only transactions closed strictly before that date are admissible as comparables; the price index used to time-adjust them is re-estimated every month from prior sales only; the adjustment grids and machine-learning benchmarks are re-estimated every quarter from prior sales only. The masked price is then compared with the estimate. Because every configuration of the method is applied to exactly the same targets, differences in accuracy are paired differences, and we quantify their uncertainty with a cluster bootstrap over municipality--quarter cells that respects spatial and temporal dependence. | |
| 9 | + | |
| 10 | +Within this engine we run \nConfigs{} configurations of the sales comparison approach. A first experiment maps the \emph{error surface} $\mathrm{Error}(k, d, t)$ over the number of comparables $k \in \{1,\dots,50\}$, the search radius $d \in \{0.5,\dots,25\}$~km and the look-back window $t \in \{3,\dots,24\}$ months, for three selection rules (nearest, most physically similar, and a compromise) and two adjustment regimes (time only, hedonic grid). A second experiment is a full factorial design over the \emph{rules} of the method at three anchor configurations: four selection criteria, eight adjustment schemes (none; time only; a stylised conventional grid; a global hedonic grid; a regional grid; a locally re-estimated grid; non-linear adjustments from a gradient-boosting model; and the assessment-ratio adjustment), five weighting schemes, and two reconciliation rules. Thirteen named models, from ``median of the three nearest sales'' to a quarterly-retrained gradient-boosting model and a hybrid, are then placed in a horse race, and five hypotheses about the trade-offs between proximity, similarity, sample size, local estimation, weighting and market liquidity are tested directly. | |
| 11 | + | |
| 12 | +Five results stand out. First, \emph{adjustments dominate}. Averaging the five nearest time-adjusted sales yields a median absolute percentage error (MdAPE) of \mdapeMtwo{}; applying a hedonic adjustment grid to the same five sales cuts it to \mdapeMseven{}, and a stylised conventional grid---half the comparable's price per square metre for living area, half the implied land value per square metre for lot size, half a percent per year of age---captures most of that gain (\mdapeMsix{}). What matters is \emph{that} one adjusts, far more than the sophistication of the grid: regional, local and non-linear refinements add little, and the locally re-estimated grid is unstable where sales are sparse. Second, \emph{similarity beats proximity}. With no physical adjustment at all, selecting the five most similar sales within a wide radius rather than the five nearest yields \mdapeMthree{}, essentially the same accuracy as adjusting the nearest ones. A nearby but dissimilar sale is a worse comparable than a more distant but similar one. Third, \emph{three comparables is too few}. On the error surface, accuracy improves monotonically from one to roughly \kOpt{} comparables when a grid is used, and the optimum increases with the density of the local market; narrow radii and three-month windows are dominated everywhere once comparables are adjusted. Fourth, \emph{weighting and reconciliation matter little}: the choice between median and mean, or between equal, distance, recency and similarity weights, moves the MdAPE by a few tenths of a point, an order of magnitude less than selection and adjustment. Fifth, \emph{the cost of transparency is small}. The best configuration of the method chosen on 2024 data reaches \mdapeMzero{} out of sample in 2025--2026, against \mdapeMelevenB{} for a gradient-boosting model retrained every quarter with access to the assessed value, and \mdapeMtwelve{} for a hybrid that combines the two. In homogeneous submarkets the gap closes. Accuracy is, above all, a function of submarket liquidity: the same method is several times more accurate where hundreds of sales occur within two kilometres in a year than where fewer than five do, and simple density-conditional rules outperform any single rule. | |
| 13 | + | |
| 14 | +The paper contributes to three literatures. It extends the small-sample studies of comparable selection and weighting \citep{colwell1983,vandell1991,gau1992,gau1994,isakson2002} to a population-scale, out-of-sample, point-in-time setting, and resolves several of their open questions with confidence intervals rather than illustrations. It speaks to the literature comparing hedonic and grid-adjustment techniques \citep{kang1991,lentz1998,pace1998} and the more recent comparisons of machine learning with traditional valuation \citep{mccluskey2013,mayer2019,bourassa2010,pace2020}, by holding the information set fixed across methods and measuring, for the first time, the price of interpretability in a transparent method tuned to the same data. And it offers the profession and its regulators a reproducible empirical basis for rules that are currently taught as conventions. We are careful about what the evidence can and cannot say: we validate the \emph{predictive accuracy of simulated professional rules} on observable characteristics, not the judgement of actual appraisers, whose files, comparables and adjustments we do not observe. | |
| 15 | + | |
| 16 | +The remainder of the paper is organised as follows. Section~\ref{sec:literature} reviews the literature on the sales comparison approach, adjustment estimation and automated valuation. Section~\ref{sec:data} describes the Vrai Prix database and the construction of the analysis sample. Section~\ref{sec:methodology} presents the point-in-time backtesting protocol, the configuration space, the benchmark models and the inference strategy. Section~\ref{sec:results} reports the results, Section~\ref{sec:robustness} examines their stability across submarkets, time and specifications, and Section~\ref{sec:conclusion} concludes. | |
added
paper/sections/literature.tex
+26 −0
@@ -0,0 +1,26 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Related literature} | |
| 3 | +\label{sec:literature} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{The sales comparison approach and its conventions} | |
| 7 | + | |
| 8 | +The sales comparison approach rests on the principle of substitution: a buyer will pay no more for a property than the cost of acquiring an equally desirable substitute \citep{appraisal2020}. Its implementation follows a sequence that has barely changed since the method was formalised: identify comparable sales, verify that they are arm's-length, adjust each sale price for differences in transaction conditions, market conditions (time), location and physical characteristics, and reconcile the adjusted indications into a point estimate. Professional texts and standards give guidance on each step but rarely quantify it. The Appraisal Institute recommends ``a sufficient number'' of comparables and treats three as a practical minimum; the International Association of Assessing Officers' standard on automated valuation models recommends that comparable-sales algorithms use ``three to ten'' sales and weight them by similarity \citep{iaao2018}. Adjustments are to be ``market-derived,'' typically through paired-sales analysis, but in practice they are frequently taken from rules of thumb, published grids, or the appraiser's experience \citep{diaz1990,diaz2001}. | |
| 9 | + | |
| 10 | +A long-standing academic literature has questioned whether these conventions are optimal. \citet{colwell1983} formalised the approach as a statistical estimator and showed that the familiar adjustment grid is a special case of a regression in which the coefficients are fixed a priori. \citet{vandell1991} derived the optimal weights to assign to comparables as a function of their adjustment variance, and showed that equal weighting is optimal only under restrictive conditions. \citet{gau1992,gau1994} extended this to the joint problem of selecting and weighting comparables, proposing a minimum-variance criterion that trades off the number of comparables against their heterogeneity. \citet{isakson1986,isakson2002} examined the nearest-neighbour logic of comparable selection and the role of distance metrics in characteristic space. \citet{lentz1998} survey the appraisal issues raised by the lending process, including the reliance on a small number of comparables and on judgemental adjustments. These contributions are theoretical or illustrated on small samples---typically a few hundred sales in one city---and none evaluates the rules out of sample on a large, heterogeneous market. Our design implements their estimators (nearest, most similar, minimum-adjustment selection; equal, distance, recency and similarity weights; median and mean reconciliation) as configurations of a single engine and measures their accuracy on hundreds of thousands of masked sales. | |
| 11 | + | |
| 12 | +\subsection{Estimating adjustments: grids, hedonics and local models} | |
| 13 | + | |
| 14 | +The estimation of adjustment magnitudes is where the sales comparison approach meets hedonic price theory. Since \citet{rosen1974}, the hedonic regression of price on characteristics has been the canonical tool for recovering implicit prices, and its coefficients are natural candidates for adjustment factors \citep{kang1991,pace1998}. \citet{kang1991} compared hedonic regression with grid-adjustment techniques on sales in a single county and found hedonic adjustments superior; \citet{pace1998} and \citet{pace1997} argued for non-parametric and spatial estimation of adjustments. More recent work has emphasised spatial heterogeneity in implicit prices \citep{bourassa2010,case2004} and the use of geographically weighted or locally estimated models. Whether locally estimated adjustments improve on a global grid in practice is an empirical question that depends on the bias--variance trade-off: local models track heterogeneity but are noisy where sales are few. We estimate adjustments at four levels of locality---a global within-municipality hedonic grid, a regional grid shrunk towards the global one, a per-valuation local regression on the candidate pool, and a non-linear gradient-boosting model---and test the trade-off directly across density strata. | |
| 15 | + | |
| 16 | +Time adjustments raise a separate issue. Price indices are routinely estimated with the full sample and applied retroactively, which leaks information from the future into every ``historical'' valuation. \citet{bailey1963} and \citet{case1989} introduced repeat-sales indices and \citet{hill2013} reviews hedonic index construction; we follow the hedonic route but re-estimate the regional monthly index every month using only prior sales, so that the time adjustment applied to a comparable on a given valuation date is one that could actually have been computed on that date. | |
| 17 | + | |
| 18 | +\subsection{Automated valuation and machine learning} | |
| 19 | + | |
| 20 | +Automated valuation models (AVMs) have moved from mass appraisal \citep{gloudemans1999} to lending and consumer applications, and the literature comparing them with traditional valuation has grown rapidly. \citet{pagourtzi2003} survey the methods; \citet{mccluskey2013} compare neural networks with hedonic models for mass appraisal; \citet{mayer2019} evaluate several machine-learning algorithms on a large Swiss dataset and find that gradient boosting and random forests dominate linear hedonics, with mixed-effects and spatial models close behind; \citet{bourassa2010} and \citet{case2004} compare spatial econometric specifications; \citet{pace2020} review the state of AVMs. A consistent finding is that tree ensembles with location features achieve median absolute errors of roughly 8--15\% on residential sales, depending on market and data richness. Comparable-sales algorithms are themselves a major class of AVM \citep{iaao2018}, and hybrid approaches that use a model to adjust comparables are common in industry but rarely evaluated in the academic literature. | |
| 21 | + | |
| 22 | +Two features of this literature motivate our design. First, machine-learning benchmarks are usually trained on random splits of the data, which overstate accuracy relative to a temporal split \citep{mayer2019}; we use strict chronological separation and quarterly retraining, which is how such models are deployed. Second, the information set is rarely held constant: AVMs often exploit variables (assessed values, listing information) that the traditional method in the same study does not. We give every method the same characteristics and the same point-in-time index, and we separately quantify the value of the assessed value as an input to both the traditional and the machine-learning methods. | |
| 23 | + | |
| 24 | +\subsection{Appraiser behaviour and the limits of simulation} | |
| 25 | + | |
| 26 | +Finally, a behavioural literature studies how appraisers actually select and adjust comparables and documents anchoring on contract prices, prior appraisals and assessed values \citep{diaz1990,diaz2001,northcraft1987}. We do not observe appraisers' files, and our results concern the predictive accuracy of \emph{rules}, not of practitioners. The distinction matters in both directions: a practitioner may do better than any rule by using information we do not observe (condition, renovations, listing history), and worse by anchoring. Our contribution is to establish what the observable record supports, so that professional rules and training can be benchmarked against it. | |
added
paper/sections/methodology.tex
+92 −0
@@ -0,0 +1,92 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Methodology} | |
| 3 | +\label{sec:methodology} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{Point-in-time backtesting} | |
| 7 | + | |
| 8 | +The core of the design is a backtest that reproduces, for each target sale $i$ with date $\tau_i$ and price $P_i$, the information set available to a valuer on $\tau_i$. Let $\mathcal{S}$ denote the clean sample. The \emph{candidate pool} of $i$ is | |
| 9 | +\begin{equation} | |
| 10 | +\mathcal{C}_i = \{\, j \in \mathcal{S} : \tau_j < \tau_i,\; \tau_i - \tau_j \le 24 \text{ months},\; d_{ij} \le 25 \text{ km},\; u_j \neq u_i \,\}, | |
| 11 | +\label{eq:pool} | |
| 12 | +\end{equation} | |
| 13 | +where $d_{ij}$ is the great-circle distance and $u_j$ the assessment unit; the last condition excludes prior sales of the subject itself, so that the method is evaluated as a comparable-sales method rather than a repeat-sales one. Every quantity entering a valuation on $\tau_i$ is computed from sales with $\tau_j < \tau_i$: the regional price index used to time-adjust comparables is re-estimated at the start of every month $m$ from all sales in months $< m$ (Section~\ref{sec:index}); adjustment coefficients and benchmark models are re-estimated at the start of every quarter $q$ from all sales in quarters $< q$ (Section~\ref{sec:config}). No configuration, hyperparameter or weight is chosen on test-period data: all are selected on the 2024 validation targets and frozen. | |
| 14 | + | |
| 15 | +A subtle but important consequence is that the price level of the target's own month is never observed. All methods, traditional and machine-learning alike, work at the level of the last fully observed month, exactly as a valuer must. This introduces a small, common and seasonal bias that we report rather than remove. | |
| 16 | + | |
| 17 | +\subsection{Point-in-time price index} | |
| 18 | +\label{sec:index} | |
| 19 | + | |
| 20 | +For each evaluation month $m$ and administrative region $r$ we estimate, on all sales of region $r$ in months $0,\dots,m-1$, the within-municipality regression | |
| 21 | +\begin{equation} | |
| 22 | +\ln P_j = \alpha_{\mu(j)} + \mathbf{x}_j'\boldsymbol{\beta} + \sum_{s<m-1} \delta_s^{(r,m)} \mathbb{1}[\text{month}_j = s] + \varepsilon_j, | |
| 23 | +\label{eq:index} | |
| 24 | +\end{equation} | |
| 25 | +where $\alpha_{\mu}$ is a municipality fixed effect, $\mathbf{x}_j$ the characteristics (log living area, log lot area, age and its square, physical link, construction type, storeys, new-build indicator) and $\delta_s^{(r,m)}$ the effect of month $s$ relative to the last observed month $m-1$. The municipality fixed effects prevent the index from being contaminated by shifts in the composition of sales between expensive and inexpensive municipalities within a region. For regions with few sales the monthly effects are shrunk towards the provincial estimate with weight $n_s/(n_s+40)$. A comparable sold in month $s$ and valued in month $m$ is time-adjusted by $\exp(-\delta_s^{(r,m)})$; comparables sold in the same month as the target are not adjusted. The index is thus a three-dimensional object (evaluation month, region, sale month), and the estimate of any historical month is allowed to change as later data arrive---as it does in reality. | |
| 26 | + | |
| 27 | +\subsection{Configurations of the sales comparison approach} | |
| 28 | +\label{sec:config} | |
| 29 | + | |
| 30 | +A configuration is a tuple $(\text{selection}, d, t, k, \text{adjustment}, \text{weighting}, \text{reconciliation})$. Given the pool $\mathcal{C}_i$, the method proceeds in four steps. | |
| 31 | + | |
| 32 | +\paragraph{Selection.} Candidates satisfying $d_{ij}\le d$ and $\tau_i-\tau_j \le t$ are ranked by a score and the $k$ best are retained. Four scores are implemented: \emph{nearest} (distance $d_{ij}$); \emph{most similar}, a physical dissimilarity $D_{ij} = \big[(\Delta\ln A/0.25)^2 + (\Delta\ln L/0.6)^2 + (\Delta\text{age}/15)^2 + \mathbb{1}[\text{link}\neq] + 0.5\,\mathbb{1}[\text{storeys}\neq]\big]^{1/2}$ in units of typical within-market standard deviations, with distance as a tie-breaker; \emph{compromise}, $D_{ij} + d_{ij}/2$; and \emph{least adjusted}, the gross hedonic adjustment $\sum_m |\hat\beta_m \Delta x_{ijm}|$, which operationalises the professional advice to prefer comparables requiring the fewest adjustments. | |
| 33 | + | |
| 34 | +\paragraph{Adjustment.} Each retained comparable's time-adjusted price $P_j^{T}$ is converted into an indication of the subject's value $\hat P_{i|j}$. Eight schemes are compared: | |
| 35 | +\begin{enumerate}[label=(\alph*)] | |
| 36 | +\item \emph{none}: $\hat P_{i|j} = P_j$ (not even time-adjusted); | |
| 37 | +\item \emph{time only}: $\hat P_{i|j} = P_j^{T}$; | |
| 38 | +\item \emph{conventional grid}: dollar adjustments in the spirit of professional practice---living area at one half of the comparable's price per square metre, lot area at one half of the comparable's implied land value per square metre (capped at $\pm 15\%$), age at $-0.5\%$ per year of difference (capped at $\pm 20\%$), and $\pm 5\%$ between detached and attached dwellings; | |
| 39 | +\item \emph{hedonic grid}: $\hat P_{i|j} = P_j^{T}\exp\!\big(\hat{\boldsymbol\beta}'(\mathbf{x}_i-\mathbf{x}_j)\big)$ with $\hat{\boldsymbol\beta}$ from a within-municipality regression on all prior sales (re-estimated quarterly); | |
| 40 | +\item \emph{regional grid}: as (d) with region-specific coefficients shrunk towards the global ones with weight $n_r/(n_r+300)$; | |
| 41 | +\item \emph{local grid}: as (d) with coefficients re-estimated on the subject's own pool within 10~km (at least 40 sales), by ridge regression penalised towards the regional grid; | |
| 42 | +\item \emph{non-linear (GBM)}: $\hat P_{i|j} = P_j^{T}\exp\!\big(f(\mathbf{z}_i)-f(\mathbf{z}_j)\big)$ where $f$ is a gradient-boosting model of the time-adjusted log price on characteristics and location (Section~\ref{sec:benchmarks}); | |
| 43 | +\item \emph{assessment ratio}: $\hat P_{i|j} = V_i \cdot P_j^{T}/V_j$ where $V$ is the assessed value, restricted to comparables in the same municipality and on the same assessment roll as the subject. | |
| 44 | +\end{enumerate} | |
| 45 | +Scheme (h) is the ratio method used by assessors and, informally, by many appraisers; it uses information (the assessed value) that schemes (a)--(g) do not, and we treat it as a distinct model in the horse race. | |
| 46 | + | |
| 47 | +\paragraph{Weighting and reconciliation.} Adjusted indications are combined by the median or the (dollar) mean, with equal weights or with weights proportional to $1/(0.25+d_{ij})$, to $\exp(-\text{months}_{ij}/6)$, to $1/(0.1+\text{gross adjustment}_{ij})$, or to their product; weighted medians are used when weights are not equal. The dispersion of the adjusted indications is recorded as a measure of uncertainty. | |
| 48 | + | |
| 49 | +\paragraph{Experiments.} Experiment E1 maps the error surface over $k \in \{1,3,5,7,10,15,20,30,50\}$, $d \in \{0.5,1,2,5,10,25\}$~km and $t \in \{3,6,12,24\}$ months, for the nearest, most-similar and compromise selections and the time-only and hedonic adjustments, with equal weights and the median (1{,}296 configurations). Experiment E2 is the full factorial of selection (4) $\times$ adjustment (8) $\times$ weighting (5) $\times$ reconciliation (2) at three anchors $(k,d,t) = (3, 2\text{ km}, 6\text{ mo})$, $(5, 5\text{ km}, 12\text{ mo})$ and $(10, 10\text{ km}, 24\text{ mo})$ (960 configurations). All \nConfigs{} configurations are evaluated on every target. | |
| 50 | + | |
| 51 | +\subsection{Benchmark models} | |
| 52 | +\label{sec:benchmarks} | |
| 53 | + | |
| 54 | +Thirteen named models are compared (Table~\ref{tab:models}). M1--M8 and M13 are configurations of the sales comparison approach that correspond to recognisable professional practices. M10 is a direct global hedonic model: the within-municipality coefficients augmented with region, county (MRC), municipality and neighbourhood-unit effects estimated as shrunken residual means, plus a median correction. M11 is a gradient-boosting model (LightGBM, 127 leaves, learning rate 0.05, L1 objective so that it targets the conditional median, number of trees chosen by early stopping on the last 10\% of the training period) of the time-adjusted log price on characteristics, coordinates, region, construction type and the month of sale; M11a excludes and M11b includes the assessed value, its land share and its roll vintage. Both are retrained at the start of every quarter on all prior sales. M9 is an \emph{optimised reconciliation}: a convex combination, estimated by non-negative least squares on the validation targets, of the seven comparable-based indications at the $(5,5,12)$ anchor. M12 is a hybrid combining M11b with the GBM-adjusted and hedonic-adjusted comparable estimates, with weights estimated the same way. M0 is the single best of the \nConfigs{} configurations on the validation targets, among those that return a value for at least 97\% of targets. | |
| 55 | + | |
| 56 | +\begin{table}[H] | |
| 57 | +\centering | |
| 58 | +\caption{Models in the horse race} | |
| 59 | +\label{tab:models} | |
| 60 | +\small | |
| 61 | +\begin{tabular}{ll} | |
| 62 | +\toprule | |
| 63 | +Model & Definition \\ | |
| 64 | +\midrule | |
| 65 | +M1 & Median of the 3 nearest sales ($\le 2$ km, $\le 6$ months), time adjustment only \\ | |
| 66 | +M2 & Mean of the 5 nearest sales ($\le 5$ km, $\le 12$ months), time adjustment only \\ | |
| 67 | +M3 & Median of the 5 most physically similar sales ($\le 5$ km, $\le 12$ months), time adjustment only \\ | |
| 68 | +M4 & 5 nearest sales, distance-weighted mean, time adjustment only \\ | |
| 69 | +M5 & 5 nearest sales, recency-weighted mean, time adjustment only \\ | |
| 70 | +M6 & 5 nearest sales, conventional adjustment grid, median \\ | |
| 71 | +M7 & 5 nearest sales, hedonic adjustment grid, median \\ | |
| 72 | +M8a & 5 nearest sales, locally re-estimated grid, median \\ | |
| 73 | +M8b & 5 nearest sales, non-linear (GBM) adjustments, median \\ | |
| 74 | +M9 & Optimised reconciliation: NNLS combination of seven comparable-based indications (weights fitted on 2024) \\ | |
| 75 | +M10 & Direct global hedonic model with region, MRC, municipality and neighbourhood effects \\ | |
| 76 | +M11a / M11b & Gradient boosting without / with the assessed value, retrained quarterly \\ | |
| 77 | +M12 & Hybrid: NNLS combination of M11b and GBM- and hedonic-adjusted comparables (weights fitted on 2024) \\ | |
| 78 | +M13 & Assessment ratio: $V_i \times$ median price-to-assessment ratio of the 5 nearest same-roll sales \\ | |
| 79 | +M0 & Best of all \nConfigs{} configurations on 2024 (coverage $\ge 97\%$), frozen \\ | |
| 80 | +\bottomrule | |
| 81 | +\end{tabular} | |
| 82 | +\end{table} | |
| 83 | + | |
| 84 | +\subsection{Accuracy measures and inference} | |
| 85 | + | |
| 86 | +For each target and method we compute the absolute percentage error $\mathrm{APE}_i = |\hat P_i - P_i|/P_i$. We report the median APE (MdAPE), the mean APE (MAPE), the median signed error (bias), the shares of estimates within $\pm5$, $\pm10$ and $\pm20\%$ of the price, and the \emph{coverage}, the share of targets for which the method returns a value. Coverage is reported because a restrictive configuration does not fail on difficult targets---it abstains---and an MdAPE computed on the targets it covers would flatter it. Configuration selection on the validation set is restricted to configurations with coverage of at least 97\%. | |
| 87 | + | |
| 88 | +Because all methods are evaluated on the same targets, comparisons are paired. For a pair of methods $(A,B)$ we report the difference in MdAPE and in MAPE together with 95\% confidence intervals from a cluster bootstrap \citep{cameron2008} that resamples municipality--quarter cells with replacement, thereby preserving the spatial and temporal dependence of valuation errors within local markets. We also report the share of targets on which $A$ is more accurate than $B$. | |
| 89 | + | |
| 90 | +\subsection{Hypotheses} | |
| 91 | + | |
| 92 | +Five hypotheses from the professional and academic debate are tested directly. \textbf{H1} (similarity versus proximity): a comparable that is very close but physically dissimilar is less informative than one that is farther away but similar. \textbf{H2} (number of comparables): adding comparables improves accuracy up to a threshold beyond which heterogeneity dominates, and the threshold rises with market density. \textbf{H3} (local adjustments): locally estimated adjustments outperform fixed grids but become unstable in small samples. \textbf{H4} (weighting and liquidity): the optimal weighting scheme depends on the liquidity of the submarket. \textbf{H5} (transparency): simple, transparent methods rival machine learning in homogeneous submarkets even if the latter dominates overall. | |
added
paper/sections/results.tex
+232 −0
@@ -0,0 +1,232 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Results} | |
| 3 | +\label{sec:results} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +All results in this section are computed on the \nTargetsTest{} test targets of January 2025 to September 2026, with every configuration, weight and model frozen on the basis of the \nTargetsVal{} validation targets of 2024. Validation-period figures are reported alongside to document the stability of the rankings. | |
| 7 | + | |
| 8 | +\subsection{How many comparables, how far, how far back?} | |
| 9 | +\label{sec:surface} | |
| 10 | + | |
| 11 | +Figure~\ref{fig:surface_k} plots the out-of-sample MdAPE against the number of comparables $k$ for each search radius, with a twelve-month window and selection by distance, without physical adjustment (left) and with the hedonic grid (right). Figure~\ref{fig:surface_dt} shows the best attainable MdAPE for each radius--window pair, and Table~\ref{tab:surface} reports the full surface for the hedonic grid. | |
| 12 | + | |
| 13 | +\begin{figure}[H] | |
| 14 | +\centering | |
| 15 | +\includegraphics[width=\textwidth]{f1_mdape_vs_k.pdf} | |
| 16 | +\caption{Out-of-sample MdAPE (2025--2026) by number of comparables and search radius; twelve-month window, selection by distance, median reconciliation. Left: time adjustment only. Right: hedonic adjustment grid.} | |
| 17 | +\label{fig:surface_k} | |
| 18 | +\end{figure} | |
| 19 | + | |
| 20 | +\begin{figure}[H] | |
| 21 | +\centering | |
| 22 | +\includegraphics[width=0.9\textwidth]{f2_surface_d_T.pdf} | |
| 23 | +\caption{Best attainable out-of-sample MdAPE (over $k$) for each search radius and look-back window; selection by distance.} | |
| 24 | +\label{fig:surface_dt} | |
| 25 | +\end{figure} | |
| 26 | + | |
| 27 | +\begin{table}[H] | |
| 28 | +\centering | |
| 29 | +\caption{Error surface $\mathrm{Error}(k,d)$: out-of-sample MdAPE (\%), selection by distance, hedonic grid, twelve-month window} | |
| 30 | +\label{tab:surface} | |
| 31 | +\input{tables/surface_geo_hedo} | |
| 32 | +\begin{tablenotes}\footnotesize | |
| 33 | +\item Note: MdAPE on the 2025--2026 test targets for which the configuration returns a value. Coverage falls below 90\% for radii of 1~km and less in low-density markets; see Table~\ref{tab:surface_cov} in the Appendix. | |
| 34 | +\end{tablenotes} | |
| 35 | +\end{table} | |
| 36 | + | |
| 37 | +Three regularities emerge. First, without adjustment, accuracy improves with $k$ only up to a handful of comparables and then flattens or deteriorates: beyond the nearest few, additional sales are increasingly dissimilar and the median of their time-adjusted prices drifts away from the subject's value. Second, with the hedonic grid the picture changes qualitatively. The error falls from \mdapeKone{} with a single comparable to \mdapeKthree{} with three, \mdapeKfive{} with five, \mdapeKten{} with ten, and reaches its minimum of \mdapeKopt{} at $k=\kOpt{}$ before rising slowly (\mdapeKfifty{} at $k=50$). Adjustment converts dissimilar sales into usable evidence, so that the variance reduction from averaging more of them dominates their heterogeneity over a much wider range. Third, the search radius and the window should be wide. The three-month window is dominated by the twelve- and twenty-four-month windows at every radius once comparables are time-adjusted, and radii of 0.5--1~km are dominated by 5--25~km once they are physically adjusted. The conventional combination---three comparables, within a kilometre, within three months---yields an MdAPE of \narrowMdape{} with the hedonic grid and returns a value for only \narrowCov{} of targets; the best configuration of the surface chosen on the 2024 validation set (\bestSel{}, $k=\bestK$, $d\le\bestD$~km, $t=\bestT$ months, \bestAdj{}) yields \bestValMdape{} on validation and \bestTestMdape{} on the test set. | |
| 38 | + | |
| 39 | +Table~\ref{tab:h2} tests H2 by stratifying the $k$-curve by market density. The optimal number of comparables rises monotonically with density: where fewer than five sales occurred within 2~km in the past year, the optimum is reached with a handful of comparables and additional ones are increasingly distant; where two hundred or more occurred, the error keeps falling well beyond twenty. The threshold at which heterogeneity starts to dominate is thus not a constant of the method but a property of the local market. | |
| 40 | + | |
| 41 | +\begin{table}[H] | |
| 42 | +\centering | |
| 43 | +\caption{Optimal number of comparables by market density (H2): out-of-sample MdAPE (\%)} | |
| 44 | +\label{tab:h2} | |
| 45 | +\input{tables/h2} | |
| 46 | +\begin{tablenotes}\footnotesize | |
| 47 | +\item Note: selection by distance, $d\le 10$~km, twelve-month window, hedonic grid, median. Density is the number of sample sales within 2~km in the twelve months preceding the target's date. The minimum of each row is in bold. | |
| 48 | +\end{tablenotes} | |
| 49 | +\end{table} | |
| 50 | + | |
| 51 | +\subsection{Which adjustments are warranted?} | |
| 52 | +\label{sec:adjustments} | |
| 53 | + | |
| 54 | +Table~\ref{tab:adjustments} and Figure~\ref{fig:adjustments} compare the eight adjustment schemes at the $(5, 5\text{ km}, 12\text{ mo})$ anchor with selection by distance, equal weights and the median. The time adjustment alone is worth \gainTime{} points of MdAPE relative to raw prices. Any physical grid is worth a further \gainGridLo{} to \gainGridHi{} points. The stylised conventional grid---which uses no estimation whatsoever---recovers most of the gain of the hedonic grid (\convMdape{} against \mdapeMseven{}), consistent with \citet{colwell1983}'s observation that the grid is a regression with pre-set coefficients that need only be roughly right. The refinements add little: the regional grid is indistinguishable from the global one, the local grid is \localVsHedo{} (\localMdape{}), and the non-linear GBM adjustments improve on the hedonic grid by \gbmVsHedo{} points. The assessment-ratio adjustment stands apart: with \mdapeMthirteen{} it is the most accurate single scheme, but it returns a value for only \covMthirteen{} of targets because it requires comparables on the same assessment roll. | |
| 55 | + | |
| 56 | +\begin{table}[H] | |
| 57 | +\centering | |
| 58 | +\caption{Adjustment schemes at the $(5,\,5\text{ km},\,12\text{ months})$ anchor} | |
| 59 | +\label{tab:adjustments} | |
| 60 | +\input{tables/adjustments} | |
| 61 | +\begin{tablenotes}\footnotesize | |
| 62 | +\item Note: five nearest sales, equal weights, median. MdAPE, MAPE, bias and the share within $\pm10\%$ in per cent; coverage is the share of targets for which the scheme returns a value. | |
| 63 | +\end{tablenotes} | |
| 64 | +\end{table} | |
| 65 | + | |
| 66 | +\begin{figure}[H] | |
| 67 | +\centering | |
| 68 | +\includegraphics[width=0.9\textwidth]{f3_ajustements.pdf} | |
| 69 | +\caption{MdAPE by adjustment scheme, validation (2024) and test (2025--2026); five nearest sales, $\le 5$~km, $\le 12$ months, median.} | |
| 70 | +\label{fig:adjustments} | |
| 71 | +\end{figure} | |
| 72 | + | |
| 73 | +Table~\ref{tab:h3} examines H3 by crossing the adjustment schemes with market density at two anchors. The local grid tracks the hedonic grid closely where sales are abundant and falls behind where they are scarce, as the bias--variance argument predicts; at the tight $(3, 2\text{ km}, 6\text{ mo})$ anchor, where the local regression is estimated on the smallest pools, the gap widens. The non-linear GBM adjustments are the most accurate scheme in dense markets and lose their advantage in sparse ones. The practical implication is that adjustment grids should be estimated on the largest pool that is homogeneous in implicit prices---in these data, the province with municipality fixed effects---rather than on the handful of sales around the subject. | |
| 74 | + | |
| 75 | +\begin{table}[H] | |
| 76 | +\centering | |
| 77 | +\caption{Adjustment schemes by market density (H3): out-of-sample MdAPE (\%)} | |
| 78 | +\label{tab:h3} | |
| 79 | +\input{tables/h3} | |
| 80 | +\begin{tablenotes}\footnotesize | |
| 81 | +\item Note: selection by distance, equal weights, median. Upper panel: $(5, 5\text{ km}, 12\text{ mo})$; lower panel: $(3, 2\text{ km}, 6\text{ mo})$. | |
| 82 | +\end{tablenotes} | |
| 83 | +\end{table} | |
| 84 | + | |
| 85 | +The estimated hedonic grid itself is of independent interest (Table~\ref{tab:grid}). Within a municipality, the elasticity of price with respect to living area is \betaFloor{}, that with respect to lot area is \betaLand{}, and depreciation is \betaAge{} per year at the mean age with a convex profile; semi-detached and row dwellings sell at discounts of \betaSemi{} and \betaRow{} relative to detached dwellings of the same size. These coefficients are stable across the eleven quarterly re-estimations to the second decimal. | |
| 86 | + | |
| 87 | +\begin{table}[H] | |
| 88 | +\centering | |
| 89 | +\caption{Hedonic adjustment grid (within-municipality coefficients, log price)} | |
| 90 | +\label{tab:grid} | |
| 91 | +\input{tables/grid} | |
| 92 | +\begin{tablenotes}\footnotesize | |
| 93 | +\item Note: coefficients of the within-municipality regression of the time-adjusted log price on characteristics, estimated on sales before 2025 (test-period grid). Living and lot areas in logs; age in years; age$^2$/100. | |
| 94 | +\end{tablenotes} | |
| 95 | +\end{table} | |
| 96 | + | |
| 97 | +\subsection{How should comparables be selected? (H1)} | |
| 98 | +\label{sec:selection} | |
| 99 | + | |
| 100 | +Table~\ref{tab:selection} compares the four selection criteria at the $(5,5,12)$ anchor with the hedonic grid. Selecting by physical similarity rather than by distance lowers the MdAPE by \gainSelHedo{} points when comparables are adjusted, and the compromise criterion is close to the similarity criterion. The ``least adjusted'' criterion, which embodies the professional advice to prefer comparables needing few adjustments, performs like the similarity criterion for which it is a proxy. | |
| 101 | + | |
| 102 | +\begin{table}[H] | |
| 103 | +\centering | |
| 104 | +\caption{Selection criteria at the $(5,\,5\text{ km},\,12\text{ months})$ anchor, hedonic grid} | |
| 105 | +\label{tab:selection} | |
| 106 | +\input{tables/selection} | |
| 107 | +\end{table} | |
| 108 | + | |
| 109 | +The direct test of H1 is starker. Without any physical adjustment, the five nearest sales within one kilometre yield an MdAPE of \hOneGeoDone{} (coverage \hOneGeoDoneCov{}); the five most similar sales within ten kilometres yield \hOneSimDten{}. The nearby-but-dissimilar comparable is worse than the distant-but-similar one, by a wide margin. The reason is that, in these data, the physical characteristics explain a larger share of the within-market variance of prices than micro-location does once the municipality is fixed. With the hedonic grid the two criteria converge (\hOneGeoDtenHedo{} against \hOneSimDtenHedo{}), because adjustment does for the nearest sales what selection does for the most similar; the compromise criterion, which uses both, is best (\hOneMixDtenHedo{}). | |
| 110 | + | |
| 111 | +\subsection{Weighting and reconciliation (H4)} | |
| 112 | +\label{sec:weighting} | |
| 113 | + | |
| 114 | +Table~\ref{tab:weighting} reports the ten weighting--reconciliation combinations at the $(5,5,12)$ anchor with the hedonic grid, and Table~\ref{tab:h4} stratifies them by density. The range of the MdAPE across all ten combinations is \wrRange{} points. The median is marginally more robust than the mean, as expected from the heavy tails of adjusted indications; distance and similarity weights help slightly in dense markets, where the nearest and most similar comparables are also the most informative, and are neutral or slightly harmful in sparse ones, where they concentrate weight on one or two sales. H4 is thus supported in direction but the effects are an order of magnitude smaller than those of selection and adjustment. The marginal effects over the whole E2 design (Table~\ref{tab:marginal}) make the hierarchy explicit: the adjustment factor spans \margAdjLo{}--\margAdjHi{} points of MdAPE, selection \margSelLo{}--\margSelHi{}, weighting \margWLo{}--\margWHi{}, and reconciliation \margReconMedian{} (median) against \margReconMean{} (mean). | |
| 115 | + | |
| 116 | +\begin{table}[H] | |
| 117 | +\centering | |
| 118 | +\caption{Weighting and reconciliation at the $(5,\,5\text{ km},\,12\text{ months})$ anchor, hedonic grid} | |
| 119 | +\label{tab:weighting} | |
| 120 | +\input{tables/weighting} | |
| 121 | +\end{table} | |
| 122 | + | |
| 123 | +\begin{table}[H] | |
| 124 | +\centering | |
| 125 | +\caption{Weighting by market density (H4): out-of-sample MdAPE (\%)} | |
| 126 | +\label{tab:h4} | |
| 127 | +\input{tables/h4} | |
| 128 | +\end{table} | |
| 129 | + | |
| 130 | +\begin{table}[H] | |
| 131 | +\centering | |
| 132 | +\caption{Marginal effects of the four rule factors in the E2 design} | |
| 133 | +\label{tab:marginal} | |
| 134 | +\input{tables/marginal} | |
| 135 | +\begin{tablenotes}\footnotesize | |
| 136 | +\item Note: mean out-of-sample MdAPE (\%) of all E2 configurations sharing the given level of the factor, averaged over the other factors and the three anchors. | |
| 137 | +\end{tablenotes} | |
| 138 | +\end{table} | |
| 139 | + | |
| 140 | +\subsection{The horse race} | |
| 141 | +\label{sec:horserace} | |
| 142 | + | |
| 143 | +Table~\ref{tab:competition} and Figure~\ref{fig:competition} report the thirteen named models and the empirically optimised configurations. The ordering is informative at every level. The traditional practices M1--M5---a few nearest sales, time-adjusted, weighted in one way or another---cluster at \mdapeMoneRange{} of MdAPE, with fewer than a third of estimates within $\pm10\%$ of the price. Adding a grid (M6, M7) or selecting by similarity (M3) brings the error into the \mdapeGridRange{} range. The assessment ratio (M13) reaches \mdapeMthirteen{} on the targets it covers. The best single transparent configuration frozen on 2024 (M0: \mZeroDesc) reaches \mdapeMzero{} with \pTenMzero{} of estimates within $\pm10\%$, and the optimised reconciliation M9 \mdapeMnine{}. Among the external benchmarks, the direct hedonic model M10 is the least accurate (\mdapeMten{}): a global model with municipality effects cannot match the local information of comparables. The gradient-boosting model without the assessed value (M11a, \mdapeMelevenA{}) does no better than a well-configured comparables method; with the assessed value (M11b) it reaches \mdapeMelevenB{} and \pTenMelevenB{} within $\pm10\%$, and the hybrid M12 \mdapeMtwelve{}. | |
| 144 | + | |
| 145 | +\begin{table}[H] | |
| 146 | +\centering | |
| 147 | +\caption{Horse race: out-of-sample accuracy, 2025--2026} | |
| 148 | +\label{tab:competition} | |
| 149 | +\footnotesize | |
| 150 | +\input{tables/competition} | |
| 151 | +\begin{tablenotes}\footnotesize | |
| 152 | +\item Note: MdAPE, MAPE, median signed error and the shares within $\pm5$, $\pm10$ and $\pm20\%$ in per cent, on the test targets for which the model returns a value; coverage is the share of such targets. The validation column gives the MdAPE on the 2024 targets. Weights of M9 and M12 and the configuration of M0 are chosen on 2024 only. | |
| 153 | +\end{tablenotes} | |
| 154 | +\end{table} | |
| 155 | + | |
| 156 | +\begin{figure}[H] | |
| 157 | +\centering | |
| 158 | +\includegraphics[width=0.95\textwidth]{f4_competition.pdf} | |
| 159 | +\caption{Out-of-sample MdAPE by model, 2025--2026. Orange: configurations of the sales comparison approach; green: empirically optimised configurations; blue: external benchmarks.} | |
| 160 | +\label{fig:competition} | |
| 161 | +\end{figure} | |
| 162 | + | |
| 163 | +Table~\ref{tab:pairs} gives the paired differences with cluster-bootstrap confidence intervals for the comparisons that matter. The gain from the hedonic grid over time-only adjustment (M7 $-$ M2), the gain from similarity selection (M3 $-$ M2), and the gain of M11b over M0 are all precisely estimated and far from zero. The difference between the conventional and hedonic grids (M7 $-$ M6) is small; so is that between local and global grids. The hybrid M12 improves significantly on M11b, which shows that comparables carry local information that a quarterly-retrained tree model with coordinates does not fully extract. | |
| 164 | + | |
| 165 | +\begin{table}[H] | |
| 166 | +\centering | |
| 167 | +\caption{Paired differences in accuracy with cluster-bootstrap 95\% confidence intervals} | |
| 168 | +\label{tab:pairs} | |
| 169 | +\footnotesize | |
| 170 | +\input{tables/pairs} | |
| 171 | +\begin{tablenotes}\footnotesize | |
| 172 | +\item Note: $\Delta$ is the difference $A-B$ in MdAPE and in MAPE (percentage points) on the common test targets; a negative value means $A$ is more accurate. Confidence intervals from 400 (MAPE) and 150 (MdAPE) cluster-bootstrap replications over municipality--quarter cells. The last column is the share of targets on which $A$ has the smaller error. | |
| 173 | +\end{tablenotes} | |
| 174 | +\end{table} | |
| 175 | + | |
| 176 | +Two findings deserve emphasis. The first is the \emph{cost of transparency}: \transparencyCost{} points of MdAPE separate the best frozen configuration of the sales comparison approach from the best opaque model with the same information plus the assessed value, and \transparencyCostNoRole{} points separate it from the same model without the assessed value. Every estimate of M0 is the median of \bestK{} identified sales adjusted by published coefficients; every estimate of M11b is the output of a thousand trees. The second is the \emph{value of the assessed value}. The ratio method M13 and the GBM with roll information M11b dominate their counterparts without it by \roleGainRatio{} and \roleGainGbm{} points respectively. The assessed value summarises characteristics that the roll extract does not expose to us---quality, condition, outbuildings---and a local price level that the assessor has already estimated; it is, in effect, a professional valuation available for every property, and the evidence says it should be used. | |
| 177 | + | |
| 178 | +\subsection{Where does the method hold, and where does it fail?} | |
| 179 | +\label{sec:strata} | |
| 180 | + | |
| 181 | +Table~\ref{tab:density} and Figure~\ref{fig:density} report accuracy by market density for the main models. The dependence is steep and common to all methods: the MdAPE of the hedonic-grid method M7 is \densHighMseven{} where two hundred or more sales occurred within 2~km in the past year and \densLowMseven{} where fewer than five did. The gap between the best transparent method and the best opaque one is widest in the sparsest markets, where the tree model's ability to borrow strength from a wider area is most valuable, and narrowest in the densest ones. Tables~\ref{tab:regions}--\ref{tab:price} in the Appendix report the same comparison by region, price quintile, age and physical link; accuracy is lowest in the Montréal region and for the highest price quintile, where the unobserved characteristics (quality, lot features, views) carry the largest share of value. | |
| 182 | + | |
| 183 | +\begin{table}[H] | |
| 184 | +\centering | |
| 185 | +\caption{Out-of-sample MdAPE (\%) by market density} | |
| 186 | +\label{tab:density} | |
| 187 | +\input{tables/density} | |
| 188 | +\end{table} | |
| 189 | + | |
| 190 | +\begin{figure}[H] | |
| 191 | +\centering | |
| 192 | +\includegraphics[width=0.85\textwidth]{f5_densite.pdf} | |
| 193 | +\caption{Out-of-sample MdAPE by market density (sales within 2~km in the preceding twelve months), selected models.} | |
| 194 | +\label{fig:density} | |
| 195 | +\end{figure} | |
| 196 | + | |
| 197 | +Table~\ref{tab:h5} addresses H5 by splitting the test targets into terciles of submarket heterogeneity, measured by the dispersion of the adjusted indications of the ten nearest comparables within 10~km and 24 months---a quantity available at valuation time. In the homogeneous tercile the transparent configuration M0 is within \hFiveGapHomog{} points of M11b; in the heterogeneous tercile the gap is \hFiveGapHeter{} points. Simple, explainable rules rival machine learning precisely where the market is informative enough for a few well-chosen sales to pin down value; machine learning earns its keep where the market is thin or heterogeneous. | |
| 198 | + | |
| 199 | +\begin{table}[H] | |
| 200 | +\centering | |
| 201 | +\caption{Accuracy by submarket heterogeneity (H5): out-of-sample MdAPE (\%)} | |
| 202 | +\label{tab:h5} | |
| 203 | +\input{tables/h5} | |
| 204 | +\begin{tablenotes}\footnotesize | |
| 205 | +\item Note: terciles of the standard deviation of the log adjusted indications of the ten nearest sales within 10~km and 24 months (hedonic grid), computed on the test targets. | |
| 206 | +\end{tablenotes} | |
| 207 | +\end{table} | |
| 208 | + | |
| 209 | +\subsection{Value intervals} | |
| 210 | +\label{sec:intervals} | |
| 211 | + | |
| 212 | +The dispersion of the adjusted indications provides an interval of value. Table~\ref{tab:intervals} reports, for the time-only, hedonic and GBM adjustments at two anchors, the coverage of the interval $\hat P \exp(\pm z\,\hat\sigma)$ with $z=1.28$ (nominal 80\% under normality) and with $z$ calibrated on the validation targets to achieve 80\% coverage, together with the median width of the calibrated interval. The nominal intervals under-cover: the dispersion of five or ten adjusted sales understates the uncertainty of the estimate, because the comparables share unobserved characteristics with each other but not with the subject. Calibrated on 2024, the intervals achieve close to their target coverage in 2025--2026 at a median width of \ivWidth{} of value, which gives the method a usable, honest measure of its own uncertainty. | |
| 213 | + | |
| 214 | +\begin{table}[H] | |
| 215 | +\centering | |
| 216 | +\caption{Coverage of value intervals derived from the dispersion of adjusted indications} | |
| 217 | +\label{tab:intervals} | |
| 218 | +\input{tables/intervals} | |
| 219 | +\end{table} | |
| 220 | + | |
| 221 | +\subsection{Towards an empirical comparable-recommendation system} | |
| 222 | +\label{sec:recommend} | |
| 223 | + | |
| 224 | +The preceding results suggest that the best rule is conditional on the market. Table~\ref{tab:cond} implements the simplest version of a recommendation system: for each density stratum, the configuration (among all \nConfigs{} that return a value for at least 95\% of the stratum's validation targets) with the lowest MdAPE on 2024 is selected and applied to the 2025--2026 targets of that stratum. The recommended rules differ systematically: sparse markets call for the most similar sales over a wide radius and a long window, dense markets for more comparables over a shorter radius. Together, the density-conditional rules achieve an out-of-sample MdAPE of \condMdape{} with \condPten{} of estimates within $\pm10\%$ and \condCov{} coverage, against \mdapeMzero{} for the best single rule. | |
| 225 | + | |
| 226 | +\begin{table}[H] | |
| 227 | +\centering | |
| 228 | +\caption{Density-conditional rules chosen on 2024 and evaluated on 2025--2026} | |
| 229 | +\label{tab:cond} | |
| 230 | +\footnotesize | |
| 231 | +\input{tables/cond_rules} | |
| 232 | +\end{table} | |
added
paper/sections/robustness.tex
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +\section{Robustness} | |
| 3 | +\label{sec:robustness} | |
| 4 | +% ============================================================================ | |
| 5 | + | |
| 6 | +\subsection{Stability over time} | |
| 7 | + | |
| 8 | +Table~\ref{tab:stability} reports the MdAPE and bias of the main models by quarter of the test period. The rankings are stable quarter by quarter, and the validation-period (2024) and test-period (2025--2026) MdAPEs of every configuration in Tables~\ref{tab:adjustments}--\ref{tab:competition} differ by less than a point, which indicates that the rules selected on 2024 did not overfit it. All methods display a common seasonal bias: estimates are too low in the first quarter and too high in the third. This is the signature of the point-in-time constraint---every method works at the level of the previous month, and Quebec prices rise into the spring and soften after the summer---and of the fact that the price level of the valuation month is not knowable on the valuation date. It is a property of the valuation problem, not of any method, and a seasonal adjustment estimated on prior years would remove most of it. | |
| 9 | + | |
| 10 | +\begin{table}[H] | |
| 11 | +\centering | |
| 12 | +\caption{Stability over the test period: MdAPE and bias (\%) by quarter} | |
| 13 | +\label{tab:stability} | |
| 14 | +\footnotesize | |
| 15 | +\input{tables/stability} | |
| 16 | +\end{table} | |
| 17 | + | |
| 18 | +\subsection{Coverage and the common-support comparison} | |
| 19 | + | |
| 20 | +Configurations with narrow radii or short windows abstain on a substantial share of targets, and their MdAPE on the targets they cover is not comparable to that of configurations that cover all targets. Table~\ref{tab:surface_cov} in the Appendix reports the coverage of every point of the error surface. Our configuration selection enforces 97\% coverage; relaxing the constraint selects a narrower configuration (\mZeroUncDesc) with a lower MdAPE on its own support (\mZeroUncMdape{}) but \mZeroUncCov{} coverage. The conclusions of Section~\ref{sec:surface} hold on the common support of all configurations with at least 97\% coverage. | |
| 21 | + | |
| 22 | +\subsection{Exclusion rules} | |
| 23 | + | |
| 24 | +The exclusion of extreme price-to-assessment ratios (rule R8) is the only rule that removes a non-trivial share of sales on a criterion related to price. Including the excluded sales as targets (but not as comparables, as an appraiser would not use them) raises the MdAPE of every method by a similar amount and leaves the rankings unchanged; the excluded sales are, by construction, those on which no method based on market evidence can succeed. Tightening the rule to $[0.5, 2]$ lowers all MdAPEs by a few tenths of a point without affecting any comparison. | |
| 25 | + | |
| 26 | +\subsection{Alternative accuracy measures} | |
| 27 | + | |
| 28 | +The MdAPE is our headline measure because the distribution of valuation errors is heavy-tailed and the median is the appraiser's natural loss. Table~\ref{tab:competition} shows that the MAPE and the shares within $\pm5$, $\pm10$ and $\pm20\%$ rank the models identically; the paired bootstrap on MAPE differences in Table~\ref{tab:pairs} yields the same conclusions as on MdAPE differences. Measuring errors in logs rather than in percentages, or symmetrising the APE by dividing by the average of estimate and price, changes no ranking. | |
| 29 | + | |
| 30 | +\subsection{Index construction} | |
| 31 | + | |
| 32 | +The point-in-time index uses an expanding window of all prior sales and municipality fixed effects. A 36-month rolling window produces an index that is noisier in small regions and leaves the comparable-based results unchanged to the first decimal, because comparables are at most 24 months old and their time adjustments are small. Omitting the municipality fixed effects biases the index downward in 2025--2026, when the composition of sales shifted towards less expensive municipalities, and produces a negative bias in the global models (M10, M11) that the comparable-based methods, which use only recent local sales, do not share. | |
| 33 | + | |
| 34 | +\subsection{Benchmark specification} | |
| 35 | + | |
| 36 | +The gradient-boosting benchmarks are trained with an L1 objective so that they estimate the conditional median, which is the quantity the MdAPE rewards; the same models trained with a squared-error objective have a median bias of about $-5\%$, because the distribution of log prices conditional on characteristics is left-skewed, and a correspondingly higher MdAPE. The quarterly retraining and the month-of-sale feature matter for the level of the benchmarks over a 21-month test period but not for the rankings. The hybrid weights of M12 and the reconciliation weights of M9 are convex combinations fitted on 2024 and applied unchanged; refitting them on each quarter of the test period, which a deployed system would do, improves M12 by a few tenths of a point. | |
added
paper/sections/titlepage.tex
+77 −0
@@ -0,0 +1,77 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Title Page | |
| 3 | +% ============================================================================ | |
| 4 | +\thispagestyle{empty} | |
| 5 | + | |
| 6 | +\begin{center} | |
| 7 | + | |
| 8 | +% --- Logo --- | |
| 9 | +\includegraphics[width=3.5cm]{uq_logo.jpg} | |
| 10 | + | |
| 11 | +\vspace{0.6cm} | |
| 12 | + | |
| 13 | +{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm] | |
| 14 | +{\footnotesize\textsc{Département des sciences administratives}} | |
| 15 | + | |
| 16 | +\vspace{0.8cm} | |
| 17 | + | |
| 18 | +{\footnotesize\textsc{Working Paper No.~\WPnumber}} | |
| 19 | + | |
| 20 | +\vspace{1.2cm} | |
| 21 | + | |
| 22 | +% --- Title --- | |
| 23 | +{\LARGE\bfseries \WPtitle\par} | |
| 24 | + | |
| 25 | +\ifx\WPsubtitle\empty\else | |
| 26 | + \vspace{0.3cm} | |
| 27 | + {\large\itshape \WPsubtitle\par} | |
| 28 | +\fi | |
| 29 | + | |
| 30 | +\vspace{1.2cm} | |
| 31 | + | |
| 32 | +% --- Author --- | |
| 33 | +{\large \WPauthor}\\[0.3cm] | |
| 34 | +{\normalsize \WPaffiliation}\\[0.15cm] | |
| 35 | +{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm] | |
| 36 | +{\small \WPaddress} | |
| 37 | + | |
| 38 | +\vspace{0.8cm} | |
| 39 | + | |
| 40 | +% --- Date & Version --- | |
| 41 | +{\normalsize \WPdate}\\[0.1cm] | |
| 42 | +{\small Version~\WPversion} | |
| 43 | + | |
| 44 | +\end{center} | |
| 45 | + | |
| 46 | +\vfill | |
| 47 | + | |
| 48 | +\newpage | |
| 49 | + | |
| 50 | +% ============================================================================ | |
| 51 | +% Abstract Page | |
| 52 | +% ============================================================================ | |
| 53 | +\thispagestyle{empty} | |
| 54 | + | |
| 55 | +\vspace*{1cm} | |
| 56 | + | |
| 57 | +\noindent\rule{\textwidth}{0.4pt} | |
| 58 | +\vspace{0.3cm} | |
| 59 | + | |
| 60 | +\noindent\textbf{Abstract} | |
| 61 | + | |
| 62 | +\vspace{0.15cm} | |
| 63 | + | |
| 64 | +\noindent\WPabstract | |
| 65 | + | |
| 66 | +\vspace{0.4cm} | |
| 67 | + | |
| 68 | +\noindent\textbf{Keywords:} \WPkeywords | |
| 69 | + | |
| 70 | +\vspace{0.15cm} | |
| 71 | + | |
| 72 | +\noindent\textbf{JEL Classification:} \WPjel | |
| 73 | + | |
| 74 | +\vspace{0.3cm} | |
| 75 | +\noindent\rule{\textwidth}{0.4pt} | |
| 76 | + | |
| 77 | +\newpage | |
added
paper/tables/adjustments.tex
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +\begin{tabular}{lrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Adjustment & MdAPE 2024 & MdAPE 2025--26 & MAPE & Bias & Within $\pm10\%$ & Coverage \\ | |
| 4 | +\midrule | |
| 5 | +none & 18.6 & 18.2 & 30.8 & -4.1 & 30.2 & 99.7 \\ | |
| 6 | +time only & 18.3 & 18.2 & 31.7 & -0.7 & 29.7 & 99.7 \\ | |
| 7 | +conventional grid & 14.5 & 13.8 & 26.8 & 0.6 & 39.1 & 99.7 \\ | |
| 8 | +hedonic grid & 14.0 & 13.5 & 25.4 & -0.7 & 39.4 & 99.7 \\ | |
| 9 | +regional grid & 14.2 & 13.3 & 25.4 & -0.4 & 39.4 & 99.7 \\ | |
| 10 | +local grid & 15.1 & 14.1 & 26.3 & -0.2 & 37.0 & 99.7 \\ | |
| 11 | +GBM adjustments & 14.0 & 12.4 & 24.9 & -0.9 & 42.5 & 99.7 \\ | |
| 12 | +assessment ratio & 12.0 & 11.6 & 22.8 & 0.2 & 40.4 & 89.7 \\ | |
| 13 | +\bottomrule | |
| 14 | +\end{tabular} | |
added
paper/tables/competition.tex
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Model & MdAPE 2024 & MdAPE 2025--26 & MAPE & Bias & $\pm5\%$ & $\pm10\%$ & $\pm20\%$ & Coverage \\ | |
| 4 | +\midrule | |
| 5 | +M12 hybrid comparables + GBM & 12.6 & 10.7 & 19.9 & -4.1 & 26.0 & 47.8 & 76.2 & 100.0 \\ | |
| 6 | +M11b gradient boosting, with assessed value & 13.1 & 10.7 & 19.9 & -4.7 & 24.4 & 47.4 & 75.5 & 100.0 \\ | |
| 7 | +M9 optimised reconciliation & 11.6 & 11.0 & 22.2 & -0.8 & 24.5 & 46.1 & 72.1 & 99.7 \\ | |
| 8 | +M0 meilleure configuration empirique (best configuration, chosen on 2024) & 11.6 & 11.3 & 24.1 & -0.1 & 23.1 & 45.0 & 70.2 & 99.0 \\ | |
| 9 | +M13 assessment ratio & 12.0 & 11.6 & 22.8 & 0.2 & 22.2 & 40.4 & 63.4 & 89.8 \\ | |
| 10 | +M8b GBM adjustments & 14.0 & 12.4 & 24.9 & -0.9 & 23.4 & 42.5 & 67.0 & 99.7 \\ | |
| 11 | +M3 5 most similar, time only & 14.6 & 12.9 & 27.0 & -0.2 & 21.2 & 40.9 & 64.4 & 99.7 \\ | |
| 12 | +M11a gradient boosting, no assessed value & 14.1 & 13.2 & 23.8 & -5.7 & 19.0 & 38.4 & 67.4 & 100.0 \\ | |
| 13 | +M7 hedonic grid & 14.0 & 13.5 & 25.4 & -0.7 & 21.7 & 39.4 & 66.1 & 99.7 \\ | |
| 14 | +M6 conventional grid & 14.5 & 13.8 & 26.8 & 0.6 & 20.5 & 39.0 & 64.3 & 99.7 \\ | |
| 15 | +M8a local grid & 15.1 & 14.1 & 26.2 & -0.2 & 19.9 & 37.0 & 64.1 & 99.7 \\ | |
| 16 | +M10 global hedonic model & 16.1 & 15.1 & 25.2 & -5.2 & 17.5 & 34.3 & 63.9 & 100.0 \\ | |
| 17 | +M4 5 nearest, distance-weighted & 19.3 & 18.6 & 32.0 & -0.3 & 14.9 & 29.6 & 53.0 & 99.7 \\ | |
| 18 | +M1 median of 3 nearest, time only & 20.2 & 18.8 & 33.8 & -0.6 & 15.2 & 28.4 & 50.3 & 96.5 \\ | |
| 19 | +M2 mean of 5 nearest, time only & 19.4 & 19.0 & 32.4 & -0.3 & 14.2 & 28.6 & 52.3 & 99.7 \\ | |
| 20 | +M5 5 nearest, recency-weighted & 19.3 & 19.2 & 33.0 & -0.2 & 14.2 & 27.7 & 51.1 & 99.7 \\ | |
| 21 | +\bottomrule | |
| 22 | +\end{tabular} | |
added
paper/tables/cond_rules.tex
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +\begin{tabular}{llrrrr} | |
| 2 | +\toprule | |
| 3 | +Density & Rule & MdAPE 2024 & MdAPE 2025--26 & Coverage 2024 & $n$ test \\ | |
| 4 | +\midrule | |
| 5 | +$<5$ & nearest, $k=5$, $\le 10$ km, 24 mo, hedonic grid, equal weights, median & 21.9 & 29.8 & 99.2 & 212 \\ | |
| 6 | +5--19 & nearest, $k=10$, $\le 10$ km, 24 mo, regional grid, 1/distance weights, median & 15.4 & 19.1 & 100.0 & 338 \\ | |
| 7 | +20--49 & nearest, $k=10$, $\le 10$ km, 24 mo, assessment ratio, equal weights, median & 12.3 & 13.5 & 95.7 & 322 \\ | |
| 8 | +50--199 & least adjusted, $k=10$, $\le 10$ km, 24 mo, assessment ratio, equal weights, median & 9.3 & 10.1 & 96.7 & 1,123 \\ | |
| 9 | +$\ge 200$ & compromise, $k=7$, $\le 0.5$ km, 12 mo, hedonic grid, equal weights, median & 9.0 & 9.6 & 100.0 & 668 \\ | |
| 10 | +\bottomrule | |
| 11 | +\end{tabular} | |
added
paper/tables/density.tex
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Density & $n$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 4 | +\midrule | |
| 5 | +$<5$ & 203 & 40.0 & 29.5 & 29.2 & 20.9 & 29.2 & 26.6 & 18.4 & 20.3 \\ | |
| 6 | +5--19 & 338 & 28.9 & 20.0 & 21.0 & 16.3 & 19.1 & 19.6 & 15.4 & 15.0 \\ | |
| 7 | +20--49 & 322 & 21.9 & 16.2 & 17.6 & 12.7 & 14.1 & 16.8 & 11.9 & 11.9 \\ | |
| 8 | +50--199 & 1,123 & 16.6 & 11.3 & 11.7 & 9.9 & 9.5 & 13.7 & 9.9 & 9.5 \\ | |
| 9 | +$\ge 200$ & 668 & 14.1 & 10.8 & 11.9 & 9.8 & 9.5 & 12.0 & 9.1 & 8.9 \\ | |
| 10 | +\bottomrule | |
| 11 | +\end{tabular} | |
added
paper/tables/density_dist.tex
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +\begin{tabular}{lrr} | |
| 2 | +\toprule | |
| 3 | +Density (sales within 2 km, 12 months) & Test targets & Share (\%) \\ | |
| 4 | +\midrule | |
| 5 | +$<5$ & 212 & 8.0 \\ | |
| 6 | +5--19 & 338 & 12.7 \\ | |
| 7 | +20--49 & 322 & 12.1 \\ | |
| 8 | +50--199 & 1,123 & 42.2 \\ | |
| 9 | +$\ge 200$ & 668 & 25.1 \\ | |
| 10 | +\bottomrule | |
| 11 | +\end{tabular} | |
added
paper/tables/descriptives.tex
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +\begin{tabular}{lrrr} | |
| 2 | +\toprule | |
| 3 | +Statistic & Development 2021--2023 & Validation 2024 & Test 2025--2026 \\ | |
| 4 | +\midrule | |
| 5 | +Sales & 0 & 0 & 0 \\ | |
| 6 | +Median price (\$) & 0 & 0 & 0 \\ | |
| 7 | +Mean price (\$) & 0 & 0 & 0 \\ | |
| 8 | +Median living area (m$^2$) & 0.0 & 0.0 & 0.0 \\ | |
| 9 | +Median lot area (m$^2$) & 0 & 0 & 0 \\ | |
| 10 | +Median age (years) & 0.0 & 0.0 & 0.0 \\ | |
| 11 | +Detached (\%) & 0.0 & 0.0 & 0.0 \\ | |
| 12 | +Median price/assessment ratio & 0.00 & 0.00 & 0.00 \\ | |
| 13 | +Median sales within 2 km, 12 months & -- & -- & -- \\ | |
| 14 | +Median candidate pool (25 km, 24 months) & -- & -- & -- \\ | |
| 15 | +\bottomrule | |
| 16 | +\end{tabular} | |
added
paper/tables/e1_best.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{llrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Selection & Adjustment & $k$ & $d$ (km) & $t$ (months) & MdAPE 2024 & MdAPE 2025--26 & Coverage \\ | |
| 4 | +\midrule | |
| 5 | +nearest & hedonic grid & 30 & 1 & 24 & 12.6 & 12.1 & 97.2 \\ | |
| 6 | +nearest & time only & 7 & 1 & 24 & 16.1 & 16.7 & 97.2 \\ | |
| 7 | +compromise & hedonic grid & 50 & 1 & 24 & 11.9 & 11.6 & 97.2 \\ | |
| 8 | +compromise & time only & 20 & 5 & 24 & 12.3 & 12.3 & 99.9 \\ | |
| 9 | +most similar & hedonic grid & 15 & 2 & 24 & 11.6 & 11.3 & 99.0 \\ | |
| 10 | +most similar & time only & 20 & 5 & 24 & 12.3 & 12.2 & 99.9 \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/exclusions.tex
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +\begin{tabular}{lr} | |
| 2 | +\toprule | |
| 3 | +Step & n \\ | |
| 4 | +\midrule | |
| 5 | +Single-family transactions, raw & 456,950 \\ | |
| 6 | +Not matched to the assessment roll & 9,255 \\ | |
| 7 | +Duplicates (same unit, same date) & 355 \\ | |
| 8 | +Roll unit not a single dwelling or more than one dwelling & 29,509 \\ | |
| 9 | +Coordinates missing or outside Quebec & 0 \\ | |
| 10 | +Living area missing or outside [40, 600] m$^2$ & 3,797 \\ | |
| 11 | +Lot area missing or outside [80, 20,000] m$^2$ & 4,278 \\ | |
| 12 | +Year built missing, before 1850 or after the sale (land sale) & 2,393 \\ | |
| 13 | +Sale in the year of construction (lot or unfinished building) & 5,427 \\ | |
| 14 | +Price outside [\$50,000, \$3,000,000] & 528 \\ | |
| 15 | +Assessed value missing or price/assessment ratio outside [0.33, 3] & 8,074 \\ | |
| 16 | +Resale within 180 days with price change above 40\% (flip) & 794 \\ | |
| 17 | +Physical link or construction type missing & 1,512 \\ | |
| 18 | +Municipality absent from the MAMH directory & 32 \\ | |
| 19 | +Final sample & 390,996 \\ | |
| 20 | +\bottomrule | |
| 21 | +\end{tabular} | |
added
paper/tables/grid.tex
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +\begin{tabular}{lr} | |
| 2 | +\toprule | |
| 3 | +Characteristic & Coefficient \\ | |
| 4 | +\midrule | |
| 5 | +log living area & 0.605 \\ | |
| 6 | +log lot area & 0.067 \\ | |
| 7 | +age (years) & -0.010 \\ | |
| 8 | +age$^2$/100 & 0.004 \\ | |
| 9 | +semi-detached & -0.074 \\ | |
| 10 | +row / integrated & -0.105 \\ | |
| 11 | +construction type: full storeys & -0.309 \\ | |
| 12 | +split-level & -0.098 \\ | |
| 13 | +mansard & 0.012 \\ | |
| 14 | +storeys & -0.094 \\ | |
| 15 | +built within 3 years & -0.015 \\ | |
| 16 | +\bottomrule | |
| 17 | +\end{tabular} | |
| 18 | +\par\footnotesize Within-$R^2$ = 0.362; $n$ = 266,437 sales before 2025-01-01. | |
added
paper/tables/h2.tex
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Density & $k=1$ & $k=3$ & $k=5$ & $k=7$ & $k=10$ & $k=15$ & $k=20$ & $k=30$ & $k=50$ \\ | |
| 4 | +\midrule | |
| 5 | +$<5$ & 29.6 & 31.8 & 29.0 & 28.3 & 27.0 & \textbf{26.7} & 27.6 & 28.5 & 28.0 \\ | |
| 6 | +5--19 & 27.2 & 23.4 & 20.0 & 19.7 & 19.2 & \textbf{18.4} & 18.6 & 18.4 & 19.7 \\ | |
| 7 | +20--49 & 20.9 & 18.1 & 16.2 & 15.4 & 15.4 & 14.6 & 14.3 & 14.8 & \textbf{13.7} \\ | |
| 8 | +50--199 & 14.8 & 12.3 & 11.3 & 11.0 & 10.6 & 10.5 & \textbf{10.2} & 10.4 & 10.4 \\ | |
| 9 | +$\ge 200$ & 13.3 & 11.5 & 10.8 & 10.6 & 10.2 & \textbf{10.1} & 10.2 & 10.1 & 10.2 \\ | |
| 10 | +\bottomrule | |
| 11 | +\end{tabular} | |
added
paper/tables/h3.tex
+19 −0
@@ -0,0 +1,19 @@ | ||
| 1 | +\begin{tabular}{lrrrrr} | |
| 2 | +\toprule | |
| 3 | +Density & conventional grid & hedonic grid & regional grid & local grid & GBM adjustments \\ | |
| 4 | +\midrule | |
| 5 | +\multicolumn{6}{l}{\emph{Anchor $(5,\,5\text{ km},\,12\text{ months})$}} \\ | |
| 6 | +$<5$ & 32.8 & 29.5 & 30.9 & \textbf{29.2} & 30.0 \\ | |
| 7 | +5--19 & 20.9 & 20.0 & \textbf{19.4} & 21.0 & 19.9 \\ | |
| 8 | +20--49 & 16.2 & 16.2 & 16.2 & 17.6 & \textbf{16.1} \\ | |
| 9 | +50--199 & 11.9 & 11.3 & 11.0 & 11.7 & \textbf{10.4} \\ | |
| 10 | +$\ge 200$ & 10.5 & 10.8 & 10.6 & 11.9 & \textbf{9.9} \\ | |
| 11 | +\midrule | |
| 12 | +\multicolumn{6}{l}{\emph{Anchor $(3,\,2\text{ km},\,6\text{ months})$}} \\ | |
| 13 | +$<5$ & 37.3 & 31.8 & 32.4 & 36.1 & \textbf{29.5} \\ | |
| 14 | +5--19 & 24.1 & 24.3 & 24.1 & 23.5 & \textbf{23.3} \\ | |
| 15 | +20--49 & 18.9 & 18.3 & 18.2 & 19.9 & \textbf{18.1} \\ | |
| 16 | +50--199 & 12.7 & 11.8 & 11.7 & 12.5 & \textbf{11.3} \\ | |
| 17 | +$\ge 200$ & 11.7 & 11.4 & 11.4 & 12.2 & \textbf{10.8} \\ | |
| 18 | +\bottomrule | |
| 19 | +\end{tabular} | |
added
paper/tables/h4.tex
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Density & equal / median & equal / mean & 1/distance / mean & recency / mean & most similar / mean & product / mean & product / median \\ | |
| 4 | +\midrule | |
| 5 | +$<5$ & 29.5 & 28.7 & 27.6 & 29.0 & \textbf{27.3} & 28.2 & 29.3 \\ | |
| 6 | +5--19 & 20.0 & 20.1 & 19.8 & 20.1 & 19.5 & \textbf{18.4} & 21.9 \\ | |
| 7 | +20--49 & \textbf{16.2} & 17.1 & 17.2 & 16.9 & 17.9 & 18.0 & 18.2 \\ | |
| 8 | +50--199 & \textbf{11.3} & 12.1 & 12.2 & 12.2 & 11.9 & 12.0 & 11.6 \\ | |
| 9 | +$\ge 200$ & \textbf{10.8} & 11.4 & 11.3 & 11.2 & 10.9 & 11.0 & 11.1 \\ | |
| 10 | +\bottomrule | |
| 11 | +\end{tabular} | |
added
paper/tables/h5.tex
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Submarket & Median $\hat\sigma$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 4 | +\midrule | |
| 5 | +homogeneous & 0.123 & 14.6 & 10.4 & 11.2 & 8.8 & 9.1 & 13.3 & 8.8 & 8.6 \\ | |
| 6 | +intermediate & 0.252 & 19.3 & 14.5 & 15.1 & 12.2 & 12.2 & 15.2 & 11.1 & 11.1 \\ | |
| 7 | +heterogeneous & 0.407 & 23.8 & 17.0 & 17.5 & 15.0 & 13.9 & 17.0 & 13.1 & 13.2 \\ | |
| 8 | +\bottomrule | |
| 9 | +\end{tabular} | |
added
paper/tables/intervals.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{lrrrrr} | |
| 2 | +\toprule | |
| 3 | +Configuration & $z$ (calibrated on 2024) & Coverage at $z=1.28$ & Coverage, calibrated $z$ & Median width (\%) & $n$ \\ | |
| 4 | +\midrule | |
| 5 | +time only, $k=5$, $\le 5$ km, 12 months & 1.89 & 66.3 & 79.3 & 89.5 & 2,645 \\ | |
| 6 | +time only, $k=10$, $\le 10$ km, 24 months & 1.39 & 76.7 & 79.2 & 85.2 & 2,662 \\ | |
| 7 | +hedonic grid, $k=5$, $\le 5$ km, 12 months & 1.93 & 67.1 & 79.8 & 67.2 & 2,645 \\ | |
| 8 | +hedonic grid, $k=10$, $\le 10$ km, 24 months & 1.28 & 77.8 & 77.9 & 63.0 & 2,662 \\ | |
| 9 | +GBM adjustments, $k=5$, $\le 5$ km, 12 months & 2.01 & 66.5 & 80.4 & 65.2 & 2,645 \\ | |
| 10 | +GBM adjustments, $k=10$, $\le 10$ km, 24 months & 1.42 & 77.1 & 80.5 & 66.4 & 2,662 \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/marginal.tex
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +\begin{tabular}{llr} | |
| 2 | +\toprule | |
| 3 | +Factor & Level & Mean MdAPE \\ | |
| 4 | +\midrule | |
| 5 | +Selection & least adjusted & 13.3 \\ | |
| 6 | +Selection & nearest & 14.6 \\ | |
| 7 | +Selection & compromise & 13.1 \\ | |
| 8 | +Selection & most similar & 13.3 \\ | |
| 9 | +Adjustment & conventional grid & 13.3 \\ | |
| 10 | +Adjustment & GBM adjustments & 13.1 \\ | |
| 11 | +Adjustment & hedonic grid & 13.3 \\ | |
| 12 | +Adjustment & regional grid & 13.3 \\ | |
| 13 | +Adjustment & local grid & 13.6 \\ | |
| 14 | +Adjustment & time only & 14.7 \\ | |
| 15 | +Adjustment & none & 15.4 \\ | |
| 16 | +Adjustment & assessment ratio & 11.8 \\ | |
| 17 | +Weighting & distance $\times$ recency $\times$ similarity & 13.6 \\ | |
| 18 | +Weighting & 1/distance & 13.6 \\ | |
| 19 | +Weighting & equal & 13.5 \\ | |
| 20 | +Weighting & recency & 13.8 \\ | |
| 21 | +Weighting & most similar & 13.3 \\ | |
| 22 | +Reconciliation & mean & 13.8 \\ | |
| 23 | +Reconciliation & median & 13.4 \\ | |
| 24 | +\bottomrule | |
| 25 | +\end{tabular} | |
added
paper/tables/numbers.tex
+91 −0
@@ -0,0 +1,91 @@ | ||
| 1 | +% généré par make_paper_tables.py | |
| 2 | +\newcommand{\nSalesRaw}{456,950} | |
| 3 | +\newcommand{\nSalesClean}{390,996} | |
| 4 | +\newcommand{\nTrain}{201,642} | |
| 5 | +\newcommand{\nVal}{64,795} | |
| 6 | +\newcommand{\nTest}{124,559} | |
| 7 | +\newcommand{\nTargetsVal}{1,337} | |
| 8 | +\newcommand{\nTargetsTest}{2,663} | |
| 9 | +\newcommand{\nTargetsAll}{4,000} | |
| 10 | +\newcommand{\nConfigs}{2,256} | |
| 11 | +\newcommand{\nTransactionsDb}{766,596} | |
| 12 | +\newcommand{\nUnits}{3.75 million} | |
| 13 | +\newcommand{\dateMin}{2021-01-05} | |
| 14 | +\newcommand{\dateMax}{2026-09-29} | |
| 15 | +\newcommand{\medianPrice}{400,000} | |
| 16 | +\newcommand{\mdapeMzero}{11.3\%} | |
| 17 | +\newcommand{\mdapeMtwo}{19.0\%} | |
| 18 | +\newcommand{\mdapeMthree}{12.9\%} | |
| 19 | +\newcommand{\mdapeMsix}{13.8\%} | |
| 20 | +\newcommand{\mdapeMseven}{13.5\%} | |
| 21 | +\newcommand{\mdapeMnine}{11.0\%} | |
| 22 | +\newcommand{\mdapeMten}{15.1\%} | |
| 23 | +\newcommand{\mdapeMelevenA}{13.2\%} | |
| 24 | +\newcommand{\mdapeMelevenB}{10.7\%} | |
| 25 | +\newcommand{\mdapeMtwelve}{10.7\%} | |
| 26 | +\newcommand{\mdapeMthirteen}{11.6\%} | |
| 27 | +\newcommand{\convMdape}{13.8\%} | |
| 28 | +\newcommand{\localMdape}{14.1\%} | |
| 29 | +\newcommand{\covMthirteen}{89.8\%} | |
| 30 | +\newcommand{\pTenMzero}{45.0\%} | |
| 31 | +\newcommand{\pTenMelevenB}{47.4\%} | |
| 32 | +\newcommand{\mdapeMoneRange}{18.6--19.2\%} | |
| 33 | +\newcommand{\mdapeGridRange}{12.9--13.8\%} | |
| 34 | +\newcommand{\transparencyCost}{0.6} | |
| 35 | +\newcommand{\transparencyCostNoRole}{-1.9} | |
| 36 | +\newcommand{\roleGainRatio}{1.8} | |
| 37 | +\newcommand{\roleGainGbm}{2.5} | |
| 38 | +\newcommand{\gainTime}{-0.0} | |
| 39 | +\newcommand{\gainGridLo}{4.2} | |
| 40 | +\newcommand{\gainGridHi}{5.8} | |
| 41 | +\newcommand{\localVsHedo}{slightly worse} | |
| 42 | +\newcommand{\gbmVsHedo}{1.0} | |
| 43 | +\newcommand{\gainSelHedo}{0.9} | |
| 44 | +\newcommand{\wrRange}{0.8} | |
| 45 | +\newcommand{\margSelLo}{13.1} | |
| 46 | +\newcommand{\margSelHi}{14.6} | |
| 47 | +\newcommand{\margAdjLo}{11.8} | |
| 48 | +\newcommand{\margAdjHi}{15.4} | |
| 49 | +\newcommand{\margWLo}{13.3} | |
| 50 | +\newcommand{\margWHi}{13.8} | |
| 51 | +\newcommand{\margReconMedian}{13.4} | |
| 52 | +\newcommand{\margReconMean}{13.8} | |
| 53 | +\newcommand{\kOpt}{50} | |
| 54 | +\newcommand{\mdapeKone}{17.0\%} | |
| 55 | +\newcommand{\mdapeKthree}{14.5\%} | |
| 56 | +\newcommand{\mdapeKfive}{13.4\%} | |
| 57 | +\newcommand{\mdapeKten}{12.5\%} | |
| 58 | +\newcommand{\mdapeKopt}{12.3\%} | |
| 59 | +\newcommand{\mdapeKfifty}{12.3\%} | |
| 60 | +\newcommand{\narrowMdape}{13.3\%} | |
| 61 | +\newcommand{\narrowCov}{89.3\%} | |
| 62 | +\newcommand{\bestSel}{most similar} | |
| 63 | +\newcommand{\bestK}{15} | |
| 64 | +\newcommand{\bestD}{2} | |
| 65 | +\newcommand{\bestT}{24} | |
| 66 | +\newcommand{\bestAdj}{hedonic grid} | |
| 67 | +\newcommand{\bestValMdape}{11.6\%} | |
| 68 | +\newcommand{\bestTestMdape}{11.3\%} | |
| 69 | +\newcommand{\mZeroDesc}{most similar, $k=15$, $\le 2$ km, 24 months, hedonic grid, equal weights, median} | |
| 70 | +\newcommand{\mZeroUncDesc}{most similar, $k=10$, $\le 10$ km, 24 months, assessment ratio} | |
| 71 | +\newcommand{\mZeroUncMdape}{10.6\%} | |
| 72 | +\newcommand{\mZeroUncCov}{84.4\%} | |
| 73 | +\newcommand{\densHighMseven}{10.8\%} | |
| 74 | +\newcommand{\densLowMseven}{29.5\%} | |
| 75 | +\newcommand{\hOneGeoDone}{17.8\%} | |
| 76 | +\newcommand{\hOneGeoDoneCov}{95.4\%} | |
| 77 | +\newcommand{\hOneSimDten}{12.9\%} | |
| 78 | +\newcommand{\hOneGeoDtenHedo}{13.4\%} | |
| 79 | +\newcommand{\hOneSimDtenHedo}{12.4\%} | |
| 80 | +\newcommand{\hOneMixDtenHedo}{12.7\%} | |
| 81 | +\newcommand{\hFiveGapHomog}{0.3} | |
| 82 | +\newcommand{\hFiveGapHeter}{0.9} | |
| 83 | +\newcommand{\ivWidth}{63.0\%} | |
| 84 | +\newcommand{\condMdape}{11.8\%} | |
| 85 | +\newcommand{\condPten}{40.9\%} | |
| 86 | +\newcommand{\condCov}{94.4\%} | |
| 87 | +\newcommand{\betaFloor}{0.60} | |
| 88 | +\newcommand{\betaLand}{0.07} | |
| 89 | +\newcommand{\betaAge}{1.0\%} | |
| 90 | +\newcommand{\betaSemi}{7.4\%} | |
| 91 | +\newcommand{\betaRow}{10.5\%} | |
added
paper/tables/pairs.tex
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +\begin{tabular}{lrrrr} | |
| 2 | +\toprule | |
| 3 | +Comparison & $\Delta$ MdAPE [95\% CI] & $\Delta$ MAPE [95\% CI] & $A$ more accurate (\%) & $n$ \\ | |
| 4 | +\midrule | |
| 5 | +M7 $-$ M2 & -5.53 [-6.38; -4.61] & -7.01 [-8.22; -5.72] & 61.7 & 2,654 \\ | |
| 6 | +M7 $-$ M6 & -0.31 [-0.94; 0.32] & -1.37 [-1.86; -0.96] & 51.6 & 2,654 \\ | |
| 7 | +M8a $-$ M7 & 0.60 [0.01; 1.04] & 0.87 [0.50; 1.28] & 47.1 & 2,654 \\ | |
| 8 | +M8b $-$ M7 & -1.01 [-1.75; -0.63] & -0.44 [-0.89; 0.01] & 54.0 & 2,654 \\ | |
| 9 | +M11b $-$ M0 & -0.68 [-1.34; 0.07] & -4.34 [-5.17; -3.55] & 55.6 & 2,637 \\ | |
| 10 | +M11a $-$ M0 & 1.79 [1.17; 2.46] & -0.58 [-1.12; -0.00] & 45.4 & 2,637 \\ | |
| 11 | +M12 $-$ M11b & -0.07 [-0.28; 0.17] & -0.05 [-0.15; 0.04] & 55.5 & 2,663 \\ | |
| 12 | +M13 $-$ M7 & -1.96 [-2.85; -1.28] & -2.59 [-3.50; -1.68] & 54.1 & 2,390 \\ | |
| 13 | +M3 $-$ M2 & -6.08 [-7.02; -5.07] & -5.41 [-6.39; -4.37] & 59.4 & 2,654 \\ | |
| 14 | +M9 $-$ M7 & -2.45 [-3.21; -1.87] & -3.21 [-3.92; -2.58] & 58.2 & 2,654 \\ | |
| 15 | +M1 $-$ M2 & 0.14 [-0.53; 1.31] & 2.01 [1.01; 2.90] & 46.1 & 2,569 \\ | |
| 16 | +\bottomrule | |
| 17 | +\end{tabular} | |
added
paper/tables/regions.tex
+23 −0
@@ -0,0 +1,23 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Region & $n$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 4 | +\midrule | |
| 5 | +Montérégie & 541 & 15.6 & 10.8 & 12.2 & 10.5 & 9.8 & 12.3 & 9.9 & 9.5 \\ | |
| 6 | +Laurentides & 287 & 20.3 & 14.6 & 14.5 & 11.8 & 11.0 & 15.1 & 10.5 & 10.4 \\ | |
| 7 | +Lanaudière & 258 & 18.7 & 14.1 & 13.8 & 11.1 & 9.8 & 14.7 & 10.9 & 11.0 \\ | |
| 8 | +Capitale-Nationale & 253 & 19.2 & 12.5 & 13.0 & 12.1 & 11.6 & 14.0 & 10.9 & 10.9 \\ | |
| 9 | +Montréal & 212 & 17.4 & 14.7 & 15.8 & 11.0 & 9.8 & 17.4 & 11.5 & 11.4 \\ | |
| 10 | +Estrie & 188 & 21.0 & 16.7 & 16.2 & 13.9 & 13.5 & 16.0 & 10.7 & 10.3 \\ | |
| 11 | +Chaudière-Appalaches & 170 & 20.0 & 17.1 & 16.9 & 13.5 & 14.9 & 19.0 & 13.7 & 14.4 \\ | |
| 12 | +Outaouais & 142 & 16.2 & 11.1 & 11.5 & 9.3 & 10.0 & 13.5 & 8.2 & 8.3 \\ | |
| 13 | +Saguenay--Lac-Saint-Jean & 126 & 23.0 & 14.3 & 14.3 & 10.5 & 15.3 & 18.0 & 11.9 & 12.1 \\ | |
| 14 | +Laval & 117 & 14.4 & 8.2 & 10.2 & 7.1 & 9.6 & 13.3 & 7.8 & 8.0 \\ | |
| 15 | +Centre-du-Québec & 94 & 21.1 & 15.5 & 16.1 & 11.5 & 14.1 & 15.1 & 9.6 & 9.4 \\ | |
| 16 | +Mauricie & 88 & 22.3 & 14.5 & 16.4 & 13.6 & 10.6 & 15.6 & 12.3 & 12.5 \\ | |
| 17 | +Bas-Saint-Laurent & 76 & 26.1 & 15.5 & 19.3 & 16.3 & 15.4 & 18.7 & 12.0 & 12.0 \\ | |
| 18 | +Abitibi-Témiscamingue & 50 & 26.1 & 16.0 & 20.3 & 13.3 & 13.9 & 20.5 & 15.2 & 15.3 \\ | |
| 19 | +Côte-Nord & 26 & 25.7 & 29.4 & 31.8 & 18.5 & 30.7 & 21.6 & 21.2 & 20.7 \\ | |
| 20 | +Gaspésie--Îles-de-la-Madeleine & 23 & 35.6 & 15.6 & 16.9 & 19.0 & 21.0 & 26.2 & 19.1 & 18.1 \\ | |
| 21 | +Nord-du-Québec & 3 & 27.1 & 16.2 & 13.9 & 15.7 & 31.7 & 3.1 & 13.6 & 9.8 \\ | |
| 22 | +\bottomrule | |
| 23 | +\end{tabular} | |
added
paper/tables/regression.tex
+37 −0
@@ -0,0 +1,37 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Table: Regression Results | |
| 3 | +% ============================================================================ | |
| 4 | +\begin{table}[htbp] | |
| 5 | + \centering | |
| 6 | + \begin{threeparttable} | |
| 7 | + \caption{Baseline Regression Results} | |
| 8 | + \label{tab:regression} | |
| 9 | + \begin{tabular}{@{}lcccc@{}} | |
| 10 | + \toprule | |
| 11 | + & (1) & (2) & (3) & (4) \\ | |
| 12 | + \midrule | |
| 13 | + $X_{i,t}$ & 0.245$^{***}$ & 0.198$^{***}$ & 0.183$^{***}$ & 0.172$^{***}$ \\ | |
| 14 | + & (0.042) & (0.038) & (0.035) & (0.033) \\[6pt] | |
| 15 | + Size & & & $-$0.032$^{**}$ & $-$0.028$^{**}$ \\ | |
| 16 | + & & & (0.014) & (0.013) \\[6pt] | |
| 17 | + Book-to-Market & & & 0.051$^{***}$ & 0.047$^{***}$ \\ | |
| 18 | + & & & (0.018) & (0.017) \\[6pt] | |
| 19 | + Leverage & & & & $-$0.089$^{**}$ \\ | |
| 20 | + & & & & (0.041) \\[6pt] | |
| 21 | + ROA & & & & 0.015$^{*}$ \\ | |
| 22 | + & & & & (0.008) \\[6pt] | |
| 23 | + \midrule | |
| 24 | + Firm FE & No & Yes & Yes & Yes \\ | |
| 25 | + Time FE & No & No & Yes & Yes \\ | |
| 26 | + Controls & No & No & Yes & Yes \\[3pt] | |
| 27 | + Observations & 125{,}400 & 125{,}400 & 125{,}400 & 125{,}400 \\ | |
| 28 | + $R^2$ & 0.032 & 0.148 & 0.215 & 0.228 \\ | |
| 29 | + Adjusted $R^2$ & 0.031 & 0.112 & 0.178 & 0.190 \\ | |
| 30 | + \bottomrule | |
| 31 | + \end{tabular} | |
| 32 | + \begin{tablenotes}[flushleft] | |
| 33 | + \footnotesize | |
| 34 | + \item \textit{Notes:} This table reports OLS estimates of Equation~\eqref{eq:baseline}. The dependent variable is $y_{i,t}$. $X_{i,t}$ is the main explanatory variable. Control variables include firm size, book-to-market ratio, leverage, and return on assets. Standard errors clustered at the firm level are reported in parentheses. $^{***}$, $^{**}$, and $^{*}$ denote statistical significance at the 1\%, 5\%, and 10\% levels, respectively. | |
| 35 | + \end{tablenotes} | |
| 36 | + \end{threeparttable} | |
| 37 | +\end{table} | |
added
paper/tables/selection.tex
+10 −0
@@ -0,0 +1,10 @@ | ||
| 1 | +\begin{tabular}{lrrrr} | |
| 2 | +\toprule | |
| 3 | +Selection & MdAPE 2024 & MdAPE 2025--26 & Bias & Within $\pm10\%$ \\ | |
| 4 | +\midrule | |
| 5 | +nearest & 14.0 & 13.5 & -0.7 & 39.4 \\ | |
| 6 | +most similar & 13.7 & 12.6 & 0.1 & 41.0 \\ | |
| 7 | +compromise & 13.8 & 12.8 & 0.0 & 41.8 \\ | |
| 8 | +least adjusted & 13.6 & 12.9 & 0.1 & 41.0 \\ | |
| 9 | +\bottomrule | |
| 10 | +\end{tabular} | |
added
paper/tables/stability.tex
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Quarter & $n$ & M2 & M7 & M0 & M11b & M12 & Bias M7 & Bias M11b \\ | |
| 4 | +\midrule | |
| 5 | +2025Q1 & 256 & 19.9 & 13.9 & 12.3 & 10.7 & 10.5 & -4.4 & -3.0 \\ | |
| 6 | +2025Q2 & 516 & 18.8 & 14.3 & 10.3 & 10.9 & 11.1 & -2.8 & -6.7 \\ | |
| 7 | +2025Q3 & 435 & 18.3 & 13.1 & 12.0 & 10.3 & 10.3 & 1.4 & -2.9 \\ | |
| 8 | +2025Q4 & 335 & 21.1 & 15.6 & 11.8 & 12.2 & 12.1 & 2.9 & -5.6 \\ | |
| 9 | +2026Q1 & 212 & 18.5 & 14.7 & 14.1 & 11.7 & 11.6 & -4.0 & -4.0 \\ | |
| 10 | +2026Q2 & 492 & 16.5 & 11.5 & 10.4 & 10.3 & 9.8 & -1.3 & -5.5 \\ | |
| 11 | +2026Q3 & 417 & 20.6 & 12.9 & 10.6 & 10.1 & 9.6 & 1.6 & -2.6 \\ | |
| 12 | +\bottomrule | |
| 13 | +\end{tabular} | |
added
paper/tables/strata_other.tex
+26 −0
@@ -0,0 +1,26 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Price quintile & $n$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 4 | +\midrule | |
| 5 | +Q1 (low) & 464 & 53.7 & 36.7 & 36.6 & 29.9 & 35.4 & 31.2 & 23.7 & 24.8 \\ | |
| 6 | +Q2 & 529 & 18.8 & 13.9 & 15.6 & 13.0 & 11.4 & 13.9 & 9.5 & 9.3 \\ | |
| 7 | +Q3 & 506 & 14.8 & 10.8 & 11.9 & 10.7 & 10.0 & 13.9 & 9.6 & 9.3 \\ | |
| 8 | +Q4 & 575 & 13.9 & 10.3 & 10.3 & 8.2 & 8.0 & 12.4 & 9.1 & 8.9 \\ | |
| 9 | +Q5 (high) & 580 & 19.5 & 12.5 & 13.3 & 9.7 & 10.1 & 14.0 & 10.4 & 10.0 \\ | |
| 10 | +\midrule | |
| 11 | +Age & $n$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 12 | +\midrule | |
| 13 | +16--40 & 803 & 17.7 & 11.3 & 11.7 & 9.6 & 9.8 & 13.1 & 9.5 & 9.0 \\ | |
| 14 | +4--15 & 388 & 18.3 & 10.5 & 11.6 & 9.6 & 8.4 & 12.8 & 9.0 & 8.7 \\ | |
| 15 | +41--70 & 1,060 & 18.3 & 14.6 & 15.3 & 13.4 & 12.5 & 16.0 & 11.8 & 11.8 \\ | |
| 16 | +$>70$ & 345 & 26.4 & 20.4 & 21.2 & 15.6 & 17.2 & 20.9 & 15.2 & 15.4 \\ | |
| 17 | +$\le 3$ years & 58 & 25.8 & 15.7 & 14.4 & 12.0 & 13.1 & 14.8 & 9.4 & 9.4 \\ | |
| 18 | +\midrule | |
| 19 | +Physical link & $n$ & M2 & M7 & M8a & M13 & M0 & M10 & M11b & M12 \\ | |
| 20 | +\midrule | |
| 21 | +0.0 & 2,161 & 20.1 & 14.6 & 15.0 & 12.4 & 12.2 & 15.5 & 11.4 & 11.4 \\ | |
| 22 | +1.0 & 324 & 15.0 & 10.3 & 11.2 & 9.2 & 9.1 & 12.8 & 9.1 & 8.5 \\ | |
| 23 | +2.0 & 165 & 13.6 & 8.6 & 10.0 & 8.5 & 7.3 & 12.8 & 7.7 & 7.6 \\ | |
| 24 | +3.0 & 4 & 30.4 & 30.0 & 21.4 & 18.5 & 23.8 & 12.9 & 16.2 & 16.9 \\ | |
| 25 | +\bottomrule | |
| 26 | +\end{tabular} | |
added
paper/tables/summary_stats.tex
+29 −0
@@ -0,0 +1,29 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Table: Summary Statistics | |
| 3 | +% ============================================================================ | |
| 4 | +\begin{table}[htbp] | |
| 5 | + \centering | |
| 6 | + \begin{threeparttable} | |
| 7 | + \caption{Summary Statistics} | |
| 8 | + \label{tab:summary_stats} | |
| 9 | + \begin{tabular}{@{}lcccccc@{}} | |
| 10 | + \toprule | |
| 11 | + Variable & $N$ & Mean & Std.\ Dev. & $p_{25}$ & Median & $p_{75}$ \\ | |
| 12 | + \midrule | |
| 13 | + \multicolumn{7}{@{}l}{\textit{Panel A: Dependent Variables}} \\[3pt] | |
| 14 | + $r_{i,t}$ (\%) & 125{,}400 & 0.82 & 4.35 & $-$1.92 & 0.65 & 3.48 \\ | |
| 15 | + $\sigma_{i,t}$ (\%) & 125{,}400 & 2.14 & 1.87 & 0.98 & 1.65 & 2.78 \\[6pt] | |
| 16 | + \multicolumn{7}{@{}l}{\textit{Panel B: Explanatory Variables}} \\[3pt] | |
| 17 | + $X_{i,t}$ & 125{,}400 & 0.45 & 0.32 & 0.21 & 0.40 & 0.65 \\ | |
| 18 | + Size (log) & 125{,}400 & 7.23 & 1.85 & 5.92 & 7.15 & 8.47 \\ | |
| 19 | + Book-to-Market & 125{,}400 & 0.68 & 0.54 & 0.30 & 0.56 & 0.91 \\ | |
| 20 | + Leverage & 125{,}400 & 0.34 & 0.21 & 0.17 & 0.32 & 0.48 \\ | |
| 21 | + ROA (\%) & 125{,}400 & 3.15 & 8.42 & 0.52 & 3.28 & 6.74 \\ | |
| 22 | + \bottomrule | |
| 23 | + \end{tabular} | |
| 24 | + \begin{tablenotes}[flushleft] | |
| 25 | + \footnotesize | |
| 26 | + \item \textit{Notes:} This table reports summary statistics for the main variables used in the analysis. The sample covers the period January~1990 to December~2023. $r_{i,t}$ is the monthly return of firm~$i$. $\sigma_{i,t}$ is the realized volatility computed from daily returns within the month. $X_{i,t}$ is the main explanatory variable. Size is the natural logarithm of market capitalization. Book-to-Market is the ratio of book equity to market equity. Leverage is total debt over total assets. ROA is return on assets. All continuous variables are winsorized at the 1st and 99th percentiles. | |
| 27 | + \end{tablenotes} | |
| 28 | + \end{threeparttable} | |
| 29 | +\end{table} | |
added
paper/tables/surface_cov.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{lrrrr} | |
| 2 | +\toprule | |
| 3 | +Radius & 3 months & 6 months & 12 months & 24 months \\ | |
| 4 | +\midrule | |
| 5 | +$\le 0.5$ km & 79.8 & 87.3 & 91.7 & 94.7 \\ | |
| 6 | +$\le 1$ km & 89.3 & 92.6 & 95.4 & 97.2 \\ | |
| 7 | +$\le 2$ km & 93.9 & 96.5 & 98.4 & 99.0 \\ | |
| 8 | +$\le 5$ km & 98.0 & 99.2 & 99.7 & 99.9 \\ | |
| 9 | +$\le 10$ km & 99.4 & 99.8 & 99.9 & 100.0 \\ | |
| 10 | +$\le 25$ km & 99.8 & 100.0 & 100.0 & 100.0 \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/surface_geo_hedo.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Radius & $k=1$ & $k=3$ & $k=5$ & $k=7$ & $k=10$ & $k=15$ & $k=20$ & $k=30$ & $k=50$ \\ | |
| 4 | +\midrule | |
| 5 | +$\le 0.5$ km & 16.2 & 13.8 & 12.9 & 12.7 & 12.3 & \textbf{12.1} & 12.2 & 12.2 & 12.2 \\ | |
| 6 | +$\le 1$ km & 16.5 & 14.1 & 13.0 & 12.8 & 12.4 & 12.1 & 12.1 & \textbf{12.0} & 12.1 \\ | |
| 7 | +$\le 2$ km & 16.7 & 14.2 & 13.2 & 12.9 & 12.4 & 12.2 & 12.2 & 12.2 & \textbf{12.1} \\ | |
| 8 | +$\le 5$ km & 17.0 & 14.4 & 13.5 & 13.0 & 12.5 & 12.4 & 12.4 & 12.4 & \textbf{12.3} \\ | |
| 9 | +$\le 10$ km & 17.0 & 14.5 & 13.4 & 13.0 & 12.5 & 12.4 & 12.5 & 12.5 & \textbf{12.3} \\ | |
| 10 | +$\le 25$ km & 17.0 & 14.5 & 13.5 & 13.0 & 12.5 & 12.3 & 12.4 & 12.5 & \textbf{12.3} \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/surface_geo_none.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Radius & $k=1$ & $k=3$ & $k=5$ & $k=7$ & $k=10$ & $k=15$ & $k=20$ & $k=30$ & $k=50$ \\ | |
| 4 | +\midrule | |
| 5 | +$\le 0.5$ km & 20.5 & 17.5 & 17.5 & 17.0 & 17.0 & 16.8 & \textbf{16.7} & 17.0 & 17.0 \\ | |
| 6 | +$\le 1$ km & 20.8 & 18.0 & 17.8 & 17.5 & \textbf{17.2} & 17.2 & 17.5 & 17.5 & 18.0 \\ | |
| 7 | +$\le 2$ km & 21.1 & 18.4 & 18.1 & 17.8 & \textbf{17.5} & 17.6 & 17.8 & 17.6 & 18.4 \\ | |
| 8 | +$\le 5$ km & 21.4 & 18.7 & 18.2 & 17.9 & \textbf{17.6} & 17.8 & 17.8 & 17.7 & 18.6 \\ | |
| 9 | +$\le 10$ km & 21.4 & 18.7 & 18.2 & 17.8 & 17.5 & 17.5 & 17.6 & \textbf{17.5} & 18.3 \\ | |
| 10 | +$\le 25$ km & 21.4 & 18.7 & 18.2 & 17.8 & \textbf{17.5} & 17.6 & 17.7 & 17.6 & 18.4 \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/surface_sim_hedo.tex
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +\begin{tabular}{lrrrrrrrrr} | |
| 2 | +\toprule | |
| 3 | +Radius & $k=1$ & $k=3$ & $k=5$ & $k=7$ & $k=10$ & $k=15$ & $k=20$ & $k=30$ & $k=50$ \\ | |
| 4 | +\midrule | |
| 5 | +$\le 0.5$ km & 14.6 & 13.1 & 12.3 & 12.1 & 11.9 & \textbf{11.7} & 12.0 & 12.2 & 12.2 \\ | |
| 6 | +$\le 1$ km & 14.9 & 12.9 & 12.2 & 12.0 & 11.7 & 11.6 & \textbf{11.6} & 11.6 & 11.9 \\ | |
| 7 | +$\le 2$ km & 15.2 & 12.9 & 12.4 & 11.9 & 11.7 & 11.7 & \textbf{11.4} & 11.4 & 11.7 \\ | |
| 8 | +$\le 5$ km & 15.7 & 13.1 & 12.6 & 12.5 & 12.0 & \textbf{11.9} & 12.0 & 12.0 & 12.1 \\ | |
| 9 | +$\le 10$ km & 16.2 & 13.4 & 12.4 & 12.4 & 12.3 & 11.9 & \textbf{11.8} & 12.1 & 12.2 \\ | |
| 10 | +$\le 25$ km & 16.3 & 13.6 & 12.8 & 12.4 & 12.5 & 12.3 & \textbf{12.0} & 12.2 & 12.3 \\ | |
| 11 | +\bottomrule | |
| 12 | +\end{tabular} | |
added
paper/tables/weighting.tex
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +\begin{tabular}{llrrrr} | |
| 2 | +\toprule | |
| 3 | +Weighting & Reconciliation & MdAPE 2024 & MdAPE 2025--26 & Bias & Within $\pm10\%$ \\ | |
| 4 | +\midrule | |
| 5 | +equal & median & 14.0 & 13.5 & -0.7 & 39.4 \\ | |
| 6 | +equal & mean & 14.8 & 13.9 & -2.3 & 36.2 \\ | |
| 7 | +1/distance & median & 13.8 & 13.3 & -0.7 & 39.8 \\ | |
| 8 | +1/distance & mean & 14.6 & 13.8 & -2.2 & 36.8 \\ | |
| 9 | +recency & median & 15.5 & 14.2 & -0.2 & 38.2 \\ | |
| 10 | +recency & mean & 15.5 & 14.1 & -2.2 & 36.5 \\ | |
| 11 | +most similar & median & 13.9 & 13.8 & 0.2 & 39.0 \\ | |
| 12 | +most similar & mean & 14.0 & 13.5 & -1.8 & 38.5 \\ | |
| 13 | +distance $\times$ recency $\times$ similarity & median & 14.7 & 13.7 & 0.4 & 38.1 \\ | |
| 14 | +distance $\times$ recency $\times$ similarity & mean & 14.7 & 13.9 & -1.6 & 38.3 \\ | |
| 15 | +\bottomrule | |
| 16 | +\end{tabular} | |
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
modified
pipeline/03_models.py
+4 −3
@@ -35,7 +35,8 @@ def ols(X, y): | ||
| 35 | 35 | |
| 36 | 36 | ASOF = list(range(M_EVAL0, int(df["month"].max()) + 1, 3)) # ré-estimation trimestrielle (2024-01, 2024-04, …) |
| 37 | 37 | save_json({"asof": ASOF}, "asof.json") |
| 38 | −for m_asof in ASOF: | |
| 38 | +TODO = [int(a) for a in os.environ["CV_ASOF"].split(",")] if os.environ.get("CV_ASOF") else ASOF # sous-ensemble (répartition sur plusieurs nœuds) | |
| 39 | +for m_asof in TODO: | |
| 39 | 40 | period = str(m_asof); start = f"{2021 + m_asof // 12}-{m_asof % 12 + 1:02d}-01" |
| 40 | 41 | tr = df[df["month"] < m_asof].copy() |
| 41 | 42 | y = time_adjusted(tr, m_asof) |
@@ -91,7 +92,7 @@ for m_asof in ASOF: | ||
| 91 | 92 | return G |
| 92 | 93 | G = gbm_frame(tr) |
| 93 | 94 | params = dict(objective="regression_l1", learning_rate=0.05, num_leaves=127, min_data_in_leaf=40, feature_fraction=0.8, |
| 94 | − bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=20) | |
| 95 | + bagging_fraction=0.8, bagging_freq=1, lambda_l2=1.0, verbose=-1, num_threads=int(os.environ.get("CV_THREADS", 20))) | |
| 95 | 96 | models = {} |
| 96 | 97 | for name, cols in [("gbm", [c for c in G.columns if c not in ("ln_role", "part_terrain", "role_vintage")]), |
| 97 | 98 | ("gbm_role", list(G.columns))]: |
@@ -101,7 +102,7 @@ for m_asof in ASOF: | ||
| 101 | 102 | dtr = lgb.Dataset(G[cols].iloc[:cut], y[:cut], categorical_feature=["rid", "btype"]) |
| 102 | 103 | dva = lgb.Dataset(G[cols].iloc[cut:], y[cut:], reference=dtr) |
| 103 | 104 | bst = lgb.train(params, dtr, num_boost_round=3000, valid_sets=[dva], callbacks=[lgb.early_stopping(100, verbose=False)]) |
| 104 | − best = bst.best_iteration | |
| 105 | + best = max(bst.best_iteration, 300) # plancher : l'arrêt précoce sur la tranche finale (mois partiels) peut être trop court | |
| 105 | 106 | bst = lgb.train(params, ds, num_boost_round=best) |
| 106 | 107 | models[name] = bst |
| 107 | 108 | Gall = gbm_frame(df); Gall["month"] = float(m_asof - 1) # f évalué au niveau du dernier mois connu pour toutes les ventes |
modified
pipeline/04_backtest.py
+5 −2
@@ -163,6 +163,9 @@ if __name__ == "__main__": | ||
| 163 | 163 | targets = df.index[df["period"].isin(["val", "test"])].to_numpy() |
| 164 | 164 | if os.environ.get("CV_SAMPLE"): |
| 165 | 165 | rng = np.random.default_rng(1); targets = np.sort(rng.choice(targets, int(os.environ["CV_SAMPLE"]), replace=False)) |
| 166 | + SUF = "" | |
| 167 | + if os.environ.get("CV_SHARD"): # répartition sur plusieurs nœuds : CV_SHARD=i/N → cibles i, i+N, i+2N, … | |
| 168 | + i_sh, n_sh = map(int, os.environ["CV_SHARD"].split("/")); targets = targets[i_sh::n_sh]; SUF = f".shard{i_sh}of{n_sh}" | |
| 166 | 169 | log(f"{len(targets):,} cibles ; {len(E1)} configurations E1, {len(E2)} E2 ; {N:,} ventes dans le bassin") |
| 167 | 170 | pd.DataFrame(E1).to_csv(OUT / "configs_E1.csv", index_label="cfg") |
| 168 | 171 | pd.DataFrame(E2).to_csv(OUT / "configs_E2.csv", index_label="cfg") |
@@ -179,12 +182,12 @@ if __name__ == "__main__": | ||
| 179 | 182 | est1[done] = e1; est2[done] = e2; sd2[done] = s2; nav[done] = nv; infos.append(info); done += 1 |
| 180 | 183 | if done % 20000 < 400: |
| 181 | 184 | log(f" {done:,}/{len(targets):,} cibles ({time.time() - t0:.0f} s)") |
| 182 | − np.save(OUT / "est_E1.npy", est1); np.save(OUT / "est_E2.npy", est2); np.save(OUT / "sd_E2.npy", sd2); np.save(OUT / "n_avail.npy", nav) | |
| 185 | + np.save(OUT / f"est_E1{SUF}.npy", est1); np.save(OUT / f"est_E2{SUF}.npy", est2); np.save(OUT / f"sd_E2{SUF}.npy", sd2); np.save(OUT / f"n_avail{SUF}.npy", nav) | |
| 183 | 186 | T = df.loc[targets, ["idx", "id", "date", "period", "amount", "ln_price", "region", "region_nom", "mrc", "mrc_nom", "code_mun", |
| 184 | 187 | "mun_nom", "unite_voisinage", "floor_area", "land_area", "age", "lien", "etages", "role_vente", "month", "lat", "lng"]].reset_index(drop=True) |
| 185 | 188 | T["n_cand"] = [x["n_cand"] for x in infos]; T["asof"] = [x["asof"] for x in infos] |
| 186 | 189 | T["n_2km_12m"] = nav[:, DS.index(2) * len(TS) + TS.index(12)] |
| 187 | 190 | T["n_5km_12m"] = nav[:, DS.index(5) * len(TS) + TS.index(12)] |
| 188 | − save_parquet(T, "targets.parquet") | |
| 191 | + save_parquet(T, f"targets{SUF}.parquet") | |
| 189 | 192 | pd.DataFrame({"DS": [DS], "TS": [TS], "KS": [KS]}).to_json(OUT / "grid_meta.json") |
| 190 | 193 | log(f"rétro-évaluation terminée en {time.time() - t0:.0f} s") |
added
pipeline/04b_gather.py
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# -*- coding: utf-8 -*- | |
| 2 | +"""04b — Rassemble les fragments (shards) de la rétro-évaluation calculés sur plusieurs nœuds, dans l'ordre des cibles.""" | |
| 3 | +import glob | |
| 4 | +from common import * | |
| 5 | +sh = sorted(glob.glob(str(OUT / "targets.shard*.parquet"))) | |
| 6 | +log(f"{len(sh)} fragments") | |
| 7 | +Ts = [pd.read_parquet(p) for p in sh] | |
| 8 | +parts = {} | |
| 9 | +for name in ["est_E1", "est_E2", "sd_E2", "n_avail"]: | |
| 10 | + parts[name] = [np.load(p.replace("targets.", f"{name}.").replace(".parquet", ".npy")) for p in sh] | |
| 11 | +T = pd.concat(Ts, ignore_index=True); order = np.argsort(T["idx"].to_numpy(), kind="stable") | |
| 12 | +T = T.iloc[order].reset_index(drop=True); save_parquet(T, "targets.parquet") | |
| 13 | +for name, arrs in parts.items(): | |
| 14 | + A = np.concatenate(arrs, axis=0)[order]; np.save(OUT / f"{name}.npy", A); log(f" {name}: {A.shape}") | |
| 15 | +log("rassemblement terminé") | |
modified
pipeline/05_analyze.py
+9 −0
@@ -351,6 +351,15 @@ for j, (r, lab_) in enumerate([("QC", "Québec"), ("06", "Montréal"), ("03", "C | ||
| 351 | 351 | ax.set_ylabel("indice hédonique (2021-01 = 100)"); ax.legend(fontsize=8); ax.set_title("Indice de prix mensuel (plein échantillon, descriptif)") |
| 352 | 352 | savefig(fig, "f7_indice") |
| 353 | 353 | res["dataset"] = load_json("dataset_stats.json") |
| 354 | +desc = {} | |
| 355 | +for per in ["train", "val", "test"]: | |
| 356 | + d = S[S["period"] == per] | |
| 357 | + desc[per] = dict(n=int(len(d)), prix_med=float(d.amount.median()), prix_moy=float(d.amount.mean()), aire_med=float(d.floor_area.median()), | |
| 358 | + terrain_med=float(d.land_area.median()), age_med=float(d.age.median()), detache=float((d.lien == 0).mean() * 100), | |
| 359 | + etages_moy=float(d.etages.mean()), ratio_role_med=float(d.ratio_role.median())) | |
| 360 | +for per, m in [("val", VAL), ("test", TEST)]: | |
| 361 | + desc[per]["dens_med"] = float(np.median(T[m].n_2km_12m)); desc[per]["n_cand_med"] = float(np.median(T[m].n_cand)) | |
| 362 | +res["descriptives"] = desc | |
| 354 | 363 | res["n_targets"] = dict(val=int(VAL.sum()), test=int(TEST.sum())) |
| 355 | 364 | res["densite_distribution"] = T[TEST]["dens"].value_counts().sort_index().to_dict() |
| 356 | 365 | res["n_cand_median"] = dict(val=float(np.median(T[VAL].n_cand)), test=float(np.median(T[TEST].n_cand))) |
added
run_distributed.sh
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +#!/bin/zsh | |
| 2 | +# Rétro-évaluation complète répartie sur le cluster : fragments proportionnels aux cœurs, rassemblement sur M2U64. | |
| 3 | +# Prérequis : 01-03 terminés (modèles trimestriels présents sur M2U64 dans out/$V), env Python sur chaque nœud. | |
| 4 | +set -e | |
| 5 | +V=${CV_VERSION:-2026-10-10} | |
| 6 | +HUB=M2U64 | |
| 7 | +typeset -A SHARDS; SHARDS=(M3U96a 4 M3U96b 4 M2U64 3 M4M64a 2 M4M64b 2) | |
| 8 | +typeset -A CORES; CORES=(M3U96a 32 M3U96b 32 M2U64 24 M4M64a 16 M4M64b 16) | |
| 9 | +N=15 | |
| 10 | +PYHUB='~/uqo-eval-terrains/env/bin/python' | |
| 11 | +echo "== 1. modèles trimestriels → $HUB → nœuds" | |
| 12 | +rm -rf /tmp/cv-models && mkdir -p /tmp/cv-models | |
| 13 | +for n in M2U64 M3U96a M3U96b M4M64a M4M64b; do ssh $n "cd ~/comparables-validation/out/$V && tar czf - models_*.json gbm_pred_*.npy gbm_role_pred_*.npy asof.json 2>/dev/null" | tar xzf - -C /tmp/cv-models; done | |
| 14 | +ls /tmp/cv-models | wc -l | |
| 15 | +for n in M2U64 M3U96a M3U96b M4M64a M4M64b; do tar czf - -C /tmp/cv-models . | ssh $n "tar xzf - -C ~/comparables-validation/out/$V" & done; wait | |
| 16 | +for n in M2U64 M3U96a M3U96b M4M64a M4M64b; do echo " $n: $(ssh $n "ls ~/comparables-validation/out/$V/models_*.json | wc -l") modèles ; rm -f shards"; ssh $n "rm -f ~/comparables-validation/out/shard-*.log ~/comparables-validation/out/$V/*shard*"; done | |
| 17 | +echo "== 2. lancement des $N fragments" | |
| 18 | +i=0 | |
| 19 | +for n in ${(k)SHARDS}; do | |
| 20 | + k=$SHARDS[$n]; procs=$(( CORES[$n] / k )) | |
| 21 | + py='../env/bin/python'; [ "$n" = "M2U64" ] && py=$PYHUB | |
| 22 | + for j in $(seq 1 $k); do | |
| 23 | + ssh $n "cd ~/comparables-validation/pipeline && CV_VERSION=$V CV_SHARD=$i/$N CV_PROCS=$procs nohup $py 04_backtest.py > ../out/shard-$i.log 2>&1 &" | |
| 24 | + i=$((i+1)) | |
| 25 | + done | |
| 26 | +done | |
| 27 | +echo "== 3. attente" | |
| 28 | +while true; do | |
| 29 | + done_=0 | |
| 30 | + for n in ${(k)SHARDS}; do done_=$(( done_ + $(ssh $n "grep -l 'rétro-évaluation terminée' ~/comparables-validation/out/shard-*.log 2>/dev/null | wc -l") )); done | |
| 31 | + err=0; for n in ${(k)SHARDS}; do err=$(( err + $(ssh $n "grep -l Traceback ~/comparables-validation/out/shard-*.log 2>/dev/null | wc -l") )); done | |
| 32 | + echo " $(date +%T) fragments terminés : $done_/$N (erreurs : $err)" | |
| 33 | + [ "$err" -gt 0 ] && { for n in ${(k)SHARDS}; do ssh $n "grep -B2 -A8 Traceback ~/comparables-validation/out/shard-*.log 2>/dev/null"; done; exit 1; } | |
| 34 | + [ "$done_" -ge "$N" ] && break | |
| 35 | + sleep 20 | |
| 36 | +done | |
| 37 | +echo "== 4. rassemblement sur $HUB" | |
| 38 | +for n in M3U96a M3U96b M4M64a M4M64b; do ssh -A $HUB "rsync -a -e 'ssh -o StrictHostKeyChecking=accept-new' $(ssh $n 'echo simon-pierreboucher@$(ipconfig getifaddr en0)'):comparables-validation/out/$V/'*shard*' ~/comparables-validation/out/$V/" & done; wait | |
| 39 | +ssh $HUB "cd ~/comparables-validation/pipeline && CV_VERSION=$V $PYHUB 04b_gather.py && CV_VERSION=$V $PYHUB 05_analyze.py && CV_VERSION=$V $PYHUB 06_report.py" 2>&1 | grep -v Warning | |
| 40 | +echo "== terminé $(date +%T)" | |
| 41 | ||