# ⚖️ Does the Sales Comparison Approach Actually Work? **UQO Working Paper No. 16** — *A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation — Evidence from 391,000 Single-Family Transactions in Quebec, 2021–2026* [-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) [](pipeline/) [](docs/PROTOCOLE.md) [](pipeline/) [](https://uqo.ca) [](mailto:simon-pierre.boucher@uqo.ca) > **TL;DR** — The sales comparison approach rests on professional convention: three comparables, a short window, a > small radius, an adjustment grid, a reconciliation rule. We backtest 2,256 configurations of the approach, point in > time, on 189,354 masked target sales drawn from 390,996 Quebec single-family transactions matched to the assessment > roll (configurations chosen on 2024, evaluated on 2025–2026; paired differences tested with a 999-replication cluster > bootstrap over municipality × quarter cells). **Adjustments dominate** (median absolute error of the five nearest > sales: 18.1% time-only → 13.0% with a hedonic grid); **similar beats near** (selecting physically similar sales gives > the same gain with no adjustment); **the optimal number of comparables is ~20–30, not 3**; windows should be wide, > radius barely matters, weighting and reconciliation matter little. The best transparent rule reaches **11.1%** out of > sample versus 9.0% for a quarterly-retrained gradient-boosting model that uses the assessed value (10.7% without it): > the cost of transparency is 2.2 points, and the chosen rule ranks 2nd of 1,370 eligible configurations out of sample. > Accuracy is governed by submarket liquidity. --- ## 📌 Status | Item | Value | |---|---| | Version | **1.1 (October 2026)** — revised 2026-10-17 after a full editorial-board review (see `docs/REVIEW-REPORT.md`, `docs/CHANGELOG-REVISION.md`); v1.0 frozen as git tag `pre-revision-20261017` and `docs/main_pre-revision-20261017.pdf` | | SSRN | abstract **7626720** (PRELIMINARY_UPLOAD, 2026-10-17) — uploaded PDF carries the old cover "Working Paper No. 11" and the v1.0 text; **re-upload `paper/main.pdf` (No. 16, v1.1)** | | Journal | target *Journal of Real Estate Finance and Economics*; fallback *Journal of Real Estate Research*. Submission copies: `Recherche-UQO/_soumissions/revues/sales-comparison-validation/` (anonymous and with-authors PDFs) | | Code & data | pipeline in `pipeline/`, run on the MacLustr node M2U64; the Vrai Prix database snapshot is not redistributed | ## 🎯 What this paper does For every target sale the price is masked and the approach is applied with only the information available on the valuation date: candidate comparables are selected (nearest / most similar / compromise / least adjusted, k ∈ {1 … 50}, radius d, window t), adjusted (none / time only / stylized conventional grid / hedonic grid estimated quarterly / regional grid / local ridge grid / GBM adjustments / assessment ratio), weighted and reconciled (median, mean, weighted). Error surfaces Error(k, d, t) are mapped, the best configuration is chosen on the 2024 validation year and evaluated out of sample on 2025–2026 against hedonic, gradient-boosting, assessment-ratio and hybrid benchmarks. Accuracy is then decomposed by submarket liquidity, region, price stratum and property type, and the selection step itself is audited (validation → test rank stability). ## 🏆 Headline results | Object | Estimate (test 2025–2026, MdAPE) | |---|---| | 5 nearest sales, time adjustment only (M2) | 18.1% | | 5 nearest sales, hedonic adjustment grid (M7) | 13.0% (M7 − M2: −5.1 pts, 95% CI [−5.26; −4.88]) | | 5 most similar sales, no physical adjustment (M3) | 13.0% | | Number of comparables with a grid, k = 1 → 3 → 5 → 10 → 20–50 | 16.5% → 13.8% → 13.0% → 12.4% → 12.1% (flat) | | Best transparent rule chosen in 2024 (M0: compromise, k = 30, ≤ 25 km, 24 months, hedonic grid) | **11.1%** [10.8; 11.5]; rank 2 / 1,370 out of sample | | Assessment-ratio comparable (M13, 90.3% coverage) | 11.5% | | Quarterly-retrained GBM without / with assessed value (M11a / M11b) | 10.7% / 9.0% [8.8; 9.2] | | Hybrid comparables + GBM (M12) | 8.9% (M12 − M11b: −0.06 pts, t = −8.3) | | Cost of transparency (M11b − M0) | 2.2 pts [2.0; 2.3], t = −36; 1.4 pts in the most homogeneous tercile | | Liquidity effect (M7): ≥ 200 sales within 2 km / year vs < 5 | 10.2% vs 25.3% | | Calibrated 80% value interval (hedonic grid, k = 10) | median width 66.5% of value (≈ ±33%) | ## 🗂 Repository layout ``` sales-comparison-validation/ ├── pipeline/ # 01_dataset → 02_index → 03_models → 04_backtest (+04b_gather) → 05_analyze → 06_report │ └── 07_revision_analyses.py # laptop-side complementary analyses (bootstrap B=999, selection stability, R8, seasonality) ├── run_all.sh # full chain on the compute node (CV_VERSION=YYYY-MM-DD) ├── run_distributed.sh # 15-shard backtest across 5 MacLustr nodes, gather on M2U64 ├── make_pdf.sh # pull out// + docs/ from M2U64, compile docs/RAPPORT.pdf (French report) ├── docs/ # PROTOCOLE.md, RAPPORT.md/.pdf (French), README-fr.md, REVIEW-REPORT.md, CHANGELOG-REVISION.md ├── results/revision-2026-10-17/ # outputs of 07_revision_analyses.py (csv/json, committed) ├── ref/ # MUN.csv — municipality → MRC → region (MAMH) ├── out// # sales.parquet, index_asof.npy, models_*.json, est_E*.npy, tables/, fig/, results.json — not committed ├── data/ # vraiprix-.db snapshot (node only) — not committed └── paper/ # main.tex (+ main_anon.tex) + sections/ + tables/ (generated) + figures/ → main.pdf ``` ## 🚀 Quick start ```bash rsync -a --exclude out --exclude data --exclude .git ./ M2U64:comparables-validation/ ssh M2U64 'cd ~/comparables-validation && CV_VERSION=2026-10-10 ./run_all.sh' # ≈ 30 min (or ./run_distributed.sh) ./make_pdf.sh # pull out/ + docs/, build RAPPORT.pdf python3 -I pipeline/07_revision_analyses.py # ≈ 80 s on the laptop: bootstrap, selection stability, R8, seasonality cd paper && python3 make_paper_tables.py && make && latexmk -pdf main_anon.tex # tables + numbers*.tex macros → main.pdf, main_anon.pdf ``` Variables: `CV_VERSION` (output folder), `CV_SAMPLE=3000` (target subsample for a dry run), `CV_PROCS` (processes, 20), `CV_BOOT` (bootstrap replications in `07_revision_analyses.py`, 999). Environment on M2U64: `~/uqo-eval-terrains/env` (Python 3.14, pandas, numpy, scipy, lightgbm, matplotlib). ## 🗃 Data Snapshot of the Vrai Prix production database (2026-10-10): 766,596 residential transactions and 3.75 M assessment-roll units. Single-family sales matched to the roll, cleaned by logged rules R0–R11 (`out//exclusions.csv`): 456,950 → **390,996** sales (2021-01-05 → 2026-09-29), split train 201,642 / validation 64,795 (2024) / test 124,559 (2025–2026). Median price $400,000. All numbers in the paper are macros in `paper/tables/numbers.tex` (regenerated by `paper/make_paper_tables.py` from `out//`) and `paper/tables/numbers_revision.tex` (regenerated by `pipeline/07_revision_analyses.py`). ## 🔬 Methodology 1. **Point-in-time index and quarterly hedonic grids** (`02_index.py`, `03_models.py`) — nothing after the valuation date is used. 2. **Backtest of 2,256 configurations** × benchmarks on 189,354 targets (`04_backtest.py`, sharded; `04b_gather.py`). 3. **Selection on 2024, evaluation on 2025–2026**; error surfaces, marginal effects of each rule, conditional rules by stratum (`05_analyze.py`). 4. **Decomposition** by liquidity, region, price, type; stability over time; prediction intervals (`06_report.py`, `paper/make_paper_tables.py`). 5. **Inference and audit of the selection step** (`07_revision_analyses.py`): cluster bootstrap (999 replications, three alternative cluster definitions), cluster-robust Diebold–Mariano-type t statistics, per-model confidence intervals, validation → test rank stability of all configurations, sensitivity to the R8 screen, seasonal bias by calendar quarter. ## 🖼 Figures & tables inventory | Figure | Content | |---|---| | `f1_mdape_vs_k` | error vs number of comparables, by adjustment | | `f2_surface_d_T` | error surface over radius and window | | `f3_ajustements` | contribution of adjustment types | | `f4_competition` | comparables vs hedonic / GBM / ratio / hybrid | | `f5_densite`, `f6_k_par_densite` | accuracy and optimal k by submarket liquidity | | `f7_indice` | descriptive price index, province and five regions | Tables (`paper/tables/*.tex`, 33 generated): descriptives, exclusions, selection, adjustments, grid, weighting, surfaces, competition (+ bootstrap CIs), paired differences (B = 999, t), cluster-definition sensitivity, density, regions, strata, stability, seasonal bias, intervals, conditional rules, hypotheses H2–H5, selection stability (top-10, near-optimal set), R8 sensitivity, regional index growth. ## ⚠️ Limitations - Adjustments simulated on observable characteristics (area, lot, age, storeys, attachment, assessed value); unobserved condition and quality adjustments made by appraisers are not modelled (within-municipality R² of the grid ≈ 0.36). - Single-family houses in Quebec only; results for condos/plexes not covered. - Registry prices may include non-arm's-length transfers surviving the price/assessment screen; sales dated by registration, publication lag ignored. - The seasonal bias of the comparables estimate (−2.4 pts in Q1, +1.6 pts in Q3 for M7) is documented but not corrected; the backtest was not re-run with alternative screening bounds for the comparable pool (target-side sensitivity only). ## 📝 Citation ```bibtex @techreport{boucher2026salescomparison, author = {Boucher, Simon-Pierre}, title = {Does the Sales Comparison Approach Actually Work? A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation}, institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, type = {Working Paper}, number = {16}, year = {2026}, month = {October}, note = {Version 1.1} } ``` ## 👤 Author & contact **Simon-Pierre Boucher** — Département des sciences administratives, Université du Québec en Outaouais (UQO), Gatineau, QC. 📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) ## 🗒 Notes - **Revision 2026-10-17 (v1.1)**: full scientific review by an editorial-board pass — see `docs/REVIEW-REPORT.md` (diagnosis, verdicts, remaining limitations, analyses recommended but not executed) and `docs/CHANGELOG-REVISION.md` (audit trail of every change, references added with DOIs, re-analyses executed). No headline estimate changed; the regional index figure, one incorrect regional-growth statement and one unsupported mechanism (GBM seasonality) were corrected. - Numbering: issued on 2026-10-10 as "Working Paper No. 11" (number already used by *Half a Million Prices, Twenty Models*); renumbered **16** on 2026-10-17. The SSRN upload 7626720 predates the renumbering and the revision. - Deviation from `Recherche-UQO/HARMONISATION.md` §2: the pipeline keeps `pipeline/` (not `scripts/`) and `out/` (not `results/`) because the same tree runs on M2U64 under `~/comparables-validation` with these paths; the laptop-side revision analyses write to `results/revision-2026-10-17/` (committed). - French protocol and report in `docs/` (`PROTOCOLE.md`, `RAPPORT.md/.pdf`, generated by `06_report.py` from the 2026-10-10 run and left unchanged); original French README in `docs/README-fr.md`. - Per-node copies of the working tree pulled on 2026-10-10 (`M*omparables-validation/`) were removed from the repository and archived in `Recherche-UQO/_archives/sales-comparison-node-copies-20261010/`.