Research plan — Can Autonomous AI Agents Discover the Fundamental Value of Real Estate?
UQO Working Paper No. 26. Protocol in CLAUDE.md; this file is the operational plan with the concrete choices made on 2026-10-27.
Object of study
Market-consistent value $V^*{i,t} = E[P{i,t} \mid \mathcal{I}_t]$ of a Quebec residential property at its sale date, conditional on an explicit, timestamped information set $\mathcal{I}t$ (attributes, roll, comparables sold before $t$, local market state). The realised price $P{i,t}$ is a noisy, selected proxy; the primary endpoint is nonetheless the absolute log error $|\log \hat P - \log P|$ because it is the only universally observed target. Section 9 of the paper triangulates with repeat sales, rents (RDL) and user cost.
Design in one table
| Element | Choice |
|---|---|
| Data | Vrai Prix registered sales 2021-01 → 2026-10 (snapshot 2026-10-21), house/condo/plex, roll-matched, geocoded; ≈ 630 k headline sales |
| Splits | train 2021-01 → 2024-06; validation 2024-07 → 2025-03; test 2025-04 → 2025-12; holdout 2026-01 → 2026-10-07 (once); prospective = registered after the freeze; spatial holdout = Outaouais, Abitibi-Témiscamingue, Bas-Saint-Laurent withheld from training |
| Baselines | A hedonic ridge (municipality × quarter FE, 1 km grid FE, attributes); B hierarchical hedonic + repeat-sales adjustment; C LightGBM (CatBoost/XGBoost robustness); D AVM = C + comparable features + spatial-temporal lags + conformalised quantile intervals |
| LLM arms | E single generalist; F 12 independent roles; G structured debate (1, 2, 4 rounds); H debate + blinded skeptic + evidence verification; I placebo roles; J cost-matched single agent (k samples + self-refine) |
| Providers | mid tier cost-matched: claude-sonnet-5-5 / gpt-6.1-sol; committees all-Anthropic, all-OpenAI, mixed-random |
| Subsamples for LLM arms | stratified (region group × class × price tercile): validation 240, test 600, holdout 300 (cost) |
| Primary endpoint | mean absolute log error on test (paired differences vs D, clustered bootstrap by municipality × month) |
| Secondary | MAE CAD, MdAPE, RMSE log, P90/P95 abs log error, 50/80/90/95 % coverage, width, pinball loss, severe mispricing rate (> 25 %), cost, latency, faithfulness scores |
| Inference | block bootstrap (municipality × month), Holm correction over the pre-registered family of 6 primary contrasts |
Phases and gates
- Inventory and gap analysis (
reports/phase_00.md) — done 2026-10-27. - Literature and novelty map (
docs/literature_map.md,paper/references.bib, all DOIs verified). - Preregistration (
docs/preregistration.md), estimands, theory (paper/sections/theory.tex), data dictionary (docs/data_dictionary.md). - Analytic layer, splits, features, comparables, leakage tests, tuned baselines (
scripts/01–14). - Agents: 5-role prototype → 12 roles → 20 roles; mock smoke test (
make smoke). - Pilot (24 properties) → validation runs (240) → aggregation weights → freeze (
results/freeze.jsonwith prompt hashes, model ids, configs, git commit). - Test (600) and holdout (300) evaluated once; prospective set fetched and evaluated.
- Human benchmark: pending (ethics). Welfare model (illustrative). Robustness: spatial holdout, rounds grid, provider committees, run-to-run variance, exclusion sensitivity.
- Manuscript, appendices, reproducibility bundle, limitations register.
Budget (list prices, configs/budgets.yaml)
Assuming ≈ 3.5 k input + 0.7 k output tokens per agent-turn at the mid tier ($2/$10): one property under G2 ≈ 12 roles × 3 turns × (3.5k·2 + 0.7k·10)/1e6 ≈ $0.50; F ≈ $0.17; E ≈ $0.015. Validation tier (240 properties × all arms × 3 committees) ≈ $150–250; final tier (600) ≈ $300–500; holdout ≈ $100–150. Prompt caching of the shared packet lowers input cost. Hard caps enforced by AV_MAX_SPEND_USD and the ledger.
Not done / pending
Human appraiser arm (ethics certificate required); mortgage-rate series (not in snapshot; can be added from the Bank of Canada if time permits); Immo-Ka listing-linked analysis limited to the prospective set.