1# Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca2# Phase 6 (addendum) — prospective evaluation set (2026-10-27)34Status at the time of writing: data layer, bundle, runner option, tests, documentation, baselines and the E and F live5runs **complete**; the G (2-round debate) live run **still running in the background** (resumable, capped). The numbers in §4 are the state6when this report was written; re-run the status commands in §5 for the final counts and append them here.78## 1. Artefacts910| artefact | content |11|---|---|12| `data/raw/prospective_2026-10-27.parquet`, `prospective_units_2026-10-27.parquet`, `.manifest.json` | raw fetch (gitignored, contains addresses — never printed) |13| `scripts/61_build_prospective.py` | WP 22 cleaning re-implemented on the new rows + WP 26 funnel → analytic contract; builds the runner bundle |14| `data/processed/prospective_analytic.parquet` | 1,182 rows, `ANALYTIC_COLUMNS`, `validate_analytic` passes |15| `results/data/prospective_summary.json` | manifest, funnel, exclusion flags, class / region / tercile counts, bundle summary |16| `data/processed/prospective_bundle/{analytic,splits,llm_subsamples,geo_features}.parquet` | snapshot ∪ prospective (633,048 rows); split `prospective`; 40-row stratified subsample (18 strata); geo features |17| `src/agentic_valuation/experiments/run_arm.py`, `scripts/30_run_arm.py` | `RunSpec.data_dir` / `--data-dir` (backward compatible; not among the freeze-hashed files) |18| `tests/test_experiments_datadir.py` | 4 tests (defaults untouched; loads redirected; mock end-to-end run on a bundle; default manifest unchanged) — 8/8 pass with `tests/test_agents_run_arm.py` |19| `scripts/62_prospective_baselines.py` | refit A/B/C/D (+CatBoost/XGBoost) with the 10_train protocol, predict the prospective rows |20| `docs/prospective_set.md` | what, when, funnel, column mapping, bundle, extension procedure |2122## 2. Manifest and funnel2324Fetch (read-only SSH, M3U96a live DB, 2026-10-27): 36,826 live rows dated ≥ 2026-07-01 → **1,444 new rows** absent25from the 2026-10-21 snapshot, dates **2026-10-06 … 2026-10-13**, **1,434 roll units** (every new row matched).26Freeze digest recorded in the manifest: `f0e9d6a94b0387f9…`.2728Funnel: 1,444 → roll matched 1,444 → no WP 22 exclusion flag 1,251 (−193: land 96, non-residential 56,29assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2;30duplicates of a snapshot sale 0) → class ∈ {house, condo, plex} 1,182 (−69) → coordinates / parcel id / price band /31municipality lookup 1,182 (−0). **1,182 rows**: house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99,32Outaouais 57, other 567; terciles 270 / 411 / 501; 321 with a prior snapshot sale; 0 new constructions; roll in force332024-07-01 (391), 2023-07-01 (595), 2022-07-01 (192), earlier (4). Class source: roll 1,175, provider 7.3435Subsample: `stratified_subsample(..., n_by_split={"prospective": 40})`, seed 20261027 → **40 rows, 18 strata**36(e.g. Montréal CMA|house|3: 7, other|house|1: 6, other|house|2: 6).3738## 3. Runner change (diff summary)3940`RunSpec.data_dir: Path | None = None`; `data_base(synthetic, data_dir=None)`; `load_analytic`, `select_subsample`,41`feature_backend` take `data_dir`; `run_arm_async` passes `spec.data_dir`; manifest gains `data_dir` and reports the42actual `data_source` / `geo_cache` paths. CLI: `--data-dir`. Frozen prompts, configs, roles, protocols and43aggregation untouched (`check_freeze` scope does not include `experiments/run_arm.py`).4445## 4. Runs (state when this report was written — background jobs still running)4647Chained by `logs/prospective_runs.sh` (log `logs/prospective_runs.log`), concurrency 3 because eight other live runs48(final, final_small, holdout, validation ablations) share the keys:4950| run id | arm | n | cap | state |51|---|---|---|---|---|52| `prospective_E_single_llm_mixed_random_mid` | E | 40 | 10 USD | **done**: 40 transcripts, 280 prediction rows, 40 calls, 0.82 USD, 0 failures, 0 turn errors |53| `prospective_F_independent_mixed_random_mid` | F | 40 | 30 USD | **done**: 40 transcripts, 280 prediction rows, 480 calls, 9.36 USD, 0 failures (1 turn error, retried) |54| `prospective_G_debate_r2_mixed_random_mid` | G r2 | 40 | 90 USD | running (143 calls, 3.36 USD at the time of writing; ≈ 1.4 USD/property expected) |5556Mock dry run on the real bundle before going live: packets of prospective subjects carry no price, comparables are57dated ≤ cutoff (sale date − 1), market index and geo context present.5859Baselines (`scripts/62_prospective_baselines.py --trials 10`, 1,090 s, `results/runs/prospective_baselines/`): feature60matrix on the bundle 452,624 rows in 62 s; **as-of invariance max |Δ| = 0.0** on train/validation rows vs61`data/processed/feature_matrix.parquet`; hyper-parameters, repeat-sales weight (0.15410) and conformal scale (1.07583)62identical to the main run. Predictions: 7,092 rows (1,182 × 6 arms, `split = "prospective"`). Console summary on the631,182 prospective sales (absolute log error; the formal evaluation with clustered uncertainty is `20_evaluate.py`):6465| arm | mean | median |66|---|---|---|67| A_hedonic | 0.2186 | 0.1430 |68| B_hierarchical | 0.2069 | 0.1252 |69| C_gbm | 0.2179 | 0.1452 |70| C_catboost | 0.2146 | 0.1346 |71| C_xgboost | 0.2218 | 0.1411 |72| D_avm | 0.2024 | 0.1297 |7374## 5. Exact commands7576```77python3 scripts/60_fetch_prospective.py78python3 scripts/61_build_prospective.py # --date 2026-10-27 ; --no-bundle79zsh logs/prospective_runs.sh # E → F → G, resumable (see docs/prospective_set.md §6)80python3 scripts/62_prospective_baselines.py --trials 1081python3 -m pytest tests/test_experiments_datadir.py tests/test_agents_run_arm.py -q82# status83grep -E "wrote|SPEND|Traceback|chain finished" logs/prospective_runs.log84for r in E_single_llm F_independent G_debate_r2; do ls results/runs/prospective_${r}_mixed_random_mid/transcripts | wc -l; done85python3 -c "import json,glob;[print(f, sum(json.loads(l)['cost_usd'] for l in open(f))) for f in glob.glob('results/ledger/prospective_*.jsonl')]"86tail -12 logs/62_prospective_baselines.log87# evaluation (after the runs)88python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet89```9091## 6. Decisions and open points9293* Geography by municipal-code lookup (not the snapshot's spatial join); boroughs imputed from the snapshot's94 municipality × 1 km cell mode (24 % filled). Documented in docs/prospective_set.md §3.95* Prospective terciles use the snapshot's class cut points (the snapshot's own terciles are rank-based within the96 table; the two coincide at the cut points).97* Sales dated 2026-10-06/07 fall inside the holdout window but were absent from the snapshot: they are prospective by98 construction (registered after the snapshot), not holdout rows.99* The prospective sample covers one week of registrations and 40 LLM subjects: a consistency check of the100 test/holdout results, not a primary endpoint. Not evaluated yet (`20_evaluate.py --split prospective` once the runs101 finish). No git commit was made.102