# Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
# Phase 6 (addendum) — prospective evaluation set (2026-10-27)

Status at the time of writing: data layer, bundle, runner option, tests, documentation, baselines and the E and F live
runs **complete**; the G (2-round debate) live run **still running in the background** (resumable, capped). The numbers in §4 are the state
when this report was written; re-run the status commands in §5 for the final counts and append them here.

## 1. Artefacts

| artefact | content |
|---|---|
| `data/raw/prospective_2026-10-27.parquet`, `prospective_units_2026-10-27.parquet`, `.manifest.json` | raw fetch (gitignored, contains addresses — never printed) |
| `scripts/61_build_prospective.py` | WP 22 cleaning re-implemented on the new rows + WP 26 funnel → analytic contract; builds the runner bundle |
| `data/processed/prospective_analytic.parquet` | 1,182 rows, `ANALYTIC_COLUMNS`, `validate_analytic` passes |
| `results/data/prospective_summary.json` | manifest, funnel, exclusion flags, class / region / tercile counts, bundle summary |
| `data/processed/prospective_bundle/{analytic,splits,llm_subsamples,geo_features}.parquet` | snapshot ∪ prospective (633,048 rows); split `prospective`; 40-row stratified subsample (18 strata); geo features |
| `src/agentic_valuation/experiments/run_arm.py`, `scripts/30_run_arm.py` | `RunSpec.data_dir` / `--data-dir` (backward compatible; not among the freeze-hashed files) |
| `tests/test_experiments_datadir.py` | 4 tests (defaults untouched; loads redirected; mock end-to-end run on a bundle; default manifest unchanged) — 8/8 pass with `tests/test_agents_run_arm.py` |
| `scripts/62_prospective_baselines.py` | refit A/B/C/D (+CatBoost/XGBoost) with the 10_train protocol, predict the prospective rows |
| `docs/prospective_set.md` | what, when, funnel, column mapping, bundle, extension procedure |

## 2. Manifest and funnel

Fetch (read-only SSH, M3U96a live DB, 2026-10-27): 36,826 live rows dated ≥ 2026-07-01 → **1,444 new rows** absent
from the 2026-10-21 snapshot, dates **2026-10-06 … 2026-10-13**, **1,434 roll units** (every new row matched).
Freeze digest recorded in the manifest: `f0e9d6a94b0387f9…`.

Funnel: 1,444 → roll matched 1,444 → no WP 22 exclusion flag 1,251 (−193: land 96, non-residential 56,
assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2;
duplicates of a snapshot sale 0) → class ∈ {house, condo, plex} 1,182 (−69) → coordinates / parcel id / price band /
municipality lookup 1,182 (−0). **1,182 rows**: house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99,
Outaouais 57, other 567; terciles 270 / 411 / 501; 321 with a prior snapshot sale; 0 new constructions; roll in force
2024-07-01 (391), 2023-07-01 (595), 2022-07-01 (192), earlier (4). Class source: roll 1,175, provider 7.

Subsample: `stratified_subsample(..., n_by_split={"prospective": 40})`, seed 20261027 → **40 rows, 18 strata**
(e.g. Montréal CMA|house|3: 7, other|house|1: 6, other|house|2: 6).

## 3. Runner change (diff summary)

`RunSpec.data_dir: Path | None = None`; `data_base(synthetic, data_dir=None)`; `load_analytic`, `select_subsample`,
`feature_backend` take `data_dir`; `run_arm_async` passes `spec.data_dir`; manifest gains `data_dir` and reports the
actual `data_source` / `geo_cache` paths. CLI: `--data-dir`. Frozen prompts, configs, roles, protocols and
aggregation untouched (`check_freeze` scope does not include `experiments/run_arm.py`).

## 4. Runs (state when this report was written — background jobs still running)

Chained by `logs/prospective_runs.sh` (log `logs/prospective_runs.log`), concurrency 3 because eight other live runs
(final, final_small, holdout, validation ablations) share the keys:

| run id | arm | n | cap | state |
|---|---|---|---|---|
| `prospective_E_single_llm_mixed_random_mid` | E | 40 | 10 USD | **done**: 40 transcripts, 280 prediction rows, 40 calls, 0.82 USD, 0 failures, 0 turn errors |
| `prospective_F_independent_mixed_random_mid` | F | 40 | 30 USD | **done**: 40 transcripts, 280 prediction rows, 480 calls, 9.36 USD, 0 failures (1 turn error, retried) |
| `prospective_G_debate_r2_mixed_random_mid` | G r2 | 40 | 90 USD | running (143 calls, 3.36 USD at the time of writing; ≈ 1.4 USD/property expected) |

Mock dry run on the real bundle before going live: packets of prospective subjects carry no price, comparables are
dated ≤ cutoff (sale date − 1), market index and geo context present.

Baselines (`scripts/62_prospective_baselines.py --trials 10`, 1,090 s, `results/runs/prospective_baselines/`): feature
matrix on the bundle 452,624 rows in 62 s; **as-of invariance max |Δ| = 0.0** on train/validation rows vs
`data/processed/feature_matrix.parquet`; hyper-parameters, repeat-sales weight (0.15410) and conformal scale (1.07583)
identical to the main run. Predictions: 7,092 rows (1,182 × 6 arms, `split = "prospective"`). Console summary on the
1,182 prospective sales (absolute log error; the formal evaluation with clustered uncertainty is `20_evaluate.py`):

| arm | mean | median |
|---|---|---|
| A_hedonic | 0.2186 | 0.1430 |
| B_hierarchical | 0.2069 | 0.1252 |
| C_gbm | 0.2179 | 0.1452 |
| C_catboost | 0.2146 | 0.1346 |
| C_xgboost | 0.2218 | 0.1411 |
| D_avm | 0.2024 | 0.1297 |

## 5. Exact commands

```
python3 scripts/60_fetch_prospective.py
python3 scripts/61_build_prospective.py                 # --date 2026-10-27 ; --no-bundle
zsh logs/prospective_runs.sh                            # E → F → G, resumable (see docs/prospective_set.md §6)
python3 scripts/62_prospective_baselines.py --trials 10
python3 -m pytest tests/test_experiments_datadir.py tests/test_agents_run_arm.py -q
# status
grep -E "wrote|SPEND|Traceback|chain finished" logs/prospective_runs.log
for r in E_single_llm F_independent G_debate_r2; do ls results/runs/prospective_${r}_mixed_random_mid/transcripts | wc -l; done
python3 -c "import json,glob;[print(f, sum(json.loads(l)['cost_usd'] for l in open(f))) for f in glob.glob('results/ledger/prospective_*.jsonl')]"
tail -12 logs/62_prospective_baselines.log
# evaluation (after the runs)
python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet
```

## 6. Decisions and open points

* Geography by municipal-code lookup (not the snapshot's spatial join); boroughs imputed from the snapshot's
  municipality × 1 km cell mode (24 % filled). Documented in docs/prospective_set.md §3.
* Prospective terciles use the snapshot's class cut points (the snapshot's own terciles are rank-based within the
  table; the two coincide at the cut points).
* Sales dated 2026-10-06/07 fall inside the holdout window but were absent from the snapshot: they are prospective by
  construction (registered after the snapshot), not holdout rows.
* The prospective sample covers one week of registrations and 40 LLM subjects: a consistency check of the
  test/holdout results, not a primary endpoint. Not evaluated yet (`20_evaluate.py --split prospective` once the runs
  finish). No git commit was made.