# Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca

# Phase 6 (addendum) — prospective evaluation set (2026-10-27)

Status at the time of writing: data layer, bundle, runner option, tests, documentation, baselines and the E and F live runs complete; the G (2-round debate) live run still running in the background (resumable, capped). The numbers in §4 are the state when this report was written; re-run the status commands in §5 for the final counts and append them here.

# 1. Artefacts

# 2. Manifest and funnel

Fetch (read-only SSH, M3U96a live DB, 2026-10-27): 36,826 live rows dated ≥ 2026-07-01 → 1,444 new rows absent from the 2026-10-21 snapshot, dates 2026-10-06 … 2026-10-13, 1,434 roll units (every new row matched). Freeze digest recorded in the manifest: f0e9d6a94b0387f9….

Funnel: 1,444 → roll matched 1,444 → no WP 22 exclusion flag 1,251 (−193: land 96, non-residential 56, assessment-ratio outlier 26, same-parcel/same-date price conflict 8, shared parcel 5, multi-parcel candidate 2; duplicates of a snapshot sale 0) → class ∈ {house, condo, plex} 1,182 (−69) → coordinates / parcel id / price band / municipality lookup 1,182 (−0). 1,182 rows: house 856, condo 161, plex 165; Montréal CMA 459, Québec CMA 99, Outaouais 57, other 567; terciles 270 / 411 / 501; 321 with a prior snapshot sale; 0 new constructions; roll in force 2024-07-01 (391), 2023-07-01 (595), 2022-07-01 (192), earlier (4). Class source: roll 1,175, provider 7.

Subsample: stratified_subsample(..., n_by_split={"prospective": 40}), seed 20261027 → 40 rows, 18 strata (e.g. Montréal CMA|house|3: 7, other|house|1: 6, other|house|2: 6).

# 3. Runner change (diff summary)

RunSpec.data_dir: Path | None = None; data_base(synthetic, data_dir=None); load_analytic, select_subsample, feature_backend take data_dir; run_arm_async passes spec.data_dir; manifest gains data_dir and reports the actual data_source / geo_cache paths. CLI: --data-dir. Frozen prompts, configs, roles, protocols and aggregation untouched (check_freeze scope does not include experiments/run_arm.py).

# 4. Runs (state when this report was written — background jobs still running)

Chained by logs/prospective_runs.sh (log logs/prospective_runs.log), concurrency 3 because eight other live runs (final, final_small, holdout, validation ablations) share the keys:

Mock dry run on the real bundle before going live: packets of prospective subjects carry no price, comparables are dated ≤ cutoff (sale date − 1), market index and geo context present.

Baselines (scripts/62_prospective_baselines.py --trials 10, 1,090 s, results/runs/prospective_baselines/): feature matrix on the bundle 452,624 rows in 62 s; as-of invariance max |Δ| = 0.0 on train/validation rows vs data/processed/feature_matrix.parquet; hyper-parameters, repeat-sales weight (0.15410) and conformal scale (1.07583) identical to the main run. Predictions: 7,092 rows (1,182 × 6 arms, split = "prospective"). Console summary on the 1,182 prospective sales (absolute log error; the formal evaluation with clustered uncertainty is 20_evaluate.py):

# 5. Exact commands

python3 scripts/60_fetch_prospective.py
python3 scripts/61_build_prospective.py                 # --date 2026-10-27 ; --no-bundle
zsh logs/prospective_runs.sh                            # E → F → G, resumable (see docs/prospective_set.md §6)
python3 scripts/62_prospective_baselines.py --trials 10
python3 -m pytest tests/test_experiments_datadir.py tests/test_agents_run_arm.py -q
# status
grep -E "wrote|SPEND|Traceback|chain finished" logs/prospective_runs.log
for r in E_single_llm F_independent G_debate_r2; do ls results/runs/prospective_${r}_mixed_random_mid/transcripts | wc -l; done
python3 -c "import json,glob;[print(f, sum(json.loads(l)['cost_usd'] for l in open(f))) for f in glob.glob('results/ledger/prospective_*.jsonl')]"
tail -12 logs/62_prospective_baselines.log
# evaluation (after the runs)
python3 scripts/20_evaluate.py --split prospective --splits data/processed/prospective_bundle/splits.parquet

# 6. Decisions and open points

• Geography by municipal-code lookup (not the snapshot's spatial join); boroughs imputed from the snapshot's municipality × 1 km cell mode (24 % filled). Documented in docs/prospective_set.md §3.
• Prospective terciles use the snapshot's class cut points (the snapshot's own terciles are rank-based within the table; the two coincide at the cut points).
• Sales dated 2026-10-06/07 fall inside the holdout window but were absent from the snapshot: they are prospective by construction (registered after the snapshot), not holdout rows.
• The prospective sample covers one week of registrations and 40 LLM subjects: a consistency check of the test/holdout results, not a primary endpoint. Not evaluated yet (20_evaluate.py --split prospective once the runs finish). No git commit was made.