1<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->2# Evaluation protocol (WP 26) — definitions cited in the paper34Code: `src/agentic_valuation/evaluation/` (`metrics.py`, `inference.py`, `calibration.py`, `stratify.py`, `faithfulness.py`, `tables.py`);5driver `scripts/20_evaluate.py`; figures `scripts/21_figures.py`; tests `tests/test_evaluation.py`.6Every number in `results/evaluation/*.csv`, `results/tables/*.tex` and `figures/*.pdf` is produced by these scripts from7`results/runs/<run_id>/predictions.parquet` (contract in `docs/architecture.md` §1) joined to `data/processed/analytic.parquet`8on `tx_id`. Run ids beginning with `_` (e.g. `_fake_demo`) are never collected unless `--include-fake` is passed.910Notation. Property $i$ has registered sale price $P_i$ (CAD) and date $t_i$. An arm produces a point estimate $\hat P_i$ and11quantiles $q_{i,\tau}$ for $\tau \in \{0.05, 0.25, 0.50, 0.75, 0.95\}$ (quantiles may be absent for arms without intervals).12$P_i$ is a noisy proxy of market value, not the latent fundamental value; all metrics below are prediction-error metrics.1314## 1. Per-record errors1516* Log error: $e_i = \log \hat P_i - \log P_i$. Absolute log error $|e_i|$ (≈ relative error for small values).17* CAD error: $|\hat P_i - P_i|$. Absolute percentage error: $|\hat P_i / P_i - 1|$.18* Severe mispricing: $\mathbb 1\{|e_i| > \log 1.25\}$ (more than 25 % off in ratio terms, symmetric in log space).19* Overvaluation: $\mathbb 1\{\hat P_i/P_i - 1 > 0.25\}$; undervaluation: $\mathbb 1\{\hat P_i/P_i - 1 < -0.25\}$ (percent space, as a20 lender would read them; note under- is $e_i < \log 0.75$, which is stricter than $-\log 1.25$).2122## 2. Accuracy summaries (per arm, over the evaluated records)2324| name in CSV | definition |25|---|---|26| `mean_abs_log_error` (**primary**) | $\frac1n\sum_i |e_i|$ |27| `mae_cad` | $\frac1n\sum_i |\hat P_i - P_i|$ |28| `mdape` | $\operatorname{median}_i |\hat P_i/P_i - 1|$ |29| `rmse_log` | $\sqrt{\frac1n\sum_i e_i^2}$ |30| `p90_abs_log_error`, `p95_abs_log_error` | empirical 90th / 95th percentiles of $|e_i|$ (NumPy linear interpolation) |31| `bias_log` | $\bar e = \frac1n\sum_i e_i$ |32| `severe_rate`, `over_rate`, `under_rate` | means of the indicators of §1 |33| `cost_usd_mean`, `cost_usd_total`, `latency_s_mean`, `tokens_mean` | per-property means (and total cost) of the recorded API cost, wall-clock latency and tokens; `NaN` for baselines that do not record them |3435## 3. Interval metrics3637### 3.1 Raw coverage (reported as primary calibration numbers)38* 50 %: $\frac1n\sum_i \mathbb 1\{q_{i,.25} \le P_i \le q_{i,.75}\}$ (`coverage_50`)39* 90 %: $\frac1n\sum_i \mathbb 1\{q_{i,.05} \le P_i \le q_{i,.95}\}$ (`coverage_90`)4041### 3.2 Interpolated 80 % and 95 % coverage42The five quantiles do not bracket an 80 % or a 95 % interval. We fit, per record, a normal distribution in log space to the43five quantiles by least squares on the probit grid: $\log q_{i,\tau} \approx \mu_i + \sigma_i z_\tau$, $z_\tau = \Phi^{-1}(\tau)$,44giving $\mu_i = \frac15\sum_\tau \log q_{i,\tau}$ and $\sigma_i = \sum_\tau z_\tau(\log q_{i,\tau} - \mu_i)/\sum_\tau z_\tau^2$45(truncated at 0). The $1-\alpha$ interval is $\exp(\mu_i \pm z_{1-\alpha/2}\sigma_i)$ (`coverage_80`, `coverage_95`).46`coverage_90_normal_fit` is the same construction at 90 % and is reported next to the raw `coverage_90` as a diagnostic of the47fit (identical when the quantiles are exactly log-normal). The 80/95 % numbers are therefore model-based interpolations and are48labelled as such in the tables.4950### 3.3 Width51`mean_log_width_90` $= \frac1n\sum_i (\log q_{i,.95} - \log q_{i,.05})$; `mean_log_width_50` analogously with $q_{.25}, q_{.75}$.52`sharpness()` also reports the median log width and the mean relative width $(q_{.95}-q_{.05})/q_{.50}$.5354### 3.4 Pinball loss and CRPS (log scale)55With $y_i = \log P_i$ and $\rho_\tau(u) = u(\tau - \mathbb 1\{u<0\})$:56`pinball_log` $= \frac1n\sum_i \frac15\sum_{\tau} \rho_\tau(y_i - \log q_{i,\tau})$.57`crps_log` $= \frac1n\sum_i\, 2\int_{0.05}^{0.95} \rho_\tau(y_i - \log q_{i,\tau})\,d\tau$ approximated by the trapezoidal rule on the58five-point $\tau$ grid. Because the integral over $\tau\in(0,0.05)\cup(0.95,1)$ is unobserved it is omitted: the quantity is a59lower-biased approximation of the CRPS, consistent across arms. `crps_log_normal_fit` is the closed-form CRPS of the fitted60normal of §3.2 (Gneiting & Raftery 2007, eq. 21), reported as a robustness check. Both are in log units.6162### 3.5 PIT and reliability63PIT$_i = F_i(\log P_i)$ where $F_i$ is interpolated in probit space: $\Phi^{-1}(F_i)$ is piecewise linear in $\log P$ between the64knots $(\log q_{i,\tau}, \Phi^{-1}(\tau))$; outside $[q_{.05}, q_{.95}]$ the tails of the fitted normal (§3.2) are used. This is exact for65log-normal forecasts. A calibrated arm has a uniform PIT histogram (`pit.csv`, `fig_pit.pdf`). Reliability of each quantile:66empirical $\frac1n\sum_i \mathbb 1\{P_i \le q_{i,\tau}\}$ versus nominal $\tau$ (`coverage.csv`, `kind = quantile`).67Conditional coverage: §3.1 recomputed within subgroups (region group, property class, price tercile) subject to the cell rule of §6.6869### 3.6 Conformal rescaling (`conformalise`, `apply_conformal`)70Split conformal with an asymmetric scaled-residual score. With $c_i = \log q_{i,.50}$, $\ell_i = c_i - \log q_{i,.05}$,71$u_i = \log q_{i,.95} - c_i$: $s_i = \max\{(c_i - y_i)/\ell_i,\ (y_i - c_i)/u_i\}$ on the **validation** records. The factor72$k_{90}$ is the $\lceil (n+1)\,0.90\rceil$-th smallest $s_i$; the test interval becomes $[\exp(c_i - k_{90}\ell_i), \exp(c_i + k_{90}u_i)]$.73Same construction for the 50 % interval with $q_{.25}, q_{.75}$. Under exchangeability the marginal coverage on test is $\ge 0.90$.74Factors are estimated on validation only and frozen (no test-set information). Point and median are never changed.7576## 4. Paired inference between arms7778All contrasts are **paired on the same `tx_id`s**. For arms A and B and a summary statistic $T(\cdot)$ computed on per-record79losses, $\Delta = T(B) - T(A)$; for a loss (all accuracy metrics) $\Delta < 0$ favours B. Pre-registered pairs80(`docs/preregistration.md`): (D, G2), (F, G2), (J, G2), (I, G2) on mean $|e|$ (Holm family), (I, G2) on P95 $|e|$; all other81pairs/metrics are exploratory.8283### 4.1 Block bootstrap84Blocks = `mun_code × calendar month` of the sale (fallback `mun_code`, then i.i.d. records when neither exists). $B = 2000$85resamples of blocks with replacement (seed 20261027), both arms resampled with the identical block draw.8695 % percentile CI = (2.5th, 97.5th) percentiles of $\Delta^*$. Bootstrap p-value: $p = \frac1B\sum_b \mathbb 1\{|\Delta^*_b - \hat\Delta| \ge |\hat\Delta|\}$87(bootstrap distribution recentred at the null; floored at $1/(B+1)$). The bootstrap SE is the SD of $\Delta^*$.88Any metric of §2–§3 can be used as $T$ (`metric=` name or callable on a per-record frame).8990### 4.2 Holm correction91For a family of $m$ p-values ordered $p_{(1)} \le \dots \le p_{(m)}$, adjusted $\tilde p_{(k)} = \max_{j\le k}\min\{1, (m-j+1)p_{(j)}\}$.92Applied within each metric across the pre-registered pairs (`p_holm`). Verified against `statsmodels.multipletests(method="holm")`.9394### 4.3 Diebold–Mariano95On the loss differential $d_i = L^B_i - L^A_i$ ($L = |e|$, or squared error for RMSE, or pinball) **sorted by sale date**,96$DM = \bar d / \sqrt{\hat\omega / n}$ where $\hat\omega$ is the Newey–West long-run variance with Bartlett kernel and bandwidth97$h = \lfloor 4(n/100)^{2/9} \rfloor$, with the Harvey–Leybourne–Newbold (1997) small-sample factor and a $t_{n-1}$ reference.98**Assumption**: $d_i$ is a covariance-stationary series in time order; sales on the same day are ordered arbitrarily (stable sort),99so the HAC correction captures calendar dependence only. The DM test is reported as a complement to the block bootstrap, which100remains the primary inference tool (it respects the municipality × month cluster structure).101102## 5. Rounds, cost and Pareto103The rounds curve (`fig_rounds.pdf`) reads `results/evaluation/rounds.csv` (`round, value[, ci_lo, ci_hi, arm, metric]`) produced by104the experiments layer; it is absent until the rounds grid 0/1/2/4 has been run. Cost–accuracy (`fig_cost_accuracy.pdf`) plots105`cost_usd_mean` (log scale; baselines with no API cost are plotted at a floor) against mean $|e|$; the Pareto frontier joins106non-dominated arms.107108## 6. Strata and the minimum-cell rule109Standard strata: `region_group` (Montréal / Québec / Outaouais / other, from `region_name` when the analytic table does not carry110it), `urban_class`, `property_class`, `price_tercile` (terciles of $P$ over the truth table), `year_built_decade`111(<1950, 1950s … 2000s, 2010+), `density_tercile` (terciles of `n_sales_3km_12m` → thin / mid / liquid, when the column exists).112A cell with fewer than **30** records is suppressed (value `NaN`, `suppressed = True`, printed `--`). Tables are long113(`arm, committee, aggregation, stratum, level, n, metric, value, suppressed`).114115## 7. Faithfulness audit (transcripts)116Operates on `DebateTranscript` + `EvidencePacket` JSON of each run. Per estimate/revision turn:117* `cited_share_valid`: share of `evidence_ids_used` present in `packet.evidence_ids()` (H6 threshold: ≥ 0.90 on average);118* `adj_share_within_40`: share of `adjustments` with $|\text{adjusted}/\text{raw} - 1| \le 0.40$ (raw = comparable price in the packet);119 `share_turns_adj_ok` = share of turns where all adjustments pass (H6: ≥ 0.80);120* `point_in_hull`: point estimate within $[\min \text{adjusted} \times 0.75,\ \max \text{adjusted} \times 1.25]$; otherwise the log121 deviation from the nearest bound and whether `key_assumptions` is non-empty (`hull_deviation_justified`);122* `quantile_sane`: positive, non-decreasing, point within $[q_{.05}, q_{.95}]$, and $q_{.95}/q_{.05} \in [1.01, 4]$;123* `summary_words` and `summary_within_limit` (≤ 120 words);124* `packet_hash_match`: `packet.content_hash() == transcript.packet_hash`.125126Committee level, per transcript and round $r$ with log estimates $x_{k,r}$ of roles $k$ and round-0 median $m_0$:127* dispersion `iqr_log` $= \operatorname{IQR}_k(x_{k,r})$; `convergence` $= \operatorname{IQR}_r / \operatorname{IQR}_0$;128* `anchoring_index` $= \operatorname{corr}_k\big(x_{k,r} - x_{k,0},\ m_0 - x_{k,0}\big)$ (1 = every revision is a move toward the initial129 committee median, i.e. pure herding; ≈ 0 = revisions unrelated to the initial consensus);130* `share_toward_median` $= \frac1K\sum_k \mathbb 1\{|x_{k,r} - m_0| < |x_{k,0} - m_0|\}$.131`faithfulness.csv` aggregates per run × arm × committee (final-round committee statistics; `committee_rounds.csv` keeps all rounds).132133## 8. Outputs and tables134`results/evaluation/`: `summary.csv` (all arm × committee × aggregation × run_id), `summary_main.csv` (one row per arm: preferred135configuration `--main-committee mixed_random --main-aggregation median`, else the baseline configuration, else the first available —136column `main_selection` records which), `records.parquet`, `contrasts.csv`, `coverage.csv`, `pit.csv`, `strata.csv`,137`faithfulness*.csv`, `committee_rounds.csv`, `manifest.json` (runs, truth path, seed, $B$, timestamp).138`results/tables/*.tex` are booktabs `tabular` fragments with `\num{}` cells, no caption, last line `\end{tabular}`; a `.csv` twin139sits next to each fragment. Figures are PDF without titles, colour-blind-safe palette with markers/hatches as secondary encoding.140141## 9. Commands142```bash143python3 -m pytest tests/test_evaluation.py -q144python3 scripts/29_fake_demo_predictions.py # FAKE run under results/runs/_fake_demo (skipped by default)145python3 scripts/20_evaluate.py --synthetic --include-fake --B 500 # smoke test of the pipeline on the fake run146python3 scripts/20_evaluate.py # real evaluation (analytic.parquet + splits.parquet, B = 2000)147python3 scripts/21_figures.py148```149