# Evaluation protocol (WP 26) — definitions cited in the paper

Code: src/agentic_valuation/evaluation/ (metrics.py, inference.py, calibration.py, stratify.py, faithfulness.py, tables.py); driver scripts/20_evaluate.py; figures scripts/21_figures.py; tests tests/test_evaluation.py. Every number in results/evaluation/*.csv, results/tables/*.tex and figures/*.pdf is produced by these scripts from results/runs/<run_id>/predictions.parquet (contract in docs/architecture.md §1) joined to data/processed/analytic.parquet on tx_id. Run ids beginning with _ (e.g. _fake_demo) are never collected unless --include-fake is passed.

Notation. Property $i$ has registered sale price $P_i$ (CAD) and date $t_i$. An arm produces a point estimate $\hat P_i$ and quantiles $q_{i,\tau}$ for $\tau \in {0.05, 0.25, 0.50, 0.75, 0.95}$ (quantiles may be absent for arms without intervals). $P_i$ is a noisy proxy of market value, not the latent fundamental value; all metrics below are prediction-error metrics.

# 1. Per-record errors

• Log error: $e_i = \log \hat P_i - \log P_i$. Absolute log error $|e_i|$ (≈ relative error for small values).
• CAD error: $|\hat P_i - P_i|$. Absolute percentage error: $|\hat P_i / P_i - 1|$.
• Severe mispricing: $\mathbb 1{|e_i| > \log 1.25}$ (more than 25 % off in ratio terms, symmetric in log space).
• Overvaluation: $\mathbb 1{\hat P_i/P_i - 1 > 0.25}$; undervaluation: $\mathbb 1{\hat P_i/P_i - 1 < -0.25}$ (percent space, as a lender would read them; note under- is $e_i < \log 0.75$, which is stricter than $-\log 1.25$).

# 2. Accuracy summaries (per arm, over the evaluated records)

# 3. Interval metrics

# 3.1 Raw coverage (reported as primary calibration numbers)

• 50 %: $\frac1n\sum_i \mathbb 1{q_{i,.25} \le P_i \le q_{i,.75}}$ (coverage_50)
• 90 %: $\frac1n\sum_i \mathbb 1{q_{i,.05} \le P_i \le q_{i,.95}}$ (coverage_90)

# 3.2 Interpolated 80 % and 95 % coverage

The five quantiles do not bracket an 80 % or a 95 % interval. We fit, per record, a normal distribution in log space to the five quantiles by least squares on the probit grid: $\log q_{i,\tau} \approx \mu_i + \sigma_i z_\tau$, $z_\tau = \Phi^{-1}(\tau)$, giving $\mu_i = \frac15\sum_\tau \log q_{i,\tau}$ and $\sigma_i = \sum_\tau z_\tau(\log q_{i,\tau} - \mu_i)/\sum_\tau z_\tau^2$ (truncated at 0). The $1-\alpha$ interval is $\exp(\mu_i \pm z_{1-\alpha/2}\sigma_i)$ (coverage_80, coverage_95). coverage_90_normal_fit is the same construction at 90 % and is reported next to the raw coverage_90 as a diagnostic of the fit (identical when the quantiles are exactly log-normal). The 80/95 % numbers are therefore model-based interpolations and are labelled as such in the tables.

# 3.3 Width

mean_log_width_90 $= \frac1n\sum_i (\log q_{i,.95} - \log q_{i,.05})$; mean_log_width_50 analogously with $q_{.25}, q_{.75}$. sharpness() also reports the median log width and the mean relative width $(q_{.95}-q_{.05})/q_{.50}$.

# 3.4 Pinball loss and CRPS (log scale)

With $y_i = \log P_i$ and $\rho_\tau(u) = u(\tau - \mathbb 1{u<0})$: pinball_log $= \frac1n\sum_i \frac15\sum_{\tau} \rho_\tau(y_i - \log q_{i,\tau})$. crps_log $= \frac1n\sum_i, 2\int_{0.05}^{0.95} \rho_\tau(y_i - \log q_{i,\tau}),d\tau$ approximated by the trapezoidal rule on the five-point $\tau$ grid. Because the integral over $\tau\in(0,0.05)\cup(0.95,1)$ is unobserved it is omitted: the quantity is a lower-biased approximation of the CRPS, consistent across arms. crps_log_normal_fit is the closed-form CRPS of the fitted normal of §3.2 (Gneiting & Raftery 2007, eq. 21), reported as a robustness check. Both are in log units.

# 3.5 PIT and reliability

PIT$i = F_i(\log P_i)$ where $F_i$ is interpolated in probit space: $\Phi^{-1}(F_i)$ is piecewise linear in $\log P$ between the knots $(\log q{i,\tau}, \Phi^{-1}(\tau))$; outside $[q_{.05}, q_{.95}]$ the tails of the fitted normal (§3.2) are used. This is exact for log-normal forecasts. A calibrated arm has a uniform PIT histogram (pit.csv, fig_pit.pdf). Reliability of each quantile: empirical $\frac1n\sum_i \mathbb 1{P_i \le q_{i,\tau}}$ versus nominal $\tau$ (coverage.csv, kind = quantile). Conditional coverage: §3.1 recomputed within subgroups (region group, property class, price tercile) subject to the cell rule of §6.

# 3.6 Conformal rescaling (conformalise, apply_conformal)

Split conformal with an asymmetric scaled-residual score. With $c_i = \log q_{i,.50}$, $\ell_i = c_i - \log q_{i,.05}$, $u_i = \log q_{i,.95} - c_i$: $s_i = \max{(c_i - y_i)/\ell_i,\ (y_i - c_i)/u_i}$ on the validation records. The factor $k_{90}$ is the $\lceil (n+1),0.90\rceil$-th smallest $s_i$; the test interval becomes $[\exp(c_i - k_{90}\ell_i), \exp(c_i + k_{90}u_i)]$. Same construction for the 50 % interval with $q_{.25}, q_{.75}$. Under exchangeability the marginal coverage on test is $\ge 0.90$. Factors are estimated on validation only and frozen (no test-set information). Point and median are never changed.

# 4. Paired inference between arms

All contrasts are paired on the same tx_ids. For arms A and B and a summary statistic $T(\cdot)$ computed on per-record losses, $\Delta = T(B) - T(A)$; for a loss (all accuracy metrics) $\Delta < 0$ favours B. Pre-registered pairs (docs/preregistration.md): (D, G2), (F, G2), (J, G2), (I, G2) on mean $|e|$ (Holm family), (I, G2) on P95 $|e|$; all other pairs/metrics are exploratory.

# 4.1 Block bootstrap

Blocks = mun_code × calendar month of the sale (fallback mun_code, then i.i.d. records when neither exists). $B = 2000$ resamples of blocks with replacement (seed 20261027), both arms resampled with the identical block draw. 95 % percentile CI = (2.5th, 97.5th) percentiles of $\Delta^$. Bootstrap p-value: $p = \frac1B\sum_b \mathbb 1{|\Delta^_b - \hat\Delta| \ge |\hat\Delta|}$ (bootstrap distribution recentred at the null; floored at $1/(B+1)$). The bootstrap SE is the SD of $\Delta^*$. Any metric of §2–§3 can be used as $T$ (metric= name or callable on a per-record frame).

# 4.2 Holm correction

For a family of $m$ p-values ordered $p_{(1)} \le \dots \le p_{(m)}$, adjusted $\tilde p_{(k)} = \max_{j\le k}\min{1, (m-j+1)p_{(j)}}$. Applied within each metric across the pre-registered pairs (p_holm). Verified against statsmodels.multipletests(method="holm").

# 4.3 Diebold–Mariano

On the loss differential $d_i = L^B_i - L^A_i$ ($L = |e|$, or squared error for RMSE, or pinball) sorted by sale date, $DM = \bar d / \sqrt{\hat\omega / n}$ where $\hat\omega$ is the Newey–West long-run variance with Bartlett kernel and bandwidth $h = \lfloor 4(n/100)^{2/9} \rfloor$, with the Harvey–Leybourne–Newbold (1997) small-sample factor and a $t_{n-1}$ reference. Assumption: $d_i$ is a covariance-stationary series in time order; sales on the same day are ordered arbitrarily (stable sort), so the HAC correction captures calendar dependence only. The DM test is reported as a complement to the block bootstrap, which remains the primary inference tool (it respects the municipality × month cluster structure).

# 5. Rounds, cost and Pareto

The rounds curve (fig_rounds.pdf) reads results/evaluation/rounds.csv (round, value[, ci_lo, ci_hi, arm, metric]) produced by the experiments layer; it is absent until the rounds grid 0/1/2/4 has been run. Cost–accuracy (fig_cost_accuracy.pdf) plots cost_usd_mean (log scale; baselines with no API cost are plotted at a floor) against mean $|e|$; the Pareto frontier joins non-dominated arms.

# 6. Strata and the minimum-cell rule

Standard strata: region_group (Montréal / Québec / Outaouais / other, from region_name when the analytic table does not carry it), urban_class, property_class, price_tercile (terciles of $P$ over the truth table), year_built_decade (<1950, 1950s … 2000s, 2010+), density_tercile (terciles of n_sales_3km_12m → thin / mid / liquid, when the column exists). A cell with fewer than 30 records is suppressed (value NaN, suppressed = True, printed --). Tables are long (arm, committee, aggregation, stratum, level, n, metric, value, suppressed).

# 7. Faithfulness audit (transcripts)

Operates on DebateTranscript + EvidencePacket JSON of each run. Per estimate/revision turn:

• cited_share_valid: share of evidence_ids_used present in packet.evidence_ids() (H6 threshold: ≥ 0.90 on average);
• adj_share_within_40: share of adjustments with $|\text{adjusted}/\text{raw} - 1| \le 0.40$ (raw = comparable price in the packet); share_turns_adj_ok = share of turns where all adjustments pass (H6: ≥ 0.80);
• point_in_hull: point estimate within $[\min \text{adjusted} \times 0.75,\ \max \text{adjusted} \times 1.25]$; otherwise the log deviation from the nearest bound and whether key_assumptions is non-empty (hull_deviation_justified);
• quantile_sane: positive, non-decreasing, point within $[q_{.05}, q_{.95}]$, and $q_{.95}/q_{.05} \in [1.01, 4]$;
• summary_words and summary_within_limit (≤ 120 words);
• packet_hash_match: packet.content_hash() == transcript.packet_hash.

Committee level, per transcript and round $r$ with log estimates $x_{k,r}$ of roles $k$ and round-0 median $m_0$:

• dispersion iqr_log $= \operatorname{IQR}k(x{k,r})$; convergence $= \operatorname{IQR}_r / \operatorname{IQR}_0$;
• anchoring_index $= \operatorname{corr}k\big(x{k,r} - x_{k,0},\ m_0 - x_{k,0}\big)$ (1 = every revision is a move toward the initial committee median, i.e. pure herding; ≈ 0 = revisions unrelated to the initial consensus);
• share_toward_median $= \frac1K\sum_k \mathbb 1{|x_{k,r} - m_0| < |x_{k,0} - m_0|}$. faithfulness.csv aggregates per run × arm × committee (final-round committee statistics; committee_rounds.csv keeps all rounds).

# 8. Outputs and tables

results/evaluation/: summary.csv (all arm × committee × aggregation × run_id), summary_main.csv (one row per arm: preferred configuration --main-committee mixed_random --main-aggregation median, else the baseline configuration, else the first available — column main_selection records which), records.parquet, contrasts.csv, coverage.csv, pit.csv, strata.csv, faithfulness*.csv, committee_rounds.csv, manifest.json (runs, truth path, seed, $B$, timestamp). results/tables/*.tex are booktabs tabular fragments with \num{} cells, no caption, last line \end{tabular}; a .csv twin sits next to each fragment. Figures are PDF without titles, colour-blind-safe palette with markers/hatches as secondary encoding.

# 9. Commands

python3 -m pytest tests/test_evaluation.py -q
python3 scripts/29_fake_demo_predictions.py                         # FAKE run under results/runs/_fake_demo (skipped by default)
python3 scripts/20_evaluate.py --synthetic --include-fake --B 500    # smoke test of the pipeline on the fake run
python3 scripts/20_evaluate.py                                       # real evaluation (analytic.parquet + splits.parquet, B = 2000)
python3 scripts/21_figures.py