1% Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca2\providecommand{\figdir}{../figures}3\section{Calibration, interval quality and faithfulness}4\label{sec:calibration}56A valuation that is used as collateral or as an appraisal is a distribution, not a point, and the welfare model of Section~\ref{sec:welfare} prices its width and its honesty. This section reports the coverage of the intervals at the four nominal levels, the probability-integral-transform histograms and the reliability of each quantile, the trade-off between sharpness and coverage and the conformal remedy, the conditional coverage by subgroup, the behaviour of the intervals under the drift of Section~\ref{subsec:results_holdout}, the severe-mispricing rates that a lender reads, and the faithfulness audit of the transcripts that H6 concerns. The 50 and 90 percent coverages are raw, since the five quantiles bracket those intervals directly; the 80 and 95 percent coverages are interpolated from a normal fit in log space and are labelled as such.78\subsection{Coverage at the four nominal levels}9\label{subsec:cal_coverage}1011Table~\ref{tab:coverage} and Figure~\ref{fig:coverage_vs_nominal} report the coverage, width and proper scores of every arm with intervals on the test partition. The hedonic and hierarchical baselines, whose intervals come from a Gaussian residual model, are close to nominal at every level, 0.489 and 0.499 at 50 percent, 0.909 and 0.913 at 90 percent, with a mean 90 percent log width of 1.00, that is, intervals spanning a factor of $e$ between their bounds. The AVM's conformal intervals, with factors estimated on validation, cover 0.896 at 90 percent and 0.936 at 95 percent with a width of 0.83, but only 0.446 at 50 percent. The mid-tier agentic arms cover 0.887 at 90 percent for E, F, G2 and I, 0.875 for H and 0.863 for J, with widths of 0.56 to 0.60; at 50 percent they cover 0.463 (E), 0.512 (F), 0.475 (G2), 0.525 (H), 0.487 (I) and 0.475 (J); at the interpolated 95 percent level they cover 0.887 to 0.912, which is the first sign that their tails are too light. H5 stated that the 90 percent interval of G2 would have a coverage closer to 0.90 than that of E and of F. The three coverages are identical, 0.8875, 71 of 80 properties in each case, and the paired contrast of G2 against E is 0.000 with interval $-0.050$ to $0.051$; H5 is not supported, and the reason is structural rather than statistical. The committee interval is the median across roles of each quantile, and Table~\ref{tab:deliberation} shows that the roles' own quantiles do not contract with the rounds, the 90 percent width being 0.590 for F and 0.602 for G2; Proposition~\ref{prop:consensus}(i) predicted that agreement-based intervals would contract while the pooled error did not, and the protocol's intervals are not agreement-based, which is why the two rounds leave coverage where it was. A linear pool of quantiles is not calibrated even when its components are \citep{ranjan2010combining}; here the components are not calibrated either.1213\begin{table}[t]14\centering15\begin{threeparttable}16\caption{Interval coverage, width and proper scores by arm, test partition.}17\label{tab:coverage}18\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/coverage}19\begin{tablenotes}[flushleft]20\footnotesize21\item \textit{Notes.} Cov.\ 50\% and 90\% are the raw shares of sale prices inside $[q_{.25},q_{.75}]$ and $[q_{.05},q_{.95}]$; Cov.\ 80\% and 95\% are interpolated from a normal distribution fitted in log space to the five quantiles on the probit grid and are model-based. Width 90\% is the mean of $\log q_{.95}-\log q_{.05}$. Pinball is the average over the five quantiles of the pinball loss in log units; CRPS the trapezoidal approximation of the continuous ranked probability score on the five-point grid, lower-biased and comparable across arms. Arm C produces no intervals. Samples as in Table~\ref{tab:main_results}. Source: \texttt{results/evaluation/summary\_main.csv}.22\end{tablenotes}23\end{threeparttable}24\end{table}2526\begin{figure}[t]27\centering28\includegraphics[width=0.62\textwidth]{\figdir/fig_coverage_vs_nominal.pdf}29\caption{Empirical against nominal interval coverage by arm on the test partition at the 50, 80, 90 and 95 percent levels (80 and 95 interpolated). The dashed diagonal is perfect calibration; arms beyond the sixth are drawn in grey. Source: \texttt{results/evaluation/coverage.csv}, \texttt{scripts/21\_figures.py}.}30\label{fig:coverage_vs_nominal}31\end{figure}3233The small-tier arms are worse calibrated on every count: 0.777, 0.787 and 0.807 at 90 percent for E, F and G2, 0.417 to 0.427 at 50 percent, 0.813 to 0.860 at the interpolated 95 percent level, with intervals that are narrower than the mid tier's, 0.45 to 0.49 against 0.56 to 0.60, while their errors are larger. The cheaper models are therefore overconfident in the ordinary sense, and the committee of twelve of them, whose interval is the median of twelve overconfident intervals, inherits the overconfidence; this is the common-component term of Proposition~\ref{prop:exchangeable} applied to the quantiles rather than to the points. The pattern is consistent with the general finding that language models' verbal confidence is poorly calibrated and that calibration improves with model scale \citep{kadavath2022language,xiong2024can}.3435\subsection{Probability integral transform and quantile reliability}36\label{subsec:cal_pit}3738Figure~\ref{fig:pit} shows the PIT histograms, the position of the realised log price in each arm's predictive distribution, interpolated in probit space between the five quantiles. A calibrated arm has a flat histogram \citep{gneiting2007strictly,gneiting2014probabilistic}. The AVM has a spike in the 0.2 to 0.3 bin, 19.5 percent of the mass against 10, and too little mass above 0.5: its distribution is shifted upward relative to the prices, which the quantile reliability confirms, with 0.578 of prices below the AVM's median quantile and 0.339 below its lower quartile, against 0.50 and 0.25 nominal. The hedonic baselines show a smaller spike in the same bin. The mid-tier agentic arms are closer to flat at eighty properties, with bins between 5 and 16 percent, and their reliability is good at the extremes and poor in the middle: for G2 the shares of prices below the 5th, 25th, 50th, 75th and 95th quantiles are 0.062, 0.225, 0.475, 0.700 and 0.950, so that the lower tail and the median are right, the upper quartile sits too low, and the interquartile band is narrow on its upper side. The small-tier histograms are U-shaped, with 15 percent of the mass in each of the two extreme bins for G2 and 18 and 15 percent for E, and reliabilities of 0.133 at the 5th and 0.910 at the 95th quantile for E: too many prices fall outside the stated 90 percent range on both sides, which is the signature of intervals that are too narrow rather than mislocated.3940\begin{figure}[t]41\centering42\includegraphics[width=0.9\textwidth]{\figdir/fig_pit.pdf}43\caption{Probability integral transform histograms by arm on the test partition, ten bins; the dashed line is the uniform reference. The PIT is $F_i(\log P_i)$ with $F_i$ interpolated in probit space between the five stated quantiles and normal tails beyond them. Source: \texttt{results/evaluation/pit.csv}, \texttt{scripts/21\_figures.py}.}44\label{fig:pit}45\end{figure}4647\subsection{Sharpness, coverage and the conformal remedy}48\label{subsec:cal_sharpness}4950The agentic intervals are sharper than the AVM's at almost the same 90 percent coverage: 0.56 to 0.60 in log width against 0.83 on the full partition and 0.846 on the eighty paired properties, 27 to 34 percent narrower, for coverages of 0.887 against 0.896 and 0.875. Sharpness subject to calibration is the goal of probabilistic forecasting \citep{gneiting2007strictly}, and on that criterion the committees' intervals are the more useful ones on the test partition. The proper scores are less favourable than the widths suggest: the pinball loss of G2 is 0.053 against 0.058 for the AVM on the full partition but 0.052 on the paired eighty, and the F--D contrast on the pinball loss is $+0.001$ with interval $-0.008$ to $0.011$; the narrower intervals are paid for by the mass that falls outside them, and on the same properties the two families' predictive distributions score the same.5152Table~\ref{tab:conformal} applies the split-conformal rescaling of the evaluation protocol, with the factors estimated on the forty validation properties of each arm and committee and frozen before the test runs \citep{vovk2005algorithmic,lei2018distribution,romano2019conformalized,angelopoulos2023gentle}. The factors are 1.55 to 1.63 for the 90 percent interval and 1.24 to 1.26 for the 50 percent interval, which says that the models' own 90 percent intervals needed to be stretched by more than half to cover 90 percent of validation prices. On the test partition the rescaled intervals cover 0.925 (E), 0.950 (F), 0.938 (G2) and 0.938 (J) at 90 percent and 0.525 to 0.637 at 50 percent, with 90 percent widths of 0.86 to 0.93: the conformal guarantee holds, and the sharpness advantage over the AVM disappears, since the rescaled widths match or exceed the AVM's 0.846 on the same properties. On the prospective set the rescaled coverages are 0.900 to 0.925. On the 2026 holdout they are 0.883 to 0.900 with widths of 0.80 to 0.82, below the guarantee for E and G2, because split conformal prediction assumes exchangeability between the calibration and the evaluation sets and the holdout is a different year with a different level of prices. The remedy restores marginal coverage where the distribution has not moved and does not where it has; the honest interval for 2026 would have had to be calibrated on 2026 data.5354\begin{table}[t]55\centering56\begin{threeparttable}57\caption{Split-conformal rescaling of the agentic intervals with validation factors.}58\label{tab:conformal}59\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_conformal}60\begin{tablenotes}[flushleft]61\footnotesize62\item \textit{Notes.} Raw columns are the stated intervals. $k_{50}$ and $k_{90}$ are the split-conformal factors of the asymmetric scaled-residual score, estimated on the 40 validation properties of the same arm and mixed committee (\texttt{results/evaluation/llm\_conformal\_scales.json}) and applied unchanged: the rescaled 90 percent interval is $[\exp(c-k_{90}\ell),\exp(c+k_{90}u)]$ with $c=\log q_{.50}$, $\ell=c-\log q_{.05}$, $u=\log q_{.95}-c$, and likewise at 50 percent with the quartiles. Under exchangeability the rescaled coverage is at least nominal. Arms H and I have no validation factors and are omitted. Source: \texttt{results/evaluation\{,\_holdout,\_prospective\}/records.parquet}, \texttt{scripts/22\_extra\_tables.py}.63\end{tablenotes}64\end{threeparttable}65\end{table}6667\subsection{Conditional coverage}68\label{subsec:cal_conditional}6970Marginal coverage can hide subgroups in which the intervals fail, and lenders care about the subgroups. Table~\ref{tab:coverage_subgroups} gives the raw 90 and 50 percent coverages by region group, property class and price tercile, under the thirty-property rule, for the AVM on the full partition cells and for the agentic arms on the subsample cells. At the mid tier the committees cover 0.944 of Montr\'eal CMA prices and 0.861 of prices elsewhere at the 90 percent level, 0.860 to 0.895 for houses and 0.906 in the top tercile, against 0.911, 0.885, 0.896 and 0.928 for the AVM; the cost-matched single agent is lower in three of the four cells, 0.917, 0.861, 0.860 and 0.844. The small-tier debate, whose seventeen populated cells reach further into the distribution, covers 0.900 of condominiums and 0.853 of the top tercile but only 0.789 of houses, 0.756 of plexes and 0.667 of the cheapest tercile, where the AVM itself covers only 0.770. The cheapest tercile is where every method's intervals fail: these are the sales for which comparables are heterogeneous and the price is most affected by condition that the packet does not observe. At the 50 percent level the mid-tier cells lie between 0.417 and 0.562 and the small-tier debate covers 0.222 of cheapest-tercile prices with its interquartile band. Conditional coverage therefore degrades in the same cells where accuracy degrades in Section~\ref{subsec:abl_strata}, and the degradation is larger for the cheaper models.7172\begin{table}[t]73\centering74\begin{threeparttable}75\caption{Conditional coverage of the 90 percent (upper panel) and 50 percent (lower panel) intervals by subgroup, test partition.}76\label{tab:coverage_subgroups}77\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_coverage_subgroups}78\begin{tablenotes}[flushleft]79\footnotesize80\item \textit{Notes.} Raw coverages within each subgroup. D is scored on the full test partition cell; the mid-tier arms E--J on the cells of the 80-property subsample ($n$ mid) and the small-tier debate on the cells of the 300-property subsample ($n$ small); cells with fewer than 30 properties are suppressed (--). Source: \texttt{results/evaluation/coverage.csv}, \texttt{scripts/22\_extra\_tables.py}.81\end{tablenotes}82\end{threeparttable}83\end{table}8485\subsection{Calibration under drift and severe mispricing}86\label{subsec:cal_drift}8788Table~\ref{tab:drift} already showed what happens to the intervals on the later evaluation sets. On the 2026 holdout the agentic arms' raw 90 percent coverage falls to 0.800, 0.833 and 0.833 and their widths to 0.50 to 0.53, narrower than on the test partition although the errors are larger, while the AVM covers 0.878 on the full partition and 0.900 on the sixty paired properties with a width of 0.82. On the prospective set the agentic coverages are 0.750, 0.800 and 0.825 against 0.866 and 0.900 for the AVM. The agents' intervals do not know that the market has moved: their width is a function of the packet, which is as of the valuation date, and of the model's habits, and neither widens when the model's bias changes sign. The AVM's conformal factor does not know either, but its interval is wide enough to absorb a bias of 0.05.8990The severe-mispricing rates are the quantity a lender reads. On the test partition the share of valuations more than 25 percent away from the price is 0.218 for the AVM on the full partition and 0.212 on the paired properties, 0.237 for the generalist, 0.212 for the independent committee, 0.225 for the debate; over- and undervaluation are balanced for the committees, 0.100 and 0.088 for G2, whereas the AVM overvalues more often than it undervalues on the full partition, 0.110 against 0.059. On the holdout the rates are 0.233 to 0.267 for the agentic arms and 0.250 for the paired AVM, but their composition changes: 0.183 of the agentic valuations are more than 25 percent above the price and 0.050 below, and the paired AVM shows the same 0.183 above. On the prospective set the agentic rates are 0.275 to 0.300 with 0.175 above, and the AVM's 0.281 on all 1,182 sales with 0.157 above. A collateral haircut calibrated on the test partition's symmetric errors would have been too small in 2026 for every method, and the committees' narrower intervals would have made the shortfall larger for them than for the AVM. Proposition~\ref{prop:lenderloss} gives the mapping from these rates to the lender's expected loss.9192\subsection{Faithfulness audit}93\label{subsec:cal_faithfulness}9495H6 is descriptive: at least ninety percent of the evidence codes cited by G2 agents should exist in their packets, and the adjusted-comparable consistency check should pass for at least eighty percent of turns. Table~\ref{tab:faithfulness} reports the audit for every run, computed on the stored transcripts and packets. The first criterion is met with room to spare: in every run of every arm, on test, holdout, prospective and validation, 100 percent of the cited evidence identifiers exist in the role's packet, the packet hash recorded in each transcript matches the packet rebuilt from the snapshot, and the mean number of schema-validation retries per turn is at most 0.033. No role cited a comparable or a section it had not been shown, and in the terms of Lemma~\ref{lem:additive} no revision was persuasion dressed as evidence. The second criterion is met for the single-agent arms and not for the committees. Between 95.4 and 99.9 percent of individual adjustments lie within forty percent of the raw comparable price, but the share of turns in which every adjustment does is 0.900 for the generalist and 0.917 for the cost-matched agent against 0.709 for the independent committee, 0.721 for the debate, 0.717 for the skeptic variant and 0.711 for the placebo on the test partition, and 0.70 to 0.75 in every other committee run. The committees fail H6's second threshold, and the full-view ablation of Section~\ref{subsec:abl_fullview}, where the share rises to 0.883 when every role sees the whole packet, locates the failure in the partition: roles whose view does not contain the comparable section reason about comparables at second hand and state adjustments that the deterministic check cannot reconcile with a price they were not shown. The verification flags of arm H, which are injected into the next prompt, do not raise the share, 0.717, because a flagged role that cannot see the comparable cannot repair the adjustment. The remaining checks pass almost uniformly: the point estimate lies within the hull of the adjusted comparables widened by a quarter in 97.5 to 100 percent of turns, the five quantiles are positive, ordered, bracket the point and span a ratio between 1.01 and 4 in 98.5 to 100 percent, and the reasoning summaries of the test, holdout and prospective runs average 71 to 80 words with none above the 120-word limit.9697\begin{table}[t]98\centering99\begin{threeparttable}100\caption{Faithfulness audit of the transcripts by run.}101\label{tab:faithfulness}102\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_faithfulness}103\begin{tablenotes}[flushleft]104\footnotesize105\item \textit{Notes.} $n$ is the number of properties and Turns the number of estimate or revision turns audited. Valid ids: share of cited evidence identifiers present in the role's packet (H6 threshold 0.90). Adj.\ $\pm$40\%: share of stated adjustments with $|\text{adjusted}/\text{raw}-1|\le0.40$ where raw is the comparable's price in the packet; Turns ok: share of turns in which every adjustment passes (H6 threshold 0.80). In hull: share of turns whose point lies in $[\min\text{adjusted}\times0.75,\max\text{adjusted}\times1.25]$. Quantiles ok: positive, non-decreasing, point inside $[q_{.05},q_{.95}]$ and $q_{.95}/q_{.05}\in[1.01,4]$. Words: mean length of the reasoning summary; $\le$120: share within the limit; Retries: mean number of schema-validation retries per turn. Every transcript's packet hash matches the rebuilt packet. The deliberation statistics of the same runs are in Table~\ref{tab:deliberation}. Source: \texttt{results/evaluation/faithfulness.csv}, \texttt{scripts/22\_extra\_tables.py}.106\end{tablenotes}107\end{threeparttable}108\end{table}109110These results bear on H6 and on nothing larger. The audit establishes that the explanations are consistent with the evidence the agents were given: the identifiers exist, the arithmetic is within bounds, the point lies among the comparables, and the stated uncertainty has the right shape. It does not establish that the stated reasons are the causes of the estimate. An explanation can cite valid evidence and be numerically consistent while the number was produced by a different process, the one that the placebo committee of Section~\ref{subsec:abl_placebo} exposes when generic prompts reach the same estimates as specialised mandates with different prose; the literature on language-model explanations documents that stated reasoning can be unfaithful to the computation that produced the answer \citep{jacovi2020towards,turpin2023language,lanham2023measuring}. The hull check is also weak by construction, since a band from three quarters of the lowest adjusted comparable to five quarters of the highest contains most defensible values. The paper therefore reports faithfulness as a precondition for the mechanism of H2, which is satisfied, and not as a property of the valuations, which the accuracy and calibration results of this and the two preceding sections measure directly. The counterfactual-stability and human-rated usefulness assessments listed in the research plan are NOT YET ESTIMATED.111