1% Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca2\providecommand{\figdir}{../figures}3\section{Ablations and robustness}4\label{sec:ablations}5\label{sec:robustness}67The ablations of this section were run on the validation subsample, forty properties for every arm and twenty for the four-round debate, before the freeze, and they are the runs on which the aggregation weights and the conformal factors were estimated. They are exploratory in the sense of the preregistration: they are reported with block-bootstrap intervals and without claims of significance, and nothing in them was re-estimated after the test results were known. Two further ablations use the test partition itself, because they involve no choice: the comparison of aggregation rules, which is a column of the prediction files, and the spatial holdout of the baselines. The stratified errors are reported for both partitions. With forty properties the minimum detectable effect for a within-protocol contrast such as G2 against G1 is of the order of 0.01 log points and for a cross-protocol contrast such as G2 against D of the order of 0.05, so that the section can rule out large effects and nothing smaller.89\subsection{Rounds of deliberation}10\label{subsec:abl_rounds}1112Figure~\ref{fig:rounds} and Table~\ref{tab:rounds} follow the committee through its rounds. For every property and every round the transcripts give the twelve current log estimates; the table reports the mean absolute error of their median and of their mean, with block-bootstrap intervals, together with the mean interquartile range of the twelve estimates. Round 0 is the independent vector of arm F. When the stopping rule of Section~\ref{subsec:arch_protocols} ends a debate early, because the interquartile range changed by less than one percent between consecutive rounds, the committee's last estimates are carried forward to the later rounds, so that every round averages the same properties and the curve is not contaminated by the selection of the committees that kept debating. In the two-round debate of the mixed committee the mean absolute error of the median goes from 0.180 at round 0 to 0.178 after one round and 0.176 after two, a change of 0.004 inside an interval of width 0.11, while the interquartile range of the estimates falls from 0.070 to 0.044 and then 0.034, a contraction to about half in two rounds. The one-round debate reproduces the first step, 0.180 to 0.178 and 0.069 to 0.043. The four-round debate on twenty properties starts lower, 0.162, because the twenty properties are a different sample, and stays at 0.163 through every later round while its interquartile range falls from 0.057 to 0.032, 0.026, 0.025 and 0.024: after the second round the committee has stopped moving and the accuracy has not changed. The all-Anthropic and all-OpenAI committees give the same reading at opposite extremes of convergence, 0.181, 0.179 and 0.178 with the interquartile range falling from 0.056 to 0.018 for the former, 0.177, 0.178 and 0.178 with the range barely moving from 0.042 to 0.038 for the latter. The full-view committee goes from 0.173 to 0.174 and 0.175. The paired contrasts on the final outputs are G2 against G1, $-0.002$ with interval $-0.008$ to $0.002$, and G4 against G2 on the same twenty properties, $-0.001$ with interval $-0.005$ to $0.002$.1314\begin{figure}[t]15\centering16\includegraphics[width=0.7\textwidth]{\figdir/fig_rounds.pdf}17\caption{Mean absolute log error of the committee median by deliberation round on the validation subsample, mixed committee (G1, G2, G4, G-fullview). Shaded bands are 95 percent block-bootstrap intervals. Committees stopped early by the interquartile-range rule keep their last estimates in the later rounds, so that every round averages the same 40 properties (20 for G4). Source: \texttt{results/evaluation/rounds.csv}, \texttt{scripts/35\_rounds\_curve.py}.}18\label{fig:rounds}19\end{figure}2021\begin{table}[t]22\centering23\begin{threeparttable}24\caption{Accuracy and dispersion of the committee by round, validation subsample.}25\label{tab:rounds}26\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_rounds}27\begin{tablenotes}[flushleft]28\footnotesize29\item \textit{Notes.} For each property and round $r$, the twelve roles' current log estimates are pooled by their median and by their mean; the columns give the mean over properties of the absolute error of each pool and the mean interquartile range (IQR, in log points) of the twelve estimates. $n$ is the number of properties; committees stopped early by the interquartile-range rule carry their last estimates forward, so that every round is computed on the same properties. 95\% CIs are percentile block-bootstrap intervals (municipality $\times$ month blocks, 2,000 resamples). Source: \texttt{results/evaluation/rounds\_validation.csv}, \texttt{scripts/35\_rounds\_curve.py}.30\end{tablenotes}31\end{threeparttable}32\end{table}3334The dispersion collapses and the accuracy does not move. This is the pattern of Proposition~\ref{prop:consensus}: opinion exchange contracts every dispersion statistic at a geometric rate whatever the error structure, while the pooled error is unchanged, so that convergence is not evidence. The transcript statistics of Table~\ref{tab:deliberation} say how the contraction happens. The anchoring index, the correlation across roles between the revision $x_{k,r}-x_{k,0}$ and the distance to the initial committee median $m_0-x_{k,0}$, is 0.67 on validation and 0.71 on the test runs of G2, 0.68 at the small tier, 0.70 on the holdout and 0.76 on the prospective set; between 43 and 52 percent of the roles end closer to the initial median than they started. Revisions are thus mostly moves toward the table that every role is shown, and the revision instruction, which states that moving toward the committee without an evidential reason is an error, does not prevent them. Under the remark on public signals in Section~\ref{subsec:static}, anchoring of this kind is harmless as long as the weight placed on the median does not exceed twice its optimal weight, and the flat accuracy curve is consistent with a weight in that range; what it is not consistent with is the evidence-exchange gain of Proposition~\ref{prop:exchange}, which would have moved the pooled error. Cited evidence identifiers that lie outside a role's view are passed by identifier only, and the roles' comparable pools are drawn from the same retrieval, so that the exchange term $\sigma_u^2(\|\omega'\|^2-\|\omega\|^2)$ has little to act on.3536\begin{table}[t]37\centering38\begin{threeparttable}39\caption{Deliberation statistics by run: dispersion, convergence and anchoring.}40\label{tab:deliberation}41\footnotesize42\input{tables/tab_deliberation}43\begin{tablenotes}[flushleft]44\footnotesize45\item \textit{Notes.} IQR$_0$ and IQR$_R$ are the mean interquartile ranges of the twelve roles' log estimates at round 0 and at the final round; Convergence is their ratio; Anchoring is the mean over properties of the correlation across roles between the revision $x_{k,R}-x_{k,0}$ and the distance $m_0-x_{k,0}$ to the initial committee median (1 = every revision is a move toward the initial median); To median is the share of roles whose final estimate is closer to $m_0$ than their initial one. Arm F has no rounds (convergence 1 by construction); arm H's skeptic sits outside the rounds and its final-round statistics are not defined. Source: \texttt{results/evaluation/faithfulness.csv}, \texttt{scripts/22\_extra\_tables.py}.46\end{tablenotes}47\end{threeparttable}48\end{table}4950\subsection{Blinded skeptic and deterministic verification}51\label{subsec:abl_skeptic}5253Arm H removes the skeptic from the rounds, blinds it to the discussion, and injects deterministic verification flags into the next prompt of any role whose estimate cites a non-existent code, adjusts a comparable by more than forty percent, or lies outside the range of the adjusted comparables widened by a quarter. On the test subsample H and G2 are identical to three decimals on the primary metric, 0.158 against 0.158, with a paired difference of $-0.000$ and interval $-0.003$ to $0.003$; the median error is 9.2 against 8.7 percent, interval for the difference $-0.017$ to $0.007$; the 95th percentile, the severe-mispricing rate and the pinball loss do not differ; the 90 percent coverage is 0.875 against 0.887. On validation the picture is the same, 0.175 against 0.176. The verification tool has little to correct because the faithfulness audit of Section~\ref{subsec:cal_faithfulness} finds that every cited identifier exists and that 98 to 99 percent of adjustments already lie within the forty percent band in the unverified arms; the flags that remain concern individual adjustments and do not move the committee median. H is, however, cheaper than G2, 1.35 against 1.52 dollars, because the skeptic's turns are removed from the rounds, so that at equal accuracy it dominates the full debate on the cost frontier.5455\subsection{Placebo roles}56\label{subsec:abl_placebo}5758Arm I keeps the two rounds, the twelve evidence views and the committee state of G2 and replaces the twelve specialised mandates by generic prompts. On the test subsample the placebo committee has a mean absolute log error of 0.157 against 0.158, a difference of 0.000 with interval $-0.005$ to $0.006$ and a Holm-adjusted $p$-value of one; its 95th percentile of $|e|$ is 0.497 against 0.493, interval for the difference $-0.022$ to $0.065$. On validation the same contrast had pointed the other way, $-0.008$ with interval $-0.023$ to $0.003$ on the mean and $-0.044$ with interval $-0.156$ to $0.011$ on the 95th percentile, which is why H4 was retained in the primary family; the test tier did not reproduce it. By Proposition~\ref{prop:placebo}, a contrast between I and G2 with views fixed to the slots identifies the complementarity between a prompt and the evidence it is paired with, and nothing else; its absence means that, on this packet, a role told to reason as a mortgage underwriter about the underwriting section does no better than a role told to value the property from the same section. The placebo committee also deliberates like the role committee: its anchoring index is 0.75 against 0.71, its interquartile range contracts from 0.087 to 0.041 against 0.088 to 0.038, and its summaries are shorter, 72 words against 78. Role labels change the prose and not the numbers.5960\subsection{Full-view committee: the value of the evidence partition}61\label{subsec:abl_fullview}6263The second component of role diversity is the partition of the packet into least-privilege views. The full-view variant of G2 keeps the twelve default prompts and gives every role the whole packet; it was run on validation only. Table~\ref{tab:contrasts_extra} reports the paired contrasts. Against G2 the full-view committee has a mean absolute log error of 0.175 against 0.176, difference $-0.001$ with interval $-0.006$ to $0.004$; the median error is unchanged, the 95th percentile differs by 0.005, and the 90 percent coverage is 0.850 against 0.825. Against the placebo committee it is better by 0.009, interval $-0.027$ to $0.004$, and against the independent committee by 0.005, interval $-0.017$ to $0.004$. The partition therefore carries no measurable value on this packet, which is what Proposition~\ref{prop:exchange}(ii) predicts when the union of the views is what a single agent would be shown anyway: after the exchange the committee's evidence error is at best that of one agent with the union view, and giving the union to every role from the start reaches the same point without communication. Two secondary observations follow. First, with the full packet the share of turns in which every stated adjustment lies within forty percent of the raw comparable price rises from 0.71 to 0.88, close to the 0.90 to 0.98 of the single-agent arms, which is consistent with part of the flagged adjustments in the partitioned committee coming from roles that reason about comparables they were not shown; the paper does not test this. Second, the full-view committee anchors slightly less, 0.65 against 0.67, with a larger share of roles moving toward the median, 0.52 against 0.44, which is the expected behaviour of agents whose views no longer differ.6465\begin{table}[t]66\centering67\begin{threeparttable}68\caption{Exploratory paired contrasts on validation: the full-view committee.}69\label{tab:contrasts_extra}70\footnotesize71\input{tables/tab_contrasts_extra}72\begin{tablenotes}[flushleft]73\footnotesize74\item \textit{Notes.} Same conventions as Table~\ref{tab:contrasts}: $\Delta$ = B $-$ A on the same 40 validation properties, mixed committee, median aggregation; percentile block-bootstrap CI (municipality $\times$ month blocks, 2,000 resamples, seed 20261027) and recentred bootstrap $p$-value. These pairs were not preregistered and are not adjusted for multiplicity. Source: \texttt{results/evaluation\_validation/records.parquet}, \texttt{scripts/22\_extra\_tables.py}.75\end{tablenotes}76\end{threeparttable}77\end{table}7879\subsection{Provider committees}80\label{subsec:abl_committees}8182Proposition~\ref{prop:nested} predicts that the mixed committee has the lowest pooled error if provider-common errors are material, and no difference if they are not. Table~\ref{tab:committees} reports E, F and G2 in the all-Anthropic, all-OpenAI and mixed committees on validation, with paired contrasts of each single-provider committee against the mixed one. For the two-round debate the three committees have mean absolute log errors of 0.178, 0.178 and 0.176, differences of 0.002 from the mixed committee with intervals $-0.014$ to $0.016$ and $-0.011$ to $0.019$; for the independent committee 0.182, 0.178 and 0.179; for the single generalist 0.194, 0.177 and 0.182, the largest spread, with the all-Anthropic generalist worse than the mixed one by 0.013, interval $-0.025$ to $0.048$. The provider-common term is thus not detectable at forty properties, where the minimum detectable effect for the G2 contrasts is about 0.02 log points, or twelve percent. What differs between providers is the shape of the error and the dynamics of the debate rather than its level. The Anthropic models carry a smaller upward bias, 0.043 to 0.062 against 0.073 to 0.080 for the OpenAI models, and wider intervals, 0.58 against 0.49 to 0.52 in log width; the mixed committee sits between. In deliberation the two providers behave very differently: the all-Anthropic committee has an anchoring index of 0.97 and contracts its interquartile range from 0.056 to 0.018, almost complete convergence on the initial median, while the all-OpenAI committee has an anchoring index of 0.38, moves only 7.5 percent of its roles toward the median, and keeps its interquartile range at 0.038 against 0.042 initially; the mixed committee is at 0.67 and 0.034 against 0.070. Neither behaviour changes the accuracy of the median, which is the point of Proposition~\ref{prop:consensus}: a committee that converges completely and a committee that barely revises reach the same pooled error, because in neither case is the revision generated by evidence. The three committees do not differ in faithfulness either, with every cited identifier valid in all of them, but the Anthropic roles write shorter summaries, 63 to 69 words against 83 to 87.8384\begin{table}[t]85\centering86\begin{threeparttable}87\caption{Provider committees on validation: accuracy, calibration, deliberation and contrasts against the mixed committee.}88\label{tab:committees}89\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_committees}90\begin{tablenotes}[flushleft]91\footnotesize92\item \textit{Notes.} All-Anthropic: every role served by \texttt{claude-sonnet-5-5}; all-OpenAI: by \texttt{gpt-6.1-sol}; mixed: seeded balanced assignment of the two. Median aggregation, 40 validation properties. Bias is the mean log error; Cov.\ 90\% and Width 90\% the raw coverage and mean log width of the 90 percent interval; IQR$_R$ and Anchoring as in Table~\ref{tab:deliberation} (G2 only). $\Delta$ vs mixed is the mean $|e|$ of the single-provider committee minus that of the mixed committee of the same arm on the same properties, with its percentile block-bootstrap CI and recentred $p$-value (2,000 resamples). Source: \texttt{results/evaluation\_validation/summary.csv}, \texttt{results/runs/validation\_*/predictions.parquet}, \texttt{scripts/22\_extra\_tables.py}.93\end{tablenotes}94\end{threeparttable}95\end{table}9697\subsection{Aggregation rules}98\label{subsec:abl_aggregation}99100Every prediction file carries one row per aggregation rule, so that the choice of rule can be examined on the test partition without any new run. Table~\ref{tab:aggregation} gives the mean absolute log error of the committee arms under the median, the arithmetic mean, the twenty-percent trimmed mean, the inverse-variance pool and the stacking pool with weights estimated on the validation run of G2, the robust Student-$t$ pool, and the equal-weight mixture of the roles' distributions. The weights themselves are nearly uniform for the inverse-variance rule, between 0.082 and 0.088 for eleven roles and 0.060 for the building-condition role, whose log-error standard deviation on validation is 0.296 against 0.244 to 0.252 for the others; the stacking weights are concentrated, 0.39 on the hedonic role, 0.24 on the geospatial role, 0.16 on the market economist, 0.13 on the data auditor and 0.08 on the underwriter, with zero on the seven others. At the mid tier the rule changes the mean absolute log error of G2 between 0.158 and 0.162 and that of F between 0.155 and 0.162, a range of 0.004 to 0.007 that is below the run-to-run noise of Section~\ref{subsec:results_repeat} and far inside the intervals of the primary contrasts; the mean is the best rule for the independent committee and the median for the debate, and stacking is the worst for both. At the small tier the ranges are 0.006 and 0.009 and stacking is the best rule for F. This is the forecast-combination result of \citet{clemen1989combining} and \citet{timmermann2006forecast} in the form of Proposition~\ref{prop:general}: with roles of nearly equal accuracy, weights estimated on forty transcripts add estimation noise and cannot be expected to help, and a rule that concentrates on five roles discards seven nearly equivalent signals. For the cost-matched single agent the mean and the median of the twelve samples give 0.164 and 0.163 and the self-refined estimate 0.161, so that a critique-and-refine step on one model is worth about as much as two rounds among twelve. The primary comparisons use the median, which Proposition~\ref{prop:overshrink} identifies as the rule that is invariant to the deliberation; the table shows that nothing in the conclusions depends on this choice.101102\begin{table}[t]103\centering104\begin{threeparttable}105\caption{Mean absolute log error by aggregation rule, test partition.}106\label{tab:aggregation}107\footnotesize108\input{tables/tab_aggregation}109\begin{tablenotes}[flushleft]110\footnotesize111\item \textit{Notes.} Each cell is the mean $|e|$ of the same run under a different pooling of the twelve roles' final log estimates (80 properties at the mid tier, 300 at the small tier). Inverse-variance and stacking weights, and the scale of the Student-$t$ pool, were estimated once on the validation run of G2 (\texttt{results/aggregation\_weights.json}) and applied unchanged; the mixture pools the roles' quantile functions with equal weights. For arm J the twelve rows are the independent samples of the generalist; ``self-refined'' is the estimate after the critique-and-refine step. Source: \texttt{results/evaluation/summary.csv}, \texttt{scripts/22\_extra\_tables.py}.112\end{tablenotes}113\end{threeparttable}114\end{table}115116\subsection{Spatial holdout of the baselines}117\label{subsec:abl_spatial}118119The spatial experiment withholds Outaouais, Abitibi-T\'emiscamingue and Bas-Saint-Laurent from training and scores the withheld-region models on the 8,988 test-window sales of those regions; Table~\ref{tab:spatial} pairs each withheld model with its in-region counterpart on the same sales. The hedonic regression suffers most: its mean absolute log error rises from 0.183 to 0.208, difference 0.025 with interval 0.019 to 0.030, and its mean log error falls from $+0.021$ to $-0.078$, a systematic undervaluation that follows from municipality fixed effects it cannot estimate for municipalities it has not seen; its 90 percent coverage falls from 0.921 to 0.886. The boosted model does not suffer at all: trained without the regions it is more accurate on them, 0.176 against 0.187, difference $-0.011$ with interval $-0.013$ to $-0.009$, a precisely estimated difference for which the paper offers no mechanism beyond the observation that the three regions are a small share of the training data and that the in-region fit may be dominated by the rest of the province. The AVM is unchanged in accuracy, 0.161 against 0.163, interval for the difference $-0.006$ to $0.001$, and in coverage, 0.907 against 0.910, with a larger upward bias, $+0.045$ against $+0.022$, interval for the difference 0.019 to 0.027. The methods that travel across regions are those whose information comes from comparables and lag features computed as of the valuation date rather than from region-specific parameters. The agentic arms were not run on the spatial holdout in this version of the paper: their packets contain comparables retrieved from the whole province and no fitted region effect, so that the experiment would withhold nothing from them, and the Outaouais stratum of the eighty-property test subsample contains two properties.120121\begin{table}[t]122\centering123\begin{threeparttable}124\caption{Spatial holdout: in-region and withheld-region baselines paired on the withheld sales.}125\label{tab:spatial}126\footnotesize127\input{tables/tab_spatial}128\begin{tablenotes}[flushleft]129\footnotesize130\item \textit{Notes.} $n$ = 8,988 test-window sales in Outaouais, Abitibi-T\'emiscamingue and Bas-Saint-Laurent. ``In-region model'' is the baseline trained on the full training partition; ``Withheld model'' the same specification trained without the three regions. $\Delta$ = withheld $-$ in-region, with percentile block-bootstrap CI and recentred $p$-value (municipality $\times$ month blocks, 2,000 resamples, seed 20261027); $p$ values below 0.001 are floored. Arm C has no intervals. Source: \texttt{results/evaluation/records.parquet}, \texttt{scripts/22\_extra\_tables.py}.131\end{tablenotes}132\end{threeparttable}133\end{table}134135\subsection{Stratified errors}136\label{subsec:abl_strata}137138Figure~\ref{fig:strata} and Table~\ref{tab:strata_compact} report the mean absolute log error by region group, urban class, property class, price tercile and decade of construction, under the rule that a cell with fewer than thirty properties is suppressed. For the AVM the cells are those of the full partition and are all populated; for the mid-tier agentic arms only five of the twenty-four cells reach thirty properties, the Montr\'eal CMA and the residual region group, the metropolitan class, houses and the top price tercile; for the small-tier debate seventeen cells do. The comparison between the AVM's full-partition cell and an agentic arm's subsample cell is not paired and should be read as indicative. Two patterns are nonetheless clear. The agentic arms do markedly better than the AVM in the Montr\'eal CMA, 0.112 to 0.118 against 0.149 at the mid tier and 0.137 against 0.149 at the small tier, and no better outside it, 0.189 to 0.206 against 0.199 in the residual region group and 0.223 at the small tier; the Montr\'eal cell is where comparables are densest and where the market section of the packet is most informative. The small-tier cells, which cover more of the distribution, show the agentic arms better than the AVM for condominiums, 0.088 against 0.115, for the middle price tercile, 0.125 against 0.131, and for houses built in the 2000s, 0.117 against 0.141, and worse for rural sales, 0.267 against 0.219, for houses, 0.200 against 0.179, for the cheapest tercile, 0.337 against 0.319, and above all for buildings of the 1970s, 0.274 against 0.185, and of the 2010s, 0.160 against 0.137. The cheapest tercile is the hardest for every method, with errors above 0.3, as the hedonic literature on thin and heterogeneous low-price segments would predict \citep{dube2023residential}. The stratification by local transaction density, thin, mid and liquid terciles of the number of sales within three kilometres in the preceding twelve months, is NOT YET ESTIMATED: the analytic table of this version does not carry the density variable, and the cell is absent from the strata file.139140\begin{figure}[t]141\centering142\includegraphics[width=0.85\textwidth]{\figdir/fig_strata.pdf}143\caption{Mean absolute log error by region group and arm on the test partition. The AVM (D) is scored on the full partition cell ($n$ in the axis labels); the mid-tier agentic arms on the cells of the 80-property subsample, of which only ``other'' and ``Montr\'eal CMA'' reach the thirty-property threshold; suppressed cells are marked $n<30$. Source: \texttt{results/evaluation/strata.csv}, \texttt{scripts/21\_figures.py}.}144\label{fig:strata}145\end{figure}146147\begin{table}[t]148\centering149\begin{threeparttable}150\caption{Mean absolute log error by stratum, test partition.}151\label{tab:strata_compact}152\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_strata_compact}153\begin{tablenotes}[flushleft]154\footnotesize155\item \textit{Notes.} $n$ full, $n$ mid and $n$ small are the cell sizes in the full test partition (on which D is scored), in the 80-property mid-tier subsample (E, F, G2, I, J) and in the 300-property small-tier subsample (G2 small). Cells with fewer than 30 properties are suppressed (--). Price terciles are terciles of the sale price over the test partition. The full stratified table with every arm and metric is \texttt{results/tables/strata.tex}. Source: \texttt{results/evaluation/strata.csv}, \texttt{scripts/22\_extra\_tables.py}.156\end{tablenotes}157\end{threeparttable}158\end{table}159160\subsection{What was not run}161\label{subsec:abl_notrun}162163Three robustness checks that the research plan lists were not run in this version and are stated as such rather than approximated. The sensitivity of the results to the exclusion flags of WP~22, non-residential, land, duplicate, assessment-ratio outlier, built-after-sale and multi-parcel transfers, has not been estimated: all partitions apply the flags and no run without them exists. The agentic arms were not run on the spatial holdout, for the reason given above. The rounds grid beyond two rounds was run at the mid tier in the mixed committee only, on twenty properties; the interaction of the exchange term with the share of cited identifiers outside the common view, which Section~\ref{subsec:implications} proposes as the direct test of Proposition~\ref{prop:exchange}, is NOT YET ESTIMATED.164