% Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
\providecommand{\figdir}{../figures}
\section{Main results}
\label{sec:results}

This section reports the preregistered evaluation on the test partition, the reading of hypotheses H1 to H4, the effect sizes and what the confidence intervals exclude, the cheaper model tier, the evaluations on the untouched 2026 holdout and on the prospective set fetched after the freeze, the run-to-run variation of the hosted models, and the cost and latency of every arm. All numbers are those of the versioned files written by the evaluation script; the preregistered contrasts were computed once, after the freeze, and nothing in this section was used to choose anything. Throughout, $|e|$ denotes the absolute log error of Section~\ref{subsec:design_estimands}, the agentic arms use the mid-tier model pair in the mixed committee with the median aggregation unless stated otherwise, and $n$ is the number of properties.

\subsection{Accuracy on the test partition}
\label{subsec:results_main}

Table~\ref{tab:main_results} gives the accuracy of every arm. The non-agentic baselines are scored on the full test partition of 96,935 sales; the agentic arms on the first eighty properties of the preregistered subsample at the mid tier, and on the first three hundred at the small tier. Among the baselines the ordering is the expected one: the hedonic regression (A) has a mean absolute log error of 0.191, the hierarchical model with the repeat-sales adjustment (B) 0.182, the three boosted models (C) between 0.177 and 0.180, and the AVM (D), which adds comparable-based and spatial-temporal lag features to the boosted model, 0.171, with a median absolute percentage error of 10.5 percent and a severe-mispricing rate, the share of sales valued more than 25 percent away from the price, of 21.8 percent. The AVM is the strongest non-agentic arm on every accuracy metric and is the H1 benchmark.

\begin{table}[t]
\centering
\begin{threeparttable}
\caption{Accuracy by arm on the test partition.}
\label{tab:main_results}
\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/main_results}
\begin{tablenotes}[flushleft]
\footnotesize
\item \textit{Notes.} $e=\log\hat P-\log P$. Mean $|e|$ is the primary metric; MAE is the mean absolute error in Canadian dollars; MdAPE the median absolute percentage error; RMSE the root mean squared log error; P90 and P95 the empirical percentiles of $|e|$; Bias the mean of $e$; the last-but-one column the share of sales with $|e|>\log 1.25$. Baselines A--D are scored on the full test partition (sales registered 2025-04-01 to 2025-12-31); rows marked \emph{spatial} are the versions trained without Outaouais, Abitibi-T\'emiscamingue and Bas-Saint-Laurent, scored on the 8,988 test sales of those regions. Agentic arms E--J use the mid-tier pair (\texttt{claude-sonnet-5-5}, \texttt{gpt-6.1-sol}) in the mixed committee with the median aggregation on the first 80 properties of the preregistered test subsample; rows marked \emph{small} use the small-tier pair (\texttt{claude-haiku-5-5}, \texttt{gpt-6-luna}) on the first 300. Source: \texttt{results/evaluation/summary\_main.csv}, \texttt{scripts/20\_evaluate.py}.
\end{tablenotes}
\end{threeparttable}
\end{table}

The six agentic arms at the mid tier are close to one another and close to the AVM. The single generalist (E) has a mean absolute log error of 0.166, the independent committee (F) 0.159, the two-round debate (G2) 0.158, the debate with the blinded skeptic (H) 0.158, the placebo committee (I) 0.157 and the cost-matched single agent (J) 0.163. The committees have a smaller median error than the generalist, 8.7 to 9.3 percent against 10.7 percent, and the lowest 95th percentile of $|e|$ among all arms, 0.493 to 0.499 against 0.508 for E; their mean log error is within 0.004 of zero. Because the eighty properties are not the full partition, the relevant AVM figure is its paired value on the same eighty, 0.163, which appears in the contrast tables and is lower than its full-partition value, so that the subsample is somewhat easier than the partition as a whole. Figure~\ref{fig:error_distribution} shows the distribution of the log error by arm: the agentic arms have a slightly tighter interquartile range than the baselines and whiskers that end earlier on the right, but the picture is one of methods that make errors of the same order on the same properties.

\begin{figure}[t]
\centering
\includegraphics[width=0.8\textwidth]{\figdir/fig_error_distribution.pdf}
\caption{Distribution of the log error $e=\log\hat P-\log P$ by arm on the test partition. Boxes are the interquartile range with the median marked; whiskers end at the 5th and 95th percentiles. Baselines are scored on the full partition (spatial rows on the 8,988 withheld-region sales), mid-tier agentic arms on 80 properties and small-tier arms on 300. Source: \texttt{results/evaluation/records.parquet}, \texttt{scripts/21\_figures.py}.}
\label{fig:error_distribution}
\end{figure}

\subsection{Preregistered contrasts and the reading of H1 to H4}
\label{subsec:results_contrasts}

Table~\ref{tab:contrasts} reports the paired contrasts. The first four rows are the primary family: G2 against D, F, J and I on the mean absolute log error, with the block bootstrap of Section~\ref{subsec:design_inference} over 65 municipality-by-month blocks and the step-down adjustment of \citet{holm1979simple}; the fifth row is the tail component of H4; the remaining rows are secondary. Every primary contrast is small and every interval contains zero. Against the AVM, the debate lowers the mean absolute log error by 0.005, with a 95 percent interval from $-0.033$ to $0.026$; against the independent committee by 0.001, interval $-0.006$ to $0.005$; against the cost-matched single agent by 0.006, interval $-0.017$ to $0.004$; against the placebo committee it raises it by 0.000, interval $-0.005$ to $0.006$. The bootstrap $p$-values are 0.75, 0.70, 0.29 and 0.95, the Holm-adjusted values are all equal to one, and the Diebold--Mariano statistics on the date-ordered loss differentials give the same reading, with $p$-values between 0.25 and 0.95. The P95 component of H4 is $-0.004$ with interval $-0.022$ to $0.065$. None of H1 to H4 is rejected.

\begin{table}[t]
\centering
\begin{threeparttable}
\caption{Paired contrasts on the test subsample: primary family and secondary pairs.}
\label{tab:contrasts}
\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/contrasts}
\begin{tablenotes}[flushleft]
\footnotesize
\item \textit{Notes.} Each row compares arm B to arm A on the same $N$ properties; $\Delta$ is the metric of B minus the metric of A, so that a negative value favours B for every loss. The 95\% CI is the percentile interval of a paired block bootstrap (blocks = municipality $\times$ calendar month, 65 blocks, 2,000 resamples, seed 20261027); $p$ (boot) is the recentred bootstrap $p$-value; $p$ (Holm) is the step-down adjusted value within the primary family of the four mean-$|e|$ contrasts (rows 1--4); $p$ (DM) is the Diebold--Mariano test on the date-ordered loss differential with Newey--West variance and the Harvey--Leybourne--Newbold correction, reported for mean $|e|$ and pinball loss only. The baselines' values are their paired values on the same properties, not their full-partition values. Cov.\ 90\% is the raw coverage of the 90 percent interval; arm C has no intervals. Source: \texttt{results/evaluation/contrasts.csv}.
\end{tablenotes}
\end{threeparttable}
\end{table}

Three readings follow from the pattern of the estimates rather than from any one test. First, a committee of twelve deliberating agents is, on these properties, about as accurate as the AVM: a difference of 0.005 in mean absolute log error, three percent in relative terms, in favour of the committee, estimated with an interval that extends from a 20 percent advantage to a 16 percent disadvantage. Second, communication adds nothing measurable to independent aggregation: the G2--F contrast is the most precisely estimated of the family because the two arms are the same twelve roles on the same views and differ only in the two rounds of exchange, and its interval, $-0.006$ to $0.005$, is narrow relative to the others. Third, the compute-matched single agent is close to the committee, 0.163 against 0.158, and the placebo committee is indistinguishable from it, which is the prediction of Proposition~\ref{prop:placebo}(ii) under additive separability of prompt and view. The secondary contrasts complete the picture: the committee beats the single generalist by 0.008, interval $-0.023$ to $0.004$, and by 0.020 in median absolute percentage error, interval $-0.060$ to $0.007$; the independent committee beats the generalist by 0.007; neither difference is distinguishable from zero at these sample sizes, but both are in the direction that the averaging of reasoning errors in Proposition~\ref{prop:exchange} predicts, and both are larger than the G2--F difference.

\begin{table}[t]
\centering
\begin{threeparttable}
\caption{Effect sizes, relative confidence intervals and minimum detectable effects for the primary family.}
\label{tab:primary_effects}
\scriptsize\setlength{\tabcolsep}{3pt}\input{tables/tab_primary_effects}
\begin{tablenotes}[flushleft]
\footnotesize
\item \textit{Notes.} $\Delta$ and its CI are those of Table~\ref{tab:contrasts}; SE is the bootstrap standard error of $\Delta$; $\Delta$ (\%) and its CI are $\Delta$ and the CI divided by the value of the reference arm A (the first-named arm in the design, e.g. D in ``G2 vs D''). MDE is the minimum detectable effect at 80 percent power for a two-sided test at the 5 percent level implied by the observed standard error, $(1.96+0.84)\,\mathrm{SE}$, in absolute log points and as a percentage of the reference value. $p$ (Holm) is reported for the four mean-$|e|$ contrasts. Source: \texttt{results/evaluation/contrasts.csv}, \texttt{scripts/22\_extra\_tables.py}.
\end{tablenotes}
\end{threeparttable}
\end{table}

Table~\ref{tab:primary_effects} states what the intervals rule out. In relative terms the G2--D contrast is $-3.1$ percent with an interval from $-20.0$ to $+16.2$ percent; G2--F is $-0.7$ percent, interval $-4.0$ to $+3.2$ percent; G2--J is $-3.5$ percent, interval $-10.6$ to $+2.5$ percent; G2--I is $+0.1$ percent, interval $-2.9$ to $+3.5$ percent. The minimum effect that the design could have detected with 80 percent power, given the bootstrap standard errors, is 0.043 log points or 26 percent of the AVM's error for H1, 0.008 or 5.1 percent for H2, 0.015 or 9.5 percent for H3, and 0.007 or 4.6 percent for H4. The design was therefore powered to detect a moderate gain from communication and from roles, and only a large gain over the AVM. The preregistration stated that the claim that deliberation adds information would be falsified if H2 and H3 were both not rejected with intervals excluding a two percent relative improvement. Both are not rejected, and the H2 interval excludes improvements larger than 4.0 percent while the H3 interval excludes improvements larger than 10.6 percent; neither excludes a two percent improvement. The falsification criterion is therefore not met in its strict form: the evidence is that any information added by two rounds of deliberation is worth less than four percent of the committee's error, not that it is zero, and the eighty-property test tier cannot separate the two. What the test tier does establish is the ordering of the channels: whatever twelve agents add over one, they add it before the first round of communication.

\subsection{The small tier and the cost frontier}
\label{subsec:results_small}

The small-tier runs provide the second point on the cost frontier and a larger sample for the central contrast. With \texttt{claude-haiku-5-5} and \texttt{gpt-6-luna} on three hundred properties, the generalist has a mean absolute log error of 0.182, the independent committee 0.176 and the two-round debate 0.180, between 0.017 and 0.022 above the mid-tier values of the same arms, with a positive mean log error of 0.028 to 0.038 and severe-mispricing rates of 25.7 to 28.0 percent against 21.2 to 23.7 percent at the mid tier. The cheaper models are also worse calibrated: their raw 90 percent coverage is 0.777, 0.787 and 0.807 against 0.887 for the three mid-tier arms, their 50 percent coverage 0.417 to 0.427 against 0.463 to 0.512, with intervals that are narrower, not wider, than those of the mid tier. Section~\ref{sec:calibration} returns to this. Within the small tier the independent committee is again the best of the three and the debate does not improve on it; the ordering E, G2, F reproduces the mid-tier finding that the gain from twelve roles over one is obtained without communication.

\begin{figure}[t]
\centering
\includegraphics[width=0.72\textwidth]{\figdir/fig_cost_accuracy.pdf}
\caption{Cost per property and mean absolute log error on the test partition. Baselines (squares) have no API cost and are plotted at a floor; agentic arms (circles) at their ledger cost per property, mid tier on 80 properties and small tier on 300. Letters are arm codes; the two points per letter for E, F and G are the small and mid tiers. The grey line joins the non-dominated arms. Source: \texttt{results/evaluation/summary\_main.csv}, \texttt{scripts/21\_figures.py}.}
\label{fig:cost_accuracy}
\end{figure}

Figure~\ref{fig:cost_accuracy} places every arm in the cost--accuracy plane. The frontier runs from the AVM, which costs nothing per property and has a mean absolute log error of 0.161 on its spatial variant and 0.171 on the full partition, through the independent committee at 23 cents and 0.159, to the debate and skeptic arms at 1.35 to 1.52 dollars and 0.158. The small-tier arms lie above the frontier at every cost: the small-tier debate costs 12.5 cents per property, six times the mid-tier generalist, and is worse than it by 0.014. The frontier is almost flat: moving from the AVM to the most expensive arm buys a reduction of about 0.005 in mean absolute log error for a cost of 1.5 dollars and five to six minutes of wall-clock time per property. These are the numbers that the lender of Section~\ref{sec:welfare} would weigh.

\subsection{Temporal holdout and prospective evaluation}
\label{subsec:results_holdout}

The 2026 holdout partition, 83,489 sales registered from 2026-01-01 to 2026-10-07, was evaluated once, after the test results were written, for the baselines on the full partition and for E, F and G2 on the first sixty properties of the holdout subsample. The prospective set comprises sales present in the live database but absent from the 2026-10-21 snapshot, registered between 2026-10-06 and 2026-10-13, whose identifiers could not have influenced any decision; the baselines are scored on all 1,182 of them and the three agentic arms on forty. Table~\ref{tab:drift} collects the accuracy and calibration of the AVM and the agentic arms on the three evaluation sets; the full tables for the holdout and prospective sets are in the repository.

\begin{table}[t]
\centering
\begin{threeparttable}
\caption{Accuracy and calibration across the test, holdout and prospective evaluation sets.}
\label{tab:drift}
\scriptsize\setlength{\tabcolsep}{2pt}\input{tables/tab_drift}
\begin{tablenotes}[flushleft]
\footnotesize
\item \textit{Notes.} Columns as in Table~\ref{tab:main_results}; Over +25\% is the share of sales valued more than 25 percent above the price; Cov.\ 50\% and 90\% are the raw coverages of the 50 and 90 percent intervals and Width 90\% their mean log width. ``D (full partition)'' is the AVM on all sales of the set; ``D (paired subsample)'' the same AVM on the properties valued by the agentic arms, so that it is directly comparable with E, F and G2. Test: sales of 2025-04-01 to 2025-12-31; holdout: 2026-01-01 to 2026-10-07; prospective: registered 2026-10-06 to 2026-10-13 and absent from the snapshot. Source: \texttt{results/evaluation\{,\_holdout,\_prospective\}/summary\_main.csv} and \texttt{records.parquet}, \texttt{scripts/22\_extra\_tables.py}.
\end{tablenotes}
\end{threeparttable}
\end{table}

The holdout reverses the test-tier ordering between the AVM and the committees and reveals a drift that the test partition did not show. On the full 2026 partition the AVM's mean absolute log error is 0.180 with a mean log error of 0.002; on the sixty paired properties it is 0.189 with a mean log error of 0.055. The agentic arms have mean absolute log errors of 0.195 (E), 0.186 (F) and 0.189 (G2), their 95th percentiles of $|e|$ rise from about 0.50 on the test subsample to 0.81--0.85, and all three carry a mean log error of $+0.10$ to $+0.11$: in 2026 they overvalue, and 18.3 percent of their valuations are more than 25 percent above the price against 5 percent below. The paired contrasts on the holdout are G2 against D $-0.000$, interval $-0.035$ to $0.037$, and G2 against F $+0.003$, interval $-0.001$ to $0.007$, with a Holm-adjusted $p$-value of 0.26; the generalist is worse than the independent committee by 0.009, interval $-0.021$ to $0.002$. The median error of the agentic arms remains low, 8.6 to 9.0 percent against 11.2 percent for the AVM on the full partition, so that the drift is in the tails and in the level, not in the typical case. One reading, which the data cannot confirm, is that the agents extrapolate the 2025 expansion into 2026 from the market section of their packets, while the AVM's lag features are re-computed as of each valuation date; the fundamental-value triangulation of Section~\ref{sec:fundamental} is the only place where a bias of this sign could be interpreted as something other than an error.

The prospective set shows a bias of the same sign in every arm, including the baselines, which suggests that the market itself moved between the training window and October 2026. On the 1,182 prospective sales the AVM has a mean absolute log error of 0.202 and a mean log error of $+0.039$, the hedonic regression 0.219, the boosted models 0.215 to 0.222 with mean log errors of $+0.050$ to $+0.095$. On the forty paired properties the AVM is at 0.197 with a mean log error of $+0.059$, and the agentic arms at 0.187 (E), 0.179 (F) and 0.187 (G2) with mean log errors of $+0.058$ to $+0.061$. The contrasts are G2 against D $-0.010$, interval $-0.057$ to $0.043$, and G2 against F $+0.009$, interval $-0.000$ to $0.021$, Holm-adjusted $p$-value 0.19. Across the three evaluation sets the independent committee is never worse than the debate and is better than it on the two later sets by 0.003 and 0.009, with intervals whose lower bounds touch zero; the direction is that of Proposition~\ref{prop:consensus}(iii), under which opinion exchange with median anchoring can only lose accuracy, but the magnitudes are at the edge of what forty to sixty properties can resolve.

\subsection{Run-to-run variation}
\label{subsec:results_repeat}

Hosted models are not deterministic, and the paper does not claim bitwise reproducibility of the agentic arms. Eight of the eighty test properties, one tenth, were re-run through the two-round debate with the same packets, the same protocol, the same committee assignment and the same model identifiers, so that only the providers' sampling differed. The mean absolute difference between the two runs' log point estimates is 0.0069, the median 0.0044, the maximum 0.0186 and the standard deviation of the difference 0.0092; the correlation between the two runs' absolute errors is 0.998, the mean absolute log error on the eight properties is 0.195 in the original run and 0.198 in the repeat, the mean 90 percent log width 0.504 and 0.518, and every repeated point lies inside the original run's 90 percent interval. Run-to-run noise of the committee output is therefore of the order of 0.005 to 0.01 log points, an order of magnitude below the differences between arms and the baselines, and about the size of the G2--F and G2--I contrasts, which is one more reason to read those two contrasts as zero. The repeat run cost 12.39 dollars.

\subsection{Cost and latency}
\label{subsec:results_cost}

Table~\ref{tab:cost} reports the ledger cost, wall-clock latency and token count per property. At the mid tier the generalist costs 2.2 cents, 20 seconds and 6,600 tokens per property; the independent committee 23 cents, 79 seconds and 74,000 tokens; the cost-matched single agent 26 cents, 94 seconds and 92,000 tokens, which confirms that J is matched to F in tokens, not to G2, as Section~\ref{subsec:arch_protocols} explains; the two-round debate 1.52 dollars, 336 seconds and 521,000 tokens, seven times the tokens of the independent committee for the same accuracy; the skeptic and placebo variants 1.35 and 1.40 dollars. The small tier is about twelve times cheaper per property at the same latency, since latency is dominated by the number of sequential turns rather than by the model. The six mid-tier test runs together cost 382 dollars, the three small-tier runs 44 dollars, and the ledgers record 1,078.82 US dollars of live API spend over every run of the paper, pilots, validation ablations, test, holdout, prospective and the repeat run included, against the envelope of roughly one thousand dollars stated in Section~\ref{subsec:design_budget}. At list prices and before caching, valuing the full test partition of 96,935 sales with the two-round debate would cost about 147,000 dollars and the AVM nothing, which is the practical content of a flat cost frontier.

\begin{table}[t]
\centering
\begin{threeparttable}
\caption{Cost, latency and tokens per property, test partition.}
\label{tab:cost}
\footnotesize
\input{tables/cost}
\begin{tablenotes}[flushleft]
\footnotesize
\item \textit{Notes.} Cost is the mean estimated API cost per property in US dollars at list prices as recorded in the ledger; Latency the mean wall-clock time per property in seconds, including validation retries; Tokens the mean input plus output tokens per property; Total cost the sum over the evaluated properties. Baselines record no API cost; their fitting time is not shown. Mean $|e|$ and N repeat Table~\ref{tab:main_results}. Source: \texttt{results/tables/cost.csv}, \texttt{results/ledger/}.
\end{tablenotes}
\end{threeparttable}
\end{table}