spb/wp11_uqo Public

1% Author: Simon-Pierre Boucher — contact@spboucher.ai2% ============================================================================3\section{The horse race}4\label{sec:results}56Table~\ref{tab:race_random} reports the full scoreboard under the random7holdout (411{,}283 training sales, 102{,}929 test sales);8Table~\ref{tab:race_temporal} repeats it under the forward-in-time split.9Figures~\ref{fig:forms}--\ref{fig:methods} visualize each axis.1011\begin{table}[p]12\centering13\begin{threeparttable}14\caption{The scoreboard, random 80/20 holdout}15\label{tab:race_random}16\small17\input{../results/tables/horserace_random}18\begin{tablenotes}[flushleft]\footnotesize19\item \textit{Notes:} All metrics on the held-out 20\% (102{,}929 sales).20MdAPE/MAPE: median/mean absolute percentage error on price levels (log21models retransformed with Duan smearing; Box--Cox inverted).22RMSE$_{\ln}$ and $R^2_{\ln}$: computed on log predicted vs.\ log actual23prices. Backbone rows repeat across axes by construction (A5 $=$ B3 $=$24C2; B4 $=$ D1). Bold: best in axis.25\end{tablenotes}26\end{threeparttable}27\end{table}2829\begin{table}[p]30\centering31\begin{threeparttable}32\caption{The scoreboard, forward-in-time holdout}33\label{tab:race_temporal}34\small35\input{../results/tables/horserace_temporal}36\begin{tablenotes}[flushleft]\footnotesize37\item \textit{Notes:} Training on sales before January 2025 (355{,}824);38testing on 2025--2026 (158{,}388). Unseen time categories are priced at39the last training period (carry-forward); the machine-learning models40receive the same information through the clamped month counter.41\end{tablenotes}42\end{threeparttable}43\end{table}4445\subsection{Axis A: functional form is second-order}4647Under the random split (Figure~\ref{fig:forms}), the six forms span4816.5--18.6\% MdAPE. The spline specification wins (16.5\%), the log-log49is second (17.0\%) --- floor area enters multiplicatively, as intuition50suggests --- and the linear, semi-log and quadratic forms cluster within51half a point of each other. The Box--Cox transformation selects52$\hat\lambda = 0.25$ by profile likelihood (Table~\ref{tab:boxcox}), i.e.\53the data reject both the linear ($\lambda=1$) and the log ($\lambda=0$)54--- and then \emph{underperforms} the simple semi-log out of sample55(18.6\%), because optimizing curvature in-sample does not survive56prediction, precisely the warning of \citet{cassel1985cost} and57\citet{cropper1988choice}. The ordering is preserved under the temporal58split. Two free lessons: curvature in \emph{attributes} (splines) is59worth more than curvature in the \emph{price} (Box--Cox); and no form60choice moves the needle by more than two points.6162\begin{figure}[t]63\centering64\includegraphics[width=0.85\textwidth]{fig_forms.png}65\caption{Axis A --- functional form. MdAPE on the random holdout;66municipality and quarter fixed effects and the base attribute set held67fixed throughout.}68\label{fig:forms}69\end{figure}7071\begin{table}[t]72\centering73\begin{threeparttable}74\caption{Box--Cox profile likelihood}75\label{tab:boxcox}76\small77\input{../results/tables/boxcox}78\begin{tablenotes}[flushleft]\footnotesize79\item \textit{Notes:} $^{\dagger}$Thousands, relative to the maximum80(zero = chosen $\lambda$). Estimated on the random-split training sample81with the A-axis design.82\end{tablenotes}83\end{threeparttable}84\end{table}8586\subsection{Axis B: time controls --- anything beats nothing}8788Omitting time controls entirely costs three points (21.1\% versus 18.1\%)89in a window containing a boom, a correction and a recovery90(Figure~\ref{fig:time}). But the granularity ladder flattens immediately:91year, quarter and month effects are indistinguishable to within a quarter92point under the random split. Under the temporal split the ordering93inverts at the bottom --- \emph{no} time effects (25.4\%) is the worst94choice again, but month effects with carry-forward (20.3\%) beat quarter95effects (20.5\%) only marginally: once the level must be extrapolated,96fine within-sample time resolution has little to offer.9798\begin{figure}[t]99\centering100\includegraphics[width=0.85\textwidth]{fig_time.png}101\caption{Axis B --- time effects (quadratic semi-log backbone,102municipality FE). MdAPE on the random holdout.}103\label{fig:time}104\end{figure}105106\subsection{Axis C: spatial controls are first-order --- up to a point}107108Location dominates every other design margin109(Figure~\ref{fig:space}). Stripping all spatial controls costs110\emph{ten} points (27.9\% versus 17.5\%) --- five times the entire111functional-form range. The granularity ladder, however, is112non-monotonic: the $\sim$5.5~km grid (5{,}255 cells, 17.5\%) beats113municipality effects (18.1\%), but the $\sim$1.1~km grid (30{,}548114cells, 19.4\%) \emph{overfits} --- its cells average seventeen K sales115each, unseen cells in the holdout revert to the baseline, and estimated116effects are noisy. The optimal spatial resolution is an interior117solution, echoing \citet{goodman1998housing} on submarket definition:118finer is better only while cells remain thick enough to estimate.119120\begin{figure}[t]121\centering122\includegraphics[width=0.85\textwidth]{fig_space.png}123\caption{Axis C --- spatial controls (quadratic semi-log backbone,124quarter FE). MdAPE on the random holdout.}125\label{fig:space}126\end{figure}127128\subsection{Axis D: the machines --- and what they actually buy}129130Under the random split (Figure~\ref{fig:methods}), gradient boosting131posts 14.4\% and the random forest 14.6\%, against 16.5\% for the best132linear model and 18.3\% for the textbook OLS: a 13--21\% error133reduction, squarely in the range reported by the AVM literature134\citep{mayer2019estimation, hong2020machine}. The decomposition rows135identify the source. Boosting \emph{without coordinates} collapses to13623.2\% --- worse than plain OLS with municipality dummies --- so the137machine's edge is overwhelmingly a flexible surface over space, not138exotic structure--attribute interactions. Pure proximity without139structure (the $k$-NN comparables rule, 21.2\%) fails too: it is the140\emph{combination} of spatial flexibility with attribute adjustment that141wins. The cross-validated ridge (16.7\%) confirms that regularizing the142high-dimensional dummy design buys a little; it cannot manufacture the143missing interactions.144145\begin{figure}[t]146\centering147\includegraphics[width=0.85\textwidth]{fig_methods.png}148\caption{Axis D --- estimation methods on the identical attribute set.149MdAPE on the random holdout.}150\label{fig:methods}151\end{figure}152153\subsection{The generalization gap: when the test set is the future}154155Table~\ref{tab:generalization} and Figure~\ref{fig:generalization}156contain the paper's central result. Moving from the random to the157forward-in-time split degrades every model --- the price level of1582025--26 must be carried forward, not interpolated --- but the159degradation is systematically larger for the flexible methods.160Gradient boosting loses 6.8 points of MdAPE (14.4 $\to$ 21.2) and the161random forest 6.7, while the spline hedonic loses 2.6 (16.5 $\to$ 19.1)162and the log-log 2.2. The ranking \emph{reverses}: the best linear models163beat the machines on the median error in the deployment scenario, and164tie them exactly on $R^2_{\ln}$ ($\approx 0.52$). The machines remain165better on MAPE (29.7 versus 30.5), i.e.\ in the tails, but the headline166random-validation gap --- the number the ML-valuation literature167reports --- overstates their deployable advantage entirely. The lesson168is methodological and general: for models whose job is to price the169future, random cross-validation is the wrong experiment170\citep{mullainathan2017machine, clapp2002predicting}.171172\begin{table}[t]173\centering174\begin{threeparttable}175\caption{Random versus forward-in-time evaluation}176\label{tab:generalization}177\small178\input{../results/tables/generalization}179\begin{tablenotes}[flushleft]\footnotesize180\item \textit{Notes:} Selected models (the backbone and the full D axis)181under both splits.182\end{tablenotes}183\end{threeparttable}184\end{table}185186\begin{figure}[t]187\centering188\includegraphics[width=0.85\textwidth]{fig_generalization.png}189\caption{The generalization gap. MdAPE under the random holdout (filled190circles) and the forward-in-time holdout (open squares); the grey191segment joins the two evaluations of the same model.}192\label{fig:generalization}193\end{figure}194