spb/wp23_uqo

1\section{Main forecasting results}\label{sec:results}23\subsection{Task A: the price of a future sale}45Table~\ref{tab:scores_a} reports the accuracy of the sale-price forecasts over \NOriginsA{} quarterly origins and \NTestA{} evaluated sales, by horizon band, with Diebold--Mariano tests against the naive municipal-median reference and the 90\% model confidence set; Figure~\ref{fig:perf_a} plots the horizon profiles. Three results are robust across horizons. First, every model that uses the dwelling's characteristics beats the municipal median by a wide margin: the MASE of the hedonic OLS is \aMASEhedonicolshOneThree{} at one to three months, so a quarter of the error of the naive forecast is composition that any hedonic model removes. Second, gradient boosting is the most accurate model at every horizon (MAE \aMAElightgbmhOneThree{} log points at one to three months, \aMAElightgbmhFourSix{} at four to six and \aMAElightgbmhSevenOneTwo{} at seven to twelve, against \aMAEhedonicolshOneThree, \aMAEhedonicolshFourSix{} and \aMAEhedonicolshSevenOneTwo{} for the hedonic OLS), XGBoost and the random forest follow within one log point, and LightGBM is the only member of the 90\% model confidence set at every horizon; the ranking is the one of the valuation literature \citep{mayer2019updating}, now in a forecasting design. Third, the hierarchical hedonic network does not improve on the trees: its three pooling variants reach \aMAEhshnpartialhOneThree, \aMAEhshnfullhOneThree{} and \aMAEhshnnonehOneThree{} at one to three months, between the linear models and the trees, and the partial-pooling variant is indistinguishable from the unpooled one, so the learned municipality embeddings add nothing beyond the municipal market features that every model receives. The unembedded quantile MLP is the weakest learned model (\aMAEmlphOneThree); its predictive intervals are far too wide (90\% coverage \aCovmlphOneThree\%).67\input{tables/tab_scores_a}89\begin{figure}[htbp]\centering10\includegraphics[width=\textwidth]{figures/fig_perf_a.pdf}11\caption{Task A: MAE and CRPS of sale-price forecasts by horizon band and model.}\label{fig:perf_a}12\end{figure}1314The horizon profile is informative about what the models learn. Accuracy deteriorates slowly with the horizon (LightGBM loses \aMAEDeltalightgbm{} log points of MAE between the first and the last band), but the bias grows: all models under-predict prices at seven to twelve months by 4 to 9 log points, which is the appreciation of the Quebec market over 2023--2026 that the models, trained on past levels, do not extrapolate. The naive median has the same bias, and the models that embed the horizon (HSHN) are not less biased than those that do not. Forecasting the level of a future sale is therefore two problems: valuing the dwelling at today's prices, which the trees do well, and forecasting the market, which none of the property-level models does better than assuming no change; Task~B isolates the second problem. Figure~\ref{fig:error_time_a} shows the MAE and bias of the main models at each of the twelve origins: the bias of the hedonic models is negative at every one of the twelve origins, between $-3$ and $-8$ log points for the origins from late 2023 to early 2025 and smallest (about $-1$ to $-3$) in mid-2023 and from mid-2025, when the market was decelerating; the unembedded quantile MLP swings from $+10$ to $-19$ and back, which is the instability of an under-regularised network on 50 to 150 thousand training sales rather than a market signal. In dollars (Table~\ref{tab:dollar_a}), the median absolute error of a one-to-three-month LightGBM forecast is \$\Dollarlightgbmhouse{} for a house sold at the median price of \$420{,}000, \$\Dollarlightgbmcondo{} for a condominium (median \$364{,}000) and \$\Dollarlightgbmplex{} for a plex (median \$510{,}000), against \$\Dollarnaivemunmedianhouse, \$\Dollarnaivemunmediancondo{} and \$\Dollarnaivemunmedianplex{} for the municipal median: the characteristics reduce the dollar error by a third for houses and by three quarters for condominiums, whose prices vary most within a municipality, and the hedonic OLS is within 10\% of LightGBM in every class.1516\input{tables/tab_dollar_a}1718\begin{figure}[htbp]\centering19\includegraphics[width=\textwidth]{figures/fig_error_time_a.pdf}20\caption{Task A: MAE (left) and bias (right) of sale-price forecasts by forecast origin, all horizons pooled.}\label{fig:error_time_a}21\end{figure}2223\subsection{Task B: the change of a quality-adjusted index}2425Table~\ref{tab:scores_b} reports the accuracy of the index-change forecasts for the 82 cells (province, 17 regions, 64 municipalities) by horizon, with the Diebold--Mariano test against the no-change forecast on per-origin mean losses, the Giacomini--White conditional test and membership of the 90\% model confidence set; Figure~\ref{fig:perf_b} plots the horizon profiles and Figure~\ref{fig:traj} shows realised and forecast three-month changes for four cells. Five findings stand out.2627First, quality-adjusted index changes are forecastable at short horizons, but modestly. At one month the best model, the pooled LightGBM, has a MASE of \bMASElightgbmpanelhOne{} relative to the no-change forecast (MAE \bMAElightgbmpanelhOne{} against \bMAEnochangehOne{} log points); at three months \bMASElightgbmpanelhThree{} and at six months \bMASElightgbmpanelhSix. The Diebold--Mariano test rejects equal accuracy with the no-change forecast at the 1\% level for LightGBM at three months and at the 5\% level for QANSS and QREFA, and the Giacomini--White test agrees for LightGBM; with 31 to 45 origins per horizon the model confidence set is wide and separates only the clearly dominated models (local linear trend, seasonal naive, random walk with drift at some horizons).2829Second, the source of the predictability is momentum and seasonality in the index, not spatial information: the AR(2) on the cell's own monthly changes, the mean-growth rule and the pooled ridge all beat the no-change forecast by similar margins at three months (\bMAEarTwocellhThree, \bMAEmeangrowthhThree{} and \bMAEridgepanelhThree{} against \bMAEnochangehThree), the simple momentum rule does not (its fitted slope is close to zero because three-month changes are dominated by a seasonal cycle that the rule extrapolates in the wrong direction), and the seasonal naive is the best forecast at the provincial level at three months, where the June--July cycle of the index is strongest. The ablations of Section~\ref{sec:ablation} confirm that the cell's own momentum features carry the signal and that the regional and provincial momentum add little.3031Third, among the proposed architectures, QANSS and the integrated QREFA are the only ones that improve on the no-change forecast at one, three and six months with statistical support (QANSS: \bMAEqansshOne, \bMAEqansshThree, \bMAEqansshSix; QREFA: \bMAEqrefahOne, \bMAEqrefahThree, \bMAEqrefahSix), and neither beats the pooled LightGBM. BHPF and RAREF are indistinguishable from the pooled ridge they extend (\bMAEbhpfhThree{} and \bMAErarefhThree{} against \bMAEridgepanelhThree): the hierarchical intercepts shrink to nearly zero in a panel where the features already include the cell's own history, and the regime gating reproduces the ridge because the three experts converge to similar coefficients. DSGFN is worse than the no-change forecast at one month and no better than the ridge at three (\bMAEdsgfnhThree), consistent with the ablation that its graph carries no information.3233Fourth, the deep sequence models (LSTM, GRU, TCN, TFT-lite) are the weakest learned models at every horizon and are worse than the no-change forecast at one month (\bMAElstmhOne, \bMAEtftlitehOne{} against \bMAEnochangehOne): with 13-month histories and 34 to 45 origins they have too little data, and the robustness section shows that their errors move by half a log point across seeds. The Quebec panel reproduces the warning of \citet{zeng2023transformers} and the M-competitions \citep{makridakis2020m4}: in short samples, simple and tree-based models dominate.3435Fifth, at twelve months every pooled model fails. The AR(2) is the best forecast (\bMAEarTwocellhOneTwo{} against \bMAEnochangehOneTwo{} for no change), the pooled models are worse than no change and carry positive biases of 6 to 11 log points: trained on origins from 2022 to mid-2024, they learned the appreciation of 2021--2023 and project it onto 2025--2026, when the market slowed. The twelve-month evaluation has only 34 origins concentrated in one macroeconomic episode, and the result says less about the models than about the impossibility of learning annual dynamics from a sample that contains one cycle; it is the strongest argument in the paper for the protocol's insistence on short-sample humility.3637\input{tables/tab_scores_b}3839\begin{figure}[htbp]\centering40\includegraphics[width=\textwidth]{figures/fig_perf_b.pdf}41\caption{Task B: MAE and CRPS of index-change forecasts by horizon and model.}\label{fig:perf_b}42\end{figure}4344\begin{figure}[htbp]\centering45\includegraphics[width=\textwidth]{figures/fig_trajectories_b.pdf}46\caption{Task B: realised three-month change of the house index (black) and forecasts of five models, by origin, for the province, the Montréal and Capitale-Nationale regions and Gatineau.}\label{fig:traj}47\end{figure}4849\subsection{Evolution of the errors over time, multiple testing and combinations}5051Figure~\ref{fig:error_time_b} and Table~\ref{tab:year_b} follow the three-month errors origin by origin. The gain of the pooled models over the no-change forecast is not uniform: it is concentrated in the origins of December to March, whose three-month windows cover the spring upswing, when the realised change is large and positive and the models' calendar-month and momentum features anticipate it (LightGBM reaches a MASE of \YearlightgbmpanelTwoZeroTwoSix{} on the 2026 origins against \YearlightgbmpanelTwoZeroTwoThree{} on those of late 2023), and it vanishes for the origins of summer and autumn, where no model does better than no change. Table~\ref{tab:regime_b} makes the same point with ex-post regimes of the provincial market: when the provincial house index rose by more than two log points over the target window (\RegimeOriginsrising{} origins) LightGBM's MASE is \Regimelightgbmpanelrising{} and every pooled model beats no change; when the window was flat (\RegimeOriginsflat{} origins) LightGBM's MASE is \Regimelightgbmpanelflat, only the momentum rule edges out the no-change forecast (\Regimemomentumflat), and DSGFN is the worst model (\Regimedsgfnflat); the single falling window, the three months from September 2025, is forecast better than no change by DSGFN, QREFA, TFT-lite, LightGBM and momentum, and worse by the AR(2), BHPF, RAREF, QANSS and the ridge, which extrapolate the positive drift. The forecastability measured in Section~\ref{sec:results} is therefore the forecastability of seasonal upswings; a model that is good in rising windows and no worse than no change in flat ones, which is LightGBM's profile, is the best that this sample supports.5253Table~\ref{tab:tests_b} in Appendix~\ref{app:tests} reports the Diebold--Mariano statistics with Holm-adjusted $p$-values across the 18 models of each horizon. The adjustment matters: at three months LightGBM's unadjusted $p$-value of \DMPlightgbmpanelhThree{} becomes \HolmPlightgbmpanelhThree{} after Holm, QANSS's \DMPqansshThree{} becomes \HolmPqansshThree, and no improvement over the no-change forecast survives a 10\% family-wise error rate at any horizon; only the deterioration of the local linear trend remains significant. With 31 to 45 origins the evidence that any model beats no change is suggestive at the 5\% level for single comparisons and absent once the number of models tried is accounted for. The Giacomini--White conditional test, which asks whether the loss difference is predictable from its own lag, rejects for LightGBM ($p = \GWPlightgbmpanelhThree$), whose advantage is itself predictable from the season, and for the random walk with drift, the seasonal naive and the mean-growth forecasts, whose relative performance alternates with the calendar.5455Directional accuracy (Table~\ref{tab:direction_b}) tells the same story in a form closer to use. At three months the pooled models predict the sign of the realised change in \DirlightgbmpanelhThree\% of cell-months (LightGBM) to \DirdsgfnhThree\% (DSGFN), against 50\% for a coin and \DirmomentumhThree\% for the momentum rule, whose sign is that of the previous quarter; at turning points, origins where the realised change reverses sign, the accuracy falls to \DirTPlightgbmpanelhThree{} to \DirTPqrefahThree\%, little above chance, and momentum, which by construction predicts the sign that has just reversed, does best (\DirTPmomentumhThree\%) for the wrong reason. At twelve months the directional accuracy of about 91\% for every pooled model is uninformative: 91\% of twelve-month changes in the sample were positive and the models predict a rise; at turning points none does better than a coin.5657Table~\ref{tab:combo_b} evaluates equal-weight combinations of the strongest models on the same cells and origins. None improves on LightGBM alone (MASE \CombolightgbmpanelhThree{} at three months against \ComboeqlgbmqansshThree{} for LightGBM + QANSS and \ComboeqlgbmqanssridgearTwohThree{} for the four-model combination), all improve on QREFA, the learned stack (\ComboqrefahThree), and adding the no-change forecast as a fifth component (\ComboeqallwithnochangehThree) costs little. Combination protects against a bad component but, with one dominant component, does not beat it: the classical result that combination helps when the components have comparable accuracy and different information \citep{timmermann2006combinations} holds here in its negative form, and the learned stack does worse than equal weights because its weights are estimated on the same short sample in which the components' relative accuracy changes from year to year (Table~\ref{tab:year_b}).5859\input{tables/tab_year_b}60\input{tables/tab_regime_b}61\input{tables/tab_direction_b}62\input{tables/tab_combo_b}6364\begin{figure}[htbp]\centering65\includegraphics[width=\textwidth]{figures/fig_error_time_b.pdf}66\caption{Task B, horizon 3: MAE of the index-change forecasts by forecast origin, selected models.}\label{fig:error_time_b}67\end{figure}68