SPB Git forge
spboucher.ai ↗

spb/wp22_uqo

Public
6commits 1branches 0releases
142.4 MBsize
maindefault branch
1 h agolast push
TeX 55.1% Python 44.2%
23.1 KB
View source Raw Blame History

# Novelty scan: close predecessors of an adaptive, hierarchical, state-space house price index

Scope: what each close predecessor does, with the citation as verified on 2026-10-21 (keys refer to paper/references.bib; metadata in docs/literature/references_verified.json), and what an adaptive hierarchical state-space index (the SHARP-HPI candidate in CLAUDE.md §6–7) would have to add to be distinct. Descriptions are based on the published abstracts retrieved from Crossref / Semantic Scholar / RePEc; where an abstract could not be retrieved this is stated and only title-level information is used. No claim of novelty is made here; the purpose is to delimit what already exists.

# 1. Rolling-time-dummy (RTD) hedonic indexes

Citations. Hill, Scholz, Shimizu & Steurer (2022), Journal of Official Statistics 38(1), 127–151, DOI 10.2478/jos-2022-0007 (hill2022rtd); Hill, Scholz, Shimizu & Steurer (2018), Economie et Statistique 500-501-502, 221–238, DOI 10.24187/ecostat.2018.500t.1953 (hill2018eval); the method is also described in the Eurostat et al. (2013) handbook (eurostat2013handbook).

What it does. A hedonic regression with time dummies is re-estimated on a rolling window of the most recent periods; the index movement for the latest period is taken from the newest window and chained (linked) onto the published series, so published values are never revised. The 2022 JOS paper (abstract) addresses three issues: a method for choosing the window length, variants of the linking rule and how to pick among them, and three modifications to make RTD robust to periods of low transaction volume. RTD "requires less data and is more adaptable than other hedonic methods", which is why several European statistical offices use it for official HPIs (Hill et al. 2018 evaluate the hedonic methods used by European NSIs).

What an adaptive hierarchical state-space index would need to add. RTD pools information over time only through a fixed (or one-off optimised) window and treats each geography separately; the hedonic coefficients are not shrunk across areas, there is no explicit model of the latent price level, and uncertainty is the regression standard error of the time dummies. To differ, the new estimator must (i) make the degree of temporal and cross-sectional pooling cell-specific and data-driven (estimated variance components rather than a chosen window), (ii) share hedonic and movement information across a spatial hierarchy with estimated shrinkage weights, (iii) provide filtered (real-time) and smoothed (retrospective) estimates with model-based intervals, and (iv) treat revision explicitly rather than by construction (RTD avoids revision by never updating). Any claim must be benchmarked against RTD with the 2022 window-length and low-volume modifications, not against a naive fixed window.

# 2. HMTS (hedonic multilateral time-series re-estimation with splicing)

Citation. Ishaak, Ouwehand & Remøy (2024), "Constructing Limited-Revisable and Stable CPPIs for Small Domains", Journal of Official Statistics 40(3), 380–408, DOI 10.1177/0282423X241246617 (ishaak2024hmts). This is the paper the official Statistics Netherlands R package REPS cites as the source of HMTS; an earlier four-author (with de Haan, 2023) version is mentioned in the package manual but could not be located bibliographically (see unverified.md).

What it does. From the abstract: for commercial property indices in small domains, where common methods give volatile series and attempts to reduce volatility cause frequent revisions of the whole series, the authors introduce "a combination of hedonic imputation, multilateral calculations, time series analysis, and window splicing", producing "stable and limited-revisable indices with the ability to detect turning points in an early stage", tested on Dutch commercial transactions. The REPS implementation (CRAN manual) makes the steps explicit: average imputed prices are computed for every period against every possible base period; each base-period series is re-estimated with a state-space (Kalman-filter) model; the series are converted to indices; a window of index figures is spliced; and the geometric average over the remaining figures gives the final index.

What an adaptive hierarchical state-space index would need to add. HMTS already contains a state-space smoothing step and already targets small domains, so the overlap is substantial. The differences available are: (i) in HMTS the Kalman filter is applied series-by-series to imputed averages after a hedonic model estimated separately, whereas a joint model would estimate the hedonic quality function, the latent movements and their variances in one likelihood with the small-domain movements shrunk toward higher-level movements; (ii) HMTS has no cross-domain hierarchy (each domain is smoothed on its own) and no robust (heavy-tailed) observation model; (iii) HMTS's stability comes from averaging over base periods and splicing a fixed window, not from an estimated signal-to-noise ratio; (iv) HMTS's uncertainty is not model-based. Because HMTS is already in official use (and the mandated benchmark G in CLAUDE.md §5), the empirical comparison must use the REPS-style implementation with its re-estimation and splicing options switched on.

# 3. Structural time-series / hierarchical repeat-sales for thin markets (Francke and co-authors)

Citations. Francke (2010), "Repeat Sales Index for Thin Markets", JREFE 41(1), 24–52, DOI 10.1007/s11146-009-9203-1 (francke2010thin); Francke & de Vos (2000), "Efficient Computation of Hierarchical Trends", JBES 18(1), 51–57 (franckedevos2000); Francke & van de Minne (2017), "The Hierarchical Repeat Sales Model for Granular Commercial Real Estate and Residential Price Indices", JREFE 55(4), 511–532, DOI 10.1007/s11146-017-9632-1 (francke2017hierarchical); Francke, Rolheiser & van de Minne (2025), "Estimating Census Tract House Price Indexes: A New Spatial Dynamic Factor Approach", JREFE 70(3), 483–514, DOI 10.1007/s11146-023-09957-w (francke2025censustract); van de Minne, Francke, Geltner & White (2020), JREFE 60(4), 514–553 (vandeminne2020revisions); Schwann (1998), JREFE 16(3), 269–287 (schwann1998thin) and Goetzmann (1992), JREFE 5(1), 5–53 (goetzmann1992) are the earlier thin-market / random-walk antecedents.

What they do. Francke (2010), abstract: replaces the time dummies of the Bailey–Muth–Nourse repeat-sales model (and Goetzmann's random walk with drift) by a local linear trend structural time-series model in which both level and drift vary over time, adds the reciprocal of the time between sales as a regressor, and obtains index estimates even for periods with no observations; it is explicitly designed for thin markets. Francke & van de Minne (2017), abstract: a repeat-sales model with "multiple stochastic log price trends having a hierarchical additive structure: one common log price trend and cluster specific log price trends in deviation from the common trend", with t-distributed errors to handle outliers; the estimated degrees of freedom are small, rejecting normality; applied to Philadelphia/Baltimore commercial property and part of Amsterdam, giving "reliable indices on a very detailed level based on a small number of observations". Francke, Rolheiser & van de Minne (2025), abstract: augments the repeat-sales model with a spatial dynamic factor model in which loadings on latent trends follow a spatial random walk, producing quarterly census-tract indexes that are less noisy than non-spatial ones and that replicate standard repeat-sales indexes where pairs are plentiful. van de Minne et al. (2020) use revisions as a quality criterion for repeat-sales indices.

What an adaptive hierarchical state-space index would need to add. This strand already has (a) state-space latent price levels, (b) an additive common-plus-deviation hierarchy, (c) heavy-tailed errors, (d) spatial sharing of latent trends, and (e) revision-based evaluation. It is the closest predecessor and must be cited as such. The remaining space is: (i) the observation equation is repeat-sales (pairs), which in Quebec municipalities with few resales discards most single-sale transactions and inherits repeat-sales selection issues (Gatzlaff & Haurin 1997, 1998; Clapp, Giaccotto & Tirtiroglu 1991); a hedonic (single-sale) observation equation with the same hierarchical trend structure would be a different estimator, not a new idea in itself (hybrid models go back to Case & Quigley 1991 and Quigley 1995); (ii) the pooling strength in these models is governed by estimated variance hyper-parameters that are constant over time and across clusters — an adaptive scheme in which the local-versus-pooled weight responds to a cell-level reliability measure (effective sample size, covariate overlap, residual spatial dependence) would need to be shown to improve on constant hyper-parameters, in simulation and out of time; (iii) a multi-level (province / region / MRC / municipality) hierarchy rather than one common trend plus one deviation level; (iv) coherence of aggregates (see §7). Without (ii)–(iv) demonstrated empirically, the proposal is a hedonic variant of Francke & van de Minne (2017).

# 4. Timeliness from listings (Anenberg & Laufer)

Citation. Anenberg & Laufer (2017), "A More Timely House Price Index", Review of Economics and Statistics 99(4), 722–734, DOI 10.1162/rest_a_00634 (anenberg2017timely). Related: Hill, Pfeifer, Steurer & Trojanek (2024), RIW 70(2), 320–344 (hill2024warning) on stale new-build transaction prices and the use of preliminary agreements; Sayag, Ben-hur & Pfeffermann (2022), IJF 38(1), 253–266 (sayag2022nowcasts).

What it does. From the RePEc abstract: using listings data, the authors build a repeat-sales index that dates house values at the contract date (when the price is determined) rather than at the closing date, which helps explain short-run serial correlation and weak correlation with stock prices and news; a version using only recent listings reveals trends several months before existing indexes such as Case–Shiller are published. Sayag et al. (2022), abstract: provisional HPIs are revised heavily because late-reported transactions differ from those reported on time; they nowcast, at sub-district level, the characteristics, average price and number of late-reported transactions and feed them into an extended hedonic model, cutting revision magnitude by more than 50% (Israel).

What an adaptive hierarchical state-space index would need to add. These contributions are about data timing (contract vs. deed date; late registrations), not about the estimator's pooling structure. A Quebec index built on registry deeds faces exactly the registration-lag problem; an explicit real-time vintage design (CLAUDE.md §11) and a late-registration model are therefore required to compare with Sayag et al., and the filtered state-space estimate is a nowcast that should be evaluated against a nowcast-augmented hedonic index, not only against the retrospective series. Nothing in a state-space hierarchy by itself addresses the contract-vs-closing timing issue.

# 5. High-frequency indices with time-series modelling (Bollerslev, Patton & Wang)

Citation. Bollerslev, Patton & Wang (2016), "Daily House Price Indices: Construction, Modeling, and Longer-run Predictions", Journal of Applied Econometrics 31(6), 1005–1025, DOI 10.1002/jae.2471 (bollerslev2016daily). Related: Bokhari & Geltner (2012), JREFE 45(2), 522–543 (bokharigeltner2012); Francke & van de Minne (2022), REE 50(5), 1257–1281 (francke2022daily).

What it does. From the abstract: daily repeat-sales indices for 10 US metropolitan areas, built with a method that "closely mimics" the monthly Case–Shiller methodology; the daily returns show mild autocorrelation and strong conditional heteroskedasticity; a multivariate time-series model for the daily returns "explicitly allowing for commonalities across cities and GARCH effects" forecasts longer-run monthly changes better than lower-frequency alternatives.

What an adaptive hierarchical state-space index would need to add. Bollerslev et al. model the index returns after construction (a two-step approach) with cross-city commonality and GARCH; the index itself remains a standard repeat-sales estimate on large metropolitan samples. A joint estimation of the index and its dynamics in sparse municipalities is a different setting; the relevant lesson to carry over is that cross-area commonality in returns is empirically strong, and that forecasting performance must be tested with proper out-of-time designs. Daily frequency is not at issue for Quebec (monthly target).

# 6. Fay–Herriot and small-area estimation (SAE)

Citations. Fay & Herriot (1979), JASA 74(366), 269–277, DOI 10.1080/01621459.1979.10482505 (fayherriot1979); Datta & Lahiri (2000), Statistica Sinica 10(2), 613–627 (dattalahiri2000); Datta, Rao & Smith (2005), Biometrika 92(1), 183–196 (datta2005variability); Pfeffermann (2013), Statistical Science 28(1), 40–68 (pfeffermann2013sae); Rao & Molina (2015), Small Area Estimation, 2nd ed., Wiley (raomolina2015). Empirical-Bayes shrinkage roots: Efron & Morris (1973, 1975) (efronmorris1973, efronmorris1975); hierarchical modelling: Gelman & Hill (2007), Gelman (2006). Housing applications at small spatial scale: Ren, Fox & Bruce (2017), Annals of Applied Statistics 11(2), 808–839, DOI 10.1214/17-AOAS1019 (renfoxbruce2017); Lopez & Hewings (2018), RSUE 70, 57–71 (lopezhewings2018); Bogin, Doerner & Larson (2019), REE 47(2), 365–398 (bogin2019local); Contat & Larson (2024), REE 52(6), 1551–1583 (contatlarson2024); Ahlfeldt, Heblich & Seidel (2023), RSUE 98, 103836 (ahlfeldt2023micro; abstract not retrieved, title-level only); Guntermann, Liu & Nowak (2016), JRER 38(1), 93–128 (guntermann2016thin; abstract not retrieved).

What they do. The Fay–Herriot area-level model combines a direct (design-based) estimate for each small area with a regression synthetic estimate, shrinking the direct estimate toward the synthetic one with a weight that depends on the ratio of sampling variance to model variance; Datta & Lahiri (2000) derive a second-order MSE approximation and estimator for EBLUPs under REML/ML, and Datta, Rao & Smith (2005) treat MSE under the basic area-level model; Pfeffermann (2013) reviews design-based, frequentist and Bayesian SAE developments including time-series and spatial extensions. In housing: Ren, Fox & Bruce (2017), abstract: a Bayesian nonparametric latent-factor model that clusters census tracts with correlated valuation dynamics to overcome sparse sales, yielding monthly tract-level indices for Seattle; Lopez & Hewings (2018), abstract: median-based HPIs are misleading for small spatial units, and they propose a matching-based Fisher HPI that controls for characteristics and is not restricted to repeat sales (Illinois); Bogin, Doerner & Larson (2019), abstract: the first public set of constant- quality (repeat-sales) indices for US counties, ZIP codes and census tracts, annual, 40 years; Contat & Larson (2024), abstract: an algorithm making granular repeat-sales indices feasible with low transaction counts, a balanced panel of 63,084 tract-level annual indices, and city indices built from tract aggregates that are robust to heterogeneous submarket appreciation and non-random sampling.

What an adaptive hierarchical state-space index would need to add. The SAE literature supplies the formal notion of borrowing strength with an estimated shrinkage weight and a valid MSE, which is exactly the "adaptive pooling" idea in CLAUDE.md §7; so adaptive shrinkage per se is not new. What is not standard is a dynamic, multi-level Fay–Herriot structure for index movements in which the direct estimate is a cell-level hedonic movement, the synthetic estimate is the parent-level movement, and the weights evolve over time through a state-space model with spatial neighbours — i.e. the combination of SAE (time-constant, area-level) with structural time-series (dynamic, single level) and a geographic hierarchy. Ren et al. (2017) already do data-driven clustering of tracts with latent factors in a Bayesian model, and Francke et al. (2025) already do spatially varying loadings; the new estimator would have to show, with MSE and interval-coverage evidence as SAE practice requires, that hierarchy-based pooling with an observable reliability score outperforms (or is at least comparable to) these latent-factor approaches in Quebec-like sparsity. Reporting direct vs. model-assisted cells, as in SAE practice, is a requirement not a contribution.

# 7. Forecast reconciliation / coherent hierarchies

Citations. Hyndman, Ahmed, Athanasopoulos & Shang (2011), CSDA 55(9), 2579–2589 (hyndman2011hierarchical); Wickramasuriya, Athanasopoulos & Hyndman (2019), JASA 114(526), 804–819, DOI 10.1080/01621459.2018.1448825 (wickramasuriya2019mint); Athanasopoulos, Hyndman, Kourentzes & Panagiotelis (2024), "Forecast reconciliation: A review", IJF 40(2), 430–456 (athanasopoulos2024reconciliation).

What they do. From the abstracts: hierarchical time series are collections that aggregate across products or geographies; "coherent" forecasts respect the aggregation constraints. Hyndman et al. (2011) forecast all series at all levels independently and combine them by a regression (GLS) projection that is unbiased and minimum-variance among combinations under stated assumptions. Wickramasuriya et al. (2019) show the coherency-error covariance of the 2011 approach is not identifiable, and propose MinT, which uses the full forecast-error covariance to minimise the MSE of the coherent forecasts under unbiasedness, with a closed form and a scalable implementation. The 2024 review covers cross-sectional, temporal and general linear-constraint hierarchies, and the Bayesian, probabilistic and machine-learning perspectives.

What an adaptive hierarchical state-space index would need to add. Reconciliation operates on forecasts of series that aggregate additively. Price indices do not aggregate additively: a provincial index is a weighted (stock- or transaction-weighted) combination of municipal indices in logs or levels, and the weights are themselves estimands (CLAUDE.md §15). Applying MinT-type projections to index estimates (not forecasts) therefore requires (i) specifying the linear constraint that defines coherence for the chosen aggregation (log-linear with fixed weights is the tractable case), (ii) an estimate of the covariance of the estimation errors across cells — available from the state-space posterior/filter, which is the natural link to §3 and §6 — and (iii) evidence that reconciliation reduces error rather than merely smoothing local series toward the aggregate (the review notes reconciliation "can also improve forecast accuracy", which is an empirical property, not a guarantee). If the hierarchical state-space model already imposes the hierarchy through its prior, reconciliation may be redundant; this has to be tested by ablation, and CLAUDE.md §7 requires reporting when reconciliation introduces bias or excessive smoothing.

# 8. Other verified work that constrains novelty claims (brief)

  • Hedonic imputation vs. time dummy. Diewert, Heravi & Silver (2009); Silver & Heravi (2007), JBES 25(2); de Haan (2010), Jahrbücher 230(6); Hill & Melser (2008), Economic Inquiry 46(4); Hill (2013), JES 27(5); Hill & Scholz (2018), RIW 64(4) (geospatial splines in hedonic imputation); Hill & Steurer (2026), RIW 72(1) (post-GFC survey of methods, data and policy). Any claim about quality adjustment must be positioned against hedonic imputation with flexible location controls (Hill & Scholz 2018).
  • Multilateral methods and non-revisable indexes. Ivancic, Diewert & Fox (2011), J. Econometrics 161(1) (rolling-window GEKS); de Haan & Krsinich (2014), JBES 32(3); Diewert & Fox (2022), JBES 40(1); de Haan, Hendriks & Scholz (2021), RIW 67(2). "Stable multilateral temporal linking" (CLAUDE.md §7) is a well-developed topic; window splicing options are catalogued there and in HMTS.
  • Robustness. Bourassa, Cantoni & Hoesli (2013), REE 41(3) and (2016), IJHMA 9(1) (robust repeat-sales and robust hedonic indexes); Francke & van de Minne (2017) (t errors); Hill, Pfeifer, Steurer & Trojanek (2024) (detrimental transaction prices). Robust likelihoods in an HPI are not new.
  • Unobserved heterogeneity / location. Francke & van de Minne (2021), REE 49(4): property and spatial random effects in hedonic models improve out-of-sample prediction, most for properties without prior sales — directly relevant to the alpha_location(i) term in CLAUDE.md §6.
  • Revisions. Clapp & Giaccotto (1999), REE 27(1); Clapham, Englund, Quigley & Redfearn (2006), REE 34(2); van de Minne et al. (2020); Croushore (2011), JEL 49(1) (real-time data analysis).
  • Quality adjustment with text / ML. Nowak & Smith (2020), AER: Insights 2(3): the constant-quality assumption of repeat-sales HPIs is violated in a time-varying way; textual features from agent remarks reduce the bias (up to 7% downward during the crisis, 20% upward after). Relevant to any use of listing-text or image features (CLAUDE.md §8).
  • Matching estimators. McMillen (2012), REE 40(4); Deng, McMillen & Sing (2012), RSUE 42(3); Lopez & Hewings (2018).
  • Canadian official/industry indices. Teranet–National Bank (repeat-sales, registry-based, smoothed composite; teranet2017methodology); CREA MLS HPI (hybrid repeat-sales/hedonic benchmark-price model; crea2023hpi); Statistics Canada RPPI (resale component = arithmetic repeat-sales, Shiller 1991; statcan2019rppi) and NHPI technical guide (statcan2022nhpi). No Bank of Canada or CMHC methodology document was found (see unverified.md).

# 9. Summary of the gap as it stands after this scan

Every individual ingredient listed in CLAUDE.md §6–7 exists in the verified literature: structural time-series index levels (Francke 2010), hierarchical common-plus-deviation trends with t errors (Francke & van de Minne 2017), spatially varying latent factors for tracts (Francke et al. 2025; Ren et al. 2017), estimated shrinkage with MSE (Fay–Herriot / SAE), state-space smoothing with splicing for small domains in official statistics (HMTS, Ishaak et al. 2024), rolling-window hedonics with low-volume safeguards (RTD, Hill et al. 2022), nowcast-based revision reduction (Sayag et al. 2022), and coherent hierarchical combination (MinT). What has not been found in a single published estimator is the combination of (a) a hedonic single-sale observation equation, (b) a multi-level administrative hierarchy (province / region / MRC / municipality × type), (c) time-varying, reliability-driven pooling weights rather than constant variance hyper-parameters, (d) real-time vintage evaluation with an explicit registration-lag model, and (e) coherence-aware aggregation evaluated by ablation. The contribution, if any, is this combination and its documented performance in Quebec-like sparse markets; it should be described as an integration and extension of Francke & van de Minne (2017) and Ishaak et al. (2024), not as a new class of model.