spb/wp14_uqo

1% ============================================================================2\section{Robustness and Limitations}3\label{sec:robustness}4% ============================================================================56\subsection{Inference and Subsample Stability}7\label{subsec:robust}89Several features of the design guard against spurious results. The two-part10specification (Section~\ref{subsec:econometrics}) ensures that estimates are not11driven by the non-random availability of living area: coefficients on attributes12common to both models retain their sign and rough magnitude whether or not area is13conditioned on. The ``\texttt{has\_}'' indicators accompanying each imputed field let14the regression separate observed from imputed information, so median imputation does15not silently attenuate the size coefficients. The price-per-square-foot trim removes16the residual parsing errors that would otherwise dominate the tails, and the17municipality fixed effects absorb the first-order spatial confound emphasized by18\citet{can1992spatial} and \citet{anselin1988spatial}.1920\paragraph{Inference.} Because listings within a municipality share unobserved21local shocks, our preferred standard errors are clustered at the municipality level22(195 clusters; \citealp{cameron2015cluster}), the same level as the fixed effects.23Clustering matters exactly where theory predicts: the text- and vision-derived24quality attributes---which vary house by house within a municipality---are25essentially unaffected (curb appeal, renovation level, luxury, garage, materials,26landscaping, and the rural-setting discount all remain significant at the 5\% level27or better in Model~A, as do the LLM condition scale, renovated kitchen, and hardwood28indicators), whereas keyword amenities with a strong spatial component lose29precision (public-transit proximity falls to the 5\% boundary in Model~A and is30insignificant in Model~B; central air conditioning is insignificant in Model~B; the31curb-appeal score, precisely estimated in Model~A, loses significance in the broader32Model~B). Heteroskedasticity-robust (HC3) standard errors33\citep{white1980heteroskedasticity,mackinnon1985some} yield the same qualitative34pattern with uniformly smaller intervals. The last column of35Table~\ref{tab:rob_ai_coefs} reports 95\% intervals from a pairs cluster bootstrap36that resamples the 195 municipalities with replacement; they are wider than the37analytical ones, as expected with clusters of very unequal size38\citep{mackinnon2017wild}, and they exclude zero for curb appeal, the renovation39level, the luxury cue, the visible garage, brick, stone, vinyl, the rural setting,40the LLM condition scale, and the renovated kitchen; the intervals for landscaping and41hardwood flooring include zero by a small margin. This asymmetry between42property-level and location-level regressors supports our main claim: the43AI-generated attributes capture within-market, property-level quality variation44rather than repackaged location effects.4546\input{tables/rob_ai_coefs}4748Table~\ref{tab:robustness} probes the stability of the predictive blend across49subsamples. Three patterns stand out. First, accuracy is comparable whether living50area is observed (median error 13.5\%) or imputed (14.4\%), so the area-free model51prices the incomplete listings almost as well as the rich model. Second, the model52fits the broker-listed Realtor.ca inventory much better ($R^2_{\text{OOS}}=0.80$)53than the smaller for-sale-by-owner DuProprio segment (0.58), though median errors are54identical at 14.0\%; the gap is consistent with the compositional and marketing55differences between brokered and FSBO platforms documented by \citet{hendel2009mls},56and with the absence of the room block for DuProprio listings. Third, the median57error is stable across price terciles (11--16\%), lowest in the mid-market and58modestly higher in the tails, where idiosyncratic and luxury attributes are harder59to capture. (Within a single tercile $R^2$ is uninformative because the price range60is restricted; the median error is the comparable metric.)6162\input{tables/robustness}6364\subsection{Language of the Description}65\label{subsec:robust_language}6667Section~\ref{subsec:text_content} showed that English-language listings are68substantially more expensive than French ones. Column~(2) of69Table~\ref{tab:rob_ai_coefs} adds the two language indicators and the description70length to Model~A. The English indicator is large and precisely estimated: an71English description is associated with a 7.2\% higher asking price conditional on72every structural, amenity, text, and vision attribute and on the municipality73(11.8\% in Model~B), and the mixed-language indicator with 4.6\%. We read this as a74location and clientele effect within municipalities---English listings cluster in the75most expensive parts of large municipalities such as Montreal, and brokers write in76the language of the clientele they target---rather than as a price of the language77itself. What matters for our purpose is that the AI-derived implicit prices are78unchanged to the third decimal when language is controlled for: curb appeal moves79from 0.023 to 0.022, the luxury cue stays at 0.051, brick at 0.032--0.033, vinyl80at $-0.044$, and the LLM condition and kitchen coefficients at 0.017 and 0.025. The81language indicators also leave the explanatory gain of the AI layers intact82(Panel~A of Table~\ref{tab:rob_language_source}: 0.719 $\to$ 0.727 $\to$ 0.757 with83language controls, against 0.717 $\to$ 0.725 $\to$ 0.755 without).8485The gains are present within each language group. Re-estimating the nested ablation86of Model~A separately on French and English descriptions87(Table~\ref{tab:rob_language_source}, Panel~A), the text layer adds 1.5 points of88$R^2$ for French listings and 0.6 for English ones, and the vision layer 3.4 and892.2 points; out of sample, gradient boosting gains 1.5 points from the text in both90groups and 3.1 and 2.6 from vision (Panel~C). The smaller text gain91for English listings is consistent with a specification that already fits English92listings better (base $R^2$ of 0.763 against 0.704), so that less is left for the93text to explain, rather than with a weaker extraction in English. Adding the94language indicators to the gradient-boosting feature set raises its out-of-sample95$R^2$ from 0.770 to 0.774, a modest gain that confirms language is informative but96largely redundant with location and the other attributes.9798\input{tables/rob_language_source}99100\subsection{Information Leakage Through the Text}101\label{subsec:robust_leakage}102103Because every DuProprio description contains the asking price, the clean test of104the text layer is the Realtor.ca subsample. The contrast in Panel~A of105Table~\ref{tab:rob_language_source} is stark. On Realtor.ca listings the text layer106adds 0.6 points of in-sample $R^2$ to Model~A and the vision layer 2.4 points; on107DuProprio listings the increments are 3.3 and 4.1 points. Out of sample108(Panel~C), gradient boosting gains 1.2 points from the text on Realtor.ca and 3.1109on DuProprio. Three remarks temper the comparison. First, the DuProprio base is much110weaker (in-sample $R^2$ of 0.564 against 0.779) because the room block is missing,111so every additional block has more left to explain there; the vision layer, which112never sees any text, also gains more on DuProprio (4.1 against 2.4 points), and the113ratio of the DuProprio to the Realtor.ca gain is about 1.7 for vision against 5 for114text. Second, the DuProprio ``description'' also embeds structured fields (cladding,115heating, inclusions) that the LLM can read verbatim and that are genuinely116informative, so part of the excess text gain is legitimate. Third, the asking price117itself cannot be reproduced by any field of the schema; what the price can do is tilt118the model's reading of ambiguous fields (condition, turnkey, luxury-type highlights)119towards the expensive or the cheap end. We cannot separate these channels without120re-running the extraction on descriptions purged of the price, which we recommend as121the first step of any replication and which the archived raw text makes possible122at a cost of about \$1. Until then, we regard the Realtor.ca increments---0.6 points123of in-sample $R^2$ and 0.6 (OLS) to 1.2 (gradient boosting) points out of sample for124the text layer---as the credible estimate of what the LLM specifications add beyond125keyword amenities, and we note that none of the paper's conclusions about the126vision layer is affected, since the vision model is exposed neither to the text nor127to the price. Column~(4) of Table~\ref{tab:rob_ai_coefs} re-estimates Model~A on128Realtor.ca listings alone with RTA fixed effects and language controls: curb appeal129(0.021), the renovation level (0.041), the luxury cue (0.042), the visible garage130(0.041), stone ($0.030$), vinyl ($-0.051$), landscaping (0.027), and the renovated131kitchen (0.030) retain their magnitude and significance, whereas brick, the rural132setting, the LLM condition scale, and hardwood flooring keep their sign but lose133significance in this smaller sample of 5{,}781 listings.134135\subsection{Finer Location Controls and Spatial Validation}136\label{subsec:robust_spatial}137138Municipalities are coarse spatial units: Montreal, Quebec City, Laval, and Gatineau139each contain neighbourhoods whose price levels differ by a factor of two or more.140Column~(3) of Table~\ref{tab:rob_ai_coefs} replaces the 195 municipality effects by141250 RTA effects. In-sample fit improves (Model~A $R^2$ from 0.755 to 0.764,142Model~B from 0.725 to 0.726), and the AI coefficients are stable: curb appeal 0.016,143renovation level 0.029, luxury 0.045, garage 0.037, brick 0.031, stone 0.037, vinyl144$-0.047$, rural $-0.122$, LLM condition 0.015, renovated kitchen 0.019, hardwood1450.016, all significant at the 5\% level with standard errors clustered by RTA. The146in-sample increments from the two AI layers under RTA effects are 0.7 and 3.2147points (Table~\ref{tab:rob_language_source}, last row of Panel~A). Finer location148controls therefore do not absorb the AI-measured quality, which is what one expects149if the layers measure property-level rather than neighbourhood-level attributes.150151Random cross-validation folds let every held-out listing borrow strength from152neighbours in the training folds. Table~\ref{tab:rob_spatial} repeats the nested153ablation with folds formed by RTA, so that an entire postal area is predicted from a154model that has never seen it. All levels fall, as they must: the full155text\,$+$\,vision OLS drops from 0.686 to 0.638 and gradient boosting from 0.770156to 0.734, with median absolute errors rising by about 1.5 percentage points. The157increments from the AI layers, however, are preserved almost exactly---the text layer158adds 0.7 (OLS) and 1.3 (gradient boosting) points under spatial folds against 0.9159and 1.4 under random folds, and the vision layer 7.0 and 3.1 points against 5.8 and1602.7. If anything, the vision layer is \emph{more} valuable when location is poorly161measured, which is consistent with the façade carrying information about the162neighbourhood (density, setting, housing vintage) as well as about the house. We163regard the random-fold numbers as the relevant ones for an AVM operating within its164coverage area, and the spatial-fold numbers as the relevant ones for extrapolation to165new areas.166167\input{tables/rob_spatial}168169\subsection{Limitations}170\label{subsec:limits}171172Seven limitations qualify the interpretation of our estimates.173174\emph{Asking prices.} The dependent variable is the asking price of properties175listed for sale, not the transaction price. The list price is a strategic choice176that anchors buyer search and bargaining177\citep{horowitz1992list,knight2002listing,han2015microstructure,han2016asking}: it is178set with sale in mind and is highly correlated with the eventual transaction price,179but list-to-sale ratios vary systematically with market conditions and overpricing180is more common in thin segments. Two implications follow. First, our implicit prices181measure how sellers and their agents \emph{price} attributes into the ask, which182converges to buyers' valuation only insofar as the market disciplines the ask; the183economically sensible magnitudes and the monotone validation of184Section~\ref{subsec:validation} suggest the discipline is substantial. Second, any185attribute that shifts the list-to-sale ratio itself (e.g., attributes of hard-to-sell186properties) will carry a coefficient that mixes valuation and listing strategy.187Replicating the analysis on registry transaction prices is the natural next step.188189\emph{Price in the text.} For the 19.7\% of listings from DuProprio, the language190model read the asking price as part of the description. The text-layer gains on191that platform are therefore an upper bound, and the Realtor.ca gains the credible192estimate (Section~\ref{subsec:robust_leakage}). The vision layer is unaffected.193194\emph{Non-causality and content endogeneity.} Descriptions and photographs are195marketing instruments chosen by sellers and agents. Richer descriptions, more196photographs, and higher visual quality correlate with higher-end listing agents and197more valuable properties \citep{nowak2017textual}. The coefficients are therefore198\emph{predictive} associations, not causal effects; they should not be read as the199price change from physically adding a given attribute. We control for two observable200dimensions of marketing effort (photo count and virtual tours) and for the photo201quality score, but cannot rule out that residual marketing quality loads on the202AI-derived attributes, and the three visual quality scores are collinear enough that203only their joint contribution is well identified.204205\emph{Machine-generated features and their reproducibility.} LLM and vision outputs206contain errors. Strict schemas and closed enumerations bound the \emph{form} of the207output but not its accuracy, and the agreement analysis of208Section~\ref{subsec:validation} shows that accuracy varies widely across fields:209high for concrete amenities, low for waterfront, the visual setting, and the visual210building type. We have no human-labelled validation set; a few hundred manually211coded listings would allow field-level precision and recall against ground truth212and are the obvious next step. Classical measurement error in the extracted scores213biases their coefficients towards zero, which makes our implicit prices214conservative, but errors correlated with price (through the leakage channel above,215or because the model rates expensive-looking houses more generously on every216dimension) need not. The text extraction ran at temperature zero, the vision217extraction at the API default temperature, and both model identifiers are provider218aliases rather than pinned snapshots; the raw outputs are archived, so the219regressors are exactly reproducible from them, but a fresh extraction with a later220model version could return different scores, and we have not measured test--retest221variability. The vision layer judges each property from a single façade photograph222and ignores interiors.223224\emph{Language.} The LLM processes French and English descriptions with the same225French-language instruction and schema. We find no evidence that the extraction is226weaker in either language and the implicit prices are robust to language controls,227but the English indicator itself carries a large conditional premium that we can228interpret only as a proxy for unobserved location and clientele.229230\emph{Selection.} Listed properties are not a random sample of the housing stock,231and listings with missing area or room detail are not missing at random (they tend232to be less complete listings, and all DuProprio listings lack the room block). The233``\texttt{has\_}'' indicators and the two-part design mitigate, but do not eliminate,234the second concern, and the markedly worse fit for DuProprio shows that the pooled235model is not equally adequate on both platforms.236237\emph{Cross-section.} The data are a single-date snapshot, so the model captures no238market dynamics, no forward-in-time holdout is possible, and the location premia are239fixed as of 3~July~2026. Municipalities with fewer than 20 listings are pooled,240reducing spatial resolution where data are sparse, and some municipality effects in241Model~A rest on very few observations (Section~\ref{subsec:location}).242