spb/wp20_uqo

1% ============================================================================2\section{Results I: what listing media reveal about the building}3\label{sec:results_extraction}4% ============================================================================56\subsection{What sellers photograph}7\label{sec:mnar}89Before asking how well attributes are extracted, we ask what there is to extract. Figure~\ref{fig:scene_position} classifies every photograph of the sample by the scene the small extractor assigns to it and by its position in the seller's order. The first photograph is an exterior front view in more than four cases out of five; living and dining rooms dominate positions three to ten; kitchens appear from position four onward; bedrooms, bathrooms and basements are pushed to the end of the sequence. Within the first ten photographs analysed by the mid-size extractor, \shownExterior\% of listings show at least one exterior view and \shownKitchen\% a kitchen, but only \shownBathroom\% a bathroom and \shownBasement\% a basement (Table~\ref{tab:scene_presence}); the median position of the first basement photograph, when there is one within twelve, is \firstPosBasement{}, against \firstPosExteriorFront{} for the front exterior and \firstPosKitchen{} for the kitchen. Figure~\ref{fig:scene_vs_k} shows how the share of listings with each room rises with the number of photographs analysed: kitchens saturate quickly, bathrooms slowly, and basements hardly at all within the budget we could afford.1011\begin{figure}[H]12\centering13\includegraphics[width=\textwidth]{f03_scene_by_position.pdf}14\caption{Scene of each photograph by its position in the seller's order (small extractor, up to twelve photographs per listing, \nHaikuImages{} photographs). The seller's sequence is a presentation choice: exterior first, principal rooms next, service rooms last.}15\label{fig:scene_position}16\end{figure}1718\begin{table}[H]19\centering20\caption{Which rooms are shown within the first ten photographs, and how listings that show them differ (mid-size extractor)}21\label{tab:scene_presence}22\footnotesize23\begin{adjustbox}{max width=\textwidth}24\input{tables/scene_presence}25\end{adjustbox}26\begin{tablenotes}\footnotesize27\item Note: ``Condition'' is the composite visual condition score (1--5). Among listings whose broker field describes the basement as fully finished, \basementShownGivenFinished\% show it within ten photographs (n$=$\nBasementRefFinished).28\end{tablenotes}29\end{table}3031\begin{figure}[H]32\centering33\includegraphics[width=0.62\textwidth]{f04_scene_presence_vs_k.pdf}34\caption{Share of listings with at least one photograph of the kitchen, the bathroom and the basement among the first $K$ photographs, by extractor.}35\label{fig:scene_vs_k}36\end{figure}3738This is the missing-not-at-random structure that the framework anticipated, and it has a direct consequence for the cost approach: basement finish, which adds up to a fifth of the hard cost of a bungalow in the schedule, cannot be read from photographs in \shownBasement\% of listings and must come from the broker's field or the text, while finish grade is read from kitchens and living rooms and bathroom grade is unavailable for four listings in five. Table~\ref{tab:scene_presence} also shows that the rooms left out are not left out at random: listings that show a basement ask a lower median price and are older than those that do not, and listings that do not show a kitchen ask more. The pattern is consistent with a presentation logic in which sellers of expensive or newer houses let the principal rooms and the exterior speak and omit service spaces, and with the warning in \citet{almaas2026giving} that image selection is strategic.3940\subsection{Agreement of visual attributes with reference fields}41\label{sec:agreement}4243Table~\ref{tab:agreement} and Figure~\ref{fig:agreement} report coverage and agreement for the attributes that have a reference. The references are broker fields, except for storeys (the roll) and building type (the listing's own type). Three facts stand out.4445\begin{table}[H]46\centering47\caption{Coverage and agreement of extracted attributes with reference fields (first ten photographs)}48\label{tab:agreement}49\scriptsize50\begin{adjustbox}{max width=\textwidth}51\input{tables/agreement}52\end{adjustbox}53\begin{tablenotes}\footnotesize54\item Note: coverage is the share of listings with a reference for which the extractor returns an informative value; agreement is computed on informative cases only. ``Cladding in broker siding set'' counts the primary visual cladding as agreeing when it belongs to the set of materials the broker lists. Storeys tolerate a half-storey difference; the strict agreement is reported in the text. Text rows are the language model reading the masked text, which contains the broker fields themselves; their high agreement is a check on the masking and parsing, not evidence about images. $\kappa$ is Cohen's kappa; it is undefined when the reference has a single class.55\end{tablenotes}56\end{table}5758\begin{figure}[H]59\centering60\includegraphics[width=\textwidth]{f05_agreement.pdf}61\caption{Coverage (left) and agreement with the reference (right) of visual attributes, by extractor, first ten photographs.}62\label{fig:agreement}63\end{figure}6465First, materials and equipment that are visible from the street are extracted reliably. The mid-size extractor returns an informative roof covering for \covroofcovering\% of listings with a broker roofing field and agrees with it in \accroofcovering\% of those cases ($\kappa=\kapparoofcovering$, n$=$\nroofcovering); it names the garage type (attached or detached) for \covgaragetypeattacheddetached\% of listings whose broker field has one and agrees in \accgaragetypeattacheddetached\% ($\kappa=\kappagaragetypeattacheddetached$); its primary cladding belongs to the broker's list of materials in \acccladdingprimarybrokersidings\% of the \ncladdingprimarybrokersidings{} comparable cases; and its building type agrees with the listing's type in \accbuildingtypevisual\% of the \nbuildingtypevisual{} cases where the type is informative ($\kappa=\kappabuildingtypevisual$). The confusion matrices (Figure~\ref{fig:confusion}) show where the residual disagreement lies: membranes on low-slope roofs are read as shingles when only a fa\c{c}ade is visible, and the ``other'' categories absorb cases the broker coded with local vocabulary.6667\begin{figure}[H]68\centering69\includegraphics[width=\textwidth]{f06_confusion.pdf}70\caption{Confusion matrices for roof covering (broker field vs image) and storeys (roll vs image), mid-size extractor; cells give counts, shading row-normalised shares.}71\label{fig:confusion}72\end{figure}7374Second, storeys are a different kind of attribute. The image gives a count in \covstoreysvisible\% of listings, which agrees with the roll within a half-storey in \accstoreysvisible\% of cases but exactly in only \accStrictstoreysvisible\% ($\kappa=\kappastoreysvisible$). The roll records whole storeys only; the image sees split levels, one-and-a-half-storey cottages and walk-out basements that the assessor codes as one or two. The text channel, which reads the broker's own storey count, agrees with the roll no better (\accTextstoreys\% within a half-storey, \accStrictTextstoreys\% exactly), so the disagreement is between definitions as much as between observers. This is why the administrative value overrides the visual one in B4, and why half-storey multipliers in the cost schedule are a source of uncertainty we vary.7576Third, the interior attributes that matter most for cost have the thinnest references. Basement finish is informative in the image channel for \covbasementfinish\% of listings with a broker basement field---the missingness of Section~\ref{sec:mnar}---and in those few cases agrees in \accbasementfinish\%. Kitchen counters and cabinets have a usable broker reference for a dozen listings only, too few to report. Finish grades and condition have no reference at all; they are evaluated in Section~\ref{sec:crossmodel}.7778\subsection{Information content of the channels}79\label{sec:channels}8081Table~\ref{tab:channels} compares, attribute by attribute, how often the image channel and the text channel return an informative value, how often both do, and how often they agree when they do. The channels are complementary in a structured way. Exterior attributes are an image matter: cladding is informative from images for \chanVisCladding\% of listings and from text for \chanTxtCladding\%, and the channels agree in \chanAgreeCladding\% of the cases where both speak; roof covering \chanVisRoofCovering\% against \chanTxtRoofCovering\%, agreeing in \chanAgreeRoofCovering\%. Basement finish is a text matter: images \chanVisBasementFinish\%, text \chanTxtBasementFinish\%, agreement when both \chanAgreeBasementFinish\%. Kitchen grade is mostly an image matter (\chanVisKitchenGrade\% against \chanTxtKitchenGrade\%) but the two channels agree only in \chanAgreeKitchenGrade\% of the cases where both grade the kitchen: the text's grade is the broker's adjective, the image's grade is the counter and the cabinets, and they measure different things. Garage and building type are available from both and agree in \chanAgreeGarage\% and \chanAgreeBuildingType\% respectively. The either-channel coverage column is what the multimodal baseline B3 works with; for the attributes that enter the cost model it exceeds 70\% everywhere except bathroom grade.8283\begin{table}[H]84\centering85\caption{Information content of the image and text channels (mid-size extractor, first ten photographs)}86\label{tab:channels}87\footnotesize88\begin{adjustbox}{max width=\textwidth}89\input{tables/channels}90\end{adjustbox}91\end{table}9293The text channel also reads two administrative quantities when brokers state them: the year built, which agrees with the roll within a year in \accTextyearbuilt\% of the \nTextyearbuilt{} listings where it is stated, and the building footprint, which, multiplied by the roll's storey count, falls within 15\% of the roll's floor area in \accTextbuildingfootprintstoreys\% of the \nTextbuildingfootprintstoreys{} listings that give dimensions, with a median ratio of \medianRatioTextbuildingfootprintstoreys{} and a median absolute deviation of \mdapeTextbuildingfootprintstoreys\%. The footprint result is a reminder that the roll's floor-area concept (sum of above-grade floor areas) and the broker's building dimensions describe the same house differently, and that neither should be inferred from a photograph.9495\subsection{Image budget}96\label{sec:imagecount}9798Because aggregation is deterministic, truncating the photograph set to the first $K$ in seller order measures what an analyst gets for a smaller image budget without new model calls (Table~\ref{tab:image_count}, Figure~\ref{fig:image_count}). With a single photograph, almost always the fa\c{c}ade, exterior attributes are nearly complete but no finish grade exists. With four photographs the small extractor has a kitchen for \khaikuFourKitchen\% of listings and a composite grade for \khaikuFourGrade\%; with eight, \khaikuEightKitchen\% and \khaikuEightGrade\%; with twelve, \khaikuOneTwoKitchen\% and \khaikuOneTwoGrade\%. The grade computed from eight photographs equals the one computed from twelve in most listings where the latter exists, and the composite condition moves by a median of a few hundredths of a point between eight and twelve. The basement share rises from \khaikuEightBasement\% at eight to \khaikuOneTwoBasement\% at twelve: more photographs help, but at the seller's pace, not the analyst's. The practical implication for a mass-appraisal deployment is that about eight photographs capture what the listing will reveal about grade and condition, and that no budget captures the basement.99100\begin{table}[H]101\centering102\caption{Attribute availability and stability by number of photographs analysed ($K$, seller order)}103\label{tab:image_count}104\footnotesize105\begin{adjustbox}{max width=\textwidth}106\input{tables/image_count}107\end{adjustbox}108\begin{tablenotes}\footnotesize109\item Note: ``Grade = full'' is the share of listings with a composite grade at the full budget whose grade at $K$ is identical; ``Cond.\ MAD'' is the median absolute difference of the composite condition at $K$ from its full-budget value.110\end{tablenotes}111\end{table}112113\begin{figure}[H]114\centering115\includegraphics[width=0.62\textwidth]{f16_image_count.pdf}116\caption{Availability and stability of the composite finish grade as a function of the number of photographs analysed.}117\label{fig:image_count}118\end{figure}119120\subsection{Reliability of judgment attributes across extractors}121\label{sec:crossmodel}122123Finish grades and condition scores have no external reference in our data, and expert labels do not exist (Section~\ref{sec:limitations}). What can be measured is whether different extractors, given the same photographs, reach the same judgments; disagreement between them bounds from below the measurement error of each. Table~\ref{tab:cross_model} and Figure~\ref{fig:cross_model} report pairwise agreement, Cohen's $\kappa$ and Krippendorff's nominal $\alpha$ across the tiers on the common photographs, and the correlation of condition scores. At the level of the individual photograph the small and mid-size extractors assign the same scene to \sceneAgreement\% of \nSceneImages{} images ($\kappa=\sceneKappa$). At the property level, factual attributes are reliable---roof covering agrees in \AgreeHaikuSonnetRoofCovering\% of listings where both are informative ($\kappa=\KappaHaikuSonnetRoofCovering$), with $\alpha=\alphaRoofCovering$ across the tiers---while judgment attributes are not: the kitchen grade agrees in \AgreeHaikuSonnetKitchenGrade\% ($\kappa=\KappaHaikuSonnetKitchenGrade$, $\alpha=\alphaKitchenGrade$) and the composite grade in \AgreeHaikuSonnetGradeComposite\% ($\alpha=\alphaGradeComposite$). The composite condition scores of the two extractors correlate at $r=\CorrHaikuSonnetConditionComposite$ with a median absolute difference of about two tenths of a point on the five-point scale, and the smaller extractor is slightly more generous on average. By the conventional thresholds for coding reliability, the factual attributes would be acceptable as reference labels and the judgment attributes would not; the latter are usable as ordinal proxies whose scale depends on the extractor, which is exactly how the cost model treats them, through a condition factor whose strength is a sensitivity parameter.124125\begin{table}[H]126\centering127\caption{Cross-extractor agreement on the common first ten photographs}128\label{tab:cross_model}129\scriptsize130\begin{adjustbox}{max width=\textwidth}131\input{tables/cross_model}132\end{adjustbox}133\begin{tablenotes}\footnotesize134\item Note: $\alpha$ is Krippendorff's nominal alpha across all available tiers; pairwise statistics are computed on listings where both extractors return an informative value. For continuous scores the pairwise statistic is the Pearson correlation.135\end{tablenotes}136\end{table}137138\begin{figure}[H]139\centering140\includegraphics[width=\textwidth]{f07_cross_model.pdf}141\caption{Left: Krippendorff's $\alpha$ across extractor tiers by attribute, with the 0.67 and 0.80 conventions marked. Right: composite condition scores of the two full-sample extractors for the same listings.}142\label{fig:cross_model}143\end{figure}144145\subsection{Confidence and abstention}146\label{sec:calibration}147148The schema asks each extractor for a confidence in its scene-relevant attributes. Figure~\ref{fig:calibration} and Table~\ref{tab:confidence} relate stated confidence, per exterior image, to agreement with the broker's roofing and siding fields. Stated confidence is informative but compressed: almost all values lie between 0.7 and 0.9, agreement rises with confidence within that range, and the models are slightly under-confident on these easy exterior facts. Selective prediction confirms the usable direction: keeping only images with confidence at least 0.8 retains \selsonnetZeroEightKept\% of the mid-size extractor's exterior images and raises roof agreement from \selsonnetZeroZeroAcc\% to \selsonnetZeroEightAcc\%; a 0.9 threshold retains only \selsonnetZeroNineKept\% of images. The abstention mechanism---answering \texttt{not\_visible}---is used heavily and appropriately for interior attributes on exterior photographs and for roofs seen from the street, as the coverage columns of Table~\ref{tab:agreement} show. We did not find stated confidence to be a reliable guide for the judgment attributes, whose disagreement across extractors is largely unrelated to either extractor's confidence; a review rule based on confidence alone would flag a large share of listings without targeting the uncertain ones (Section~\ref{sec:review}).149150\begin{figure}[H]151\centering152\includegraphics[width=\textwidth]{f08_calibration.pdf}153\caption{Left: agreement with broker fields by stated-confidence bin (per exterior image). Right: selective prediction---roof agreement among images retained at increasing confidence thresholds.}154\label{fig:calibration}155\end{figure}156157\begin{table}[H]158\centering159\caption{Stated confidence and agreement with broker fields, per exterior image}160\label{tab:confidence}161\footnotesize162\begin{adjustbox}{max width=\textwidth}163\input{tables/confidence_calibration}164\end{adjustbox}165\end{table}166