spb/wp20_uqo

1% ============================================================================2\section{Pipeline implementation, baselines and validation protocol}3\label{sec:methodology}4% ============================================================================56\subsection{Pipeline}78Figure~\ref{fig:pipeline} shows the pipeline as implemented. Listings are matched to roll units and filtered (Section~\ref{sec:sample}); the first twelve photographs are sent one at a time, by URL, to a vision--language model with a frozen system prompt and a strict JSON schema of 22 fields (Appendix~\ref{app:prompts}); the masked text is sent once per listing to a language model with a second schema of 22 fields; the per-image outputs are aggregated by the deterministic rules of Section~\ref{sec:aggregation}; visual, text and administrative attributes are reconciled by \eqref{eq:reconcile}; the cost schedule and depreciation model of Section~\ref{sec:schedule} produce the indication; and a Monte Carlo layer propagates attribute and parameter uncertainty. Every raw model response, prompt version, schema version and model identifier is persisted; no step after the model calls involves any fitted parameter.910\begin{figure}[H]11\centering12\begin{tikzpicture}[node distance=4mm and 6mm, every node/.style={font=\footnotesize},13  box/.style={draw, rounded corners=2pt, align=center, minimum height=8mm, text width=27mm, fill=white},14  src/.style={box, fill=blue!6}, ai/.style={box, fill=orange!10}, det/.style={box, fill=green!8}, outp/.style={box, fill=gray!10},15  arr/.style={-{Latex[length=2mm]}, thick, gray!70!black}]16\node[src] (list) {Listing: photos (URL), description, broker fields};17\node[src, right=of list] (roll) {Assessment roll unit (15 m match)};18\node[det, below=of list, xshift=10mm] (qc) {Source quality control: rules R1--R10, price masking};19\node[ai, below left=of qc, xshift=-4mm] (vlm) {Per-image VLM extraction (3 tiers, strict schema)};20\node[ai, below right=of qc, xshift=4mm] (llm) {Text LLM extraction (masked text)};21\node[det, below=of vlm] (agg) {Cross-image aggregation (votes, medians, means)};22\node[det, below right=of agg, xshift=4mm] (rec) {Reconciliation: admin $\succ$ visual $\succ$ text; status flags};23\node[det, below=of rec] (cost) {Cost schedule: $RCN$, add-ons, mark-ups, regional/temporal index};24\node[det, left=of cost] (dep) {Depreciation: condition-adjusted effective age, age--life};25\node[src, right=of cost] (land) {Land: roll land value, indexed};26\node[outp, below=of cost] (ind) {Cost indication $\widehat V$ with Monte Carlo interval};27\node[outp, right=of ind] (bench) {Benchmarks: assessor building value, asking price, recorded sales};28\draw[arr] (list) -- (qc); \draw[arr] (roll) -- (qc); \draw[arr] (qc) -- (vlm); \draw[arr] (qc) -- (llm);29\draw[arr] (vlm) -- (agg); \draw[arr] (agg) -- (rec); \draw[arr] (llm) -- (rec); \draw[arr] (roll.east) to[out=0,in=90] ([xshift=10mm]rec.north);30\draw[arr] (rec) -- (cost); \draw[arr] (rec) to[out=-150,in=90] (dep.north); \draw[arr] (dep) -- (cost); \draw[arr] (land) -- (cost);31\draw[arr] (cost) -- (ind); \draw[arr] (ind) -- (bench);32\end{tikzpicture}33\caption{The extraction-to-valuation pipeline. Blue: data sources; orange: model calls; green: deterministic, parameter-free or sourced steps; grey: outputs and benchmarks.}34\label{fig:pipeline}35\end{figure}3637\subsection{Extractors}3839Three model tiers of one provider family were used so that extractor capability is itself a treatment: a small model (Claude Haiku~5.5) on all \nSample{} listings and up to twelve photographs each (\nHaikuImages{} image calls), a mid-size model (Claude Sonnet~5.5) on all listings and the first ten photographs (\nSonnetImages{} calls), and the largest model (Claude Opus~5.5) on a nested random subsample of 200 listings and ten photographs. The largest tier's text batch was returned and is used in the text-channel statistics; its image batch (1,994 requests) had not been returned by the provider when this version was compiled, so the cross-extractor tables cover the two full-sample tiers and the image-based Opus results are deferred to the next version. All calls went through the provider's asynchronous batch interface with the same prompt and schema; sampling parameters were left at their defaults and reasoning effort at its lowest setting, since the task is per-image classification. The schema forces every field to be filled, with explicit \texttt{not\_visible} and \texttt{not\_applicable} values, and asks for a confidence in $[0,1]$ and a short statement of the visual cues used; the system prompt instructs the model never to infer hidden conditions, exact year or area. Because comparisons between tiers must hold the evidence fixed, all cross-tier statistics use the common first ten photographs; image-count sensitivity up to twelve uses the small model only. Image calls succeeded for \okSonnetPct\% (mid-size) and \okHaikuPct\% (small) of requests; failures are dead URLs and a few residual \texttt{robots.txt} refusals on secondary hosts, and are treated as missing images. Table~\ref{tab:runs} summarises the runs; total provider cost was about US\$\apiCostUSD{} at batch prices. Text calls used the same three tiers on the masked text.4041\begin{table}[H]42\centering43\caption{Extraction runs}44\label{tab:runs}45\footnotesize46\begin{adjustbox}{max width=\textwidth}47\input{tables/extraction_runs}48\end{adjustbox}49\begin{tablenotes}\footnotesize50\item Note: ``URL failures'' are images the provider could not download; ``robots.txt refusals'' are hosts whose directives disallow retrieval of the specific path. Cost is the provider's batch price for the tokens actually billed.51\end{tablenotes}52\end{table}5354\subsection{Baselines}55\label{sec:baselines}5657Five nested information sets define the baselines (Table~\ref{tab:baselines}). B0 uses only the roll and the typical assumptions an appraiser would make without an inspection: standard grade, average condition, unfinished basement, no garage, no pool, storeys and building type from the record. B1 adds the text channel---broker fields and the masked description, through the language model---for grade, condition claims, renovations, basement, garage, pool and, where stated, storeys and type. B2 adds the image channel instead of the text channel and is restricted to visually supported attributes; where no interior photograph exists it falls back to the B0 default and flags the listing. B3 reconciles images and text by \eqref{eq:reconcile}. B4 adds the administrative override (storeys from the roll) and the uncertainty flags used by the Monte Carlo layer. No baseline sees prices or assessed building values; all share the roll floor area and the roll land value, so differences between them isolate the specification of the improvements. The comparison is fair in the sense that each baseline is given exactly the information its label says and nothing else; it is unfavourable to B2 to the extent that sellers do not photograph what matters.5859\begin{table}[H]60\centering61\caption{Information sets of the baselines}62\label{tab:baselines}63\footnotesize64\begin{tabular}{llp{8.6cm}}65\toprule66Level & Channels & Attributes beyond the roll \\67\midrule68B0 & roll & none: standard grade, condition 3/5, unfinished basement, no garage, no pool \\69B1 & roll + text & grade from kitchen/bathroom claims, condition claim, renovation claim, basement, garage, pool, storeys/type if stated \\70B2 & roll + images & composite grade and condition, renovation evidence, basement finish, garage, storeys and type as seen \\71B3 & roll + images + text & B2 with text filling what images do not show; contradictions flagged \\72B4 & B3 + administrative reconciliation & storeys from the roll; uncertainty flags for missing grade/condition \\73\bottomrule74\end{tabular}75\end{table}7677\subsection{Validation protocol and metrics}7879The protocol was written before any result was examined and is released with the code. For \textsc{visual} attributes with a reference field we report \emph{coverage} (the share of listings for which the extractor gives an informative value when a reference exists), \emph{agreement} among informative cases, Cohen's $\kappa$ and macro-F1, with confusion matrices for the key attributes; broker fields are claims, so we speak of agreement, not accuracy, except for the roll's storeys and the listing's property type. For \textsc{proxy} attributes we report cross-extractor agreement (pairwise $\kappa$, Krippendorff's nominal $\alpha$, correlations of condition scores) and the effect of the attribute on the cost indication. Confidence calibration is assessed by binning per-image stated confidence against agreement with the broker field and by selective-prediction curves. The missing-not-at-random structure of photographs is measured directly: the scene of every photograph by position in seller order, the share of listings showing each room within the first $K$ photographs, and the association of showing a room with price, age, condition and the broker's own description of that room.8081For the cost indication we report, for every baseline: the median indication and its decomposition; the ratio of depreciated improvements to the assessor's building value indexed from the roll's market-conditions date to the valuation quarter with the metropolitan-area construction price index (\emph{assessor alignment}: median ratio, median absolute percentage deviation, coefficient of dispersion); and the ratio of the indication to the asking price (\emph{market alignment}: median ratio, median absolute percentage deviation, share within 20\%). Confidence intervals come from 499 cluster bootstrap replications by municipality, because listings in one municipality share an assessor and a market. A single depreciation or level parameter is calibrated against the assessor benchmark on five metropolitan areas and evaluated on the Québec City area as a geographic hold-out; prices are never used for calibration. Recorded sale prices after listing are used for a small market-alignment test, labelled as such. Everything that is a scenario (renovation, grade shift, external obsolescence) is labelled a scenario. All experiments, including those that could not be run for lack of data (human expert agreement, human-in-the-loop gains), are listed in the pre-registered plan with their status.82