spb/wp20_uqo

1# Research log — WP20 (From Listing Images to Property Valuation)23Chronological, append-only. Each entry: date, decision or finding, evidence/path. API keys are never written here.45## 2026-10-20 07:20 — Kick-off and environment67- Request: build WP20 in `~/Desktop/Recherche-UQO` following the author's CLAUDE.md specification.8- Folder created: `immobilier/multimodal-cost-approach-valuation` (HARMONISATION.md §1: next free number is 20; WP18 was created9  earlier today in `immobilier/housing-leverage-intergenerational-inequality`; no local or remote WP19 was found — the number 20 is10  the one requested by the author and is used as-is; note kept for the README). Target spbgit repo `wp20_uqo.git` (to be created11  when the gateway is reachable again).12- Blocker: laptop off the cluster LAN (IP 192.0.0.2, Tailscale logged out) → M1M32/spbgit and M3U96a (Vrai Prix production code)13  unreachable. Work proceeds on local copies (see `docs/implementation_audit.md` §1). `git push` deferred.14- Tooling: TeX Live (latexmk, pdflatex), Python 3.14 with numpy 2.4 / pandas 3.0 / scipy / statsmodels / scikit-learn / lightgbm /15  matplotlib / pyarrow / geopandas; project venv `.venv` (system-site-packages) with `anthropic` 1.15.0 added.1617## 2026-10-20 07:30 — Gate 1: repository and data audit (done)1819- Verified local copies of Vrai Prix data (`vraiprix.db`, `immoka.db` copied 2026-10-17 for the Toveli project) and of the Vrai Prix20  estimation engine (`toveli/src/lib/engine.ts`). Details and counts in `docs/implementation_audit.md`.21- Finding: Vrai Prix has no inspectable cost-approach code locally; `vp_eval.cost_est` exists as an output (54,364 listings) of a22  script not available offline. WP20 will build its own transparent cost model and treat `cost_est` as an external production23  benchmark of unknown formula.24- Finding: 20,123 house listings join to a roll unit within 15 m with complete roll fields and live images; roll vs listing year25  built agree within ±1 year in 95.3 % of cases.26- Finding: listing image URLs still resolve (HTTP 200) for all major sources, 3 months after the WP14 crawl and 3 days after the27  Immo-Ka copy.28- Decision: primary sample = Immo-Ka house listings matched to the roll (verified administrative fields available), stratified by29  region, price tier, building age, and source; WP14's Realtor/DuProprio sample kept as a secondary, text-leakage-aware sample only30  if time permits (it has no roll link).31- Decision on API use: Anthropic Message Batches API (50 % price) for all extractions; three extractor tiers (Haiku 5.5 on the full32  sample, Sonnet 5.5 on the full sample, Opus 5.5 on a subsample) so that extractor capability is itself a treatment. Spend is33  logged per batch in `results/raw/api_usage.jsonl` (tokens only).34- Decision on rights: images are fetched transiently for analysis only, never stored in the repository, never reproduced in the35  manuscript; `data/images/` is git-ignored and deleted at the end. See `docs/data_rights.md`.3637## 2026-10-20 07:40 — Sampling frame and sample (Gate 3/4 preparation)3839- `src/ingestion/build_frame.py`: 39,278 house listings evaluated by Vrai Prix → 7,745 after rules R1–R10 (cascade in40  `results/tables/frame_exclusions.csv`). Rule R10 (new, rights-driven): image hosts whose robots.txt disallows all automated41  retrieval (`media.remax-quebec.com`: `User-agent: * / Disallow: /`, checked 2026-10-20, `results/raw/robots_check.txt`) are42  excluded; the Anthropic URL fetcher independently refused those URLs ("disallowed by the website's robots.txt"). We do not43  route around this by downloading the files ourselves. 5,287 listings lost (RE/MAX Québec ≈ 40 % of the frame); balance check44  in `results/tables/robots_exclusion_balance.csv` (excluded listings: median price $589,900 vs $563,000 retained, land value45  $163,900 vs $132,600, floor area 125 vs 121 m²) → the retained frame under-represents RE/MAX's more urban, higher-land-value46  listings; reported as a limitation.47- `src/ingestion/sample.py`: stratified sample of 1,200 listings (cma × age band × price quartile, floor 20 per stratum, seed48  20261020), design weights saved (`weight`), 14,245 image URLs (first ≤ 12 in seller order), Opus subsample of 200.49- Smoke test (`scripts/03_smoke_test.py`): structured-output schemas must not use min/max/maxItems/maxLength (API 400) → scales50  encoded as integer enums. Mean tokens per image call ≈ 4,800 input / 330 output; per text call ≈ 3,400 / 280.51- Text masking bug found and fixed before submission: DuProprio `features` carried "Évaluation municipale : 429 000 $" → all52  feature strings are now masked and assessment/tax/price keys dropped; 0 / 1,200 listings retain a dollar amount.5354## 2026-10-20 07:55 — Extraction batches submitted (Anthropic Message Batches API, 50 % price)5556| batch | requests | model | images per listing |57|---|---|---|---|58| img_haiku | 14,245 | claude-haiku-5-5 | ≤ 12 |59| img_sonnet | 11,916 | claude-sonnet-5-5 | ≤ 10 |60| img_opus | 1,994 | claude-opus-5-5 | ≤ 10 (200 listings) |61| txt_haiku / txt_sonnet / txt_opus | 1,200 / 1,200 / 200 | — | masked text |6263Batch ids in `results/raw/batches/*.batch_ids.txt`; schema `wp20-schema-1.0`, prompts `wp20-prompts-1.0` (`run_meta.json`).64Expected spend ≈ US$110 at batch prices (dominant: Sonnet images); actual usage logged in `results/raw/api_usage.jsonl`.6566## 2026-10-20 09:10 — Gate 2 (literature) status and an honesty note6768- 104 candidate references → 104 verified against primary records (Crossref works API by DOI or bibliographic query with title69  similarity ≥ 0.80 and year ±1; arXiv API for 9 preprints/PMLR-NeurIPS papers), raw JSON/Atom kept in `docs/verification/`;70  `paper/references.bib` is generated from the verified metadata only. Two initial failures were resolved manually: Lindenthal &71  Johnson (2021) is published as « Machine Learning, Architectural Styles and Property Values » (JREFE, DOI 10.1007/s11146-021-09845-1),72  not under the working title first typed; Gal & Ghahramani (2016) verified via arXiv 1506.02142 after an arXiv 500/429.73  Dropped: Goodman & Thibodeau (1995, JHR, no record found → replaced by their 1998 REE article) and Cheng (1974, Public Finance, no record).74- `docs/literature_matrix.csv` content notes are written from the team's prior knowledge of each work at abstract level; the75  works were **not re-read in this session**. Consequently the manuscript cites them for general contributions only and never76  attributes specific figures to them.77- Opus 5.5 batches (1,994 image calls, 200 text calls) still processing at 09:10; all other tiers collected. API spend to date78  ≈ US$98 (Sonnet images ≈ US$79).7980## 2026-10-20 10:30 — Gates 4–7: experiments, manuscript, validation8182- Collection gap found and fixed: one Haiku image batch (4,000 requests, msgbatch_01QQiU…) had ended after the last collect83  and was missing from the first full run; collected, pipeline re-run end to end (05 → 06 → 07 → figures → tables → paper).84  Final counts: Haiku 14,245 image calls (97.1 % ok), Sonnet 11,916 (96.4 % ok), Opus text 200; Opus image batch (1,994)85  still `in_progress` at 10:30 (3 h) — manuscript v1.0 states this and defers Opus image results to v1.186  (`make collect aggregate experiments figures tables paper`).87- Key design decisions taken during analysis (all pre-dating the reading of results where they affect inference): anchor B88  (Québec 2026 guides) as headline unit cost with anchor A (Altus Montréal) as alternative; assessor-based calibration of a89  single parameter with Québec City hold-out as a diagnostic, never as the method; garage "none" reference from parking90  fields abandoned (shown unreliable against both channels); scenario effects reported among affected listings.91- Manuscript: 66 pages (main text ≈ 46 + appendices), 15 figures, 23 generated table fragments, 1,183 macros; 104 references;92  latexmk exit 0; 0 undefined references/citations; 3 overfull boxes left (table widths) to tidy.93- Validation (`scripts/12_validate_results.py`): 18/18 independent recomputations match the macros (`docs/results_validation.md`).94- Spend: ≈ US$98 before Opus images (Opus images ≈ US$18 more when billed).95- Not done / blocked: spbgit remote (gateway unreachable), production code re-inspection, MEFQ 3E text, human labels (see96  `docs/CHECKLIST.md`).9798## 2026-10-20 11:05 — Remote repository99100- Gateway still unreachable by LAN/Tailscale, but M1M32 answers on the MacLustr Tunnel hub (BHS64 `tunnelctl peer ls` → 10.67.0.2, handshake < 60 s). Bare repo `srv/git/wp20_uqo.git` created and `main` pushed with `GIT_SSH_COMMAND="ssh -o ProxyJump=BHS64 -o HostName=10.67.0.2"`; verified on the remote (7ccbf65, 2.78 MiB pack).101