# Data — WP20

Nothing under `data/raw/` or `data/processed/` is committed except this file, `data/raw/statcan/` (public Statistics Canada
series, 40 kB) and `.gitkeep` files. See `docs/data_rights.md` for licences and conditions.

| Path | Content | Origin | Committed |
|---|---|---|---|
| `raw/statcan/bcpi_18100289_raw.json` | StatCan table 18-10-0289-01, single-detached house, division composite, Québec / Montréal / Ottawa–Gatineau / 15-CMA composite, 40 latest quarters (to 2026-Q2) | WDS API `getDataFromCubePidCoordAndLatestNPeriods`, 2026-10-20 | yes |
| (external) `immoka.db`, `vraiprix.db` | Vrai Prix production copies: listings (77,511) + `vp_eval`; assessment-roll units (3.75 M), transactions (769 k), market index | `sqlite3 .backup` from `M3U96a:~/apps/vrai-prix/data/`, 2026-10-17; paths in `configs/data.yaml` | no (3 GB; rights) |
| `processed/frame.parquet` | 7,745 house listings joined to their roll unit with broker claims parsed | `src/ingestion/build_frame.py` | no |
| `processed/sample.parquet`, `sample_images.parquet` | stratified sample of 1,200 listings, 14,245 image URLs, design weights, tier flags | `src/ingestion/sample.py` (seed 20261020) | no |
| `processed/sample_sold.parquet` | 146 frame listings whose unit sold after listing (supplement) | ad hoc, see research log | no |
| `processed/images_<tier>.parquet`, `text_<tier>.parquet` | parsed raw extraction results per image / per listing | `scripts/05_aggregate.py` | no (regenerable from `results/raw/batches/*.results.jsonl`) |
| `processed/props_<tier>_K<k>.parquet` | property-level visual attributes at image budget K | idem | no |
| `processed/levels_<tier>.parquet` | B0–B4 cost-model inputs | idem | no |
| `processed/cost_<tier>_<level>.parquet`, `exp14_mc_*.parquet`, `exp6_effects_*.parquet` | cost indications and Monte Carlo draws | `scripts/07_exp_cost.py` | no |
| `processed/market_index.csv` | Vrai Prix monthly single-family price index 2021-01 → 2026-07 | extract of `vraiprix.db` | no |
| `processed/frame_sales_after_listing.parquet` | recorded sales of frame units after listing (170) | extract of `vraiprix.db` | no |

The raw model responses (`results/raw/batches/*.results.jsonl`, ≈ 30 MB, no images) are kept outside git as well; they are
available from the author together with the manifest hashes (`results/manifest.json`).

To rebuild everything from the two databases: `make frame sample`, then `make batches` (API key required) and `make collect`
once the batches have ended, then `make aggregate experiments figures tables paper`.