data/ — not tracked by git
This directory is excluded from version control because:
raw/louka.db(268 MB) exceeds GitHub's 100 MB per-file limit;- the database contains a leftover application
userstable (credentials) that must not be published; - the listing texts are proprietary platform content (Realtor.ca / Centris).
Restoring the data locally
Place the SQLite database at:
data/raw/louka.db # table `properties`: 46,479 listings, 5 categories(or set the environment variable WP2_DB_PATH to its location), then run the
pipeline from the repository root:
python3 scripts/01_prepare_data.py
python3 scripts/02_similarities.py
python3 scripts/03_models.py
python3 scripts/04_robustness.py
python3 scripts/05_figures.py
python3 scripts/06_tables.py
python3 scripts/07_revision_extensions.py # 2026-10-17 revision: language, FE, OOS, sensitivity, multilingual encoder
python3 scripts/08_revision_tables.py
./run_all.sh # Stage D batches (scripts 09–20)The pipeline recreates everything under data/processed/:
| File | Content |
|---|---|
houses.parquet |
analysis sample (17,087 × 14, incl. remarks text) |
embeddings_maisons.npy |
17,087 × 384 sentence embeddings (cache) |
sim_matrix_maisons.npy |
17,087 × 20 cosine-similarity matrix |
hedonic_maison_results.csv |
analysis dataset used by all models/figures |
embeddings_multilingual.npy |
17,087 × 384 embeddings from paraphrase-multilingual-MiniLM-L12-v2 (cache, step 07) |
revision_fields.csv |
language class, FSA, municipality, lot and floor area (m²) per listing (step 07) |
houses_v2.parquet |
Stage D frame (17,087 × 37: float counts, raw strings, missing indicators, MLS, scrape time, FSA, municipality, lot/floor m², flags; step 09) |
fields_v2.parquet |
samples (S_main / S_nodup / S_floor), language labels (rule, cleaned rule, langdetect), region, flags (step 09) |
embeddings_{minilm_256,minilm_chunk,ml_128,ml_512,ml_chunk}[_masked].npy |
17,087 × 384 encoder variants of the revision (step 10; SHA-256 in environment/checksums.txt) |
references_{minilm,ml}.npy, token_lengths.parquet |
reference embeddings; word-piece counts per listing (step 10) |
sim_<variant>_{raw,centred,lowdin,mean}.npy |
17,087 × 20 similarity variants (step 11) |
language_sample_300.csv, dimension_sample_300.csv (committed) |
ids + labels of the LLM-coded validation samples (no text) |
_llm_sample_texts.csv, _reference_nearest_texts.csv |
texts for the coding passes / nearest listings — never committed |
The pre-refactor legacy/ folder of the author's checkout is produced by no script and is not part of the pipeline.
What can be shared and what cannot (including the application users table, never read and never extracted) is stated
in DATA_AVAILABILITY.md.
Embeddings are deterministic for a given model version, so a fresh run reproduces the
committed results exactly (see docs/CHANGES.md §3 and docs/CHANGELOG-REVISION.md §1). Both
sentence-transformers models are downloaded from the Hugging Face hub on first use (about 90 MB
and 470 MB).