<!-- Author: Simon-Pierre Boucher — contact@spboucher.ai -->

# data/ — not tracked by git

This directory is excluded from version control because:

1. `raw/louka.db` (268 MB) exceeds GitHub's 100 MB per-file limit;
2. the database contains a leftover application `users` table (credentials) that
   must not be published;
3. the listing texts are proprietary platform content (Realtor.ca / Centris).

## Restoring the data locally

Place the SQLite database at:

```
data/raw/louka.db        # table `properties`: 46,479 listings, 5 categories
```

(or set the environment variable `WP2_DB_PATH` to its location), then run the
pipeline from the repository root:

```bash
python3 scripts/01_prepare_data.py
python3 scripts/02_similarities.py
python3 scripts/03_models.py
python3 scripts/04_robustness.py
python3 scripts/05_figures.py
python3 scripts/06_tables.py
python3 scripts/07_revision_extensions.py   # 2026-10-17 revision: language, FE, OOS, sensitivity, multilingual encoder
python3 scripts/08_revision_tables.py
./run_all.sh                            # Stage D batches (scripts 09–20)
```

The pipeline recreates everything under `data/processed/`:

| File | Content |
|---|---|
| `houses.parquet` | analysis sample (17,087 × 14, incl. remarks text) |
| `embeddings_maisons.npy` | 17,087 × 384 sentence embeddings (cache) |
| `sim_matrix_maisons.npy` | 17,087 × 20 cosine-similarity matrix |
| `hedonic_maison_results.csv` | analysis dataset used by all models/figures |
| `embeddings_multilingual.npy` | 17,087 × 384 embeddings from paraphrase-multilingual-MiniLM-L12-v2 (cache, step 07) |
| `revision_fields.csv` | language class, FSA, municipality, lot and floor area (m²) per listing (step 07) |
| `houses_v2.parquet` | Stage D frame (17,087 × 37: float counts, raw strings, missing indicators, MLS, scrape time, FSA, municipality, lot/floor m², flags; step 09) |
| `fields_v2.parquet` | samples (S_main / S_nodup / S_floor), language labels (rule, cleaned rule, langdetect), region, flags (step 09) |
| `embeddings_{minilm_256,minilm_chunk,ml_128,ml_512,ml_chunk}[_masked].npy` | 17,087 × 384 encoder variants of the revision (step 10; SHA-256 in `environment/checksums.txt`) |
| `references_{minilm,ml}.npy`, `token_lengths.parquet` | reference embeddings; word-piece counts per listing (step 10) |
| `sim_<variant>_{raw,centred,lowdin,mean}.npy` | 17,087 × 20 similarity variants (step 11) |
| `language_sample_300.csv`, `dimension_sample_300.csv` (committed) | ids + labels of the LLM-coded validation samples (no text) |
| `_llm_sample_texts.csv`, `_reference_nearest_texts.csv` | texts for the coding passes / nearest listings — never committed |

The pre-refactor `legacy/` folder of the author's checkout is produced by no script and is not part of the pipeline.
What can be shared and what cannot (including the application `users` table, never read and never extracted) is stated
in `DATA_AVAILABILITY.md`.

Embeddings are deterministic for a given model version, so a fresh run reproduces the
committed results exactly (see `docs/CHANGES.md` §3 and `docs/CHANGELOG-REVISION.md` §1). Both
sentence-transformers models are downloaded from the Hugging Face hub on first use (about 90 MB
and 470 MB).