R03-DATA-01/EDIT-05/ADVE-07/ADVE-10/DATA-06/THEO-15/DATA-02/DATA-08: rebuild the upstream stage (00_build_sample, 00b_design_matrix, 00c --legacy, tests, DAS, environment)

Commit

- scripts/00_build_sample.py + src/wp3/build.py: DuckDB -> analytical sample; all 68 columns equal
  analytical_sample.pkl bit-for-bit (max |d| = 0); attrition, descriptives, corrected Table 3 rules
  (stories/parking = state median), lot reconciliation (123,942 + 6,115 + 211 + 3 negative = 130,271),
  duplicates (6,787 rows / 3,152 groups), listing-type shares, data window (snapshot 2026-01-08)
- data/ref/category_map.csv: frozen raw-string -> category lookup (30,038 strings)
- scripts/00b_design_matrix.py + src/wp3/design.py: one design matrix (design_v2.parquet); legacy mode
  equals model_data.pkl::X_const (max |d| = 0, means/stds ddof=1); v2 = ln lot recoded + lot_missing,
  parking/stories winsorised, implausible flag (4,970), six missingness indicators
- scripts/00c_primary_models.py --legacy --ols-only: baseline OLS reproduced (params rel 7e-12, HC3 SE
  rel 9e-8, R2 0.634277, F 18,712.47, skew 0.1776, kurtosis 5.5452; state-FE test R2 0.67757 vs 0.67754)
- tests/: 13 assertions green; environment/: requirements.lock (laptop + M4M64c), README
- data/DATA_AVAILABILITY.md: source, window, licence, SHA-256, sizes, listing types

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 6 days ago · Nov 4, 2026 parent 026a55f

3 changed files +19,996 −1 Browse files

modified .gitignore +4 −1

		`@@ -35,5 +35,8 @@ logs/`
35	35	.DS_Store
36	36
37	37	# Project-specific: DuckDB + pickles (3.2 GB), available on request
38		−data/
	38	+data/raw/*
	39	+data/processed/*
	40	+!data/ref/
39	41	results/*.pkl
	42	+!data/DATA_AVAILABILITY.md

added data/DATA_AVAILABILITY.md +46 −0

		`@@ -0,0 +1,46 @@`
	1	+# Data Availability Statement — UQO Working Paper No. 3
	2	+
	3	+Written in the November 2026 review (Stage D, items WP03-DATA-02, DATA-08, DATA-09, ADVE-18, EDIT-12, EDIT-05). Every number below is computed by `scripts/00_build_sample.py` from the raw extract (`results/build/build_meta.json`, `results/build/listing_types_raw_shares.csv`, `results/build/attrition.csv`).
	4	+
	5	+## 1. Source and access
	6	+
	7	+\| Item \| Value \|
	8	+\|---\|---\|
	9	+\| Source \| Zillow active for-sale listing pages (public web pages), collected by the author with a custom extractor into a DuckDB database. \|
	10	+\| Access method \| Web extraction of listing pages (property details, price history, nearby schools, tax history). No Zillow API key, no bulk licence; the extract is a research snapshot of publicly displayed listing attributes. \|
	11	+\| Collection window (inferred from the data; no extraction log survives) \| Last `price_history.event_date` = 2026-01-08; last `ph_first_listed_date` = 2026-01-08; last `date_sold` = 2026-01-07; earliest `price_history.event_date` = 1961-05-10 (historical sale events). The snapshot was therefore completed on 8 January 2026. `days_on_zillow` at extraction: median 72 days, IQR 34–135 (v1.2 text said 35–134: corrected). \|
	12	+\| Coverage \| 839,313 listings (zpid unique: 839,313 = 839,313 rows, i.e. no duplicate zpid in the raw table), 116 columns; all with `home_status = FOR_SALE`; 52 state codes incl. PR and VI. \|
	13	+\| Listing type (raw shares) \| For Sale by Agent 87.43 %; New Construction Plan 3.36 %; New Construction 2.94 %; Unknown Listed By 2.76 %; For Sale by Owner 1.69 %; New Construction Spec 0.71 %; Pre-Foreclosure 0.43 %; Coming Soon 0.38 %; other < 0.3 % (`results/build/listing_types_raw_shares.csv`). \|
	14	+\| Home type (analytical sample) \| SINGLE_FAMILY 85.38 %, CONDO 14.62 % (`results/build/home_types_sample.csv`). \|
	15	+\| Licence / terms \| Zillow's Terms of Use do not permit redistribution of the extracted data. The raw database and the derived pickles/parquet files are therefore not distributed; they are available from the author for verification purposes (simon-pierre.boucher@uqo.ca), subject to the same terms. The paper reports only aggregate statistics and model estimates. \|
	16	+\| Citation \| Zillow Group (2026). Active for-sale listings, United States, snapshot of 8 January 2026 [research extract, 839,313 listings]. Collected by S.-P. Boucher. \|
	17	+
	18	+## 2. Files, sizes and checksums (SHA-256)
	19	+
	20	+\| File \| Size (bytes) \| SHA-256 \| Role \|
	21	+\|---\|---\|---\|---\|
	22	+\| `data/raw/us_housing.duckdb` \| 1,626,615,808 \| `c66bb552ae3c1256792b20b986abaf6583019f11da7cedaea4ba90e19cc44dd3` \| raw extract (5 tables: `properties` 839,313 × 116, `price_history`, `schools`, `tax_history`, `meta_columns`) \|
	23	+\| `data/processed/analytical_sample.pkl` \| 456,891,130 \| `55253eb18d2ef56cdbfcd131767a5c3273cddc4ac0d3120ae311bfbbdce0d9ac` \| v1.1 analytical sample (788,842 × 68), reproduced bit-for-bit by `scripts/00_build_sample.py` \|
	24	+\| `data/processed/model_data.pkl` \| 867,098,809 \| `6876a185cab76047692554574e3b95137132beeb62b0e8f6df1092b186b807be` \| v1.1 standardised design matrix + OLS, reproduced by `scripts/00b_design_matrix.py --check-legacy` and `00c_primary_models.py --legacy` \|
	25	+\| `data/processed/shap_data.pkl` \| 7,526,409 \| `a313663e5046ff80565f0b99e33494f7684fcfed78c537ba4c75633538ab8c17` \| v1.1 SHAP sample (10,000 rows of `X_test`) \|
	26	+\| `results/ml_results.pkl` \| 436,972,197 \| `dfafa321d9249955be976daa13499856e04fafd7f336127eaefdf80652253c4a` \| v1.1 XGBoost / LightGBM models and predictions \|
	27	+\| `results/extended_results.pkl` \| 7,754 \| `99eda8204af0abb5bfd8acfe034faebba96bd227d7e8406a168b337071f4a5c8` \| v1.1 natural-unit OLS, state-FE R², RF, geographic holdout (xgb 0.5468, rf 0.5423), Moran \|
	28	+\| `results/qr_results.pkl` \| 17,041 \| `cd325395bc0ac01f984aa4821723c9df5109c324c65d5aff839f0cfa7ab4cd89` \| v1.1 quantile regressions \|
	29	+\| `results/v3_qr_imputation_results.pkl` \| 5,457 \| `1b836f2aa72061fbd6c497b0dfa48e063c66ba945115a43ed7c34e7ceb14dde8` \| v1.1 QR stability / imputation / winsorisation \|
	30	+\| `results/v3_shap_stability.pkl` \| 9,005 \| `f02934a9fb83df235f83de03da32162d9c99dbd4bd150358c92dc34f489d9a92` \| v1.1 cross-model SHAP \|
	31	+\| `results/v3_spatial_results.pkl` \| 3,843 \| `9caeb2edbf1dcb28a52ddb736a0befbc5aeb738ab500b2fa2da7c2fe303e8b9d` \| v1.1 ZIP3 FE, Moran, ablation \|
	32	+\| `data/processed/analytical_sample_v2.parquet` (generated) \| see `results/build/sha256.txt` \| — \| rebuilt sample + raw-missingness flags, lot classes, duplicate-group ids, `implausible` flag, `zip3` \|
	33	+\| `data/ref/category_map.csv` (committed) \| 1.96 MB \| `results/build/sha256.txt` \| frozen raw-string → category lookup for roof / construction / foundation (30,038 strings) \|
	34	+
	35	+## 3. From raw to analysis data
	36	+
	37	+1. `scripts/00_build_sample.py` — five filters (Table 2: 839,313 → 817,473 → 797,380 → 790,799 → 789,199 → 788,842), 68 engineered columns (rules in `src/wp3/build.py`, all state medians computed on the filtered sample), frozen category lookup, raw-missingness flags, lot-size classes (123,942 null + 6,115 zero + 211 equal to 1 sq ft + 3 negative = 130,271 listings with ln(lot) = 0 in the v1.1 coding), exact-duplicate groups (6,787 listings in 3,152 groups on ZIP × area × beds × baths × year built × price), `implausible` flag (4,970 listings: parking > 20, stories > 10, lot > 640 acres or age ≥ 200).
	38	+2. `scripts/00b_design_matrix.py` — one design matrix for every model (`design_v2.parquet`): v1.1 columns reproduced (`ln_lot_v1`, `parking_spaces_v1`, `stories_v1`) and the corrected v2 coding (ln lot recoded + indicator, parking winsorised at 20, stories at 10, six missingness indicators).
	39	+3. `scripts/00c_primary_models.py --legacy --ols-only` — baseline OLS re-estimated and asserted against the stored estimates (`results/primary/legacy_ols_check.json`).
	40	+4. `tests/` — `pytest` asserts 1–3 (`tests/test_build.py`, `test_design.py`, `test_primary_legacy.py`).
	41	+
	42	+Corrections to v1.2 surfaced by the rebuild: Table 3's imputation rule for `stories` ("set to 1") and `parking_spaces` ("set to 0") — both are state medians (67,129 of the 202,087 missing stories receive 2; 187,059 of the 191,492 missing parking counts receive 2); the "6,329 listings with a reported lot size of zero" (Table 28 note, §6.3) are 6,115 zeros + 211 lots of 1 sq ft + 3 negative lots; `days_on_zillow` IQR 34–135.
	43	+
	44	+## 4. What is not recoverable
	45	+
	46	+The v1.1 training code of the Random Forest, Ridge, Lasso and Elastic Net rows of Table 17 (hyperparameters unknown; only `r2_rf` = 0.7842 and the ten-state `rf` = 0.5423 survive in `extended_results.pkl`). These rows are re-specified in the v2 pipeline (`00c_primary_models.py --v2`) and labelled as such. The keyword precedence that collapsed the raw roof / construction / foundation strings is replaced by the frozen lookup (an ordered-keyword approximation disagrees for 1,262 / 35,007 / 43,965 listings respectively, see `improvements/WP-03/implemented_changes.md`).

added data/ref/category_map.csv +19946 −0

Large diff (19,946 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

spb wp3_uqo

R03-DATA-01/EDIT-05/ADVE-07/ADVE-10/DATA-06/THEO-15/DATA-02/DATA-08: rebuild the upstream stage (00_build_sample, 00b_design_matrix, 00c --legacy, tests, DAS, environment)

Theme