COMPUTE_LOG.md
2026-07-13 — Stage A pilot benchmarks (local: Apple Silicon, macOS, Python 3.14, polars 1.40)
| Operation | Input | Wall time | Notes |
|---|---|---|---|
| Download 1992-06 (24 MB) | HF anonymous | 3.3 s | ~7 MB/s |
| Download 2009-06 (270 MB) | HF anonymous | 24.4 s | ~11 MB/s |
| Download 2025-12 (401 MB) | HF anonymous | 19.0 s | ~21 MB/s |
| Full audit of 3 months (58.8M rows) | polars eager | ~40 s total | fits comfortably in RAM per month |
Projections for full run
- Full download 87.7 GB: ~1.5–2.5 h serial anonymous; ~10× faster sharded across MacLustr
nodes with
--shard k/n(downloader supports node sharding natively). - Storage: raw 88 GB; clean RTH panel est. 60–80 GB; features est. 100–150 GB; total ≲ 350 GB. Local free: ~1.0 TB (checked 2026-07-13).
- Largest single month 464 MB compressed (~2–4 GB in RAM) — per-month processing is single-node safe on every cluster node with ≥16 GB RAM.