# Incident log — WP44 (reproducibility record)

| # | When (ET) | What | Impact | Recovery |
|---|---|---|---|---|
| I-1 | 2026-11-03 ~15:45 | `openalex_snapshot.py --id-list` second pass: workers had `ID_SET = None` (macOS spawn start method does not inherit module globals) | 68 part files marked error, no data loss | pool initializer loads the id set per worker; ledger reset; rerun |
| I-2 | 2026-11-03 ~15:50 | DuckDB "Too many open files" reading 1,898 Parquet files per table | linkage step failed | `compact_tables.py` (one Parquet file per table) + `ulimit -n 65536` |
| I-3 | 2026-11-03 ~16:30 | `multiprocessing.Pool` + pyarrow in spawned workers hangs on macOS (0 % CPU, no progress) for the classifier | classification test stalled 10 min | shard mode: independent processes started by `run_classify_shards.sh` (two nodes) |
| I-4 | 2026-11-03 ~16:58 | **M3U96b froze** (ping 100 % loss, SSH "host is down") while running the second snapshot pass (8 curl+parse workers), the SPECTER2 embedding job (MPS) and earlier DuckDB jobs concurrently | second pass (≈ 1,400/1,898 files done), the chained corpus v2 / classification, and the M3U96b embedding share interrupted; the three other embedding nodes unaffected | automatic restart expected (`mld harden`); all jobs are resumable from ledgers/shards; rule adopted: never run ingestion, embeddings and classification concurrently on one node |
| I-5 | 2026-11-03 17:16 | **M3U96b froze again** two minutes after the second pass was restarted alone (8 download+parse workers) on a freshly rebooted node | pass 2 interrupted a second time; chain and classification watchers lost | pass 2 moved to the idle node M5M36 (10 workers, resumable ledger copied); M3U96b reserved for DuckDB/classification; chain now waits for a flag raised after the pass-2 outputs are copied back; Apple diagnostics show autobugcapture_Performance reports, no kernel panic found in the quick log search |
| I-6 | 2026-11-03 17:17 | **M4M64a froze a second time** within two minutes of restarting its SPECTER2 embedding share (batch 32); M3U96b rebooted at the same minute while running pass 2 alone | M4M64a embedding share lost again | M4M64a and M3U96b withdrawn from GPU/embedding and bulk-download work; ranges reassigned to M4M64c (58–77 queued) and M3U96a (0–29 queued); pass 2 on M5M36 |