| I-1 |
2026-11-03 ~15:45 |
openalex_snapshot.py --id-list second pass: workers had ID_SET = None (macOS spawn start method does not inherit module globals) |
68 part files marked error, no data loss |
pool initializer loads the id set per worker; ledger reset; rerun |
| I-2 |
2026-11-03 ~15:50 |
DuckDB "Too many open files" reading 1,898 Parquet files per table |
linkage step failed |
compact_tables.py (one Parquet file per table) + ulimit -n 65536 |
| I-3 |
2026-11-03 ~16:30 |
multiprocessing.Pool + pyarrow in spawned workers hangs on macOS (0 % CPU, no progress) for the classifier |
classification test stalled 10 min |
shard mode: independent processes started by run_classify_shards.sh (two nodes) |
| I-4 |
2026-11-03 ~16:58 |
M3U96b froze (ping 100 % loss, SSH "host is down") while running the second snapshot pass (8 curl+parse workers), the SPECTER2 embedding job (MPS) and earlier DuckDB jobs concurrently |
second pass (≈ 1,400/1,898 files done), the chained corpus v2 / classification, and the M3U96b embedding share interrupted; the three other embedding nodes unaffected |
automatic restart expected (mld harden); all jobs are resumable from ledgers/shards; rule adopted: never run ingestion, embeddings and classification concurrently on one node |
| I-5 |
2026-11-03 17:16 |
M3U96b froze again two minutes after the second pass was restarted alone (8 download+parse workers) on a freshly rebooted node |
pass 2 interrupted a second time; chain and classification watchers lost |
pass 2 moved to the idle node M5M36 (10 workers, resumable ledger copied); M3U96b reserved for DuckDB/classification; chain now waits for a flag raised after the pass-2 outputs are copied back; Apple diagnostics show autobugcapture_Performance reports, no kernel panic found in the quick log search |
| I-6 |
2026-11-03 17:17 |
M4M64a froze a second time within two minutes of restarting its SPECTER2 embedding share (batch 32); M3U96b rebooted at the same minute while running pass 2 alone |
M4M64a embedding share lost again |
M4M64a and M3U96b withdrawn from GPU/embedding and bulk-download work; ranges reassigned to M4M64c (58–77 queued) and M3U96a (0–29 queued); pass 2 on M5M36 |