Incident I1: M3U96b and M4M64a frozen under MPS load; mitigations
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +22 −0
added
docs/INCIDENTS.md
+22 −0
@@ -0,0 +1,22 @@ | |||||
| 1 | +# Incidents | ||||
| 2 | + | ||||
| 3 | +## I1 — 2026-11-09 ~23:30 UTC: M3U96b and M4M64a unreachable during Medium-scale denoising pretraining | ||||
| 4 | + | ||||
| 5 | +- **Symptoms.** Both nodes stopped answering ping and SSH (banner timeout) within the same hour. M3U96a and M1M32 | ||||
| 6 | + answered normally, so it is not a LAN outage. Same signature as the WP44 incidents (2026-11-03): Apple Silicon nodes | ||||
| 7 | + freeze under sustained MPS load close to the unified-memory limit. | ||||
| 8 | +- **What was running.** M4M64a: `denoise_m1_small_medium` (130 M params; the first attempt logged 62.9 GB of MPS | ||||
| 9 | + driver memory on a 64 GB machine before being restarted with a smaller batch budget) plus a queued fine-tune chain. | ||||
| 10 | + M3U96b: `denoise_m1_tiny_medium` (21 GB) at step ≈ 7 000 plus a queued fine-tune chain; both nodes also held the | ||||
| 11 | + 784 MB `passages.jsonl` of medium_v0 in RAM through the `DenoiseDataset` (258 k protected passages in memory). | ||||
| 12 | +- **Impact.** Both Medium denoisers interrupted. `denoise_m1_tiny_medium` has a resumable `last` checkpoint (evaluations | ||||
| 13 | + every 1 000 steps; best val CE 2.26 at step 7 000); `denoise_m1_small_medium` had not reached its first evaluation | ||||
| 14 | + (no checkpoint). No data lost (raw/parsed corpora live on the laptop and in git-tracked summaries). | ||||
| 15 | +- **Mitigations adopted.** | ||||
| 16 | + 1. Cap MPS memory explicitly: `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.6` for every remote training run, and keep | ||||
| 17 | + `max_tokens × model size` well under half of the node's RAM (the trainer logs `mem_gb`). | ||||
| 18 | + 2. Never run data loading + training of two jobs on the same node; one training process per node. | ||||
| 19 | + 3. Stream the denoising dataset lazily (do not materialise 258 k protected passages) — see `DenoiseDataset` TODO. | ||||
| 20 | + 4. Resume from `last` where available; otherwise restart with the capped settings. | ||||
| 21 | +- **Follow-up.** Nodes rebooted automatically or manually (see the next entry when they return); the cluster's | ||||
| 22 | + `mld heal` runs every 5 min on M1M32 and restarts PM2/launchd services but not these ad-hoc jobs. | ||||
| 23 | |||||