spb/wpai0002_uqo

1# Incidents23## I1 — 2026-11-09 ~23:30 UTC: M3U96b and M4M64a unreachable during Medium-scale denoising pretraining45- **Symptoms.** Both nodes stopped answering ping and SSH (banner timeout) within the same hour. M3U96a and M1M326  answered normally, so it is not a LAN outage. Same signature as the WP44 incidents (2026-11-03): Apple Silicon nodes7  freeze under sustained MPS load close to the unified-memory limit.8- **What was running.** M4M64a: `denoise_m1_small_medium` (130 M params; the first attempt logged 62.9 GB of MPS9  driver memory on a 64 GB machine before being restarted with a smaller batch budget) plus a queued fine-tune chain.10  M3U96b: `denoise_m1_tiny_medium` (21 GB) at step ≈ 7 000 plus a queued fine-tune chain; both nodes also held the11  784 MB `passages.jsonl` of medium_v0 in RAM through the `DenoiseDataset` (258 k protected passages in memory).12- **Impact.** Both Medium denoisers interrupted. `denoise_m1_tiny_medium` has a resumable `last` checkpoint (evaluations13  every 1 000 steps; best val CE 2.26 at step 7 000); `denoise_m1_small_medium` had not reached its first evaluation14  (no checkpoint). No data lost (raw/parsed corpora live on the laptop and in git-tracked summaries).15- **Mitigations adopted.**16  1. Cap MPS memory explicitly: `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5` (both!) for every remote training run, and keep17     `max_tokens × model size` well under half of the node's RAM (the trainer logs `mem_gb`).18  2. Never run data loading + training of two jobs on the same node; one training process per node.19  3. Stream the denoising dataset lazily (do not materialise 258 k protected passages) — see `DenoiseDataset` TODO.20  4. Resume from `last` where available; otherwise restart with the capped settings.21- **Follow-up.** Nodes rebooted automatically or manually (see the next entry when they return); the cluster's22  `mld heal` runs every 5 min on M1M32 and restarts PM2/launchd services but not these ad-hoc jobs.2324## I2 — 2026-11-10: invalid MPS watermark setting aborted queued runs2526Setting only `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.6` makes PyTorch raise `invalid low watermark ratio 1.4` (the default low ratio must not exceed the high ratio). The laptop fine-tune chain and the M4M64c denoiser chain died at start; no data affected. Fix: always set both, `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5`. Chains relaunched.2728**I2 addendum.** On the 48 GB laptop a 0.7 high-watermark (≈ 26 GB) is too tight when CPU prediction jobs also hold torch memory: the first Medium fine-tune OOMed at step ≈ 200. Laptop runs now use the default watermark with a 3 072-token batch budget; the explicit caps stay for the 64 GB cluster nodes (which froze instead of raising an error).2930## I3 — 2026-11-10: cuDNN scaled-dot-product attention on the H100 (torch 2.11.0+cu128)3132- **Symptom A (speed).** HTR-Base pretraining ran at 3.3 s/step with the GPU at 0–30 % utilisation; a synthetic batch of the33  same size took 0.21 s. Profiling showed `aten::_cudnn_attention_forward/backward` taking ~3.5 s **the first time each34  (batch, length) shape is seen**: the cuDNN SDPA backend builds an execution plan per shape, and the token-budget bucket35  sampler produces hundreds of shapes. Fix: `torch.backends.cuda.enable_cudnn_sdp(False)` in `Trainer` (CUDA only); step36  time 0.24–0.28 s, GPU 90–99 %. (Also added `--num-workers` to the loaders, which was not the bottleneck: noising costs 0.9 ms/item.)37- **Symptom B (NaN).** `roberta-large` and `bart-large` fine-tuning produced NaN gradients from the 2nd–6th step under bf1638  autocast with the default (cuDNN) backend; with cuDNN SDPA disabled the same loop is finite. Same fix applied in39  `hf_finetune.py` and `train_discriminator.py`.40- **Related, not cuDNN.** `facebook/bart-large` still NaNs from step 2 even in fp32 (bart-base trains normally) → the41  backbone track uses `google/flan-t5-large`. `resize_token_embeddings` for the placeholder tokens breaks T5 v1.142  (initial CE ≈ 800 vs 1.3 without resizing; transformers 5.22) → placeholders are spelled `[[C1]]` for HF backbones43  (`ph_to_ascii`/`ascii_to_ph`). The reported B2 T5-small numbers used the resized vocabulary and are left as reported.44- **Symptom C (OOM).** Base (16 384-token batches, 62–76 GB) and Large cannot share one 80 GB H100; Large needs 8 192-token45  batches (71 GB at 12 288). Chains were made sequential / per-GPU.46- **Operational.** A remote `pkill -f <pattern>` kills the SSH shell itself when the same shell's command line contains the47  pattern (e.g. the relaunch command): issue kills and relaunches in separate SSH sessions.4849## I4 — 2026-11-10: HTR-Large (1.07 B) is unstable at this recipe5051Three pretraining stages on one H100 (bf16 autocast, 8 192-token batches, AdamW, cosine, grad-clip 1.0): stage 1 (lr 2e-4) diverged at52step ≈ 11 500 (best val CE 1.739); stage 2 restarted from that best with lr 8e-5 diverged at ≈ 2 500 (best 1.628); stage 3 restarted from53the stage-2 best with lr 5e-5 **and the copy mixture / log-softmax in fp32** reached 1.534 at 5 000 steps, then drifted up again54(1.59 at 13 000, 1.68 at 14 000) and its pair fine-tuning (lr 6e-5) blew up (val CE 3.2–3.4). The only usable Large system is the55stage-2 fine-tune (`medium_m1_large_lambda_stage2ft`, system `htr-m1-large-lambda`, test −0.71 vs identity): under-trained, reported as56a negative scaling data point. Stage-3 artifacts are kept as `*_stage3*` / `outputs_htr-m1-large-lambda-stage3.jsonl` and are not57scored. Likely causes (not resolved; out of budget for this round): no QK-normalisation / pre-LN only, bf16 attention at 24 heads ×58d 64, AdamW eps 1e-8 with d_model 1 536. A future attempt should use fp32 attention logits or QK-norm, a lower peak lr (≤ 3e-5) and59a longer warm-up.60