# Incidents

# I1 — 2026-11-09 ~23:30 UTC: M3U96b and M4M64a unreachable during Medium-scale denoising pretraining

• Symptoms. Both nodes stopped answering ping and SSH (banner timeout) within the same hour. M3U96a and M1M32 answered normally, so it is not a LAN outage. Same signature as the WP44 incidents (2026-11-03): Apple Silicon nodes freeze under sustained MPS load close to the unified-memory limit.
• What was running. M4M64a: denoise_m1_small_medium (130 M params; the first attempt logged 62.9 GB of MPS driver memory on a 64 GB machine before being restarted with a smaller batch budget) plus a queued fine-tune chain. M3U96b: denoise_m1_tiny_medium (21 GB) at step ≈ 7 000 plus a queued fine-tune chain; both nodes also held the 784 MB passages.jsonl of medium_v0 in RAM through the DenoiseDataset (258 k protected passages in memory).
• Impact. Both Medium denoisers interrupted. denoise_m1_tiny_medium has a resumable last checkpoint (evaluations every 1 000 steps; best val CE 2.26 at step 7 000); denoise_m1_small_medium had not reached its first evaluation (no checkpoint). No data lost (raw/parsed corpora live on the laptop and in git-tracked summaries).
• Mitigations adopted.
  1. Cap MPS memory explicitly: PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5 (both!) for every remote training run, and keep max_tokens × model size well under half of the node's RAM (the trainer logs mem_gb).
  2. Never run data loading + training of two jobs on the same node; one training process per node.
  3. Stream the denoising dataset lazily (do not materialise 258 k protected passages) — see DenoiseDataset TODO.
  4. Resume from last where available; otherwise restart with the capped settings.
• Follow-up. Nodes rebooted automatically or manually (see the next entry when they return); the cluster's mld heal runs every 5 min on M1M32 and restarts PM2/launchd services but not these ad-hoc jobs.

# I2 — 2026-11-10: invalid MPS watermark setting aborted queued runs

Setting only PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.6 makes PyTorch raise invalid low watermark ratio 1.4 (the default low ratio must not exceed the high ratio). The laptop fine-tune chain and the M4M64c denoiser chain died at start; no data affected. Fix: always set both, PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5. Chains relaunched.

I2 addendum. On the 48 GB laptop a 0.7 high-watermark (≈ 26 GB) is too tight when CPU prediction jobs also hold torch memory: the first Medium fine-tune OOMed at step ≈ 200. Laptop runs now use the default watermark with a 3 072-token batch budget; the explicit caps stay for the 64 GB cluster nodes (which froze instead of raising an error).

# I3 — 2026-11-10: cuDNN scaled-dot-product attention on the H100 (torch 2.11.0+cu128)

• Symptom A (speed). HTR-Base pretraining ran at 3.3 s/step with the GPU at 0–30 % utilisation; a synthetic batch of the same size took 0.21 s. Profiling showed aten::_cudnn_attention_forward/backward taking ~3.5 s the first time each (batch, length) shape is seen: the cuDNN SDPA backend builds an execution plan per shape, and the token-budget bucket sampler produces hundreds of shapes. Fix: torch.backends.cuda.enable_cudnn_sdp(False) in Trainer (CUDA only); step time 0.24–0.28 s, GPU 90–99 %. (Also added --num-workers to the loaders, which was not the bottleneck: noising costs 0.9 ms/item.)
• Symptom B (NaN). roberta-large and bart-large fine-tuning produced NaN gradients from the 2nd–6th step under bf16 autocast with the default (cuDNN) backend; with cuDNN SDPA disabled the same loop is finite. Same fix applied in hf_finetune.py and train_discriminator.py.
• Related, not cuDNN. facebook/bart-large still NaNs from step 2 even in fp32 (bart-base trains normally) → the backbone track uses google/flan-t5-large. resize_token_embeddings for the placeholder tokens breaks T5 v1.1 (initial CE ≈ 800 vs 1.3 without resizing; transformers 5.22) → placeholders are spelled [[C1]] for HF backbones (ph_to_ascii/ascii_to_ph). The reported B2 T5-small numbers used the resized vocabulary and are left as reported.
• Symptom C (OOM). Base (16 384-token batches, 62–76 GB) and Large cannot share one 80 GB H100; Large needs 8 192-token batches (71 GB at 12 288). Chains were made sequential / per-GPU.
• Operational. A remote pkill -f <pattern> kills the SSH shell itself when the same shell's command line contains the pattern (e.g. the relaunch command): issue kills and relaunches in separate SSH sessions.

# I4 — 2026-11-10: HTR-Large (1.07 B) is unstable at this recipe

Three pretraining stages on one H100 (bf16 autocast, 8 192-token batches, AdamW, cosine, grad-clip 1.0): stage 1 (lr 2e-4) diverged at step ≈ 11 500 (best val CE 1.739); stage 2 restarted from that best with lr 8e-5 diverged at ≈ 2 500 (best 1.628); stage 3 restarted from the stage-2 best with lr 5e-5 and the copy mixture / log-softmax in fp32 reached 1.534 at 5 000 steps, then drifted up again (1.59 at 13 000, 1.68 at 14 000) and its pair fine-tuning (lr 6e-5) blew up (val CE 3.2–3.4). The only usable Large system is the stage-2 fine-tune (medium_m1_large_lambda_stage2ft, system htr-m1-large-lambda, test −0.71 vs identity): under-trained, reported as a negative scaling data point. Stage-3 artifacts are kept as *_stage3* / outputs_htr-m1-large-lambda-stage3.jsonl and are not scored. Likely causes (not resolved; out of budget for this round): no QK-normalisation / pre-LN only, bf16 attention at 24 heads × d 64, AdamW eps 1e-8 with d_model 1 536. A future attempt should use fp32 attention logits or QK-norm, a lower peak lr (≤ 3e-5) and a longer warm-up.