1# FAILURES.md — failed hypotheses, tests and attempts (never deleted)23| date | what failed | evidence | resolution |4|---|---|---|---|5| 2026-10-29 | `test_incremental_equals_full_prefix_forced`: cached decoding ≠ full-prefix forward. | pytest, atol 1e-4 | Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). |6| 2026-10-29 | rsync exclusion `experiments` also dropped `configs/experiments/`; baseline launch on M3U96b failed twice (`FileNotFoundError`). | `experiments/logs/pretrain_enlarged102m_seed1.log` (first attempts) | anchored pattern `/experiments`. |7| 2026-10-29 | Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01–0.02. Diagnosis: scale mismatch — backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by ×100 before edits can affect logits. | `experiments/runs/p4_editor_v1_seed01_*/metrics.jsonl` (stopped at step ~750) | D9: fixed non-learned `readout_scale` = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. |8| 2026-10-30 | Editor collapse at lr 3e-3 (w = 128 mixed budgets): after 500 steps every budget gives exactly the budget-0 loss (0.4005) and the trainable gradient norm is 0.00 — the adapter learned to make all edits inert (gate/U → 0) instead of useful. | M4M64a `experiments/logs/p4_diag_mixed_w128_lr3e3.log`, run `p4_diag_mixed_w128_lr3e3_seed01_*` | "harmless collapse" is a real failure mode of the bootstrap; lr 1e-3 retained. |9| 2026-10-30 | Global-recurrence baseline run crashed at the final exact-match evaluation (`AttributeError: 'GlobalRecurrenceModel' object has no attribute 'W_K'`): the cached generator assumed PRISME caches. Training and the step-3000 checkpoint are intact. | M5M36 `experiments/logs/p4_gr_mixed_w64.log` | added `generate_uncached` fallback (full recomputation) for models without the bank; EM re-run from the checkpoint. |10| 2026-10-30 | Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. | M4M64a `experiments/logs/p4_joint_mixed_w64.log` | Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). |11| 2026-10-30 | v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. | `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) | 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). |12| 2026-10-30 | Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). | manifests remain in `experiments/runs/` on the nodes (state: running → killed) | — |13| 2026-10-30 | Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 %, captured fraction −8 %; on dev-iid the locked policy (λ=0.3, κ=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); λ=0 is worse (+1.3 %). | `experiments/runs/controller_v1rank_seed01_*`, `results/raw/routing_v1rank.json` | Within-state ranking is not learnable from these labels at this scale; H3 null stands. |14| 2026-10-30 | Stage A ran 6,000 steps instead of the intended 12,000 (stale config on the node, see rsync exclusion bug above); the enlarged baseline ran the 12,000-step config until step ~6,540 and was killed. | `experiments/runs/pretrain103m_seed01_20261029232706/manifest.json` (resolved_train.steps = 6000), M3U96b log | Backbone budget fixed at 98.3M tokens (D13); baseline relaunched with the same 6,000-step recipe. |15| 2026-10-30 | 103M editor pilot at lr 1e-3 collapsed to inert edits within 500 steps (val loss identical at every budget, 3.9889; trainable gradient norm 0.00 from step 160) — the same failure as the tiny w=128 lr 3e-3 run. | M3U96a `experiments/logs/editor_pilot_103m_seed1.log`, run `editor_pilot_103m_seed01_*` (killed at step ~650) | relaunched with lr 3e-4 (D14). |16