SPB Git

spb/wp31_uqo

Public
4.9 KB · 24 lines latex
1\subsection*{FAILURES.md --- failed hypotheses, tests and attempts (never deleted)}\label{failures.md-failed-hypotheses-tests-and-attempts-never-deleted}23\begin{itemize}4\tightlist5\item6  \textbf{date}: 2026-10-29; \textbf{what failed}: \texttt{test\_incremental\_equals\_full\_prefix\_forced}: cached decoding $\neq$ full-prefix forward.; \textbf{evidence}: pytest, atol 1e-4; \textbf{resolution}: Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4).7\item8  \textbf{date}: 2026-10-29; \textbf{what failed}: rsync exclusion \texttt{experiments} also dropped \texttt{configs/experiments/}; baseline launch on M3U96b failed twice (\texttt{FileNotFoundError}).; \textbf{evidence}: \texttt{experiments/logs/pretrain\_enlarged102m\_seed1.log} (first attempts); \textbf{resolution}: anchored pattern \texttt{/experiments}.9\item10  \textbf{date}: 2026-10-29; \textbf{what failed}: Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01--0.02. Diagnosis: scale mismatch --- backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by $\times$100 before edits can affect logits.; \textbf{evidence}: \texttt{experiments/runs/p4\_editor\_v1\_seed01\_*/metrics.jsonl} (stopped at step \textasciitilde750); \textbf{resolution}: D9: fixed non-learned \texttt{readout\_scale} = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence.11\item12  \textbf{date}: 2026-10-30; \textbf{what failed}: Editor collapse at lr 3e-3 (w = 128 mixed budgets): after 500 steps every budget gives exactly the budget-0 loss (0.4005) and the trainable gradient norm is 0.00 --- the adapter learned to make all edits inert (gate/U $\rightarrow$ 0) instead of useful.; \textbf{evidence}: M4M64a \texttt{experiments/logs/p4\_diag\_mixed\_w128\_lr3e3.log}, run \texttt{p4\_diag\_mixed\_w128\_lr3e3\_seed01\_*}; \textbf{resolution}: ``harmless collapse'' is a real failure mode of the bootstrap; lr 1e-3 retained.13\item14  \textbf{date}: 2026-10-30; \textbf{what failed}: Global-recurrence baseline run crashed at the final exact-match evaluation (\texttt{AttributeError:\ \textquotesingle{}GlobalRecurrenceModel\textquotesingle{}\ object\ has\ no\ attribute\ \textquotesingle{}W\_K\textquotesingle{}}): the cached generator assumed PRISME caches. Training and the step-3000 checkpoint are intact.; \textbf{evidence}: M5M36 \texttt{experiments/logs/p4\_gr\_mixed\_w64.log}; \textbf{resolution}: added \texttt{generate\_uncached} fallback (full recomputation) for models without the bank; EM re-run from the checkpoint.15\item16  \textbf{date}: 2026-10-30; \textbf{what failed}: Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps --- the backbone degraded under the narrower objective and the large learning rate.; \textbf{evidence}: M4M64a \texttt{experiments/logs/p4\_joint\_mixed\_w64.log}; \textbf{resolution}: Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison).17\item18  \textbf{date}: 2026-10-30; \textbf{what failed}: v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k $\rightarrow$ 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone.; \textbf{evidence}: \texttt{experiments/logs/p4\_tiny\_backbone\_v2.log}, run \texttt{p4\_tiny\_backbone\_v2\_seed01\_*} (state: stopped); \textbf{resolution}: 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator).19\item20  \textbf{date}: 2026-10-30; \textbf{what failed}: Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix \texttt{p4v2\_dense}).; \textbf{evidence}: manifests remain in \texttt{experiments/runs/} on the nodes (state: running $\rightarrow$ killed)21\item22  \textbf{date}: 2026-10-30; \textbf{what failed}: Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 \%, captured fraction --8 \%; on dev-iid the locked policy ($\lambda$=0.3, $\kappa$=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); $\lambda$=0 is worse (+1.3 \%).; \textbf{evidence}: \texttt{experiments/runs/controller\_v1rank\_seed01\_*}, \texttt{results/raw/routing\_v1rank.json}; \textbf{resolution}: Within-state ranking is not learnable from these labels at this scale; H3 null stands.23\end{itemize}24
Files · a2c46f03643fdd75e8e87595931d35e18c590d57
Menu

Theme