# PROGRESS.md — WP31 PRISME phase log

| date | phase | status | commands / evidence | measurements | gate | blockers / next |
|---|---|---|---|---|---|---|
| 2026-10-29 | 0 env audit | done | `scripts/environment_audit.py --smoke` → `hardware/device_inventory.json`; `scripts/cluster_audit.sh` (read-only SSH) | laptop M5 Max 18c/40 GPU/48 GB, torch 2.15 MPS+CPU smoke OK; 12 LAN nodes reachable incl. 2× M3 Ultra 96 GB, M2 Ultra 64 GB, 3× M4 Max 64 GB | PASS | — |
| 2026-10-29 | 1 literature | done | `literature/papers.bib` (77 verified entries), `comparison_matrix.csv`, `novelty_audit.md` | all 12 mandated refs exist incl. RecurTrace 2609.03379 and ELR 2610.12582; C3 (differential zero-init bypass) judged not novel; C2 novel only in its multi-action+propensity form | PASS | preregistration frozen before any test split is read (see protocol/) |
| 2026-10-29 | 2 backbone + counts | done | `scripts/count_params.py --all-configs --device meta`; `scripts/estimate_flops.py` | backbone counts match brief exactly (99,502,848 / 446,744,064 / 1,149,339,648); workspace 2,631,173 vs brief 3,088,260 (reconciled, D3); ops table reproduces brief within 1.3 % | PASS | — |
| 2026-10-29 | 3 invariants | done | `python -m pytest tests -q` → 22 passed | 15 mandated invariants + controller target-freedom + dense ablation + 5 cache-equivalence tests | PASS | — |
| 2026-10-29 | 4 learnability (backbone) | done | `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) | exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) | PASS (weak backbone, usable comparator) | stronger v2 backbone (5.6M, 15k steps) running |
| 2026-10-29 | 4 learnability (overfit) | done | `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` | 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) | PASS (gradient path + capacity) | — |
| 2026-10-29 | 4 learnability (generalisation, v1 backbone) | done | zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) | after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) | PASS (weak, learnable; mechanism effect ≈ 1 %) | decisive 5-seed matrix on v2 backbone |
| 2026-10-30 | 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence}; dense running) | done (PRISME, GR) | `scripts/launch_seed_matrix.sh`; `scripts/statistics.py` → `results/analyzed/{seed_matrix_v2,statistics_v2}.json` | dev-iid loss vs budget 0 (0.4621): PRISME −2.1/−2.8/−3.0/−1.0 % at b=1/2/4/8 (seed SD ≤ 0.003); GR −3.6/−4.1/−3.7/−3.3 %; EM dev-iid 37.7 → PRISME 39.6 (b2, +1.9 pp, t-CI [1.6, 2.2]) vs GR 39.6 (b2) / 40.0 (b8); PRISME − GR at b2 +0.06 pp [−0.53, +0.65]; OOD regimes unchanged (≤ 1 pp) | H1 weak PASS (< 3 pp prereg minimum); sparse ≤ global at every budget | dense seeds pending (M4BP36 lost; relaunched) |
| 2026-10-30 | 6 controller on v2 editors (4 seeds) | done | `scripts/run_stage_cd.sh` → `results/analyzed/routing_seeds_v2.json` | locked controller: dev-iid loss +0.5 % ± 1.1 vs budget 0 (0.05–4.3 edits/token); best fixed schedule −4.0 %; oracle b8 −15.0 %; calib sign acc 0.50–0.63, captured 0.30–0.52 on calib audit but no transfer to dev rollouts | H3 null replicated across seeds | ranking-loss variant (D12) |
| 2026-10-30 | 5 counterfactual labels + oracle audit (tiny) | done | `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` | gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 | PASS (oracle margin exists; action choice is ≈40 % of it) | controller fit |
| 2026-10-30 | 6 controller (tiny) | partial | `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) | controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity | OPEN (H3 null so far at tiny scale) | rerun with early stopping; more labels |
| 2026-10-30 | 7 inference engine + latency (tiny) | done | `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` | batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits | PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) | re-measure at 103M |
| 2026-10-30 | 8 pilot Stage A (103M backbone) | done | M3U96a run `pretrain103m_seed01_20261029232706` | 6,000 steps = 98.3M tokens (not 196.6M: stale config, D13), 2.3 h, 13.8k tok/s, peak 28.0 GB, val loss 3.875 (ppl 48.2); enlarged 102.2M baseline relaunched with the identical 6k-step recipe on M3U96b | PASS | Stage B pilots |
| 2026-10-30 | 8 pilot Stage B (103M adapters) | running | M3U96a editor lr 1e-3 → collapse (killed), relaunch lr 3e-4 (`editor_pilot_103m_lr3e4`); M4M64a GR pilot done (`gr_pilot_103m_seed01_20261030015117`, 20.5 min, 24.6M tokens, peak 9.7 GB); M2U64 dense probe (reduced budget) | GR: val loss 3.9889 → 3.9858 at b=4 (−0.08 %); editor lr 3e-4 at step 500: −0.02 % | adapters on web text ≈ no effect so far | labels/controller/routing/latency at 103M after the editor completes |
| 2026-10-30 | 5/6 routing evaluation (tiny v1 editor, dev-iid, 640 items / 3,417 answer tokens) | done | `evaluation.routing` → `results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json` | budget 0: 0.4334; uniform b1 0.4301 (−0.8 %), random b1 0.4305, fixed pair b1 0.4306; b8 schedules +0.6 to +2.8 %; entropy router ≥ budget 0; norm router (thr 0.2) 0.4309; learned controller (locked λ=0.3) 0.4404 (+1.6 %) and worse at every λ; greedy one-step oracle 0.4025 (b1) → 0.3666 (b8, 2.26 edits/token, −15 %); two-step oracle gain 0.034 vs one-step 0.0233 vs greedy-then-best 0.0337 (myopia regret ≈ 0) | H2 null (uniform = random = pair); H3 null (controller < no routing on dev) at this scale; oracle opportunity large | labels from a split unseen by the editor (D11); 5-seed v2 matrix |
| 2026-10-30 | 6 ranking-loss controller (D12, tiny v1 editor, editor-unseen labels) | done | `controller_v1rank.yaml` → `results/raw/routing_v1rank.json` | locked λ=0.3 κ=0.5: dev-iid loss 0.4333 vs 0.4334 (0.07 edits/token); λ=0: +1.3 % | H3 null stands | — |
| 2026-10-30 | 4 dense-writes ablation (v2 backbone, 5 seeds) | done | `p4v2_dense.yaml` → `results/analyzed/seed_matrix_v2.json` | dev-iid loss −2.5/−3.2/−3.2/−1.6 % at b=1/2/4/8 (vs sparse −2.1/−2.8/−3.0/−1.0, GR −3.6/−4.1/−3.7/−3.3); EM +2.0 pp (b1), t-CI [1.6, 2.4] | sparse local writes ≤ dense (H2 second half null; 5 seeds: −2.4/−3.1/−3.1/−1.5 %, EM +2.0 pp [1.6, 2.4]) | — |
| 2026-10-30 | 8–10 pilot stages B–D + routing + latency (103M) | done | `editor_pilot_103m_lr3e4` (3000 steps, 90 min, peak 25.9 GB), `labels_103m` (491k labels, 32,768 audit states), `controller_103m`, `routing_103m_val.json`, `latency_103m_mps.json`, `eval_gr_103m_val.json` | editor −0.04/−0.06/−0.08/−0.06 % at b=1/2/4/8; GR −0.08 % (ppl 48.16 → 48.01); controller locked (λ=0.3, κ=0.5) −0.01 % with 0.61 edits/token, calib sign acc 51 %, captured 7 %; oracle b8 −3.6 % (4.3 edits/token); two-step oracle 0.0162 vs one-step 0.0088 nats; latency batch 1: 7.55 ms (b0) → 22.9 (8 single) / 26.9 (8 dense) / 11.0 (controller) ms; dense 103M probe collapsed twice | H1 not supported on web text; H3 null; dispatch overhead still dominant (sparse 3.0× vs dense 3.6× measured, analytic 1.6× vs 3.1×) | enlarged baseline eval, final audit |
| 2026-10-30 | 9 locked test evaluation (synthetic, pre-registered primary outcome) | done | prereg frozen v1.0 04:04, manifest unsealed 04:04; `test_eval_v2_s*.yaml` on 5 nodes → `results/raw/eval_p4v2_*_test_*.json`, `results/analyzed/test_statistics.json` | pooled EM (iid+template+depth) b2 vs b0: PRISME +0.9 pp [+0.9, +1.0]; PRISME − GR −0.3 [−0.6, +0.1]; PRISME − dense +0.4 [+0.1, +0.7]; compose 4–5 % all systems | pre-registered threshold (3 pp) NOT met → do not advance | FineWeb test perplexity (4 systems) running |
| 2026-10-30 | 9 locked FineWeb test (4 systems) | done | `locked_evaluation.yaml` → `results/raw/eval_{backbone_103m,prisme_103m_editor,gr_103m,enlarged_102m}_{val,test}.json` | test ppl: backbone 45.72; editor/GR −0.08 % loss at b=4; enlarged plain decoder 45.64 (val 47.89 vs 48.16) | larger ordinary Transformer wins at matched training (pre-registered null) | — |
| 2026-10-30 | 11 MLX / MacLustr engineering | not executed | — | PyTorch/MPS measurements only | SKIPPED (resource/time; declared in paper) | — |
| 2026-10-30 | 12 scaling (459M / 1.18B) | meta-instantiation only | `scripts/count_params.py`, `scripts/estimate_flops.py` | exact counts + ops table | no training (declared) | — |
| 2026-10-30 | 13 statistics + audit | done | `scripts/statistics.py`, `scripts/verify_manuscript_claims.py` → AUDIT PASS (25 claims), `reproducibility/audit_report.md`, `checksums.sha256` | — | PASS | — |
| 2026-10-30 | 14 manuscript | done | `paper/main.pdf` 48 p., 0 undefined refs, 78 verified refs, tables/figures generated | — | PASS | — |
| 2026-10-30 | 15 handoff | done | `REPRODUCE.md`, `scripts/reproduce_pilot.sh` (56 s CPU end-to-end), `reproducibility/run_index.md` (52 runs), model/data cards | — | PASS | next experiment: within-state ranking labels at a scale where arithmetic dominates dispatch, GR as baseline |