WP31: final assembly — locked evaluations, audit PASS, 48-page manuscript, handoff
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
44 changed files +778 −111
modified
PROGRESS.md
+6 −0
@@ -21,3 +21,9 @@ | ||
| 21 | 21 | | 2026-10-30 | 4 dense-writes ablation (v2 backbone, 5 seeds) | done | `p4v2_dense.yaml` → `results/analyzed/seed_matrix_v2.json` | dev-iid loss −2.5/−3.2/−3.2/−1.6 % at b=1/2/4/8 (vs sparse −2.1/−2.8/−3.0/−1.0, GR −3.6/−4.1/−3.7/−3.3); EM +2.0 pp (b1), t-CI [1.6, 2.4] | sparse local writes ≤ dense (H2 second half null; 5 seeds: −2.4/−3.1/−3.1/−1.5 %, EM +2.0 pp [1.6, 2.4]) | — | |
| 22 | 22 | | 2026-10-30 | 8–10 pilot stages B–D + routing + latency (103M) | done | `editor_pilot_103m_lr3e4` (3000 steps, 90 min, peak 25.9 GB), `labels_103m` (491k labels, 32,768 audit states), `controller_103m`, `routing_103m_val.json`, `latency_103m_mps.json`, `eval_gr_103m_val.json` | editor −0.04/−0.06/−0.08/−0.06 % at b=1/2/4/8; GR −0.08 % (ppl 48.16 → 48.01); controller locked (λ=0.3, κ=0.5) −0.01 % with 0.61 edits/token, calib sign acc 51 %, captured 7 %; oracle b8 −3.6 % (4.3 edits/token); two-step oracle 0.0162 vs one-step 0.0088 nats; latency batch 1: 7.55 ms (b0) → 22.9 (8 single) / 26.9 (8 dense) / 11.0 (controller) ms; dense 103M probe collapsed twice | H1 not supported on web text; H3 null; dispatch overhead still dominant (sparse 3.0× vs dense 3.6× measured, analytic 1.6× vs 3.1×) | enlarged baseline eval, final audit | |
| 23 | 23 | | 2026-10-30 | 9 locked test evaluation (synthetic, pre-registered primary outcome) | done | prereg frozen v1.0 04:04, manifest unsealed 04:04; `test_eval_v2_s*.yaml` on 5 nodes → `results/raw/eval_p4v2_*_test_*.json`, `results/analyzed/test_statistics.json` | pooled EM (iid+template+depth) b2 vs b0: PRISME +0.9 pp [+0.9, +1.0]; PRISME − GR −0.3 [−0.6, +0.1]; PRISME − dense +0.4 [+0.1, +0.7]; compose 4–5 % all systems | pre-registered threshold (3 pp) NOT met → do not advance | FineWeb test perplexity (4 systems) running | |
| 24 | +| 2026-10-30 | 9 locked FineWeb test (4 systems) | done | `locked_evaluation.yaml` → `results/raw/eval_{backbone_103m,prisme_103m_editor,gr_103m,enlarged_102m}_{val,test}.json` | test ppl: backbone 45.72; editor/GR −0.08 % loss at b=4; enlarged plain decoder 45.64 (val 47.89 vs 48.16) | larger ordinary Transformer wins at matched training (pre-registered null) | — | | |
| 25 | +| 2026-10-30 | 11 MLX / MacLustr engineering | not executed | — | PyTorch/MPS measurements only | SKIPPED (resource/time; declared in paper) | — | | |
| 26 | +| 2026-10-30 | 12 scaling (459M / 1.18B) | meta-instantiation only | `scripts/count_params.py`, `scripts/estimate_flops.py` | exact counts + ops table | no training (declared) | — | | |
| 27 | +| 2026-10-30 | 13 statistics + audit | done | `scripts/statistics.py`, `scripts/verify_manuscript_claims.py` → AUDIT PASS (25 claims), `reproducibility/audit_report.md`, `checksums.sha256` | — | PASS | — | | |
| 28 | +| 2026-10-30 | 14 manuscript | done | `paper/main.pdf` 48 p., 0 undefined refs, 78 verified refs, tables/figures generated | — | PASS | — | | |
| 29 | +| 2026-10-30 | 15 handoff | done | `REPRODUCE.md`, `scripts/reproduce_pilot.sh` (56 s CPU end-to-end), `reproducibility/run_index.md` (52 runs), model/data cards | — | PASS | next experiment: within-state ranking labels at a scale where arithmetic dominates dispatch, GR as baseline | | |
modified
experiments/runs/p4_editor_v1_seed01_20261029233624/manifest.json
+1 −1
@@ -9,7 +9,7 @@ | ||
| 9 | 9 | "python": "3.12.13", |
| 10 | 10 | "torch": "2.15.0", |
| 11 | 11 | "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py --config configs/experiments/p4_editor_bootstrap.yaml --device mps", |
| 12 | − "state": "running", | |
| 12 | + "state": "stopped_by_operator_scale_mismatch_diagnosed", | |
| 13 | 13 | "wall_seconds": 0.02, |
| 14 | 14 | "device": "mps", |
| 15 | 15 | "backend": "mps", |
modified
paper/appendix/runs_index.tex
+38 −4
@@ -2,6 +2,8 @@ | ||
| 2 | 2 | |
| 3 | 3 | \begin{itemize} |
| 4 | 4 | \tightlist |
| 5 | +\item | |
| 6 | + \textbf{run\_id}: controller\_103m\_seed01\_20261030034451; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 315.08; \textbf{commit}: unknown | |
| 5 | 7 | \item |
| 6 | 8 | \textbf{run\_id}: controller\_mini\_seed01\_20261030005154; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: cpu; \textbf{wall s}: 3.14; \textbf{commit}: ede67055 |
| 7 | 9 | \item |
@@ -10,6 +12,10 @@ | ||
| 10 | 12 | \textbf{run\_id}: controller\_p4\_hent\_seed01\_20261030002015; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 584.36; \textbf{commit}: unknown |
| 11 | 13 | \item |
| 12 | 14 | \textbf{run\_id}: controller\_v1ctrl\_seed01\_20261030004131; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 164.25; \textbf{commit}: 33298d96 |
| 15 | +\item | |
| 16 | + \textbf{run\_id}: controller\_v1rank\_seed01\_20261030005431; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 171.86; \textbf{commit}: d6746e03 | |
| 17 | +\item | |
| 18 | + \textbf{run\_id}: controller\_v2s1\_seed01\_20261030005758; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 141.05; \textbf{commit}: unknown | |
| 13 | 19 | \item |
| 14 | 20 | \textbf{run\_id}: controller\_v2s2\_seed01\_20261030004717; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 193.23; \textbf{commit}: unknown |
| 15 | 21 | \item |
@@ -18,6 +24,14 @@ | ||
| 18 | 24 | \textbf{run\_id}: controller\_v2s4\_seed01\_20261030004719; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 217.1; \textbf{commit}: unknown |
| 19 | 25 | \item |
| 20 | 26 | \textbf{run\_id}: controller\_v2s5\_seed01\_20261030004719; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 214.63; \textbf{commit}: unknown |
| 27 | +\item | |
| 28 | + \textbf{run\_id}: editor\_pilot\_103m\_lr3e4\_seed01\_20261030021200; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 5397.54; \textbf{train tokens}: 24576000; \textbf{peak mem (GB)}: 25.9; \textbf{commit}: unknown; \textbf{notes}: prisme | |
| 29 | +\item | |
| 30 | + \textbf{run\_id}: editor\_pilot\_103m\_seed01\_20261030015139; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.24; \textbf{commit}: unknown; \textbf{notes}: prisme | |
| 31 | +\item | |
| 32 | + \textbf{run\_id}: gr\_pilot\_103m\_seed01\_20261030015117; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1230.52; \textbf{train tokens}: 24576000; \textbf{peak mem (GB)}: 9.7; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence | |
| 33 | +\item | |
| 34 | + \textbf{run\_id}: labels\_103m\_seed01\_20261030034314; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 94.09; \textbf{commit}: unknown | |
| 21 | 35 | \item |
| 22 | 36 | \textbf{run\_id}: labels\_mini\_seed01\_20261030005150; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: cpu; \textbf{wall s}: 3.44; \textbf{commit}: ede67055 |
| 23 | 37 | \item |
@@ -26,6 +40,8 @@ | ||
| 26 | 40 | \textbf{run\_id}: labels\_v1ctrl\_seed01\_20261030003957; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: 33298d96 |
| 27 | 41 | \item |
| 28 | 42 | \textbf{run\_id}: labels\_v1ctrl\_seed01\_20261030004018; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 72.1; \textbf{commit}: 33298d96 |
| 43 | +\item | |
| 44 | + \textbf{run\_id}: labels\_v2s1\_seed01\_20261030005712; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 44.65; \textbf{commit}: unknown | |
| 29 | 45 | \item |
| 30 | 46 | \textbf{run\_id}: labels\_v2s2\_seed01\_20261030004543; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 92.45; \textbf{commit}: unknown |
| 31 | 47 | \item |
@@ -43,7 +59,7 @@ | ||
| 43 | 59 | \item |
| 44 | 60 | \textbf{run\_id}: p4\_editor\_v1\_scaled\_seed01\_20261029234334; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.16; \textbf{commit}: HEAD; \textbf{notes}: prisme |
| 45 | 61 | \item |
| 46 | − \textbf{run\_id}: p4\_editor\_v1\_seed01\_20261029233624; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: prisme | |
| 62 | + \textbf{run\_id}: p4\_editor\_v1\_seed01\_20261029233624; \textbf{state}: stopped\_by\_operator\_scale\_mismatch\_diagnosed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: prisme | |
| 47 | 63 | \item |
| 48 | 64 | \textbf{run\_id}: p4\_overfit64\_b8\_seed01\_20261029235111; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 519.6; \textbf{train tokens}: 95773; \textbf{peak mem (GB)}: 6.7; \textbf{commit}: a9d94e8a; \textbf{notes}: prisme |
| 49 | 65 | \item |
@@ -51,7 +67,23 @@ | ||
| 51 | 67 | \item |
| 52 | 68 | \textbf{run\_id}: p4\_tiny\_backbone\_v2\_seed01\_20261029233624; \textbf{state}: stopped\_overfitting\_step9300\_checkpoint\_step5000\_used; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: backbone\_only |
| 53 | 69 | \item |
| 54 | − \textbf{run\_id}: p4v2\_dense\_seed05\_20261030001839; \textbf{state}: running; \textbf{host}: M4BP36; \textbf{device}: mps; \textbf{wall s}: 0.33; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 70 | + \textbf{run\_id}: p4v2\_dense\_seed01\_20261030013421; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2465.35; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 71 | +\item | |
| 72 | + \textbf{run\_id}: p4v2\_dense\_seed02\_20261030001749; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.51; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 73 | +\item | |
| 74 | + \textbf{run\_id}: p4v2\_dense\_seed02\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2601.32; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 75 | +\item | |
| 76 | + \textbf{run\_id}: p4v2\_dense\_seed03\_20261030001749; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.54; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 77 | +\item | |
| 78 | + \textbf{run\_id}: p4v2\_dense\_seed03\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2620.1; \textbf{train tokens}: 499017; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 79 | +\item | |
| 80 | + \textbf{run\_id}: p4v2\_dense\_seed04\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 3983.98; \textbf{train tokens}: 497821; \textbf{peak mem (GB)}: 28.4; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 81 | +\item | |
| 82 | + \textbf{run\_id}: p4v2\_dense\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: M4BP36; \textbf{device}: mps; \textbf{wall s}: 3249.7; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 83 | +\item | |
| 84 | + \textbf{run\_id}: p4v2\_dense\_seed05\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2620.45; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 28.2; \textbf{commit}: unknown; \textbf{notes}: dense | |
| 85 | +\item | |
| 86 | + \textbf{run\_id}: p4v2\_gr\_seed01\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1120.54; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 2.3; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence | |
| 55 | 87 | \item |
| 56 | 88 | \textbf{run\_id}: p4v2\_gr\_seed02\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 951.12; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 3.4; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence |
| 57 | 89 | \item |
@@ -60,6 +92,8 @@ | ||
| 60 | 92 | \textbf{run\_id}: p4v2\_gr\_seed04\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 430.63; \textbf{train tokens}: 497821; \textbf{peak mem (GB)}: 2.3; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence |
| 61 | 93 | \item |
| 62 | 94 | \textbf{run\_id}: p4v2\_gr\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 370.27; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 3.4; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence |
| 95 | +\item | |
| 96 | + \textbf{run\_id}: p4v2\_prisme\_seed01\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2333.13; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 8.9; \textbf{commit}: unknown; \textbf{notes}: prisme | |
| 63 | 97 | \item |
| 64 | 98 | \textbf{run\_id}: p4v2\_prisme\_seed02\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1529.74; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 8.8; \textbf{commit}: unknown; \textbf{notes}: prisme |
| 65 | 99 | \item |
@@ -69,7 +103,7 @@ | ||
| 69 | 103 | \item |
| 70 | 104 | \textbf{run\_id}: p4v2\_prisme\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 939.64; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 9.9; \textbf{commit}: unknown; \textbf{notes}: prisme |
| 71 | 105 | \item |
| 72 | − \textbf{run\_id}: pretrain103m\_seed01\_20261029232706; \textbf{state}: running; \textbf{host}: M3U96a; \textbf{device}: mps; \textbf{wall s}: 0.03; \textbf{commit}: unknown | |
| 106 | + \textbf{run\_id}: pretrain103m\_seed01\_20261029232706; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 8279.79; \textbf{train tokens}: 98304000; \textbf{peak mem (GB)}: 28.0; \textbf{commit}: unknown | |
| 73 | 107 | \item |
| 74 | − \textbf{run\_id}: pretrain\_enlarged102m\_seed01\_20261029232914; \textbf{state}: running; \textbf{host}: M3U96b; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: unknown | |
| 108 | + \textbf{run\_id}: pretrain\_enlarged102m\_seed01\_20261030015017; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 8527.03; \textbf{train tokens}: 98304000; \textbf{peak mem (GB)}: 28.5; \textbf{commit}: unknown | |
| 75 | 109 | \end{itemize} |
modified
paper/main.log
+42 −45
@@ -1,4 +1,4 @@ | ||
| 1 | −This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 04:15 | |
| 1 | +This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 04:21 | |
| 2 | 2 | entering extended mode |
| 3 | 3 | restricted \write18 enabled. |
| 4 | 4 | %&-line parsing enabled. |
@@ -1297,18 +1297,19 @@ Overfull \hbox (36.62411pt too wide) in paragraph at lines 19--24 | ||
| 1297 | 1297 | t/m/n/12 data/processed/fineweb_edu_300m/manifest.json\T1/ptm/m/n/12 (-20) ). |
| 1298 | 1298 | [] |
| 1299 | 1299 | |
| 1300 | −[16] | |
| 1301 | − | |
| 1302 | −LaTeX Warning: Reference `tab:controls' on page 17 undefined on input line 27. | |
| 1300 | +[16] (./tables/controls.tex | |
| 1301 | +Overfull \hbox (344.15573pt too wide) in paragraph at lines 7--16 | |
| 1302 | + [][] | |
| 1303 | + [] | |
| 1303 | 1304 | |
| 1304 | −) (./sections/results.tex | |
| 1305 | −Overfull \hbox (71.70984pt too wide) in paragraph at lines 38--1 | |
| 1305 | +)) (./sections/results.tex | |
| 1306 | +Overfull \hbox (71.70984pt too wide) in paragraph at lines 39--1 | |
| 1306 | 1307 | \T1/ptm/m/n/12 (-20) The pri-mary out-come, met-rics, de-ci-sion thresh-olds an |
| 1307 | 1308 | d seed plan were writ-ten to \T1/cmtt/m/n/12 protocol/preregistration.md |
| 1308 | 1309 | [] |
| 1309 | 1310 | |
| 1310 | 1311 | [17] |
| 1311 | −<../results/figures/learnability_bootstrap.pdf, id=494, 353.75287pt x 191.45125 | |
| 1312 | +<../results/figures/learnability_bootstrap.pdf, id=496, 353.75287pt x 191.45125 | |
| 1312 | 1313 | pt> |
| 1313 | 1314 | File: ../results/figures/learnability_bootstrap.pdf Graphic file (type pdf) |
| 1314 | 1315 | <use ../results/figures/learnability_bootstrap.pdf> |
@@ -1321,14 +1322,14 @@ Overfull \hbox (133.05286pt too wide) in paragraph at lines 7--21 | ||
| 1321 | 1322 | [] |
| 1322 | 1323 | |
| 1323 | 1324 | ) |
| 1324 | −<../results/figures/frontier_v2.pdf, id=499, 305.58919pt x 208.87035pt> | |
| 1325 | +<../results/figures/frontier_v2.pdf, id=501, 305.58919pt x 208.87035pt> | |
| 1325 | 1326 | File: ../results/figures/frontier_v2.pdf Graphic file (type pdf) |
| 1326 | 1327 | <use ../results/figures/frontier_v2.pdf> |
| 1327 | 1328 | Package pdftex.def Info: ../results/figures/frontier_v2.pdf used on input line |
| 1328 | 1329 | 59. |
| 1329 | 1330 | (pdftex.def) Requested size: 338.22418pt x 231.19547pt. |
| 1330 | − [18 <../results/figures/learnability_bootstrap.pdf>] [19] | |
| 1331 | −<../results/figures/labels_analysis.pdf, id=536, 548.77519pt x 192.17598pt> | |
| 1331 | + [18] [19 <../results/figures/learnability_bootstrap.pdf>] | |
| 1332 | +<../results/figures/labels_analysis.pdf, id=538, 548.77519pt x 192.17598pt> | |
| 1332 | 1333 | File: ../results/figures/labels_analysis.pdf Graphic file (type pdf) |
| 1333 | 1334 | <use ../results/figures/labels_analysis.pdf> |
| 1334 | 1335 | Package pdftex.def Info: ../results/figures/labels_analysis.pdf used on input |
@@ -1340,12 +1341,12 @@ Overfull \hbox (109.74734pt too wide) in paragraph at lines 8--15 | ||
| 1340 | 1341 | [][] |
| 1341 | 1342 | [] |
| 1342 | 1343 | |
| 1343 | −) [20 <../results/figures/frontier_v2.pdf>] (./tables/stats.tex) [21 <../result | |
| 1344 | −s/figures/labels_analysis.pdf>] (./tables/test.tex) (./tables/latency.tex) | |
| 1345 | −(./sections/results_pilot.tex [22] (./tables/pilot_ppl.tex) | |
| 1346 | −(./tables/pilot_routing.tex) (./tables/pilot_latency.tex))) | |
| 1347 | −(./sections/mechanistic.tex [23]) (./sections/limitations.tex [24]) | |
| 1348 | −(./sections/conclusion.tex [25]) (./sections/reproducibility.tex) | |
| 1344 | +) [20] (./tables/stats.tex) [21 <../results/figures/frontier_v2.pdf>] | |
| 1345 | +(./tables/test.tex) [22 <../results/figures/labels_analysis.pdf>] | |
| 1346 | +(./tables/latency.tex) (./sections/results_pilot.tex (./tables/pilot_ppl.tex) | |
| 1347 | +[23] (./tables/pilot_routing.tex) (./tables/pilot_latency.tex))) | |
| 1348 | +(./sections/mechanistic.tex [24]) (./sections/limitations.tex [25]) | |
| 1349 | +(./sections/conclusion.tex) (./sections/reproducibility.tex) | |
| 1349 | 1350 | Overfull \hbox (51.18217pt too wide) in paragraph at lines 2--101 |
| 1350 | 1351 | \T1/ptm/m/n/12 (-20) num-ber in the manuscript is pro-duced by \T1/cmtt/m/n/12 |
| 1351 | 1352 | scripts/make_tables.py \T1/ptm/m/n/12 (-20) and \T1/cmtt/m/n/12 scripts/reprodu |
@@ -1359,18 +1360,18 @@ cript_claims.py \T1/ptm/m/n/12 (-20) checks the chain. \T1/cmtt/m/n/12 REPRODUC | ||
| 1359 | 1360 | E.md |
| 1360 | 1361 | [] |
| 1361 | 1362 | |
| 1362 | −[26] (./main.bbl [27] [28] | |
| 1363 | +[26] [27] (./main.bbl [28] [29] | |
| 1363 | 1364 | Underfull \hbox (badness 1297) in paragraph at lines 208--211 |
| 1364 | 1365 | []\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe |
| 1365 | 1366 | i Ning, Ning Ding, and |
| 1366 | 1367 | [] |
| 1367 | 1368 | |
| 1368 | −[29]) (./appendix/appendix.tex (./appendix/notation.tex | |
| 1369 | +[30]) (./appendix/appendix.tex (./appendix/notation.tex | |
| 1369 | 1370 | Overfull \hbox (12.19864pt too wide) in paragraph at lines 2--16 |
| 1370 | 1371 | [] |
| 1371 | 1372 | [] |
| 1372 | 1373 | |
| 1373 | −) (./appendix/proofs.tex [30]) (./appendix/params.tex [31] | |
| 1374 | +) (./appendix/proofs.tex [31]) (./appendix/params.tex [32] | |
| 1374 | 1375 | Overfull \hbox (92.32437pt too wide) detected at line 12 |
| 1375 | 1376 | \OML/ztmcm/m/it/12 N[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/it/12 d \OT1/ztmcm/m/n/ |
| 1376 | 1377 | 12 + 3\OML/ztmcm/m/it/12 dw \OT1/ztmcm/m/n/12 + \OML/ztmcm/m/it/12 Mw \OT1/ztmc |
@@ -1479,7 +1480,7 @@ LaTeX Warning: `h' float specifier changed to `ht'. | ||
| 1479 | 1480 | ) (./appendix/prereg.tex |
| 1480 | 1481 | LaTeX Font Info: Font shape `T1/cmtt/bx/n' in size <12> not available |
| 1481 | 1482 | (Font) Font shape `T1/cmtt/m/n' tried instead on input line 1. |
| 1482 | − [32] | |
| 1483 | + [33] | |
| 1483 | 1484 | LaTeX Font Info: Trying to load font information for TS1+ptm on input line 8 |
| 1484 | 1485 | . |
| 1485 | 1486 | |
@@ -1492,7 +1493,7 @@ Overfull \hbox (80.09856pt too wide) in paragraph at lines 24--26 | ||
| 1492 | 1493 | T1/cmtt/m/n/12 data/generators/synthetic.py\T1/ptm/m/n/12 (-20) , |
| 1493 | 1494 | [] |
| 1494 | 1495 | |
| 1495 | −[33] [34]) (./appendix/runs.tex | |
| 1496 | +[34] [35]) (./appendix/runs.tex | |
| 1496 | 1497 | Overfull \hbox (121.91187pt too wide) in paragraph at lines 1--14 |
| 1497 | 1498 | \T1/cmtt/m/n/12 training.pretrain\T1/ptm/m/n/12 (-20) , \T1/cmtt/m/n/12 trainin |
| 1498 | 1499 | g.train_editor\T1/ptm/m/n/12 (-20) , \T1/cmtt/m/n/12 training.collect_intervent |
@@ -1507,14 +1508,14 @@ tion.latency \T1/ptm/m/n/12 (-20) or \T1/cmtt/m/n/12 evaluation.run\T1/ptm/m/n/ | ||
| 1507 | 1508 | 12 (-20) ; launch scripts un-der \T1/cmtt/m/n/12 scripts/ |
| 1508 | 1509 | [] |
| 1509 | 1510 | |
| 1510 | −(./appendix/runs_index.tex [35] [36] | |
| 1511 | −Overfull \hbox (25.12868pt too wide) in paragraph at lines 52--54 | |
| 1511 | +(./appendix/runs_index.tex [36] [37] | |
| 1512 | +Overfull \hbox (25.12868pt too wide) in paragraph at lines 68--70 | |
| 1512 | 1513 | []\T1/ptm/b/n/10 (-20) run_id\T1/ptm/m/n/10 (-20) : p4_tiny_backbone_v2_seed01_ |
| 1513 | 1514 | 20261029233624; \T1/ptm/b/n/10 (-20) state\T1/ptm/m/n/10 (-20) : stopped_overfi |
| 1514 | 1515 | tting_step9300_checkpoint_step5000_used; |
| 1515 | 1516 | [] |
| 1516 | 1517 | |
| 1517 | −)) (./appendix/results.tex (./tables/em_families.tex)) | |
| 1518 | +[38])) (./appendix/results.tex (./tables/em_families.tex)) | |
| 1518 | 1519 | Overfull \hbox (168.4564pt too wide) in paragraph at lines 2--16 |
| 1519 | 1520 | \T1/ptm/m/n/12 (-20) records (\T1/cmtt/m/n/12 experiments/runs/*/em_*.json\T1/p |
| 1520 | 1521 | tm/m/n/12 (-20) , \T1/cmtt/m/n/12 results/raw/items_*.json\T1/ptm/m/n/12 (-20) |
@@ -1538,14 +1539,14 @@ ileNotFoundError\T1/ptm/m/n/12 (-20) ).; \T1/ptm/b/n/12 (-20) ev-i-dence\T1/ptm | ||
| 1538 | 1539 | g |
| 1539 | 1540 | [] |
| 1540 | 1541 | |
| 1541 | −[37] | |
| 1542 | + | |
| 1542 | 1543 | Overfull \hbox (49.9369pt too wide) in paragraph at lines 10--12 |
| 1543 | 1544 | \T1/ptm/m/n/12 (-20) be-fore ed-its can af-fect log-its.; \T1/ptm/b/n/12 (-20) |
| 1544 | 1545 | ev-i-dence\T1/ptm/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/runs/p4_editor_v1_ |
| 1545 | 1546 | seed01_*/metrics.jsonl |
| 1546 | 1547 | [] |
| 1547 | 1548 | |
| 1548 | − | |
| 1549 | +[39] | |
| 1549 | 1550 | Overfull \hbox (47.44559pt too wide) in paragraph at lines 12--14 |
| 1550 | 1551 | \T1/cmtt/m/n/12 experiments/logs/p4_diag_mixed_w128_lr3e3.log\T1/ptm/m/n/12 (-2 |
| 1551 | 1552 | 0) , run \T1/cmtt/m/n/12 p4_diag_mixed_w128_lr3e3_seed01_*\T1/ptm/m/n/12 (-20) |
@@ -1570,15 +1571,15 @@ Overfull \hbox (12.79033pt too wide) in paragraph at lines 18--20 | ||
| 1570 | 1571 | run \T1/cmtt/m/n/12 p4_tiny_backbone_v2_seed01_* \T1/ptm/m/n/12 (-20) (state: |
| 1571 | 1572 | [] |
| 1572 | 1573 | |
| 1573 | −[38{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1.en | |
| 1574 | −c}] | |
| 1574 | + | |
| 1575 | 1575 | Overfull \hbox (12.01894pt too wide) in paragraph at lines 22--24 |
| 1576 | 1576 | \T1/ptm/m/n/12 (-20) 0.4334); $\OML/ztmcm/m/it/12 ^^U$\T1/ptm/m/n/12 (-20) =0 i |
| 1577 | 1577 | s worse (+1.3 %).; \T1/ptm/b/n/12 (-20) ev-i-dence\T1/ptm/m/n/12 (-20) : \T1/cm |
| 1578 | 1578 | tt/m/n/12 experiments/runs/controller_v1rank_seed01_*\T1/ptm/m/n/12 (-20) , |
| 1579 | 1579 | [] |
| 1580 | 1580 | |
| 1581 | − | |
| 1581 | +[40{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1.en | |
| 1582 | +c}] | |
| 1582 | 1583 | Overfull \hbox (146.55865pt too wide) in paragraph at lines 24--26 |
| 1583 | 1584 | \T1/ptm/m/n/12 (-20) un-til step ~6,540 and was killed.; \T1/ptm/b/n/12 (-20) e |
| 1584 | 1585 | v-i-dence\T1/ptm/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/runs/pretrain103m_s |
@@ -1599,26 +1600,22 @@ dence\T1/ptm/m/n/12 (-20) : M2U64 \T1/cmtt/m/n/12 experiments/logs/dense_probe_ | ||
| 1599 | 1600 | 103m*_seed1.log\T1/ptm/m/n/12 (-20) ; |
| 1600 | 1601 | [] |
| 1601 | 1602 | |
| 1602 | −)) [39] [40] [41] [42] [43] [44] [45] (./main.aux) | |
| 1603 | +)) [41] [42] [43] [44] [45] [46] [47] [48] (./main.aux) | |
| 1603 | 1604 | *********** |
| 1604 | 1605 | LaTeX2e <2025-11-01> |
| 1605 | 1606 | L3 programming layer <2026-01-19> |
| 1606 | 1607 | *********** |
| 1607 | − | |
| 1608 | − | |
| 1609 | −LaTeX Warning: There were undefined references. | |
| 1610 | − | |
| 1611 | 1608 | Package rerunfilecheck Info: File `main.out' has not changed. |
| 1612 | −(rerunfilecheck) Checksum: 5C078824F258CF894B89F741D112AD48;9386. | |
| 1609 | +(rerunfilecheck) Checksum: 2518BAC872CA29A47E8E1B5E242EBEEE;9386. | |
| 1613 | 1610 | ) |
| 1614 | 1611 | Here is how much of TeX's memory you used: |
| 1615 | − 36174 strings out of 467525 | |
| 1616 | − 703202 string characters out of 5418982 | |
| 1617 | − 1152637 words of memory out of 5000000 | |
| 1618 | − 63458 multiletter control sequences out of 15000+600000 | |
| 1619 | − 793836 words of font info for 673 fonts, out of 8000000 for 9000 | |
| 1612 | + 36217 strings out of 467525 | |
| 1613 | + 703862 string characters out of 5418982 | |
| 1614 | + 1163186 words of memory out of 5000000 | |
| 1615 | + 63476 multiletter control sequences out of 15000+600000 | |
| 1616 | + 794607 words of font info for 685 fonts, out of 8000000 for 9000 | |
| 1620 | 1617 | 1141 hyphenation exceptions out of 8191 |
| 1621 | − 84i,13n,104p,1001b,1484s stack positions out of 10000i,1000n,20000p,200000b,200000s | |
| 1618 | + 84i,13n,104p,1001b,1530s stack positions out of 10000i,1000n,20000p,200000b,200000s | |
| 1622 | 1619 | </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb |
| 1623 | 1620 | ></usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb> |
| 1624 | 1621 | </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></ |
@@ -1634,10 +1631,10 @@ exlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026 | ||
| 1634 | 1631 | e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti |
| 1635 | 1632 | mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8 |
| 1636 | 1633 | a.pfb> |
| 1637 | −Output written on main.pdf (45 pages, 447044 bytes). | |
| 1634 | +Output written on main.pdf (48 pages, 450927 bytes). | |
| 1638 | 1635 | PDF statistics: |
| 1639 | − 824 PDF objects out of 1000 (max. 8388607) | |
| 1640 | − 709 compressed objects within 8 object streams | |
| 1641 | − 167 named destinations out of 1000 (max. 500000) | |
| 1636 | + 839 PDF objects out of 1000 (max. 8388607) | |
| 1637 | + 721 compressed objects within 8 object streams | |
| 1638 | + 171 named destinations out of 1000 (max. 500000) | |
| 1642 | 1639 | 143785 words of extra memory for PDF output out of 154059 (max. 10000000) |
| 1643 | 1640 | |
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/sections/conclusion.tex
+1 −1
@@ -1,7 +1,7 @@ | ||
| 1 | 1 | \section{Conclusion}\label{sec:conclusion} |
| 2 | 2 | The central question was whether counterfactual supervision of \emph{where} to edit a small latent workspace |
| 3 | 3 | improves the quality--compute frontier beyond uniform workspace computation and adaptive global recurrence. |
| 4 | −On the evidence collected here the answer is no. The sparse editor is learnable and helps a frozen backbone | |
| 4 | +On the evidence collected here the answer is no, and the pre-registered decision rule says so: on the locked test regimes the primary effect is $\testprismebtwovsbzero$ points against a three-point threshold. The sparse editor is learnable and helps a frozen backbone | |
| 5 | 5 | by a few percent of loss and two points of exact match when it is applied once or twice at every position, but |
| 6 | 6 | a global-recurrence adapter trained with the identical recipe helps at least as much at every budget; where to |
| 7 | 7 | write barely matters for fixed schedules; an exhaustive oracle shows large opportunity (up to |
modified
paper/sections/design.tex
+1 −0
@@ -33,6 +33,7 @@ recipe and the identical symmetry-breaking device. Reference routers (random, un | ||
| 33 | 33 | entropy threshold, norm-change threshold) reuse the trained \prisme{} editor and are therefore matched in |
| 34 | 34 | everything except the choice of action. The exhaustive one-step and two-step oracles read the target and are |
| 35 | 35 | reported as ceilings only. |
| 36 | +\input{tables/controls.tex} | |
| 36 | 37 | |
| 37 | 38 | \subsection{Pre-registration, seeds and uncertainty} |
| 38 | 39 | The primary outcome, metrics, decision thresholds and seed plan were written to |
modified
paper/sections/results_pilot.tex
+6 −2
@@ -17,8 +17,12 @@ global adapter, which did not collapse at $10^{-3}$, gives $-\pilotgrgainbone$, | ||
| 17 | 17 | $-\pilotgrgainbfour$ and $-\pilotgrgainbeight\,\%$ in \pilotgrminutes{} min. The dense sixteen-slot probe |
| 18 | 18 | collapsed at both learning rates and yields no 103M result. On general web text, therefore, neither adapter |
| 19 | 19 | moves a frozen 100M backbone by more than a tenth of a percent of loss with 24.6M adapter tokens---less than the |
| 20 | −run-to-run noise of pretraining itself---so H1 is not supported outside the synthetic suite, and the | |
| 21 | −parameter-matched plain decoder is the relevant comparison (Table~\ref{tab:pilotppl}). | |
| 20 | +run-to-run noise of pretraining itself---so H1 is not supported outside the synthetic suite. On the locked | |
| 21 | +FineWeb-Edu test split (327,680 tokens) the backbone's perplexity is \pilotpplbackbonemtest{}; forced edits | |
| 22 | +change the test loss by \pilottestprismemeditorbfour\,\% (sparse, budget 4) and \pilottestgrmbfour\,\% | |
| 23 | +(global); the parameter-matched plain decoder trained with the identical recipe reaches \pilotpplenlargedmtest{} | |
| 24 | +(Table~\ref{tab:pilotppl})---the proposal's null ``a larger ordinary Transformer wins after equalising training | |
| 25 | +compute'' holds at this budget. | |
| 22 | 26 | \input{tables/pilot_ppl.tex} |
| 23 | 27 | |
| 24 | 28 | \paragraph{Stages C--D and routing.} Counterfactual labels were collected on a validation-disjoint stream of |
modified
paper/tables/controls.tex
+11 −10
@@ -1,15 +1,16 @@ | ||
| 1 | 1 | \begin{table}[t]\centering\small |
| 2 | −\caption{Systems and controls. ``bb'' = backbone, ``6NP'' = the usual training-FLOP estimate $6\,N\,P$ for $N$ tokens | |
| 3 | −and $P$ parameters; adapter training additionally executes up to 8 editor applications per position. Adapter | |
| 4 | −parameters are unique parameters outside the backbone (counted by instantiation).} | |
| 2 | +\caption{Systems and controls. ``bb'' = backbone, ``$6NP$'' = the usual training-FLOP estimate for $N$ tokens and $P$ | |
| 3 | +parameters; adapter training additionally executes up to 8 editor applications per position. Adapter parameters are | |
| 4 | +unique parameters outside the backbone (counted by instantiation; the 103M editor includes the residual-plus-entropy | |
| 5 | +controller features).} | |
| 5 | 6 | \label{tab:controls} |
| 6 | 7 | \begin{tabular}{lrrlll}\toprule |
| 7 | 8 | System & backbone & adapter & training exposure & training compute & inference regime\\\midrule |
| 8 | −Frozen backbone v2 (budget 0) & 4,754,688 & 182,725 & synthetic train, 5k steps x 64 & backbone only & no workspace (bitwise bypass)\\ | |
| 9 | −PRISME sparse editor (tiny) & 4,754,688 & 182,725 & same + 3k adapter steps x 32 & frozen bb fwd + editor & 1-2 slots/action, 0-8 actions\\ | |
| 10 | −Global-recurrence adapter (tiny) & 4,754,688 & 172,608 & same & frozen bb fwd + block & whole latent/action, 0-8 actions\\ | |
| 11 | −Dense 16-slot editor (tiny) & 4,754,688 & 182,725 & same & frozen bb fwd + 16 lanes & 16 slots/action, 0-8 actions\\ | |
| 12 | −PRISME-103M (editor pilot) & 99,502,848 & 2,762,757 & FineWeb-Edu 196.6M tok + 24.6M adapter tok & 6NP + editor & 1-2 slots/action\\ | |
| 13 | −Global-recurrence adapter (103M) & 99,502,848 & 2,623,488 & same & 6NP + block & whole latent/action\\ | |
| 14 | −Plain decoder, parameter-matched (102.2M) & 102,157,056 & 0 & FineWeb-Edu 196.6M tok (same recipe) & 6NP & no workspace\\ | |
| 9 | +Frozen backbone v2 (budget 0) & 4,754,688 & 182,725 & synthetic train, 5k steps $\times$ 64 & backbone only & no workspace (bitwise bypass)\\ | |
| 10 | +PRISME sparse editor (tiny) & 4,754,688 & 182,725 & same + 3k adapter steps $\times$ 32 & frozen bb fwd + editor & 1--2 slots/action, 0--8 actions\\ | |
| 11 | +Global-recurrence adapter (tiny) & 4,754,688 & 172,608 & same & frozen bb fwd + block & whole latent/action, 0--8 actions\\ | |
| 12 | +Dense 16-slot editor (tiny) & 4,754,688 & 182,725 & same & frozen bb fwd + 16 lanes & 16 slots/action, 0--8 actions\\ | |
| 13 | +PRISME-103M sparse editor & 99,502,848 & 2,762,757 & FineWeb-Edu 98.3M tok + 24.6M adapter tok & $6NP$ + editor & 1--2 slots/action\\ | |
| 14 | +Global-recurrence adapter (103M) & 99,502,848 & 2,623,488 & same & $6NP$ + block & whole latent/action\\ | |
| 15 | +Plain decoder, parameter-matched (102.2M) & 102,157,056 & 0 & FineWeb-Edu 98.3M tok (same recipe) & $6NP$ & no workspace\\ | |
| 15 | 16 | \bottomrule\end{tabular}\end{table} |
modified
paper/tables/numbers.tex
+16 −1
@@ -63,12 +63,27 @@ | ||
| 63 | 63 | \newcommand{\pilotlatctrl}{11.0} |
| 64 | 64 | \newcommand{\pilotlatdeight}{26.9} |
| 65 | 65 | \newcommand{\pilotlatseight}{22.9} |
| 66 | −\newcommand{\pilotpplgrm}{48.16} | |
| 66 | +\newcommand{\pilotpplbackbonemtest}{45.72} | |
| 67 | +\newcommand{\pilotpplbackbonemval}{48.16} | |
| 68 | +\newcommand{\pilotpplenlargedmtest}{45.64} | |
| 69 | +\newcommand{\pilotpplenlargedmval}{47.89} | |
| 70 | +\newcommand{\pilotpplgrmtest}{45.72} | |
| 71 | +\newcommand{\pilotpplgrmval}{48.16} | |
| 72 | +\newcommand{\pilotpplprismemeditortest}{45.72} | |
| 73 | +\newcommand{\pilotpplprismemeditorval}{48.16} | |
| 67 | 74 | \newcommand{\pilotroutingctrl}{-0.01} |
| 68 | 75 | \newcommand{\pilotroutingctrledits}{0.61} |
| 69 | 76 | \newcommand{\pilotroutingoracle}{-3.6} |
| 70 | 77 | \newcommand{\pilotroutingoracleedits}{4.28} |
| 71 | 78 | \newcommand{\pilotroutinguniformone}{-0.02} |
| 79 | +\newcommand{\pilottestbackbonembfour}{+0.00} | |
| 80 | +\newcommand{\pilottestbackbonembtwo}{+0.00} | |
| 81 | +\newcommand{\pilottestenlargedmbfour}{+0.00} | |
| 82 | +\newcommand{\pilottestenlargedmbtwo}{+0.00} | |
| 83 | +\newcommand{\pilottestgrmbfour}{-0.08} | |
| 84 | +\newcommand{\pilottestgrmbtwo}{-0.06} | |
| 85 | +\newcommand{\pilottestprismemeditorbfour}{-0.08} | |
| 86 | +\newcommand{\pilottestprismemeditorbtwo}{-0.06} | |
| 72 | 87 | \newcommand{\routingbzero}{0.4334} |
| 73 | 88 | \newcommand{\routingctrllambda}{0.3} |
| 74 | 89 | \newcommand{\routingctrlledits}{0.22} |
modified
paper/tables/pilot_ppl.tex
+45 −9
@@ -1,12 +1,48 @@ | ||
| 1 | 1 | \begin{table}[t]\centering\small |
| 2 | −\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens | |
| 3 | −(40 $\times$ 16 $\times$ 512) under forced random budgets and, for PRISME, under the locked controller.} | |
| 2 | +\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens and | |
| 3 | +on the LOCKED test split (40 $\times$ 16 $\times$ 512 tokens each) under forced random budgets. Budget 0 of the adapters | |
| 4 | +equals the backbone bitwise.} | |
| 4 | 5 | \label{tab:pilotppl} |
| 5 | −\begin{tabular}{llrr}\toprule | |
| 6 | −System & policy & loss & ppl\\\midrule | |
| 7 | −gr 103m & forced b0 & 3.8746 & 48.16\\ | |
| 8 | −gr 103m & forced b1 & 3.8731 & 48.09\\ | |
| 9 | −gr 103m & forced b2 & 3.8722 & 48.05\\ | |
| 10 | −gr 103m & forced b4 & 3.8714 & 48.01\\ | |
| 11 | −gr 103m & forced b8 & 3.8718 & 48.03\\ | |
| 6 | +\begin{tabular}{lllrr}\toprule | |
| 7 | +System & split & policy & loss & ppl\\\midrule | |
| 8 | +backbone 103m & val & forced b0 & 3.8746 & 48.16\\ | |
| 9 | +backbone 103m & val & forced b1 & 3.8746 & 48.16\\ | |
| 10 | +backbone 103m & val & forced b2 & 3.8746 & 48.16\\ | |
| 11 | +backbone 103m & val & forced b4 & 3.8746 & 48.16\\ | |
| 12 | +backbone 103m & val & forced b8 & 3.8746 & 48.16\\ | |
| 13 | +enlarged 102m & val & forced b0 & 3.8689 & 47.89\\ | |
| 14 | +enlarged 102m & val & forced b1 & 3.8689 & 47.89\\ | |
| 15 | +enlarged 102m & val & forced b2 & 3.8689 & 47.89\\ | |
| 16 | +enlarged 102m & val & forced b4 & 3.8689 & 47.89\\ | |
| 17 | +enlarged 102m & val & forced b8 & 3.8689 & 47.89\\ | |
| 18 | +gr 103m & val & forced b0 & 3.8746 & 48.16\\ | |
| 19 | +gr 103m & val & forced b1 & 3.8731 & 48.09\\ | |
| 20 | +gr 103m & val & forced b2 & 3.8722 & 48.05\\ | |
| 21 | +gr 103m & val & forced b4 & 3.8714 & 48.01\\ | |
| 22 | +gr 103m & val & forced b8 & 3.8718 & 48.03\\ | |
| 23 | +prisme 103m editor & val & forced b0 & 3.8746 & 48.16\\ | |
| 24 | +prisme 103m editor & val & forced b1 & 3.8731 & 48.09\\ | |
| 25 | +prisme 103m editor & val & forced b2 & 3.8722 & 48.05\\ | |
| 26 | +prisme 103m editor & val & forced b4 & 3.8714 & 48.01\\ | |
| 27 | +prisme 103m editor & val & forced b8 & 3.8718 & 48.03\\ | |
| 28 | +backbone 103m & test & forced b0 & 3.8226 & 45.72\\ | |
| 29 | +backbone 103m & test & forced b1 & 3.8226 & 45.72\\ | |
| 30 | +backbone 103m & test & forced b2 & 3.8226 & 45.72\\ | |
| 31 | +backbone 103m & test & forced b4 & 3.8226 & 45.72\\ | |
| 32 | +backbone 103m & test & forced b8 & 3.8226 & 45.72\\ | |
| 33 | +gr 103m & test & forced b0 & 3.8226 & 45.72\\ | |
| 34 | +gr 103m & test & forced b1 & 3.8211 & 45.66\\ | |
| 35 | +gr 103m & test & forced b2 & 3.8203 & 45.62\\ | |
| 36 | +gr 103m & test & forced b4 & 3.8195 & 45.58\\ | |
| 37 | +gr 103m & test & forced b8 & 3.8201 & 45.61\\ | |
| 38 | +prisme 103m editor & test & forced b0 & 3.8226 & 45.72\\ | |
| 39 | +prisme 103m editor & test & forced b1 & 3.8211 & 45.66\\ | |
| 40 | +prisme 103m editor & test & forced b2 & 3.8202 & 45.61\\ | |
| 41 | +prisme 103m editor & test & forced b4 & 3.8195 & 45.58\\ | |
| 42 | +prisme 103m editor & test & forced b8 & 3.8200 & 45.60\\ | |
| 43 | +enlarged 102m & test & forced b0 & 3.8207 & 45.64\\ | |
| 44 | +enlarged 102m & test & forced b1 & 3.8207 & 45.64\\ | |
| 45 | +enlarged 102m & test & forced b2 & 3.8207 & 45.64\\ | |
| 46 | +enlarged 102m & test & forced b4 & 3.8207 & 45.64\\ | |
| 47 | +enlarged 102m & test & forced b8 & 3.8207 & 45.64\\ | |
| 12 | 48 | \bottomrule\end{tabular}\end{table} |
added
reproducibility/audit_report.md
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +# Results-to-claim audit report (WP31, 2026-10-30) | |
| 2 | + | |
| 3 | +* `scripts/verify_manuscript_claims.py`: **AUDIT PASS** — 25 registered claims; every artefact path exists; every run id has a | |
| 4 | + manifest (states `completed`, or `stopped-run-used` for the two deliberately stopped runs cited as failures); every numeric | |
| 5 | + macro used in the manuscript is generated by `scripts/make_tables.py`; no promotional phrase; no placeholder section. | |
| 6 | +* Tests: `python -m pytest tests -q` → 23 passed (15 mandated invariants, controller target-freedom with and without the | |
| 7 | + residual/entropy features, dense-write ablation, 5 cache-equivalence tests). | |
| 8 | +* Manuscript: `paper/main.pdf`, 48 pages, latexmk with 0 undefined references/citations (one harmless font-shape warning). | |
| 9 | +* Pre-registration: `protocol/preregistration.md` frozen v1.0 at 2026-10-30T04:04-0400; `protocol/locked_test_manifest.json` | |
| 10 | + unsealed at the same time; test splits were read only afterwards (synthetic test regimes: 5 nodes; FineWeb test: 4 systems). | |
| 11 | +* Deviations: `protocol/deviations.md` (bootstrap init/scale, controller features, label stream, dense 103M infeasible, | |
| 12 | + backbone budget 98.3M instead of 196.6M tokens). | |
| 13 | +* Not executed: Phase 11 (MLX port), Phase 12 training (459M/1.18B only instantiated on meta), GSM8K/EvalPlus, M ∈ {4,8,32}, | |
| 14 | + editor depth ∈ {1,4}, rematched pairs, quantisation, distillation, joint fine-tuning (one degraded attempt), dense editor at 103M | |
| 15 | + (collapsed twice), MLX/Metal profiling. | |
| 16 | +* Checksums of the key artefacts: `reproducibility/checksums.sha256`. | |
added
reproducibility/checksums.sha256
+213 −0
@@ -0,0 +1,213 @@ | ||
| 1 | +7d2419ffa1f2892bc7dafc95685f767cc79de6554e219f7f1a76570c8d3790b1 paper/main.pdf | |
| 2 | +a1a3468ffe50fba1beb150871b8d5ad575163ba1cd96373838d7ff989380853d results/claim_registry.csv | |
| 3 | +70cb86ea329cb0052e0956db0374d8e55b43305056edd778e7512cf5ee6fef02 results/analyzed/labels_labels_p4_seed01_20261030000543.json | |
| 4 | +e5765cbde70d9faf34b4aec875a44184b8ae368531d5ccca29e5ecfddb4a020e results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json | |
| 5 | +77f6c3406df0ed977b90c4b5dbeccfecfff63a5054a87dbe1c6ddb3e23acc53d results/analyzed/routing_seeds_v2.json | |
| 6 | +c121c2bfd141a3ffbc161c8213bb9189507a965434c22b65005432f9cb723a58 results/analyzed/seed_matrix_v2.json | |
| 7 | +de0677e8600a9d82c293ad49469fdb5346a3fe55f05d8f6d24315dadcd5f147c results/analyzed/statistics_v2.json | |
| 8 | +b7923aef253503c8d8ea1740c9791b691a2b620519eba798c1d34fc12dd1e64d results/analyzed/test_statistics.json | |
| 9 | +7945a43fde621b133add69976be3d1f50bfa6a3e976bacae6a8075f6663d4312 results/raw/eval_backbone_103m_test.json | |
| 10 | +1db7d53edf42e2065ee66749b5b1231a0e793a51d3b7b3bad0475e907fd65fad results/raw/eval_backbone_103m_val.json | |
| 11 | +56efdc8762fefc5ee2a6dd0e3ee3a88edfe51450576838862ca4528f85df8f99 results/raw/eval_enlarged_102m_test.json | |
| 12 | +4ed5f6d040e0bb84741a54de6de385a76a4db78d26a883c31ccd4442223ae877 results/raw/eval_enlarged_102m_val.json | |
| 13 | +24a17a932585a6a46116375b71c4e72847892b58070d3d979e1c17e41cee13db results/raw/eval_gr_103m_test.json | |
| 14 | +94d76e5cdf6da940f16ec113eca2daf25fe9069b35c6537f83eea513ea6caad6 results/raw/eval_gr_103m_val.json | |
| 15 | +240a85d62d9ebe4a274314e0d50a2e8c2ad4b4b581bfdb5f95d0579d132c5604 results/raw/eval_p4v2_dense_s1_test_compose.json | |
| 16 | +b4601f1bfb75d0618c8a48e2a7b9f15d8b9f0906192c1235c3a355d0c95d25e9 results/raw/eval_p4v2_dense_s1_test_depth.json | |
| 17 | +0de49b5dc700f1db63066d69d8bc5fcfb6dc12a1efaefb72be033840f1be13c7 results/raw/eval_p4v2_dense_s1_test_iid.json | |
| 18 | +a4bf328b91b69fd77878f7d278d3d81e8d708ea809e84eb82e55dd08e3e580ae results/raw/eval_p4v2_dense_s1_test_template.json | |
| 19 | +59348dced041971ace2836f13d744ae07e1e1be9ecf21ab35b254db499d0ea7e results/raw/eval_p4v2_dense_s2_test_compose.json | |
| 20 | +f9b5bd5d98040d5556d5b624987c1d4f10f89f3af74480ce909167ab14b9465f results/raw/eval_p4v2_dense_s2_test_depth.json | |
| 21 | +e0c772078e0277a185890e6a0f37bee555be1e2856fd181f62ce5b067a9a36e9 results/raw/eval_p4v2_dense_s2_test_iid.json | |
| 22 | +b8a8f4bd4feccf6468755ee8eee6cc73baf50726ff3e4a7384a849d845ce9460 results/raw/eval_p4v2_dense_s2_test_template.json | |
| 23 | +ff25f532fa987ba08027d677b77e730308eb98debe5b2861e9d63ad896daf047 results/raw/eval_p4v2_dense_s3_test_compose.json | |
| 24 | +4c22fe9f58e85b80c7a04d66a367dc9d8468df7a37feab0a55e96a488d86108c results/raw/eval_p4v2_dense_s3_test_depth.json | |
| 25 | +ccae7be5c414503e6916af1b496ac5d76e8ccf10ed02938bc302b3063dd55bcd results/raw/eval_p4v2_dense_s3_test_iid.json | |
| 26 | +19a05738052fdb253b6b0cca58386c80a98b4a460d399d4eb86587b499760d7e results/raw/eval_p4v2_dense_s3_test_template.json | |
| 27 | +553412dbdae6f569ff2a4aaba12cebbaf46b545e63f17b2c92af727db3f2cacc results/raw/eval_p4v2_dense_s4_test_compose.json | |
| 28 | +9c9a7cb00ba1e9e2edf419961966508cf00fb8bd9b8c5e19bc4297f875d4f3db results/raw/eval_p4v2_dense_s4_test_depth.json | |
| 29 | +6d16a9462d9e90ed1210f095b30445a1abc8e95fabf4d5262d26b8bcc9f75821 results/raw/eval_p4v2_dense_s4_test_iid.json | |
| 30 | +cb601cd8f310f29e19ce6b880d19dd68178d83970d8eb2659c79dd24bcef7f55 results/raw/eval_p4v2_dense_s4_test_template.json | |
| 31 | +a6cabd657169d423a1d49befb485d161f79ca7d7c31c457e6467870a315fc60b results/raw/eval_p4v2_dense_s5_test_compose.json | |
| 32 | +40cb9aa807935ef72c86d1cd2a41d3a3a76fa933c9f36c50a5216aa31c50245f results/raw/eval_p4v2_dense_s5_test_depth.json | |
| 33 | +7d6bc6039c4c2be34c18d9016350765e35bccc4f2a8e7f7b13dbddabe22af1ce results/raw/eval_p4v2_dense_s5_test_iid.json | |
| 34 | +6a18b127adeb7267d2a41c3113215c3d9bd56aa48189461bba2ba383fc131437 results/raw/eval_p4v2_dense_s5_test_template.json | |
| 35 | +a89e20b97e580615df0d3818d5280af6a722d8ae442d898bd5c945ea0550936d results/raw/eval_p4v2_gr_s1_test_compose.json | |
| 36 | +bd42a30c976c62752907bb84e71dd43a3634a27c937d5e82093ce7dd6fe87905 results/raw/eval_p4v2_gr_s1_test_depth.json | |
| 37 | +5636d906e8c4aadf73b89a2e10fd4e0a6dd1c62099949eda5bf9ee92fa630e47 results/raw/eval_p4v2_gr_s1_test_iid.json | |
| 38 | +88c25f1afa4e344e2a09f95fa4356bef066288459a5237a4efa2fc6cbcb4deaf results/raw/eval_p4v2_gr_s1_test_template.json | |
| 39 | +e83025dcb455a1e417098583271de0a5f460d86086a0b8beb034fef12bf304cd results/raw/eval_p4v2_gr_s2_test_compose.json | |
| 40 | +c93d645871dd63aebaef1e9c5b43725bbf9c7b0cd9805ebe5c7c67683742ef89 results/raw/eval_p4v2_gr_s2_test_depth.json | |
| 41 | +4adae524b80f03a7b3db761f6449e28a5d191b9d35d864eef208a2ad014cd9b5 results/raw/eval_p4v2_gr_s2_test_iid.json | |
| 42 | +ff3b6070ecca2de27276c2b6c81a045bc05cc32ed696d30c581b7b5b4c490a86 results/raw/eval_p4v2_gr_s2_test_template.json | |
| 43 | +cc3310f947c2ab0556978bd721d68d11b8cb252fb52a4e5c44710956b4339415 results/raw/eval_p4v2_gr_s3_test_compose.json | |
| 44 | +c48b3b0ba3533819c092ebcde1a578f69d44cec34a38247e2a2eb1c59489e01a results/raw/eval_p4v2_gr_s3_test_depth.json | |
| 45 | +f9df369e3f358110e301413d6a6071aefbed562c08f06957d71514af738f9827 results/raw/eval_p4v2_gr_s3_test_iid.json | |
| 46 | +b2281f2b745aa5be1972dbb0d111e60f3b39ec6b2e0a7e228a852a5a158b629b results/raw/eval_p4v2_gr_s3_test_template.json | |
| 47 | +daff4678d7c288ffc1c891dd8fc73ea63e9c00ef165fee1c786f7ab6910d27ef results/raw/eval_p4v2_gr_s4_test_compose.json | |
| 48 | +edcdfebbc348086b2578c00c4ffd757e0d135bf24b67c202598b09961e39080a results/raw/eval_p4v2_gr_s4_test_depth.json | |
| 49 | +5275701b43e763cea01617bbc6ba406622d7dada9975e17d152543e4d86e9b37 results/raw/eval_p4v2_gr_s4_test_iid.json | |
| 50 | +781772b860a0d8171b135122d05bcbef9402bfa75e302d64f75a5eff54d297bd results/raw/eval_p4v2_gr_s4_test_template.json | |
| 51 | +3f44da7d05b20404c12639942152172942faf325b81b89391e1b7d300ac1ec6d results/raw/eval_p4v2_gr_s5_test_compose.json | |
| 52 | +01ea31f8424a5a00b90164cb641e6e8af66dbb0e8176bdb4da8aa572a7f1268f results/raw/eval_p4v2_gr_s5_test_depth.json | |
| 53 | +8dd943be0d13f10b3eab74249e96bf474e8cb761f111323df42b3619e46ae875 results/raw/eval_p4v2_gr_s5_test_iid.json | |
| 54 | +6dbccfdd4ea90d91b2bc37adba864c4d44866b2de489956e379e7f6f5e106e04 results/raw/eval_p4v2_gr_s5_test_template.json | |
| 55 | +21fd0d5905ca637f9b79a7be000909865197774dca6f47252d007caa620e834a results/raw/eval_p4v2_prisme_s1_test_compose.json | |
| 56 | +f313c6d0183813f2a1273d7a79f34d91436c1951933482c368392df81d779f31 results/raw/eval_p4v2_prisme_s1_test_depth.json | |
| 57 | +2912cf4bb956a7d1483e8246909e0f6fcf44b4fd66cdf41c87dd70f89403407f results/raw/eval_p4v2_prisme_s1_test_iid.json | |
| 58 | +e7ce172fdbcd7daee5b41c1cce7e7935201f55a862b748b47f767c5f0fbde186 results/raw/eval_p4v2_prisme_s1_test_template.json | |
| 59 | +9cf33ba1b80f7bd9e0740f2de0bbb51a9044b573094a6178c629447e386aba41 results/raw/eval_p4v2_prisme_s2_test_compose.json | |
| 60 | +7b09000cca0ce54acdd7b989ee52ebbe8c977af5967949d4ff2d6f052b152640 results/raw/eval_p4v2_prisme_s2_test_depth.json | |
| 61 | +7b029010b158cb1654792bb03c31c3348ebbc3d1e1fb2e02dc0ecc2b4ca9d4d2 results/raw/eval_p4v2_prisme_s2_test_iid.json | |
| 62 | +432e089082d068e591934353180b450c6f1b6899e8146af8f058fdbc0e041705 results/raw/eval_p4v2_prisme_s2_test_template.json | |
| 63 | +32d899d80fe6e96c75145a0cad12cbb0b1d46743e0bc5097ebb2b3a1dceba931 results/raw/eval_p4v2_prisme_s3_test_compose.json | |
| 64 | +0cb5219b36308c768f96fd23008284b2470d773e87c9cf54200cdc5b95ee53c3 results/raw/eval_p4v2_prisme_s3_test_depth.json | |
| 65 | +9f614b3f111cd9f97e5b5fe376f508f96311a1db33c67a3862dcd0eb8cf8e2e4 results/raw/eval_p4v2_prisme_s3_test_iid.json | |
| 66 | +aa3eeac5cc2c680170fa82e19e5e6a748f92771285d110fc80b001f04b7f7a5d results/raw/eval_p4v2_prisme_s3_test_template.json | |
| 67 | +508ba5332b8d84219703c07971c6212f9e8d8c5fee0e7e15bec90bb19bac836c results/raw/eval_p4v2_prisme_s4_test_compose.json | |
| 68 | +b667a0ddc3794ee2412a1ff183631a5ba9f67c417ea21c8a46a849ce6c04d888 results/raw/eval_p4v2_prisme_s4_test_depth.json | |
| 69 | +833f5eed3a48185c17ea58dfa76e297d93f6b875bb08651bc324120a30f8b3bf results/raw/eval_p4v2_prisme_s4_test_iid.json | |
| 70 | +b2e41313e851338880f5ae5ac181fb665775f5ef90ffe8a458e79952b60079da results/raw/eval_p4v2_prisme_s4_test_template.json | |
| 71 | +5a78fa1db60b6f2d699e404540c43cdfffa8264290d76804d7cc88741d7d663c results/raw/eval_p4v2_prisme_s5_test_compose.json | |
| 72 | +6fae55bfa3d1531da0047057efab493fad5d59387631a6af5ce50d59c2f17898 results/raw/eval_p4v2_prisme_s5_test_depth.json | |
| 73 | +26b429fd5eb464ad19d7502d4de4bfc1bf8ff3c138c63e035da2d36bb8ac0dc2 results/raw/eval_p4v2_prisme_s5_test_iid.json | |
| 74 | +169577d4954ca6786db7e95a0b10d3f5e67821154908097895fd67e54ab62020 results/raw/eval_p4v2_prisme_s5_test_template.json | |
| 75 | +83200604d660282d28acbf8eca212562915edd1bff3ed19d18010195ce98fb54 results/raw/eval_prisme_103m_editor_test.json | |
| 76 | +bb1080bbcfb2ff25471be1dc13409dab4531b7a07f4f133e1427f41b2c4d047c results/raw/eval_prisme_103m_editor_val.json | |
| 77 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s1_test_compose_budget0.json | |
| 78 | +2e999f28ca2c658f23598b80bc9952815f9f23d385c960bd1d5d95e51cca1d17 results/raw/items_p4v2_dense_s1_test_compose_forced_b2.json | |
| 79 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s1_test_depth_budget0.json | |
| 80 | +c509d6ebf76ed9fd8f3ba93c7c0bed1d9dc062f599f1572f9b814b25b76e1ece results/raw/items_p4v2_dense_s1_test_depth_forced_b2.json | |
| 81 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s1_test_iid_budget0.json | |
| 82 | +c767436f7a6c79a2c93cdf99096249263cd1432d7a1a4562945ddf407eaca69d results/raw/items_p4v2_dense_s1_test_iid_forced_b2.json | |
| 83 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s1_test_template_budget0.json | |
| 84 | +8448cc79e9d1e9ab28274143142207f345427d12dccea5a23b5a4417eece564b results/raw/items_p4v2_dense_s1_test_template_forced_b2.json | |
| 85 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s2_test_compose_budget0.json | |
| 86 | +16a7d8b011b8a10caef1ba95393d488e19a2d850eeaf80a8aa95ba14d5b04c1d results/raw/items_p4v2_dense_s2_test_compose_forced_b2.json | |
| 87 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s2_test_depth_budget0.json | |
| 88 | +a02eaf64716ddddcfb10708cdd215597f525323881fff3e1831731be22814924 results/raw/items_p4v2_dense_s2_test_depth_forced_b2.json | |
| 89 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s2_test_iid_budget0.json | |
| 90 | +dd07b669a62500527d704d123584184be2aa19e032e1b933e3a83d3d7c466fc1 results/raw/items_p4v2_dense_s2_test_iid_forced_b2.json | |
| 91 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s2_test_template_budget0.json | |
| 92 | +f51973ecfd545bc9780240e219a2ad5bd98cf83ca1085add5c65eb800a8653fe results/raw/items_p4v2_dense_s2_test_template_forced_b2.json | |
| 93 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s3_test_compose_budget0.json | |
| 94 | +307a8c7781f3c75a5a65edb17140ef94be91fbf2d5776607a4f75a32a2f3f575 results/raw/items_p4v2_dense_s3_test_compose_forced_b2.json | |
| 95 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s3_test_depth_budget0.json | |
| 96 | +7ef70865bf95cd33268c5e2124ab6f7a649e128f09d51053891d16ca63e950b8 results/raw/items_p4v2_dense_s3_test_depth_forced_b2.json | |
| 97 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s3_test_iid_budget0.json | |
| 98 | +ab88e74b8581317c7e4fbd0c6e3f4a55c51d4209f344fcbf90d004787fbef4c0 results/raw/items_p4v2_dense_s3_test_iid_forced_b2.json | |
| 99 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s3_test_template_budget0.json | |
| 100 | +60a01032ada2b17ae0b1cc80bee58da6167eba49f678e206f6dca3e832eca4fe results/raw/items_p4v2_dense_s3_test_template_forced_b2.json | |
| 101 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s4_test_compose_budget0.json | |
| 102 | +f93c20b054f2760e287a8dab70fd6fed984f0d7b4e137a488d7a515e4748ca0b results/raw/items_p4v2_dense_s4_test_compose_forced_b2.json | |
| 103 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s4_test_depth_budget0.json | |
| 104 | +a1c523c689ee99410142d4e89cb3997a10342a00e712dab11bd37d560d1a5928 results/raw/items_p4v2_dense_s4_test_depth_forced_b2.json | |
| 105 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s4_test_iid_budget0.json | |
| 106 | +c9cfad851190b133393b90636bb74625f1434226cc1c13c7744fad5714bb1b46 results/raw/items_p4v2_dense_s4_test_iid_forced_b2.json | |
| 107 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s4_test_template_budget0.json | |
| 108 | +ddd0fe249aa523c6c8b5e65666829710af812c4503001b425a8c33af637a6761 results/raw/items_p4v2_dense_s4_test_template_forced_b2.json | |
| 109 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s5_test_compose_budget0.json | |
| 110 | +1ed8e804351e8c2daf199cba00355d83db19d363f4830ab024f7fd0a37f6fe60 results/raw/items_p4v2_dense_s5_test_compose_forced_b2.json | |
| 111 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s5_test_depth_budget0.json | |
| 112 | +c99c501680904e3cf2d2f6845a6bc4f4a15b10a581ecc7592da8bba5a9d60924 results/raw/items_p4v2_dense_s5_test_depth_forced_b2.json | |
| 113 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s5_test_iid_budget0.json | |
| 114 | +91568b5ec9200783cf78320454f1cf988d14d418a93d5198e5bd1e62a1cd1a29 results/raw/items_p4v2_dense_s5_test_iid_forced_b2.json | |
| 115 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s5_test_template_budget0.json | |
| 116 | +505f27c6fffe6875e91f62221bb14693a3bf0467fd091bfd41b2b1e032872dd9 results/raw/items_p4v2_dense_s5_test_template_forced_b2.json | |
| 117 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s1_test_compose_budget0.json | |
| 118 | +e464fdc8a2b60c7609843dce2a8c22747bd897c1adb1886616eb9d4210f35c17 results/raw/items_p4v2_gr_s1_test_compose_forced_b2.json | |
| 119 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s1_test_depth_budget0.json | |
| 120 | +8aad9b15f27d52155e4928ebe77e36aea8bf19390f044531fcf0733dd5f5adfc results/raw/items_p4v2_gr_s1_test_depth_forced_b2.json | |
| 121 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s1_test_iid_budget0.json | |
| 122 | +5fd4cde26848fb97275e93925c6e24c8777da268be39dad2e8962dc604c4aaa8 results/raw/items_p4v2_gr_s1_test_iid_forced_b2.json | |
| 123 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s1_test_template_budget0.json | |
| 124 | +3e77bad1a73c4a811c92611002499a7e948f19c2bdec93c442ad4578f2e8eaec results/raw/items_p4v2_gr_s1_test_template_forced_b2.json | |
| 125 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s2_test_compose_budget0.json | |
| 126 | +7fee922e3864adcf6eb2d7d66353393ec4df5948769c1c8adb06d920ded6c0cb results/raw/items_p4v2_gr_s2_test_compose_forced_b2.json | |
| 127 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s2_test_depth_budget0.json | |
| 128 | +93db1e816557fdcfbdf374cfc12f6eb08cc8b88d1bfce4c49a594756f5247b07 results/raw/items_p4v2_gr_s2_test_depth_forced_b2.json | |
| 129 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s2_test_iid_budget0.json | |
| 130 | +187f6a7fc6131419ac322a7005e11b58bd829938042d5ab0eae81285ed62c189 results/raw/items_p4v2_gr_s2_test_iid_forced_b2.json | |
| 131 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s2_test_template_budget0.json | |
| 132 | +c9f640c16d349590b8dde575cc8218e93e653e0e9f5f060b87e144dee4a2d8af results/raw/items_p4v2_gr_s2_test_template_forced_b2.json | |
| 133 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s3_test_compose_budget0.json | |
| 134 | +b05f703226a7c4db4484e46c027bbbe768f8e41d8a4cdcdff07a64aaf0d67e7f results/raw/items_p4v2_gr_s3_test_compose_forced_b2.json | |
| 135 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s3_test_depth_budget0.json | |
| 136 | +ecbb22971316f411cccc02692f4eefe40a297771824c8b94a74645399f1a3379 results/raw/items_p4v2_gr_s3_test_depth_forced_b2.json | |
| 137 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s3_test_iid_budget0.json | |
| 138 | +d292ff34868f76e572e1ad4daf6545e222f0240f5fcb8fdd3a8921e3840cdb0d results/raw/items_p4v2_gr_s3_test_iid_forced_b2.json | |
| 139 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s3_test_template_budget0.json | |
| 140 | +8b43517446e9f59c9153606c135bf048feec04406b30bc9d26f4a4a43c8d9484 results/raw/items_p4v2_gr_s3_test_template_forced_b2.json | |
| 141 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s4_test_compose_budget0.json | |
| 142 | +ab3e9552cd2a6a230cdb8f33a4973893bbb107ba4f5cdb1c31cc74c86d8c2a26 results/raw/items_p4v2_gr_s4_test_compose_forced_b2.json | |
| 143 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s4_test_depth_budget0.json | |
| 144 | +487cd30e77ff443531b33cd047f9eb401a9e4177cbad513bb6e75e8e09e68abf results/raw/items_p4v2_gr_s4_test_depth_forced_b2.json | |
| 145 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s4_test_iid_budget0.json | |
| 146 | +1d353dfdbc37d45ecf8a90d8846c5325830c91fc4ac3f69a06ce8fe265a28595 results/raw/items_p4v2_gr_s4_test_iid_forced_b2.json | |
| 147 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s4_test_template_budget0.json | |
| 148 | +476b357454472720aeb0730eb10903c7251683fe8d3a7991b0df2eeaffb70aad results/raw/items_p4v2_gr_s4_test_template_forced_b2.json | |
| 149 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s5_test_compose_budget0.json | |
| 150 | +b79c36c2b71c270f90e54776c0e470fce0d15f8376bf751b7b7781c6fcf63332 results/raw/items_p4v2_gr_s5_test_compose_forced_b2.json | |
| 151 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s5_test_depth_budget0.json | |
| 152 | +f2f016d4cd3346e60eacc131cc100d462a21638d3490c5a240d51a9d46315dc4 results/raw/items_p4v2_gr_s5_test_depth_forced_b2.json | |
| 153 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s5_test_iid_budget0.json | |
| 154 | +2d5df71b91fe76a80463efde1a216de1a2a6339aacd9e0c584cfbfa0938a629d results/raw/items_p4v2_gr_s5_test_iid_forced_b2.json | |
| 155 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s5_test_template_budget0.json | |
| 156 | +32eb5e11f354321177fbd7dbd0841473cf2270e4cf9df5a0f9464dcbfb59ad9e results/raw/items_p4v2_gr_s5_test_template_forced_b2.json | |
| 157 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s1_test_compose_budget0.json | |
| 158 | +80586b49e01e888dfb56a0a5db4e54d58cb4856668ca21278c86ebe4e11506aa results/raw/items_p4v2_prisme_s1_test_compose_forced_b2.json | |
| 159 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s1_test_depth_budget0.json | |
| 160 | +4ab1a75242a89a0af0e6d024749d2cf088c3b8e58c9f2f663940be9564c52b2b results/raw/items_p4v2_prisme_s1_test_depth_forced_b2.json | |
| 161 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s1_test_iid_budget0.json | |
| 162 | +8391d02c1a270e306ee86ebfc6fed306ab2817bbef784e6b807b993a08b9e5bf results/raw/items_p4v2_prisme_s1_test_iid_forced_b2.json | |
| 163 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s1_test_template_budget0.json | |
| 164 | +d969667cd1fd32f07dd4033ede38e756200b6af096f9d2aa5764b5848013e9ac results/raw/items_p4v2_prisme_s1_test_template_forced_b2.json | |
| 165 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s2_test_compose_budget0.json | |
| 166 | +81f46e521c20c05cc1981017682681adf39aee9e685f6e3aaf9888d3768bd639 results/raw/items_p4v2_prisme_s2_test_compose_forced_b2.json | |
| 167 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s2_test_depth_budget0.json | |
| 168 | +b82f82f9dda0335045995f92f005abe6cd48c27e54b5de65ed453c9f0c16dd51 results/raw/items_p4v2_prisme_s2_test_depth_forced_b2.json | |
| 169 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s2_test_iid_budget0.json | |
| 170 | +8761df4b6ae82ce5848e5b8736f6d4d6cb3453ec1daac23317416e791fa59236 results/raw/items_p4v2_prisme_s2_test_iid_forced_b2.json | |
| 171 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s2_test_template_budget0.json | |
| 172 | +b84990037836a760f101b381216c8ba32455ee65f7ae14871b0ad72373399b15 results/raw/items_p4v2_prisme_s2_test_template_forced_b2.json | |
| 173 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s3_test_compose_budget0.json | |
| 174 | +2233020039ad7f833d596c4d29b0ea8544302eff1a418eb051d9a7c4692c8c74 results/raw/items_p4v2_prisme_s3_test_compose_forced_b2.json | |
| 175 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s3_test_depth_budget0.json | |
| 176 | +c38c9207609aaf1d8012712bfbf69898ceb8c2e3947517131c00f574e2df0a19 results/raw/items_p4v2_prisme_s3_test_depth_forced_b2.json | |
| 177 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s3_test_iid_budget0.json | |
| 178 | +474f66cc8bae2acaffb15163852b1a32565706637ddc349a8434870778f168cd results/raw/items_p4v2_prisme_s3_test_iid_forced_b2.json | |
| 179 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s3_test_template_budget0.json | |
| 180 | +07c6750a3bdd19e78fce4a252440a150ce45b2b47f98feefe058d16ede0fd9fb results/raw/items_p4v2_prisme_s3_test_template_forced_b2.json | |
| 181 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s4_test_compose_budget0.json | |
| 182 | +bd79b85556e9d87ea59daa7429b932d03f8c5557e24318e99a953a934a0ebfe3 results/raw/items_p4v2_prisme_s4_test_compose_forced_b2.json | |
| 183 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s4_test_depth_budget0.json | |
| 184 | +f03d3e5be849eda84999cea23d73ce170cf99fbb5286816763bcdd229317577a results/raw/items_p4v2_prisme_s4_test_depth_forced_b2.json | |
| 185 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s4_test_iid_budget0.json | |
| 186 | +89999afb68660788f080c070e8a53c5377700403ba40c6deafa70162f4834046 results/raw/items_p4v2_prisme_s4_test_iid_forced_b2.json | |
| 187 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s4_test_template_budget0.json | |
| 188 | +128df74db3be1d568b86ac2e158af651dd8ff5390cef15f81de6f81afbd600a2 results/raw/items_p4v2_prisme_s4_test_template_forced_b2.json | |
| 189 | +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s5_test_compose_budget0.json | |
| 190 | +e604df74deb203b3d80f0fd271237506da912b87d04dc969e6f1fee513556aca results/raw/items_p4v2_prisme_s5_test_compose_forced_b2.json | |
| 191 | +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s5_test_depth_budget0.json | |
| 192 | +99e37b0a00330ebdb8790092fa2b194fb065e27fc03c4d58dd0886048e8545d0 results/raw/items_p4v2_prisme_s5_test_depth_forced_b2.json | |
| 193 | +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s5_test_iid_budget0.json | |
| 194 | +d5eb96b933c0b67966734d733579e0885ce3710f782d101eee95d629fda4bdc9 results/raw/items_p4v2_prisme_s5_test_iid_forced_b2.json | |
| 195 | +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s5_test_template_budget0.json | |
| 196 | +4a273387f2204f907717961a3e5d1f58133ef2b3b03f97f8a947931d4f8f4eae results/raw/items_p4v2_prisme_s5_test_template_forced_b2.json | |
| 197 | +5ac057e7323760fe0774e0016dad4e4480a3b0fb2bcb2a71fab9b595596f00fa results/raw/latency_103m_mps.json | |
| 198 | +17fd8d751c915fa60c506718da6e862fb4e973b1b4996599d9ade54512982634 results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json | |
| 199 | +c7f8878b9b007546d7312cb29b1904992235c243d2d29446c1bec64bcfc9b1e4 results/raw/routing_103m_val.json | |
| 200 | +a2d6f4fc3f5b1591e6f02215d9d72a537bedc6afec076210b04eab44550a2822 results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json | |
| 201 | +f7748c20a6446a22b1e307c80cea49a30cc89dc2c4c951d2486659cdbcfc3cef results/raw/routing_mini.json | |
| 202 | +f20c434c42090ba17603819bdfc127da73ca46163fc6823b6a78013dda08e555 results/raw/routing_v1ctrl.json | |
| 203 | +95cf28b285b26f6e1a54564b2a2b8ee691e89886df9875c850a1b5606d8831b1 results/raw/routing_v1rank.json | |
| 204 | +bb85e8001ba798a203fc01e55f212777430ae816acf893eb32ebdfc88a779e4b results/raw/routing_v2s1.json | |
| 205 | +224965164912b215d4e0be99c6a470b8b6d42a7fc390198424cfeb1a4e8be893 results/raw/routing_v2s2.json | |
| 206 | +152654d5ded3c0a643dcb2c296580ee8c330877b299d484057eec06f2a32b6a1 results/raw/routing_v2s3.json | |
| 207 | +909442a2374d2a2929719a7dc0142933ed1a4bf6f1feba2c82c547751d711b7e results/raw/routing_v2s4.json | |
| 208 | +38c615486b0c298ba1b179327f4e03827d26f7db3662f00657f2ddbfc30a10c3 results/raw/routing_v2s5.json | |
| 209 | +06620e1cf60b7250aac71f4fb5ee74e3705f4b9aab102e22b5c5bdbed76c4ae2 data/splits/synthetic_v1/manifest.json | |
| 210 | +d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54 data/processed/fineweb_edu_300m/manifest.json | |
| 211 | +54ef965fa85e96ebfd88bcdc745a8d19efa56c461d73444a4592dbedc678ac4e protocol/preregistration.md | |
| 212 | +acd9f9c178b9ab8979d527fed75733100a11eec63fb201f892c4ea016daeb1b9 protocol/locked_test_manifest.json | |
| 213 | +e9763b53e11f13ed41ccdba86fe95da896bc7b249762be6e4136ccc8a318b71b requirements-lock.txt | |
modified
reproducibility/run_index.md
+21 −4
@@ -2,18 +2,26 @@ | ||
| 2 | 2 | |
| 3 | 3 | | run_id | state | host | device | wall s | train tokens | peak mem (GB) | commit | notes | |
| 4 | 4 | |---|---|---|---|---|---|---|---|---| |
| 5 | +| controller_103m_seed01_20261030034451 | completed | laptop | mps | 315.08 | | | unknown | | | |
| 5 | 6 | | controller_mini_seed01_20261030005154 | completed | laptop | cpu | 3.14 | | | ede67055 | | |
| 6 | 7 | | controller_p4_hent_seed01_20261030001419 | running | laptop | mps | 0.02 | | | unknown | | |
| 7 | 8 | | controller_p4_hent_seed01_20261030002015 | completed | laptop | mps | 584.36 | | | unknown | | |
| 8 | 9 | | controller_v1ctrl_seed01_20261030004131 | completed | laptop | mps | 164.25 | | | 33298d96 | | |
| 10 | +| controller_v1rank_seed01_20261030005431 | completed | laptop | mps | 171.86 | | | d6746e03 | | | |
| 11 | +| controller_v2s1_seed01_20261030005758 | completed | laptop | mps | 141.05 | | | unknown | | | |
| 9 | 12 | | controller_v2s2_seed01_20261030004717 | completed | laptop | mps | 193.23 | | | unknown | | |
| 10 | 13 | | controller_v2s3_seed01_20261030004720 | completed | laptop | mps | 197.97 | | | unknown | | |
| 11 | 14 | | controller_v2s4_seed01_20261030004719 | completed | laptop | mps | 217.1 | | | unknown | | |
| 12 | 15 | | controller_v2s5_seed01_20261030004719 | completed | laptop | mps | 214.63 | | | unknown | | |
| 16 | +| editor_pilot_103m_lr3e4_seed01_20261030021200 | completed | laptop | mps | 5397.54 | 24576000 | 25.9 | unknown | prisme | | |
| 17 | +| editor_pilot_103m_seed01_20261030015139 | running | laptop | mps | 0.24 | | | unknown | prisme | | |
| 18 | +| gr_pilot_103m_seed01_20261030015117 | completed | laptop | mps | 1230.52 | 24576000 | 9.7 | unknown | global_recurrence | | |
| 19 | +| labels_103m_seed01_20261030034314 | completed | laptop | mps | 94.09 | | | unknown | | | |
| 13 | 20 | | labels_mini_seed01_20261030005150 | completed | laptop | cpu | 3.44 | | | ede67055 | | |
| 14 | 21 | | labels_p4_seed01_20261030000543 | completed | laptop | mps | 33.33 | | | unknown | | |
| 15 | 22 | | labels_v1ctrl_seed01_20261030003957 | running | laptop | mps | 0.02 | | | 33298d96 | | |
| 16 | 23 | | labels_v1ctrl_seed01_20261030004018 | completed | laptop | mps | 72.1 | | | 33298d96 | | |
| 24 | +| labels_v2s1_seed01_20261030005712 | completed | laptop | mps | 44.65 | | | unknown | | | |
| 17 | 25 | | labels_v2s2_seed01_20261030004543 | completed | laptop | mps | 92.45 | | | unknown | | |
| 18 | 26 | | labels_v2s3_seed01_20261030004544 | completed | laptop | mps | 95.57 | | | unknown | | |
| 19 | 27 | | labels_v2s4_seed01_20261030004544 | completed | laptop | mps | 94.03 | | | unknown | | |
@@ -22,18 +30,27 @@ | ||
| 22 | 30 | | mini_editor_seed01_20261030005118 | completed | laptop | cpu | 30.59 | 9847 | 2.0 | ede67055 | prisme | |
| 23 | 31 | | p4_editor_v1_Ustd02_seed01_20261029235050 | completed | laptop | mps | 650.11 | 331026 | 8.8 | unknown | prisme | |
| 24 | 32 | | p4_editor_v1_scaled_seed01_20261029234334 | running | laptop | mps | 0.16 | | | HEAD | prisme | |
| 25 | −| p4_editor_v1_seed01_20261029233624 | running | laptop | mps | 0.02 | | | HEAD | prisme | | |
| 33 | +| p4_editor_v1_seed01_20261029233624 | stopped_by_operator_scale_mismatch_diagnosed | laptop | mps | 0.02 | | | HEAD | prisme | | |
| 26 | 34 | | p4_overfit64_b8_seed01_20261029235111 | completed | laptop | mps | 519.6 | 95773 | 6.7 | a9d94e8a | prisme | |
| 27 | 35 | | p4_tiny_backbone_seed01_20261029232626 | completed | laptop | mps | 457.67 | 44367329 | 3.4 | HEAD | backbone_only | |
| 28 | 36 | | p4_tiny_backbone_v2_seed01_20261029233624 | stopped_overfitting_step9300_checkpoint_step5000_used | laptop | mps | 0.02 | | | HEAD | backbone_only | |
| 29 | −| p4v2_dense_seed05_20261030001839 | running | M4BP36 | mps | 0.33 | | | unknown | dense | | |
| 37 | +| p4v2_dense_seed01_20261030013421 | completed | laptop | mps | 2465.35 | 496298 | 30.3 | unknown | dense | | |
| 38 | +| p4v2_dense_seed02_20261030001749 | running | laptop | mps | 0.51 | | | unknown | dense | | |
| 39 | +| p4v2_dense_seed02_20261030004544 | completed | laptop | mps | 2601.32 | 498543 | 30.3 | unknown | dense | | |
| 40 | +| p4v2_dense_seed03_20261030001749 | running | laptop | mps | 0.54 | | | unknown | dense | | |
| 41 | +| p4v2_dense_seed03_20261030004544 | completed | laptop | mps | 2620.1 | 499017 | 30.3 | unknown | dense | | |
| 42 | +| p4v2_dense_seed04_20261030001749 | completed | laptop | mps | 3983.98 | 497821 | 28.4 | unknown | dense | | |
| 43 | +| p4v2_dense_seed05_20261030001839 | completed | M4BP36 | mps | 3249.7 | 497078 | 30.3 | unknown | dense | | |
| 44 | +| p4v2_dense_seed05_20261030004544 | completed | laptop | mps | 2620.45 | 497078 | 28.2 | unknown | dense | | |
| 45 | +| p4v2_gr_seed01_20261030001749 | completed | laptop | mps | 1120.54 | 496298 | 2.3 | unknown | global_recurrence | | |
| 30 | 46 | | p4v2_gr_seed02_20261030001749 | completed | laptop | mps | 951.12 | 498543 | 3.4 | unknown | global_recurrence | |
| 31 | 47 | | p4v2_gr_seed03_20261030001749 | completed | laptop | mps | 915.78 | 499017 | 2.3 | unknown | global_recurrence | |
| 32 | 48 | | p4v2_gr_seed04_20261030001749 | completed | laptop | mps | 430.63 | 497821 | 2.3 | unknown | global_recurrence | |
| 33 | 49 | | p4v2_gr_seed05_20261030001839 | completed | laptop | mps | 370.27 | 497078 | 3.4 | unknown | global_recurrence | |
| 50 | +| p4v2_prisme_seed01_20261030001749 | completed | laptop | mps | 2333.13 | 496298 | 8.9 | unknown | prisme | | |
| 34 | 51 | | p4v2_prisme_seed02_20261030001749 | completed | laptop | mps | 1529.74 | 498543 | 8.8 | unknown | prisme | |
| 35 | 52 | | p4v2_prisme_seed03_20261030001749 | completed | laptop | mps | 1578.01 | 499017 | 10.0 | unknown | prisme | |
| 36 | 53 | | p4v2_prisme_seed04_20261030001749 | completed | laptop | mps | 1504.41 | 497821 | 9.3 | unknown | prisme | |
| 37 | 54 | | p4v2_prisme_seed05_20261030001839 | completed | laptop | mps | 939.64 | 497078 | 9.9 | unknown | prisme | |
| 38 | −| pretrain103m_seed01_20261029232706 | running | M3U96a | mps | 0.03 | | | unknown | | | |
| 39 | −| pretrain_enlarged102m_seed01_20261029232914 | running | M3U96b | mps | 0.02 | | | unknown | | | |
| 55 | +| pretrain103m_seed01_20261029232706 | completed | laptop | mps | 8279.79 | 98304000 | 28.0 | unknown | | | |
| 56 | +| pretrain_enlarged102m_seed01_20261030015017 | completed | laptop | mps | 8527.03 | 98304000 | 28.5 | unknown | | | |
modified
results/claim_registry.csv
+26 −25
@@ -1,25 +1,26 @@ | ||
| 1 | −claim,metric,value,benchmark,run_ids,artifact,analysis_script,status | |
| 2 | −Backbone parameter counts equal the proposal for 103M/459M/1.18B,unique parameters,"99502848;446744064;1149339648",meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete | |
| 3 | −Implemented workspace+controller counts differ from the proposal by a closed-form 7w^2,unique parameters,"2631173;10505221;21655813",meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete | |
| 4 | −Analytic ops table reproduces the proposal within 1.3%,ops/token at 459M T=4096,"1.296;1.666;1.787;2.219;5.633 e9",analytic,,results/tables/ops_459m_T4096.json,scripts/estimate_flops.py,complete | |
| 5 | −15 mandated invariants + cache equivalence pass,tests,23 passed,pytest,,tests/,python -m pytest tests,complete | |
| 6 | −Zero-U bootstrap does not learn (scale mismatch),val loss by budget after 700 steps,0.4144/0.4143/0.4143/0.4146/0.4164,synthetic dev-iid v1 backbone,p4_editor_v1_seed01,experiments/runs/p4_editor_v1_seed01_20261029233315/metrics.jsonl,,stopped-run-used | |
| 7 | −Overfitting test passes through the workspace path,train loss 64 items budget 8,0.347->0.0024 in 200 steps,synthetic train subset,p4_overfit64_b8_seed01,experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl,,complete | |
| 8 | −Sparse editor lowers held-out loss vs frozen backbone (H1 weak),dev-iid answer loss relative change by budget,see numbers.tex vtwoprismegainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_prisme,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete | |
| 9 | −Global-recurrence adapter matches or exceeds the sparse editor at every budget,dev-iid answer loss by budget,see numbers.tex vtwogrgainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_gr,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete | |
| 10 | −Variance of intervention gains is mostly between states,within-state share,see numbers.tex labelscalibwithin,editor-unseen label stream (calib audit),labels_v1ctrl_seed01_20261030004018,results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json,evaluation/label_analysis.py,complete | |
| 11 | −One-step oracle lowers dev-iid loss by ~15% at budget 8 (diagnostic),loss relative change,see numbers.tex routingoraclebeight,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 12 | −Learned controller is worse than no routing on dev-iid (H3 null),loss relative change at locked lambda,see numbers.tex routingctrllocked;vtwoctrlrel,synthetic dev-iid v1 and v2 editors,controller_v1ctrl_seed01;controller_v2s2;controller_v2s3;controller_v2s4;controller_v2s5,results/raw/routing_v1ctrl.json;results/analyzed/routing_seeds_v2.json,evaluation/routing.py,complete | |
| 13 | −Fixed schedules: random = uniform = fixed pair (H2 null),loss at budget 1,see numbers.tex routing*bone,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 14 | −Two-step oracle vs one-step: myopia regret small,gain nats,see numbers.tex oracletwostep;oracleonestep,synthetic dev-iid v1 editor (1 batch),controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 15 | −Dispatch overhead dominates at tiny scale; dense not slower than sparse,median ms/token batch 1,see paper/tables/latency.tex,laptop MPS tiny model,p4_editor_v1_Ustd02_seed01,results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json,evaluation/latency.py,complete | |
| 16 | −Miniature end-to-end pipeline runs on CPU,wall seconds,56,mini configs,mini_backbone_seed01;mini_editor_seed01;labels_mini_seed01;controller_mini_seed01,results/raw/routing_mini.json,scripts/reproduce_pilot.sh,complete | |
| 17 | −103M backbone pretrained on 98.3M tokens (ppl 48.2),val loss/ppl,3.8746 / 48.16,FineWeb-Edu val,pretrain103m_seed01_20261029232706,experiments/runs/pretrain103m_seed01_20261029232706/manifest.json,training/pretrain.py,complete | |
| 18 | −103M sparse editor (lr 3e-4) changes held-out loss by less than 0.1% at any budget,val loss relative change,see numbers.tex piloted*,FineWeb-Edu val (training-time eval),editor_pilot_103m_lr3e4_seed01_20261030021200,experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/metrics.jsonl,training/train_editor.py,complete | |
| 19 | −103M sparse editor at lr 1e-3 collapsed to inert edits,val loss by budget,identical 3.9889,FineWeb-Edu val,editor_pilot_103m_seed01_20261030015139,experiments/runs/editor_pilot_103m_seed01_20261030015139/metrics.jsonl,,stopped-run-used | |
| 20 | −103M global-recurrence adapter changes held-out loss by less than 0.1%,val ppl by budget,48.16 -> 48.01,FineWeb-Edu val,gr_pilot_103m_seed01_20261030015117,results/raw/eval_gr_103m_val.json,evaluation/run.py,complete | |
| 21 | −103M controller: oracle opportunity exists (-3.6% at b8) but the controller captures none (-0.01%),val loss relative change,see numbers.tex pilotrouting*,FineWeb-Edu val,controller_103m_seed01,results/raw/routing_103m_val.json,evaluation/routing.py,complete | |
| 22 | −103M latency: 8 sparse edits 3.0x budget-0 median latency; dense 3.6x; controller 1.5x,median ms/token batch 1,7.55 / 22.9 / 26.9 / 11.0,M3 Ultra MPS fp32,controller_103m_seed01,results/raw/latency_103m_mps.json,evaluation/latency.py,complete | |
| 23 | −Pre-registered primary outcome: pooled test EM gain of sparse editor at b2 is +0.9 pp (< 3 pp threshold),exact match pp,see numbers.tex testprismebtwovsbzero*,synthetic locked test regimes (1000 items each),p4v2_prisme (5 seeds),results/analyzed/test_statistics.json;paper/tables/test.tex,scripts/make_results_tables.py,complete | |
| 24 | −Sparse editor vs global recurrence on locked test: -0.3 pp [-0.6 +0.1]; vs dense +0.4 pp [+0.1 +0.7],exact match pp,see numbers.tex testprismevs*,synthetic locked test regimes,p4v2_prisme;p4v2_gr;p4v2_dense,results/analyzed/test_statistics.json,scripts/make_results_tables.py,complete | |
| 25 | −Parameter-matched plain decoder (102.2M) beats backbone+adapters on validation text,val loss,3.8689 vs 3.8746,FineWeb-Edu val,pretrain_enlarged102m_seed01_20261030015017,experiments/runs/pretrain_enlarged102m_seed01_20261030015017/manifest.json,training/pretrain.py,complete | |
| 1 | +claim,metric,value,benchmark,run_ids,artifact,analysis_script,status | |
| 2 | +Backbone parameter counts equal the proposal for 103M/459M/1.18B,unique parameters,99502848;446744064;1149339648,meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete | |
| 3 | +Implemented workspace+controller counts differ from the proposal by a closed-form 7w^2,unique parameters,2631173;10505221;21655813,meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete | |
| 4 | +Analytic ops table reproduces the proposal within 1.3%,ops/token at 459M T=4096,1.296;1.666;1.787;2.219;5.633 e9,analytic,,results/tables/ops_459m_T4096.json,scripts/estimate_flops.py,complete | |
| 5 | +15 mandated invariants + cache equivalence pass,tests,23 passed,pytest,,tests/,python -m pytest tests,complete | |
| 6 | +Zero-U bootstrap does not learn (scale mismatch),val loss by budget after 700 steps,0.4144/0.4143/0.4143/0.4146/0.4164,synthetic dev-iid v1 backbone,p4_editor_v1_seed01_20261029233624,experiments/runs/p4_editor_v1_seed01_20261029233624/metrics.jsonl,,stopped-run-used | |
| 7 | +Overfitting test passes through the workspace path,train loss 64 items budget 8,0.347->0.0024 in 200 steps,synthetic train subset,p4_overfit64_b8_seed01,experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl,,complete | |
| 8 | +Sparse editor lowers held-out loss vs frozen backbone (H1 weak),dev-iid answer loss relative change by budget,see numbers.tex vtwoprismegainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_prisme,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete | |
| 9 | +Global-recurrence adapter matches or exceeds the sparse editor at every budget,dev-iid answer loss by budget,see numbers.tex vtwogrgainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_gr,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete | |
| 10 | +Variance of intervention gains is mostly between states,within-state share,see numbers.tex labelscalibwithin,editor-unseen label stream (calib audit),labels_v1ctrl_seed01_20261030004018,results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json,evaluation/label_analysis.py,complete | |
| 11 | +One-step oracle lowers dev-iid loss by ~15% at budget 8 (diagnostic),loss relative change,see numbers.tex routingoraclebeight,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 12 | +Learned controller is worse than no routing on dev-iid (H3 null),loss relative change at locked lambda,see numbers.tex routingctrllocked;vtwoctrlrel,synthetic dev-iid v1 and v2 editors,controller_v1ctrl_seed01;controller_v2s2;controller_v2s3;controller_v2s4;controller_v2s5,results/raw/routing_v1ctrl.json;results/analyzed/routing_seeds_v2.json,evaluation/routing.py,complete | |
| 13 | +Fixed schedules: random = uniform = fixed pair (H2 null),loss at budget 1,see numbers.tex routing*bone,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 14 | +Two-step oracle vs one-step: myopia regret small,gain nats,see numbers.tex oracletwostep;oracleonestep,synthetic dev-iid v1 editor (1 batch),controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete | |
| 15 | +Dispatch overhead dominates at tiny scale; dense not slower than sparse,median ms/token batch 1,see paper/tables/latency.tex,laptop MPS tiny model,p4_editor_v1_Ustd02_seed01,results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json,evaluation/latency.py,complete | |
| 16 | +Miniature end-to-end pipeline runs on CPU,wall seconds,56,mini configs,mini_backbone_seed01;mini_editor_seed01;labels_mini_seed01;controller_mini_seed01,results/raw/routing_mini.json,scripts/reproduce_pilot.sh,complete | |
| 17 | +103M backbone pretrained on 98.3M tokens (ppl 48.2),val loss/ppl,3.8746 / 48.16,FineWeb-Edu val,pretrain103m_seed01_20261029232706,experiments/runs/pretrain103m_seed01_20261029232706/manifest.json,training/pretrain.py,complete | |
| 18 | +103M sparse editor (lr 3e-4) changes held-out loss by less than 0.1% at any budget,val loss relative change,see numbers.tex piloted*,FineWeb-Edu val (training-time eval),editor_pilot_103m_lr3e4_seed01_20261030021200,experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/metrics.jsonl,training/train_editor.py,complete | |
| 19 | +103M sparse editor at lr 1e-3 collapsed to inert edits,val loss by budget,identical 3.9889,FineWeb-Edu val,editor_pilot_103m_seed01_20261030015139,experiments/runs/editor_pilot_103m_seed01_20261030015139/metrics.jsonl,,stopped-run-used | |
| 20 | +103M global-recurrence adapter changes held-out loss by less than 0.1%,val ppl by budget,48.16 -> 48.01,FineWeb-Edu val,gr_pilot_103m_seed01_20261030015117,results/raw/eval_gr_103m_val.json,evaluation/run.py,complete | |
| 21 | +103M controller: oracle opportunity exists (-3.6% at b8) but the controller captures none (-0.01%),val loss relative change,see numbers.tex pilotrouting*,FineWeb-Edu val,controller_103m_seed01,results/raw/routing_103m_val.json,evaluation/routing.py,complete | |
| 22 | +103M latency: 8 sparse edits 3.0x budget-0 median latency; dense 3.6x; controller 1.5x,median ms/token batch 1,7.55 / 22.9 / 26.9 / 11.0,M3 Ultra MPS fp32,controller_103m_seed01,results/raw/latency_103m_mps.json,evaluation/latency.py,complete | |
| 23 | +Pre-registered primary outcome: pooled test EM gain of sparse editor at b2 is +0.9 pp (< 3 pp threshold),exact match pp,see numbers.tex testprismebtwovsbzero*,synthetic locked test regimes (1000 items each),p4v2_prisme_seed01_20261030001749;p4v2_prisme_seed02_20261030001749;p4v2_prisme_seed03_20261030001749;p4v2_prisme_seed04_20261030001749;p4v2_prisme_seed05_20261030001839,results/analyzed/test_statistics.json;paper/tables/test.tex,scripts/make_results_tables.py,complete | |
| 24 | +Sparse editor vs global recurrence on locked test: -0.3 pp [-0.6 +0.1]; vs dense +0.4 pp [+0.1 +0.7],exact match pp,see numbers.tex testprismevs*,synthetic locked test regimes,p4v2_prisme;p4v2_gr;p4v2_dense,results/analyzed/test_statistics.json,scripts/make_results_tables.py,complete | |
| 25 | +Parameter-matched plain decoder (102.2M) beats backbone+adapters on validation text,val loss,3.8689 vs 3.8746,FineWeb-Edu val,pretrain_enlarged102m_seed01_20261030015017,experiments/runs/pretrain_enlarged102m_seed01_20261030015017/manifest.json,training/pretrain.py,complete | |
| 26 | +Locked FineWeb test: backbone ppl 45.72; sparse editor and global adapter change test loss by -0.08% at budget 4; enlarged decoder lower,test ppl / relative loss,see numbers.tex pilotppl*test;pilottest*,FineWeb-Edu locked test (327680 tokens),pretrain103m_seed01_20261029232706;editor_pilot_103m_lr3e4_seed01_20261030021200;gr_pilot_103m_seed01_20261030015117;pretrain_enlarged102m_seed01_20261030015017,results/raw/eval_backbone_103m_test.json;results/raw/eval_prisme_103m_editor_test.json;results/raw/eval_gr_103m_test.json;results/raw/eval_enlarged_102m_test.json,evaluation/run.py,complete | |
added
results/figures/exact_match_regimes.pdf
+0 −0
Binary file not shown.
added
results/figures/exact_match_regimes.png
+0 −0
Binary file not shown.
modified
results/figures/frontier_v2.pdf
+0 −0
Binary file not shown.
modified
results/figures/labels_analysis.pdf
+0 −0
Binary file not shown.
modified
results/figures/latency_103m_mps.pdf
+0 −0
Binary file not shown.
modified
results/figures/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.pdf
+0 −0
Binary file not shown.
modified
results/figures/learnability_bootstrap.pdf
+0 −0
Binary file not shown.
modified
results/figures/learnability_v2.pdf
+0 −0
Binary file not shown.
modified
results/figures/pretrain_curves.pdf
+0 −0
Binary file not shown.
modified
results/figures/pretrain_curves.png
+0 −0
Binary file not shown.
modified
results/figures/routing_103m_val.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_controller_p4_hent_seed01_20261030002015_dev_iid.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_mini.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v1ctrl.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v1rank.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v2s1.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v2s2.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v2s3.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v2s4.pdf
+0 −0
Binary file not shown.
modified
results/figures/routing_v2s5.pdf
+0 −0
Binary file not shown.
added
results/raw/eval_backbone_103m_test.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "backbone_103m", | |
| 3 | + "split": "test", | |
| 4 | + "checkpoint": "experiments/runs/pretrain103m_seed01_20261029232706/checkpoint", | |
| 5 | + "checkpoint_sha256": "1f05af70bb9223b63dacb765c1816212ec828ff6e81bf8c0d6d46f33ebc77db2", | |
| 6 | + "config_sha256": "3962409f5a94a4dcd6a9898a024ac618f1fbacb74e5f1466318b8989dcf3857b", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:15:27-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8226005017757414, | |
| 15 | + "ppl": 45.722956481048506, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.8226005017757414, | |
| 20 | + "ppl": 45.722956481048506, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.8226005017757414, | |
| 25 | + "ppl": 45.722956481048506, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.8226005017757414, | |
| 30 | + "ppl": 45.722956481048506, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.8226005017757414, | |
| 35 | + "ppl": 45.722956481048506, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 38.6 | |
| 40 | +} | |
| \ No newline at end of file | ||
added
results/raw/eval_backbone_103m_val.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "backbone_103m", | |
| 3 | + "split": "val", | |
| 4 | + "checkpoint": "experiments/runs/pretrain103m_seed01_20261029232706/checkpoint", | |
| 5 | + "checkpoint_sha256": "1f05af70bb9223b63dacb765c1816212ec828ff6e81bf8c0d6d46f33ebc77db2", | |
| 6 | + "config_sha256": "3962409f5a94a4dcd6a9898a024ac618f1fbacb74e5f1466318b8989dcf3857b", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:14:49-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8746298491954803, | |
| 15 | + "ppl": 48.16486672694994, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.8746298491954803, | |
| 20 | + "ppl": 48.16486672694994, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.8746298491954803, | |
| 25 | + "ppl": 48.16486672694994, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.8746298491954803, | |
| 30 | + "ppl": 48.16486672694994, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.8746298491954803, | |
| 35 | + "ppl": 48.16486672694994, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 38.7 | |
| 40 | +} | |
| \ No newline at end of file | ||
added
results/raw/eval_enlarged_102m_test.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "enlarged_102m", | |
| 3 | + "split": "test", | |
| 4 | + "checkpoint": "experiments/runs/pretrain_enlarged102m_seed01_20261030015017/checkpoint", | |
| 5 | + "checkpoint_sha256": "1552f1ec984ec895a40814cd466bde59abaf35e4bfe1ea35b9d43ff4cfe9b9d0", | |
| 6 | + "config_sha256": "0cc9406173e04dcb44f4facbf0dc448da684d5e7a0ba3362385d292e6a125357", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:20:19-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8207094609737395, | |
| 15 | + "ppl": 45.63657420669954, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.8207094609737395, | |
| 20 | + "ppl": 45.63657420669954, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.8207094609737395, | |
| 25 | + "ppl": 45.63657420669954, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.8207094609737395, | |
| 30 | + "ppl": 45.63657420669954, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.8207094609737395, | |
| 35 | + "ppl": 45.63657420669954, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 36.8 | |
| 40 | +} | |
| \ No newline at end of file | ||
added
results/raw/eval_enlarged_102m_val.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "enlarged_102m", | |
| 3 | + "split": "val", | |
| 4 | + "checkpoint": "experiments/runs/pretrain_enlarged102m_seed01_20261030015017/checkpoint", | |
| 5 | + "checkpoint_sha256": "1552f1ec984ec895a40814cd466bde59abaf35e4bfe1ea35b9d43ff4cfe9b9d0", | |
| 6 | + "config_sha256": "0cc9406173e04dcb44f4facbf0dc448da684d5e7a0ba3362385d292e6a125357", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:19:44-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.868902176618576, | |
| 15 | + "ppl": 47.88978268831879, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.868902176618576, | |
| 20 | + "ppl": 47.88978268831879, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.868902176618576, | |
| 25 | + "ppl": 47.88978268831879, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.868902176618576, | |
| 30 | + "ppl": 47.88978268831879, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.868902176618576, | |
| 35 | + "ppl": 47.88978268831879, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 35.2 | |
| 40 | +} | |
| \ No newline at end of file | ||
added
results/raw/eval_gr_103m_test.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "gr_103m", | |
| 3 | + "split": "test", | |
| 4 | + "checkpoint": "experiments/runs/gr_pilot_103m_seed01_20261030015117/checkpoint", | |
| 5 | + "checkpoint_sha256": "1e3e5cdd103672376ad23bbfaafc50bcc68b937d808fae74308285a54564938a", | |
| 6 | + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:16:54-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8226005017757414, | |
| 15 | + "ppl": 45.722956481048506, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.8211239874362946, | |
| 20 | + "ppl": 45.65549569582661, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.8202611207962036, | |
| 25 | + "ppl": 45.61611808291132, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.8195126056671143, | |
| 30 | + "ppl": 45.581986503991125, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.8201463341712953, | |
| 35 | + "ppl": 45.61088226318195, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 45.8 | |
| 40 | +} | |
| \ No newline at end of file | ||
modified
results/raw/eval_gr_103m_val.json
+2 −2
@@ -7,7 +7,7 @@ | ||
| 7 | 7 | "code_commit": "unknown", |
| 8 | 8 | "device": "mps", |
| 9 | 9 | "precision": "fp32", |
| 10 | − "evaluated_at": "2026-10-30T03:43:14-0400", | |
| 10 | + "evaluated_at": "2026-10-30T04:16:08-0400", | |
| 11 | 11 | "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", |
| 12 | 12 | "perplexity": { |
| 13 | 13 | "forced_b0": { |
@@ -36,5 +36,5 @@ | ||
| 36 | 36 | "n_tokens": 327680 |
| 37 | 37 | } |
| 38 | 38 | }, |
| 39 | − "seconds": 45.8 | |
| 39 | + "seconds": 45.9 | |
| 40 | 40 | } |
| \ No newline at end of file | ||
added
results/raw/eval_prisme_103m_editor_test.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "prisme_103m_editor", | |
| 3 | + "split": "test", | |
| 4 | + "checkpoint": "experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/checkpoint", | |
| 5 | + "checkpoint_sha256": "1203ea2e786b258268ea56204e9676a6b900d41628d9fce32af5a01ab0181ea3", | |
| 6 | + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:16:30-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8226005017757414, | |
| 15 | + "ppl": 45.722956481048506, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.821114146709442, | |
| 20 | + "ppl": 45.65504641477478, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.8202012896537783, | |
| 25 | + "ppl": 45.61338890009928, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.819450855255127, | |
| 30 | + "ppl": 45.57917188444815, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.8199622869491576, | |
| 35 | + "ppl": 45.60248847945214, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 86.6 | |
| 40 | +} | |
| \ No newline at end of file | ||
added
results/raw/eval_prisme_103m_editor_val.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "system": "prisme_103m_editor", | |
| 3 | + "split": "val", | |
| 4 | + "checkpoint": "experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/checkpoint", | |
| 5 | + "checkpoint_sha256": "1203ea2e786b258268ea56204e9676a6b900d41628d9fce32af5a01ab0181ea3", | |
| 6 | + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373", | |
| 7 | + "code_commit": "unknown", | |
| 8 | + "device": "mps", | |
| 9 | + "precision": "fp32", | |
| 10 | + "evaluated_at": "2026-10-30T04:14:49-0400", | |
| 11 | + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54", | |
| 12 | + "perplexity": { | |
| 13 | + "forced_b0": { | |
| 14 | + "loss": 3.8746298491954803, | |
| 15 | + "ppl": 48.16486672694994, | |
| 16 | + "n_tokens": 327680 | |
| 17 | + }, | |
| 18 | + "forced_b1": { | |
| 19 | + "loss": 3.8731415927410127, | |
| 20 | + "ppl": 48.093238367070285, | |
| 21 | + "n_tokens": 327680 | |
| 22 | + }, | |
| 23 | + "forced_b2": { | |
| 24 | + "loss": 3.872187703847885, | |
| 25 | + "ppl": 48.04738463431747, | |
| 26 | + "n_tokens": 327680 | |
| 27 | + }, | |
| 28 | + "forced_b4": { | |
| 29 | + "loss": 3.871379178762436, | |
| 30 | + "ppl": 48.00855281891424, | |
| 31 | + "n_tokens": 327680 | |
| 32 | + }, | |
| 33 | + "forced_b8": { | |
| 34 | + "loss": 3.871813231706619, | |
| 35 | + "ppl": 48.02939559571834, | |
| 36 | + "n_tokens": 327680 | |
| 37 | + } | |
| 38 | + }, | |
| 39 | + "seconds": 100.4 | |
| 40 | +} | |
| \ No newline at end of file | ||
modified
scripts/make_results_tables.py
+53 −7
@@ -345,22 +345,29 @@ def table_pilot(macro): | ||
| 345 | 345 | macro(f"{tag}gainb{bn}", "n/a") |
| 346 | 346 | macro(f"{tag}minutes", "n/a"); macro(f"{tag}peakgb", "n/a"); macro(f"{tag}params", "n/a") |
| 347 | 347 | # perplexity table from evaluation.run outputs |
| 348 | − evs = sorted(glob.glob("results/raw/eval_*_val.json")) | |
| 348 | + evs = sorted(glob.glob("results/raw/eval_*_val.json")) + sorted(glob.glob("results/raw/eval_*103m*_test.json")) + sorted(glob.glob("results/raw/eval_enlarged*_test.json")) | |
| 349 | + evs = [f for f in evs if "p4v2" not in f] | |
| 349 | 350 | if evs: |
| 350 | 351 | head = r"""\begin{table}[t]\centering\small |
| 351 | −\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens | |
| 352 | −(40 $\times$ 16 $\times$ 512) under forced random budgets and, for PRISME, under the locked controller.} | |
| 352 | +\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens and | |
| 353 | +on the LOCKED test split (40 $\times$ 16 $\times$ 512 tokens each) under forced random budgets. Budget 0 of the adapters | |
| 354 | +equals the backbone bitwise.} | |
| 353 | 355 | \label{tab:pilotppl} |
| 354 | −\begin{tabular}{llrr}\toprule | |
| 355 | −System & policy & loss & ppl\\\midrule | |
| 356 | +\begin{tabular}{lllrr}\toprule | |
| 357 | +System & split & policy & loss & ppl\\\midrule | |
| 356 | 358 | """ |
| 357 | 359 | body = "" |
| 358 | 360 | for f in evs: |
| 359 | 361 | d = json.load(open(f)) |
| 360 | 362 | for pol, r in d.get("perplexity", {}).items(): |
| 361 | − body += f"{d['system'].replace('_', ' ')} & {pol.replace('_', ' ')} & {r['loss']:.4f} & {r['ppl']:.2f}\\\\\n" | |
| 363 | + if pol == "controller": | |
| 364 | + continue | |
| 365 | + body += f"{d['system'].replace('_', ' ')} & {d['split']} & {pol.replace('_', ' ')} & {r['loss']:.4f} & {r['ppl']:.2f}\\\\\n" | |
| 362 | 366 | if pol == "forced_b0": |
| 363 | − macro("pilotppl" + re.sub(r"[^a-z]", "", d["system"]), f"{r['ppl']:.2f}") | |
| 367 | + macro("pilotppl" + re.sub(r"[^a-z]", "", d["system"]) + d["split"], f"{r['ppl']:.2f}") | |
| 368 | + if d["split"] == "test" and pol in ("forced_b2", "forced_b4"): | |
| 369 | + b0 = d["perplexity"]["forced_b0"]["loss"] | |
| 370 | + macro("pilottest" + re.sub(r"[^a-z]", "", d["system"]) + {"forced_b2": "btwo", "forced_b4": "bfour"}[pol], f"{100*(r['loss']/b0-1):+.2f}") | |
| 364 | 371 | (OUT / "pilot_ppl.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") |
| 365 | 372 | rt = Path("results/raw/routing_103m_val.json") |
| 366 | 373 | if rt.exists(): |
@@ -485,3 +492,42 @@ System & budget & iid & template & depth & compose & pooled\\\midrule | ||
| 485 | 492 | _old_make_all4 = make_all |
| 486 | 493 | def make_all(macro): # noqa: F811 |
| 487 | 494 | _old_make_all4(macro); table_test(macro) |
| 495 | + | |
| 496 | + | |
| 497 | +def table_controls(macro): | |
| 498 | + import torch | |
| 499 | + from training.train_editor import build_model | |
| 500 | + from training.common import load_yaml | |
| 501 | + rows = [] | |
| 502 | + def count(exp, label, data, compute, regime): | |
| 503 | + cfg, m = build_model(exp) | |
| 504 | + nb = sum(p.numel() for p in m.backbone.parameters()); tot = sum(p.numel() for p in m.parameters()) | |
| 505 | + rows.append((label, nb, tot - nb, data, compute, regime)) | |
| 506 | + with torch.device("meta"): | |
| 507 | + count(load_yaml("configs/experiments/p4v2_prisme.yaml"), "Frozen backbone v2 (budget 0)", "synthetic train, 5k steps $\\times$ 64", "backbone only", "no workspace (bitwise bypass)") | |
| 508 | + count(load_yaml("configs/experiments/p4v2_prisme.yaml"), "PRISME sparse editor (tiny)", "same + 3k adapter steps $\\times$ 32", "frozen bb fwd + editor", "1--2 slots/action, 0--8 actions") | |
| 509 | + count(load_yaml("configs/experiments/p4v2_gr.yaml"), "Global-recurrence adapter (tiny)", "same", "frozen bb fwd + block", "whole latent/action, 0--8 actions") | |
| 510 | + count(load_yaml("configs/experiments/p4v2_dense.yaml"), "Dense 16-slot editor (tiny)", "same", "frozen bb fwd + 16 lanes", "16 slots/action, 0--8 actions") | |
| 511 | + e = load_yaml("configs/experiments/editor_pilot_103m_lr3e4.yaml"); e["init"] = {} | |
| 512 | + count(e, "PRISME-103M sparse editor", "FineWeb-Edu 98.3M tok + 24.6M adapter tok", "$6NP$ + editor", "1--2 slots/action") | |
| 513 | + g = load_yaml("configs/experiments/gr_pilot_103m.yaml"); g["init"] = {} | |
| 514 | + count(g, "Global-recurrence adapter (103M)", "same", "$6NP$ + block", "whole latent/action") | |
| 515 | + count({"model": "configs/baselines/transformer_enlarged_102m.yaml", "mode": "backbone_only"}, "Plain decoder, parameter-matched (102.2M)", "FineWeb-Edu 98.3M tok (same recipe)", "$6NP$", "no workspace") | |
| 516 | + with open(OUT / "controls.tex", "w") as f: | |
| 517 | + f.write(r"""\begin{table}[t]\centering\small | |
| 518 | +\caption{Systems and controls. ``bb'' = backbone, ``$6NP$'' = the usual training-FLOP estimate for $N$ tokens and $P$ | |
| 519 | +parameters; adapter training additionally executes up to 8 editor applications per position. Adapter parameters are | |
| 520 | +unique parameters outside the backbone (counted by instantiation; the 103M editor includes the residual-plus-entropy | |
| 521 | +controller features).} | |
| 522 | +\label{tab:controls} | |
| 523 | +\begin{tabular}{lrrlll}\toprule | |
| 524 | +System & backbone & adapter & training exposure & training compute & inference regime\\\midrule | |
| 525 | +""") | |
| 526 | + for lab, nb, ad, data, comp, reg in rows: | |
| 527 | + f.write(f"{lab} & {nb:,} & {ad:,} & {data} & {comp} & {reg}\\\\\n") | |
| 528 | + f.write(r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 529 | + | |
| 530 | + | |
| 531 | +_old_make_all5 = make_all | |
| 532 | +def make_all(macro): # noqa: F811 | |
| 533 | + _old_make_all5(macro); table_controls(macro) | |
| 488 | 534 | |