SPB Git

spb/wp31_uqo

Public

WP31: final assembly — locked evaluations, audit PASS, 48-page manuscript, handoff

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent dc69523 Browse files

44 changed files +778 −111

44 changed files
Unified Split
modified PROGRESS.md +6 −0
@@ -21,3 +21,9 @@
21 21 | 2026-10-30 | 4 dense-writes ablation (v2 backbone, 5 seeds) | done | `p4v2_dense.yaml` → `results/analyzed/seed_matrix_v2.json` | dev-iid loss −2.5/−3.2/−3.2/−1.6 % at b=1/2/4/8 (vs sparse −2.1/−2.8/−3.0/−1.0, GR −3.6/−4.1/−3.7/−3.3); EM +2.0 pp (b1), t-CI [1.6, 2.4] | sparse local writes ≤ dense (H2 second half null; 5 seeds: −2.4/−3.1/−3.1/−1.5 %, EM +2.0 pp [1.6, 2.4]) | — |
22 22 | 2026-10-30 | 8–10 pilot stages B–D + routing + latency (103M) | done | `editor_pilot_103m_lr3e4` (3000 steps, 90 min, peak 25.9 GB), `labels_103m` (491k labels, 32,768 audit states), `controller_103m`, `routing_103m_val.json`, `latency_103m_mps.json`, `eval_gr_103m_val.json` | editor −0.04/−0.06/−0.08/−0.06 % at b=1/2/4/8; GR −0.08 % (ppl 48.16 → 48.01); controller locked (λ=0.3, κ=0.5) −0.01 % with 0.61 edits/token, calib sign acc 51 %, captured 7 %; oracle b8 −3.6 % (4.3 edits/token); two-step oracle 0.0162 vs one-step 0.0088 nats; latency batch 1: 7.55 ms (b0) → 22.9 (8 single) / 26.9 (8 dense) / 11.0 (controller) ms; dense 103M probe collapsed twice | H1 not supported on web text; H3 null; dispatch overhead still dominant (sparse 3.0× vs dense 3.6× measured, analytic 1.6× vs 3.1×) | enlarged baseline eval, final audit |
23 23 | 2026-10-30 | 9 locked test evaluation (synthetic, pre-registered primary outcome) | done | prereg frozen v1.0 04:04, manifest unsealed 04:04; `test_eval_v2_s*.yaml` on 5 nodes → `results/raw/eval_p4v2_*_test_*.json`, `results/analyzed/test_statistics.json` | pooled EM (iid+template+depth) b2 vs b0: PRISME +0.9 pp [+0.9, +1.0]; PRISME − GR −0.3 [−0.6, +0.1]; PRISME − dense +0.4 [+0.1, +0.7]; compose 4–5 % all systems | pre-registered threshold (3 pp) NOT met → do not advance | FineWeb test perplexity (4 systems) running |
24 +| 2026-10-30 | 9 locked FineWeb test (4 systems) | done | `locked_evaluation.yaml` → `results/raw/eval_{backbone_103m,prisme_103m_editor,gr_103m,enlarged_102m}_{val,test}.json` | test ppl: backbone 45.72; editor/GR −0.08 % loss at b=4; enlarged plain decoder 45.64 (val 47.89 vs 48.16) | larger ordinary Transformer wins at matched training (pre-registered null) | — |
25 +| 2026-10-30 | 11 MLX / MacLustr engineering | not executed | — | PyTorch/MPS measurements only | SKIPPED (resource/time; declared in paper) | — |
26 +| 2026-10-30 | 12 scaling (459M / 1.18B) | meta-instantiation only | `scripts/count_params.py`, `scripts/estimate_flops.py` | exact counts + ops table | no training (declared) | — |
27 +| 2026-10-30 | 13 statistics + audit | done | `scripts/statistics.py`, `scripts/verify_manuscript_claims.py` → AUDIT PASS (25 claims), `reproducibility/audit_report.md`, `checksums.sha256` | — | PASS | — |
28 +| 2026-10-30 | 14 manuscript | done | `paper/main.pdf` 48 p., 0 undefined refs, 78 verified refs, tables/figures generated | — | PASS | — |
29 +| 2026-10-30 | 15 handoff | done | `REPRODUCE.md`, `scripts/reproduce_pilot.sh` (56 s CPU end-to-end), `reproducibility/run_index.md` (52 runs), model/data cards | — | PASS | next experiment: within-state ranking labels at a scale where arithmetic dominates dispatch, GR as baseline |
modified experiments/runs/p4_editor_v1_seed01_20261029233624/manifest.json +1 −1
@@ -9,7 +9,7 @@
9 9 "python": "3.12.13",
10 10 "torch": "2.15.0",
11 11 "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py --config configs/experiments/p4_editor_bootstrap.yaml --device mps",
12 − "state": "running",
12 + "state": "stopped_by_operator_scale_mismatch_diagnosed",
13 13 "wall_seconds": 0.02,
14 14 "device": "mps",
15 15 "backend": "mps",
modified paper/appendix/runs_index.tex +38 −4
@@ -2,6 +2,8 @@
2 2
3 3 \begin{itemize}
4 4 \tightlist
5 +\item
6 + \textbf{run\_id}: controller\_103m\_seed01\_20261030034451; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 315.08; \textbf{commit}: unknown
5 7 \item
6 8 \textbf{run\_id}: controller\_mini\_seed01\_20261030005154; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: cpu; \textbf{wall s}: 3.14; \textbf{commit}: ede67055
7 9 \item
@@ -10,6 +12,10 @@
10 12 \textbf{run\_id}: controller\_p4\_hent\_seed01\_20261030002015; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 584.36; \textbf{commit}: unknown
11 13 \item
12 14 \textbf{run\_id}: controller\_v1ctrl\_seed01\_20261030004131; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 164.25; \textbf{commit}: 33298d96
15 +\item
16 + \textbf{run\_id}: controller\_v1rank\_seed01\_20261030005431; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 171.86; \textbf{commit}: d6746e03
17 +\item
18 + \textbf{run\_id}: controller\_v2s1\_seed01\_20261030005758; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 141.05; \textbf{commit}: unknown
13 19 \item
14 20 \textbf{run\_id}: controller\_v2s2\_seed01\_20261030004717; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 193.23; \textbf{commit}: unknown
15 21 \item
@@ -18,6 +24,14 @@
18 24 \textbf{run\_id}: controller\_v2s4\_seed01\_20261030004719; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 217.1; \textbf{commit}: unknown
19 25 \item
20 26 \textbf{run\_id}: controller\_v2s5\_seed01\_20261030004719; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 214.63; \textbf{commit}: unknown
27 +\item
28 + \textbf{run\_id}: editor\_pilot\_103m\_lr3e4\_seed01\_20261030021200; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 5397.54; \textbf{train tokens}: 24576000; \textbf{peak mem (GB)}: 25.9; \textbf{commit}: unknown; \textbf{notes}: prisme
29 +\item
30 + \textbf{run\_id}: editor\_pilot\_103m\_seed01\_20261030015139; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.24; \textbf{commit}: unknown; \textbf{notes}: prisme
31 +\item
32 + \textbf{run\_id}: gr\_pilot\_103m\_seed01\_20261030015117; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1230.52; \textbf{train tokens}: 24576000; \textbf{peak mem (GB)}: 9.7; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence
33 +\item
34 + \textbf{run\_id}: labels\_103m\_seed01\_20261030034314; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 94.09; \textbf{commit}: unknown
21 35 \item
22 36 \textbf{run\_id}: labels\_mini\_seed01\_20261030005150; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: cpu; \textbf{wall s}: 3.44; \textbf{commit}: ede67055
23 37 \item
@@ -26,6 +40,8 @@
26 40 \textbf{run\_id}: labels\_v1ctrl\_seed01\_20261030003957; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: 33298d96
27 41 \item
28 42 \textbf{run\_id}: labels\_v1ctrl\_seed01\_20261030004018; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 72.1; \textbf{commit}: 33298d96
43 +\item
44 + \textbf{run\_id}: labels\_v2s1\_seed01\_20261030005712; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 44.65; \textbf{commit}: unknown
29 45 \item
30 46 \textbf{run\_id}: labels\_v2s2\_seed01\_20261030004543; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 92.45; \textbf{commit}: unknown
31 47 \item
@@ -43,7 +59,7 @@
43 59 \item
44 60 \textbf{run\_id}: p4\_editor\_v1\_scaled\_seed01\_20261029234334; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.16; \textbf{commit}: HEAD; \textbf{notes}: prisme
45 61 \item
46 − \textbf{run\_id}: p4\_editor\_v1\_seed01\_20261029233624; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: prisme
62 + \textbf{run\_id}: p4\_editor\_v1\_seed01\_20261029233624; \textbf{state}: stopped\_by\_operator\_scale\_mismatch\_diagnosed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: prisme
47 63 \item
48 64 \textbf{run\_id}: p4\_overfit64\_b8\_seed01\_20261029235111; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 519.6; \textbf{train tokens}: 95773; \textbf{peak mem (GB)}: 6.7; \textbf{commit}: a9d94e8a; \textbf{notes}: prisme
49 65 \item
@@ -51,7 +67,23 @@
51 67 \item
52 68 \textbf{run\_id}: p4\_tiny\_backbone\_v2\_seed01\_20261029233624; \textbf{state}: stopped\_overfitting\_step9300\_checkpoint\_step5000\_used; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: HEAD; \textbf{notes}: backbone\_only
53 69 \item
54 − \textbf{run\_id}: p4v2\_dense\_seed05\_20261030001839; \textbf{state}: running; \textbf{host}: M4BP36; \textbf{device}: mps; \textbf{wall s}: 0.33; \textbf{commit}: unknown; \textbf{notes}: dense
70 + \textbf{run\_id}: p4v2\_dense\_seed01\_20261030013421; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2465.35; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense
71 +\item
72 + \textbf{run\_id}: p4v2\_dense\_seed02\_20261030001749; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.51; \textbf{commit}: unknown; \textbf{notes}: dense
73 +\item
74 + \textbf{run\_id}: p4v2\_dense\_seed02\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2601.32; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense
75 +\item
76 + \textbf{run\_id}: p4v2\_dense\_seed03\_20261030001749; \textbf{state}: running; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 0.54; \textbf{commit}: unknown; \textbf{notes}: dense
77 +\item
78 + \textbf{run\_id}: p4v2\_dense\_seed03\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2620.1; \textbf{train tokens}: 499017; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense
79 +\item
80 + \textbf{run\_id}: p4v2\_dense\_seed04\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 3983.98; \textbf{train tokens}: 497821; \textbf{peak mem (GB)}: 28.4; \textbf{commit}: unknown; \textbf{notes}: dense
81 +\item
82 + \textbf{run\_id}: p4v2\_dense\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: M4BP36; \textbf{device}: mps; \textbf{wall s}: 3249.7; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 30.3; \textbf{commit}: unknown; \textbf{notes}: dense
83 +\item
84 + \textbf{run\_id}: p4v2\_dense\_seed05\_20261030004544; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2620.45; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 28.2; \textbf{commit}: unknown; \textbf{notes}: dense
85 +\item
86 + \textbf{run\_id}: p4v2\_gr\_seed01\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1120.54; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 2.3; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence
55 87 \item
56 88 \textbf{run\_id}: p4v2\_gr\_seed02\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 951.12; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 3.4; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence
57 89 \item
@@ -60,6 +92,8 @@
60 92 \textbf{run\_id}: p4v2\_gr\_seed04\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 430.63; \textbf{train tokens}: 497821; \textbf{peak mem (GB)}: 2.3; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence
61 93 \item
62 94 \textbf{run\_id}: p4v2\_gr\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 370.27; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 3.4; \textbf{commit}: unknown; \textbf{notes}: global\_recurrence
95 +\item
96 + \textbf{run\_id}: p4v2\_prisme\_seed01\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 2333.13; \textbf{train tokens}: 496298; \textbf{peak mem (GB)}: 8.9; \textbf{commit}: unknown; \textbf{notes}: prisme
63 97 \item
64 98 \textbf{run\_id}: p4v2\_prisme\_seed02\_20261030001749; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 1529.74; \textbf{train tokens}: 498543; \textbf{peak mem (GB)}: 8.8; \textbf{commit}: unknown; \textbf{notes}: prisme
65 99 \item
@@ -69,7 +103,7 @@
69 103 \item
70 104 \textbf{run\_id}: p4v2\_prisme\_seed05\_20261030001839; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 939.64; \textbf{train tokens}: 497078; \textbf{peak mem (GB)}: 9.9; \textbf{commit}: unknown; \textbf{notes}: prisme
71 105 \item
72 − \textbf{run\_id}: pretrain103m\_seed01\_20261029232706; \textbf{state}: running; \textbf{host}: M3U96a; \textbf{device}: mps; \textbf{wall s}: 0.03; \textbf{commit}: unknown
106 + \textbf{run\_id}: pretrain103m\_seed01\_20261029232706; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 8279.79; \textbf{train tokens}: 98304000; \textbf{peak mem (GB)}: 28.0; \textbf{commit}: unknown
73 107 \item
74 − \textbf{run\_id}: pretrain\_enlarged102m\_seed01\_20261029232914; \textbf{state}: running; \textbf{host}: M3U96b; \textbf{device}: mps; \textbf{wall s}: 0.02; \textbf{commit}: unknown
108 + \textbf{run\_id}: pretrain\_enlarged102m\_seed01\_20261030015017; \textbf{state}: completed; \textbf{host}: laptop; \textbf{device}: mps; \textbf{wall s}: 8527.03; \textbf{train tokens}: 98304000; \textbf{peak mem (GB)}: 28.5; \textbf{commit}: unknown
75 109 \end{itemize}
modified paper/main.log +42 −45
@@ -1,4 +1,4 @@
1 −This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 04:15
1 +This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 04:21
2 2 entering extended mode
3 3 restricted \write18 enabled.
4 4 %&-line parsing enabled.
@@ -1297,18 +1297,19 @@ Overfull \hbox (36.62411pt too wide) in paragraph at lines 19--24
1297 1297 t/m/n/12 data/processed/fineweb_edu_300m/manifest.json\T1/ptm/m/n/12 (-20) ).
1298 1298 []
1299 1299
1300 −[16]
1301 −
1302 −LaTeX Warning: Reference `tab:controls' on page 17 undefined on input line 27.
1300 +[16] (./tables/controls.tex
1301 +Overfull \hbox (344.15573pt too wide) in paragraph at lines 7--16
1302 + [][]
1303 + []
1303 1304
1304 −) (./sections/results.tex
1305 −Overfull \hbox (71.70984pt too wide) in paragraph at lines 38--1
1305 +)) (./sections/results.tex
1306 +Overfull \hbox (71.70984pt too wide) in paragraph at lines 39--1
1306 1307 \T1/ptm/m/n/12 (-20) The pri-mary out-come, met-rics, de-ci-sion thresh-olds an
1307 1308 d seed plan were writ-ten to \T1/cmtt/m/n/12 protocol/preregistration.md
1308 1309 []
1309 1310
1310 1311 [17]
1311 −<../results/figures/learnability_bootstrap.pdf, id=494, 353.75287pt x 191.45125
1312 +<../results/figures/learnability_bootstrap.pdf, id=496, 353.75287pt x 191.45125
1312 1313 pt>
1313 1314 File: ../results/figures/learnability_bootstrap.pdf Graphic file (type pdf)
1314 1315 <use ../results/figures/learnability_bootstrap.pdf>
@@ -1321,14 +1322,14 @@ Overfull \hbox (133.05286pt too wide) in paragraph at lines 7--21
1321 1322 []
1322 1323
1323 1324 )
1324 −<../results/figures/frontier_v2.pdf, id=499, 305.58919pt x 208.87035pt>
1325 +<../results/figures/frontier_v2.pdf, id=501, 305.58919pt x 208.87035pt>
1325 1326 File: ../results/figures/frontier_v2.pdf Graphic file (type pdf)
1326 1327 <use ../results/figures/frontier_v2.pdf>
1327 1328 Package pdftex.def Info: ../results/figures/frontier_v2.pdf used on input line
1328 1329 59.
1329 1330 (pdftex.def) Requested size: 338.22418pt x 231.19547pt.
1330 − [18 <../results/figures/learnability_bootstrap.pdf>] [19]
1331 −<../results/figures/labels_analysis.pdf, id=536, 548.77519pt x 192.17598pt>
1331 + [18] [19 <../results/figures/learnability_bootstrap.pdf>]
1332 +<../results/figures/labels_analysis.pdf, id=538, 548.77519pt x 192.17598pt>
1332 1333 File: ../results/figures/labels_analysis.pdf Graphic file (type pdf)
1333 1334 <use ../results/figures/labels_analysis.pdf>
1334 1335 Package pdftex.def Info: ../results/figures/labels_analysis.pdf used on input
@@ -1340,12 +1341,12 @@ Overfull \hbox (109.74734pt too wide) in paragraph at lines 8--15
1340 1341 [][]
1341 1342 []
1342 1343
1343 −) [20 <../results/figures/frontier_v2.pdf>] (./tables/stats.tex) [21 <../result
1344 −s/figures/labels_analysis.pdf>] (./tables/test.tex) (./tables/latency.tex)
1345 −(./sections/results_pilot.tex [22] (./tables/pilot_ppl.tex)
1346 −(./tables/pilot_routing.tex) (./tables/pilot_latency.tex)))
1347 −(./sections/mechanistic.tex [23]) (./sections/limitations.tex [24])
1348 −(./sections/conclusion.tex [25]) (./sections/reproducibility.tex)
1344 +) [20] (./tables/stats.tex) [21 <../results/figures/frontier_v2.pdf>]
1345 +(./tables/test.tex) [22 <../results/figures/labels_analysis.pdf>]
1346 +(./tables/latency.tex) (./sections/results_pilot.tex (./tables/pilot_ppl.tex)
1347 +[23] (./tables/pilot_routing.tex) (./tables/pilot_latency.tex)))
1348 +(./sections/mechanistic.tex [24]) (./sections/limitations.tex [25])
1349 +(./sections/conclusion.tex) (./sections/reproducibility.tex)
1349 1350 Overfull \hbox (51.18217pt too wide) in paragraph at lines 2--101
1350 1351 \T1/ptm/m/n/12 (-20) num-ber in the manuscript is pro-duced by \T1/cmtt/m/n/12
1351 1352 scripts/make_tables.py \T1/ptm/m/n/12 (-20) and \T1/cmtt/m/n/12 scripts/reprodu
@@ -1359,18 +1360,18 @@ cript_claims.py \T1/ptm/m/n/12 (-20) checks the chain. \T1/cmtt/m/n/12 REPRODUC
1359 1360 E.md
1360 1361 []
1361 1362
1362 −[26] (./main.bbl [27] [28]
1363 +[26] [27] (./main.bbl [28] [29]
1363 1364 Underfull \hbox (badness 1297) in paragraph at lines 208--211
1364 1365 []\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe
1365 1366 i Ning, Ning Ding, and
1366 1367 []
1367 1368
1368 −[29]) (./appendix/appendix.tex (./appendix/notation.tex
1369 +[30]) (./appendix/appendix.tex (./appendix/notation.tex
1369 1370 Overfull \hbox (12.19864pt too wide) in paragraph at lines 2--16
1370 1371 []
1371 1372 []
1372 1373
1373 −) (./appendix/proofs.tex [30]) (./appendix/params.tex [31]
1374 +) (./appendix/proofs.tex [31]) (./appendix/params.tex [32]
1374 1375 Overfull \hbox (92.32437pt too wide) detected at line 12
1375 1376 \OML/ztmcm/m/it/12 N[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/it/12 d \OT1/ztmcm/m/n/
1376 1377 12 + 3\OML/ztmcm/m/it/12 dw \OT1/ztmcm/m/n/12 + \OML/ztmcm/m/it/12 Mw \OT1/ztmc
@@ -1479,7 +1480,7 @@ LaTeX Warning: `h' float specifier changed to `ht'.
1479 1480 ) (./appendix/prereg.tex
1480 1481 LaTeX Font Info: Font shape `T1/cmtt/bx/n' in size <12> not available
1481 1482 (Font) Font shape `T1/cmtt/m/n' tried instead on input line 1.
1482 − [32]
1483 + [33]
1483 1484 LaTeX Font Info: Trying to load font information for TS1+ptm on input line 8
1484 1485 .
1485 1486
@@ -1492,7 +1493,7 @@ Overfull \hbox (80.09856pt too wide) in paragraph at lines 24--26
1492 1493 T1/cmtt/m/n/12 data/generators/synthetic.py\T1/ptm/m/n/12 (-20) ,
1493 1494 []
1494 1495
1495 −[33] [34]) (./appendix/runs.tex
1496 +[34] [35]) (./appendix/runs.tex
1496 1497 Overfull \hbox (121.91187pt too wide) in paragraph at lines 1--14
1497 1498 \T1/cmtt/m/n/12 training.pretrain\T1/ptm/m/n/12 (-20) , \T1/cmtt/m/n/12 trainin
1498 1499 g.train_editor\T1/ptm/m/n/12 (-20) , \T1/cmtt/m/n/12 training.collect_intervent
@@ -1507,14 +1508,14 @@ tion.latency \T1/ptm/m/n/12 (-20) or \T1/cmtt/m/n/12 evaluation.run\T1/ptm/m/n/
1507 1508 12 (-20) ; launch scripts un-der \T1/cmtt/m/n/12 scripts/
1508 1509 []
1509 1510
1510 −(./appendix/runs_index.tex [35] [36]
1511 −Overfull \hbox (25.12868pt too wide) in paragraph at lines 52--54
1511 +(./appendix/runs_index.tex [36] [37]
1512 +Overfull \hbox (25.12868pt too wide) in paragraph at lines 68--70
1512 1513 []\T1/ptm/b/n/10 (-20) run_id\T1/ptm/m/n/10 (-20) : p4_tiny_backbone_v2_seed01_
1513 1514 20261029233624; \T1/ptm/b/n/10 (-20) state\T1/ptm/m/n/10 (-20) : stopped_overfi
1514 1515 tting_step9300_checkpoint_step5000_used;
1515 1516 []
1516 1517
1517 −)) (./appendix/results.tex (./tables/em_families.tex))
1518 +[38])) (./appendix/results.tex (./tables/em_families.tex))
1518 1519 Overfull \hbox (168.4564pt too wide) in paragraph at lines 2--16
1519 1520 \T1/ptm/m/n/12 (-20) records (\T1/cmtt/m/n/12 experiments/runs/*/em_*.json\T1/p
1520 1521 tm/m/n/12 (-20) , \T1/cmtt/m/n/12 results/raw/items_*.json\T1/ptm/m/n/12 (-20)
@@ -1538,14 +1539,14 @@ ileNotFoundError\T1/ptm/m/n/12 (-20) ).; \T1/ptm/b/n/12 (-20) ev-i-dence\T1/ptm
1538 1539 g
1539 1540 []
1540 1541
1541 −[37]
1542 +
1542 1543 Overfull \hbox (49.9369pt too wide) in paragraph at lines 10--12
1543 1544 \T1/ptm/m/n/12 (-20) be-fore ed-its can af-fect log-its.; \T1/ptm/b/n/12 (-20)
1544 1545 ev-i-dence\T1/ptm/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/runs/p4_editor_v1_
1545 1546 seed01_*/metrics.jsonl
1546 1547 []
1547 1548
1548 −
1549 +[39]
1549 1550 Overfull \hbox (47.44559pt too wide) in paragraph at lines 12--14
1550 1551 \T1/cmtt/m/n/12 experiments/logs/p4_diag_mixed_w128_lr3e3.log\T1/ptm/m/n/12 (-2
1551 1552 0) , run \T1/cmtt/m/n/12 p4_diag_mixed_w128_lr3e3_seed01_*\T1/ptm/m/n/12 (-20)
@@ -1570,15 +1571,15 @@ Overfull \hbox (12.79033pt too wide) in paragraph at lines 18--20
1570 1571 run \T1/cmtt/m/n/12 p4_tiny_backbone_v2_seed01_* \T1/ptm/m/n/12 (-20) (state:
1571 1572 []
1572 1573
1573 −[38{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1.en
1574 −c}]
1574 +
1575 1575 Overfull \hbox (12.01894pt too wide) in paragraph at lines 22--24
1576 1576 \T1/ptm/m/n/12 (-20) 0.4334); $\OML/ztmcm/m/it/12 ^^U$\T1/ptm/m/n/12 (-20) =0 i
1577 1577 s worse (+1.3 %).; \T1/ptm/b/n/12 (-20) ev-i-dence\T1/ptm/m/n/12 (-20) : \T1/cm
1578 1578 tt/m/n/12 experiments/runs/controller_v1rank_seed01_*\T1/ptm/m/n/12 (-20) ,
1579 1579 []
1580 1580
1581 −
1581 +[40{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1.en
1582 +c}]
1582 1583 Overfull \hbox (146.55865pt too wide) in paragraph at lines 24--26
1583 1584 \T1/ptm/m/n/12 (-20) un-til step ~6,540 and was killed.; \T1/ptm/b/n/12 (-20) e
1584 1585 v-i-dence\T1/ptm/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/runs/pretrain103m_s
@@ -1599,26 +1600,22 @@ dence\T1/ptm/m/n/12 (-20) : M2U64 \T1/cmtt/m/n/12 experiments/logs/dense_probe_
1599 1600 103m*_seed1.log\T1/ptm/m/n/12 (-20) ;
1600 1601 []
1601 1602
1602 −)) [39] [40] [41] [42] [43] [44] [45] (./main.aux)
1603 +)) [41] [42] [43] [44] [45] [46] [47] [48] (./main.aux)
1603 1604 ***********
1604 1605 LaTeX2e <2025-11-01>
1605 1606 L3 programming layer <2026-01-19>
1606 1607 ***********
1607 −
1608 −
1609 −LaTeX Warning: There were undefined references.
1610 −
1611 1608 Package rerunfilecheck Info: File `main.out' has not changed.
1612 −(rerunfilecheck) Checksum: 5C078824F258CF894B89F741D112AD48;9386.
1609 +(rerunfilecheck) Checksum: 2518BAC872CA29A47E8E1B5E242EBEEE;9386.
1613 1610 )
1614 1611 Here is how much of TeX's memory you used:
1615 − 36174 strings out of 467525
1616 − 703202 string characters out of 5418982
1617 − 1152637 words of memory out of 5000000
1618 − 63458 multiletter control sequences out of 15000+600000
1619 − 793836 words of font info for 673 fonts, out of 8000000 for 9000
1612 + 36217 strings out of 467525
1613 + 703862 string characters out of 5418982
1614 + 1163186 words of memory out of 5000000
1615 + 63476 multiletter control sequences out of 15000+600000
1616 + 794607 words of font info for 685 fonts, out of 8000000 for 9000
1620 1617 1141 hyphenation exceptions out of 8191
1621 − 84i,13n,104p,1001b,1484s stack positions out of 10000i,1000n,20000p,200000b,200000s
1618 + 84i,13n,104p,1001b,1530s stack positions out of 10000i,1000n,20000p,200000b,200000s
1622 1619 </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb
1623 1620 ></usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb>
1624 1621 </usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></
@@ -1634,10 +1631,10 @@ exlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026
1634 1631 e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti
1635 1632 mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8
1636 1633 a.pfb>
1637 −Output written on main.pdf (45 pages, 447044 bytes).
1634 +Output written on main.pdf (48 pages, 450927 bytes).
1638 1635 PDF statistics:
1639 − 824 PDF objects out of 1000 (max. 8388607)
1640 − 709 compressed objects within 8 object streams
1641 − 167 named destinations out of 1000 (max. 500000)
1636 + 839 PDF objects out of 1000 (max. 8388607)
1637 + 721 compressed objects within 8 object streams
1638 + 171 named destinations out of 1000 (max. 500000)
1642 1639 143785 words of extra memory for PDF output out of 154059 (max. 10000000)
1643 1640
modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/sections/conclusion.tex +1 −1
@@ -1,7 +1,7 @@
1 1 \section{Conclusion}\label{sec:conclusion}
2 2 The central question was whether counterfactual supervision of \emph{where} to edit a small latent workspace
3 3 improves the quality--compute frontier beyond uniform workspace computation and adaptive global recurrence.
4 −On the evidence collected here the answer is no. The sparse editor is learnable and helps a frozen backbone
4 +On the evidence collected here the answer is no, and the pre-registered decision rule says so: on the locked test regimes the primary effect is $\testprismebtwovsbzero$ points against a three-point threshold. The sparse editor is learnable and helps a frozen backbone
5 5 by a few percent of loss and two points of exact match when it is applied once or twice at every position, but
6 6 a global-recurrence adapter trained with the identical recipe helps at least as much at every budget; where to
7 7 write barely matters for fixed schedules; an exhaustive oracle shows large opportunity (up to
modified paper/sections/design.tex +1 −0
@@ -33,6 +33,7 @@ recipe and the identical symmetry-breaking device. Reference routers (random, un
33 33 entropy threshold, norm-change threshold) reuse the trained \prisme{} editor and are therefore matched in
34 34 everything except the choice of action. The exhaustive one-step and two-step oracles read the target and are
35 35 reported as ceilings only.
36 +\input{tables/controls.tex}
36 37
37 38 \subsection{Pre-registration, seeds and uncertainty}
38 39 The primary outcome, metrics, decision thresholds and seed plan were written to
modified paper/sections/results_pilot.tex +6 −2
@@ -17,8 +17,12 @@ global adapter, which did not collapse at $10^{-3}$, gives $-\pilotgrgainbone$,
17 17 $-\pilotgrgainbfour$ and $-\pilotgrgainbeight\,\%$ in \pilotgrminutes{} min. The dense sixteen-slot probe
18 18 collapsed at both learning rates and yields no 103M result. On general web text, therefore, neither adapter
19 19 moves a frozen 100M backbone by more than a tenth of a percent of loss with 24.6M adapter tokens---less than the
20 −run-to-run noise of pretraining itself---so H1 is not supported outside the synthetic suite, and the
21 −parameter-matched plain decoder is the relevant comparison (Table~\ref{tab:pilotppl}).
20 +run-to-run noise of pretraining itself---so H1 is not supported outside the synthetic suite. On the locked
21 +FineWeb-Edu test split (327,680 tokens) the backbone's perplexity is \pilotpplbackbonemtest{}; forced edits
22 +change the test loss by \pilottestprismemeditorbfour\,\% (sparse, budget 4) and \pilottestgrmbfour\,\%
23 +(global); the parameter-matched plain decoder trained with the identical recipe reaches \pilotpplenlargedmtest{}
24 +(Table~\ref{tab:pilotppl})---the proposal's null ``a larger ordinary Transformer wins after equalising training
25 +compute'' holds at this budget.
22 26 \input{tables/pilot_ppl.tex}
23 27
24 28 \paragraph{Stages C--D and routing.} Counterfactual labels were collected on a validation-disjoint stream of
modified paper/tables/controls.tex +11 −10
@@ -1,15 +1,16 @@
1 1 \begin{table}[t]\centering\small
2 −\caption{Systems and controls. ``bb'' = backbone, ``6NP'' = the usual training-FLOP estimate $6\,N\,P$ for $N$ tokens
3 −and $P$ parameters; adapter training additionally executes up to 8 editor applications per position. Adapter
4 −parameters are unique parameters outside the backbone (counted by instantiation).}
2 +\caption{Systems and controls. ``bb'' = backbone, ``$6NP$'' = the usual training-FLOP estimate for $N$ tokens and $P$
3 +parameters; adapter training additionally executes up to 8 editor applications per position. Adapter parameters are
4 +unique parameters outside the backbone (counted by instantiation; the 103M editor includes the residual-plus-entropy
5 +controller features).}
5 6 \label{tab:controls}
6 7 \begin{tabular}{lrrlll}\toprule
7 8 System & backbone & adapter & training exposure & training compute & inference regime\\\midrule
8 −Frozen backbone v2 (budget 0) & 4,754,688 & 182,725 & synthetic train, 5k steps x 64 & backbone only & no workspace (bitwise bypass)\\
9 −PRISME sparse editor (tiny) & 4,754,688 & 182,725 & same + 3k adapter steps x 32 & frozen bb fwd + editor & 1-2 slots/action, 0-8 actions\\
10 −Global-recurrence adapter (tiny) & 4,754,688 & 172,608 & same & frozen bb fwd + block & whole latent/action, 0-8 actions\\
11 −Dense 16-slot editor (tiny) & 4,754,688 & 182,725 & same & frozen bb fwd + 16 lanes & 16 slots/action, 0-8 actions\\
12 −PRISME-103M (editor pilot) & 99,502,848 & 2,762,757 & FineWeb-Edu 196.6M tok + 24.6M adapter tok & 6NP + editor & 1-2 slots/action\\
13 −Global-recurrence adapter (103M) & 99,502,848 & 2,623,488 & same & 6NP + block & whole latent/action\\
14 −Plain decoder, parameter-matched (102.2M) & 102,157,056 & 0 & FineWeb-Edu 196.6M tok (same recipe) & 6NP & no workspace\\
9 +Frozen backbone v2 (budget 0) & 4,754,688 & 182,725 & synthetic train, 5k steps $\times$ 64 & backbone only & no workspace (bitwise bypass)\\
10 +PRISME sparse editor (tiny) & 4,754,688 & 182,725 & same + 3k adapter steps $\times$ 32 & frozen bb fwd + editor & 1--2 slots/action, 0--8 actions\\
11 +Global-recurrence adapter (tiny) & 4,754,688 & 172,608 & same & frozen bb fwd + block & whole latent/action, 0--8 actions\\
12 +Dense 16-slot editor (tiny) & 4,754,688 & 182,725 & same & frozen bb fwd + 16 lanes & 16 slots/action, 0--8 actions\\
13 +PRISME-103M sparse editor & 99,502,848 & 2,762,757 & FineWeb-Edu 98.3M tok + 24.6M adapter tok & $6NP$ + editor & 1--2 slots/action\\
14 +Global-recurrence adapter (103M) & 99,502,848 & 2,623,488 & same & $6NP$ + block & whole latent/action\\
15 +Plain decoder, parameter-matched (102.2M) & 102,157,056 & 0 & FineWeb-Edu 98.3M tok (same recipe) & $6NP$ & no workspace\\
15 16 \bottomrule\end{tabular}\end{table}
modified paper/tables/numbers.tex +16 −1
@@ -63,12 +63,27 @@
63 63 \newcommand{\pilotlatctrl}{11.0}
64 64 \newcommand{\pilotlatdeight}{26.9}
65 65 \newcommand{\pilotlatseight}{22.9}
66 −\newcommand{\pilotpplgrm}{48.16}
66 +\newcommand{\pilotpplbackbonemtest}{45.72}
67 +\newcommand{\pilotpplbackbonemval}{48.16}
68 +\newcommand{\pilotpplenlargedmtest}{45.64}
69 +\newcommand{\pilotpplenlargedmval}{47.89}
70 +\newcommand{\pilotpplgrmtest}{45.72}
71 +\newcommand{\pilotpplgrmval}{48.16}
72 +\newcommand{\pilotpplprismemeditortest}{45.72}
73 +\newcommand{\pilotpplprismemeditorval}{48.16}
67 74 \newcommand{\pilotroutingctrl}{-0.01}
68 75 \newcommand{\pilotroutingctrledits}{0.61}
69 76 \newcommand{\pilotroutingoracle}{-3.6}
70 77 \newcommand{\pilotroutingoracleedits}{4.28}
71 78 \newcommand{\pilotroutinguniformone}{-0.02}
79 +\newcommand{\pilottestbackbonembfour}{+0.00}
80 +\newcommand{\pilottestbackbonembtwo}{+0.00}
81 +\newcommand{\pilottestenlargedmbfour}{+0.00}
82 +\newcommand{\pilottestenlargedmbtwo}{+0.00}
83 +\newcommand{\pilottestgrmbfour}{-0.08}
84 +\newcommand{\pilottestgrmbtwo}{-0.06}
85 +\newcommand{\pilottestprismemeditorbfour}{-0.08}
86 +\newcommand{\pilottestprismemeditorbtwo}{-0.06}
72 87 \newcommand{\routingbzero}{0.4334}
73 88 \newcommand{\routingctrllambda}{0.3}
74 89 \newcommand{\routingctrlledits}{0.22}
modified paper/tables/pilot_ppl.tex +45 −9
@@ -1,12 +1,48 @@
1 1 \begin{table}[t]\centering\small
2 −\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens
3 −(40 $\times$ 16 $\times$ 512) under forced random budgets and, for PRISME, under the locked controller.}
2 +\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens and
3 +on the LOCKED test split (40 $\times$ 16 $\times$ 512 tokens each) under forced random budgets. Budget 0 of the adapters
4 +equals the backbone bitwise.}
4 5 \label{tab:pilotppl}
5 −\begin{tabular}{llrr}\toprule
6 −System & policy & loss & ppl\\\midrule
7 −gr 103m & forced b0 & 3.8746 & 48.16\\
8 −gr 103m & forced b1 & 3.8731 & 48.09\\
9 −gr 103m & forced b2 & 3.8722 & 48.05\\
10 −gr 103m & forced b4 & 3.8714 & 48.01\\
11 −gr 103m & forced b8 & 3.8718 & 48.03\\
6 +\begin{tabular}{lllrr}\toprule
7 +System & split & policy & loss & ppl\\\midrule
8 +backbone 103m & val & forced b0 & 3.8746 & 48.16\\
9 +backbone 103m & val & forced b1 & 3.8746 & 48.16\\
10 +backbone 103m & val & forced b2 & 3.8746 & 48.16\\
11 +backbone 103m & val & forced b4 & 3.8746 & 48.16\\
12 +backbone 103m & val & forced b8 & 3.8746 & 48.16\\
13 +enlarged 102m & val & forced b0 & 3.8689 & 47.89\\
14 +enlarged 102m & val & forced b1 & 3.8689 & 47.89\\
15 +enlarged 102m & val & forced b2 & 3.8689 & 47.89\\
16 +enlarged 102m & val & forced b4 & 3.8689 & 47.89\\
17 +enlarged 102m & val & forced b8 & 3.8689 & 47.89\\
18 +gr 103m & val & forced b0 & 3.8746 & 48.16\\
19 +gr 103m & val & forced b1 & 3.8731 & 48.09\\
20 +gr 103m & val & forced b2 & 3.8722 & 48.05\\
21 +gr 103m & val & forced b4 & 3.8714 & 48.01\\
22 +gr 103m & val & forced b8 & 3.8718 & 48.03\\
23 +prisme 103m editor & val & forced b0 & 3.8746 & 48.16\\
24 +prisme 103m editor & val & forced b1 & 3.8731 & 48.09\\
25 +prisme 103m editor & val & forced b2 & 3.8722 & 48.05\\
26 +prisme 103m editor & val & forced b4 & 3.8714 & 48.01\\
27 +prisme 103m editor & val & forced b8 & 3.8718 & 48.03\\
28 +backbone 103m & test & forced b0 & 3.8226 & 45.72\\
29 +backbone 103m & test & forced b1 & 3.8226 & 45.72\\
30 +backbone 103m & test & forced b2 & 3.8226 & 45.72\\
31 +backbone 103m & test & forced b4 & 3.8226 & 45.72\\
32 +backbone 103m & test & forced b8 & 3.8226 & 45.72\\
33 +gr 103m & test & forced b0 & 3.8226 & 45.72\\
34 +gr 103m & test & forced b1 & 3.8211 & 45.66\\
35 +gr 103m & test & forced b2 & 3.8203 & 45.62\\
36 +gr 103m & test & forced b4 & 3.8195 & 45.58\\
37 +gr 103m & test & forced b8 & 3.8201 & 45.61\\
38 +prisme 103m editor & test & forced b0 & 3.8226 & 45.72\\
39 +prisme 103m editor & test & forced b1 & 3.8211 & 45.66\\
40 +prisme 103m editor & test & forced b2 & 3.8202 & 45.61\\
41 +prisme 103m editor & test & forced b4 & 3.8195 & 45.58\\
42 +prisme 103m editor & test & forced b8 & 3.8200 & 45.60\\
43 +enlarged 102m & test & forced b0 & 3.8207 & 45.64\\
44 +enlarged 102m & test & forced b1 & 3.8207 & 45.64\\
45 +enlarged 102m & test & forced b2 & 3.8207 & 45.64\\
46 +enlarged 102m & test & forced b4 & 3.8207 & 45.64\\
47 +enlarged 102m & test & forced b8 & 3.8207 & 45.64\\
12 48 \bottomrule\end{tabular}\end{table}
added reproducibility/audit_report.md +16 −0
@@ -0,0 +1,16 @@
1 +# Results-to-claim audit report (WP31, 2026-10-30)
2 +
3 +* `scripts/verify_manuscript_claims.py`: **AUDIT PASS** — 25 registered claims; every artefact path exists; every run id has a
4 + manifest (states `completed`, or `stopped-run-used` for the two deliberately stopped runs cited as failures); every numeric
5 + macro used in the manuscript is generated by `scripts/make_tables.py`; no promotional phrase; no placeholder section.
6 +* Tests: `python -m pytest tests -q` → 23 passed (15 mandated invariants, controller target-freedom with and without the
7 + residual/entropy features, dense-write ablation, 5 cache-equivalence tests).
8 +* Manuscript: `paper/main.pdf`, 48 pages, latexmk with 0 undefined references/citations (one harmless font-shape warning).
9 +* Pre-registration: `protocol/preregistration.md` frozen v1.0 at 2026-10-30T04:04-0400; `protocol/locked_test_manifest.json`
10 + unsealed at the same time; test splits were read only afterwards (synthetic test regimes: 5 nodes; FineWeb test: 4 systems).
11 +* Deviations: `protocol/deviations.md` (bootstrap init/scale, controller features, label stream, dense 103M infeasible,
12 + backbone budget 98.3M instead of 196.6M tokens).
13 +* Not executed: Phase 11 (MLX port), Phase 12 training (459M/1.18B only instantiated on meta), GSM8K/EvalPlus, M ∈ {4,8,32},
14 + editor depth ∈ {1,4}, rematched pairs, quantisation, distillation, joint fine-tuning (one degraded attempt), dense editor at 103M
15 + (collapsed twice), MLX/Metal profiling.
16 +* Checksums of the key artefacts: `reproducibility/checksums.sha256`.
added reproducibility/checksums.sha256 +213 −0
@@ -0,0 +1,213 @@
1 +7d2419ffa1f2892bc7dafc95685f767cc79de6554e219f7f1a76570c8d3790b1 paper/main.pdf
2 +a1a3468ffe50fba1beb150871b8d5ad575163ba1cd96373838d7ff989380853d results/claim_registry.csv
3 +70cb86ea329cb0052e0956db0374d8e55b43305056edd778e7512cf5ee6fef02 results/analyzed/labels_labels_p4_seed01_20261030000543.json
4 +e5765cbde70d9faf34b4aec875a44184b8ae368531d5ccca29e5ecfddb4a020e results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json
5 +77f6c3406df0ed977b90c4b5dbeccfecfff63a5054a87dbe1c6ddb3e23acc53d results/analyzed/routing_seeds_v2.json
6 +c121c2bfd141a3ffbc161c8213bb9189507a965434c22b65005432f9cb723a58 results/analyzed/seed_matrix_v2.json
7 +de0677e8600a9d82c293ad49469fdb5346a3fe55f05d8f6d24315dadcd5f147c results/analyzed/statistics_v2.json
8 +b7923aef253503c8d8ea1740c9791b691a2b620519eba798c1d34fc12dd1e64d results/analyzed/test_statistics.json
9 +7945a43fde621b133add69976be3d1f50bfa6a3e976bacae6a8075f6663d4312 results/raw/eval_backbone_103m_test.json
10 +1db7d53edf42e2065ee66749b5b1231a0e793a51d3b7b3bad0475e907fd65fad results/raw/eval_backbone_103m_val.json
11 +56efdc8762fefc5ee2a6dd0e3ee3a88edfe51450576838862ca4528f85df8f99 results/raw/eval_enlarged_102m_test.json
12 +4ed5f6d040e0bb84741a54de6de385a76a4db78d26a883c31ccd4442223ae877 results/raw/eval_enlarged_102m_val.json
13 +24a17a932585a6a46116375b71c4e72847892b58070d3d979e1c17e41cee13db results/raw/eval_gr_103m_test.json
14 +94d76e5cdf6da940f16ec113eca2daf25fe9069b35c6537f83eea513ea6caad6 results/raw/eval_gr_103m_val.json
15 +240a85d62d9ebe4a274314e0d50a2e8c2ad4b4b581bfdb5f95d0579d132c5604 results/raw/eval_p4v2_dense_s1_test_compose.json
16 +b4601f1bfb75d0618c8a48e2a7b9f15d8b9f0906192c1235c3a355d0c95d25e9 results/raw/eval_p4v2_dense_s1_test_depth.json
17 +0de49b5dc700f1db63066d69d8bc5fcfb6dc12a1efaefb72be033840f1be13c7 results/raw/eval_p4v2_dense_s1_test_iid.json
18 +a4bf328b91b69fd77878f7d278d3d81e8d708ea809e84eb82e55dd08e3e580ae results/raw/eval_p4v2_dense_s1_test_template.json
19 +59348dced041971ace2836f13d744ae07e1e1be9ecf21ab35b254db499d0ea7e results/raw/eval_p4v2_dense_s2_test_compose.json
20 +f9b5bd5d98040d5556d5b624987c1d4f10f89f3af74480ce909167ab14b9465f results/raw/eval_p4v2_dense_s2_test_depth.json
21 +e0c772078e0277a185890e6a0f37bee555be1e2856fd181f62ce5b067a9a36e9 results/raw/eval_p4v2_dense_s2_test_iid.json
22 +b8a8f4bd4feccf6468755ee8eee6cc73baf50726ff3e4a7384a849d845ce9460 results/raw/eval_p4v2_dense_s2_test_template.json
23 +ff25f532fa987ba08027d677b77e730308eb98debe5b2861e9d63ad896daf047 results/raw/eval_p4v2_dense_s3_test_compose.json
24 +4c22fe9f58e85b80c7a04d66a367dc9d8468df7a37feab0a55e96a488d86108c results/raw/eval_p4v2_dense_s3_test_depth.json
25 +ccae7be5c414503e6916af1b496ac5d76e8ccf10ed02938bc302b3063dd55bcd results/raw/eval_p4v2_dense_s3_test_iid.json
26 +19a05738052fdb253b6b0cca58386c80a98b4a460d399d4eb86587b499760d7e results/raw/eval_p4v2_dense_s3_test_template.json
27 +553412dbdae6f569ff2a4aaba12cebbaf46b545e63f17b2c92af727db3f2cacc results/raw/eval_p4v2_dense_s4_test_compose.json
28 +9c9a7cb00ba1e9e2edf419961966508cf00fb8bd9b8c5e19bc4297f875d4f3db results/raw/eval_p4v2_dense_s4_test_depth.json
29 +6d16a9462d9e90ed1210f095b30445a1abc8e95fabf4d5262d26b8bcc9f75821 results/raw/eval_p4v2_dense_s4_test_iid.json
30 +cb601cd8f310f29e19ce6b880d19dd68178d83970d8eb2659c79dd24bcef7f55 results/raw/eval_p4v2_dense_s4_test_template.json
31 +a6cabd657169d423a1d49befb485d161f79ca7d7c31c457e6467870a315fc60b results/raw/eval_p4v2_dense_s5_test_compose.json
32 +40cb9aa807935ef72c86d1cd2a41d3a3a76fa933c9f36c50a5216aa31c50245f results/raw/eval_p4v2_dense_s5_test_depth.json
33 +7d6bc6039c4c2be34c18d9016350765e35bccc4f2a8e7f7b13dbddabe22af1ce results/raw/eval_p4v2_dense_s5_test_iid.json
34 +6a18b127adeb7267d2a41c3113215c3d9bd56aa48189461bba2ba383fc131437 results/raw/eval_p4v2_dense_s5_test_template.json
35 +a89e20b97e580615df0d3818d5280af6a722d8ae442d898bd5c945ea0550936d results/raw/eval_p4v2_gr_s1_test_compose.json
36 +bd42a30c976c62752907bb84e71dd43a3634a27c937d5e82093ce7dd6fe87905 results/raw/eval_p4v2_gr_s1_test_depth.json
37 +5636d906e8c4aadf73b89a2e10fd4e0a6dd1c62099949eda5bf9ee92fa630e47 results/raw/eval_p4v2_gr_s1_test_iid.json
38 +88c25f1afa4e344e2a09f95fa4356bef066288459a5237a4efa2fc6cbcb4deaf results/raw/eval_p4v2_gr_s1_test_template.json
39 +e83025dcb455a1e417098583271de0a5f460d86086a0b8beb034fef12bf304cd results/raw/eval_p4v2_gr_s2_test_compose.json
40 +c93d645871dd63aebaef1e9c5b43725bbf9c7b0cd9805ebe5c7c67683742ef89 results/raw/eval_p4v2_gr_s2_test_depth.json
41 +4adae524b80f03a7b3db761f6449e28a5d191b9d35d864eef208a2ad014cd9b5 results/raw/eval_p4v2_gr_s2_test_iid.json
42 +ff3b6070ecca2de27276c2b6c81a045bc05cc32ed696d30c581b7b5b4c490a86 results/raw/eval_p4v2_gr_s2_test_template.json
43 +cc3310f947c2ab0556978bd721d68d11b8cb252fb52a4e5c44710956b4339415 results/raw/eval_p4v2_gr_s3_test_compose.json
44 +c48b3b0ba3533819c092ebcde1a578f69d44cec34a38247e2a2eb1c59489e01a results/raw/eval_p4v2_gr_s3_test_depth.json
45 +f9df369e3f358110e301413d6a6071aefbed562c08f06957d71514af738f9827 results/raw/eval_p4v2_gr_s3_test_iid.json
46 +b2281f2b745aa5be1972dbb0d111e60f3b39ec6b2e0a7e228a852a5a158b629b results/raw/eval_p4v2_gr_s3_test_template.json
47 +daff4678d7c288ffc1c891dd8fc73ea63e9c00ef165fee1c786f7ab6910d27ef results/raw/eval_p4v2_gr_s4_test_compose.json
48 +edcdfebbc348086b2578c00c4ffd757e0d135bf24b67c202598b09961e39080a results/raw/eval_p4v2_gr_s4_test_depth.json
49 +5275701b43e763cea01617bbc6ba406622d7dada9975e17d152543e4d86e9b37 results/raw/eval_p4v2_gr_s4_test_iid.json
50 +781772b860a0d8171b135122d05bcbef9402bfa75e302d64f75a5eff54d297bd results/raw/eval_p4v2_gr_s4_test_template.json
51 +3f44da7d05b20404c12639942152172942faf325b81b89391e1b7d300ac1ec6d results/raw/eval_p4v2_gr_s5_test_compose.json
52 +01ea31f8424a5a00b90164cb641e6e8af66dbb0e8176bdb4da8aa572a7f1268f results/raw/eval_p4v2_gr_s5_test_depth.json
53 +8dd943be0d13f10b3eab74249e96bf474e8cb761f111323df42b3619e46ae875 results/raw/eval_p4v2_gr_s5_test_iid.json
54 +6dbccfdd4ea90d91b2bc37adba864c4d44866b2de489956e379e7f6f5e106e04 results/raw/eval_p4v2_gr_s5_test_template.json
55 +21fd0d5905ca637f9b79a7be000909865197774dca6f47252d007caa620e834a results/raw/eval_p4v2_prisme_s1_test_compose.json
56 +f313c6d0183813f2a1273d7a79f34d91436c1951933482c368392df81d779f31 results/raw/eval_p4v2_prisme_s1_test_depth.json
57 +2912cf4bb956a7d1483e8246909e0f6fcf44b4fd66cdf41c87dd70f89403407f results/raw/eval_p4v2_prisme_s1_test_iid.json
58 +e7ce172fdbcd7daee5b41c1cce7e7935201f55a862b748b47f767c5f0fbde186 results/raw/eval_p4v2_prisme_s1_test_template.json
59 +9cf33ba1b80f7bd9e0740f2de0bbb51a9044b573094a6178c629447e386aba41 results/raw/eval_p4v2_prisme_s2_test_compose.json
60 +7b09000cca0ce54acdd7b989ee52ebbe8c977af5967949d4ff2d6f052b152640 results/raw/eval_p4v2_prisme_s2_test_depth.json
61 +7b029010b158cb1654792bb03c31c3348ebbc3d1e1fb2e02dc0ecc2b4ca9d4d2 results/raw/eval_p4v2_prisme_s2_test_iid.json
62 +432e089082d068e591934353180b450c6f1b6899e8146af8f058fdbc0e041705 results/raw/eval_p4v2_prisme_s2_test_template.json
63 +32d899d80fe6e96c75145a0cad12cbb0b1d46743e0bc5097ebb2b3a1dceba931 results/raw/eval_p4v2_prisme_s3_test_compose.json
64 +0cb5219b36308c768f96fd23008284b2470d773e87c9cf54200cdc5b95ee53c3 results/raw/eval_p4v2_prisme_s3_test_depth.json
65 +9f614b3f111cd9f97e5b5fe376f508f96311a1db33c67a3862dcd0eb8cf8e2e4 results/raw/eval_p4v2_prisme_s3_test_iid.json
66 +aa3eeac5cc2c680170fa82e19e5e6a748f92771285d110fc80b001f04b7f7a5d results/raw/eval_p4v2_prisme_s3_test_template.json
67 +508ba5332b8d84219703c07971c6212f9e8d8c5fee0e7e15bec90bb19bac836c results/raw/eval_p4v2_prisme_s4_test_compose.json
68 +b667a0ddc3794ee2412a1ff183631a5ba9f67c417ea21c8a46a849ce6c04d888 results/raw/eval_p4v2_prisme_s4_test_depth.json
69 +833f5eed3a48185c17ea58dfa76e297d93f6b875bb08651bc324120a30f8b3bf results/raw/eval_p4v2_prisme_s4_test_iid.json
70 +b2e41313e851338880f5ae5ac181fb665775f5ef90ffe8a458e79952b60079da results/raw/eval_p4v2_prisme_s4_test_template.json
71 +5a78fa1db60b6f2d699e404540c43cdfffa8264290d76804d7cc88741d7d663c results/raw/eval_p4v2_prisme_s5_test_compose.json
72 +6fae55bfa3d1531da0047057efab493fad5d59387631a6af5ce50d59c2f17898 results/raw/eval_p4v2_prisme_s5_test_depth.json
73 +26b429fd5eb464ad19d7502d4de4bfc1bf8ff3c138c63e035da2d36bb8ac0dc2 results/raw/eval_p4v2_prisme_s5_test_iid.json
74 +169577d4954ca6786db7e95a0b10d3f5e67821154908097895fd67e54ab62020 results/raw/eval_p4v2_prisme_s5_test_template.json
75 +83200604d660282d28acbf8eca212562915edd1bff3ed19d18010195ce98fb54 results/raw/eval_prisme_103m_editor_test.json
76 +bb1080bbcfb2ff25471be1dc13409dab4531b7a07f4f133e1427f41b2c4d047c results/raw/eval_prisme_103m_editor_val.json
77 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s1_test_compose_budget0.json
78 +2e999f28ca2c658f23598b80bc9952815f9f23d385c960bd1d5d95e51cca1d17 results/raw/items_p4v2_dense_s1_test_compose_forced_b2.json
79 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s1_test_depth_budget0.json
80 +c509d6ebf76ed9fd8f3ba93c7c0bed1d9dc062f599f1572f9b814b25b76e1ece results/raw/items_p4v2_dense_s1_test_depth_forced_b2.json
81 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s1_test_iid_budget0.json
82 +c767436f7a6c79a2c93cdf99096249263cd1432d7a1a4562945ddf407eaca69d results/raw/items_p4v2_dense_s1_test_iid_forced_b2.json
83 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s1_test_template_budget0.json
84 +8448cc79e9d1e9ab28274143142207f345427d12dccea5a23b5a4417eece564b results/raw/items_p4v2_dense_s1_test_template_forced_b2.json
85 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s2_test_compose_budget0.json
86 +16a7d8b011b8a10caef1ba95393d488e19a2d850eeaf80a8aa95ba14d5b04c1d results/raw/items_p4v2_dense_s2_test_compose_forced_b2.json
87 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s2_test_depth_budget0.json
88 +a02eaf64716ddddcfb10708cdd215597f525323881fff3e1831731be22814924 results/raw/items_p4v2_dense_s2_test_depth_forced_b2.json
89 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s2_test_iid_budget0.json
90 +dd07b669a62500527d704d123584184be2aa19e032e1b933e3a83d3d7c466fc1 results/raw/items_p4v2_dense_s2_test_iid_forced_b2.json
91 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s2_test_template_budget0.json
92 +f51973ecfd545bc9780240e219a2ad5bd98cf83ca1085add5c65eb800a8653fe results/raw/items_p4v2_dense_s2_test_template_forced_b2.json
93 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s3_test_compose_budget0.json
94 +307a8c7781f3c75a5a65edb17140ef94be91fbf2d5776607a4f75a32a2f3f575 results/raw/items_p4v2_dense_s3_test_compose_forced_b2.json
95 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s3_test_depth_budget0.json
96 +7ef70865bf95cd33268c5e2124ab6f7a649e128f09d51053891d16ca63e950b8 results/raw/items_p4v2_dense_s3_test_depth_forced_b2.json
97 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s3_test_iid_budget0.json
98 +ab88e74b8581317c7e4fbd0c6e3f4a55c51d4209f344fcbf90d004787fbef4c0 results/raw/items_p4v2_dense_s3_test_iid_forced_b2.json
99 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s3_test_template_budget0.json
100 +60a01032ada2b17ae0b1cc80bee58da6167eba49f678e206f6dca3e832eca4fe results/raw/items_p4v2_dense_s3_test_template_forced_b2.json
101 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s4_test_compose_budget0.json
102 +f93c20b054f2760e287a8dab70fd6fed984f0d7b4e137a488d7a515e4748ca0b results/raw/items_p4v2_dense_s4_test_compose_forced_b2.json
103 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s4_test_depth_budget0.json
104 +a1c523c689ee99410142d4e89cb3997a10342a00e712dab11bd37d560d1a5928 results/raw/items_p4v2_dense_s4_test_depth_forced_b2.json
105 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s4_test_iid_budget0.json
106 +c9cfad851190b133393b90636bb74625f1434226cc1c13c7744fad5714bb1b46 results/raw/items_p4v2_dense_s4_test_iid_forced_b2.json
107 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s4_test_template_budget0.json
108 +ddd0fe249aa523c6c8b5e65666829710af812c4503001b425a8c33af637a6761 results/raw/items_p4v2_dense_s4_test_template_forced_b2.json
109 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_dense_s5_test_compose_budget0.json
110 +1ed8e804351e8c2daf199cba00355d83db19d363f4830ab024f7fd0a37f6fe60 results/raw/items_p4v2_dense_s5_test_compose_forced_b2.json
111 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_dense_s5_test_depth_budget0.json
112 +c99c501680904e3cf2d2f6845a6bc4f4a15b10a581ecc7592da8bba5a9d60924 results/raw/items_p4v2_dense_s5_test_depth_forced_b2.json
113 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_dense_s5_test_iid_budget0.json
114 +91568b5ec9200783cf78320454f1cf988d14d418a93d5198e5bd1e62a1cd1a29 results/raw/items_p4v2_dense_s5_test_iid_forced_b2.json
115 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_dense_s5_test_template_budget0.json
116 +505f27c6fffe6875e91f62221bb14693a3bf0467fd091bfd41b2b1e032872dd9 results/raw/items_p4v2_dense_s5_test_template_forced_b2.json
117 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s1_test_compose_budget0.json
118 +e464fdc8a2b60c7609843dce2a8c22747bd897c1adb1886616eb9d4210f35c17 results/raw/items_p4v2_gr_s1_test_compose_forced_b2.json
119 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s1_test_depth_budget0.json
120 +8aad9b15f27d52155e4928ebe77e36aea8bf19390f044531fcf0733dd5f5adfc results/raw/items_p4v2_gr_s1_test_depth_forced_b2.json
121 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s1_test_iid_budget0.json
122 +5fd4cde26848fb97275e93925c6e24c8777da268be39dad2e8962dc604c4aaa8 results/raw/items_p4v2_gr_s1_test_iid_forced_b2.json
123 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s1_test_template_budget0.json
124 +3e77bad1a73c4a811c92611002499a7e948f19c2bdec93c442ad4578f2e8eaec results/raw/items_p4v2_gr_s1_test_template_forced_b2.json
125 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s2_test_compose_budget0.json
126 +7fee922e3864adcf6eb2d7d66353393ec4df5948769c1c8adb06d920ded6c0cb results/raw/items_p4v2_gr_s2_test_compose_forced_b2.json
127 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s2_test_depth_budget0.json
128 +93db1e816557fdcfbdf374cfc12f6eb08cc8b88d1bfce4c49a594756f5247b07 results/raw/items_p4v2_gr_s2_test_depth_forced_b2.json
129 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s2_test_iid_budget0.json
130 +187f6a7fc6131419ac322a7005e11b58bd829938042d5ab0eae81285ed62c189 results/raw/items_p4v2_gr_s2_test_iid_forced_b2.json
131 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s2_test_template_budget0.json
132 +c9f640c16d349590b8dde575cc8218e93e653e0e9f5f060b87e144dee4a2d8af results/raw/items_p4v2_gr_s2_test_template_forced_b2.json
133 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s3_test_compose_budget0.json
134 +b05f703226a7c4db4484e46c027bbbe768f8e41d8a4cdcdff07a64aaf0d67e7f results/raw/items_p4v2_gr_s3_test_compose_forced_b2.json
135 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s3_test_depth_budget0.json
136 +ecbb22971316f411cccc02692f4eefe40a297771824c8b94a74645399f1a3379 results/raw/items_p4v2_gr_s3_test_depth_forced_b2.json
137 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s3_test_iid_budget0.json
138 +d292ff34868f76e572e1ad4daf6545e222f0240f5fcb8fdd3a8921e3840cdb0d results/raw/items_p4v2_gr_s3_test_iid_forced_b2.json
139 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s3_test_template_budget0.json
140 +8b43517446e9f59c9153606c135bf048feec04406b30bc9d26f4a4a43c8d9484 results/raw/items_p4v2_gr_s3_test_template_forced_b2.json
141 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s4_test_compose_budget0.json
142 +ab3e9552cd2a6a230cdb8f33a4973893bbb107ba4f5cdb1c31cc74c86d8c2a26 results/raw/items_p4v2_gr_s4_test_compose_forced_b2.json
143 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s4_test_depth_budget0.json
144 +487cd30e77ff443531b33cd047f9eb401a9e4177cbad513bb6e75e8e09e68abf results/raw/items_p4v2_gr_s4_test_depth_forced_b2.json
145 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s4_test_iid_budget0.json
146 +1d353dfdbc37d45ecf8a90d8846c5325830c91fc4ac3f69a06ce8fe265a28595 results/raw/items_p4v2_gr_s4_test_iid_forced_b2.json
147 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s4_test_template_budget0.json
148 +476b357454472720aeb0730eb10903c7251683fe8d3a7991b0df2eeaffb70aad results/raw/items_p4v2_gr_s4_test_template_forced_b2.json
149 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_gr_s5_test_compose_budget0.json
150 +b79c36c2b71c270f90e54776c0e470fce0d15f8376bf751b7b7781c6fcf63332 results/raw/items_p4v2_gr_s5_test_compose_forced_b2.json
151 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_gr_s5_test_depth_budget0.json
152 +f2f016d4cd3346e60eacc131cc100d462a21638d3490c5a240d51a9d46315dc4 results/raw/items_p4v2_gr_s5_test_depth_forced_b2.json
153 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_gr_s5_test_iid_budget0.json
154 +2d5df71b91fe76a80463efde1a216de1a2a6339aacd9e0c584cfbfa0938a629d results/raw/items_p4v2_gr_s5_test_iid_forced_b2.json
155 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_gr_s5_test_template_budget0.json
156 +32eb5e11f354321177fbd7dbd0841473cf2270e4cf9df5a0f9464dcbfb59ad9e results/raw/items_p4v2_gr_s5_test_template_forced_b2.json
157 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s1_test_compose_budget0.json
158 +80586b49e01e888dfb56a0a5db4e54d58cb4856668ca21278c86ebe4e11506aa results/raw/items_p4v2_prisme_s1_test_compose_forced_b2.json
159 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s1_test_depth_budget0.json
160 +4ab1a75242a89a0af0e6d024749d2cf088c3b8e58c9f2f663940be9564c52b2b results/raw/items_p4v2_prisme_s1_test_depth_forced_b2.json
161 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s1_test_iid_budget0.json
162 +8391d02c1a270e306ee86ebfc6fed306ab2817bbef784e6b807b993a08b9e5bf results/raw/items_p4v2_prisme_s1_test_iid_forced_b2.json
163 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s1_test_template_budget0.json
164 +d969667cd1fd32f07dd4033ede38e756200b6af096f9d2aa5764b5848013e9ac results/raw/items_p4v2_prisme_s1_test_template_forced_b2.json
165 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s2_test_compose_budget0.json
166 +81f46e521c20c05cc1981017682681adf39aee9e685f6e3aaf9888d3768bd639 results/raw/items_p4v2_prisme_s2_test_compose_forced_b2.json
167 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s2_test_depth_budget0.json
168 +b82f82f9dda0335045995f92f005abe6cd48c27e54b5de65ed453c9f0c16dd51 results/raw/items_p4v2_prisme_s2_test_depth_forced_b2.json
169 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s2_test_iid_budget0.json
170 +8761df4b6ae82ce5848e5b8736f6d4d6cb3453ec1daac23317416e791fa59236 results/raw/items_p4v2_prisme_s2_test_iid_forced_b2.json
171 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s2_test_template_budget0.json
172 +b84990037836a760f101b381216c8ba32455ee65f7ae14871b0ad72373399b15 results/raw/items_p4v2_prisme_s2_test_template_forced_b2.json
173 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s3_test_compose_budget0.json
174 +2233020039ad7f833d596c4d29b0ea8544302eff1a418eb051d9a7c4692c8c74 results/raw/items_p4v2_prisme_s3_test_compose_forced_b2.json
175 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s3_test_depth_budget0.json
176 +c38c9207609aaf1d8012712bfbf69898ceb8c2e3947517131c00f574e2df0a19 results/raw/items_p4v2_prisme_s3_test_depth_forced_b2.json
177 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s3_test_iid_budget0.json
178 +474f66cc8bae2acaffb15163852b1a32565706637ddc349a8434870778f168cd results/raw/items_p4v2_prisme_s3_test_iid_forced_b2.json
179 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s3_test_template_budget0.json
180 +07c6750a3bdd19e78fce4a252440a150ce45b2b47f98feefe058d16ede0fd9fb results/raw/items_p4v2_prisme_s3_test_template_forced_b2.json
181 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s4_test_compose_budget0.json
182 +bd79b85556e9d87ea59daa7429b932d03f8c5557e24318e99a953a934a0ebfe3 results/raw/items_p4v2_prisme_s4_test_compose_forced_b2.json
183 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s4_test_depth_budget0.json
184 +f03d3e5be849eda84999cea23d73ce170cf99fbb5286816763bcdd229317577a results/raw/items_p4v2_prisme_s4_test_depth_forced_b2.json
185 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s4_test_iid_budget0.json
186 +89999afb68660788f080c070e8a53c5377700403ba40c6deafa70162f4834046 results/raw/items_p4v2_prisme_s4_test_iid_forced_b2.json
187 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s4_test_template_budget0.json
188 +128df74db3be1d568b86ac2e158af651dd8ff5390cef15f81de6f81afbd600a2 results/raw/items_p4v2_prisme_s4_test_template_forced_b2.json
189 +3d20172b7720ee1d094c7cd1d2c897832c16361d28c40f3fd41969177bf8ef06 results/raw/items_p4v2_prisme_s5_test_compose_budget0.json
190 +e604df74deb203b3d80f0fd271237506da912b87d04dc969e6f1fee513556aca results/raw/items_p4v2_prisme_s5_test_compose_forced_b2.json
191 +4f9f12c4648eba2ca8b664faadb3d797ea85fd97b812952aa6dc24fe567eba34 results/raw/items_p4v2_prisme_s5_test_depth_budget0.json
192 +99e37b0a00330ebdb8790092fa2b194fb065e27fc03c4d58dd0886048e8545d0 results/raw/items_p4v2_prisme_s5_test_depth_forced_b2.json
193 +7bc153555ff0b72561aea0e33d85c96c3e65e4f77cfd065d694bb81c6032766a results/raw/items_p4v2_prisme_s5_test_iid_budget0.json
194 +d5eb96b933c0b67966734d733579e0885ce3710f782d101eee95d629fda4bdc9 results/raw/items_p4v2_prisme_s5_test_iid_forced_b2.json
195 +2f00de69a1f10a905c93d3d6f01488e6590303e7e59db903785acece1bce9025 results/raw/items_p4v2_prisme_s5_test_template_budget0.json
196 +4a273387f2204f907717961a3e5d1f58133ef2b3b03f97f8a947931d4f8f4eae results/raw/items_p4v2_prisme_s5_test_template_forced_b2.json
197 +5ac057e7323760fe0774e0016dad4e4480a3b0fb2bcb2a71fab9b595596f00fa results/raw/latency_103m_mps.json
198 +17fd8d751c915fa60c506718da6e862fb4e973b1b4996599d9ade54512982634 results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json
199 +c7f8878b9b007546d7312cb29b1904992235c243d2d29446c1bec64bcfc9b1e4 results/raw/routing_103m_val.json
200 +a2d6f4fc3f5b1591e6f02215d9d72a537bedc6afec076210b04eab44550a2822 results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json
201 +f7748c20a6446a22b1e307c80cea49a30cc89dc2c4c951d2486659cdbcfc3cef results/raw/routing_mini.json
202 +f20c434c42090ba17603819bdfc127da73ca46163fc6823b6a78013dda08e555 results/raw/routing_v1ctrl.json
203 +95cf28b285b26f6e1a54564b2a2b8ee691e89886df9875c850a1b5606d8831b1 results/raw/routing_v1rank.json
204 +bb85e8001ba798a203fc01e55f212777430ae816acf893eb32ebdfc88a779e4b results/raw/routing_v2s1.json
205 +224965164912b215d4e0be99c6a470b8b6d42a7fc390198424cfeb1a4e8be893 results/raw/routing_v2s2.json
206 +152654d5ded3c0a643dcb2c296580ee8c330877b299d484057eec06f2a32b6a1 results/raw/routing_v2s3.json
207 +909442a2374d2a2929719a7dc0142933ed1a4bf6f1feba2c82c547751d711b7e results/raw/routing_v2s4.json
208 +38c615486b0c298ba1b179327f4e03827d26f7db3662f00657f2ddbfc30a10c3 results/raw/routing_v2s5.json
209 +06620e1cf60b7250aac71f4fb5ee74e3705f4b9aab102e22b5c5bdbed76c4ae2 data/splits/synthetic_v1/manifest.json
210 +d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54 data/processed/fineweb_edu_300m/manifest.json
211 +54ef965fa85e96ebfd88bcdc745a8d19efa56c461d73444a4592dbedc678ac4e protocol/preregistration.md
212 +acd9f9c178b9ab8979d527fed75733100a11eec63fb201f892c4ea016daeb1b9 protocol/locked_test_manifest.json
213 +e9763b53e11f13ed41ccdba86fe95da896bc7b249762be6e4136ccc8a318b71b requirements-lock.txt
modified reproducibility/run_index.md +21 −4
@@ -2,18 +2,26 @@
2 2
3 3 | run_id | state | host | device | wall s | train tokens | peak mem (GB) | commit | notes |
4 4 |---|---|---|---|---|---|---|---|---|
5 +| controller_103m_seed01_20261030034451 | completed | laptop | mps | 315.08 | | | unknown | |
5 6 | controller_mini_seed01_20261030005154 | completed | laptop | cpu | 3.14 | | | ede67055 | |
6 7 | controller_p4_hent_seed01_20261030001419 | running | laptop | mps | 0.02 | | | unknown | |
7 8 | controller_p4_hent_seed01_20261030002015 | completed | laptop | mps | 584.36 | | | unknown | |
8 9 | controller_v1ctrl_seed01_20261030004131 | completed | laptop | mps | 164.25 | | | 33298d96 | |
10 +| controller_v1rank_seed01_20261030005431 | completed | laptop | mps | 171.86 | | | d6746e03 | |
11 +| controller_v2s1_seed01_20261030005758 | completed | laptop | mps | 141.05 | | | unknown | |
9 12 | controller_v2s2_seed01_20261030004717 | completed | laptop | mps | 193.23 | | | unknown | |
10 13 | controller_v2s3_seed01_20261030004720 | completed | laptop | mps | 197.97 | | | unknown | |
11 14 | controller_v2s4_seed01_20261030004719 | completed | laptop | mps | 217.1 | | | unknown | |
12 15 | controller_v2s5_seed01_20261030004719 | completed | laptop | mps | 214.63 | | | unknown | |
16 +| editor_pilot_103m_lr3e4_seed01_20261030021200 | completed | laptop | mps | 5397.54 | 24576000 | 25.9 | unknown | prisme |
17 +| editor_pilot_103m_seed01_20261030015139 | running | laptop | mps | 0.24 | | | unknown | prisme |
18 +| gr_pilot_103m_seed01_20261030015117 | completed | laptop | mps | 1230.52 | 24576000 | 9.7 | unknown | global_recurrence |
19 +| labels_103m_seed01_20261030034314 | completed | laptop | mps | 94.09 | | | unknown | |
13 20 | labels_mini_seed01_20261030005150 | completed | laptop | cpu | 3.44 | | | ede67055 | |
14 21 | labels_p4_seed01_20261030000543 | completed | laptop | mps | 33.33 | | | unknown | |
15 22 | labels_v1ctrl_seed01_20261030003957 | running | laptop | mps | 0.02 | | | 33298d96 | |
16 23 | labels_v1ctrl_seed01_20261030004018 | completed | laptop | mps | 72.1 | | | 33298d96 | |
24 +| labels_v2s1_seed01_20261030005712 | completed | laptop | mps | 44.65 | | | unknown | |
17 25 | labels_v2s2_seed01_20261030004543 | completed | laptop | mps | 92.45 | | | unknown | |
18 26 | labels_v2s3_seed01_20261030004544 | completed | laptop | mps | 95.57 | | | unknown | |
19 27 | labels_v2s4_seed01_20261030004544 | completed | laptop | mps | 94.03 | | | unknown | |
@@ -22,18 +30,27 @@
22 30 | mini_editor_seed01_20261030005118 | completed | laptop | cpu | 30.59 | 9847 | 2.0 | ede67055 | prisme |
23 31 | p4_editor_v1_Ustd02_seed01_20261029235050 | completed | laptop | mps | 650.11 | 331026 | 8.8 | unknown | prisme |
24 32 | p4_editor_v1_scaled_seed01_20261029234334 | running | laptop | mps | 0.16 | | | HEAD | prisme |
25 −| p4_editor_v1_seed01_20261029233624 | running | laptop | mps | 0.02 | | | HEAD | prisme |
33 +| p4_editor_v1_seed01_20261029233624 | stopped_by_operator_scale_mismatch_diagnosed | laptop | mps | 0.02 | | | HEAD | prisme |
26 34 | p4_overfit64_b8_seed01_20261029235111 | completed | laptop | mps | 519.6 | 95773 | 6.7 | a9d94e8a | prisme |
27 35 | p4_tiny_backbone_seed01_20261029232626 | completed | laptop | mps | 457.67 | 44367329 | 3.4 | HEAD | backbone_only |
28 36 | p4_tiny_backbone_v2_seed01_20261029233624 | stopped_overfitting_step9300_checkpoint_step5000_used | laptop | mps | 0.02 | | | HEAD | backbone_only |
29 −| p4v2_dense_seed05_20261030001839 | running | M4BP36 | mps | 0.33 | | | unknown | dense |
37 +| p4v2_dense_seed01_20261030013421 | completed | laptop | mps | 2465.35 | 496298 | 30.3 | unknown | dense |
38 +| p4v2_dense_seed02_20261030001749 | running | laptop | mps | 0.51 | | | unknown | dense |
39 +| p4v2_dense_seed02_20261030004544 | completed | laptop | mps | 2601.32 | 498543 | 30.3 | unknown | dense |
40 +| p4v2_dense_seed03_20261030001749 | running | laptop | mps | 0.54 | | | unknown | dense |
41 +| p4v2_dense_seed03_20261030004544 | completed | laptop | mps | 2620.1 | 499017 | 30.3 | unknown | dense |
42 +| p4v2_dense_seed04_20261030001749 | completed | laptop | mps | 3983.98 | 497821 | 28.4 | unknown | dense |
43 +| p4v2_dense_seed05_20261030001839 | completed | M4BP36 | mps | 3249.7 | 497078 | 30.3 | unknown | dense |
44 +| p4v2_dense_seed05_20261030004544 | completed | laptop | mps | 2620.45 | 497078 | 28.2 | unknown | dense |
45 +| p4v2_gr_seed01_20261030001749 | completed | laptop | mps | 1120.54 | 496298 | 2.3 | unknown | global_recurrence |
30 46 | p4v2_gr_seed02_20261030001749 | completed | laptop | mps | 951.12 | 498543 | 3.4 | unknown | global_recurrence |
31 47 | p4v2_gr_seed03_20261030001749 | completed | laptop | mps | 915.78 | 499017 | 2.3 | unknown | global_recurrence |
32 48 | p4v2_gr_seed04_20261030001749 | completed | laptop | mps | 430.63 | 497821 | 2.3 | unknown | global_recurrence |
33 49 | p4v2_gr_seed05_20261030001839 | completed | laptop | mps | 370.27 | 497078 | 3.4 | unknown | global_recurrence |
50 +| p4v2_prisme_seed01_20261030001749 | completed | laptop | mps | 2333.13 | 496298 | 8.9 | unknown | prisme |
34 51 | p4v2_prisme_seed02_20261030001749 | completed | laptop | mps | 1529.74 | 498543 | 8.8 | unknown | prisme |
35 52 | p4v2_prisme_seed03_20261030001749 | completed | laptop | mps | 1578.01 | 499017 | 10.0 | unknown | prisme |
36 53 | p4v2_prisme_seed04_20261030001749 | completed | laptop | mps | 1504.41 | 497821 | 9.3 | unknown | prisme |
37 54 | p4v2_prisme_seed05_20261030001839 | completed | laptop | mps | 939.64 | 497078 | 9.9 | unknown | prisme |
38 −| pretrain103m_seed01_20261029232706 | running | M3U96a | mps | 0.03 | | | unknown | |
39 −| pretrain_enlarged102m_seed01_20261029232914 | running | M3U96b | mps | 0.02 | | | unknown | |
55 +| pretrain103m_seed01_20261029232706 | completed | laptop | mps | 8279.79 | 98304000 | 28.0 | unknown | |
56 +| pretrain_enlarged102m_seed01_20261030015017 | completed | laptop | mps | 8527.03 | 98304000 | 28.5 | unknown | |
modified results/claim_registry.csv +26 −25
@@ -1,25 +1,26 @@
1 −claim,metric,value,benchmark,run_ids,artifact,analysis_script,status
2 −Backbone parameter counts equal the proposal for 103M/459M/1.18B,unique parameters,"99502848;446744064;1149339648",meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete
3 −Implemented workspace+controller counts differ from the proposal by a closed-form 7w^2,unique parameters,"2631173;10505221;21655813",meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete
4 −Analytic ops table reproduces the proposal within 1.3%,ops/token at 459M T=4096,"1.296;1.666;1.787;2.219;5.633 e9",analytic,,results/tables/ops_459m_T4096.json,scripts/estimate_flops.py,complete
5 −15 mandated invariants + cache equivalence pass,tests,23 passed,pytest,,tests/,python -m pytest tests,complete
6 −Zero-U bootstrap does not learn (scale mismatch),val loss by budget after 700 steps,0.4144/0.4143/0.4143/0.4146/0.4164,synthetic dev-iid v1 backbone,p4_editor_v1_seed01,experiments/runs/p4_editor_v1_seed01_20261029233315/metrics.jsonl,,stopped-run-used
7 −Overfitting test passes through the workspace path,train loss 64 items budget 8,0.347->0.0024 in 200 steps,synthetic train subset,p4_overfit64_b8_seed01,experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl,,complete
8 −Sparse editor lowers held-out loss vs frozen backbone (H1 weak),dev-iid answer loss relative change by budget,see numbers.tex vtwoprismegainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_prisme,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete
9 −Global-recurrence adapter matches or exceeds the sparse editor at every budget,dev-iid answer loss by budget,see numbers.tex vtwogrgainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_gr,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete
10 −Variance of intervention gains is mostly between states,within-state share,see numbers.tex labelscalibwithin,editor-unseen label stream (calib audit),labels_v1ctrl_seed01_20261030004018,results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json,evaluation/label_analysis.py,complete
11 −One-step oracle lowers dev-iid loss by ~15% at budget 8 (diagnostic),loss relative change,see numbers.tex routingoraclebeight,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
12 −Learned controller is worse than no routing on dev-iid (H3 null),loss relative change at locked lambda,see numbers.tex routingctrllocked;vtwoctrlrel,synthetic dev-iid v1 and v2 editors,controller_v1ctrl_seed01;controller_v2s2;controller_v2s3;controller_v2s4;controller_v2s5,results/raw/routing_v1ctrl.json;results/analyzed/routing_seeds_v2.json,evaluation/routing.py,complete
13 −Fixed schedules: random = uniform = fixed pair (H2 null),loss at budget 1,see numbers.tex routing*bone,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
14 −Two-step oracle vs one-step: myopia regret small,gain nats,see numbers.tex oracletwostep;oracleonestep,synthetic dev-iid v1 editor (1 batch),controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
15 −Dispatch overhead dominates at tiny scale; dense not slower than sparse,median ms/token batch 1,see paper/tables/latency.tex,laptop MPS tiny model,p4_editor_v1_Ustd02_seed01,results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json,evaluation/latency.py,complete
16 −Miniature end-to-end pipeline runs on CPU,wall seconds,56,mini configs,mini_backbone_seed01;mini_editor_seed01;labels_mini_seed01;controller_mini_seed01,results/raw/routing_mini.json,scripts/reproduce_pilot.sh,complete
17 −103M backbone pretrained on 98.3M tokens (ppl 48.2),val loss/ppl,3.8746 / 48.16,FineWeb-Edu val,pretrain103m_seed01_20261029232706,experiments/runs/pretrain103m_seed01_20261029232706/manifest.json,training/pretrain.py,complete
18 −103M sparse editor (lr 3e-4) changes held-out loss by less than 0.1% at any budget,val loss relative change,see numbers.tex piloted*,FineWeb-Edu val (training-time eval),editor_pilot_103m_lr3e4_seed01_20261030021200,experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/metrics.jsonl,training/train_editor.py,complete
19 −103M sparse editor at lr 1e-3 collapsed to inert edits,val loss by budget,identical 3.9889,FineWeb-Edu val,editor_pilot_103m_seed01_20261030015139,experiments/runs/editor_pilot_103m_seed01_20261030015139/metrics.jsonl,,stopped-run-used
20 −103M global-recurrence adapter changes held-out loss by less than 0.1%,val ppl by budget,48.16 -> 48.01,FineWeb-Edu val,gr_pilot_103m_seed01_20261030015117,results/raw/eval_gr_103m_val.json,evaluation/run.py,complete
21 −103M controller: oracle opportunity exists (-3.6% at b8) but the controller captures none (-0.01%),val loss relative change,see numbers.tex pilotrouting*,FineWeb-Edu val,controller_103m_seed01,results/raw/routing_103m_val.json,evaluation/routing.py,complete
22 −103M latency: 8 sparse edits 3.0x budget-0 median latency; dense 3.6x; controller 1.5x,median ms/token batch 1,7.55 / 22.9 / 26.9 / 11.0,M3 Ultra MPS fp32,controller_103m_seed01,results/raw/latency_103m_mps.json,evaluation/latency.py,complete
23 −Pre-registered primary outcome: pooled test EM gain of sparse editor at b2 is +0.9 pp (< 3 pp threshold),exact match pp,see numbers.tex testprismebtwovsbzero*,synthetic locked test regimes (1000 items each),p4v2_prisme (5 seeds),results/analyzed/test_statistics.json;paper/tables/test.tex,scripts/make_results_tables.py,complete
24 −Sparse editor vs global recurrence on locked test: -0.3 pp [-0.6 +0.1]; vs dense +0.4 pp [+0.1 +0.7],exact match pp,see numbers.tex testprismevs*,synthetic locked test regimes,p4v2_prisme;p4v2_gr;p4v2_dense,results/analyzed/test_statistics.json,scripts/make_results_tables.py,complete
25 −Parameter-matched plain decoder (102.2M) beats backbone+adapters on validation text,val loss,3.8689 vs 3.8746,FineWeb-Edu val,pretrain_enlarged102m_seed01_20261030015017,experiments/runs/pretrain_enlarged102m_seed01_20261030015017/manifest.json,training/pretrain.py,complete
1 +claim,metric,value,benchmark,run_ids,artifact,analysis_script,status
2 +Backbone parameter counts equal the proposal for 103M/459M/1.18B,unique parameters,99502848;446744064;1149339648,meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete
3 +Implemented workspace+controller counts differ from the proposal by a closed-form 7w^2,unique parameters,2631173;10505221;21655813,meta instantiation,,results/tables/param_counts.json,scripts/count_params.py,complete
4 +Analytic ops table reproduces the proposal within 1.3%,ops/token at 459M T=4096,1.296;1.666;1.787;2.219;5.633 e9,analytic,,results/tables/ops_459m_T4096.json,scripts/estimate_flops.py,complete
5 +15 mandated invariants + cache equivalence pass,tests,23 passed,pytest,,tests/,python -m pytest tests,complete
6 +Zero-U bootstrap does not learn (scale mismatch),val loss by budget after 700 steps,0.4144/0.4143/0.4143/0.4146/0.4164,synthetic dev-iid v1 backbone,p4_editor_v1_seed01_20261029233624,experiments/runs/p4_editor_v1_seed01_20261029233624/metrics.jsonl,,stopped-run-used
7 +Overfitting test passes through the workspace path,train loss 64 items budget 8,0.347->0.0024 in 200 steps,synthetic train subset,p4_overfit64_b8_seed01,experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl,,complete
8 +Sparse editor lowers held-out loss vs frozen backbone (H1 weak),dev-iid answer loss relative change by budget,see numbers.tex vtwoprismegainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_prisme,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete
9 +Global-recurrence adapter matches or exceeds the sparse editor at every budget,dev-iid answer loss by budget,see numbers.tex vtwogrgainb*,synthetic dev-iid v2 backbone 5 seeds,p4v2_gr,results/analyzed/seed_matrix_v2.json,scripts/make_results_tables.py,complete
10 +Variance of intervention gains is mostly between states,within-state share,see numbers.tex labelscalibwithin,editor-unseen label stream (calib audit),labels_v1ctrl_seed01_20261030004018,results/analyzed/labels_labels_v1ctrl_seed01_20261030004018.json,evaluation/label_analysis.py,complete
11 +One-step oracle lowers dev-iid loss by ~15% at budget 8 (diagnostic),loss relative change,see numbers.tex routingoraclebeight,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
12 +Learned controller is worse than no routing on dev-iid (H3 null),loss relative change at locked lambda,see numbers.tex routingctrllocked;vtwoctrlrel,synthetic dev-iid v1 and v2 editors,controller_v1ctrl_seed01;controller_v2s2;controller_v2s3;controller_v2s4;controller_v2s5,results/raw/routing_v1ctrl.json;results/analyzed/routing_seeds_v2.json,evaluation/routing.py,complete
13 +Fixed schedules: random = uniform = fixed pair (H2 null),loss at budget 1,see numbers.tex routing*bone,synthetic dev-iid v1 editor,controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
14 +Two-step oracle vs one-step: myopia regret small,gain nats,see numbers.tex oracletwostep;oracleonestep,synthetic dev-iid v1 editor (1 batch),controller_v1ctrl_seed01,results/raw/routing_v1ctrl.json,evaluation/routing.py,complete
15 +Dispatch overhead dominates at tiny scale; dense not slower than sparse,median ms/token batch 1,see paper/tables/latency.tex,laptop MPS tiny model,p4_editor_v1_Ustd02_seed01,results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json,evaluation/latency.py,complete
16 +Miniature end-to-end pipeline runs on CPU,wall seconds,56,mini configs,mini_backbone_seed01;mini_editor_seed01;labels_mini_seed01;controller_mini_seed01,results/raw/routing_mini.json,scripts/reproduce_pilot.sh,complete
17 +103M backbone pretrained on 98.3M tokens (ppl 48.2),val loss/ppl,3.8746 / 48.16,FineWeb-Edu val,pretrain103m_seed01_20261029232706,experiments/runs/pretrain103m_seed01_20261029232706/manifest.json,training/pretrain.py,complete
18 +103M sparse editor (lr 3e-4) changes held-out loss by less than 0.1% at any budget,val loss relative change,see numbers.tex piloted*,FineWeb-Edu val (training-time eval),editor_pilot_103m_lr3e4_seed01_20261030021200,experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/metrics.jsonl,training/train_editor.py,complete
19 +103M sparse editor at lr 1e-3 collapsed to inert edits,val loss by budget,identical 3.9889,FineWeb-Edu val,editor_pilot_103m_seed01_20261030015139,experiments/runs/editor_pilot_103m_seed01_20261030015139/metrics.jsonl,,stopped-run-used
20 +103M global-recurrence adapter changes held-out loss by less than 0.1%,val ppl by budget,48.16 -> 48.01,FineWeb-Edu val,gr_pilot_103m_seed01_20261030015117,results/raw/eval_gr_103m_val.json,evaluation/run.py,complete
21 +103M controller: oracle opportunity exists (-3.6% at b8) but the controller captures none (-0.01%),val loss relative change,see numbers.tex pilotrouting*,FineWeb-Edu val,controller_103m_seed01,results/raw/routing_103m_val.json,evaluation/routing.py,complete
22 +103M latency: 8 sparse edits 3.0x budget-0 median latency; dense 3.6x; controller 1.5x,median ms/token batch 1,7.55 / 22.9 / 26.9 / 11.0,M3 Ultra MPS fp32,controller_103m_seed01,results/raw/latency_103m_mps.json,evaluation/latency.py,complete
23 +Pre-registered primary outcome: pooled test EM gain of sparse editor at b2 is +0.9 pp (< 3 pp threshold),exact match pp,see numbers.tex testprismebtwovsbzero*,synthetic locked test regimes (1000 items each),p4v2_prisme_seed01_20261030001749;p4v2_prisme_seed02_20261030001749;p4v2_prisme_seed03_20261030001749;p4v2_prisme_seed04_20261030001749;p4v2_prisme_seed05_20261030001839,results/analyzed/test_statistics.json;paper/tables/test.tex,scripts/make_results_tables.py,complete
24 +Sparse editor vs global recurrence on locked test: -0.3 pp [-0.6 +0.1]; vs dense +0.4 pp [+0.1 +0.7],exact match pp,see numbers.tex testprismevs*,synthetic locked test regimes,p4v2_prisme;p4v2_gr;p4v2_dense,results/analyzed/test_statistics.json,scripts/make_results_tables.py,complete
25 +Parameter-matched plain decoder (102.2M) beats backbone+adapters on validation text,val loss,3.8689 vs 3.8746,FineWeb-Edu val,pretrain_enlarged102m_seed01_20261030015017,experiments/runs/pretrain_enlarged102m_seed01_20261030015017/manifest.json,training/pretrain.py,complete
26 +Locked FineWeb test: backbone ppl 45.72; sparse editor and global adapter change test loss by -0.08% at budget 4; enlarged decoder lower,test ppl / relative loss,see numbers.tex pilotppl*test;pilottest*,FineWeb-Edu locked test (327680 tokens),pretrain103m_seed01_20261029232706;editor_pilot_103m_lr3e4_seed01_20261030021200;gr_pilot_103m_seed01_20261030015117;pretrain_enlarged102m_seed01_20261030015017,results/raw/eval_backbone_103m_test.json;results/raw/eval_prisme_103m_editor_test.json;results/raw/eval_gr_103m_test.json;results/raw/eval_enlarged_102m_test.json,evaluation/run.py,complete
added results/figures/exact_match_regimes.pdf +0 −0

Binary file not shown.

added results/figures/exact_match_regimes.png +0 −0

Binary file not shown.

modified results/figures/frontier_v2.pdf +0 −0

Binary file not shown.

modified results/figures/labels_analysis.pdf +0 −0

Binary file not shown.

modified results/figures/latency_103m_mps.pdf +0 −0

Binary file not shown.

modified results/figures/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.pdf +0 −0

Binary file not shown.

modified results/figures/learnability_bootstrap.pdf +0 −0

Binary file not shown.

modified results/figures/learnability_v2.pdf +0 −0

Binary file not shown.

modified results/figures/pretrain_curves.pdf +0 −0

Binary file not shown.

modified results/figures/pretrain_curves.png +0 −0

Binary file not shown.

modified results/figures/routing_103m_val.pdf +0 −0

Binary file not shown.

modified results/figures/routing_controller_p4_hent_seed01_20261030002015_dev_iid.pdf +0 −0

Binary file not shown.

modified results/figures/routing_mini.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v1ctrl.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v1rank.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v2s1.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v2s2.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v2s3.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v2s4.pdf +0 −0

Binary file not shown.

modified results/figures/routing_v2s5.pdf +0 −0

Binary file not shown.

added results/raw/eval_backbone_103m_test.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "backbone_103m",
3 + "split": "test",
4 + "checkpoint": "experiments/runs/pretrain103m_seed01_20261029232706/checkpoint",
5 + "checkpoint_sha256": "1f05af70bb9223b63dacb765c1816212ec828ff6e81bf8c0d6d46f33ebc77db2",
6 + "config_sha256": "3962409f5a94a4dcd6a9898a024ac618f1fbacb74e5f1466318b8989dcf3857b",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:15:27-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8226005017757414,
15 + "ppl": 45.722956481048506,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.8226005017757414,
20 + "ppl": 45.722956481048506,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.8226005017757414,
25 + "ppl": 45.722956481048506,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.8226005017757414,
30 + "ppl": 45.722956481048506,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.8226005017757414,
35 + "ppl": 45.722956481048506,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 38.6
40 +}
\ No newline at end of file
added results/raw/eval_backbone_103m_val.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "backbone_103m",
3 + "split": "val",
4 + "checkpoint": "experiments/runs/pretrain103m_seed01_20261029232706/checkpoint",
5 + "checkpoint_sha256": "1f05af70bb9223b63dacb765c1816212ec828ff6e81bf8c0d6d46f33ebc77db2",
6 + "config_sha256": "3962409f5a94a4dcd6a9898a024ac618f1fbacb74e5f1466318b8989dcf3857b",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:14:49-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8746298491954803,
15 + "ppl": 48.16486672694994,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.8746298491954803,
20 + "ppl": 48.16486672694994,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.8746298491954803,
25 + "ppl": 48.16486672694994,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.8746298491954803,
30 + "ppl": 48.16486672694994,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.8746298491954803,
35 + "ppl": 48.16486672694994,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 38.7
40 +}
\ No newline at end of file
added results/raw/eval_enlarged_102m_test.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "enlarged_102m",
3 + "split": "test",
4 + "checkpoint": "experiments/runs/pretrain_enlarged102m_seed01_20261030015017/checkpoint",
5 + "checkpoint_sha256": "1552f1ec984ec895a40814cd466bde59abaf35e4bfe1ea35b9d43ff4cfe9b9d0",
6 + "config_sha256": "0cc9406173e04dcb44f4facbf0dc448da684d5e7a0ba3362385d292e6a125357",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:20:19-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8207094609737395,
15 + "ppl": 45.63657420669954,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.8207094609737395,
20 + "ppl": 45.63657420669954,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.8207094609737395,
25 + "ppl": 45.63657420669954,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.8207094609737395,
30 + "ppl": 45.63657420669954,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.8207094609737395,
35 + "ppl": 45.63657420669954,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 36.8
40 +}
\ No newline at end of file
added results/raw/eval_enlarged_102m_val.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "enlarged_102m",
3 + "split": "val",
4 + "checkpoint": "experiments/runs/pretrain_enlarged102m_seed01_20261030015017/checkpoint",
5 + "checkpoint_sha256": "1552f1ec984ec895a40814cd466bde59abaf35e4bfe1ea35b9d43ff4cfe9b9d0",
6 + "config_sha256": "0cc9406173e04dcb44f4facbf0dc448da684d5e7a0ba3362385d292e6a125357",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:19:44-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.868902176618576,
15 + "ppl": 47.88978268831879,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.868902176618576,
20 + "ppl": 47.88978268831879,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.868902176618576,
25 + "ppl": 47.88978268831879,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.868902176618576,
30 + "ppl": 47.88978268831879,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.868902176618576,
35 + "ppl": 47.88978268831879,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 35.2
40 +}
\ No newline at end of file
added results/raw/eval_gr_103m_test.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "gr_103m",
3 + "split": "test",
4 + "checkpoint": "experiments/runs/gr_pilot_103m_seed01_20261030015117/checkpoint",
5 + "checkpoint_sha256": "1e3e5cdd103672376ad23bbfaafc50bcc68b937d808fae74308285a54564938a",
6 + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:16:54-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8226005017757414,
15 + "ppl": 45.722956481048506,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.8211239874362946,
20 + "ppl": 45.65549569582661,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.8202611207962036,
25 + "ppl": 45.61611808291132,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.8195126056671143,
30 + "ppl": 45.581986503991125,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.8201463341712953,
35 + "ppl": 45.61088226318195,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 45.8
40 +}
\ No newline at end of file
modified results/raw/eval_gr_103m_val.json +2 −2
@@ -7,7 +7,7 @@
7 7 "code_commit": "unknown",
8 8 "device": "mps",
9 9 "precision": "fp32",
10 − "evaluated_at": "2026-10-30T03:43:14-0400",
10 + "evaluated_at": "2026-10-30T04:16:08-0400",
11 11 "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 12 "perplexity": {
13 13 "forced_b0": {
@@ -36,5 +36,5 @@
36 36 "n_tokens": 327680
37 37 }
38 38 },
39 − "seconds": 45.8
39 + "seconds": 45.9
40 40 }
\ No newline at end of file
added results/raw/eval_prisme_103m_editor_test.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "prisme_103m_editor",
3 + "split": "test",
4 + "checkpoint": "experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/checkpoint",
5 + "checkpoint_sha256": "1203ea2e786b258268ea56204e9676a6b900d41628d9fce32af5a01ab0181ea3",
6 + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:16:30-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8226005017757414,
15 + "ppl": 45.722956481048506,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.821114146709442,
20 + "ppl": 45.65504641477478,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.8202012896537783,
25 + "ppl": 45.61338890009928,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.819450855255127,
30 + "ppl": 45.57917188444815,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.8199622869491576,
35 + "ppl": 45.60248847945214,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 86.6
40 +}
\ No newline at end of file
added results/raw/eval_prisme_103m_editor_val.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "system": "prisme_103m_editor",
3 + "split": "val",
4 + "checkpoint": "experiments/runs/editor_pilot_103m_lr3e4_seed01_20261030021200/checkpoint",
5 + "checkpoint_sha256": "1203ea2e786b258268ea56204e9676a6b900d41628d9fce32af5a01ab0181ea3",
6 + "config_sha256": "a3f0c8dc75a01e38b6eb8a9d199aa2e29f574b887bdd0e62092e2f550e7a6373",
7 + "code_commit": "unknown",
8 + "device": "mps",
9 + "precision": "fp32",
10 + "evaluated_at": "2026-10-30T04:14:49-0400",
11 + "data_sha256": "d8753bdfa9609fce56a7ec2da7c108666185d021e3f921721e4d2241e1d4fe54",
12 + "perplexity": {
13 + "forced_b0": {
14 + "loss": 3.8746298491954803,
15 + "ppl": 48.16486672694994,
16 + "n_tokens": 327680
17 + },
18 + "forced_b1": {
19 + "loss": 3.8731415927410127,
20 + "ppl": 48.093238367070285,
21 + "n_tokens": 327680
22 + },
23 + "forced_b2": {
24 + "loss": 3.872187703847885,
25 + "ppl": 48.04738463431747,
26 + "n_tokens": 327680
27 + },
28 + "forced_b4": {
29 + "loss": 3.871379178762436,
30 + "ppl": 48.00855281891424,
31 + "n_tokens": 327680
32 + },
33 + "forced_b8": {
34 + "loss": 3.871813231706619,
35 + "ppl": 48.02939559571834,
36 + "n_tokens": 327680
37 + }
38 + },
39 + "seconds": 100.4
40 +}
\ No newline at end of file
modified scripts/make_results_tables.py +53 −7
@@ -345,22 +345,29 @@ def table_pilot(macro):
345 345 macro(f"{tag}gainb{bn}", "n/a")
346 346 macro(f"{tag}minutes", "n/a"); macro(f"{tag}peakgb", "n/a"); macro(f"{tag}params", "n/a")
347 347 # perplexity table from evaluation.run outputs
348 − evs = sorted(glob.glob("results/raw/eval_*_val.json"))
348 + evs = sorted(glob.glob("results/raw/eval_*_val.json")) + sorted(glob.glob("results/raw/eval_*103m*_test.json")) + sorted(glob.glob("results/raw/eval_enlarged*_test.json"))
349 + evs = [f for f in evs if "p4v2" not in f]
349 350 if evs:
350 351 head = r"""\begin{table}[t]\centering\small
351 −\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens
352 −(40 $\times$ 16 $\times$ 512) under forced random budgets and, for PRISME, under the locked controller.}
352 +\caption{103M pilot: held-out cross-entropy (nats/token) and perplexity on 327,680 FineWeb-Edu validation tokens and
353 +on the LOCKED test split (40 $\times$ 16 $\times$ 512 tokens each) under forced random budgets. Budget 0 of the adapters
354 +equals the backbone bitwise.}
353 355 \label{tab:pilotppl}
354 −\begin{tabular}{llrr}\toprule
355 −System & policy & loss & ppl\\\midrule
356 +\begin{tabular}{lllrr}\toprule
357 +System & split & policy & loss & ppl\\\midrule
356 358 """
357 359 body = ""
358 360 for f in evs:
359 361 d = json.load(open(f))
360 362 for pol, r in d.get("perplexity", {}).items():
361 − body += f"{d['system'].replace('_', ' ')} & {pol.replace('_', ' ')} & {r['loss']:.4f} & {r['ppl']:.2f}\\\\\n"
363 + if pol == "controller":
364 + continue
365 + body += f"{d['system'].replace('_', ' ')} & {d['split']} & {pol.replace('_', ' ')} & {r['loss']:.4f} & {r['ppl']:.2f}\\\\\n"
362 366 if pol == "forced_b0":
363 − macro("pilotppl" + re.sub(r"[^a-z]", "", d["system"]), f"{r['ppl']:.2f}")
367 + macro("pilotppl" + re.sub(r"[^a-z]", "", d["system"]) + d["split"], f"{r['ppl']:.2f}")
368 + if d["split"] == "test" and pol in ("forced_b2", "forced_b4"):
369 + b0 = d["perplexity"]["forced_b0"]["loss"]
370 + macro("pilottest" + re.sub(r"[^a-z]", "", d["system"]) + {"forced_b2": "btwo", "forced_b4": "bfour"}[pol], f"{100*(r['loss']/b0-1):+.2f}")
364 371 (OUT / "pilot_ppl.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n")
365 372 rt = Path("results/raw/routing_103m_val.json")
366 373 if rt.exists():
@@ -485,3 +492,42 @@ System & budget & iid & template & depth & compose & pooled\\\midrule
485 492 _old_make_all4 = make_all
486 493 def make_all(macro): # noqa: F811
487 494 _old_make_all4(macro); table_test(macro)
495 +
496 +
497 +def table_controls(macro):
498 + import torch
499 + from training.train_editor import build_model
500 + from training.common import load_yaml
501 + rows = []
502 + def count(exp, label, data, compute, regime):
503 + cfg, m = build_model(exp)
504 + nb = sum(p.numel() for p in m.backbone.parameters()); tot = sum(p.numel() for p in m.parameters())
505 + rows.append((label, nb, tot - nb, data, compute, regime))
506 + with torch.device("meta"):
507 + count(load_yaml("configs/experiments/p4v2_prisme.yaml"), "Frozen backbone v2 (budget 0)", "synthetic train, 5k steps $\\times$ 64", "backbone only", "no workspace (bitwise bypass)")
508 + count(load_yaml("configs/experiments/p4v2_prisme.yaml"), "PRISME sparse editor (tiny)", "same + 3k adapter steps $\\times$ 32", "frozen bb fwd + editor", "1--2 slots/action, 0--8 actions")
509 + count(load_yaml("configs/experiments/p4v2_gr.yaml"), "Global-recurrence adapter (tiny)", "same", "frozen bb fwd + block", "whole latent/action, 0--8 actions")
510 + count(load_yaml("configs/experiments/p4v2_dense.yaml"), "Dense 16-slot editor (tiny)", "same", "frozen bb fwd + 16 lanes", "16 slots/action, 0--8 actions")
511 + e = load_yaml("configs/experiments/editor_pilot_103m_lr3e4.yaml"); e["init"] = {}
512 + count(e, "PRISME-103M sparse editor", "FineWeb-Edu 98.3M tok + 24.6M adapter tok", "$6NP$ + editor", "1--2 slots/action")
513 + g = load_yaml("configs/experiments/gr_pilot_103m.yaml"); g["init"] = {}
514 + count(g, "Global-recurrence adapter (103M)", "same", "$6NP$ + block", "whole latent/action")
515 + count({"model": "configs/baselines/transformer_enlarged_102m.yaml", "mode": "backbone_only"}, "Plain decoder, parameter-matched (102.2M)", "FineWeb-Edu 98.3M tok (same recipe)", "$6NP$", "no workspace")
516 + with open(OUT / "controls.tex", "w") as f:
517 + f.write(r"""\begin{table}[t]\centering\small
518 +\caption{Systems and controls. ``bb'' = backbone, ``$6NP$'' = the usual training-FLOP estimate for $N$ tokens and $P$
519 +parameters; adapter training additionally executes up to 8 editor applications per position. Adapter parameters are
520 +unique parameters outside the backbone (counted by instantiation; the 103M editor includes the residual-plus-entropy
521 +controller features).}
522 +\label{tab:controls}
523 +\begin{tabular}{lrrlll}\toprule
524 +System & backbone & adapter & training exposure & training compute & inference regime\\\midrule
525 +""")
526 + for lab, nb, ad, data, comp, reg in rows:
527 + f.write(f"{lab} & {nb:,} & {ad:,} & {data} & {comp} & {reg}\\\\\n")
528 + f.write(r"\bottomrule\end{tabular}\end{table}" + "\n")
529 +
530 +
531 +_old_make_all5 = make_all
532 +def make_all(macro): # noqa: F811
533 + _old_make_all5(macro); table_controls(macro)
488 534
Menu

Theme