WP31: 5-seed routing summary, statistics, ranking controller result

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 4 days ago (Oct 30, 2026) parent 512b980 Browse files

26 changed files +1,850 −34

modified FAILURES.md +1 −0

		`@@ -10,3 +10,4 @@`
10	10	\| 2026-10-30 \| Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. \| M4M64a `experiments/logs/p4_joint_mixed_w64.log` \| Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). \|
11	11	\| 2026-10-30 \| v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. \| `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) \| 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). \|
12	12	\| 2026-10-30 \| Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). \| manifests remain in `experiments/runs/` on the nodes (state: running → killed) \| — \|
	13	+\| 2026-10-30 \| Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 %, captured fraction −8 %; on dev-iid the locked policy (λ=0.3, κ=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); λ=0 is worse (+1.3 %). \| `experiments/runs/controller_v1rank_seed01_*`, `results/raw/routing_v1rank.json` \| Within-state ranking is not learnable from these labels at this scale; H3 null stands. \|

modified PROGRESS.md +3 −1

		`@@ -9,9 +9,11 @@`
9	9	\| 2026-10-29 \| 4 learnability (backbone) \| done \| `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) \| exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) \| PASS (weak backbone, usable comparator) \| stronger v2 backbone (5.6M, 15k steps) running \|
10	10	\| 2026-10-29 \| 4 learnability (overfit) \| done \| `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` \| 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) \| PASS (gradient path + capacity) \| — \|
11	11	\| 2026-10-29 \| 4 learnability (generalisation, v1 backbone) \| done \| zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) \| after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) \| PASS (weak, learnable; mechanism effect ≈ 1 %) \| decisive 5-seed matrix on v2 backbone \|
12		−\| 2026-10-30 \| 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence, dense}) \| running \| `scripts/launch_seed_matrix.sh` → 15 runs on M2U64/M4M64c/M4M64a/M5M36/M4BP48/M4BP36 \| — \| — \| aggregate with `scripts/make_tables.py` \|
	12	+\| 2026-10-30 \| 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence}; dense running) \| done (PRISME, GR) \| `scripts/launch_seed_matrix.sh`; `scripts/statistics.py` → `results/analyzed/{seed_matrix_v2,statistics_v2}.json` \| dev-iid loss vs budget 0 (0.4621): PRISME −2.1/−2.8/−3.0/−1.0 % at b=1/2/4/8 (seed SD ≤ 0.003); GR −3.6/−4.1/−3.7/−3.3 %; EM dev-iid 37.7 → PRISME 39.6 (b2, +1.9 pp, t-CI [1.6, 2.2]) vs GR 39.6 (b2) / 40.0 (b8); PRISME − GR at b2 +0.06 pp [−0.53, +0.65]; OOD regimes unchanged (≤ 1 pp) \| H1 weak PASS (< 3 pp prereg minimum); sparse ≤ global at every budget \| dense seeds pending (M4BP36 lost; relaunched) \|
	13	+\| 2026-10-30 \| 6 controller on v2 editors (4 seeds) \| done \| `scripts/run_stage_cd.sh` → `results/analyzed/routing_seeds_v2.json` \| locked controller: dev-iid loss +0.5 % ± 1.1 vs budget 0 (0.05–4.3 edits/token); best fixed schedule −4.0 %; oracle b8 −15.0 %; calib sign acc 0.50–0.63, captured 0.30–0.52 on calib audit but no transfer to dev rollouts \| H3 null replicated across seeds \| ranking-loss variant (D12) \|
13	14	\| 2026-10-30 \| 5 counterfactual labels + oracle audit (tiny) \| done \| `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` \| gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 \| PASS (oracle margin exists; action choice is ≈40 % of it) \| controller fit \|
14	15	\| 2026-10-30 \| 6 controller (tiny) \| partial \| `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) \| controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity \| OPEN (H3 null so far at tiny scale) \| rerun with early stopping; more labels \|
15	16	\| 2026-10-30 \| 7 inference engine + latency (tiny) \| done \| `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` \| batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits \| PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) \| re-measure at 103M \|
16	17	\| 2026-10-29 \| 8 pilot (Stage A) \| running \| M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` \| 14.8k tok/s, 196.6M tokens planned (~3.7 h each) \| — \| editor/controller stages after completion \|
17	18	\| 2026-10-30 \| 5/6 routing evaluation (tiny v1 editor, dev-iid, 640 items / 3,417 answer tokens) \| done \| `evaluation.routing` → `results/raw/routing_controller_p4_hent_seed01_20261030002015_dev_iid.json` \| budget 0: 0.4334; uniform b1 0.4301 (−0.8 %), random b1 0.4305, fixed pair b1 0.4306; b8 schedules +0.6 to +2.8 %; entropy router ≥ budget 0; norm router (thr 0.2) 0.4309; learned controller (locked λ=0.3) 0.4404 (+1.6 %) and worse at every λ; greedy one-step oracle 0.4025 (b1) → 0.3666 (b8, 2.26 edits/token, −15 %); two-step oracle gain 0.034 vs one-step 0.0233 vs greedy-then-best 0.0337 (myopia regret ≈ 0) \| H2 null (uniform = random = pair); H3 null (controller < no routing on dev) at this scale; oracle opportunity large \| labels from a split unseen by the editor (D11); 5-seed v2 matrix \|
	19	+\| 2026-10-30 \| 6 ranking-loss controller (D12, tiny v1 editor, editor-unseen labels) \| done \| `controller_v1rank.yaml` → `results/raw/routing_v1rank.json` \| locked λ=0.3 κ=0.5: dev-iid loss 0.4333 vs 0.4334 (0.07 edits/token); λ=0: +1.3 % \| H3 null stands \| — \|

modified configs/experiments/routing_v1rank.yaml +1 −1

		`@@ -1,7 +1,7 @@`
1	1	seed: 1
2	2	mode: prisme
3	3	model: from_checkpoint
4		−checkpoint: CONTROLLER_CKPT
	4	+checkpoint: experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint
5	5	data:
6	6	kind: synthetic
7	7	tokenizer: char

modified experiments/logs/controller_v1rank.log +10 −0

		`@@ -33,3 +33,13 @@ step 1999 loss 3.74421 gain 0.00241 harm 0.5247`
33	33	restored controller from step 750 (calibration score -0.01242)
34	34	sweep {'lambda': 0.0, 'kappa': 0.0, 'loss': 0.44029377250626134, 'edits_per_token': 3.0255041518386716, 'extra_ops_ratio': 0.9985018783334029}
35	35	sweep {'lambda': 0.0, 'kappa': 0.5, 'loss': 0.43276796307003795, 'edits_per_token': 1.5634638196915778, 'extra_ops_ratio': 0.5107896924725914}
	36	+sweep {'lambda': 0.01, 'kappa': 0.0, 'loss': 0.4399834968865554, 'edits_per_token': 2.7941874258600237, 'extra_ops_ratio': 0.9065133996422746}
	37	+sweep {'lambda': 0.01, 'kappa': 0.5, 'loss': 0.4311416666702868, 'edits_per_token': 1.3849347568208779, 'extra_ops_ratio': 0.44641059432024066}
	38	+sweep {'lambda': 0.03, 'kappa': 0.0, 'loss': 0.4382320236622368, 'edits_per_token': 2.3718861209964412, 'extra_ops_ratio': 0.7466290384566374}
	39	+sweep {'lambda': 0.03, 'kappa': 0.5, 'loss': 0.43146511272462096, 'edits_per_token': 1.120403321470937, 'extra_ops_ratio': 0.3504910203202905}
	40	+sweep {'lambda': 0.1, 'kappa': 0.0, 'loss': 0.43422956681619196, 'edits_per_token': 1.2627520759193358, 'extra_ops_ratio': 0.371382664672562}
	41	+sweep {'lambda': 0.1, 'kappa': 0.5, 'loss': 0.4304978969116935, 'edits_per_token': 0.5195729537366548, 'extra_ops_ratio': 0.15240548768501688}
	42	+sweep {'lambda': 0.3, 'kappa': 0.0, 'loss': 0.4298561159543437, 'edits_per_token': 0.3309608540925267, 'extra_ops_ratio': 0.09634915280596641}
	43	+sweep {'lambda': 0.3, 'kappa': 0.5, 'loss': 0.4300144485165923, 'edits_per_token': 0.11684460260972716, 'extra_ops_ratio': 0.034015740434752945}
	44	+selected {'lambda': 0.3, 'kappa': 0.5, 'loss': 0.4300144485165923, 'edits_per_token': 0.11684460260972716, 'extra_ops_ratio': 0.034015740434752945, 'loss_reduction': 0.001048727250466841, 'reduction_per_ops': 0.03083064596163796}
	45	+run controller_v1rank_seed01_20261030005431

added experiments/logs/routing_v1rank.log +36 −0

		`@@ -0,0 +1,36 @@`
	1	+{"policy": "budget0", "loss": 0.43343788690653534, "edits_per_token": 0.0, "extra_ops_ratio": 0.0, "n_tokens": 3417}
	2	+/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:57: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.)
	3	+ return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device)
	4	+{"policy": "random_forced", "budget": 1, "loss": 0.43049107206331766, "edits_per_token": 1.0, "extra_ops_ratio": 0.33739269034430613, "n_tokens": 3417}
	5	+{"policy": "uniform", "budget": 1, "loss": 0.430082754325476, "edits_per_token": 1.0, "extra_ops_ratio": 0.2911194881431394, "n_tokens": 3417}
	6	+{"policy": "fixed_pair", "budget": 1, "loss": 0.4306084942950259, "edits_per_token": 1.0, "extra_ops_ratio": 0.38326141294904154, "n_tokens": 3417}
	7	+{"policy": "random_forced", "budget": 2, "loss": 0.43129151572879726, "edits_per_token": 2.0, "extra_ops_ratio": 0.6752437784687831, "n_tokens": 3417}
	8	+{"policy": "uniform", "budget": 2, "loss": 0.43061076446970825, "edits_per_token": 2.0, "extra_ops_ratio": 0.5822389762862789, "n_tokens": 3417}
	9	+{"policy": "fixed_pair", "budget": 2, "loss": 0.43226337237493456, "edits_per_token": 2.0, "extra_ops_ratio": 0.7665228258980831, "n_tokens": 3417}
	10	+{"policy": "random_forced", "budget": 4, "loss": 0.4350665061488499, "edits_per_token": 4.0, "extra_ops_ratio": 1.3500830851558498, "n_tokens": 3417}
	11	+{"policy": "uniform", "budget": 4, "loss": 0.43226772879940184, "edits_per_token": 4.0, "extra_ops_ratio": 1.1644779525725577, "n_tokens": 3417}
	12	+{"policy": "fixed_pair", "budget": 4, "loss": 0.43608775067685257, "edits_per_token": 4.0, "extra_ops_ratio": 1.5330456517961661, "n_tokens": 3417}
	13	+{"policy": "random_forced", "budget": 8, "loss": 0.44534724575753865, "edits_per_token": 8.0, "extra_ops_ratio": 2.6992224028210283, "n_tokens": 3417}
	14	+{"policy": "uniform", "budget": 8, "loss": 0.4360636224654602, "edits_per_token": 8.0, "extra_ops_ratio": 2.3289559051451154, "n_tokens": 3417}
	15	+{"policy": "fixed_pair", "budget": 8, "loss": 0.4454028259636821, "edits_per_token": 8.0, "extra_ops_ratio": 3.0660913035923323, "n_tokens": 3417}
	16	+{"policy": "entropy", "threshold": 0.1, "budget": 8, "loss": 0.44345039856653656, "edits_per_token": 2.893473807433421, "extra_ops_ratio": 0.9313065950585138, "n_tokens": 3417}
	17	+{"policy": "entropy", "threshold": 0.5, "budget": 8, "loss": 0.44244280132174246, "edits_per_token": 2.097453906935909, "extra_ops_ratio": 0.6750847836283169, "n_tokens": 3417}
	18	+{"policy": "entropy", "threshold": 1.0, "budget": 8, "loss": 0.4375149400603328, "edits_per_token": 1.1785191688615744, "extra_ops_ratio": 0.3794397201180842, "n_tokens": 3417}
	19	+{"policy": "entropy", "threshold": 2.0, "budget": 8, "loss": 0.43457157094819804, "edits_per_token": 0.20749195200468246, "extra_ops_ratio": 0.06636437945523596, "n_tokens": 3417}
	20	+{"policy": "norm", "threshold": 0.02, "budget": 8, "loss": 0.4431566372928056, "edits_per_token": 7.994146912496342, "extra_ops_ratio": 2.576118717076383, "n_tokens": 3417}
	21	+{"policy": "norm", "threshold": 0.05, "budget": 8, "loss": 0.4419520246658124, "edits_per_token": 7.054141059408838, "extra_ops_ratio": 2.2736114108984604, "n_tokens": 3417}
	22	+{"policy": "norm", "threshold": 0.1, "budget": 8, "loss": 0.4397261778374042, "edits_per_token": 4.717588527948493, "extra_ops_ratio": 1.5199677263884506, "n_tokens": 3417}
	23	+{"policy": "norm", "threshold": 0.2, "budget": 8, "loss": 0.430901125784258, "edits_per_token": 1.0491659350307287, "extra_ops_ratio": 0.33984092665514054, "n_tokens": 3417}
	24	+{"policy": "controller_locked", "lam": 0.3, "kap": 0.5, "budget": 8, "loss": 0.43332059026845965, "edits_per_token": 0.06613988879133743, "extra_ops_ratio": 0.01925461050499103, "n_tokens": 3417}
	25	+{"policy": "controller", "lam": 0.0, "kap": 0.5, "budget": 8, "loss": 0.4392198074760582, "edits_per_token": 1.4641498390400935, "extra_ops_ratio": 0.48109084149707576, "n_tokens": 3417}
	26	+{"policy": "controller", "lam": 0.01, "kap": 0.5, "budget": 8, "loss": 0.43936958569757484, "edits_per_token": 1.3075797483172373, "extra_ops_ratio": 0.42456216117182355, "n_tokens": 3417}
	27	+{"policy": "controller", "lam": 0.03, "kap": 0.5, "budget": 8, "loss": 0.438084998115448, "edits_per_token": 1.0398009950248757, "extra_ops_ratio": 0.3312630400798598, "n_tokens": 3417}
	28	+{"policy": "controller", "lam": 0.1, "kap": 0.5, "budget": 8, "loss": 0.4354970293991183, "edits_per_token": 0.4433713784021071, "extra_ops_ratio": 0.13166276062654758, "n_tokens": 3417}
	29	+{"policy": "controller", "lam": 0.3, "kap": 0.5, "budget": 8, "loss": 0.43332059026845965, "edits_per_token": 0.06613988879133743, "extra_ops_ratio": 0.01925461050499103, "n_tokens": 3417}
	30	+{"policy": "oracle1_greedy", "budget": 1, "diagnostic": true, "loss": 0.4025459464804867, "edits_per_token": 0.5237366003062787, "extra_ops_ratio": 0.19226168855956283, "n_tokens": 653}
	31	+{"policy": "oracle1_greedy", "budget": 2, "diagnostic": true, "loss": 0.390496985653093, "edits_per_token": 0.9509954058192955, "extra_ops_ratio": 0.34655941646277993, "n_tokens": 653}
	32	+{"policy": "oracle1_greedy", "budget": 4, "diagnostic": true, "loss": 0.37995706181431255, "edits_per_token": 1.5313935681470139, "extra_ops_ratio": 0.552776494398599, "n_tokens": 653}
	33	+{"policy": "oracle1_greedy", "budget": 8, "diagnostic": true, "loss": 0.3665898426751083, "edits_per_token": 2.2664624808575806, "extra_ops_ratio": 0.8196839220125861, "n_tokens": 653}
	34	+{"policy": "oracle_two_step", "diagnostic": true, "one_step_gain": 0.023339285247627347, "two_step_gain": 0.03395615775009681, "greedy_then_best_gain": 0.03374234835306803, "n_tokens": 174}
	35	+wrote results/raw/routing_v1rank.json
	36	+RANK_DONE

added experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint/checkpoint.json +49 −0

		`@@ -0,0 +1,49 @@`
	1	+{
	2	+ "config": {
	3	+ "name": "tiny_p4",
	4	+ "vocab_size": 128,
	5	+ "d_model": 192,
	6	+ "n_layers": 4,
	7	+ "n_heads": 6,
	8	+ "n_kv_heads": 2,
	9	+ "ffn_width": 512,
	10	+ "max_seq_len": 320,
	11	+ "rope_theta": 10000.0,
	12	+ "norm_eps": 1e-05,
	13	+ "tie_embeddings": true,
	14	+ "init_std": 0.02,
	15	+ "use_workspace": true,
	16	+ "w": 64,
	17	+ "ffn_w": 192,
	18	+ "n_slots": 16,
	19	+ "n_editor_blocks": 2,
	20	+ "n_editor_heads": 2,
	21	+ "r_max": 8,
	22	+ "eta_max": 0.25,
	23	+ "clip_c": 1.0,
	24	+ "clip_eps": 1e-06,
	25	+ "readout_zero_init": false,
	26	+ "readout_init_std": 0.2,
	27	+ "readout_scale": 5.251033782958984,
	28	+ "controller_hidden": 0,
	29	+ "ctrl_input": "h",
	30	+ "ctrl_use_entropy": true,
	31	+ "lambda_cost": 0.3,
	32	+ "kappa_risk": 0.5,
	33	+ "dense_writes": false
	34	+ },
	35	+ "saved_at": "2026-10-30T00:57:23-0400",
	36	+ "run_id": "controller_v1rank_seed01_20261030005431",
	37	+ "parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint",
	38	+ "selected": {
	39	+ "lambda": 0.3,
	40	+ "kappa": 0.5,
	41	+ "loss": 0.4300144485165923,
	42	+ "edits_per_token": 0.11684460260972716,
	43	+ "extra_ops_ratio": 0.034015740434752945,
	44	+ "loss_reduction": 0.001048727250466841,
	45	+ "reduction_per_ops": 0.03083064596163796
	46	+ },
	47	+ "locked": true,
	48	+ "weights_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9"
	49	+}
		\ No newline at end of file

modified experiments/runs/controller_v1rank_seed01_20261030005431/manifest.json +136 −3

		`@@ -9,10 +9,143 @@`
9	9	"python": "3.12.13",
10	10	"torch": "2.15.0",
11	11	"command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_controller.py --config configs/experiments/controller_v1rank.yaml --device mps",
12		− "state": "running",
13		− "wall_seconds": 0.02,
	12	+ "state": "completed",
	13	+ "wall_seconds": 171.86,
14	14	"device": "mps",
15	15	"checkpoint_parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint",
16	16	"labels_train": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_train.pt",
17		− "labels_calib": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_calib.pt"
	17	+ "labels_calib": "experiments/runs/labels_v1ctrl_seed01_20261030004018/labels_calib.pt",
	18	+ "finished_at": "2026-10-30T00:57:23-0400",
	19	+ "eval_train": {
	20	+ "huber": 0.0022109991413126954,
	21	+ "sign_acc": 0.5578782919208451,
	22	+ "harm_brier": 0.18087974557421105,
	23	+ "harm_ece": 0.05984468385577202,
	24	+ "n_labels": 26884,
	25	+ "oracle_opportunity_per_state": 0.015146345818821804,
	26	+ "regret_per_state": 0.012184017977496993,
	27	+ "captured_fraction": 0.19558036484573305,
	28	+ "top1_match": 0.05240443896424168,
	29	+ "n_audit_states": 1622
	30	+ },
	31	+ "eval_calib": {
	32	+ "huber": 0.002075914168963728,
	33	+ "sign_acc": 0.5701813048001189,
	34	+ "harm_brier": 0.18367661889833475,
	35	+ "harm_ece": 0.06215643882751465,
	36	+ "n_labels": 26916,
	37	+ "oracle_opportunity_per_state": 0.013603728955277556,
	38	+ "regret_per_state": 0.012418674625524272,
	39	+ "captured_fraction": 0.08711246259383487,
	40	+ "top1_match": 0.06930091185410335,
	41	+ "n_audit_states": 1645
	42	+ },
	43	+ "sweep": [
	44	+ {
	45	+ "lambda": 0.0,
	46	+ "kappa": 0.0,
	47	+ "loss": 0.44029377250626134,
	48	+ "edits_per_token": 3.0255041518386716,
	49	+ "extra_ops_ratio": 0.9985018783334029,
	50	+ "loss_reduction": -0.00923059673920218,
	51	+ "reduction_per_ops": -0.00924444604411656
	52	+ },
	53	+ {
	54	+ "lambda": 0.0,
	55	+ "kappa": 0.5,
	56	+ "loss": 0.43276796307003795,
	57	+ "edits_per_token": 1.5634638196915778,
	58	+ "extra_ops_ratio": 0.5107896924725914,
	59	+ "loss_reduction": -0.001704787302978794,
	60	+ "reduction_per_ops": -0.003337552280521541
	61	+ },
	62	+ {
	63	+ "lambda": 0.01,
	64	+ "kappa": 0.0,
	65	+ "loss": 0.4399834968865554,
	66	+ "edits_per_token": 2.7941874258600237,
	67	+ "extra_ops_ratio": 0.9065133996422746,
	68	+ "loss_reduction": -0.008920321119496255,
	69	+ "reduction_per_ops": -0.009840252910785835
	70	+ },
	71	+ {
	72	+ "lambda": 0.01,
	73	+ "kappa": 0.5,
	74	+ "loss": 0.4311416666702868,
	75	+ "edits_per_token": 1.3849347568208779,
	76	+ "extra_ops_ratio": 0.44641059432024066,
	77	+ "loss_reduction": -7.849090322764596e-05,
	78	+ "reduction_per_ops": -0.00017582670354668846
	79	+ },
	80	+ {
	81	+ "lambda": 0.03,
	82	+ "kappa": 0.0,
	83	+ "loss": 0.4382320236622368,
	84	+ "edits_per_token": 2.3718861209964412,
	85	+ "extra_ops_ratio": 0.7466290384566374,
	86	+ "loss_reduction": -0.0071688478951776324,
	87	+ "reduction_per_ops": -0.009601619446782316
	88	+ },
	89	+ {
	90	+ "lambda": 0.03,
	91	+ "kappa": 0.5,
	92	+ "loss": 0.43146511272462096,
	93	+ "edits_per_token": 1.120403321470937,
	94	+ "extra_ops_ratio": 0.3504910203202905,
	95	+ "loss_reduction": -0.00040193695756179926,
	96	+ "reduction_per_ops": -0.0011467824687619549
	97	+ },
	98	+ {
	99	+ "lambda": 0.1,
	100	+ "kappa": 0.0,
	101	+ "loss": 0.43422956681619196,
	102	+ "edits_per_token": 1.2627520759193358,
	103	+ "extra_ops_ratio": 0.371382664672562,
	104	+ "loss_reduction": -0.0031663910491327996,
	105	+ "reduction_per_ops": -0.008525952744521666
	106	+ },
	107	+ {
	108	+ "lambda": 0.1,
	109	+ "kappa": 0.5,
	110	+ "loss": 0.4304978969116935,
	111	+ "edits_per_token": 0.5195729537366548,
	112	+ "extra_ops_ratio": 0.15240548768501688,
	113	+ "loss_reduction": 0.0005652788553656696,
	114	+ "reduction_per_ops": 0.0037090452840776718
	115	+ },
	116	+ {
	117	+ "lambda": 0.3,
	118	+ "kappa": 0.0,
	119	+ "loss": 0.4298561159543437,
	120	+ "edits_per_token": 0.3309608540925267,
	121	+ "extra_ops_ratio": 0.09634915280596641,
	122	+ "loss_reduction": 0.0012070598127154608,
	123	+ "reduction_per_ops": 0.012527975364208016
	124	+ },
	125	+ {
	126	+ "lambda": 0.3,
	127	+ "kappa": 0.5,
	128	+ "loss": 0.4300144485165923,
	129	+ "edits_per_token": 0.11684460260972716,
	130	+ "extra_ops_ratio": 0.034015740434752945,
	131	+ "loss_reduction": 0.001048727250466841,
	132	+ "reduction_per_ops": 0.03083064596163796
	133	+ }
	134	+ ],
	135	+ "baseline_budget0": {
	136	+ "loss": 0.43106317576705916,
	137	+ "edits_per_token": 0.0,
	138	+ "extra_ops_ratio": 0.0
	139	+ },
	140	+ "selected": {
	141	+ "lambda": 0.3,
	142	+ "kappa": 0.5,
	143	+ "loss": 0.4300144485165923,
	144	+ "edits_per_token": 0.11684460260972716,
	145	+ "extra_ops_ratio": 0.034015740434752945,
	146	+ "loss_reduction": 0.001048727250466841,
	147	+ "reduction_per_ops": 0.03083064596163796
	148	+ },
	149	+ "checkpoint_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9",
	150	+ "selection_rule": "max_loss_reduction_per_ops"
18	151	}
		\ No newline at end of file

added experiments/runs/controller_v2s1_seed01_20261030005758/checkpoint/checkpoint.json +49 −0

		`@@ -0,0 +1,49 @@`
	1	+{
	2	+ "config": {
	3	+ "name": "tiny_p4v2",
	4	+ "vocab_size": 128,
	5	+ "d_model": 256,
	6	+ "n_layers": 6,
	7	+ "n_heads": 8,
	8	+ "n_kv_heads": 4,
	9	+ "ffn_width": 768,
	10	+ "max_seq_len": 320,
	11	+ "rope_theta": 10000.0,
	12	+ "norm_eps": 1e-05,
	13	+ "tie_embeddings": true,
	14	+ "init_std": 0.02,
	15	+ "use_workspace": true,
	16	+ "w": 64,
	17	+ "ffn_w": 192,
	18	+ "n_slots": 16,
	19	+ "n_editor_blocks": 2,
	20	+ "n_editor_heads": 2,
	21	+ "r_max": 8,
	22	+ "eta_max": 0.25,
	23	+ "clip_c": 1.0,
	24	+ "clip_eps": 1e-06,
	25	+ "readout_zero_init": false,
	26	+ "readout_init_std": 0.2,
	27	+ "readout_scale": 38.47460174560547,
	28	+ "controller_hidden": 0,
	29	+ "ctrl_input": "h",
	30	+ "ctrl_use_entropy": true,
	31	+ "lambda_cost": 0.3,
	32	+ "kappa_risk": 0.5,
	33	+ "dense_writes": false
	34	+ },
	35	+ "saved_at": "2026-10-30T01:00:19-0400",
	36	+ "run_id": "controller_v2s1_seed01_20261030005758",
	37	+ "parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
	38	+ "selected": {
	39	+ "lambda": 0.3,
	40	+ "kappa": 0.5,
	41	+ "loss": 0.48617894680615986,
	42	+ "edits_per_token": 0.15361803084223014,
	43	+ "extra_ops_ratio": 0.016404977543356864,
	44	+ "loss_reduction": 0.001227108467798943,
	45	+ "reduction_per_ops": 0.07480098430831782
	46	+ },
	47	+ "locked": true,
	48	+ "weights_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198"
	49	+}
		\ No newline at end of file

added experiments/runs/controller_v2s1_seed01_20261030005758/config.yaml +67 −0

		`@@ -0,0 +1,67 @@`
	1	+experiment:
	2	+ name: controller_v2s1
	3	+ seed: 1
	4	+ mode: prisme
	5	+ model: from_checkpoint
	6	+ checkpoint: experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint
	7	+ labels:
	8	+ train: experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt
	9	+ calib: experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt
	10	+ train:
	11	+ steps: 2000
	12	+ lr: 0.001
	13	+ weight_decay: 0.01
	14	+ alpha: 1.0
	15	+ beta: 0.5
	16	+ gamma: 0.1
	17	+ huber_delta: 0.1
	18	+ log_every: 100
	19	+ eval_every: 250
	20	+ seq_len: 320
	21	+ lambda_grid:
	22	+ - 0.0
	23	+ - 0.01
	24	+ - 0.03
	25	+ - 0.1
	26	+ - 0.3
	27	+ kappa_grid:
	28	+ - 0.0
	29	+ - 0.5
	30	+ sweep_batches: 10
	31	+ max_budget: 8
	32	+ selection_rule: max_loss_reduction_per_ops
	33	+ model_overrides:
	34	+ ctrl_input: h
	35	+ ctrl_use_entropy: true
	36	+model:
	37	+ name: tiny_p4v2
	38	+ vocab_size: 128
	39	+ d_model: 256
	40	+ n_layers: 6
	41	+ n_heads: 8
	42	+ n_kv_heads: 4
	43	+ ffn_width: 768
	44	+ max_seq_len: 320
	45	+ rope_theta: 10000.0
	46	+ norm_eps: 1.0e-05
	47	+ tie_embeddings: true
	48	+ init_std: 0.02
	49	+ use_workspace: true
	50	+ w: 64
	51	+ ffn_w: 192
	52	+ n_slots: 16
	53	+ n_editor_blocks: 2
	54	+ n_editor_heads: 2
	55	+ r_max: 8
	56	+ eta_max: 0.25
	57	+ clip_c: 1.0
	58	+ clip_eps: 1.0e-06
	59	+ readout_zero_init: false
	60	+ readout_init_std: 0.2
	61	+ readout_scale: 38.47460174560547
	62	+ controller_hidden: 0
	63	+ ctrl_input: h
	64	+ ctrl_use_entropy: true
	65	+ lambda_cost: 0.0
	66	+ kappa_risk: 0.0
	67	+ dense_writes: false

added experiments/runs/controller_v2s1_seed01_20261030005758/manifest.json +151 −0

		`@@ -0,0 +1,151 @@`
	1	+{
	2	+ "run_id": "controller_v2s1_seed01_20261030005758",
	3	+ "name": "controller_v2s1",
	4	+ "seed": 1,
	5	+ "code_commit": "unknown",
	6	+ "config_sha256": "1b270868a323af3e33c1bf1c047828f7743cb5844138936bbaee52634435e230",
	7	+ "started_at": "2026-10-30T00:57:58-0400",
	8	+ "host": "MacStudonpierre",
	9	+ "python": "3.12.14",
	10	+ "torch": "2.15.0",
	11	+ "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/train_controller.py --config configs/experiments/controller_v2s1.yaml --device mps",
	12	+ "state": "completed",
	13	+ "wall_seconds": 141.05,
	14	+ "device": "mps",
	15	+ "checkpoint_parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
	16	+ "labels_train": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt",
	17	+ "labels_calib": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt",
	18	+ "finished_at": "2026-10-30T01:00:19-0400",
	19	+ "eval_train": {
	20	+ "huber": 0.0011783403564361462,
	21	+ "sign_acc": 0.6147150721618807,
	22	+ "harm_brier": 0.15530524830647194,
	23	+ "harm_ece": 0.031277772039175034,
	24	+ "n_labels": 26884,
	25	+ "oracle_opportunity_per_state": 0.01902814696367219,
	26	+ "regret_per_state": 0.01239542852347641,
	27	+ "captured_fraction": 0.3485740599365095,
	28	+ "top1_match": 0.06288532675709001,
	29	+ "n_audit_states": 1622
	30	+ },
	31	+ "eval_calib": {
	32	+ "huber": 0.0011220869855551932,
	33	+ "sign_acc": 0.6217491454896715,
	34	+ "harm_brier": 0.15860497322598402,
	35	+ "harm_ece": 0.02261553332209587,
	36	+ "n_labels": 26916,
	37	+ "oracle_opportunity_per_state": 0.01836320621989056,
	38	+ "regret_per_state": 0.010739420540064664,
	39	+ "captured_fraction": 0.41516637065089457,
	40	+ "top1_match": 0.06504559270516717,
	41	+ "n_audit_states": 1645
	42	+ },
	43	+ "sweep": [
	44	+ {
	45	+ "lambda": 0.0,
	46	+ "kappa": 0.0,
	47	+ "loss": 0.4843786382166092,
	48	+ "edits_per_token": 3.912811387900356,
	49	+ "extra_ops_ratio": 0.49989572029521034,
	50	+ "loss_reduction": 0.0030274170573496018,
	51	+ "reduction_per_ops": 0.00605609717074949
	52	+ },
	53	+ {
	54	+ "lambda": 0.0,
	55	+ "kappa": 0.5,
	56	+ "loss": 0.478396611422972,
	57	+ "edits_per_token": 2.082443653618031,
	58	+ "extra_ops_ratio": 0.24916935193298267,
	59	+ "loss_reduction": 0.009009443850986787,
	60	+ "reduction_per_ops": 0.036157913407464305
	61	+ },
	62	+ {
	63	+ "lambda": 0.01,
	64	+ "kappa": 0.0,
	65	+ "loss": 0.48403411629926835,
	66	+ "edits_per_token": 3.8208778173190985,
	67	+ "extra_ops_ratio": 0.4855178864683828,
	68	+ "loss_reduction": 0.0033719389746904582,
	69	+ "reduction_per_ops": 0.0069450355355961555
	70	+ },
	71	+ {
	72	+ "lambda": 0.01,
	73	+ "kappa": 0.5,
	74	+ "loss": 0.47863737855234745,
	75	+ "edits_per_token": 1.9727164887307236,
	76	+ "extra_ops_ratio": 0.23549998838847783,
	77	+ "loss_reduction": 0.008768676721611357,
	78	+ "reduction_per_ops": 0.03723429789366553
	79	+ },
	80	+ {
	81	+ "lambda": 0.03,
	82	+ "kappa": 0.0,
	83	+ "loss": 0.4810810790508809,
	84	+ "edits_per_token": 3.6109134045077105,
	85	+ "extra_ops_ratio": 0.4554090081302014,
	86	+ "loss_reduction": 0.006324976223077927,
	87	+ "reduction_per_ops": 0.013888561952357378
	88	+ },
	89	+ {
	90	+ "lambda": 0.03,
	91	+ "kappa": 0.5,
	92	+ "loss": 0.47920335201747616,
	93	+ "edits_per_token": 1.7520759193357058,
	94	+ "extra_ops_ratio": 0.20777568952893022,
	95	+ "loss_reduction": 0.008202703256482646,
	96	+ "reduction_per_ops": 0.03947864774305331
	97	+ },
	98	+ {
	99	+ "lambda": 0.1,
	100	+ "kappa": 0.0,
	101	+ "loss": 0.47742267803223815,
	102	+ "edits_per_token": 2.8718861209964412,
	103	+ "extra_ops_ratio": 0.35113779930033295,
	104	+ "loss_reduction": 0.009983377241720659,
	105	+ "reduction_per_ops": 0.028431508261466718
	106	+ },
	107	+ {
	108	+ "lambda": 0.1,
	109	+ "kappa": 0.5,
	110	+ "loss": 0.48005905920641845,
	111	+ "edits_per_token": 1.0676156583629892,
	112	+ "extra_ops_ratio": 0.1214192702676902,
	113	+ "loss_reduction": 0.007346996067540357,
	114	+ "reduction_per_ops": 0.06050930837701963
	115	+ },
	116	+ {
	117	+ "lambda": 0.3,
	118	+ "kappa": 0.0,
	119	+ "loss": 0.47899278570607323,
	120	+ "edits_per_token": 1.0723606168446027,
	121	+ "extra_ops_ratio": 0.11869999069901689,
	122	+ "loss_reduction": 0.008413269567885573,
	123	+ "reduction_per_ops": 0.07087843493786604
	124	+ },
	125	+ {
	126	+ "lambda": 0.3,
	127	+ "kappa": 0.5,
	128	+ "loss": 0.48617894680615986,
	129	+ "edits_per_token": 0.15361803084223014,
	130	+ "extra_ops_ratio": 0.016404977543356864,
	131	+ "loss_reduction": 0.001227108467798943,
	132	+ "reduction_per_ops": 0.07480098430831782
	133	+ }
	134	+ ],
	135	+ "baseline_budget0": {
	136	+ "loss": 0.4874060552739588,
	137	+ "edits_per_token": 0.0,
	138	+ "extra_ops_ratio": 0.0
	139	+ },
	140	+ "selected": {
	141	+ "lambda": 0.3,
	142	+ "kappa": 0.5,
	143	+ "loss": 0.48617894680615986,
	144	+ "edits_per_token": 0.15361803084223014,
	145	+ "extra_ops_ratio": 0.016404977543356864,
	146	+ "loss_reduction": 0.001227108467798943,
	147	+ "reduction_per_ops": 0.07480098430831782
	148	+ },
	149	+ "checkpoint_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198",
	150	+ "selection_rule": "max_loss_reduction_per_ops"
	151	+}
		\ No newline at end of file

added experiments/runs/controller_v2s1_seed01_20261030005758/metrics.jsonl +30 −0

		`@@ -0,0 +1,30 @@`
	1	+{"step": 0, "loss": 0.6217543482780457, "gain": 0.11268353462219238, "harm": 0.8242179155349731, "nll": 0.969618558883667, "t": 1.51}
	2	+{"step": 100, "loss": 0.19241824746131897, "gain": 0.003979177679866552, "harm": 0.5136688351631165, "nll": -0.6839534044265747, "t": 8.68}
	3	+{"step": 200, "loss": 0.05916588008403778, "gain": 0.0034271397162228823, "harm": 0.5102487206459045, "nll": -1.9938561916351318, "t": 14.33}
	4	+{"step": 249, "calib_huber": 0.010517006714784439, "calib_sign_acc": 0.6941967602912765, "calib_harm_brier": 0.1939330946408829, "calib_harm_ece": 0.07769986987113953, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.015281991320902816, "calib_captured_fraction": 0.16779286046737585, "calib_top1_match": 0.05410334346504559, "calib_n_audit_states": 1645, "t": 18.79}
	5	+{"step": 300, "loss": 0.14486049115657806, "gain": 0.003396645188331604, "harm": 0.6058646440505981, "nll": -1.6146845817565918, "t": 21.57}
	6	+{"step": 400, "loss": 0.07704775035381317, "gain": 0.0020318725146353245, "harm": 0.5852528810501099, "nll": -2.176105499267578, "t": 26.95}
	7	+{"step": 499, "calib_huber": 0.0015250327621141929, "calib_sign_acc": 0.5629365433199583, "calib_harm_brier": 0.17094168093994974, "calib_harm_ece": 0.022884827107191086, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01282272375101014, "calib_captured_fraction": 0.30171650868240585, "calib_top1_match": 0.0547112462006079, "calib_n_audit_states": 1645, "t": 33.86}
	8	+{"step": 500, "loss": 0.001370161771774292, "gain": 0.001527803367935121, "harm": 0.5307777523994446, "nll": -2.6554653644561768, "t": 33.92}
	9	+{"step": 600, "loss": 0.10702954232692719, "gain": 0.006142090540379286, "harm": 0.4747239649295807, "nll": -1.3647453784942627, "t": 39.36}
	10	+{"step": 700, "loss": -0.010433346033096313, "gain": 0.0026992247439920902, "harm": 0.42587319016456604, "nll": -2.2606916427612305, "t": 44.74}
	11	+{"step": 749, "calib_huber": 0.0016405455295319259, "calib_sign_acc": 0.4800861940853024, "calib_harm_brier": 0.16535962802514506, "calib_harm_ece": 0.01039370521903038, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.012461327057116662, "calib_captured_fraction": 0.3213969876557359, "calib_top1_match": 0.0547112462006079, "calib_n_audit_states": 1645, "t": 49.04}
	12	+{"step": 800, "loss": -0.019962161779403687, "gain": 0.0013843975029885769, "harm": 0.48733922839164734, "nll": -2.6501615047454834, "t": 51.84}
	13	+{"step": 900, "loss": -0.058852121233940125, "gain": 0.0009405212476849556, "harm": 0.4586194157600403, "nll": -2.8910233974456787, "t": 57.39}
	14	+{"step": 999, "calib_huber": 0.0012022649852991068, "calib_sign_acc": 0.6564125427255164, "calib_harm_brier": 0.1600398019582879, "calib_harm_ece": 0.01565980352461338, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.011427309013065233, "calib_captured_fraction": 0.3777062199145015, "calib_top1_match": 0.06322188449848025, "calib_n_audit_states": 1645, "t": 64.34}
	15	+{"step": 1000, "loss": -0.04631441831588745, "gain": 0.0013401865726336837, "harm": 0.4754399061203003, "nll": -2.853745460510254, "t": 64.4}
	16	+{"step": 1100, "loss": -0.07637041807174683, "gain": 0.0010725471656769514, "harm": 0.4387507736682892, "nll": -2.9681835174560547, "t": 69.79}
	17	+{"step": 1200, "loss": -0.06496578454971313, "gain": 0.0010426120134070516, "harm": 0.4698534905910492, "nll": -3.0093514919281006, "t": 75.21}
	18	+{"step": 1249, "calib_huber": 0.0011220869855551932, "calib_sign_acc": 0.6217491454896715, "calib_harm_brier": 0.15860497322598402, "calib_harm_ece": 0.02261553332209587, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.010739420540064664, "calib_captured_fraction": 0.41516637065089457, "calib_top1_match": 0.06504559270516717, "calib_n_audit_states": 1645, "t": 79.37}
	19	+{"step": 1300, "loss": -0.1207217276096344, "gain": 0.0005600811564363539, "harm": 0.4300370216369629, "nll": -3.3630030155181885, "t": 82.05}
	20	+{"step": 1400, "loss": -0.04722009599208832, "gain": 0.0011644494952633977, "harm": 0.45957231521606445, "nll": -2.7817070484161377, "t": 87.3}
	21	+{"step": 1499, "calib_huber": 0.0009785579579212032, "calib_sign_acc": 0.42041908158716007, "calib_harm_brier": 0.16143824849020302, "calib_harm_ece": 0.013009194284677505, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01221910088257949, "calib_captured_fraction": 0.3345878308906606, "calib_top1_match": 0.06869300911854104, "calib_n_audit_states": 1645, "t": 94.15}
	22	+{"step": 1500, "loss": -0.06804871559143066, "gain": 0.000664825551211834, "harm": 0.5305606126785278, "nll": -3.3399384021759033, "t": 94.21}
	23	+{"step": 1600, "loss": -0.07556450366973877, "gain": 0.001154533470980823, "harm": 0.5068498849868774, "nll": -3.3014397621154785, "t": 99.54}
	24	+{"step": 1700, "loss": -0.10204572975635529, "gain": 0.001108089811168611, "harm": 0.4906119704246521, "nll": -3.48459792137146, "t": 104.94}
	25	+{"step": 1749, "calib_huber": 0.0009797979088112332, "calib_sign_acc": 0.6669638876504681, "calib_harm_brier": 0.16354970610288794, "calib_harm_ece": 0.049120619893074036, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.012027452854399986, "calib_captured_fraction": 0.34502435411457977, "calib_top1_match": 0.08875379939209727, "calib_n_audit_states": 1645, "t": 109.14}
	26	+{"step": 1800, "loss": -0.11715850234031677, "gain": 0.0008175690891221166, "harm": 0.4343639016151428, "nll": -3.3515801429748535, "t": 111.91}
	27	+{"step": 1900, "loss": -0.1207641214132309, "gain": 0.0008810321451164782, "harm": 0.4911242127418518, "nll": -3.672072649002075, "t": 117.51}
	28	+{"step": 1999, "loss": -0.11400634050369263, "gain": 0.0009010079083964229, "harm": 0.47006553411483765, "nll": -3.499401092529297, "t": 122.79}
	29	+{"step": 1999, "calib_huber": 0.0009146865885242485, "calib_sign_acc": 0.4517387427552385, "calib_harm_brier": 0.15814475145820417, "calib_harm_ece": 0.018627993762493134, "calib_n_labels": 26916, "calib_oracle_opportunity_per_state": 0.01836320621989056, "calib_regret_per_state": 0.01151024432892495, "calib_captured_fraction": 0.37318983454766497, "calib_top1_match": 0.07963525835866261, "calib_n_audit_states": 1645, "t": 124.31}
	30	+{"event": "restored_best_calibration_checkpoint", "step": 1250, "score": -0.010739420540064664, "t": 124.31}

added experiments/runs/labels_v2s1_seed01_20261030005712/config.yaml +58 −0

		`@@ -0,0 +1,58 @@`
	1	+experiment:
	2	+ name: labels_v2s1
	3	+ seed: 1
	4	+ mode: prisme
	5	+ model: from_checkpoint
	6	+ checkpoint: experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint
	7	+ use_policy: false
	8	+ data:
	9	+ kind: synthetic
	10	+ tokenizer: char
	11	+ splits_dir: data/splits/synthetic_v1
	12	+ loss_span: answer
	13	+ collect:
	14	+ batch_size: 32
	15	+ seq_len: 320
	16	+ max_prefix: 3
	17	+ audit_every: 4
	18	+ partitions:
	19	+ train:
	20	+ split: ctrl_train
	21	+ n_batches: 300
	22	+ stream_seed: 11
	23	+ calib:
	24	+ split: calib
	25	+ n_batches: 40
	26	+ stream_seed: 12
	27	+model:
	28	+ name: tiny_p4v2
	29	+ vocab_size: 128
	30	+ d_model: 256
	31	+ n_layers: 6
	32	+ n_heads: 8
	33	+ n_kv_heads: 4
	34	+ ffn_width: 768
	35	+ max_seq_len: 320
	36	+ rope_theta: 10000.0
	37	+ norm_eps: 1.0e-05
	38	+ tie_embeddings: true
	39	+ init_std: 0.02
	40	+ use_workspace: true
	41	+ w: 64
	42	+ ffn_w: 192
	43	+ n_slots: 16
	44	+ n_editor_blocks: 2
	45	+ n_editor_heads: 2
	46	+ r_max: 8
	47	+ eta_max: 0.25
	48	+ clip_c: 1.0
	49	+ clip_eps: 1.0e-06
	50	+ readout_zero_init: false
	51	+ readout_init_std: 0.2
	52	+ readout_scale: 38.47460174560547
	53	+ controller_hidden: 0
	54	+ ctrl_input: z
	55	+ ctrl_use_entropy: false
	56	+ lambda_cost: 0.0
	57	+ kappa_risk: 0.0
	58	+ dense_writes: false

added experiments/runs/labels_v2s1_seed01_20261030005712/manifest.json +40 −0

		`@@ -0,0 +1,40 @@`
	1	+{
	2	+ "run_id": "labels_v2s1_seed01_20261030005712",
	3	+ "name": "labels_v2s1",
	4	+ "seed": 1,
	5	+ "code_commit": "unknown",
	6	+ "config_sha256": "d8a7c72237e220d4d2c97a848283a4500bd5af90fdd38e416f3c63cbf99d54af",
	7	+ "started_at": "2026-10-30T00:57:12-0400",
	8	+ "host": "MacStudonpierre",
	9	+ "python": "3.12.14",
	10	+ "torch": "2.15.0",
	11	+ "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/collect_interventions.py --config configs/experiments/labels_v2s1.yaml --device mps",
	12	+ "state": "completed",
	13	+ "wall_seconds": 44.65,
	14	+ "device": "mps",
	15	+ "checkpoint_parent": "experiments/runs/p4v2_prisme_seed01_20261030001749/checkpoint",
	16	+ "checkpoint_sha256": "63501504cc2f44e41c51c3c2ec941160a997e2e5035167f0cf79e8de7dc00d37",
	17	+ "finished_at": "2026-10-30T00:57:57-0400",
	18	+ "partitions": {
	19	+ "train": {
	20	+ "file": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_train.pt",
	21	+ "n_states": 49965,
	22	+ "n_labels": 199860,
	23	+ "mean_gain": 0.004881912376731634,
	24	+ "frac_positive": 0.3325177729129791,
	25	+ "frac_harm": 0.5166516304016113,
	26	+ "seconds": 39.4,
	27	+ "n_audit_batches": 75
	28	+ },
	29	+ "calib": {
	30	+ "file": "experiments/runs/labels_v2s1_seed01_20261030005712/labels_calib.pt",
	31	+ "n_states": 6729,
	32	+ "n_labels": 26916,
	33	+ "mean_gain": 0.005064019002020359,
	34	+ "frac_positive": 0.33912914991378784,
	35	+ "frac_harm": 0.5117030739784241,
	36	+ "seconds": 5.2,
	37	+ "n_audit_batches": 10
	38	+ }
	39	+ }
	40	+}
		\ No newline at end of file

added experiments/runs/labels_v2s1_seed01_20261030005712/metrics.jsonl +0 −0

modified paper/appendix/failures.tex +2 −0

		`@@ -18,4 +18,6 @@`
18	18	\textbf{date}: 2026-10-30; \textbf{what failed}: v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k $\rightarrow$ 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone.; \textbf{evidence}: \texttt{experiments/logs/p4\_tiny\_backbone\_v2.log}, run \texttt{p4\_tiny\_backbone\_v2\_seed01\_*} (state: stopped); \textbf{resolution}: 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator).
19	19	\item
20	20	\textbf{date}: 2026-10-30; \textbf{what failed}: Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix \texttt{p4v2\_dense}).; \textbf{evidence}: manifests remain in \texttt{experiments/runs/} on the nodes (state: running $\rightarrow$ killed)
	21	+\item
	22	+ \textbf{date}: 2026-10-30; \textbf{what failed}: Ranking-loss controller (D12: pairwise + listwise on top of Huber/BCE/NLL, editor-unseen labels): calibration sign accuracy fell to 45 \%, captured fraction --8 \%; on dev-iid the locked policy ($\lambda$=0.3, $\kappa$=0.5) edits 0.07 tokens and matches budget 0 (0.4333 vs 0.4334); $\lambda$=0 is worse (+1.3 \%).; \textbf{evidence}: \texttt{experiments/runs/controller\_v1rank\_seed01\_*}, \texttt{results/raw/routing\_v1rank.json}; \textbf{resolution}: Within-state ranking is not learnable from these labels at this scale; H3 null stands.
21	23	\end{itemize}

modified paper/main.log +19 −19

		`@@ -1,4 +1,4 @@`
1		−This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 00:55
	1	+This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 00:58
2	2	entering extended mode
3	3	restricted \write18 enabled.
4	4	%&-line parsing enabled.
		`@@ -1319,22 +1319,22 @@ Overfull \hbox (133.05286pt too wide) in paragraph at lines 7--18`
1319	1319	[]
1320	1320
1321	1321	) [16] [17 <../results/figures/learnability_bootstrap.pdf>]
1322		−<../results/figures/labels_analysis.pdf, id=518, 478.87544pt x 181.0464pt>
	1322	+<../results/figures/labels_analysis.pdf, id=518, 548.77519pt x 192.17598pt>
1323	1323	File: ../results/figures/labels_analysis.pdf Graphic file (type pdf)
1324	1324	<use ../results/figures/labels_analysis.pdf>
1325	1325	Package pdftex.def Info: ../results/figures/labels_analysis.pdf used on input
1326	1326	line 72.
1327		−(pdftex.def) Requested size: 469.75502pt x 177.60933pt.
	1327	+(pdftex.def) Requested size: 469.75502pt x 164.50862pt.
1328	1328
1329	1329	(./tables/routing.tex) (./tables/controller.tex) (./tables/routing_seeds.tex
1330	1330	Overfull \hbox (109.74734pt too wide) in paragraph at lines 8--14
1331	1331	[][]
1332	1332	[]
1333	1333
1334		−) [18] [19 <../results/figures/labels_analysis.pdf>] (./tables/latency.tex)
1335		−(./sections/results_pilot.tex)) (./sections/mechanistic.tex [20])
1336		−(./sections/limitations.tex [21]) (./sections/conclusion.tex)
1337		−(./sections/reproducibility.tex)
	1334	+) [18] (./tables/stats.tex) [19 <../results/figures/labels_analysis.pdf>]
	1335	+(./tables/latency.tex) (./sections/results_pilot.tex))
	1336	+(./sections/mechanistic.tex [20]) (./sections/limitations.tex [21])
	1337	+(./sections/conclusion.tex) (./sections/reproducibility.tex [22])
1338	1338	Overfull \hbox (51.18217pt too wide) in paragraph at lines 2--101
1339	1339	\T1/ptm/m/n/12 (-20) num-ber in the manuscript is pro-duced by \T1/cmtt/m/n/12
1340	1340	scripts/make_tables.py \T1/ptm/m/n/12 (-20) and \T1/cmtt/m/n/12 scripts/reprodu
		`@@ -1348,7 +1348,7 @@ cript_claims.py \T1/ptm/m/n/12 (-20) checks the chain. \T1/cmtt/m/n/12 REPRODUC`
1348	1348	E.md
1349	1349	[]
1350	1350
1351		−[22] [23] (./main.bbl [24] [25]
	1351	+[23] (./main.bbl [24] [25]
1352	1352	Underfull \hbox (badness 1297) in paragraph at lines 208--211
1353	1353	[]\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe
1354	1354	i Ning, Ning Ding, and
		`@@ -1561,7 +1561,7 @@ run \T1/cmtt/m/n/12 p4_tiny_backbone_v2_seed01_* \T1/ptm/m/n/12 (-20) (state:`
1561	1561	[]
1562	1562
1563	1563	)) [35{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1
1564		−.enc}] [36] [37] [38] (./main.aux)
	1564	+.enc}] [36] [37] [38] [39] (./main.aux)
1565	1565	***********
1566	1566	LaTeX2e <2025-11-01>
1567	1567	L3 programming layer <2026-01-19>
		`@@ -1571,14 +1571,14 @@ L3 programming layer <2026-01-19>`
1571	1571	LaTeX Warning: There were undefined references.
1572	1572
1573	1573	Package rerunfilecheck Info: File `main.out' has not changed.
1574		−(rerunfilecheck) Checksum: 09A4B2F3436223C5D37E56F8E45DB6B6;9020.
	1574	+(rerunfilecheck) Checksum: A0F1ADB953B7C9C76AC4BF5EDF00287D;9020.
1575	1575	)
1576	1576	Here is how much of TeX's memory you used:
1577		− 36051 strings out of 467525
1578		− 701004 string characters out of 5418982
1579		− 1125706 words of memory out of 5000000
1580		− 63382 multiletter control sequences out of 15000+600000
1581		− 793071 words of font info for 661 fonts, out of 8000000 for 9000
	1577	+ 36063 strings out of 467525
	1578	+ 701177 string characters out of 5418982
	1579	+ 1129055 words of memory out of 5000000
	1580	+ 63386 multiletter control sequences out of 15000+600000
	1581	+ 793198 words of font info for 663 fonts, out of 8000000 for 9000
1582	1582	1141 hyphenation exceptions out of 8191
1583	1583	84i,13n,104p,1001b,1501s stack positions out of 10000i,1000n,20000p,200000b,200000s
1584	1584	</usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb
		`@@ -1596,10 +1596,10 @@ exlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026`
1596	1596	e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti
1597	1597	mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8
1598	1598	a.pfb>
1599		−Output written on main.pdf (38 pages, 415081 bytes).
	1599	+Output written on main.pdf (39 pages, 416998 bytes).
1600	1600	PDF statistics:
1601		− 759 PDF objects out of 1000 (max. 8388607)
1602		− 659 compressed objects within 7 object streams
1603		− 153 named destinations out of 1000 (max. 500000)
	1601	+ 764 PDF objects out of 1000 (max. 8388607)
	1602	+ 663 compressed objects within 7 object streams
	1603	+ 155 named destinations out of 1000 (max. 500000)
1604	1604	143772 words of extra memory for PDF output out of 154059 (max. 10000000)
1605	1605

modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/sections/results.tex +13 −4

		`@@ -103,15 +103,24 @@ over \vtwoctrlseeds{} seeds the locked controller changes the dev-iid loss by \v`
103	103	\vtwoctrlrelsd{}) while editing a small fraction of tokens, the best fixed schedule lowers it by
104	104	\vtwobestfixedrel{}\,\% and the greedy oracle by \vtwooraclerel{}\,\%; the controller beats the frozen backbone
105	105	on \vtwoctrlbetter{} of \vtwoctrlseeds{} seeds and never approaches the fixed schedules at any cost.
	106	+(v) Adding within-state ranking losses (pairwise on the logged candidates, listwise on the exhaustive audits;
	107	+Appendix~\ref{app:failures}) did not change the picture: the locked policy edits 0.07 tokens and matches
	108	+budget 0 (0.4333 vs 0.4334), and every smaller penalty is worse.
106	109	\input{tables/routing.tex}
107	110	\input{tables/controller.tex}
108	111	\input{tables/routing_seeds.tex}
109	112
110	113	\subsection{Token loss versus task success}\label{sec:results-reasoning}
111		−Exact match of free greedy generation moves with the loss on dev-iid (Table~\ref{tab:seedmatrix}: $+1.8$ points
112		−for the sparse editor at budget 2, $+2.3$ for global recurrence at budget 8, seed SD $\le 0.5$) but not on the
113		−held-out regimes; per-family transitions are reported in Appendix~\ref{app:results}. No adapter turns the
114		−held-out compose family from near zero (2--3\,\%) into anything else.
	114	+Exact match of free greedy generation moves with the loss on dev-iid but by less than the pre-registered
	115	+minimum effect of three points (Table~\ref{tab:stats}): the sparse editor gains $+1.9$ points at budget 2
	116	+(95\,\% $t$-interval over five seeds $[+1.6,+2.2]$; every per-seed paired item bootstrap excludes zero), global
	117	+recurrence gains $+1.9$ at budget 2 and $+2.3$ at budget 8, and the paired difference between the two systems
	118	+at budget 2 is $+0.1$ points $[-0.5,+0.7]$---a null---while at budget 1 global recurrence is ahead by $0.6$
	119	+points $[0.3,0.9]$. Transitions are two-sided: at budget 2 the sparse editor turns \vtwoprismetransplus{}
	120	+items per thousand from wrong to right and \vtwoprismetransminus{} from right to wrong
	121	+(Table~\ref{tab:emfamilies}); gains concentrate in \textsf{fsm} and \textsf{bind}, \textsf{excl} does not move.
	122	+No adapter changes the held-out regimes by more than a point, and the compose family stays at 2--3\,\%.
	123	+\input{tables/stats.tex}
115	124
116	125	\subsection{Latency and memory}\label{sec:results-latency}
117	126	Table~\ref{tab:latency} gives measured batch-1 and batch-8 decode latencies for the tiny model on the

modified paper/tables/numbers.tex +6 −6

		`@@ -57,12 +57,12 @@`
57	57	\newcommand{\routinguniformbeight}{+0.6}
58	58	\newcommand{\routinguniformbone}{-0.8}
59	59	\newcommand{\routinguniformbtwo}{-0.7}
60		−\newcommand{\vtwobestfixedrel}{-4.0}
	60	+\newcommand{\vtwobestfixedrel}{-4.2}
61	61	\newcommand{\vtwobzeroloss}{0.4621}
62		−\newcommand{\vtwoctrlbetter}{2}
63		−\newcommand{\vtwoctrlrel}{+0.5}
64		−\newcommand{\vtwoctrlrelsd}{1.1}
65		−\newcommand{\vtwoctrlseeds}{4}
	62	+\newcommand{\vtwoctrlbetter}{3}
	63	+\newcommand{\vtwoctrlrel}{+0.3}
	64	+\newcommand{\vtwoctrlrelsd}{1.0}
	65	+\newcommand{\vtwoctrlseeds}{5}
66	66	\newcommand{\vtwodenseembeight}{n/a}
67	67	\newcommand{\vtwodenseembtwo}{n/a}
68	68	\newcommand{\vtwodenseembzero}{n/a}
		`@@ -81,7 +81,7 @@`
81	81	\newcommand{\vtwogrseeds}{5}
82	82	\newcommand{\vtwogrtransminus}{39}
83	83	\newcommand{\vtwogrtransplus}{57}
84		−\newcommand{\vtwooraclerel}{-15.0}
	84	+\newcommand{\vtwooraclerel}{-15.1}
85	85	\newcommand{\vtwoprismeembeight}{39.2}
86	86	\newcommand{\vtwoprismeembtwo}{39.6}
87	87	\newcommand{\vtwoprismeembzero}{37.7}

modified paper/tables/routing_seeds.tex +1 −0

		`@@ -7,6 +7,7 @@ opportunity for each controller.}`
7	7	\label{tab:routingseeds}
8	8	\begin{tabular}{lrlrrrrrr}\toprule
9	9	seed & budget 0 & best fixed & entropy & norm & controller (edits) & oracle b=8 (edits) & sign acc. & captured\\\midrule
	10	+1 & 0.4966 & 0.4738 (random forced b=4) & 0.4780 & 0.4774 & 0.4950 (0.18) & 0.4198 (2.16) & 0.622 & +0.415\\
10	11	2 & 0.4966 & 0.4754 (random forced b=4) & 0.4799 & 0.4794 & 0.4965 (0.05) & 0.4114 (1.90) & 0.413 & +0.333\\
11	12	3 & 0.4966 & 0.4775 (uniform b=8) & 0.4825 & 0.4814 & 0.5074 (4.32) & 0.4242 (1.84) & 0.504 & +0.305\\
12	13	4 & 0.4966 & 0.4762 (uniform b=8) & 0.4795 & 0.4790 & 0.4974 (0.17) & 0.4251 (2.19) & 0.633 & +0.521\\

added paper/tables/stats.tex +17 −0

		`@@ -0,0 +1,17 @@`
	1	+\begin{table}[t]\centering\small
	2	+\caption{Pre-registered uncertainty analysis of dev-iid exact match (percentage points). ``t-CI'' is the 95\,\%
	3	+$t$-interval over independent adapter seeds of the per-seed paired difference; ``item CI range'' spans the
	4	+smallest lower and largest upper bound of the per-seed paired item bootstraps (10,000 resamples, 1,000 items).}
	5	+\label{tab:stats}
	6	+\begin{tabular}{lrrll}\toprule
	7	+Comparison & seeds & mean & t-CI over seeds & item CI range\\\midrule
	8	+prisme b1 vs b0 & 5 & +1.58 & [+1.21, +1.95] & [+0.30, +3.30]\\
	9	+prisme b2 vs b0 & 5 & +1.92 & [+1.60, +2.24] & [+0.10, +3.90]\\
	10	+prisme b8 vs b0 & 5 & +1.50 & [+0.94, +2.06] & [-1.10, +4.50]\\
	11	+gr b1 vs b0 & 5 & +2.14 & [+1.76, +2.52] & [+0.20, +4.10]\\
	12	+gr b2 vs b0 & 5 & +1.86 & [+1.37, +2.35] & [-0.30, +4.50]\\
	13	+gr b8 vs b0 & 5 & +2.34 & [+0.97, +3.71] & [-1.40, +6.20]\\
	14	+prisme minus gr b1 & 5 & -0.56 & [-0.85, -0.27] & [-2.20, +0.90]\\
	15	+prisme minus gr b2 & 5 & +0.06 & [-0.53, +0.65] & [-2.20, +2.00]\\
	16	+prisme minus gr b8 & 5 & -0.84 & [-2.43, +0.75] & [-4.40, +2.50]\\
	17	+\bottomrule\end{tabular}\end{table}

modified results/analyzed/routing_seeds_v2.json +16 −0

		`@@ -1,4 +1,20 @@`
1	1	[
	2	+ {
	3	+ "seed": "1",
	4	+ "b0": 0.4965549189338985,
	5	+ "best_fixed": 0.4737940016007751,
	6	+ "best_fixed_name": "random forced b=4",
	7	+ "entropy": 0.47797408758425247,
	8	+ "norm": 0.4774494416747904,
	9	+ "ctrl": 0.4950156743532738,
	10	+ "ctrl_edits": 0.18027509511267192,
	11	+ "ctrl_lam": 0.3,
	12	+ "ctrl_kap": 0.5,
	13	+ "oracle8": 0.4197931946875673,
	14	+ "oracle8_edits": 2.1623277182235836,
	15	+ "sign_acc": 0.6217491454896715,
	16	+ "captured": 0.41516637065089457
	17	+ },
2	18	{
3	19	"seed": "2",
4	20	"b0": 0.49655490497905047,

added results/analyzed/statistics_v2.json +506 −0

		`@@ -0,0 +1,506 @@`
	1	+{
	2	+ "prisme_b1_vs_b0": {
	3	+ "per_seed": [
	4	+ {
	5	+ "diff": 0.021,
	6	+ "ci_low": 0.01,
	7	+ "ci_high": 0.033,
	8	+ "n": 1000,
	9	+ "p_two_sided": 0.0,
	10	+ "run_id": "p4v2_prisme_seed01_20261030001749"
	11	+ },
	12	+ {
	13	+ "diff": 0.014,
	14	+ "ci_low": 0.003,
	15	+ "ci_high": 0.025,
	16	+ "n": 1000,
	17	+ "p_two_sided": 0.0154,
	18	+ "run_id": "p4v2_prisme_seed02_20261030001749"
	19	+ },
	20	+ {
	21	+ "diff": 0.014,
	22	+ "ci_low": 0.003,
	23	+ "ci_high": 0.026,
	24	+ "n": 1000,
	25	+ "p_two_sided": 0.0182,
	26	+ "run_id": "p4v2_prisme_seed03_20261030001749"
	27	+ },
	28	+ {
	29	+ "diff": 0.015,
	30	+ "ci_low": 0.004,
	31	+ "ci_high": 0.026,
	32	+ "n": 1000,
	33	+ "p_two_sided": 0.0104,
	34	+ "run_id": "p4v2_prisme_seed04_20261030001749"
	35	+ },
	36	+ {
	37	+ "diff": 0.015,
	38	+ "ci_low": 0.004,
	39	+ "ci_high": 0.027,
	40	+ "n": 1000,
	41	+ "p_two_sided": 0.0098,
	42	+ "run_id": "p4v2_prisme_seed05_20261030001839"
	43	+ }
	44	+ ],
	45	+ "seeds": {
	46	+ "mean": 0.0158,
	47	+ "sd": 0.0029495762407505256,
	48	+ "ci_low": 0.012137617371902876,
	49	+ "ci_high": 0.019462382628097127,
	50	+ "n": 5
	51	+ }
	52	+ },
	53	+ "prisme_b2_vs_b0": {
	54	+ "per_seed": [
	55	+ {
	56	+ "diff": 0.023,
	57	+ "ci_low": 0.008,
	58	+ "ci_high": 0.039,
	59	+ "n": 1000,
	60	+ "p_two_sided": 0.0042,
	61	+ "run_id": "p4v2_prisme_seed01_20261030001749"
	62	+ },
	63	+ {
	64	+ "diff": 0.016,
	65	+ "ci_low": 0.001,
	66	+ "ci_high": 0.031,
	67	+ "n": 1000,
	68	+ "p_two_sided": 0.0416,
	69	+ "run_id": "p4v2_prisme_seed02_20261030001749"
	70	+ },
	71	+ {
	72	+ "diff": 0.02,
	73	+ "ci_low": 0.005,
	74	+ "ci_high": 0.035,
	75	+ "n": 1000,
	76	+ "p_two_sided": 0.0112,
	77	+ "run_id": "p4v2_prisme_seed03_20261030001749"
	78	+ },
	79	+ {
	80	+ "diff": 0.018,
	81	+ "ci_low": 0.004,
	82	+ "ci_high": 0.032,
	83	+ "n": 1000,
	84	+ "p_two_sided": 0.0124,
	85	+ "run_id": "p4v2_prisme_seed04_20261030001749"
	86	+ },
	87	+ {
	88	+ "diff": 0.019,
	89	+ "ci_low": 0.004,
	90	+ "ci_high": 0.035,
	91	+ "n": 1000,
	92	+ "p_two_sided": 0.0136,
	93	+ "run_id": "p4v2_prisme_seed05_20261030001839"
	94	+ }
	95	+ ],
	96	+ "seeds": {
	97	+ "mean": 0.019200000000000002,
	98	+ "sd": 0.002588435821108957,
	99	+ "ci_low": 0.01598603242926801,
	100	+ "ci_high": 0.022413967570731993,
	101	+ "n": 5
	102	+ }
	103	+ },
	104	+ "prisme_b8_vs_b0": {
	105	+ "per_seed": [
	106	+ {
	107	+ "diff": 0.013,
	108	+ "ci_low": -0.008,
	109	+ "ci_high": 0.035,
	110	+ "n": 1000,
	111	+ "p_two_sided": 0.2546,
	112	+ "run_id": "p4v2_prisme_seed01_20261030001749"
	113	+ },
	114	+ {
	115	+ "diff": 0.012,
	116	+ "ci_low": -0.011,
	117	+ "ci_high": 0.035,
	118	+ "n": 1000,
	119	+ "p_two_sided": 0.327,
	120	+ "run_id": "p4v2_prisme_seed02_20261030001749"
	121	+ },
	122	+ {
	123	+ "diff": 0.023,
	124	+ "ci_low": 0.001,
	125	+ "ci_high": 0.045,
	126	+ "n": 1000,
	127	+ "p_two_sided": 0.0492,
	128	+ "run_id": "p4v2_prisme_seed03_20261030001749"
	129	+ },
	130	+ {
	131	+ "diff": 0.013,
	132	+ "ci_low": -0.009,
	133	+ "ci_high": 0.036,
	134	+ "n": 1000,
	135	+ "p_two_sided": 0.2778,
	136	+ "run_id": "p4v2_prisme_seed04_20261030001749"
	137	+ },
	138	+ {
	139	+ "diff": 0.014,
	140	+ "ci_low": -0.008,
	141	+ "ci_high": 0.036,
	142	+ "n": 1000,
	143	+ "p_two_sided": 0.2264,
	144	+ "run_id": "p4v2_prisme_seed05_20261030001839"
	145	+ }
	146	+ ],
	147	+ "seeds": {
	148	+ "mean": 0.015,
	149	+ "sd": 0.004527692569068708,
	150	+ "ci_low": 0.009378127142052675,
	151	+ "ci_high": 0.020621872857947324,
	152	+ "n": 5
	153	+ }
	154	+ },
	155	+ "gr_b1_vs_b0": {
	156	+ "per_seed": [
	157	+ {
	158	+ "diff": 0.025,
	159	+ "ci_low": 0.01,
	160	+ "ci_high": 0.041,
	161	+ "n": 1000,
	162	+ "p_two_sided": 0.0008,
	163	+ "run_id": "p4v2_gr_seed01_20261030001749"
	164	+ },
	165	+ {
	166	+ "diff": 0.018,
	167	+ "ci_low": 0.002,
	168	+ "ci_high": 0.034,
	169	+ "n": 1000,
	170	+ "p_two_sided": 0.0286,
	171	+ "run_id": "p4v2_gr_seed02_20261030001749"
	172	+ },
	173	+ {
	174	+ "diff": 0.021,
	175	+ "ci_low": 0.005,
	176	+ "ci_high": 0.037,
	177	+ "n": 1000,
	178	+ "p_two_sided": 0.011,
	179	+ "run_id": "p4v2_gr_seed03_20261030001749"
	180	+ },
	181	+ {
	182	+ "diff": 0.024,
	183	+ "ci_low": 0.008,
	184	+ "ci_high": 0.04,
	185	+ "n": 1000,
	186	+ "p_two_sided": 0.0036,
	187	+ "run_id": "p4v2_gr_seed04_20261030001749"
	188	+ },
	189	+ {
	190	+ "diff": 0.019,
	191	+ "ci_low": 0.004,
	192	+ "ci_high": 0.035,
	193	+ "n": 1000,
	194	+ "p_two_sided": 0.0178,
	195	+ "run_id": "p4v2_gr_seed05_20261030001839"
	196	+ }
	197	+ ],
	198	+ "seeds": {
	199	+ "mean": 0.0214,
	200	+ "sd": 0.003049590136395382,
	201	+ "ci_low": 0.017613433718360546,
	202	+ "ci_high": 0.025186566281639452,
	203	+ "n": 5
	204	+ }
	205	+ },
	206	+ "gr_b2_vs_b0": {
	207	+ "per_seed": [
	208	+ {
	209	+ "diff": 0.018,
	210	+ "ci_low": -0.001,
	211	+ "ci_high": 0.037,
	212	+ "n": 1000,
	213	+ "p_two_sided": 0.0676,
	214	+ "run_id": "p4v2_gr_seed01_20261030001749"
	215	+ },
	216	+ {
	217	+ "diff": 0.016,
	218	+ "ci_low": -0.002,
	219	+ "ci_high": 0.035,
	220	+ "n": 1000,
	221	+ "p_two_sided": 0.1028,
	222	+ "run_id": "p4v2_gr_seed02_20261030001749"
	223	+ },
	224	+ {
	225	+ "diff": 0.019,
	226	+ "ci_low": -0.001,
	227	+ "ci_high": 0.039,
	228	+ "n": 1000,
	229	+ "p_two_sided": 0.0656,
	230	+ "run_id": "p4v2_gr_seed03_20261030001749"
	231	+ },
	232	+ {
	233	+ "diff": 0.025,
	234	+ "ci_low": 0.005,
	235	+ "ci_high": 0.045,
	236	+ "n": 1000,
	237	+ "p_two_sided": 0.0126,
	238	+ "run_id": "p4v2_gr_seed04_20261030001749"
	239	+ },
	240	+ {
	241	+ "diff": 0.015,
	242	+ "ci_low": -0.003,
	243	+ "ci_high": 0.034,
	244	+ "n": 1000,
	245	+ "p_two_sided": 0.1178,
	246	+ "run_id": "p4v2_gr_seed05_20261030001839"
	247	+ }
	248	+ ],
	249	+ "seeds": {
	250	+ "mean": 0.018600000000000002,
	251	+ "sd": 0.00391152144312159,
	252	+ "ci_low": 0.013743204645873891,
	253	+ "ci_high": 0.023456795354126113,
	254	+ "n": 5
	255	+ }
	256	+ },
	257	+ "gr_b8_vs_b0": {
	258	+ "per_seed": [
	259	+ {
	260	+ "diff": 0.024,
	261	+ "ci_low": 0.002,
	262	+ "ci_high": 0.046,
	263	+ "n": 1000,
	264	+ "p_two_sided": 0.0338,
	265	+ "run_id": "p4v2_gr_seed01_20261030001749"
	266	+ },
	267	+ {
	268	+ "diff": 0.025,
	269	+ "ci_low": 0.003,
	270	+ "ci_high": 0.047,
	271	+ "n": 1000,
	272	+ "p_two_sided": 0.0298,
	273	+ "run_id": "p4v2_gr_seed02_20261030001749"
	274	+ },
	275	+ {
	276	+ "diff": 0.021,
	277	+ "ci_low": -0.002,
	278	+ "ci_high": 0.044,
	279	+ "n": 1000,
	280	+ "p_two_sided": 0.0818,
	281	+ "run_id": "p4v2_gr_seed03_20261030001749"
	282	+ },
	283	+ {
	284	+ "diff": 0.039,
	285	+ "ci_low": 0.017,
	286	+ "ci_high": 0.062,
	287	+ "n": 1000,
	288	+ "p_two_sided": 0.0014,
	289	+ "run_id": "p4v2_gr_seed04_20261030001749"
	290	+ },
	291	+ {
	292	+ "diff": 0.008,
	293	+ "ci_low": -0.014,
	294	+ "ci_high": 0.03,
	295	+ "n": 1000,
	296	+ "p_two_sided": 0.5004,
	297	+ "run_id": "p4v2_gr_seed05_20261030001839"
	298	+ }
	299	+ ],
	300	+ "seeds": {
	301	+ "mean": 0.023400000000000004,
	302	+ "sd": 0.011058933040759403,
	303	+ "ci_low": 0.009668520984742975,
	304	+ "ci_high": 0.037131479015257034,
	305	+ "n": 5
	306	+ }
	307	+ },
	308	+ "prisme_minus_gr_b1": {
	309	+ "per_seed": [
	310	+ {
	311	+ "diff": -0.004,
	312	+ "ci_low": -0.017,
	313	+ "ci_high": 0.009,
	314	+ "n": 1000,
	315	+ "p_two_sided": 0.6038,
	316	+ "run_ids": [
	317	+ "p4v2_prisme_seed01_20261030001749",
	318	+ "p4v2_gr_seed01_20261030001749"
	319	+ ]
	320	+ },
	321	+ {
	322	+ "diff": -0.004,
	323	+ "ci_low": -0.017,
	324	+ "ci_high": 0.009,
	325	+ "n": 1000,
	326	+ "p_two_sided": 0.603,
	327	+ "run_ids": [
	328	+ "p4v2_prisme_seed02_20261030001749",
	329	+ "p4v2_gr_seed02_20261030001749"
	330	+ ]
	331	+ },
	332	+ {
	333	+ "diff": -0.007,
	334	+ "ci_low": -0.021,
	335	+ "ci_high": 0.006,
	336	+ "n": 1000,
	337	+ "p_two_sided": 0.3406,
	338	+ "run_ids": [
	339	+ "p4v2_prisme_seed03_20261030001749",
	340	+ "p4v2_gr_seed03_20261030001749"
	341	+ ]
	342	+ },
	343	+ {
	344	+ "diff": -0.009,
	345	+ "ci_low": -0.022,
	346	+ "ci_high": 0.004,
	347	+ "n": 1000,
	348	+ "p_two_sided": 0.2072,
	349	+ "run_ids": [
	350	+ "p4v2_prisme_seed04_20261030001749",
	351	+ "p4v2_gr_seed04_20261030001749"
	352	+ ]
	353	+ },
	354	+ {
	355	+ "diff": -0.004,
	356	+ "ci_low": -0.017,
	357	+ "ci_high": 0.008,
	358	+ "n": 1000,
	359	+ "p_two_sided": 0.5746,
	360	+ "run_ids": [
	361	+ "p4v2_prisme_seed05_20261030001839",
	362	+ "p4v2_gr_seed05_20261030001839"
	363	+ ]
	364	+ }
	365	+ ],
	366	+ "seeds": {
	367	+ "mean": -0.0056,
	368	+ "sd": 0.0023021728866442675,
	369	+ "ci_low": -0.008458525190987023,
	370	+ "ci_high": -0.002741474809012978,
	371	+ "n": 5
	372	+ }
	373	+ },
	374	+ "prisme_minus_gr_b2": {
	375	+ "per_seed": [
	376	+ {
	377	+ "diff": 0.005,
	378	+ "ci_low": -0.01,
	379	+ "ci_high": 0.02,
	380	+ "n": 1000,
	381	+ "p_two_sided": 0.5636,
	382	+ "run_ids": [
	383	+ "p4v2_prisme_seed01_20261030001749",
	384	+ "p4v2_gr_seed01_20261030001749"
	385	+ ]
	386	+ },
	387	+ {
	388	+ "diff": 0.0,
	389	+ "ci_low": -0.015,
	390	+ "ci_high": 0.015,
	391	+ "n": 1000,
	392	+ "p_two_sided": 1.043,
	393	+ "run_ids": [
	394	+ "p4v2_prisme_seed02_20261030001749",
	395	+ "p4v2_gr_seed02_20261030001749"
	396	+ ]
	397	+ },
	398	+ {
	399	+ "diff": 0.001,
	400	+ "ci_low": -0.015,
	401	+ "ci_high": 0.018,
	402	+ "n": 1000,
	403	+ "p_two_sided": 0.955,
	404	+ "run_ids": [
	405	+ "p4v2_prisme_seed03_20261030001749",
	406	+ "p4v2_gr_seed03_20261030001749"
	407	+ ]
	408	+ },
	409	+ {
	410	+ "diff": -0.007,
	411	+ "ci_low": -0.022,
	412	+ "ci_high": 0.008,
	413	+ "n": 1000,
	414	+ "p_two_sided": 0.3848,
	415	+ "run_ids": [
	416	+ "p4v2_prisme_seed04_20261030001749",
	417	+ "p4v2_gr_seed04_20261030001749"
	418	+ ]
	419	+ },
	420	+ {
	421	+ "diff": 0.004,
	422	+ "ci_low": -0.011,
	423	+ "ci_high": 0.019,
	424	+ "n": 1000,
	425	+ "p_two_sided": 0.6398,
	426	+ "run_ids": [
	427	+ "p4v2_prisme_seed05_20261030001839",
	428	+ "p4v2_gr_seed05_20261030001839"
	429	+ ]
	430	+ }
	431	+ ],
	432	+ "seeds": {
	433	+ "mean": 0.0006000000000000001,
	434	+ "sd": 0.004722287581247038,
	435	+ "ci_low": -0.00526349447879918,
	436	+ "ci_high": 0.006463494478799179,
	437	+ "n": 5
	438	+ }
	439	+ },
	440	+ "prisme_minus_gr_b8": {
	441	+ "per_seed": [
	442	+ {
	443	+ "diff": -0.011,
	444	+ "ci_low": -0.029,
	445	+ "ci_high": 0.007,
	446	+ "n": 1000,
	447	+ "p_two_sided": 0.264,
	448	+ "run_ids": [
	449	+ "p4v2_prisme_seed01_20261030001749",
	450	+ "p4v2_gr_seed01_20261030001749"
	451	+ ]
	452	+ },
	453	+ {
	454	+ "diff": -0.013,
	455	+ "ci_low": -0.032,
	456	+ "ci_high": 0.005,
	457	+ "n": 1000,
	458	+ "p_two_sided": 0.1828,
	459	+ "run_ids": [
	460	+ "p4v2_prisme_seed02_20261030001749",
	461	+ "p4v2_gr_seed02_20261030001749"
	462	+ ]
	463	+ },
	464	+ {
	465	+ "diff": 0.002,
	466	+ "ci_low": -0.014,
	467	+ "ci_high": 0.018,
	468	+ "n": 1000,
	469	+ "p_two_sided": 0.8416,
	470	+ "run_ids": [
	471	+ "p4v2_prisme_seed03_20261030001749",
	472	+ "p4v2_gr_seed03_20261030001749"
	473	+ ]
	474	+ },
	475	+ {
	476	+ "diff": -0.026,
	477	+ "ci_low": -0.044,
	478	+ "ci_high": -0.008,
	479	+ "n": 1000,
	480	+ "p_two_sided": 0.0044,
	481	+ "run_ids": [
	482	+ "p4v2_prisme_seed04_20261030001749",
	483	+ "p4v2_gr_seed04_20261030001749"
	484	+ ]
	485	+ },
	486	+ {
	487	+ "diff": 0.006,
	488	+ "ci_low": -0.012,
	489	+ "ci_high": 0.025,
	490	+ "n": 1000,
	491	+ "p_two_sided": 0.5602,
	492	+ "run_ids": [
	493	+ "p4v2_prisme_seed05_20261030001839",
	494	+ "p4v2_gr_seed05_20261030001839"
	495	+ ]
	496	+ }
	497	+ ],
	498	+ "seeds": {
	499	+ "mean": -0.008400000000000001,
	500	+ "sd": 0.01277888884058391,
	501	+ "ci_low": -0.02426708621040088,
	502	+ "ci_high": 0.0074670862104008785,
	503	+ "n": 5
	504	+ }
	505	+ }
	506	+}
		\ No newline at end of file

added results/raw/routing_v1rank.json +290 −0

		`@@ -0,0 +1,290 @@`
	1	+{
	2	+ "config": "configs/experiments/routing_v1rank.yaml",
	3	+ "checkpoint": "experiments/runs/controller_v1rank_seed01_20261030005431/checkpoint",
	4	+ "checkpoint_sha256": "64570e628212ba370db86265ffaa23f655bfee6e58120829c5bbe5881c7aadd9",
	5	+ "split": "dev_iid",
	6	+ "code_commit": "512b98066b4c794b04c46f9b43023c6135f1adc1",
	7	+ "device": "mps",
	8	+ "seconds": 84.3,
	9	+ "records": [
	10	+ {
	11	+ "policy": "budget0",
	12	+ "loss": 0.43343788690653534,
	13	+ "edits_per_token": 0.0,
	14	+ "extra_ops_ratio": 0.0,
	15	+ "n_tokens": 3417
	16	+ },
	17	+ {
	18	+ "policy": "random_forced",
	19	+ "budget": 1,
	20	+ "loss": 0.43049107206331766,
	21	+ "edits_per_token": 1.0,
	22	+ "extra_ops_ratio": 0.33739269034430613,
	23	+ "n_tokens": 3417
	24	+ },
	25	+ {
	26	+ "policy": "uniform",
	27	+ "budget": 1,
	28	+ "loss": 0.430082754325476,
	29	+ "edits_per_token": 1.0,
	30	+ "extra_ops_ratio": 0.2911194881431394,
	31	+ "n_tokens": 3417
	32	+ },
	33	+ {
	34	+ "policy": "fixed_pair",
	35	+ "budget": 1,
	36	+ "loss": 0.4306084942950259,
	37	+ "edits_per_token": 1.0,
	38	+ "extra_ops_ratio": 0.38326141294904154,
	39	+ "n_tokens": 3417
	40	+ },
	41	+ {
	42	+ "policy": "random_forced",
	43	+ "budget": 2,
	44	+ "loss": 0.43129151572879726,
	45	+ "edits_per_token": 2.0,
	46	+ "extra_ops_ratio": 0.6752437784687831,
	47	+ "n_tokens": 3417
	48	+ },
	49	+ {
	50	+ "policy": "uniform",
	51	+ "budget": 2,
	52	+ "loss": 0.43061076446970825,
	53	+ "edits_per_token": 2.0,
	54	+ "extra_ops_ratio": 0.5822389762862789,
	55	+ "n_tokens": 3417
	56	+ },
	57	+ {
	58	+ "policy": "fixed_pair",
	59	+ "budget": 2,
	60	+ "loss": 0.43226337237493456,
	61	+ "edits_per_token": 2.0,
	62	+ "extra_ops_ratio": 0.7665228258980831,
	63	+ "n_tokens": 3417
	64	+ },
	65	+ {
	66	+ "policy": "random_forced",
	67	+ "budget": 4,
	68	+ "loss": 0.4350665061488499,
	69	+ "edits_per_token": 4.0,
	70	+ "extra_ops_ratio": 1.3500830851558498,
	71	+ "n_tokens": 3417
	72	+ },
	73	+ {
	74	+ "policy": "uniform",
	75	+ "budget": 4,
	76	+ "loss": 0.43226772879940184,
	77	+ "edits_per_token": 4.0,
	78	+ "extra_ops_ratio": 1.1644779525725577,
	79	+ "n_tokens": 3417
	80	+ },
	81	+ {
	82	+ "policy": "fixed_pair",
	83	+ "budget": 4,
	84	+ "loss": 0.43608775067685257,
	85	+ "edits_per_token": 4.0,
	86	+ "extra_ops_ratio": 1.5330456517961661,
	87	+ "n_tokens": 3417
	88	+ },
	89	+ {
	90	+ "policy": "random_forced",
	91	+ "budget": 8,
	92	+ "loss": 0.44534724575753865,
	93	+ "edits_per_token": 8.0,
	94	+ "extra_ops_ratio": 2.6992224028210283,
	95	+ "n_tokens": 3417
	96	+ },
	97	+ {
	98	+ "policy": "uniform",
	99	+ "budget": 8,
	100	+ "loss": 0.4360636224654602,
	101	+ "edits_per_token": 8.0,
	102	+ "extra_ops_ratio": 2.3289559051451154,
	103	+ "n_tokens": 3417
	104	+ },
	105	+ {
	106	+ "policy": "fixed_pair",
	107	+ "budget": 8,
	108	+ "loss": 0.4454028259636821,
	109	+ "edits_per_token": 8.0,
	110	+ "extra_ops_ratio": 3.0660913035923323,
	111	+ "n_tokens": 3417
	112	+ },
	113	+ {
	114	+ "policy": "entropy",
	115	+ "threshold": 0.1,
	116	+ "budget": 8,
	117	+ "loss": 0.44345039856653656,
	118	+ "edits_per_token": 2.893473807433421,
	119	+ "extra_ops_ratio": 0.9313065950585138,
	120	+ "n_tokens": 3417
	121	+ },
	122	+ {
	123	+ "policy": "entropy",
	124	+ "threshold": 0.5,
	125	+ "budget": 8,
	126	+ "loss": 0.44244280132174246,
	127	+ "edits_per_token": 2.097453906935909,
	128	+ "extra_ops_ratio": 0.6750847836283169,
	129	+ "n_tokens": 3417
	130	+ },
	131	+ {
	132	+ "policy": "entropy",
	133	+ "threshold": 1.0,
	134	+ "budget": 8,
	135	+ "loss": 0.4375149400603328,
	136	+ "edits_per_token": 1.1785191688615744,
	137	+ "extra_ops_ratio": 0.3794397201180842,
	138	+ "n_tokens": 3417
	139	+ },
	140	+ {
	141	+ "policy": "entropy",
	142	+ "threshold": 2.0,
	143	+ "budget": 8,
	144	+ "loss": 0.43457157094819804,
	145	+ "edits_per_token": 0.20749195200468246,
	146	+ "extra_ops_ratio": 0.06636437945523596,
	147	+ "n_tokens": 3417
	148	+ },
	149	+ {
	150	+ "policy": "norm",
	151	+ "threshold": 0.02,
	152	+ "budget": 8,
	153	+ "loss": 0.4431566372928056,
	154	+ "edits_per_token": 7.994146912496342,
	155	+ "extra_ops_ratio": 2.576118717076383,
	156	+ "n_tokens": 3417
	157	+ },
	158	+ {
	159	+ "policy": "norm",
	160	+ "threshold": 0.05,
	161	+ "budget": 8,
	162	+ "loss": 0.4419520246658124,
	163	+ "edits_per_token": 7.054141059408838,
	164	+ "extra_ops_ratio": 2.2736114108984604,
	165	+ "n_tokens": 3417
	166	+ },
	167	+ {
	168	+ "policy": "norm",
	169	+ "threshold": 0.1,
	170	+ "budget": 8,
	171	+ "loss": 0.4397261778374042,
	172	+ "edits_per_token": 4.717588527948493,
	173	+ "extra_ops_ratio": 1.5199677263884506,
	174	+ "n_tokens": 3417
	175	+ },
	176	+ {
	177	+ "policy": "norm",
	178	+ "threshold": 0.2,
	179	+ "budget": 8,
	180	+ "loss": 0.430901125784258,
	181	+ "edits_per_token": 1.0491659350307287,
	182	+ "extra_ops_ratio": 0.33984092665514054,
	183	+ "n_tokens": 3417
	184	+ },
	185	+ {
	186	+ "policy": "controller_locked",
	187	+ "lam": 0.3,
	188	+ "kap": 0.5,
	189	+ "budget": 8,
	190	+ "loss": 0.43332059026845965,
	191	+ "edits_per_token": 0.06613988879133743,
	192	+ "extra_ops_ratio": 0.01925461050499103,
	193	+ "n_tokens": 3417
	194	+ },
	195	+ {
	196	+ "policy": "controller",
	197	+ "lam": 0.0,
	198	+ "kap": 0.5,
	199	+ "budget": 8,
	200	+ "loss": 0.4392198074760582,
	201	+ "edits_per_token": 1.4641498390400935,
	202	+ "extra_ops_ratio": 0.48109084149707576,
	203	+ "n_tokens": 3417
	204	+ },
	205	+ {
	206	+ "policy": "controller",
	207	+ "lam": 0.01,
	208	+ "kap": 0.5,
	209	+ "budget": 8,
	210	+ "loss": 0.43936958569757484,
	211	+ "edits_per_token": 1.3075797483172373,
	212	+ "extra_ops_ratio": 0.42456216117182355,
	213	+ "n_tokens": 3417
	214	+ },
	215	+ {
	216	+ "policy": "controller",
	217	+ "lam": 0.03,
	218	+ "kap": 0.5,
	219	+ "budget": 8,
	220	+ "loss": 0.438084998115448,
	221	+ "edits_per_token": 1.0398009950248757,
	222	+ "extra_ops_ratio": 0.3312630400798598,
	223	+ "n_tokens": 3417
	224	+ },
	225	+ {
	226	+ "policy": "controller",
	227	+ "lam": 0.1,
	228	+ "kap": 0.5,
	229	+ "budget": 8,
	230	+ "loss": 0.4354970293991183,
	231	+ "edits_per_token": 0.4433713784021071,
	232	+ "extra_ops_ratio": 0.13166276062654758,
	233	+ "n_tokens": 3417
	234	+ },
	235	+ {
	236	+ "policy": "controller",
	237	+ "lam": 0.3,
	238	+ "kap": 0.5,
	239	+ "budget": 8,
	240	+ "loss": 0.43332059026845965,
	241	+ "edits_per_token": 0.06613988879133743,
	242	+ "extra_ops_ratio": 0.01925461050499103,
	243	+ "n_tokens": 3417
	244	+ },
	245	+ {
	246	+ "policy": "oracle1_greedy",
	247	+ "budget": 1,
	248	+ "diagnostic": true,
	249	+ "loss": 0.4025459464804867,
	250	+ "edits_per_token": 0.5237366003062787,
	251	+ "extra_ops_ratio": 0.19226168855956283,
	252	+ "n_tokens": 653
	253	+ },
	254	+ {
	255	+ "policy": "oracle1_greedy",
	256	+ "budget": 2,
	257	+ "diagnostic": true,
	258	+ "loss": 0.390496985653093,
	259	+ "edits_per_token": 0.9509954058192955,
	260	+ "extra_ops_ratio": 0.34655941646277993,
	261	+ "n_tokens": 653
	262	+ },
	263	+ {
	264	+ "policy": "oracle1_greedy",
	265	+ "budget": 4,
	266	+ "diagnostic": true,
	267	+ "loss": 0.37995706181431255,
	268	+ "edits_per_token": 1.5313935681470139,
	269	+ "extra_ops_ratio": 0.552776494398599,
	270	+ "n_tokens": 653
	271	+ },
	272	+ {
	273	+ "policy": "oracle1_greedy",
	274	+ "budget": 8,
	275	+ "diagnostic": true,
	276	+ "loss": 0.3665898426751083,
	277	+ "edits_per_token": 2.2664624808575806,
	278	+ "extra_ops_ratio": 0.8196839220125861,
	279	+ "n_tokens": 653
	280	+ },
	281	+ {
	282	+ "policy": "oracle_two_step",
	283	+ "diagnostic": true,
	284	+ "one_step_gain": 0.023339285247627347,
	285	+ "two_step_gain": 0.03395615775009681,
	286	+ "greedy_then_best_gain": 0.03374234835306803,
	287	+ "n_tokens": 174
	288	+ }
	289	+ ]
	290	+}
		\ No newline at end of file

added results/raw/routing_v2s1.json +290 −0

		`@@ -0,0 +1,290 @@`
	1	+{
	2	+ "config": "configs/experiments/routing_v2s1.yaml",
	3	+ "checkpoint": "experiments/runs/controller_v2s1_seed01_20261030005758/checkpoint",
	4	+ "checkpoint_sha256": "61a4dda3056e6047dd0671e53f2bb195aaa6d90f3fc112cc93fcf80d84ba4198",
	5	+ "split": "dev_iid",
	6	+ "code_commit": "unknown",
	7	+ "device": "mps",
	8	+ "seconds": 50.9,
	9	+ "records": [
	10	+ {
	11	+ "policy": "budget0",
	12	+ "loss": 0.4965549189338985,
	13	+ "edits_per_token": 0.0,
	14	+ "extra_ops_ratio": 0.0,
	15	+ "n_tokens": 3417
	16	+ },
	17	+ {
	18	+ "policy": "random_forced",
	19	+ "budget": 1,
	20	+ "loss": 0.48360088745822744,
	21	+ "edits_per_token": 1.0,
	22	+ "extra_ops_ratio": 0.12365152489067972,
	23	+ "n_tokens": 3417
	24	+ },
	25	+ {
	26	+ "policy": "uniform",
	27	+ "budget": 1,
	28	+ "loss": 0.4858294641340131,
	29	+ "edits_per_token": 1.0,
	30	+ "extra_ops_ratio": 0.10679070643245528,
	31	+ "n_tokens": 3417
	32	+ },
	33	+ {
	34	+ "policy": "fixed_pair",
	35	+ "budget": 1,
	36	+ "loss": 0.48118645258036347,
	37	+ "edits_per_token": 1.0,
	38	+ "extra_ops_ratio": 0.14036496061660547,
	39	+ "n_tokens": 3417
	40	+ },
	41	+ {
	42	+ "policy": "random_forced",
	43	+ "budget": 2,
	44	+ "loss": 0.4784298638066668,
	45	+ "edits_per_token": 2.0,
	46	+ "extra_ops_ratio": 0.24747007926514458,
	47	+ "n_tokens": 3417
	48	+ },
	49	+ {
	50	+ "policy": "uniform",
	51	+ "budget": 2,
	52	+ "loss": 0.48122175722956695,
	53	+ "edits_per_token": 2.0,
	54	+ "extra_ops_ratio": 0.21358141286491056,
	55	+ "n_tokens": 3417
	56	+ },
	57	+ {
	58	+ "policy": "fixed_pair",
	59	+ "budget": 2,
	60	+ "loss": 0.47615878792021193,
	61	+ "edits_per_token": 2.0,
	62	+ "extra_ops_ratio": 0.28072992123321094,
	63	+ "n_tokens": 3417
	64	+ },
	65	+ {
	66	+ "policy": "random_forced",
	67	+ "budget": 4,
	68	+ "loss": 0.4737940016007751,
	69	+ "edits_per_token": 4.0,
	70	+ "extra_ops_ratio": 0.4947927651923059,
	71	+ "n_tokens": 3417
	72	+ },
	73	+ {
	74	+ "policy": "uniform",
	75	+ "budget": 4,
	76	+ "loss": 0.4761925692582012,
	77	+ "edits_per_token": 4.0,
	78	+ "extra_ops_ratio": 0.4271628257298211,
	79	+ "n_tokens": 3417
	80	+ },
	81	+ {
	82	+ "policy": "fixed_pair",
	83	+ "budget": 4,
	84	+ "loss": 0.47439178989266384,
	85	+ "edits_per_token": 4.0,
	86	+ "extra_ops_ratio": 0.5614598424664219,
	87	+ "n_tokens": 3417
	88	+ },
	89	+ {
	90	+ "policy": "random_forced",
	91	+ "budget": 8,
	92	+ "loss": 0.479933912027709,
	93	+ "edits_per_token": 8.0,
	94	+ "extra_ops_ratio": 0.9892416248762507,
	95	+ "n_tokens": 3417
	96	+ },
	97	+ {
	98	+ "policy": "uniform",
	99	+ "budget": 8,
	100	+ "loss": 0.4744012490468727,
	101	+ "edits_per_token": 8.0,
	102	+ "extra_ops_ratio": 0.8543256514596422,
	103	+ "n_tokens": 3417
	104	+ },
	105	+ {
	106	+ "policy": "fixed_pair",
	107	+ "budget": 8,
	108	+ "loss": 0.48843245580247313,
	109	+ "edits_per_token": 8.0,
	110	+ "extra_ops_ratio": 1.1229196849328438,
	111	+ "n_tokens": 3417
	112	+ },
	113	+ {
	114	+ "policy": "entropy",
	115	+ "threshold": 0.1,
	116	+ "budget": 8,
	117	+ "loss": 0.47797408758425247,
	118	+ "edits_per_token": 3.0755048287971904,
	119	+ "extra_ops_ratio": 0.3631002193380595,
	120	+ "n_tokens": 3417
	121	+ },
	122	+ {
	123	+ "policy": "entropy",
	124	+ "threshold": 0.5,
	125	+ "budget": 8,
	126	+ "loss": 0.4820516032017146,
	127	+ "edits_per_token": 2.4451273046532047,
	128	+ "extra_ops_ratio": 0.28835356525067246,
	129	+ "n_tokens": 3417
	130	+ },
	131	+ {
	132	+ "policy": "entropy",
	133	+ "threshold": 1.0,
	134	+ "budget": 8,
	135	+ "loss": 0.4866514847855628,
	136	+ "edits_per_token": 1.5993561603745976,
	137	+ "extra_ops_ratio": 0.1888264363224408,
	138	+ "n_tokens": 3417
	139	+ },
	140	+ {
	141	+ "policy": "entropy",
	142	+ "threshold": 2.0,
	143	+ "budget": 8,
	144	+ "loss": 0.4954935511476574,
	145	+ "edits_per_token": 0.4890254609306409,
	146	+ "extra_ops_ratio": 0.05748009566574164,
	147	+ "n_tokens": 3417
	148	+ },
	149	+ {
	150	+ "policy": "norm",
	151	+ "threshold": 0.02,
	152	+ "budget": 8,
	153	+ "loss": 0.47761174995437994,
	154	+ "edits_per_token": 8.0,
	155	+ "extra_ops_ratio": 0.9450654991893417,
	156	+ "n_tokens": 3417
	157	+ },
	158	+ {
	159	+ "policy": "norm",
	160	+ "threshold": 0.05,
	161	+ "budget": 8,
	162	+ "loss": 0.4775530145570902,
	163	+ "edits_per_token": 7.89844893181153,
	164	+ "extra_ops_ratio": 0.9331399762982823,
	165	+ "n_tokens": 3417
	166	+ },
	167	+ {
	168	+ "policy": "norm",
	169	+ "threshold": 0.1,
	170	+ "budget": 8,
	171	+ "loss": 0.4774494416747904,
	172	+ "edits_per_token": 5.593210418495756,
	173	+ "extra_ops_ratio": 0.6607864040500908,
	174	+ "n_tokens": 3417
	175	+ },
	176	+ {
	177	+ "policy": "norm",
	178	+ "threshold": 0.2,
	179	+ "budget": 8,
	180	+ "loss": 0.484189420057592,
	181	+ "edits_per_token": 1.0523851331577407,
	182	+ "extra_ops_ratio": 0.12498143781930479,
	183	+ "n_tokens": 3417
	184	+ },
	185	+ {
	186	+ "policy": "controller_locked",
	187	+ "lam": 0.3,
	188	+ "kap": 0.5,
	189	+ "budget": 8,
	190	+ "loss": 0.4950156743532738,
	191	+ "edits_per_token": 0.18027509511267192,
	192	+ "extra_ops_ratio": 0.019261529109336355,
	193	+ "n_tokens": 3417
	194	+ },
	195	+ {
	196	+ "policy": "controller",
	197	+ "lam": 0.0,
	198	+ "kap": 0.5,
	199	+ "budget": 8,
	200	+ "loss": 0.4933141067009207,
	201	+ "edits_per_token": 2.0714076675446296,
	202	+ "extra_ops_ratio": 0.24507358529814022,
	203	+ "n_tokens": 3417
	204	+ },
	205	+ {
	206	+ "policy": "controller",
	207	+ "lam": 0.01,
	208	+ "kap": 0.5,
	209	+ "budget": 8,
	210	+ "loss": 0.4924799355631653,
	211	+ "edits_per_token": 1.9534679543459175,
	212	+ "extra_ops_ratio": 0.23033672708986375,
	213	+ "n_tokens": 3417
	214	+ },
	215	+ {
	216	+ "policy": "controller",
	217	+ "lam": 0.03,
	218	+ "kap": 0.5,
	219	+ "budget": 8,
	220	+ "loss": 0.49286034918682664,
	221	+ "edits_per_token": 1.748902546093064,
	222	+ "extra_ops_ratio": 0.2050815436586777,
	223	+ "n_tokens": 3417
	224	+ },
	225	+ {
	226	+ "policy": "controller",
	227	+ "lam": 0.1,
	228	+ "kap": 0.5,
	229	+ "budget": 8,
	230	+ "loss": 0.4940115225604657,
	231	+ "edits_per_token": 1.0570676031606672,
	232	+ "extra_ops_ratio": 0.12090272048010196,
	233	+ "n_tokens": 3417
	234	+ },
	235	+ {
	236	+ "policy": "controller",
	237	+ "lam": 0.3,
	238	+ "kap": 0.5,
	239	+ "budget": 8,
	240	+ "loss": 0.4950156743532738,
	241	+ "edits_per_token": 0.18027509511267192,
	242	+ "extra_ops_ratio": 0.019261529109336355,
	243	+ "n_tokens": 3417
	244	+ },
	245	+ {
	246	+ "policy": "oracle1_greedy",
	247	+ "budget": 1,
	248	+ "diagnostic": true,
	249	+ "loss": 0.48106685130190885,
	250	+ "edits_per_token": 0.4333843797856049,
	251	+ "extra_ops_ratio": 0.05939235219911631,
	252	+ "n_tokens": 653
	253	+ },
	254	+ {
	255	+ "policy": "oracle1_greedy",
	256	+ "budget": 2,
	257	+ "diagnostic": true,
	258	+ "loss": 0.4616401316013307,
	259	+ "edits_per_token": 0.7978560490045942,
	260	+ "extra_ops_ratio": 0.10911176690281259,
	261	+ "n_tokens": 653
	262	+ },
	263	+ {
	264	+ "policy": "oracle1_greedy",
	265	+ "budget": 4,
	266	+ "diagnostic": true,
	267	+ "loss": 0.4398064138695803,
	268	+ "edits_per_token": 1.3905053598774886,
	269	+ "extra_ops_ratio": 0.18839140069795057,
	270	+ "n_tokens": 653
	271	+ },
	272	+ {
	273	+ "policy": "oracle1_greedy",
	274	+ "budget": 8,
	275	+ "diagnostic": true,
	276	+ "loss": 0.4197931946875673,
	277	+ "edits_per_token": 2.1623277182235836,
	278	+ "extra_ops_ratio": 0.2941060263383991,
	279	+ "n_tokens": 653
	280	+ },
	281	+ {
	282	+ "policy": "oracle_two_step",
	283	+ "diagnostic": true,
	284	+ "one_step_gain": 0.018501395466683924,
	285	+ "two_step_gain": 0.03580966489068393,
	286	+ "greedy_then_best_gain": 0.03554030122428105,
	287	+ "n_tokens": 174
	288	+ }
	289	+ ]
	290	+}
		\ No newline at end of file

added scripts/statistics.py +59 −0

		`@@ -0,0 +1,59 @@`
	1	+#!/usr/bin/env python
	2	+"""Pre-registered uncertainty analysis: paired item-level bootstrap (10,000 resamples) of exact-match differences
	3	+within each seed, and t-intervals over seeds. Writes results/analyzed/statistics_v2.json and paper/tables/stats.tex."""
	4	+import glob, json, sys
	5	+from pathlib import Path
	6	+import numpy as np
	7	+sys.path.insert(0, ".")
	8	+from evaluation.statistical_analysis import paired_bootstrap, seed_interval
	9	+
	10	+def load(pre):
	11	+ out = []
	12	+ for run in sorted(glob.glob(f"experiments/runs/{pre}_seed*")):
	13	+ if json.load(open(Path(run) / "manifest.json")).get("state") != "completed":
	14	+ continue
	15	+ recs = {}
	16	+ for b in (0, 1, 2, 8):
	17	+ p = Path(run) / f"em_dev_iid_b{b}.json"
	18	+ if p.exists():
	19	+ recs[b] = {r["uid"]: r["correct"] for r in json.load(open(p))}
	20	+ out.append((Path(run).name, recs))
	21	+ return out
	22	+
	23	+res = {}
	24	+systems = {"prisme": load("p4v2_prisme"), "gr": load("p4v2_gr"), "dense": load("p4v2_dense")}
	25	+for name, runs in systems.items():
	26	+ for b in (1, 2, 8):
	27	+ diffs, boots = [], []
	28	+ for rid, recs in runs:
	29	+ if b in recs and 0 in recs:
	30	+ common = sorted(set(recs[0]) & set(recs[b]))
	31	+ a = np.array([recs[b][u] for u in common]); c = np.array([recs[0][u] for u in common])
	32	+ bs = paired_bootstrap(a, c); bs["run_id"] = rid; boots.append(bs); diffs.append(bs["diff"])
	33	+ if diffs:
	34	+ res[f"{name}_b{b}_vs_b0"] = {"per_seed": boots, "seeds": seed_interval(diffs)}
	35	+# PRISME vs GR at the same budget (paired by item, per seed index)
	36	+for b in (1, 2, 8):
	37	+ diffs, boots = [], []
	38	+ for (rp, rec_p), (rg, rec_g) in zip(systems["prisme"], systems["gr"]):
	39	+ if b in rec_p and b in rec_g:
	40	+ common = sorted(set(rec_p[b]) & set(rec_g[b]))
	41	+ a = np.array([rec_p[b][u] for u in common]); c = np.array([rec_g[b][u] for u in common])
	42	+ bs = paired_bootstrap(a, c); bs["run_ids"] = [rp, rg]; boots.append(bs); diffs.append(bs["diff"])
	43	+ if diffs:
	44	+ res[f"prisme_minus_gr_b{b}"] = {"per_seed": boots, "seeds": seed_interval(diffs)}
	45	+Path("results/analyzed").mkdir(exist_ok=True, parents=True)
	46	+Path("results/analyzed/statistics_v2.json").write_text(json.dumps(res, indent=1))
	47	+rows = []
	48	+for k, v in res.items():
	49	+ s = v["seeds"]; lo = min(b["ci_low"] for b in v["per_seed"]); hi = max(b["ci_high"] for b in v["per_seed"])
	50	+ rows.append(f"{k.replace('_', ' ')} & {s['n']} & {100s['mean']:+.2f} & [{100s['ci_low']:+.2f}, {100s['ci_high']:+.2f}] & [{100lo:+.2f}, {100*hi:+.2f}]\\\\")
	51	+ print(k, "seeds", s["n"], f"mean {100s['mean']:+.2f} pp, t-CI [{100s['ci_low']:+.2f}, {100*s['ci_high']:+.2f}]")
	52	+Path("paper/tables/stats.tex").write_text(r"""\begin{table}[t]\centering\small
	53	+\caption{Pre-registered uncertainty analysis of dev-iid exact match (percentage points). ``t-CI'' is the 95\,\%
	54	+$t$-interval over independent adapter seeds of the per-seed paired difference; ``item CI range'' spans the
	55	+smallest lower and largest upper bound of the per-seed paired item bootstraps (10,000 resamples, 1,000 items).}
	56	+\label{tab:stats}
	57	+\begin{tabular}{lrrll}\toprule
	58	+Comparison & seeds & mean & t-CI over seeds & item CI range\\\midrule
	59	+""" + "\n".join(rows) + "\n" + r"\bottomrule\end{tabular}\end{table}" + "\n")
	60

spb/wp31_uqo

WP31: 5-seed routing summary, statistics, ranking controller result

Theme